Bayesian Spam Filtering | Jason Palmer, CPA, CITP

Amazing, as it seems, some Internet email providers do not offer an Antispam service for filtering out Unsolicited Commercial Email (UCE) – the proper name for what is affectionately called “SPAM” or Junk email.

Some email client software such as Microsoft Outlook and Mozilla Thunderbird include their own Antispam filters but you may want something more robust. Many of the Antivirus software vendors in their “Internet Security Suite” products include an Antispam component. For the most part, the Antispam component, like the rest of the Suite is “set it and forget it.” However, since no automated process is perfect at detecting UCE, most usually have the ability to create whitelists (always accept) and blacklists (always deny) specific senders. Many dedicated desktop Antispam solutions exist as well and some are listed below.

Antispam filters use a combination of the following techniques to differentiate between legitimate email and UCE. Some use a form of Heuristic pattern matching. The filter looks for a combination of known phrases used in UCE messages such as the ever popular “In deepest confidence” and “the sum of X million dollars” and “need your assistance.” These may not be the actual phrases tested but they demonstrate the concept of the type of language used in the classic scam email of someone contacting you to assist with the movement of money in/out of the country if you will just show good faith with money of your own. In all cases, the phrases are scored with either with positive (more likely spam) or negative (more likely legitimate) and the net number determines if the message is allowed through or moves to your junk folder.

Another technique is the straight automatic blocking of messages that originate from specific IP (Internet) addresses and senders that are known to be bulk Spammers. The Antispam program will check with a well-known service such as Spamhaus.org or the DNS Black List, which maintain a continuously updated list of known originators of Spam and act accordingly.

Many will apply Bayesian content filtering which is a content filtering technique that looks at the words in the body of the message, the email message headers (detailed information about the sender and the path the message took to be delivered to your IN box), the amount of HTML code (colors and graphics), word pairs, phrases, and the general location and context of the words and phrases and assigns a score that determines if the message is or is not UCE/Spam. What makes Bayesian content filtering reasonably successful is that the initial analysis of UCE/Spam is from a pool of email that you personally classify as UCE/Spam. In this way, the program knows what you deem as UCE/Spam so it can analyze the messages received and score them appropriately as UCE/Spam. At the same time, the Bayesian content filters also look at known good non-spam email to create similar scores as a basis of comparison.

Technical NOTE: Bayesian filters work best against a pool of homogeneous mail for a single person or single company. Since the scoring is based on a large population and the algorithm is looking for patterns and trends, Bayesian filters break down when Good email can be confused with Bad email. Let us assume that a husband who is an Accountant and wife who is a Doctor share the same family email account address. The wife may receive a large number of email messages from Big Pharma that discuss well know drugs such as Celebrex or Viagra. The Bayesian filter can get easily confused because the husband might classify all Pharma email as UCE/Spam when in fact it is legitimate to the wife who is the Doctor. But how is the filter to tell the difference between an offer to purchase Celebrex (illegally) over the Internet and a legitimate email from Pfizer the makers of Celebrex? The answer is the Bayesian filters usually goof.

The last method I will discuss is called Challenge-Response, which maintains a list of permitted senders. Every time you receive an email, if the sender is not already whitelisted (permitted), the Antispam Component will send an automatic auto-reply to the sender and ask them to visit a web site to enter in a “challenge” like two plus two equals (fill in the blank) or some other simple test that verifies that the email was sent by a human. If there is no response, as would be the case from a list server (vendor mailing list program), then the message is placed in the quarantine or junk folder for later review by you. The use of Challenge-Response, although extremely reliable, can be problematic as every Challenge email sent out, if sent to a sender that was a fake address, will just bounce back and create even more mail traffic.

You can find extensive in-depth details about the above techniques and the more advanced ones by searching out “Antispam Filtering Techniques” in your favorite search engine.

If your Internet Service Provider, email host, or email client do not filter for UCE/Spam or you want a more robust solution at the individual level, consider the “Internet Security Suites” or Dedicated packages from the well known providers below.

Popular Internet Security Suites

NOTE: These are the Consumer Product Listings – Equivalents Exist for Business

Vipre Internet Security
http://www.vipreantivirus.com/VIPRE-Internet-Security/

Kaspersky Internet Security
http://usa.kaspersky.com/products-services/home-computer-security/internet-security

Trend Micro Titanium Internet Security
http://www.trendmicro.com/us/home/products/titanium/internet-security/index.html

McAfee Internet Security
http://home.mcafee.com/store/internet-security

Norton Internet Security
http://us.norton.com/internet-security/

Dedicated Antispam Solutions

MailFrontier Desktop
http://www.mailfrontier.com/products_matador.html

Cloudmark DesktopOne
http://www.cloudmark.com/en/products/cloudmark-desktopone/index

SpamFighter
http://www.spamfighter.com/Product_Info.asp

Mailshell
http://www.mailshell.com/mail/client/oem2.html/step/client