markwilson.it

It may just be a co-incidence, but since I switched my e-mail from my ISP’s servers to my own server a few months back, I’ve been seeing a huge increase in the amount of unsolicited commercial e-mail (UCE) – commonly known as spam – in my mailbox.

At the time of writing, statistics from MessageLabs show a decline in the volumes of spam over the last 12 months (although they still indicate that 58.39% of all e-mails sent were spam). Postini’s statistics suggest that 73% of e-mail is spam.

If you think those statistics are bad, according to Microsoft, Bill Gates receives 4 million spam messages a day, making him probably the most spammed man in the world (it’s no surprise then that he is rumoured to have his own mail server at Microsoft).

Any effective strategy for dealing with UCE (specifically for Exchange Server 2003, but the generic advice is the same for all mail servers) needs to operate a multiple levels within the e-mail transport (these are defined on the Message Delivery Settings under Global Settings in Exchange System Manager but need to be imposed using the properties for each SMTP virtual server):

• Server-level accept/deny lists can be used to always accept, or always deny, messages from certain domains. The trouble with this method of trapping e-mail is that I occasionally receive non-delivery reports (NDRs) for messages that were allegedly sent from markwilson.co.uk but that actually never came near my servers, so without a real-time DNS lookup mechanism to verify the sender’s domain (such as Sender ID), these are of limited use.
• Connection filtering using real-time block lists (RBLs) is the next level of protection, using a DNS query against a RBL provider’s servers, such as the SpamHaus project.
• Sender filtering can be used to drop any messages that claim to come from a particular e-mail address, optionally archiving them.
• Recipient filtering is a method of rejecting certain e-mail addresses (e.g. for people who have left the organisation, or for non-existent addresses). One option is to filter messages for recipients who are not in the directory; however this can leave an organisation open to a directory harvest attack as the server gives different responses for valid and invalid addresses. To avoid such attacks, a “tarpit” (see Microsoft knowledge base article 842851) can be employed, to delay responses to bad addresses by a few seconds, slowing down any directory harvest attacks significantly (it would normally be possible to harvest all four-character address combinations within a few minutes – with a 5 second tarpit delay this is increased to a couple of months – and most addresses have much longer aliases than 4 characters).
• Finally, the intelligent message filter (IMF – previously a separate download but now included with Exchange Server service pack 2) employs a Microsoft-proprietary algorithm (SmartScreen) to scan each message and mark it with a spam confidence level (SCL), which is then used to process the mail accordingly at the gateway or mailbox level.

Each of these tools filters out less obvious types of UCE with increasing levels of cost in terms of server resource. Whilst the junk e-mail filters in Outlook 2003/2007 and Entourage 2004, which are also based on SmartScreen but doesn’t use the SCL mechanism, are pretty good at filtering messages, they are far from perfect (in my experience, Outlook seems to be better at this than Entourage). Activating the IMF on my server has provided an additional level of filtering which has greatly reduced the volume of UCE making it through as far as my mailbox.

The IMF uses 11 SCL ratings, set as an attribute in the message header:

• -1 is used for messages submitted internally with an authenticated connection – eliminating false positives for internal e-mail.
• 0 is used for messages that are marked as not spam.
• 1-9 are used to highlight varying levels of probability that a message is spam (9 being the most likely).

Within Exchange, the SCL value can be used to filter UCE on gateway servers as well as with a lower level SCL used by the information store to move messages to the user’s junk e-mail folder – therefore allowing for the most obvious UCE to be trapped at the gateway (least chance of false positives) and for users to retrieve any messages in the mid-range that are incorrectly marked as junk. The gateway blocking action is also configurable – with options for archival, deletion (without NDR), no action, or rejection.

Archived messages will be saved (by default) to %programfiles%\Exchsrvr\Mailroot\vsi 1\UCEArchive. Each message is archived as an .EML file, which can be viewed with a text editor. To resubmit a message for delivery it can simply be moved to the corresponding %programfiles%\Exchsrvr\Mailroot\vsi 1\Pickup folder. Obviously, viewing individual messages in a text file is time-consuming and the IMF Archive Manager is a great tool for managing IMF-archived messages.

The SCL at which to block messages for a particular organisation will vary according to the profile of e-mail sent to/from the organisation – I have my SCL level for gateway blocking set to 7 with archiving enabled and so far I have only had one false positive – but clearly for organisations receiving more e-mail than I do, this will be a bigger issue! At the store level (set to move messages with an SCL greater than 4) things are not working quite so well but that is to be expected as in the grey area between good and bad mail, some legitimate (good) messages will inevitably get marked with the same SCL as the (bad) UCE. It’s worth noting that marking a sender as safe in Outlook will only override the SCL at the mailbox-level – it has no effect at the gateway.

To assist in judging the SCL levels to use for filtering, it is possible to expose the SCL in Outlook and in Outlook Web Access (OWA). Also useful may be (temporarily) enabling diagnostic logging on the MSExchangeTransport\SMTP Protocol for a server, such that SMTP events are logged. Performance monitor counters from the MSExchange Intelligent Message Filter object can also be used to log the amount of spam filtered or acted upon, the relative SCL levels and overall IMF performance. Based on the performance monitor data, the IMF gateway blocking configuration can be reduced from no action to archive, and then finally (once confident that the levels are correct) to delete, as the appropriate SCL levels are determined.

It’s also possible to mark the SCL on archived messages by creating an new registry key called ContentFilter at HKEY_LOCAL_MACHINE\Software\Microsoft\Exchange\ and a corresponding DWORD value named ArchiveSCL set to 1. A string value named ArchiveDir can also be used to change the archive folder. Both of these settings are detailed in the Microsoft Exchange Server TechCenter along with details for applying the IMF to trusted (authenticated) connections and increasing the size limit for the rule used to process spam at mailbox level (allowing more blocked and safe senders).

Suggested further reading
IMF release notes (Microsoft knowledge base article 867633).
Microsoft Exchange Team Blog.