Controlling spam using the Microsoft Exchange intelligent message filter

Posted on By Mark Wilson

It may just be a co-incidence, but since I switched my e-mail from my ISP’s servers to my own server a few months back, I’ve been seeing a huge increase in the amount of unsolicited commercial e-mail (UCE) – commonly known as spam – in my mailbox.

At the time of writing, statistics from MessageLabs show a decline in the volumes of spam over the last 12 months (although they still indicate that 58.39% of all e-mails sent were spam). Postini’s statistics suggest that 73% of e-mail is spam.

If you think those statistics are bad, according to Microsoft, Bill Gates receives 4 million spam messages a day, making him probably the most spammed man in the world (it’s no surprise then that he is rumoured to have his own mail server at Microsoft).

Any effective strategy for dealing with UCE (specifically for Exchange Server 2003, but the generic advice is the same for all mail servers) needs to operate a multiple levels within the e-mail transport (these are defined on the Message Delivery Settings under Global Settings in Exchange System Manager but need to be imposed using the properties for each SMTP virtual server):

  • Server-level accept/deny lists can be used to always accept, or always deny, messages from certain domains. The trouble with this method of trapping e-mail is that I occasionally receive non-delivery reports (NDRs) for messages that were allegedly sent from markwilson.co.uk but that actually never came near my servers, so without a real-time DNS lookup mechanism to verify the sender’s domain (such as Sender ID), these are of limited use.
  • Connection filtering using real-time block lists (RBLs) is the next level of protection, using a DNS query against a RBL provider’s servers, such as the SpamHaus project.
  • Sender filtering can be used to drop any messages that claim to come from a particular e-mail address, optionally archiving them.
  • Recipient filtering is a method of rejecting certain e-mail addresses (e.g. for people who have left the organisation, or for non-existent addresses). One option is to filter messages for recipients who are not in the directory; however this can leave an organisation open to a directory harvest attack as the server gives different responses for valid and invalid addresses. To avoid such attacks, a “tarpit” (see Microsoft knowledge base article 842851) can be employed, to delay responses to bad addresses by a few seconds, slowing down any directory harvest attacks significantly (it would normally be possible to harvest all four-character address combinations within a few minutes – with a 5 second tarpit delay this is increased to a couple of months – and most addresses have much longer aliases than 4 characters).
  • Finally, the intelligent message filter (IMF – previously a separate download but now included with Exchange Server service pack 2) employs a Microsoft-proprietary algorithm (SmartScreen) to scan each message and mark it with a spam confidence level (SCL), which is then used to process the mail accordingly at the gateway or mailbox level.

Each of these tools filters out less obvious types of UCE with increasing levels of cost in terms of server resource. Whilst the junk e-mail filters in Outlook 2003/2007 and Entourage 2004, which are also based on SmartScreen but doesn’t use the SCL mechanism, are pretty good at filtering messages, they are far from perfect (in my experience, Outlook seems to be better at this than Entourage). Activating the IMF on my server has provided an additional level of filtering which has greatly reduced the volume of UCE making it through as far as my mailbox.

The IMF uses 11 SCL ratings, set as an attribute in the message header:

  • -1 is used for messages submitted internally with an authenticated connection – eliminating false positives for internal e-mail.
  • 0 is used for messages that are marked as not spam.
  • 1-9 are used to highlight varying levels of probability that a message is spam (9 being the most likely).

Within Exchange, the SCL value can be used to filter UCE on gateway servers as well as with a lower level SCL used by the information store to move messages to the user’s junk e-mail folder – therefore allowing for the most obvious UCE to be trapped at the gateway (least chance of false positives) and for users to retrieve any messages in the mid-range that are incorrectly marked as junk. The gateway blocking action is also configurable – with options for archival, deletion (without NDR), no action, or rejection.

Archived messages will be saved (by default) to %programfiles%\Exchsrvr\Mailroot\vsi 1\UCEArchive. Each message is archived as an .EML file, which can be viewed with a text editor. To resubmit a message for delivery it can simply be moved to the corresponding %programfiles%\Exchsrvr\Mailroot\vsi 1\Pickup folder. Obviously, viewing individual messages in a text file is time-consuming and the IMF Archive Manager is a great tool for managing IMF-archived messages.

The SCL at which to block messages for a particular organisation will vary according to the profile of e-mail sent to/from the organisation – I have my SCL level for gateway blocking set to 7 with archiving enabled and so far I have only had one false positive – but clearly for organisations receiving more e-mail than I do, this will be a bigger issue! At the store level (set to move messages with an SCL greater than 4) things are not working quite so well but that is to be expected as in the grey area between good and bad mail, some legitimate (good) messages will inevitably get marked with the same SCL as the (bad) UCE. It’s worth noting that marking a sender as safe in Outlook will only override the SCL at the mailbox-level – it has no effect at the gateway.

To assist in judging the SCL levels to use for filtering, it is possible to expose the SCL in Outlook and in Outlook Web Access (OWA). Also useful may be (temporarily) enabling diagnostic logging on the MSExchangeTransport\SMTP Protocol for a server, such that SMTP events are logged. Performance monitor counters from the MSExchange Intelligent Message Filter object can also be used to log the amount of spam filtered or acted upon, the relative SCL levels and overall IMF performance. Based on the performance monitor data, the IMF gateway blocking configuration can be reduced from no action to archive, and then finally (once confident that the levels are correct) to delete, as the appropriate SCL levels are determined.

It’s also possible to mark the SCL on archived messages by creating an new registry key called ContentFilter at HKEY_LOCAL_MACHINE\Software\Microsoft\Exchange\ and a corresponding DWORD value named ArchiveSCL set to 1. A string value named ArchiveDir can also be used to change the archive folder. Both of these settings are detailed in the Microsoft Exchange Server TechCenter along with details for applying the IMF to trusted (authenticated) connections and increasing the size limit for the rule used to process spam at mailbox level (allowing more blocked and safe senders).

Suggested further reading
IMF release notes (Microsoft knowledge base article 867633).
Microsoft Exchange Team Blog.

Creating a customised Windows XP CD using nLite

Posted on By Mark Wilson

Last night, when I was installing Windows on my Mac, I needed a Windows XP CD with service pack 2 included (i.e. a slipstreamed service pack as Apple Boot Camp doesn’t allow the use of a non-SP2 CD). I didn’t have one – only a Windows XP (RTM) CD, an integrated SP1 CD, and an SP2 update CD – but that’s no problem, as you can create your own slipstreamed XP SP2 CD.

The official method linked above works well, but (as highlighted in the August 2006 edition of Personal Computer World magazine) there is an easier way – using the excellent (and free) nLite deployment tool for unattended Windows. After copying the contents of my original Windows XP (RTM) CD to a temporary location on my hard disk, I was able to use nLite to integrate the service pack (from my SP2 CD) and make a bootable .ISO image of the new distribution (ready for burning to CD using the software of my choice) using just a few mouse clicks. I could also have integrated drivers (e.g. the ones from the Macintosh driver CD that Boot Camp creates), included updates/patches, removed components, applied tweaks and generally customised the Windows XP installation to suit – all using one simple wizard.

Thanks to Dino Nuhagic (Nuhi) for creating nLite (and for making it free) – it really is a very useful tool.

Installing Windows on my Mac

Posted on By Mark Wilson

Tonight, I committed heresy – I installed Windows on my Mac.

Ironically (and this is where I need to eat a small amount of humble pie, having previously criticised the OS X interface – although I did also say that I don’t like the new Windows Vista Aero interface or KDE), when I bought my Intel-based Mac the intention was to run Windows but then I decided to give Mac OS X a spin and I quite like it. There is a big caveat though – most Mac users zealots will say that once you switch you’ll never want to go back and I don’t fall into that camp. I now run Windows XP SP2, Windows Server 2003 SP1, Windows Vista beta 2, Mac OS X 10.4.7 and SUSE Linux 10 on my various machines (some virtual, some physical) and each has it’s place. The fact that I can dual-boot between the two that I use for my desktop work is an added bonus.

Although Mac OS X, iLife 06, Microsoft Office 2004 for Macintosh and Microsoft Messenger for Mac provide enough features to cover at least 90% of my daily computing needs, I do still need to use Adobe Photoshop (and that’s not yet optmised for MacIntels) and Microsoft Money 2000(although I’m sure there’s something available for the Mac that I could use instead). I also have legacy (and partially complete) digital video that I edited using Windows Movie Maker 2.0 and I don’t have the time to re-edit it. For that reason, Windows will be on my Mac for a while.

I chose to use Apple Boot Camp (v1.0.2 Beta) – other methods of installing Windows XP on a MacIntel are available – and the rest of this post summarises my experiences of this (relatively straightforward) operation.

The first thing to note is that Boot Camp is currently beta software and although no indication is given of how long it will continue to work for, the licensing agreement does make it clear that use of the software is for a limited time only. It’s also unsupported.

The Boot Camp beta is provided in a disk image file called BootCamp102.dmg. This contains three files:

  • Boot Camp Beta Installation & Setup Guide.pdf
  • BootCampAssistant.pkg
  • Read Before You Install.app

The first of these files is an extremely readable, 17-page, document that describes the basic steps to install and configure Boot Camp; however there are some extra points highlighted below that might be useful.

Firstly, my brand new Mac didn’t have the latest firmware on it. Although Software Update said I was up-to-date from a software perspective, I also needed to download and install Mac mini (early 2006) Firmware Update 1.0.1. This successfully brought my firmware up from MM11.004B.B00 to MM11.0055.B03 but it’s also worth planning for less successful updates. Apple’s advice for dealing with failed firmware upgrades requires the Firmware Restoration CD v1.0. As this is supplied in an Apple disk image (.DMG) file, it’s probably worth burning a copy before attempting to upgrade the firmware on your Mac (unless you have another Mac available – .DMG files aren’t much help if you have blown up your Mac and need to download/burn a CD using another operating system).

Once all the prerequisites have been met, running the Boot Camp Assistant is straightforward enough, guiding the operator through the process of creating a Macintosh Drivers CD and creating a disk partition for Windows; however before Boot Camp would let me start the Windows XP installation it insisted on restarting the Mac (using the Power button), resulting in an unclean shutdown (which thankfully didn’t cause any major issues later).

The Windows XP installation is just like any other – although I noticed that it detected my external hard disk (I don’t remember any previous Windows installations recognising USB-attached drives but I may be wrong – I’ve done so many over the years that I probably don’t notice any more). I followed Apple’s advice and installed Windows on the third partition on my internal hard disk (C:) and formatted the disk using NTFS. One downside of the installation is that because the drivers for the Marvell Yukon 88E8053 PCI-E Gigabit Ethernet Controller are not present within the Windows media, there was no network available during installation to join a domain – not a problem as I could install in workgroup mode and join the domain later.

Windows XP installation on an Intel Mac Mini

After installing the Macintosh drivers and software (with one reboot required part-way through), everything was looking good; however beware that there are three unrecognised devices shown in Device Manager:

  • USB Human Interface Device (USB\VID_05AC&PID_8240\5&12F9C752&0&2).
  • PCI Device (PCI\VEN_8086&DEV_27A3&SUBSYS_00000000&REV_03\3&B1BFB68&0&38).
  • Unknown Device (ACPI\IFX0101\1).

Apple does point out that certain devices are not supported under Windows XP and for the Mac Mini that includes the Apple Remote – I suspect that’s the USB device. At the time of writing, Craig Hart’s PCI and AGP vendors, devices and subsystems identification file doesn’t recognise the PCI device although the vendor class is Intel. The ACPI device is a mystery.

I also found that the headphone socket doesn’t mute the internal speakers when running Windows (it’s fine with Mac OS X) but I can live with that.

Having installed Windows there was some basic housekeeping to be done: join my Active Directory domain (to pick up group policy for Windows updates); install anti-virus software; label the Windows partition to give it a sensible name; and set the default operating system to be Mac OS X. Finally, I installed MacDrive v6.1.4 to allow read/write access from Windows to the external hard disk that holds my data files and is formatted as Mac OS Extended (Journalled) (I previously found the 4GB file size limit with FAT32 to be too restrictive).

So that’s it. After months of talking about it, I finally have Windows running on a Mac – albeit not the Media Center Edition, and without the use of my remote control.

(My digital) life is good.

Problems connecting to Windows Server 2003 shares from within MacOS X

Posted on By Mark Wilson

Although I’ve been connecting to Windows XP clients with no issues, each time I attempted to connect to my Windows Server 2003 (SP1) server from the Finder in MacOS X 10.4.7, I was greeted with the following message:

The alias servername could not be opened because the original item cannot be found.

There was nothing wrong with the alias (it was created automatically by OS X when browsing the network) but, as Drew McLellan outlines in his blog, the issue turns out to be related to digitally-signed SMB traffic, which must be disabled.

Strangely, the option to digitally sign communications (if client agrees) didn’t seem to make any difference, so it really is necessary to disable digitally signed communications (always). Although it would seem logical to make the change via Group Policy, this is a computer setting (so is not applied to a user account) and as Macs are not domain members they are not affected by group policy either (although the policy for the target server could be set at domain level)

Beware that if editing local policies, these are overridden by site and domain-level policies; however in this case, it’s probably best to make the change only on those servers to which access is required from a computer that doesn’t support SMB signing as the need for digitally signed communications is intended to prevent man-in-the-middle attacks from occuring and disabling this represents a security risk. Further details can be found in the Microsoft Windows Server TechCenter.

Warning – buy your upgrades when you buy your Mac

Posted on By Mark Wilson

A few weeks back, I bought a Mac Mini. Because I wanted it shipped immediately (and because the upgrade prices sounded a bit steep), I stuck with the standard 80GB hard disk and 512MB of RAM and now I’m finding performance to be a little sluggish – I suspect that’s due to a lack of memory.

When I ordered the Mac, the cost of specifying 2x1GB 667MHz DDR2 SDRAM SODIMMs instead of 2x256MB was £210.01. Likewise, to take the SATA hard disk from 80GB to 120GB would cost £89.99. Those are (very) high prices for standard PC components but nothing compared to the quote I just had from the Apple Store for 2GB of RAM (with “free” installation) – over £420! Mac:Upgrades can do a similar deal (but not while I wait) for around £325 but when I look at the memory prices using the Crucial Memory Advisor Tool to I get two options that will work for me, each at a much lower price:

  1. I could drop one of my 256MB SODIMMS and replace it with a 1GB module, giving me a total of 1.25GB for just £98.69.
  2. Alternatively, I could take out all of the existing memory and add a 2GB kit (2x1GB of matched memory) for £186.81.

…so, I guess there will be bits of MacIntel all over my desk in a few days time…

Crucial recommend the matched pair option for reasons of performance (Apple say it allows memory interleaving), and if I’m going to open up my Mac (which looks to be a delicate operation) then I’d rather only do it once – that means option 2, which is only a few pounds less than the original upgrade would have been (although I will have 512MB of spare memory afterwards).

In all, for the sake of my warranty (and sanity), it looks as if the best option would have been to specify extra RAM at the time of purchase, but I guess if I do wreck the machine in the process of upgrading, the cost of replacing it is not much more than Apple would charge me for 2GB of RAM!

Rumour has it that the new Intel Core 2 Duo processors are socket compatible with my Core Duo (and quad core chips should be available by the end of the year) so a return to the operating table for a processor upgrade is a distinct possibility for the future.

How safe is your personal information?

Posted on By Mark Wilson

I recently wrote about why I’m cautious of all the hype surrounding what has become known as Web 2.0. One of my major concerns related to data security is that if my data is held on someone else’s servers, how can I control what it is being used for? Well, last week, back in the Web 1.0 world I experienced exactly the kind of issue which just underlines these concerns, when my ISP accidentally sent my account information to 1800 customers.

The first I knew was an e-mail from the Marketing Director which read (in part):

“This afternoon, whilst the marketing team was in the process of sending you a Customer Service Update, an email was sent in error to 1,800 customers. The email sent in error contained information relating to your Force9 service.The specific information was: our internal reference number, username, name, product name, subscription amount, Force9 email, alternative email, marketing preference and active status.

No address details, credit card details, payment details or phone numbers have been disclosed.

We have contacted the customers who received your information, asked them to disregard the contents and delete the email.

I would like to apologise. Although this was a result of a regrettable human error, we will be updating our systems and processes immediately to prevent this from ever reoccurring.

Once again, please accept my apologies for any inconvenience this has caused.”

Of course, my ISP should be commended for “‘fessing up” on this one – how many other organisations would have just kept quiet? But the accidental disclosure of information held about me by third parties is not an isolated incident – last year I experienced a similar problem when the Spread Firefox database was compromised. Some protection can be gained when registering with websites by using false details (watch those mandatory fields and think “why do they need my mailing address and telephone number?”); however there are practical reasons why many service providers need to be given real information.

In these days of direct marketing and (even worse) identity fraud, it seems to me that being concerned about the use of your personal details when they are supplied to a service provider is not being paranoid – it’s just common sense.

Apple Stores inside Tesco… not a marriage made in heaven

Posted on By Mark Wilson

My local Tesco contains an Apple Store as a proof of concept with potential to be rolled out nationwide. I just called with a sales enquiry and this is an approximate transcript of the conversation:

Tesco Customer Service: “Hello, Tesco Kingston – how can I help you?”
Me: “Hello, I understand you have a dedicated Apple Store within the store.”
Tesco Customer Service: “Yes sir, we do. Would you like to speak to someone there?”
Me: “Yes please.”

(… short wait …)

Tesco Electrical Department: “Electrical.” (imagine estuary English “am I bovvered” accent)
Me: “Hello, I understand that you have an Apple Store – please can you tell me is that just for new sales or is do you offer upgrades?”
Tesco Electrical Department: “Upgrades – what is that? [like]”
Me: “Can I speak to someone in the Apple Store please?”
Tesco Electrical Department: “There’s no-one here from that section – what did you want to know?”

(Luckily, after I was insistent that I wanted to speak to someone in the Apple Store, they suddenly became available and confirmed that they do not offer an upgrade service).

Let’s look at this proof of concept in a little more detail. Apple is a brand with a tremendous image and huge customer loyalty (albeit with less-than-brilliant technical support) looking to gain an increased market presence. On the other hand, Tesco is known as a supermarket monopolist (accounting for more than £1 in every £8 spent on the high street in the UK – hence the attraction for Apple) and (in my experience) also delivers shocking customer service. Not exactly a marriage made in heaven… at least not if Apple wants to retain its reputation.

Helping spiders to crawl around my bit of the web

Posted on By Mark Wilson

A few months back, I blogged that my Google PageRank had fallen through the floor on certain pages. I was also concerned that the Google index only contained about half the content on my website.

I don’t engage in search engine optimisation, but I have found out a few things which have made a huge difference to both the quality of the site and Google’s ability to find my content – mostly from Gina Trapani’s excellent help the GoogleBot understand your website article (it’s also worth checking out 9 things you can do to make your web site better).

So, what did I do? Well, I read that the GoogleBot can’t read JavaScript – that would account for archive pages that were not being picked up – so I removed the drop-down archive list; however it didn’t seem to make much difference so I’ll be reinstating it again soon. What does seem likely is that my archive page links were appearing too far down the page code (as another theory is that the GoogleBot only follows the first 100 links on a page – actually, that’s one of the Google webmaster guidelines) and a quick look with the Smart IT Consulting GoogleBot spoofer confirmed that the archive links do indeed appear way down the code. I need to rework the site sometime (better CSS and an improved site layout… though goodness knows when I’ll get the time) and when I do, I’ll set the archive links higher up the code (I’ll need to if I want the site to degrade nicely). By far and away the most significant change I made to the site was joining the Google Sitemaps program. I now use the unlimited version of the XML Sitemap Generator to produce a new sitemap each time I write a new post and Google is finding every one of my pages (there is also a free online sitemap generator available).

Today, entering site:markwilson.co.uk as a Google search brings back 626 results (up from around 250 in March) and the webstats also show an increase in the number of site visitors, so forget search engine optimisation – just give the spiders a little bit of help to crawl your site.

Why the BBC should stick to TV programming

Posted on By Mark Wilson

Windows PCs come in for a lot of critism about reliability but most of that is unfounded. You see, it’s not that Windows is particularly bad, but it’s actually down to the sheer number of permutations of hardware and software that are available and quality of the applications that we load on top of Windows (or inside Windows in the case of device drivers).

My wife’s PC is an old Compaq Deskpro EN6350 SFF that I gave her when she set up her public relations consultancy business a few years back. I originally installed Windows 2000 Professional on it before I moved to Australia in 2001. When I returned to the UK the next year I upgraded it to Windows XP, and since then I’ve applied software patches, anti-virus and anti-spyware updates and added a Wireless network adapter. That’s it. And although it’s a bit on the slow side now, it’s fine for Internet access, e-mail and a bit of word processing – which covers 99% of what my wife does with the PC. My point is, that after 5 or 6 years I haven’t had to rebuild Windows, clean out the registry, or do anything else with it, and it generally only needs a reboot when a software update requires that the system is restarted. I have similarly reliable PCs of my own – basically well maintained, with nothing that’s likely to upset system stability – and it all works very nicely.

The trouble is that now I’m entering the world of cheap software packages for consumer use (you know the sort of thing – the CD/DVD that’s free with the Sunday newspaper or found for a fiver or less in the bargain bin at PC World) and having written about my experiences of installing childrens’ software last weekend, I then experienced the other extreme of educational software – a little package called “Noddy – Let’s Get Ready for School“.

Noddy - Let's Get Ready for School

It didn’t start off well as running the application installer initially produced a really unhelpful dialog with a red stop icon and an OK button (no title or error message), so I tried again as Administrator and the InstallShield installer ran as expected. Once installed, I launched the application to find that a) it wasn’t really installed at all (the CD was still required throughout and the installer appears to have just created a few icons) and b) the program reset the screen to 640×480 mode before crashing.

I checked the specifications on the box: Pentium processor at 90MHz or better (I was using a 1.4GHz Pentium 4 M); 16-bit colour (I was using 32-bit color) and Windows 95 or 98 (I was using XP – Windows 9x operating systems are now unsupported so I would hope that vendors would stop selling appllications that rely on them, even if they do cost only a few pounds).

Running the application in Windows XP’s Windows 95 compatibility mode solved the crash issue but even so, it still insisted on me downgrading the graphics to 16 or 24-bit colour. After running successfully as Administrator, I logged out and logged back on with my son’s (unprivileged) account to find that running the application produced the following error message:

Director Player 6.0

Unable to copy the driver file C:\WINDOWS\xobglu16.dll to your Windows directory.

Your disk may be full.

The disk was far from full, but writing to the system root folder would be subject to NTFS access permissions. Indeed, using RunAs to elevate my permissions let me run the application with no apparant issues (except that I don’t want a toddler to have to enter a password to run a game), so I tried to copy the xobglu16.dll file myself (the file doesn’t appear on the CD, but is present in %systemroot% whilst running the application using an account with the necessary privileges – e.g. Administrator – along with a similarly-named xobglu32.dll). It seems crazy that a program would copy DLLs to %systemroot% each time it is run but, nevertheless, that seems to be the case; however if I copy them myself it crashes.

In the end I resorted to making my son’s account a power user on the machine (running a sandboxed Windows 98 installation in a virtual machine would have been another option, but less user friendly). Still, at least I didn’t have to make him an administrator.

In fairness, I should have been ready for this, having spent many hours trying to get various items of software aimed at little people to run successfully on my friends’ PCs but I did think the fact that this particular package was produced by BBC Multimedia would be a good thing. Clearly I was wrong and the BBC should stick to television programming (maybe it’s no coincidence that BBC Multimedia no longer publishes computer and video games).