Fighting back to the spammers: charging for removal of blog spam links…

This content is 11 years old. I don't routinely update old blog posts as they are only intended to represent a view at a particular point in time. Please be warned that the information here may be out of date.

Right from (almost) the start, this blog has suffered from spam. I guess it just goes with the territory but I’ve written in the past about people who’ve left spam comments and then found Google’s index quotes them out of context or tech companies criticising their competitors “anonymously” in blog comments.

Even when I was helping my then-CTO to raise his social media presence, my employer’s PR agency was encouraging the use of comments on blogs to generate backlinks and now the tide is turning as Google cracks down on low-quality backlinks.

As a result, I’m getting an increasing number of emails from digital agencies including phrases like the one below:

“I’m writing to request the removal of a link to my clients’ [sic] site which is located at the following page:”

They’re (or their clients are) wasting my time, so I reserve the right to charge for removing such links.

The irony is that, over the last few years, Google’s index changes have penalised original content creators like myself in favour of corporate websites and this blog has just a fraction of the traffic it once enjoyed (oh, those were the days)…

Would be blog spammers at this site should check out the Rules for Comments.

Office 365 message filtering (and a horrible little bug that leaves email addresses exposed…)

This content is 13 years old. I don't routinely update old blog posts as they are only intended to represent a view at a particular point in time. Please be warned that the information here may be out of date.

One of my concerns with my recent switch from Google Apps Mail to Microsoft Office 365 was about spam email. You see, I get none.  Well, when I say I get none, I get plenty but it’s all trapped for me. With no effort on my part. Only a handful of missed spam messages in the last 2 or 3 years and almost as few false positives too.

I’ve had the same email address for about 12 years now (I think), and it’s been used all over the web. Some of my friends are more particular though – and, perhaps understandably, were annoyed when I accidentally emailed around 40 people with e-mail addresses visible in the To: field today. Except that I hadn’t intended to.

I think I’ve found a bug in Office 365’s Outlook Web App (at least, I hope it’s not closed as “by design”, assuming I find out how to file a bug report). If I send to a distribution group, it automatically expands the addresses and displays them to all recipients. That’s bad.

The annoying thing is that, previously, I had been BCCing the recipients. I have a feeling that at least one organisation was rejecting my mail because there was nothing in the To: field (although it didn’t like Google’s propensity to send mail from one domain “on behalf of” another address either), so I thought I’d use a list instead and the recipients would see the list name, rather than the actual email addresses. Thankfully it was only sent to my closest freinds and family (although that’s not really the point).

So, back to spam and Office 365 – does it live up to my previous experience with Google Apps Mail? Actually, yes I think it does. I’ve had to teach it a couple of safe senders and block a couple of others, but it really was just a handful and it’s settled down nicely.

All of Microsoft’s cloud-based e-mail services use Forefront Online Protection for Exchange. Enterprise administrators have some additional functionality (adapting SCL thresholds, etc.) but things seem to be working pretty well on my small business account too. Digging around in the various servers that the mail passes through sees hosts at and – Frontbridge was an aquisition that has become part of Exchange Hosted Services (and it started out as Bigfish Communications) – so the technology is established, and another Microsoft property (Hotmail) is a pretty good test bed to find and filter the world’s spam.

Working around AOL’s short-sighted antispam measures

This content is 17 years old. I don't routinely update old blog posts as they are only intended to represent a view at a particular point in time. Please be warned that the information here may be out of date.

For the last year or so, I’ve been running my own mail server. This provides me with a number of advantages:

  • I have no storage limits (other than the physical limits of my hardware).
  • I have complete control over the mail server configuration.
  • I’m not at the mercy of my ISP’s e-mail server issues and delays if the mail queues get full
  • I can use my home system to try out new features and functionality.

It’s been working well. A few newsletters stopped arriving after I installed the Exchange Server intelligent message filter (IMF) and there is no way to whitelist addresses or subjects with the IMF that I’m aware of but, together with the realtime block lists that I use, it generally does a good job of trapping spam with only a few false positives.

I’ve been wondering for a while why there always seemed to be a problem with e-mailing AOL users but it’s such a rare occurrence that I didn’t get too hung up on it. Then I needed to send something over to someone and it bounced back (actually, AOL just refused to accept a connection from my mail server), so I started to look into the problem more closely.

I checked out AOL’s postmaster best practice guidelines and there didn’t seem to be a problem at first. I use a combination of Microsoft Exchange Server 2003 and Entourage 2004 so the mail should be RFC-compliant; I verified the connecting IP address (using AOL’s own tools); I checked that I have a valid reverse DNS entry (again, using AOL’s own tools); my mail server is not operating as an open relay; there were no links (let alone invalid ones) in the bounced messages and I have a static IP address on my business ADSL connection.

Having checked all of AOL’s technical guidelines for whitelisting I applied to join the AOL whitelist using AOL’s spam feedback form. AOL’s postmaster replied with the following:

AOL does not accept email sent directly from dynamic IPs. If you have a static IP, please contact your ISP and have them send us an updated list of dynamic and static IP ranges. If you are on a dynamic IP, please send through your ISP’s mail server.

It seems that the problem is my IP address. AOL’s senders’ FAQ states that:

Customers with residential IP addresses should use the provider’s SMTP servers and should not be sending email directly to another ISP’s SMTP servers.

I do have a static IP address and a business account (i.e. it is non-residential) but that’s not the point – AOL has the audacity to prevent anyone mailing them from an IP address that they have recorded on their database as residential! Admittedly my configuration is not normal but why should AOL dictate how I provide my e-mail service? I do understand that much of the world’s spam will originate from zombie-infected PCs on people’s home networks (i.e. residential) but I have an SPF record implemented in order to verify that e-mail purporting to originate from my mail server really is from my server. In any case, AOL’s reason for blocking me was nothing to do with authenticating my e-mail server but was purely based on their assertion that I have a residential IP address – something that doesn’t seem to bother any other mail hosting provider.

Not really wanting to get into the situation where my ISP says it’s AOL’s problem and AOL says it’s up to my ISP, I decided to work around the problem through reconfiguring Exchange Server:

  1. Firstly, I changed the cost on my existing SMTP connector (set to use DNS to route e-mail and using an address space of *) from 1 to 2.
  2. Next, I created a new SMTP connector for mail to be forwarded to my ISP’s relay, gave this a lower cost (1) and added to the address space.

Now e-mail for will be sent using my ISPs servers and all other external e-mail will go directly based on the MX records that are specified for the recipient’s domain. Amset IT solutions have a page on their website which explains the configuration in detail.

I needed a quick way of testing the message flow, so I sent a test message via my mail server to my e-mail address at work. Checking the headers on receipt showed that it had gone straight from my server to my employer’s e-mail gateway. Next, I added the domain name for my work e-mail address to the address spaces on the connector for e-mail to be routed via my ISP and repeated the test. Again, checking the headers verified that the message had indeed passed through my ISP’s relay. Finally, I took my employer’s domain name out of the address space on the new connector and verified that e-mail was directly routed once more.

It’s not difficult but it is a further complication in my mail server configuration, just to satisfy the requirements of one (admittedly large) ISP …and just one more reason for me to cringe when I hear that someone is an AOL subscriber.

Implementing SenderID Framework records for my e-mail server

This content is 17 years old. I don't routinely update old blog posts as they are only intended to represent a view at a particular point in time. Please be warned that the information here may be out of date.

I recently read Craig Spiezle and Alexander Nikolayev’s TechNet Magazine article about the SenderID Framework (SIDF) – one of the available schemes to validate mail servers in the fight to reduce unsolicited commercial e-mail (UCE), more commonly known as spam.

SIDF is similar to the Sender Policy Framework (SPF) in that it uses specially-formatted TXT records in DNS (called SPF records) to detail the mail exchange (MX) servers that SMTP e-mail may originate from for a given domain, and any other domain names that may be used.

I’d decided some time ago to implement an SPF record for my domains but my hosting service provider at the time did not support the use of TXT records. Since I moved to ascomi a few months back that’s not been an issue and last night I finally requested that the changes were made.

There are a variety of tools online to help create SPF records, but the first problem I had was the need to decide whether to use OpenSPF, SenderID, or an alternative (such as Domain Keys). In the end, I decided to go with SenderID – largely because the Microsoft SenderID website helped me create an SPF record which supported the both SenderID Mail From method (identical to the SPF method) and the SenderID Purportedly Responsible Address (PRA) header method. Finally, to validate that my record was correct, I sent an e-mail to and used the Email Service Provider Coalition verification tools – Microsoft also publishes a short implementation guide for SIDF which is worth a read.

The differences between SPF formats are discussed on the OpenSPF site too (and OpenSPF also has tools to help create the necessary records) but the OpenSPF guys seem to be more interested in saying why SenderID violates the standards and shouldn’t really be called SPF (I call that the “not invented here” syndrome) than in actually helping people work out how to stop spam.

It’s also worth pointing out that my SPF record will not directly affect the volume of spam that I receive; it will, however, help others who perform SPF lookups to determine if mail that appears to come from one of my domains really originated from a server which I authorised. Even then, they may elect to retain the message, or they may drop it – that’s no different to today but as more and more SPF records are published, the volume of spam on the Internet should drop considerably as all messages are effectively authenticated as having passed through an authorised MX for the stated domain name.

Sender verify failed with incorrect reverse DNS record

This content is 17 years old. I don't routinely update old blog posts as they are only intended to represent a view at a particular point in time. Please be warned that the information here may be out of date.

What a week! Switching hosting providers, setting up a new content management system for this blog (more on that as soon as it’s ready) and all at the same time as suffering e-mail problems as, since the middle of the week, every e-mail that I’ve sent to a particular contact has bounced back with the following message:

This is an automatically generated Delivery Status Notification.

Delivery to the following recipients failed.

Reporting-MTA: dns;

Final-Recipient: rfc822;
Action: failed
Status: 5.5.0
Diagnostic-Code: smtp;550-Verification failed for <>
550-No Such User Here
550 Sender verify failed

I have various anti-spam measures on my mail server, but this appeared to be a problem when sending mail to a particular external host – e-mail sent to the same contact via a different mail server was received with no problems.

I set about researching the 550 Sender verify failed message and found various suggestions as to what might cause such an error – the most useful of which was a message on a newsgroup post which suggested it may be caused by an incorrect reverse DNS (PTR) record (thanks to Ben Winzenz for replying to that group a couple of years ago).

Even though much of my mail was being delivered successfully, that seemed like a perfectly reasonable explanation – the reverse lookup for my IP address would have returned a hostname in the format, rather than (as confirmed by a DNS report on my domain, which also commented that “RFC1912 2.1 says you should have a reverse DNS for all your mail servers. It is strongly urged that you have them, as many mailservers will not accept mail from mailservers with no reverse DNS entry”), so I set about getting the record updated by my ISP (it has to be done by the owner of the IP address block).

Initially I asked my ISP to add my mail server’s DNS name as a second PTR record for my IP address but in practice I found that DNS responded in a round robin pattern (rather than returning all the matching records) so I couldn’t rely on a consistent response and was still experiencing mail delivery failures. Finally, after reverting to a single PTR record for my IP address and waiting for DNS propagation (again), I was able to successfully send e-mail to the contact with whom I’d previously experienced issues (phew!).

As more and more hosts take action to prevent unsolicited commercial e-mail (UCE – also known as spam), this is likely to be a more common occurrence and it just underlines how important a correct DNS configuration is.

Implementing real time block lists for spam control

This content is 18 years old. I don't routinely update old blog posts as they are only intended to represent a view at a particular point in time. Please be warned that the information here may be out of date.

The Spamhaus Project
A couple of months back, I wrote a post about controlling spam using the Microsoft Exchange Intelligent Message Filter. Whilst it has to be said that the IMF has been effective in reducing my spam volumes (with very low false positives – strangely enough my blog posts are the ones it has most trouble with) it’s still not catching all of the unsolicited commercial e-mail (UCE) that I receive, so this week I resorted to another spam control – real time DNS block lists.

Various lists exist with details of known spam relays and the one I’m using is from the Spamhaus project. Actually I’m using two of their lists – the Spamhaus block list (SBL) and the Spamhaus exploits block list (XBL), both of which are free for non-commercial use – I may add other services later.

Setting up the block lists within Microsoft Exchange Server was reasonably straightforward, following advice from Daniel Petri (further information can be found in Microsoft knowledge base article 823866). I then tested the service as recommended at Crynwr Software’s spam blocking resources page. After initial problems testing the service as my mail was being routed via my ISP’s relays (but I could see the conversation when I telnetted to Crynwr’s servers) I switched to DNS-based routing and received a satisfactory response to the e-mail tests – most importantly showing the following text in the SMTP conversation:

550 5.7.1 knownspamserveripaddress has been blocked by Spamhaus
Terminating conversation

So, that’s another tool in my anti-spam arsenal. The UCE levels appear to be tailing off now… hopefully I’m not dropping too much “real e-mail”. One day I hope to be able to say (in the style of John C Dvorak) “I get no spam”.

E-mail protected by SBL advisory E-mail protected by SBL advisory

Message hygiene principles for Microsoft Exchange Server

This content is 18 years old. I don't routinely update old blog posts as they are only intended to represent a view at a particular point in time. Please be warned that the information here may be out of date.

Whilst researching my post on the Microsoft Exchange intelligent message filter a couple of months back, I came across the following message hygiene architectural principles, which Microsoft promotes as best practice:

  • Anti-spam filtering must be performed before anti-virus filtering.
  • Anti-spam filtering should be performed for inbound mail only.
  • Anti-spam filtering should remove messages (cf. quarantining messages).
  • Anti-virus filtering must scan both inbound and outbound mail.
  • Anti-virus filtering must be mail-direction aware.
  • Anti-virus filtering must block messages that it cannot scan.
  • Anti-virus and anti-spam filtering system must integrate with Exchange Server.

Avoiding compulsory website registration

This content is 18 years old. I don't routinely update old blog posts as they are only intended to represent a view at a particular point in time. Please be warned that the information here may be out of date.

I’m sick of giving out my personal details (even false ones) to websites that require me to register. A few months back I wrote about using a temporary e-mail address to avoid spammers but now (thanks to a comment on a post at 4sysops), I’ve discovered BugMeNot a site that allows ‘net users to bypass compulsory registration. Simply enter the URL for the website that requires registration and the site will tell you if it has a set of credentials on file that you can use.

Of course, there are sites that I do register with because they provide a service that I consume, but as Michael Pietroforte notes in his never sign up for ZDNet white papers post, sometimes it’s just a way to get your details (in this case from a company which has been accused of being a Spamhaus) and then refer you to a vendor’s own freely-available information.

Controlling spam using the Microsoft Exchange intelligent message filter

This content is 18 years old. I don't routinely update old blog posts as they are only intended to represent a view at a particular point in time. Please be warned that the information here may be out of date.

It may just be a co-incidence, but since I switched my e-mail from my ISP’s servers to my own server a few months back, I’ve been seeing a huge increase in the amount of unsolicited commercial e-mail (UCE) – commonly known as spam – in my mailbox.

At the time of writing, statistics from MessageLabs show a decline in the volumes of spam over the last 12 months (although they still indicate that 58.39% of all e-mails sent were spam). Postini’s statistics suggest that 73% of e-mail is spam.

If you think those statistics are bad, according to Microsoft, Bill Gates receives 4 million spam messages a day, making him probably the most spammed man in the world (it’s no surprise then that he is rumoured to have his own mail server at Microsoft).

Any effective strategy for dealing with UCE (specifically for Exchange Server 2003, but the generic advice is the same for all mail servers) needs to operate a multiple levels within the e-mail transport (these are defined on the Message Delivery Settings under Global Settings in Exchange System Manager but need to be imposed using the properties for each SMTP virtual server):

  • Server-level accept/deny lists can be used to always accept, or always deny, messages from certain domains. The trouble with this method of trapping e-mail is that I occasionally receive non-delivery reports (NDRs) for messages that were allegedly sent from but that actually never came near my servers, so without a real-time DNS lookup mechanism to verify the sender’s domain (such as Sender ID), these are of limited use.
  • Connection filtering using real-time block lists (RBLs) is the next level of protection, using a DNS query against a RBL provider’s servers, such as the SpamHaus project.
  • Sender filtering can be used to drop any messages that claim to come from a particular e-mail address, optionally archiving them.
  • Recipient filtering is a method of rejecting certain e-mail addresses (e.g. for people who have left the organisation, or for non-existent addresses). One option is to filter messages for recipients who are not in the directory; however this can leave an organisation open to a directory harvest attack as the server gives different responses for valid and invalid addresses. To avoid such attacks, a “tarpit” (see Microsoft knowledge base article 842851) can be employed, to delay responses to bad addresses by a few seconds, slowing down any directory harvest attacks significantly (it would normally be possible to harvest all four-character address combinations within a few minutes – with a 5 second tarpit delay this is increased to a couple of months – and most addresses have much longer aliases than 4 characters).
  • Finally, the intelligent message filter (IMF – previously a separate download but now included with Exchange Server service pack 2) employs a Microsoft-proprietary algorithm (SmartScreen) to scan each message and mark it with a spam confidence level (SCL), which is then used to process the mail accordingly at the gateway or mailbox level.

Each of these tools filters out less obvious types of UCE with increasing levels of cost in terms of server resource. Whilst the junk e-mail filters in Outlook 2003/2007 and Entourage 2004, which are also based on SmartScreen but doesn’t use the SCL mechanism, are pretty good at filtering messages, they are far from perfect (in my experience, Outlook seems to be better at this than Entourage). Activating the IMF on my server has provided an additional level of filtering which has greatly reduced the volume of UCE making it through as far as my mailbox.

The IMF uses 11 SCL ratings, set as an attribute in the message header:

  • -1 is used for messages submitted internally with an authenticated connection – eliminating false positives for internal e-mail.
  • 0 is used for messages that are marked as not spam.
  • 1-9 are used to highlight varying levels of probability that a message is spam (9 being the most likely).

Within Exchange, the SCL value can be used to filter UCE on gateway servers as well as with a lower level SCL used by the information store to move messages to the user’s junk e-mail folder – therefore allowing for the most obvious UCE to be trapped at the gateway (least chance of false positives) and for users to retrieve any messages in the mid-range that are incorrectly marked as junk. The gateway blocking action is also configurable – with options for archival, deletion (without NDR), no action, or rejection.

Archived messages will be saved (by default) to %programfiles%\Exchsrvr\Mailroot\vsi 1\UCEArchive. Each message is archived as an .EML file, which can be viewed with a text editor. To resubmit a message for delivery it can simply be moved to the corresponding %programfiles%\Exchsrvr\Mailroot\vsi 1\Pickup folder. Obviously, viewing individual messages in a text file is time-consuming and the IMF Archive Manager is a great tool for managing IMF-archived messages.

The SCL at which to block messages for a particular organisation will vary according to the profile of e-mail sent to/from the organisation – I have my SCL level for gateway blocking set to 7 with archiving enabled and so far I have only had one false positive – but clearly for organisations receiving more e-mail than I do, this will be a bigger issue! At the store level (set to move messages with an SCL greater than 4) things are not working quite so well but that is to be expected as in the grey area between good and bad mail, some legitimate (good) messages will inevitably get marked with the same SCL as the (bad) UCE. It’s worth noting that marking a sender as safe in Outlook will only override the SCL at the mailbox-level – it has no effect at the gateway.

To assist in judging the SCL levels to use for filtering, it is possible to expose the SCL in Outlook and in Outlook Web Access (OWA). Also useful may be (temporarily) enabling diagnostic logging on the MSExchangeTransport\SMTP Protocol for a server, such that SMTP events are logged. Performance monitor counters from the MSExchange Intelligent Message Filter object can also be used to log the amount of spam filtered or acted upon, the relative SCL levels and overall IMF performance. Based on the performance monitor data, the IMF gateway blocking configuration can be reduced from no action to archive, and then finally (once confident that the levels are correct) to delete, as the appropriate SCL levels are determined.

It’s also possible to mark the SCL on archived messages by creating an new registry key called ContentFilter at HKEY_LOCAL_MACHINE\Software\Microsoft\Exchange\ and a corresponding DWORD value named ArchiveSCL set to 1. A string value named ArchiveDir can also be used to change the archive folder. Both of these settings are detailed in the Microsoft Exchange Server TechCenter along with details for applying the IMF to trusted (authenticated) connections and increasing the size limit for the rule used to process spam at mailbox level (allowing more blocked and safe senders).

Suggested further reading
IMF release notes (Microsoft knowledge base article 867633).
Microsoft Exchange Team Blog.

Using a temporary e-mail address to avoid spammers

This content is 18 years old. I don't routinely update old blog posts as they are only intended to represent a view at a particular point in time. Please be warned that the information here may be out of date.

A colleague just sent me a link to Mailinator – a service for creating temporary mailboxes that are valid for just a few hours in order to receive (but not send) e-mail, e.g. when registering on a website and needing to see the initial registration e-mail, but wanting to guard against receiving unsolicited commercial e-mail (UCE – more commonly known as spam) afterwards.

This is how the guys at Mailinator describe it:

“It’s like super-instant, always-ready, any-email-you-want email. Right now. It’s your personal disposable email account. Here is how it works: You are on the web, at a party, or talking to your favorite insurance salesman. Wherever you are, someone (or some webpage) asks for your email. You know if you give it, you’re gambling with your privacy. On the other hand, you do want at least one message from that person. The answer is to give them a mailinator address. You don’t need to sign-up. You just make it up on the spot[…] — pick anything you want.

Later, come to [the Mailinator] site and check that account. Its that easy. Mailinator accounts are created when mail arrives for them. No signup, no personal information, and when you’re done — you can walk away — an instant solution to one way spammers get your address. It’s an anti-spam solution for everyone. Your temporary email account will be automatically deleted for you after a few hours.

Let’em spam…”

I haven’t tried it yet, but it sounds like a great idea!