Attempting to reduce my website’s bandwidth usage

Posted on By Mark Wilson

This website is in a spot of trouble. Over the last few months, I’ve seen the bandwidth usage grow dramatically although it seems to have grown faster than the number of subscribers/readers. We’re not talking vast volumes here and my hosting provider has been very understanding, but even so it’s time to do something about it.

So I had a think, and came up with three options:

  1. Don’t write anything. Tried that for half of June (when I was on holiday). No noticeable change in the webstats!
  2. Write rubbish. It’s debatable as to whether that’s a continuation of the status quo.
  3. Drp ll th vwls s f wrtng txt mssgs.
  4. Optimise the site to reduce bandwidth usage, without a major rewrite. Yeah! That sounds like a challenge.

So, option four sounded like the best course of action. There are two main elements to consider in this:

  1. Site performance (i.e. how fast pages load).
  2. Bandwidth usage.

As far as I can tell, my site performance is not blindingly fast but it’s OK. I could use something like the WP-Cache plugin to cache content but, although that should reduce the load on the server, it won’t actually decrease my bandwidth usage. In fact it might increase it as I’d need to turn off HTTP compression.

That led me to concentrate on the bandwidth issues. This is what I tried (based mostly on Jeff Atwood’s experience of reducing his site’s bandwidth usage):

  • Shut out the spammers. Akismet had blocked over 7000 spam messages in 15 days and each of these would have loaded pages and leeched some bandwidth in the process. Using the Bad Behaviour plugin started to reduce that, blocking IP known spammers based on their IP address. Hopefully it hasn’t blocked legitimate users too. Please let me know if it has blocked you (assuming you can read this!).
  • Compress the content. Check that HTTP compression is enabled for the site (it was). According to Port 80 Software’s real-time compression check, this both reduces my file size by about 77% and decreases download times by 410%. It’s also possible to compress (i.e. remove whitespace and comments) in CSS and JavaScript (as well as tools for HTML compression) but in my opinion, the benefits are slim (as these files are already compressed with HTTP compression) and code readability is more important to me (although at 12.7KB, my main stylesheet is a little on the bloated side of things – and it is one file that gets loaded frequently by clients).
  • Optimise the graphics. I already use Adobe Photoshop/ImageReady to save web optimised graphics but I used a Macintosh utility that Alex pointed me to called Ping to optimise the .PNG files that make up about half the graphics on this site (I still need to do something with the .JPGs and .GIFs) and that shaved just over 10% off their file size – not a huge reduction but it should help.
  • Outsource. Switching the main RSS feed to FeedBurner made me nervous. I’d rather have all my readers come to my domain than to one over which I have no control but then again FeedBurner gives me some great analysis tools. Then I found out about Feedburner’s MyBrand feature (previously a chargable option but free since Feedburner was acquired by Google) which lets me use feeds.markwilson.co.uk (i.e. a domain under my control) instead of feeds.feedburner.com. Combined with the FeedSmith plugin, this has let me keep control over all of my feeds. One more option is to use an external image provider (Jeff Atwood recommends Amazon S3 but I haven’t tried that yet).

At the moment it’s still early days but I do have a feeling that I’m not eating up my bandwidth quite as quickly as I was. I’ll watch the webstats over the coming days and weeks and hope to see a downward trend.

Working around AOL’s short-sighted antispam measures

Posted on By Mark Wilson

For the last year or so, I’ve been running my own mail server. This provides me with a number of advantages:

  • I have no storage limits (other than the physical limits of my hardware).
  • I have complete control over the mail server configuration.
  • I’m not at the mercy of my ISP’s e-mail server issues and delays if the mail queues get full
  • I can use my home system to try out new features and functionality.

It’s been working well. A few newsletters stopped arriving after I installed the Exchange Server intelligent message filter (IMF) and there is no way to whitelist addresses or subjects with the IMF that I’m aware of but, together with the realtime block lists that I use, it generally does a good job of trapping spam with only a few false positives.

I’ve been wondering for a while why there always seemed to be a problem with e-mailing AOL users but it’s such a rare occurrence that I didn’t get too hung up on it. Then I needed to send something over to someone and it bounced back (actually, AOL just refused to accept a connection from my mail server), so I started to look into the problem more closely.

I checked out AOL’s postmaster best practice guidelines and there didn’t seem to be a problem at first. I use a combination of Microsoft Exchange Server 2003 and Entourage 2004 so the mail should be RFC-compliant; I verified the connecting IP address (using AOL’s own tools); I checked that I have a valid reverse DNS entry (again, using AOL’s own tools); my mail server is not operating as an open relay; there were no links (let alone invalid ones) in the bounced messages and I have a static IP address on my business ADSL connection.

Having checked all of AOL’s technical guidelines for whitelisting I applied to join the AOL whitelist using AOL’s spam feedback form. AOL’s postmaster replied with the following:

AOL does not accept email sent directly from dynamic IPs. If you have a static IP, please contact your ISP and have them send us an updated list of dynamic and static IP ranges. If you are on a dynamic IP, please send through your ISP’s mail server.

It seems that the problem is my IP address. AOL’s senders’ FAQ states that:

Customers with residential IP addresses should use the provider’s SMTP servers and should not be sending email directly to another ISP’s SMTP servers.

I do have a static IP address and a business account (i.e. it is non-residential) but that’s not the point – AOL has the audacity to prevent anyone mailing them from an IP address that they have recorded on their database as residential! Admittedly my configuration is not normal but why should AOL dictate how I provide my e-mail service? I do understand that much of the world’s spam will originate from zombie-infected PCs on people’s home networks (i.e. residential) but I have an SPF record implemented in order to verify that e-mail purporting to originate from my mail server really is from my server. In any case, AOL’s reason for blocking me was nothing to do with authenticating my e-mail server but was purely based on their assertion that I have a residential IP address – something that doesn’t seem to bother any other mail hosting provider.

Not really wanting to get into the situation where my ISP says it’s AOL’s problem and AOL says it’s up to my ISP, I decided to work around the problem through reconfiguring Exchange Server:

  1. Firstly, I changed the cost on my existing SMTP connector (set to use DNS to route e-mail and using an address space of *) from 1 to 2.
  2. Next, I created a new SMTP connector for mail to be forwarded to my ISP’s relay, gave this a lower cost (1) and added aol.com to the address space.

Now e-mail for anyone@aol.com will be sent using my ISPs servers and all other external e-mail will go directly based on the MX records that are specified for the recipient’s domain. Amset IT solutions have a page on their website which explains the configuration in detail.

I needed a quick way of testing the message flow, so I sent a test message via my mail server to my e-mail address at work. Checking the headers on receipt showed that it had gone straight from my server to my employer’s e-mail gateway. Next, I added the domain name for my work e-mail address to the address spaces on the connector for e-mail to be routed via my ISP and repeated the test. Again, checking the headers verified that the message had indeed passed through my ISP’s relay. Finally, I took my employer’s domain name out of the address space on the new connector and verified that e-mail was directly routed once more.

It’s not difficult but it is a further complication in my mail server configuration, just to satisfy the requirements of one (admittedly large) ISP …and just one more reason for me to cringe when I hear that someone is an AOL subscriber.