It’s time to take patch management seriously

Windows updates don’t normally feature highly on this blog… after all, they come along every month, you test them (perhaps?), install them, and leave things along for a few weeks. Sometimes there’s an out of band patch release and that ought to indicate that there is a significant problem that requires attention. So why have I been hearing so much over the last few weeks about the Win32/Conflicker.B worm with people panicking to update systems, install the latest AV updates, and generally try and catch up after being so lackadaisical in the first place?

Let me explain what I mean… according to an e-mail I received from Microsoft last week:

Win32/Conficker.B exploits a vulnerability in the Windows Server service (SVCHOST.EXE) for Windows 2000, Windows XP, Windows Vista, Windows Server 2003, and Windows 2008. While Microsoft addressed this issue in October with Microsoft Security Bulletin MS08-067, and Forefront antivirus and OneCare (as well as other vendor’s anit-virus products) helped protect against infections, many systems that have not been patched manually through Server Update Services and Microsoft/Windows Update or through Automatic Updates have recently come under attack by this worm. Attacked systems may lock out users, disable our update services and block access to security-related Web sites:

In response to this threat, Microsoft has:

It is our hope that these resources can assist you in resolving issues with unpatched, infected systems and that you can apply MS08-067 to any other unpatched systems as soon as possible to avoid this threat.”

I’m sure there are some people who feel that applying updates is an intrusion, an unnecessary interruption into the day (these are probably the same people that advocate turning off user account control…). Others will claim that other operating systems don’t need patching so often (I don’t know about the frequency of updates but patches on my Macs always seem pretty big and Linux is in one big patch cycle as the open source model is one of continuous improvement). Personally, I’m glad that Microsoft settled down to a predictable monthly cycle and for those who think that’s a problem because it gives hackers a predictable timeframe for reverse engineering patches and attacking weaknesses in unpatched systems it’s all the more reason why every organisation’s IT security people should be ready to look at the update announcements on the second Tuesday of every month and then to act accordingly. And when a patch comes along outside that predictable schedule to consider that, yes it’s a pain in the neck, but it might just be important…

Which brings me back to the point. Conficker (also known as Downadup). As F-Secure put it:

“First — It was an out-of-band update.

Second — It was given an ‘Exploitability Index Assessment’ of ‘1 – Consistent exploit code likely’.

That kind of speaks for itself, doesn’t it?

Third — It allows for Remote Code Execution, in numerous versions of Windows (particularly critical for 2000, XP, and Server 2003).

All of these combined factors equals something quite serious that should be patched as soon as possible. If you are having difficulties with Automatic Updates, the bulletin links to manual downloads.

Security Update for Windows XP
Security Update for Windows Server 2003

It’s always a good idea to be ready for out-of-band updates. You can subscribe to Microsoft Security Notifications here.”

The other thing that this worm has awakened is corporate IT departments saying things like “how can we check that all our machines are updated with the Microsoft update and with the latest antivirus signatures?”. Well guys, there’s a feature called Network Access Protection (NAP) and it’s implemented in Windows XP SP3, Windows Vista and Windows Server 2008. Whilst you’ve all been bleating about how Vista is bad, perhaps you should have looked a bit further and seen some of the advantages it could bring. If you still can’t stomach a Vista upgrade because somehow you think that Windows 7 will be easier from an application compatibility standpoint (I have news for you…) or think that Microsoft and security in the same sentence indicates an oxymoron then there are plenty of third party endpoint security systems with similar controls…

Perhaps we need an outbreak like this from time to time to wake up the IT Managers and persuade them to spend some money on security improvements within the infrastructure.

Here endeth the lesson. Now go and update your systems.

For more information, check out Centralised information about the Conficker Worm and MS08-067 Conflicker worm update.

Microsoft Update failure

I’ve been building a Windows XP virtual machine for test purposes and needed to apply the latest updates (even with Windows XP service pack 3 it required over 20 updates to be applied). Unfortunately, Microsoft Update hit a problem and refused to install some of the updates, telling me that “a problem on your computer is preventing updates from being downloaded or installed“. I tried disabling my anti-virus software (AVG Free) but that made no difference.

Microsoft Update: Failed Updates

Microsoft’s advice is to re-register a number of DLLs using the following commands:

regsvr32 wuapi.dll
regsvr32 wuaueng.dll
regsvr32 wuaueng1.dll
regsvr32 wucltui.dll
regsvr32 wups.dll
regsvr32 wups2.dll
regsvr32 wuweb.dll

For each successful registration, Windows should return “DllRegisterServer in filename.dll succeeded” but wucltui.dll didn’t seem to exist on my system. Even so, after re-registering the remaining DLLs, Microsoft Update successfully installed the problem updates.

Caching Microsoft updates with ISA Server

I used to use WSUS to update the machines on my home network but after a botched server upgrade, it all went screwy and I didn’t really want to have to pull all the updates down over my ADSL connection again (would probably blow away my month’s worth of “fair usage”). In any case all I was doing was blindly approving updates for installation so I might as well use the Microsoft Update servers instead.

The only downside of using the Microsoft Update servers to update several computers is that there is a lot of duplication in the network traffic. That’s why ISA Server 2006 includes a cache rule that enables caching of Microsoft updates using the Background Intelligent Transfer Service (BITS). For those who aren’t aware, BITS allows the transfer of large volumes of data without degrading network performance as it transfers the data in small chunks to utilise unused bandwidth as it becomes available and reassembles the data at the destination. The BITS feature is not available for any other ISA cache rule but the Microsoft Update Cache Rule is installed by default and all I needed to do was enable caching.

After setting up the cache rules and updating one of my servers, I wanted to see that the cache was being used. I’ve previously mentioned the cachedir.exe tool that can be used to examine ISA Server caches and I downloaded the latest version from Microsoft’s ISA Server 2006 tools page. After extracting the tool, I ran it and was presented with an error:

CACHEDIR.exe – Unable to Locate Component
This application has failed to start because msfpc.DLL was not found. Re-installing the application may fix the problem.

Then I remembered that cachedir.exe needs to be copied to the ISA Server installation folder (on my system that is %programfiles%\Microsoft ISA Server) – after moving the file to the correct folder, it fired up as expected. Just remember that this utility can only display the cache contents that have been written to disk. To flush the memory cache to disk you will need to restart the Microsoft Firewall service and re-run cachedir.exe to view the contents.

When Windows Updates turn bad

Last night, as I got ready to shut down the notebook PC that I use for work, I noticed that it had some Windows updates to apply. I left Windows doing its thing and went to bed, stopping this morning only for long enough to put the PC into my bag as I headed off for the station. Only when I was on the train did I fire it up to find that the PC would not boot, greeting me instead with the following message:

Windows Boot Manager

Windows failed to start. A recent hardware or software change might be the cause. To fix the problem:

1. Insert your Windows installation disc and restart your computer.
2. Choose you language settings and then click “Next.”
3. Click “Repair your computer.”

If you do not have this disc, contact our system administrator or computer manufacturer for assistance.

File: \Windows\system32\winload.exe

Status: 0xc000000f

Info: The selected entry couple not be loaded because the application is missing or corrupt.

I spent the rest of the journey to London contacting colleagues to see if anyone could bring a Vista DVD in with them (with no success). After that failed, I asked the local IT support guys (no chance – they view anyone who doesn’t run the corporately-sanctioned Windows XP build as a renegade who can make their own support arrangements). A colleague used his MSDN subscription to start downloading a DVD image for me onto another colleague’s computer, but after almost 3 hours it was still only 60% downloaded (and he needed to leave the office). So I gave up and headed home.

Once home, the recovery process was straightforward. I booted from DVD, followed the directions for a startup repair and, after a reboot or two, I could log on as normal but it does leave me wondering whether, as I finally get stuck into today’s work at 4pm (after leaving home for the office at 6.30am), blindly applying updates is such a good idea?

I don’t think there is a single “correct” answer to this. On one hand, I run a risk that an update turns bad on me – and losing a day’s productivity is fairly minor in the scheme of things (next time it could be far worse). On the other hand, what is the risk of waiting to apply updates until after they have been tested (even critical ones)? After all, at home I’m on a NATted network segment, protected by a firewall, and at work the protection from the outside world is even stronger. But what about protection from the inside – from colleagues and internal servers? What about when I work on a public 3G or WiFi network? I guess, like any security decision, its a balance between risk of a security breach and the convenience of continued system stability.

In the meantime, I’ll carry on applying updates when Microsoft pushes them at me. It’s the first time an update has turned bad on me (and that system is operating with around 1.5% free disk space, which may be a factor in the issues that I experience with it). Hopefully next week I’ll finally get my new notebook and start the switch to using Windows Server 2008 as my daily computing platform for work.

Windows service pack roadmap

Those of us whose history goes back to Windows NT remember when a service pack was exactly what its name suggests – no new features, just bug fixes, thoroughly tested (usually) – and when application of the latest service pack was no big deal (application of any other updates was not normally required, unless addressing a specific issues). Today the landscape is different, with irregular service packs often bringing major operating system changes and requiring extensive testing, and frequent updates issued on the second Tuesday of almost every month.

A couple of weeks ago I was at a Microsoft event where one of the presenters (Microsoft UK’s James O’Neill) suggested that service packs are irrelevant and that they actually serve to put some people off deploying new operating system releases. To be fair to James, he was specifically talking about the “don’t deploy until the first service pack has shipped” doubters and to some extent he is right – the many updates that are applied to a Windows Vista installation today have provided numerous incremental improvements to the operating system since the original RTM last year. Even so, I can’t help thinking that Microsoft has muddied the water to some extent – I always understood that service packs had a higher level of integration testing than individual updates but it seems the current Microsoft advice is to apply all applicable “patch Tuesday” updates but only to apply other hotfixes (those updates produced to patch a specific customer scenario) where they are absolutely necessary.

Regardless of this confusion around the different forms of update, service packs are not dead – far from it – with both Windows Vista SP1 and Windows XP SP3 in beta at the time of writing. Although largely update rollups, these service packs do introduce some new features (new networking features for XP, and a kernel change for Vista to bring it in line with Windows Server 2008) but I’ve been of the opinion for some time now that XP SP3 is long overdue.

Going forward, it’s interesting to note that Windows Server 2008 is expected to launch with SP1 included. If that sounds odd, remember that both Windows Vista and Windows Server 2008 were originally both codenamed Longhorn and that they are very closely related – it’s anticipated that the next Windows service pack (let’s call it SP2 for the sake of argument) will be equally applicable to both the client and server operating system releases.

Problems with Microsoft Update

Over the last few days, I’ve been having problems connecting to Microsoft Update from a newly built Windows Server 2003 R2 server. Whilst searching for updates, it’s was hanging (green progress bar pulsing across the screen) before eventually reporting:

[Error number: 0x80244023]

The website has encountered a problem and cannot display the page you are trying to view. The options provided below might help you solve the problem.

For self-help options:

Frequently Asked Questions
Find Solutions
Windows Update Newsgroup

For assisted support options:

Microsoft Online Assisted Support (no-cost for Windows Update issues)

Sadly, Microsoft’s Online Assisted Support didn’t do much assisting – it only pointed me back to the knowledge base, newsgroups, or paid incident support but the problem did go away all by itself (yes, really)!

In the meantime, I’d asked for help on the Microsoft Windows Update discussion group and a really helpful MVP called TaurArian replied pointing me in the direction of the MSDN reference for Windows Update agent networking error codes. Useful stuff – and it turns out that 80244023 is WU_E_PT_HTTP_STATUS_GATEWAY_TIMEOUT – i.e. proxy server problems (and probably the reason why the problem seemed to cure itself).

Beware of automatic updates and hosted virtual machines

Whilst many organisations will have strict policies regarding patching, others will not and I’ve lost count of the number of times I’ve found myself troubleshooting strange errors in a virtual machine, only to find that the underlying host operating system has automatically updated itself and is waiting for a restart. Consequently, it’s worth mentioning that automatic updates and hosted virtualisation server products (e.g. Microsoft Virtual Server or VMware Server) do not mix well. Of course, those running a non-hosted virtualisation solution (like VMware ESX server) won’t have this issue; although even ESX needs patching from time to time.

WSUS 3.0 delivers huge improvements for the deployment of Microsoft updates

I’ve been an advocate of Microsoft SUS/WSUS since the v1.0 release. Sure, there are better enterprise software deployment products out there (Microsoft even has one – Systems Management Server) but as a low cost (free) patch management solution for Windows, it’s hard to beat Windows Software Update Services (which, since version 2.0, will update more than just Windows – WSUS 2.0 can act as a local cache for all updates that are available through the Microsoft Update servers). Except that now it has been beaten – by Windows Server Update Services (note the subtle name change) 3.0.

WSUS 3.0 was launched a couple of months ago and I finally installed it this afternoon. Not only does it include some great new features (like e-mail notification, improved reporting and computer management) but it finally gets an MMC administration interface (a huge improvement on the previous web administration interface). There are database changes too – WSUS no longer supports SQL Server 2000/MSDE (after all, those products are shortly to be retired), although it will upgrade an existing database.

The only downside that I can see is that the product still relies on clients connecting to the server and pulling updates (there is no option to force updates on clients – at least not as far as I can see). That’s fine but it does introduce some latency into the process (i.e. if there is an urgent patch to deploy, then WSUS is probably not the right tool to use); however, for the basic operational task of keeping a Windows infrastructure patched (for Microsoft products) and reporting on the current state, WSUS is definitely worth considering.

Further Information

WSUS 3.0 distributed network improvements (white paper).
WSUS 3.0 Usability improvements (white paper).

Running Red Hat Enterprise Linux without a subscription

I’ve written previously about why open source software is not really free (as in monetary value), just free (as in freedom). Companies such as Red Hat and Novell (SUSE) make their money from support and during Red Hat Enterprise Linux (RHEL) setup, it is “strongly recommended” that the system is set up for software updates via Red Hat Network (RHN), citing the benefits of an RHEL subscription as:

  • “Security and updates: receive the latest software updates, including security updates, keeping [a] Red Hat Enterprise Linux system updated and secure.
  • Downloads and upgrades: download installation images for Red Hat Enterprise Linux releases, including new releases.
  • Support: Access to the technical support experts at Red Hat or Red Hat’s partners for help with any issues you might encounter with [a] system.
  • Compliance: Stay in compliance with your subscription agreement and manage subscriptions for systems connected to [an] account at http://rhn.redhat.com/

You will not be able to take advantage of these subscriptions privileges without connecting [a] system to Red Hat Network.”

Red Hat Enterprise Linux 5 installer

Take a look at Red Hat Enterprise Linux (RHEL) and you’ll see that it’s actually quite expensive – a standard subscription for a machine with up to 2 processor sockets including 1 year’s 12×5 telephone support, 1 year of web access and unlimited incidents is €773.19 [source: Red Hat Online Shop, Europe]. That is not something that I can afford and even though Red Hat gave me a copy of RHEL 5 as part of my recent training, it only includes a 30-day subscription. Now they have launched Red Hat Exchange – a new service whereby third party open source software solutions are purchased, delivered and supported via a single, standardized Red Hat subscription agreement with consolidated billing covering the complete application stack. It’s a great idea, but the pricing for some of the packages makes using proprietary alternatives seem quite competitive.

In fairness to Red Hat, they sponsor the Fedora Project for users like me, who could probably make do with a community-supported release (Fedora is free for anyone to use modify and distribute) but there is another option – CentOS (the community enterprise operating system), which claims to be:

“An Enterprise-class Linux Distribution derived from sources freely provided to the public by a prominent North American Enterprise Linux vendor. CentOS conforms fully with the upstream vendor[‘]s redistribution policy and aims to be 100% binary compatible. (CentOS mainly changes packages to remove upstream vendor branding and artwork.) CentOS is free.”

Hmm… so which North American Enterprise Linux vendor might that be then ;-)

So what about RHEL systems for which the subscription has expired? I’m not sure what the legal standpoint is but there is a way to receive updated software using an unregistered copy of RHEL. Firstly, configuring additional repositories like Dag Wieer’s RPMForgethere are even RPMs available to set up the correct repository! Then, there are the various RPM search sites on the ‘net, including:

I’ve found that using these, even if there is not an appropriate RHEL or generic RPM available, there is often a CentOS RPM (which often still carries the el5 identifier in the filename). These should be safe to install on an RHEL system and in those rare cases when a bleeding edge package is required, there may well be a Fedora version that can be used. So it seems that I can continue to run a Linux distribution that is recognised by most software vendors, even when my RHN subscription expires.

Windows Update error 80245003

One of my Windows Vista PCs has been refusing to download updates from Windows Update, reporting that:

Windows could not search for new updates
Error(s) found:
Code 80245003

A bit of googling turned up various forum threads/blog posts about this article but most of them recommend stopping the Windows Update service, renaming/removing the %systemroot%\SoftwareDistribution folder, restarting the Windows Update service and attempting an update. That seems to work but Jeroen Jansen’s post on the subject included a very useful comment with this little gem:

“Actually you don’t have to delete the entire SoftwareDistribution folder, just the folders inside it with update cache. This way you can keep the update history.”

I renamed each folder one at a time and it seems that it was WuRedir that was causing the error on my system (that is to say that after that folder was renamed, Windows Update ran successfully, even after restoring all of the other folders, therefore maintaining my history and other configuration).

I’m not sure if it was as a direct result, but I’m pretty sure Vista switched from using Windows Update to Microsoft Update at the same time.