Troubleshooting an MS-DOS application which hangs the NTVDM subsystem in Windows XP and Windows Server 2003

I’ve been working on an intriguing (and frustrating) issue for a few weeks now and a couple of days back we finally resolved the issue.

My client has an MS-DOS (FoxPro 2.6a database) application running within an NTVDM on Windows XP. Every now and then, the application will hang – seemingly randomly. Windows XP did have service pack 2 applied, but the issue also occurs on service pack 1 PCs (I didn’t try the RTM version). Only the application hangs – it is possible to terminate the NTVDM process and carry on working as normally.

Normal actions for troubleshooting MS-DOS applications in Windows XP were not helping to resolve the issue, but the software vendor managed to narrow the issue down to FoxPro waiting for input. Occasionally, the input does not timeout and return control to the calling program – it seems that this is the root cause of the NTVDM hang. Identifying this allowed them to construct a test program which polled for input, timing out every few seconds and would reliably hang an NTVDM at a seemingly random time, but always within an hour.

Using their test program on a variety of PCs, the software vendor found that the problem was related to the Intel hyper-threading technology (my client has standardised on a version of the IBM ThinkCentre M50 which includes a single 3GHz Intel Pentium 4 processor with HT technology). Whilst disabling hyper-threading is unlikely to result in any significant performance degradation (hyper-threading only provides an average 10-20% performance gain as most applications do not fit completely with the hyper-threading model), it was still considered by IBM, Microsoft, my client and myself as a tactical workaround, rather than a strategic fix.

After seeking advice from Microsoft, I ran the test program on a Compaq ProLiant DL380 G2 server with two Pentium III 1.26GHz processors and found that the issue is not limited to Windows XP and hyper-threading, but to both Windows XP and Windows Server 2003 when running with an ACPI Multiprocessor PC HAL. Turning off hyper-threading on the PCs was no longer good enough as we can expect to see multiple processor cores constructed on a single die in the near future, leading to a rise in the use of multi-threaded applications (the logical processor provided by the hyper-threading technology in the Intel Pentium 4 processor is simply a precursor to this).

So why does an MS-DOS application running within an NTVDM on a 32-bit version of Windows use multiple processors? The answer it seems is that although the MS-DOS application is not multi-threaded, modern versions of Windows are, and can allocate parts of the NTVDM process to any available processor. With that in mind we re-ran the test program with processor affinity set to use only CPU0 in Task Manager. The results were the same as disabling hyper-threading – no NTVDM hang! Obviously, setting processor affinity manually is not sustainable outside the test environment, and short of running the application on Windows Server 2003 Enterprise Edition (with the Windows System Resource Monitor to control processor affinity) we needed to find an alternative solution.

That solution came in the form of the imagecfg.exe tool provided with the Microsoft Windows 2000 Resource Kit (supplement one). This can be used to edit an executable file and permanently set the processor affinity for a given application:

Using the imagecfg -a 0x1 c:\windows\system32\ntvdm.exe command did the trick, although Windows File Protection/System File Checker quickly restored the original ntvdm.exe file so I needed to perform this on a copy of ntvdm.exe in a temporary folder, and then overwrite both c:\windows\system32\ntvdm.exe and c:\windows\system32\dllcache\ntvdm.exe.

Once updated, the NTVDM process runs on CPU0. Of course, this limits all programs under the control of the NTVDM subsystem but it is far more preferable to disabling logical or physical processors in the BIOS; however, as this is a change to an operating system file, it must be considered alongside the implementation of any service packs and/or hotfixes from now on. Reversing the change is simply a case of restoring the original ntvdm.exe file.

Batch file command reference

Even though modern versions of Windows have rich scripting capabilities I regularly find myself writing batch (.bat) or command (.cmd) files for automating system tasks, sometimes during migrations from older versions of MS-DOS or Windows which do not have the same command set. I generally consider my batch file writing skills to be pretty good, but I have found the Computer Hope Batch File Help website to be a useful resource for checking syntax between different operating system versions.

Gartner’s top predictions for 2005 and beyond

I’ve been reading Gartner’s top predictions for 2005 and beyond.

Some of the predictions, make interesting reading, for example the rise in “microcommerce” opportunities and the prediction that “by 2009, counterfeit reality will account for at least one major media and political scandal” (scary when translated into real-world connotations).

Of more interest to me directly are the predictions that:

  • “Cyberattacks against software flaws will double by 2006” (although Gartner note that attacks against misconfigured software will actually decrease because vendors are beginning to ship software with more secure default configurations).
  • “By 2007, three of the top 10 PC vendors will exit the market” (IBM have already sold their PC business since the original Gartner research note was written in November 2004).
  • “By 2008, the technological differences between PCs, mobile devices, e-books, TVs and cellular phones will be eradicated”.
  • “By 2015, 40% of today’s IT job roles will be lost to automation”.

If that last prediction is correct, it looks like I’d better start thinking about retraining as a plumber/electrician/builder…

Discovering unknown devices in Windows

Whilst developing my unattended Windows XP build, I came across a number of devices that were not automatically detected by Windows.

Sometimes, right-clicking the unknown device in Device Manager and selecting the Update driver… option allowed Windows Update to connect to the Internet and locate updated drivers (which in turn identified the device and allowed me to download and integrate the OEM drivers into the Windows XP installation source).

On other occasions it was not so simple, and I needed to do some research to identify the device using the PCI device instance ID (found on the details page of the device properties).

A couple of years back I was introduced to Craig Hart’s PCI and AGP bus sniffer. I could have just run the utility, but in this case I chose to search its companion file (pcidevs.txt) for the vendor and device PCI IDs. Using this technique, I was able to identify my unknown device as the Broadcom (vendor 14E4) BCM4306 802.11g Wireless NIC (device 4320), which is also known as a Dell Wireless 1350. Once I had that information, all that was required was to download the drivers for integration.

Make your own iPod advert

I confess.

I don’t own an iPod.

I think they look great, but I just don’t need one (and I have concerns about the use of Apple’s proprietary AAC media format).

I like technology.

I am also a photographer, which makes me passionate about some of the fantastic images I see around me every day – like some of those featured in advertisements – and I think the silhouetted Apple iPod ads are cool.

Now there is iPod My Photo, which lets you turn any image into a mock iPod advert. You can choose one of 5 background colours, add a caption, and optionally add an iPod into the photo. Look closely and they’re not just 2-tone – check out the samples to see the shadow details etc.

All I need to do is select the right photo…

Microsoft’s new malware removal and anti-spyware products

This week, alongside the January security updates, Microsoft released the first version of its malware removal tool, called the Microsoft Windows Malicious Software Removal Tool (MSRT). New versions will be released on the second Tuesday of each month (with the monthly security updates) and each version will be cumulative.

Note that this is not the Microsoft Windows AntiSpyware tool (a separate beta of that product was released last week, based on the anti-spyware application gained in the purchase of Giant Company), nor is it an anti-virus tool – MSRT is simply a rollup of all the malware removal utilities that Microsoft has previously released.

Problems with certain NICs and a RIS-based Windows XP installation

After my hard disk failure last month, I decided to resurrect a project that I had shelved some time ago – implementing an unattended setup for my PCs at home. I have a variety of computers from HP (Compaq), IBM and Dell, which makes things slightly more complicated than it might otherwise be (although not impossible), so this was an opportunity to implement some of the business desktop deployment (BDD) technologies that I practise at work to implement a standard operating environment (SOE) and allow me to rebuild PCs at will.

My previous experience with unattended installations has largely been on the server side, basically amending and appending OEM installation scripts (e.g. Compaq/HP SmartStart or the HP ProLiant Essentials Rapid Deployment Pack (RDP)). This time I had a plethora of drivers to consider, and a limited (zero) budget. To allow for a repeatable, customisable, build I decided not to use any imaging technologies but instead to create a standard unattended setup, including all the drivers needed for the various PCs and a common set of applications. One thing I could rely on the presence of was Pre-boot eXecution Environment (PXE)-enabled workstations, so I set up and configured Microsoft Remote Installation Services (RIS) to serve my Windows XP + SP2 installation “image” (not really an image, but that’s the RIS terminology).

Incidentally, the most complete resource for information on creating unattended builds that I am aware of is the Microsoft Software Forum Network’s “Creating the Ultimate Unattended XP CD”. Although CD based, this gives much of the information needed for a successful RIS-based installation.

Everything was looking good until I tried to perform a PXE network service boot and connect to the RIS server. I could see that my DHCP server was issuing IP addresses to clients but they received an error:

PXE-E53: No Boot Filename Received

Basically, the PXE clients couldn’t find the RIS server. DHCP was being served from an ADSL router and I couldn’t find any way to configure the router to redirect PXE clients. Logically, interaction between the PXE client, the DHCP server and the RIS server should not have been affected by the router because PXE uses DHCP broadcast requests and all the computers were all on the same subnet but once DHCP was migrated to the RIS server, the error disappeared and the RIS Client Installation Wizard ran as expected. Since then, I’ve found Microsoft PSS’ Technical Guide to Remote Installation Services, which suggests various troubleshooting actions but for now it works, so maybe I’ll investigate further some other time.

The next issue was that Windows XP setup failed as the network drivers for the Broadcom BC570x NIC in my Dell Latitude D600 were not available from the Windows XP installation source:

The operating system image you selected does not contain the necessary drivers for your network adapter. Try selecting a different operating system image. If the problem persists, contact your administrator. Setup cannot continue. Press any key to exit.

Microsoft state that a hot fix is required to resolve this issue; however the Broadcom driver FAQ gives an alternative resolution which involves editing the B57WIN32.INF setup information file. I didn’t want to do this as it would break the digital signature and I would prefer to construct the build using signed drivers only. Instead, I used the latest drivers (v7.86) from Broadcom rather than the Dell-packaged version and once I had integrated the network drivers with the RIS installation source, deleted any instances of precompiled setup information (.PNF) files and restarted the Boot Information Negotiation Layer service, I was able to commence my unattended Windows setup.

This time, a new error halted text-mode setup:

File b57w2k.sys caused an unexpected error (21) at line 3788 in d:\xpsprtm\base\boot\setup\setup.c. Press any key to continue.

Some posts in the and MSFN forums led me to a solution for this by copying the Windows 2000 version of the drivers (B57W2K.SYS) to the Windows XP installation source \i386 folder alongside the Windows XP driver (B57XP32.SYS) and the setup information file (B57WIN32.INF).

Once the Dell PC was working, I had the same issue with an IBM ThinkPad T40 with an Intel PRO/100 VE card and so it seems logical to assume that this issue may apply to a variety of NICs.

For the BC570x, a Windows User Group (Nordic) article which discusses integration of Intel and Broadcom drivers with RIS images suggests rewriting the B57WIN32.INF file to replace all references to B57W2K.SYS with B57XP32.SYS, but again, I avoided this to prevent issues with unsigned drivers. Intel’s solution to installing PRO/100 or PRO/1000 NICs via RIS requires a further download but I got it working by applying the same resolution as for the Broadcom drivers – i.e. using IBM’s distribution of the Intel drivers (v7.0.28.0) and including the Windows 2000 E100BNT5.SYS driver in the Windows XP installation source \i386 folder for text-mode setup.

I should point out that it was only necessary to add these network drivers to the \i386 folder on the Windows XP installation source in order to use the NIC to copy files during setup and it is still necessary to add OEM device drivers to the Windows XP installation source for all undetected devices in order to allow the drivers to be used during the plug and play (PnP) section of setup.

After a couple of days downloading, integrating and testing drivers, my RIS-based Windows XP installation works for all of my computers and now I can focus on the finer points of the build, tuning the Windows XP installation and adding applications to my SOE.