Troubleshooting an MS-DOS application which hangs the NTVDM subsystem in Windows XP and Windows Server 2003
I’ve been working on an intriguing (and frustrating) issue for a few weeks now and a couple of days back we finally resolved the issue.
My client has an MS-DOS (FoxPro 2.6a database) application running within an NTVDM on Windows XP. Every now and then, the application will hang - seemingly randomly. Windows XP did have service pack 2 applied, but the issue also occurs on service pack 1 PCs (I didn’t try the RTM version). Only the application hangs - it is possible to terminate the NTVDM process and carry on working as normally.
Normal actions for troubleshooting MS-DOS applications in Windows XP were not helping to resolve the issue, but the software vendor managed to narrow the issue down to FoxPro waiting for input. Occasionally, the input does not timeout and return control to the calling program - it seems that this is the root cause of the NTVDM hang. Identifying this allowed them to construct a test program which polled for input, timing out every few seconds and would reliably hang an NTVDM at a seemingly random time, but always within an hour.
Using their test program on a variety of PCs, the software vendor found that the problem was related to the Intel hyper-threading technology (my client has standardised on a version of the IBM ThinkCentre M50 which includes a single 3GHz Intel Pentium 4 processor with HT technology). Whilst disabling hyper-threading is unlikely to result in any significant performance degradation (hyper-threading only provides an average 10-20% performance gain as most applications do not fit completely with the hyper-threading model), it was still considered by IBM, Microsoft, my client and myself as a tactical workaround, rather than a strategic fix.
After seeking advice from Microsoft, I ran the test program on a Compaq ProLiant DL380 G2 server with two Pentium III 1.26GHz processors and found that the issue is not limited to Windows XP and hyper-threading, but to both Windows XP and Windows Server 2003 when running with an ACPI Multiprocessor PC HAL. Turning off hyper-threading on the PCs was no longer good enough as we can expect to see multiple processor cores constructed on a single die in the near future, leading to a rise in the use of multi-threaded applications (the logical processor provided by the hyper-threading technology in the Intel Pentium 4 processor is simply a precursor to this).
So why does an MS-DOS application running within an NTVDM on a 32-bit version of Windows use multiple processors? The answer it seems is that although the MS-DOS application is not multi-threaded, modern versions of Windows are, and can allocate parts of the NTVDM process to any available processor. With that in mind we re-ran the test program with processor affinity set to use only CPU0 in Task Manager. The results were the same as disabling hyper-threading - no NTVDM hang! Obviously, setting processor affinity manually is not sustainable outside the test environment, and short of running the application on Windows Server 2003 Enterprise Edition (with the Windows System Resource Monitor to control processor affinity) we needed to find an alternative solution.
That solution came in the form of the imagecfg.exe tool provided with the Microsoft Windows 2000 Resource Kit (supplement one). This can be used to edit an executable file and permanently set the processor affinity for a given application:
Using the imagecfg -a 0x1 c:\windows\system32\ntvdm.exe command did the trick, although Windows File Protection/System File Checker quickly restored the original ntvdm.exe file so I needed to perform this on a copy of ntvdm.exe in a temporary folder, and then overwrite both c:\windows\system32\ntvdm.exe and c:\windows\system32\dllcache\ntvdm.exe.
Once updated, the NTVDM process runs on CPU0. Of course, this limits all programs under the control of the NTVDM subsystem but it is far more preferable to disabling logical or physical processors in the BIOS; however, as this is a change to an operating system file, it must be considered alongside the implementation of any service packs and/or hotfixes from now on. Reversing the change is simply a case of restoring the original ntvdm.exe file.
Posted: 14:12 on Friday 28 January 2005 under MS-DOS, Windows Server 2003, Windows XP.
Comments: 49
RSS (for comments on this post only)Share This
Comments
Comment from Priss
Time: Saturday 19 February 2005, 15:12
hi, just wanted to ask whether there is any other known cause of ntvdm.exe cpu overload - even when running on virtual CPU 0 or modified with imagecfg tool. is there any possibility of directx 9 and ntvdm inter-un-operability? thanks a bunch for any clue.
Comment from Blazej Kotelko
Time: Tuesday 6 December 2005, 13:00
We experienced the same problems with our FoxPro/DOS based applications and wrote small a program which starts a DOS *.bat file in one-processor only mode. This does the trick! Usage for our tiny utility is:
StartHT.exe [-processor number from 0 to 9 (default 0)] name.bat
For instance:
StartHT.exe subiekt.bat
And it works!
blazej@insert.com.pl
our software: http://www.insert.com.pl/en/products.html
Comment from Anonymous
Time: Tuesday 21 February 2006, 2:12
My 16-bit app (lotus 123R5)crashes (ntdvm loads again, CPU >100%) on any print function. This app has run for years without problems. I suspect installing a new program (Business Vison32) has changed my registry to cause this but I can’t restore. Does anyone have any ideas?.. I’m desparate.
Comment from Bernard
Time: Tuesday 21 February 2006, 14:15
Thanks Mark. (Re: anonymous)I’m not aware of a Windows update and I have tried earlier compatibiltiy modes.This problem doesn’t manifest on other xps. There seems to be a unique setting in my XP that now causes Lotus print functions (and, only print) to try to load Lotus a second time.
Comment from bernard
Time: Tuesday 21 February 2006, 20:41
I think my problems may have disappeared. I made a number of changes. Unfortunately, I didn’t test after each change. However this observation might help. A few weeks ago, on adding a wireless router to my office’s small wired network, the system created a large list of everyones’ shared printers (maybe 40 printers)in my control panel. Today, I deleted all these printers except a few local and a few network printers…and now Lotus is not crashing (for now). Is it possible that this large list was causing the crash? Thanks for your earlier comments and this blog… a great comfort in this great unsupport wasteland.
Comment from Erik V. Olson
Time: Thursday 16 March 2006, 19:34
Just wanted to say thanks. Had a very similar problem with a dual-core processor PowerEdge 850, and manually setting the CPU affinity to one processor fixed it.
Comment from Anonymous
Time: Wednesday 7 June 2006, 16:26
it’s seem that this tool doesn’t work with dual-core cpu.
I’ve try it on a dell optiplex gx620 with a pentium D820 inside but my apps seem to always use the two cpu and hangs randomly.
Another idea ?
Thanks, Mathieu.
Comment from WoZ
Time: Friday 16 June 2006, 0:47
Hi, my english is not so good… so excuseme. I have an windows 2003 server with 50 terminal services users, they run a foxpro 2.6 app but is for windows (foxpro 2.6 for windows
Comment from WoZ
Time: Friday 16 June 2006, 0:53
I have a terminal server (windows 2003) with 50 clients running ERP software developed on foxpro 2.6 for windows. That software run over ntvdm.exe and hang randomly… why? I don’t now. When ntvdm hang the cpu usage is 100% and the server is very slow, the “solution” is end the ntvdm process for the hung user and the server wake up again. My server was an Athlon 64 2800+ 2 GB ram 450 GB hd. I think that an application of 1994 should run very well on that system, but is not possible yet.
I hope that you can help me with my big problem. Greetings byebye
Comment from Anonymous
Time: Friday 4 August 2006, 3:59
Does anyone know where I can find Startht.exe?
Comment from Anonymous
Time: Sunday 24 September 2006, 17:53
FOR CLARIFICATION:
This problem does NOT occur on Windows 2000 (HT, Dual CPU, or Dual Core, etc). Only XP and 2003 are affected!
HOW TO REPRODUCE THE PROBLEM QUICKLY:
Launch Foxpro:
FOXPROX.EXE [ENTER]
Now move the Command Window around in a circle with the mouse. In under 30 seconds:
HANG! (usually takes under 10 seconds on my X2 4400+)
Try the same thing under Windows 2000:
No problems! (I just tried this on a dual cpu Xeon system with hyperthreading enabled, W2K server, and a W2K P4 HT system).
XP Blows.
Comment from Jeff
Time: Wednesday 27 September 2006, 21:08
Hi, I tried Active+ runfirst utility, but it does not appear to help with Foxpro. ntvdm still runs on both CPUs.
I am trying to use imagecfg with Terminal Server 2003, but Windows keeps on resetting the file even though I modified system32 & dllcache. I even put an entry in i386. Any ideas?
Comment from Anonymous
Time: Friday 20 October 2006, 12:08
Thanks for the assistance with this problem. I’ve been driven mad by this for days.
Our issues were resolved by running Foxpro with the -X switch instead of the +X which loads an extended version into memory, .25Gb of it in our instance.
Also, utilising NTVDM version 5.1.2600.1106 resolved all issues without changing +X or affinity.
I think this version is from SP1 but don’t quote me.
Thanks,
Martin Hobson - NHS IT Manchester
Comment from Anonymous
Time: Tuesday 21 November 2006, 14:46
Does anyone know if copying the NTVDM.EXE file from a Windows2000 machine to an XP machine will help?
Comment from Vikas Aggarwal (midtowngrand@gmail.com)
Time: Thursday 23 November 2006, 9:09
Thanks a lot for solving this one of my long pending headache.
I have not been able to solve it since i got myself a new pc with Intel 945 Motherboard. Most of my applications are in Foxpro 2.6 for Dos and I had to run them on Windows 98.
At last I could fix the problem by reading your beautifully explained issue.
Hats off to u.. Mark !!
Vikas
Comment from JHC
Time: Thursday 8 February 2007, 20:54
Even after copying ntvdm.exe to a temp folder, making the change there, and replacing the files in System32 and DLLCache, I still see in Event Viewer that Windows File protection has overwritten the files, and no change to my CPU Usage stats - immediately to 100% after launching the ntvdm process.
Is there a certain way I should perform this in a temp folder and do the overwrite other than copy/paste?
Thanks so much for this post.
Windows Server 2003
Comment from EDDIE
Time: Tuesday 13 March 2007, 20:16
Hey JHC!
You need to shut down th Systems Restore Capability, reboot the machine, apply the affinity w/imagecfg. Delete the other copy in DLLCache and then put back the System Restore. This will do it.
Comment from Mark C
Time: Wednesday 11 April 2007, 1:31
This was a great article - But. I tried to implement this kludge on Windows Vista.
1) FP2.6 DOS was crashing, corrupting data (yay!)
2) Windows Resource Protection quite successfully prevented me from changing ntvdm.exe
The simple fix: I found the START command in Vista and W2k3 server (Not sure about XP):
— display —
C:\Documents and Settings\Administrator>start /?
Starts a separate window to run a specified program or command.
START [”title”] [/D path] [/I] [/MIN] [/MAX] [/SEPARATE | /SHARED]
[/LOW | /NORMAL | /HIGH | /REALTIME | /ABOVENORMAL | /BELOWNORMAL]
[/AFFINITY ] [/WAIT] [/B] [command/program]
[parameters]
(more explanation…)
— end display —
… so my batch file looks something like:
start “my title” /D “C:/MyFoxApp” /AFFINITY 0×1 C:/FP/FOXPROX.EXE “C:/MyFoxApp/app.exe”
Comment from fmcgowan
Time: Friday 1 June 2007, 19:09
I am having similar problems with FPD apps that ran correctly for years on W2k. Now on XP, they are apparently updating the CDX but not the DBF on data updates.
Though “Start” does exist on XP, the “affinity” keyword is not supported… too bad, it could have made life much simpler.
Comment from Anonymous
Time: Sunday 3 June 2007, 3:12
Update:
Using any version of ntvdm.exe from XP or 2003 does NOT work. The only solution is set the processor affinity to one cpu or: upgrade to Windows 2000.
If you are using a telnet server such as Georgia Softworks to run FPD apps. Then you have no choice: Windows 2000 Server. Locking the affinity down to one CPU practically defeats the purpose of a telnet server.
Facts:
- This affects Foxpro for DOS Extended 32 Bit (FOX +x, FOXPROX)
- It does not affect regular FPD 16 bit apps (your apps might run with the -x switch, but they will run slower, or may not run at all; if your FPD apps were developed to run using the 32 bit extender like mine have, and no 16 bit testing, I guarantee you will have problems)
- This problem has persisted for over 5 years, Microsoft obviously doesn’t care and won’t fix it!
- For multiuser apps, lots of folks are using DOSEMU running under Linux
To reproduce the problem:
Launch Foxpro extended 32 bit: FOXPROX.EXE [ENTER]
Move the Command Window around in a circle with the mouse. In under 30 seconds the app will hang causing 100% cpu utilization on one processor.
Comment from Norman Eastwood
Time: Tuesday 26 June 2007, 22:48
ok, I have tried to follow this thread, but must be too slow. I too have an old fox pro DB that runs well on Win 2000 sp4, but hangs, crashes, or i/o errors out in three other XP machines. I don’t know if the processors are dual processors, but they are older machines so I don’t think so. 1), What would be the work around for a non-dual processor machine and 2), asuming I want to try the March 13 solution of making changes to the NTVDM file and copying back into the OS, what changes do I need to make to the file, and how do I disable windows defenses so I can copy ( and it will stay ) back into the os? thanks
Comment from Bruce
Time: Monday 16 July 2007, 18:04
Mark,
Mainly wanting to say THANK YOU VERY MUCH for this information. I have some screens I still need to modify with FoxPro for DOS.
In case this may be useful to others:
I couldn’t find the imagecfg.exe readily on the link above, but this one has it:
http://www.robpol86.com/Pages/imagecfg.php
Also, the c:\Windows\System32\dllcache folder seems very well hidden. Even with View options set to show it, I could not find it via Windows Explorer. So I got to it via the Command prompt.
Oh yeah, the fastest way I found to test for the problem is similar to what was said above, though slightly different. Open a screen via:
MODIFY SCREEN ANYNAME
Then hold the mouse key down and drag its “selection” box back and forth - it freezes for me in seconds without the above fix.
I am running XP Pro SP2 with an AMD X2 4200+.
Thanks again for these instructions as they helped me resolve some serious issues.
Comment from Terry
Time: Sunday 5 August 2007, 22:20
My DOS based Lotus123 worked AOK in Windows 98SE and XP. Now in Windows Vista, trying to load a “large” (only 260K) .WK1 file causes “Memory full” error message. Opening 123 in Compatibility Mode for 98SE or XP fails to resolve this. Suggestions for a fix?
Comment from Kent
Time: Friday 17 August 2007, 18:30
I tried the imagecfg patch. That worked ok on a temp copy of NTVDM.EXE but I can not get it to stick in the c:\windows\system32 folder. I turned of System Restore in the My Computer / System Restore page and rebooted but it still will not let me copy the patched file in the c:\windows\system32 folder.
What is the trick to get the patch to stick ?
Thanks,
Kent
Comment from Kent
Time: Friday 17 August 2007, 20:01
I have solved the sticking problem without having to turn off system restore. I updated the c:\i386\ntvdm.exe with the patched version first, then i was able to update the c:\windows\system32\ntvdm.exe and the patch sticks. When I checked my program in the task manager it shows it running on only one cpu.
Thanks again for your help,
Kent
Comment from Naish
Time: Thursday 18 October 2007, 12:41
Mark,
I’m another one writing to thank you so much for this very valuable info.
I support a Fox2.6 DOS application on a number of different PC’s, and had been having problems with the application hanging on newer PC’s.
I tried your suggestion of running imagecfg on the ntvdm.exe, and XP kept putting it back to it’s old state.
When I followed Kent’s suggestion (17-Aug-07), and moved c:\i386\ntcdm.ex_ away, the modified version of ntvdm persists and all is well. MARVELLOUS !!
I have also tried to resolve the same problem on Vista, using MarkC’s entry (11-Apr-2007) ie using the start command.
This works fine from a command prompt, and resolves the hanging problem. Unfortunately, I don’t think the start command is allowed in a PIF file.
I need to invoke the program through a PIF file, so I can specify my own Config file for the app to increase FILES, and also to configure the Extended Memory.
I wonder if you have any ideas.
Once again
Thank you
Comment from Mark Wilson
Time: Friday 19 October 2007, 10:15
.PIF files - now we’re going back into the deep recesses of my mind 
The only thing I can think is to have a chain of files so that the .PIF is used to launch another batch file which then calls the start command. It’s a bit messy but it may work.
Comment from Lucian
Time: Friday 11 January 2008, 8:27
I have a similar problem trying to run a dbase program in windows 98, which is in a virtual pc on a windows xp machine. The problem appears when i move the mouse or press the keyboard. I found on some sites it could be a problem of input polling/grabbing. This solution of yours does not applies to me, as i don;t have either hyper or multi processors.
Comment from Sujit Kumar
Time: Thursday 17 January 2008, 10:51
Hi.
I have an application which runs properly on the Win 2000. But when I tried to run on XP Dual Processor machine, it simply hangs after 4 transactin. But after changing its affinity to one of the processor application works till 45 transactions and later hangs. Can any body tell me what is the solution for this problem. We are running MFC application on XP and Win 2k.
Regards,
Sujit kumar
Comment from Anonymous
Time: Tuesday 5 February 2008, 22:15
The solution:
Convert one site, 18 users to Linux/DOS EMU/Samba/Putty - Success.
Convert one site, 7 users to Linux/DOS EMU/Samba/Putty - Success.
Convert large multi site, total 61 users Linux/DOS EMU/Samba/Putty - Success.
Nine sites to go, 144 users.
All because: Microsoft will not fix the problem.
This solution is very nice, but not easy to setup. The linux kernel handles these apps surprisingly well. Better than NTLM as it doesn’t have that CPU hogging issue which slows the system down and makes your keystrokes unresponsive (unrelated to the lockup issue). Network file access is also slightly faster.
Comment from Manokar
Time: Monday 11 February 2008, 0:08
Hi,
Pl. try out this, Make a DOS Batch file to run the Foxprox Exe file and make check the Compatibility Mode (Run in ) for Win98SE/Millenium will solve this problem.
Comment from Anant Jain
Time: Saturday 17 May 2008, 14:57
I am using foxpro 2.6a (Dos Based) when system running in any process contineously then it goes to hang,
Comment from Imtiaz
Time: Thursday 22 May 2008, 8:28
The imagecfg is a good tool to set processor affinity, CAN ANY ONE
KINDLY HELP ME TO USE INTERNET EXPLORER ALWAYS ON SINGLE CORE or help permanently disable one core in a C2D Processor
Comment from Alex
Time: Monday 16 June 2008, 14:33
Here is a simple & elegant solution:
http://mx.esc.ru/~assur/soft/vaffinity/
No binary patching, no startup script changes, random CPU selection for multiple ntvdm instances. Sources included.
Comment from Sharon
Time: Friday 20 June 2008, 8:42
I am working with Foxpro 2.5a X (DOS) under Win98, XP Prof, and with many different CPU’s types. Nowadays my PC is a Quadcore Dell.
Others are DualCore.
Even when I experience troubles (from time to time) they’re most focussed in ACTIVATE WINDOW sentences and when TRACING (debugging) the program (just in some PC’s). Recently I’ve experienced more troubles with debugging under QuadCore CPU but it seems that working with FULL SCREEN option reduces considerably the trouble.
Anyway, we are working with up to 10 PC, all XP Prof. and just some of them, from time to time, get hanged. I would say 1 PC hangs 6 times per year. I start with a .bat with C:\FOXPRO\FOXPROX BEGIN.
HTH
Sharon
Write a comment
Please note the rules for comments and the privacy policy and data protection notice. I'm sorry but, because not everyone sticks to the rules, I've had to implement some spam prevention measures - if you're experiencing difficulties leaving a comment, please let me know.