Windows Server 2008 R2 Hyper-V crash turns out to be an Intel driver issue

This content is 14 years old. I don't routinely update old blog posts as they are only intended to represent a view at a particular point in time. Please be warned that the information here may be out of date.

A few weeks ago, I rebuilt a recently decommissioned server to run as an infrastructure test and development rig at home.  I installed Windows Server 2008 R2, enabled the Hyper-V role and all was good until I started to configure my networks, during which I experienced a “blue screen of death” (BSOD) – never a good thing on your virtualisation host, especially when it does the same thing again on reboot:

“Oh dear, my freshly built Windows Server 2008 R2 machine has just thrown 3 BSODs in a row… after running normally for an hour or so :-(“

The server is a Dell PowerEdge 840 (a small, workgroup server that I bought a couple of years ago) with 8GB RAM and a quad core Xeon CPU.  The hardware is nothing special – but fine for my infrastructure testing – and it had been running with Windows Server 2008 Hyper-V since new (with no issues) but this was the first time I’d tried R2. 

I have 3 network adapters in the server: a built in Broadcom NetXtreme Gigabit card (which I’ve reserved for remote access); and 2 Intel PRO/100s (for VM workloads).  Ideally I’d use Gigabit Ethernet cards for the VM workload too, but this is only my home network and they were what I had available!

Trying to find out the cause of the problem, I ran WhoCrashed, which gave me the following information:

This was likely caused by the following module: efe5b32e.sys
Bugcheck code: 0xD1 (0x0, 0x2, 0x0, 0xFFFFF88002C4A3F1)
Error: DRIVER_IRQL_NOT_LESS_OR_EQUAL
Dump file: C:\Windows\Minidump\020410-15397-01.dmp
file path: C:\Windows\system32\drivers\efe5b32e.sys
product: Intel(R) PRO/100 Adapter
company: Intel Corporation
description: Intel(R) PRO/100 Adapter NDIS 5.1 driver

That confirmed that the issue was with the Intel NIC driver, which sounded right as, after enabling the Hyper-V role, I connected an Ethernet cable to one of the Intel NICs and got a BSOD each time the server came up. If I disconnected the cable, no BSOD.  Back to the twitters:

“Does anyone know of any problems with Intel NICs and Hyper-V R2 (that might cause a BSOD)?”

I switched the in-box (Microsoft) drivers for some (older) Intel ones.  That didn’t fix things, so I switched back to the latest drivers.  Eventually I found that the issue was caused by the checkbox for “Allow management operating system to share this network adapter” and that,  if the NIC is live and I selected this, I could reproduce the error:

“Found the source of yesterday’s WS08R2 Hyper-V crash… any idea why enabling this option http://twitpic.com/11b64y would trip a BSOD?”

Even though I could work around the issue (because I don’t want to share a NIC between the parent partition and the children anyway – I have the Broadcom NIC for remote access) it seemed strange that this behaviour should occur.  There was no NIC teaming involved and the server was still a straightforward UK installation (aside from enabling Hyper-V and setting up virtual networks). 

Based on suggestions from other Virtual Machine MVPs I also:

  • Flashed the NICs to the latest release of the Intel Boot Agent (these cards don’t have a BIOS).
  • Updated the Broadcom NIC to the latest drivers too.
  • Attempted to turn off Jumbo frames but the the option was not available in the properties so I could rule that out.

Thankfully, @stufox (from Microsoft in New Zealand) saw my tweets and was kind enough to step in to offer assistance.  It took us a few days, thanks to timezone differences and my work schedule, but we got there in the end.

First up, I sent Stu a minidump from the crash, which he worked on with one of the Windows Server kernel developers. They suggested running the driver verifier (verifier.exe) against the various physical network adapters (and against vmswitch.sys).  More details of this tool can be found in Microsoft knowledge base article 244617 but the response to the verifier /query command was as follows:

09/02/2010, 23:19:33
Level: 000009BB
RaiseIrqls: 0
AcquireSpinLocks: 44317
SynchronizeExecutions: 2
AllocationsAttempted: 152850
AllocationsSucceeded: 152850
AllocationsSucceededSpecialPool: 152850
AllocationsWithNoTag: 0
AllocationsFailed: 0
AllocationsFailedDeliberately: 0
Trims: 41047
UnTrackedPool: 141544
 
Verified drivers:
 
Name: efe5b32e.sys, loads: 1, unloads: 0
CurrentPagedPoolAllocations: 0
CurrentNonPagedPoolAllocations: 0
PeakPagedPoolAllocations: 0
PeakNonPagedPoolAllocations: 0
PagedPoolUsageInBytes: 0
NonPagedPoolUsageInBytes: 0
PeakPagedPoolUsageInBytes: 0
PeakNonPagedPoolUsageInBytes: 0
 
Name: ndis.sys, loads: 1, unloads: 0
CurrentPagedPoolAllocations: 6
CurrentNonPagedPoolAllocations: 1926
PeakPagedPoolAllocations: 8
PeakNonPagedPoolAllocations: 1928
PagedPoolUsageInBytes: 984
NonPagedPoolUsageInBytes: 1381456
PeakPagedPoolUsageInBytes: 1296
PeakNonPagedPoolUsageInBytes: 1381968
 
Name: b57nd60a.sys, loads: 1, unloads: 0
CurrentPagedPoolAllocations: 0
CurrentNonPagedPoolAllocations: 3
PeakPagedPoolAllocations: 0
PeakNonPagedPoolAllocations: 3
PagedPoolUsageInBytes: 0
NonPagedPoolUsageInBytes: 188448
PeakPagedPoolUsageInBytes: 0
PeakNonPagedPoolUsageInBytes: 188448
 
Name: vmswitch.sys, loads: 1, unloads: 0
CurrentPagedPoolAllocations: 1
CurrentNonPagedPoolAllocations: 18
PeakPagedPoolAllocations: 2
PeakNonPagedPoolAllocations: 24
PagedPoolUsageInBytes: 108
NonPagedPoolUsageInBytes: 50352
PeakPagedPoolUsageInBytes: 632
PeakNonPagedPoolUsageInBytes: 54464

To be honest, I haven’t a clue what half of that means but the guys at Microsoft did – and they also asked me for a kernel dump (Dirk A D Smith has written an article at Network World that gives a good description of the various types of memory dump: minidump; kernel; and full). Transmitting this file caused some issues (it was 256MB in size – too big for e-mail) but it compressed well, and 7-zip allowed me to split it into chunks to get under the 50GB file size limit on Windows Live SkyDrive.  Using this, Stu and his kernel developer colleagues were able to see that there is a bug in the Intel driver I’m using but it turns out there is another workaround too – turning off Large Send Offload in the network adapter properties.  Since I did this, the server has run without a hiccup (as I would have expected).

“Thanks to @stufox for helping me fix the BSOD on my Hyper-V R2 server. Turned out to be an Intel device driver issue – I will blog details”

It’s good to know that Hyper-V was not at fault here: sure, it shows that a rogue device driver can bring down a Windows system but that’s hardly breaking news – the good thing about the Hyper-V architecture is that I can easily update network device drivers.  And, let’s face it, I was running enterprise-class software on a workgroup server with some old, unsupported, hardware – you could say that I was asking for trouble…

Why Windows Server User Group meetings are a bit like London buses

This content is 14 years old. I don't routinely update old blog posts as they are only intended to represent a view at a particular point in time. Please be warned that the information here may be out of date.

There’s a saying in the UK when multiples of something come along at the same time… “like London buses… nothing and then three at once” (based on the principle of bunching, for high frequency services – incidentally, that’s an alien concept where I live – we’re lucky if the bus shows up at all…).

Anyway, back to the point – hot on the heels of the Windows Server User Group (WSUG) meeting with Joey Snow, Mark Parris has arranged a second meeting to co-incide with the Microsoft UK TechDays – this time it’s on 13 April 2010 at Microsoft’s Offices in London (map and directions) and the speaker will be Dan Pearson, from David Solomon’s Expert Seminars.

Dan was formerly a Senior Escalation Lead at Microsoft, where he worked in the Windows Base OS team supporting Microsoft customers. Dan will be talking about Windows crash dump analysis as well as Windows performance troubleshooting and analysis.

Check out the event registration site for more details.

User group meeting (Windows Server User Group)

This content is 14 years old. I don't routinely update old blog posts as they are only intended to represent a view at a particular point in time. Please be warned that the information here may be out of date.

A few weeks ago, I blogged about Microsoft’s UK TechDays and mentioned that the Windows Server User Group was planning to run an evening event that week.

Now we’re ready to announce the details: whilst he was at the MVP Summit last month, Mark Parris managed to persuade Joey Snow to come along and speak to us on the evening of 12 April 2010 at Microsoft’s Offices in London (map and directions).  Joey is a technical evangelist for the Worldwide Developer and Platform Evangelism team at Microsoft focusing on Windows Server, IIS and SQL Server and he’ll be presenting on Windows Server 2008 R2’s new BranchCache functionality in as well as migrating server roles to Windows Server 2008 R2.

Check out the event registration site for more details.

Distributing camera raw files along with their development history from Adobe Lightroom

This content is 14 years old. I don't routinely update old blog posts as they are only intended to represent a view at a particular point in time. Please be warned that the information here may be out of date.

I’ve written previously about how Adobe’s photo management applications such as Bridge and Lightroom use Sidecar (.XMP) files to store details of raw file edits without affecting the original image (and how that doesn’t quite work for JPEG or TIFF images).  On my system though, I found that there were no .XMP files because I had been storing the history inside my Lightroom catalog (I’ve since adjusted the catalog settings to automatically write changes into XMP) but it’s easy enough to generate an extensible metadata platform (.XMP) file for an image by either, exporting the image and selecting Original as the format in the file settings (this will save the .XMP file alongside the raw image), or, by selecting Save Metadata to File from the Metadata menu.  Either way, the resulting .XMP will be available for use in other applications (e.g. Bridge) and can be distributed with the raw image file if further processing is to be carried out on another computer.