markwilson.it [formerly known as Mark's (we)Blog]

For the last few days, I’ve been trying to get an ISA Server 2006 installation working and it’s been driving me nuts. I was pretty sure that I had my networking sorted, following Jim Harrison’s article on configuring ISA Server interface settings (although a colleague did need to point out to me that I didn’t have a static route defined on my ADSL router back to the ISA Server’s internal network - doh!) but even once this was checked there was still something up with the configuration.

My server has three NICs - a Broadcom NetXtreme Gigabit Ethernet card, connected to my Netgear ProSafe GS108 switch and two Intel PRO/100+ Management Adapters - one connected to a NetGear DS108 hub and the other disconnected at the moment but reserved for remote management of the server (the first two are both bound to Hyper-V) virtual switches.

The theory is that the Gigabit connection will be used for all my internal IT resources and the Fast Ethernet hub is just connected to the ADSL router. The server will run a few virtual machines (VMs) - the ISA Server (running with Windows Server 2003 R2 and connected to both virtual switches), another VM with Active Directory and DNS (also running Windows Server 2003 R2), my mail server and various test/development machines.

According to Microsoft:

“There are two rules to remember when setting up DNS on ISA Server. These rules apply to any Windows-based DNS configuration:

• No matter how many network adapters you have, only assign DNS servers to a single adapter (it doesn’t matter which one). There is no need to set up DNS on all network adapters.
• Always point DNS to either internal servers or external servers, never to both.”

[Configuring DNS Servers for ISA Server 2004]

Following this advice, my internal DNS Server is set to forward any requests that it can’t resolve to my ISP’s servers. The problem was that this DNS server couldn’t access the Internet through the ISA Server. ISA Server could ping hosts on all networks (so the network configuration was sound) and monitoring the traffic across the ISA Server showed the outbound DNS traffic on port 53 but nothing seemed to be coming back from the ISP’s DNS servers.

I checked another colleague’s working ISA Server 2006 configuration and found nothing major that was different (only an alternative DNS configuration - with the external NIC pointing to the internal DNS server where my external NIC has no DNS server specified - and the addition of the Local Host network in the source list for the Unrestricted Internet Access firewall access rule that is included in the Edge Firewall network template).

Then, after seeking advice from more colleagues and spending the entire day (and evening) on the problem, I finally cracked it…

Because the ISA Server was configured to use the internal DNS server for lookups (which, in turn, couldn’t get back through the ISA Server), nslookup domainname.tld didn’t work; however nslookup domainname.tld alternativednsserveripaddress did (e.g. nslookup www.google.com 4.2.2.2). HTTP(S) traffic seemed fine though - if I used IP addresses instead of domain names, I could access websites via the web proxy client.

Meanwhile, on the ISA Server, I could use nslookup for local name resolution but not for anything on the Internet. And pinging servers on the external side of the ISA server gave some very strange results - The first packet would receive a reply but not the subsequent ones.

After hours of Googling, I came across some good advice in a TechNet forum thread - download and run the ISA Server Best Practices Analyzer (BPA) tool. The ISA BPA presented me with a number of minor warnings (for example, that running ISA Server in a virtual environment can’t protect the underlying operating system) but two seemed particularly significant:

“Receive-side scaling (RSS) is enabled by the Windows Server operating system. If a network adapter installed on the local ISA Server computer supports RSS, ISA Server may function incorrectly. […]”

and:

“TCP-Acceleration (TCPA) is enabled by the Windows Server operating system. If a network adapter installed on the local ISA Server computer supports TCPA, ISA Server may function incorrectly. […]”

I made the registry edits to disable RSS and TCPA (Further details are available in Microsoft knowledge base articles 927695 and 936594), restarted the computer and crossed my fingers.

Even after this change, I still couldn’t successfully ping resources on the external side of the ISA Server from the private network, but I was sure I was onto something. I stopped looking for problems with ISA Server and DNS, and instead I focused my efforts on TCP Offload issues with Hyper-V. That’s when I found Stefaan Pouseele’s post about ISA Server and Windows Server 2003 service pack 2. Stefaan recommends not only disabling RSS and TCPA but also turning off TCP offload and the TCP chimney.

A big more googling and I found a TechNet Forum thread about ISA Server 2006 in a virtual environment where (Virtual PC Guy) Ben Armstrong and VistaGuyRay (Raymond Comvalius) had discussed disabling TCP offloading in the VM. As it happens, only yesterday, Ray blogged about how disabling TCP offloading in the virtual machine (not on the host) had resolved his problems with a Broadcom gigabit Ethernet adapter and Hyper-V (further details are available in Microsoft knowledge base article 888750). So, after making this change (but not doing anything with the TCP chimney) and a final reboot of my ISA server, I noticed that Windows wanted to apply some updates. That meant that name resolution was working, which in turn meant that the internal DNS server was successfully forwarding requests to the ISP servers via the ISA Server and my ADSL router. Result.

The final set of registry changes that I made were as follows:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters]
"EnableTCPA"=dword:00000000
"EnableRSS"=dword:00000000
"DisableTaskOffload"=dword:00000001

I’ve only made the registry changes on the ISA Server at the moment and the VM running AD/DNS seems to be fine, so this might not be an issue for all virtual machines connected to the Hyper-V virtual switch bound to the Broadcom NetXtreme NIC. What does seem reasonably certain though is that Hyper-V, ISA Server 2006 and TCP offloading don’t play nicely together in this scenario.