For the last few days, I’ve been trying to get an ISA Server 2006 installation working and it’s been driving me nuts. I was pretty sure that I had my networking sorted, following Jim Harrison’s article on configuring ISA Server interface settings (although a colleague did need to point out to me that I didn’t have a static route defined on my ADSL router back to the ISA Server’s internal network – doh!) but even once this was checked there was still something up with the configuration.
My server has three NICs – a Broadcom NetXtreme Gigabit Ethernet card, connected to my Netgear ProSafe GS108 switch and two Intel PRO/100+ Management Adapters – one connected to a NetGear DS108 hub and the other disconnected at the moment but reserved for remote management of the server (the first two are both bound to Hyper-V) virtual switches.
The theory is that the Gigabit connection will be used for all my internal IT resources and the Fast Ethernet hub is just connected to the ADSL router. The server will run a few virtual machines (VMs) – the ISA Server (running with Windows Server 2003 R2 and connected to both virtual switches), another VM with Active Directory and DNS (also running Windows Server 2003 R2), my mail server and various test/development machines.
According to Microsoft:
“There are two rules to remember when setting up DNS on ISA Server. These rules apply to any Windows-based DNS configuration:
- No matter how many network adapters you have, only assign DNS servers to a single adapter (it doesn’t matter which one). There is no need to set up DNS on all network adapters.
- Always point DNS to either internal servers or external servers, never to both.”
Following this advice, my internal DNS Server is set to forward any requests that it can’t resolve to my ISP’s servers. The problem was that this DNS server couldn’t access the Internet through the ISA Server. ISA Server could ping hosts on all networks (so the network configuration was sound) and monitoring the traffic across the ISA Server showed the outbound DNS traffic on port 53 but nothing seemed to be coming back from the ISP’s DNS servers.
I checked another colleague’s working ISA Server 2006 configuration and found nothing major that was different (only an alternative DNS configuration – with the external NIC pointing to the internal DNS server where my external NIC has no DNS server specified – and the addition of the Local Host network in the source list for the Unrestricted Internet Access firewall access rule that is included in the Edge Firewall network template).
Then, after seeking advice from more colleagues and spending the entire day (and evening) on the problem, I finally cracked it…
Because the ISA Server was configured to use the internal DNS server for lookups (which, in turn, couldn’t get back through the ISA Server), nslookup domainname.tld didn’t work; however nslookup domainname.tld alternativednsserveripaddress did (e.g. nslookup www.google.com 4.2.2.2). HTTP(S) traffic seemed fine though – if I used IP addresses instead of domain names, I could access websites via the web proxy client.
Meanwhile, on the ISA Server, I could use nslookup for local name resolution but not for anything on the Internet. And pinging servers on the external side of the ISA server gave some very strange results – The first packet would receive a reply but not the subsequent ones.
After hours of Googling, I came across some good advice in a TechNet forum thread – download and run the ISA Server Best Practices Analyzer (BPA) tool. The ISA BPA presented me with a number of minor warnings (for example, that running ISA Server in a virtual environment can’t protect the underlying operating system) but two seemed particularly significant:
“Receive-side scaling (RSS) is enabled by the Windows Server operating system. If a network adapter installed on the local ISA Server computer supports RSS, ISA Server may function incorrectly. […]”
and:
“TCP-Acceleration (TCPA) is enabled by the Windows Server operating system. If a network adapter installed on the local ISA Server computer supports TCPA, ISA Server may function incorrectly. […]”
I made the registry edits to disable RSS and TCPA (Further details are available in Microsoft knowledge base articles 927695 and 936594), restarted the computer and crossed my fingers.
Even after this change, I still couldn’t successfully ping resources on the external side of the ISA Server from the private network, but I was sure I was onto something. I stopped looking for problems with ISA Server and DNS, and instead I focused my efforts on TCP Offload issues with Hyper-V. That’s when I found Stefaan Pouseele’s post about ISA Server and Windows Server 2003 service pack 2. Stefaan recommends not only disabling RSS and TCPA but also turning off TCP offload and the TCP chimney.
A big more googling and I found a TechNet Forum thread about ISA Server 2006 in a virtual environment where (Virtual PC Guy) Ben Armstrong and VistaGuyRay (Raymond Comvalius) had discussed disabling TCP offloading in the VM. As it happens, only yesterday, Ray blogged about how disabling TCP offloading in the virtual machine (not on the host) had resolved his problems with a Broadcom gigabit Ethernet adapter and Hyper-V (further details are available in Microsoft knowledge base article 888750). So, after making this change (but not doing anything with the TCP chimney) and a final reboot of my ISA server, I noticed that Windows wanted to apply some updates. That meant that name resolution was working, which in turn meant that the internal DNS server was successfully forwarding requests to the ISP servers via the ISA Server and my ADSL router. Result.
The final set of registry changes that I made were as follows:
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters]
"EnableTCPA"=dword:00000000
"EnableRSS"=dword:00000000
"DisableTaskOffload"=dword:00000001
I’ve only made the registry changes on the ISA Server at the moment and the VM running AD/DNS seems to be fine, so this might not be an issue for all virtual machines connected to the Hyper-V virtual switch bound to the Broadcom NetXtreme NIC. What does seem reasonably certain though is that Hyper-V, ISA Server 2006 and TCP offloading don’t play nicely together in this scenario.
Mark,
One question on this – http://www.markwilson.co.uk/blog/2008/03/problems-with-hyper-v-isa-server-2006-and-tcp-offloading.htm
What template did you use on ISA – was it the back firewall template builtin into ISA 2006 – if so, did you create a network for the perimeter section or that would have been included as part of the External network?
Your input will be very much appreciated.
Rgds,
Christian.
I had the same problem as you, but I’m still not there yet…
Whenever I try Remote Desktop connection to the virtual ISA I get 0x80074e21 FWX_E_ABORTIVE_SHUTDOWN error in the ISA log
You mind giving it a shot on your setup and post the result?
@Christian – I was using the Back Firewall template but I’m now using the Edge Firewall – certainly with the Edge template the perimeter network is treated as external (basically external is anything that’s not defined as internal).
@Dennis – I just enabled Remote Desktop on my ISA server and tried to connect to it from another machine on the internal network. The ISA log says that it was denied by the default rule, result code 0xc004000d FWX_E_POLICY_RULES_DENIED.
FWX_E_POLICY_RULES_DENIED means no policy allow it :o)
Did you add your RDP client to the ‘Remote Manager Computers’ computer set on the ISA?
Sorry for being a pain but I’d really love to see if this is a general problem with this kind of setup
Dennis – no worries – after adding my internal subnet to the Remote Management Computers computer set, I was able to initiate an RDP connection to the ISA server with no problems at all. Good news for me, but I guess not so good for you. Hope you manage to find the answer. Mark
Thank you for testing it :o) – guess it’s back to the drawing board for me then…
Ohh… and thank you for the nice guide above, I’m sure many will find it usefull :o)
Thanks Mark, you saved one life today (with this post) – I’d get a stroke or heart attack for sure ;) I tried installing multiple 2003 R2’s with terminal services under hyper-v and I couldn’t figure out what’s wrong because network performance was terrible – and it seems the registry change you proposed helped a lot. Something’s probably wrong with this TCP offloading….
Mark, I am fairly new at this. I am setting up a home network with ADSL router, hyper-v environment, with ISA 2006 + cached only DNS on Windows 2003 R2 and ADDS + DNS on Windows 2008.
On my guest ISA server, I see the following registry entries:
“EnableTCPA”=dword:00000000
“EnableRSS”=dword:00000000
But I cant find “DisableTaskOffload”. Should I add the DisableTaskOffload entry? OR Start again :-) Thanks!
I forgot to mention, I can ping external names from the client m/c on my internal ntwk. However I cannot browse to the websites for some reason – and thats what I am tying to resolve. Thanks !
Scratch that ! I added the DisableTaskOffload entry and restarted the box. And now I am able to access the internet from the internal client m/c. Thanks.
Hi Nitin – sorry it took me a while to respond (public holiday here in the UK) but glad you got it working.
i installed ISAServer 2006 for the first time using 3-leg parameter. There after my isa host containing the 3-network adapter stop browsing the internet. i was able to get some materials on net that help me escalate to the level that i could ping some domain names but could not browse. please i need ur support to resolve this issue. the 3-leg parameters are: internal(192.168.1.0-192.168.1.255), while external(internet is 192.168.1.1 as my Gateway) while DMZ is 192.168.2.0-192.168.2.255). Thanks
Mustapha
This isn’t a support website – try the Microsoft forums or isaserver.org for that – but from the info you’ve given there I’d say that the problem is that your external IP address resides on the network you have specified as internal.
HTH, Mark
whenever i want to visit the site ISA server returns fwx_e_abortive_shutdown error