Migrating DHCP databases between Windows servers

One side effect of rebuilding the server that runs pretty much everything on my home network was that I had to migrate the DHCP database (twice – first to a virtual machine operating as a temporary server, and then back to the original hardware after it had been rebuilt).

I knew that it was possible (I did it from NT 4.0 to Windows 2000 for a client few years back) but hadn’t done it recently.

It turned out to be pretty straightforward – all of the details are in Microsoft knowledge base article 325473 but basically on the source (Windows 2000 Server) server, stop the DHCP service and use jetpack.exe to tidy up the database, then use the DHCP database export/import resource kit tool (dhcpexim.exe) to dump the database and finally import it on the target (Windows Server 2003) server using the network shell (netsh.exe). The second migration was even quicker – for a Windows Server 2003 source and target it just involves a couple of netsh commands. Finally, don’t forget to disable redundant DHCP services (or deauthorise the servers in Active Directory) to prevent multiple DHCP servers from servicing clients simultaneously.

Using netsh to set multiple DNS server addresses in Windows

During my recent two days of torment caused by a flaky Java application, I had to change the preferred and alternate DNS server entries for one of my network cards. Ordinarily that would be simple, but with an unresponsive Explorer interface refusing to open any network connection dialogs I needed to do it from the command line.

Enter the network shell (netsh) – a fantastic command line utility that has sneaked into recent versions of Windows and seems to have more and more functionality added with each new release.

After entering the netsh shell, interface ip got me to the TCP/IP interface settings; then show dns gave me the details of the current DNS servers; set dns "Local Area Connection" ipaddress allowed me to set the preferred DNS server and add dns "Local Area Connection" ipaddress index=2 set the alternate DNS server (that was the difficult one to work out – I had tried to set dns with a list of IP addresses but that does not work!); finally, exit the network shell and type ipconfig -all to check settings the normal way.

I love the command prompt!

This is why I’m not a fan of Java

I just wasted 2 days (one of which was on my weekend), and a lot of sleep, trying to work out why I couldn’t upgrade the Windows 2000 server which looks after my domain, DHCP, RIS, SUS and a whole load of other bits at home.

Every time I tried to run Windows Server 2003 setup it seemed to hang – and everything else was pretty slow too. I had to launch control panel applets using their .cpl filenames (e.g. appwiz.cpl for the Add or Remove Programs applet) and services would not stop cleanly.

I decided that my system was badly broken and quickly built a virtual machine on another piece of hardware, promoting that to a domain controller to provide a live backup of Active Directory. As in-place upgrades weren’t working, I resigned myself to the fact that I was going to have to migrate everything to the virtual server, then rebuild the original box but I wanted to cleanly remove the original domain controller from the directory.

Every time I ran the Active Directory installation wizard (dcpromo.exe) it failed – usually with the following error.

Active Directory Installation Failed

The operation failed because:

Failed to prepare for or remove the sysvol replication “The file replication service cannot be stopped.”

(Even though logged events with IDs 13502 and 13503 suggested that the FRS had indeed stopped).

Microsoft knowledge base article 332199 led me to try the dcpromo /forceremoval command but that failed in exactly the same way. I ran dcdiag /s:localhost on each server to look for any issues, checked that each server could ping the other one, that net view \\servername returned a list of shares, and all required DNS entries were present. I checked the DNS settings (to make sure that each server was using itself as the primary DNS server and the other domain controller as a secondary) and restarted just to be sure but all to no avail.

To cut a long story short, I found the answer purely by fluke. I couldn’t get the DHCP server service to stop cleanly (to let me migrate the database to my virtual machine) so I did a Google search for “windows services hang on stop”. This turned up a TechRepublic thread titled APC Java issues cause services to hang. I realised that I do have an APC UPS attached to the server, and that I was using a version of PowerChute Business Edition (PBE) that had been sitting there happily for a couple of years (v6.2.2) – I hadn’t upgraded to 7.x as recommended by APC knowledge base article 7202 because APC had never e-mailed me to notify me of a problem and services that aren’t broken (and that don’t have an inbuilt patching mechanism) generally get left well alone on my systems!

Lo and behold, the APC services had hung on startup and there were various events logged with ID 7022 (the APC PBE Agent service hung on starting). I disabled both the APC PBE client and server services, using the registry (as the services console was inoperable) to locate HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\servicename\ and set Start to 0x00000004 for disabled (0x00000002 is automatic and 0x00000003 is manual), restarted the server and had the fastest boot sequence in days! My Windows installation was responsive again and I was able to remove the offending applications in a few short clicks.

My problems were nothing to do with Active Directory, DNS or even Windows – they all boiled down to an expired Sun Java Runtime Environment (JRE) certificate and sloppy coding from APC which meant that if their services hung, then so did all subsequent ones. I’ve never been a fan of Java applications on Windows – generally they are slow and have a poor user interface – and this experience has done nothing to change my mind.

Once the APC PBE agent, client and server had been removed, I was able to successfully (and cleanly) demote the original domain controller (avoiding having to follow the steps in Microsoft knowledge base article 216498 to remove data left in the directory after an unsuccessful demotion) but having migrated all the services to my virtual machine, I decided to go ahead and perform a clean installation of Windows on the original hardware anyway. I’m currently mid-way through patching the rebuilt server but I’m so glad that P McGrath from Rocky Mount, VA posted his experience on TechRepublic and Google did it’s thing.

Remind me again – how did we ever manage to find things out before we had the web?

My two minutes of fame

I was on the telly today!

It all started a week or so back, when I read my regular e-mail up date from the BBC Working Lunch programme and saw this:

WORKING LUNCH NEEDS YOU!
Some of you have told us about incidents of falling slightly into overdraft and then being hit by disproportionate penalties from your bank. If you’re in the same boat we want to hear from you.

I had exactly that experience a few weeks back and so I dropped them a quick e-mail, never expecting to hear anything more. Then this morning, I got a call from one of the producers – they wanted to interview me and asked if I could come in to a local studio for a live link up!

After a couple of quick calls to clear it with my bosses (nothing about IT, no links to my employer, only an hour out of my day – call it my lunch break), I was off to the BBC’s Northampton studios. Everything seemed to go okay although I was thinking that I probably sounded like a right bumbling fool because the link from Northampton to London went via Cambridge and Norwich making my voice echo in my earpiece (which is really distracting). Then when I got home I saw this on the Working Lunch website [my underlining]:

Screen shot from the BBC Working Lunch website

“One viewer’s unfair bank charges” – that’s little Me! I thought they’d have loads of stories and I’d be the good news one because First Direct did at least drop the extortionate £105 they charged us for a minor error on our part. Imagine my surprise when I was featured in the very first piece on today’s show!).

None of this is anything to do with technology – but it did make me happy! The next bit is the techie thing (and hence the reason for blogging it here)…

Of course, I recorded the programme but only on VHS cassette which is not fantastic quality so I decided to find out how to get the online version down to my PC (the show is available on the web for 24 hours after broadcast, but only as a Real Media stream). Thanks to the advice on Swen’s Blog, I have a copy of my two minutes of fame to keep for all time (although I still need to convert the .RM file to something which doesn’t need a bug-ridden piece of spyware to read it).

One view on organising digital photos

I’m really bad at organising my digital photos. I’m paranoid about losing irreplaceable photos (like the ones of my son as he grows up fast) and end up copying them to a variety of locations, but I’m not as smart about it as I should be.

In this week’s Connected Home Media update, Paul Thurrott writes about getting organised with digital photography using nothing more than the features within Windows XP. Not everyone will agree that this is how to do it (especially those with a commitment to third party applications which work for them) but its an interesting read, and will probably work for anyone who doesn’t have massive requirements for their digital workflow.

Is the release of Windows XP service pack 3 imminent?

I heard a rumour yesterday that Windows XP service pack 3 (SP3) will be released soon. I haven’t been able to substantiate it yet, but it does sound plausible.

It’s been over a year since SP2 was released and, because it was such a massive update, many organisations still haven’t adopted it because they are worried about the impact that the new security features will have on their infrastructure.

I guess because we’ve heard so little about SP3, when it arrives we can expect it to consist basically of a rollup of hotfixes since SP2, plus SP2 itself; but its arrival will mean is that, based on the Microsoft support lifecycle, SP1 will effectively be out of support. Now’s the time to think seriously about SP2/3 adoption!

Windows Vista Product Overview for IT Professionals

Although I have Windows Vista beta 1 installed on one of my computers, I haven’t spent as much time looking at it as I would like to (although I will say that I don’t think much of the new Aero interface – it all feels a bit too much like the Linux KDE desktop to me – but I guess many people I know don’t like the Windows XP Lunar interface and elect to run in classic mode so maybe I’m just turning into a dinosaur too…).

For those who don’t have access to the beta, or just anyone who wants an overview of what Windows Vista should bring us next year, Microsoft have a Windows Vista product overview for IT professionals on their website.

A couple of useful tools for AD administrators

I just read in this week’s Windows IT Pro Windows Tips and Tricks Update about a couple of little-known tools from Microsoft which could potentially make life easier for many Active Directory (AD) administrators.

The remote control add-on for Active Directory users and computers (rcontrolad.exe) is a small add-on that provides an administrator with the option to right-click a computer account in the AD Users and Computers console and opening a terminal services/remote desktop connection to that computer. Remote control relies on the remote desktop connection software within Windows. Further information can be found at Windows IT Pro.

The limit logon tool (details at Windows IT Pro) can be used to limit the number of concurrent sessions which a user can maintain.

Symantec and Veritas – after the merger

Symantec/Veritas merger completion

Last December, I blogged about the merger between Symantec and Veritas. Then, a couple of weeks ago, I got the chance to see Mark Seager, Symantec‘s VP Technology (EMEA), present about the new organisation. Apologies if what follows appears to be a marketing plug for Symantec, but bear in mind where the information came from – I still think it makes some valid points.

Symantec’s view is that information is the “fuel” driving the global economy. Often, this information is irreplaceable and the IT department is the its custodian. According to the UK Department of Trade and Industry, 70 percent of organisations that experience serious data loss go out of business within 18 months.

Symantec quotes the following fast facts:

  • A University of California at Berkeley study suggests that we will create more data in the next 3 years than we did in the last 40,000.
  • The number of Internet users is expected to triple between 2001 and 2007 to 1.5 billion.
  • It is estimated that corporate data storage requirements are doubling every six to nine months and the resulting cost of managing new storage is five to seven times the price of the storage.
  • In the second half of 2005, the average time between the disclosure of a vulnerability and the release of an associated exploit was 6.0 days.

On the surface, some of these statistics may seem a little unbelievable (after all they do originate from a vendor of security and storage management products) but taking the data growth statistic, consider the growth in broadband Internet services and the mobile phone operators who have reached complete market saturation but still have huge costs to cover for third generation (3G) mobile phone licenses. The networks need to get users to transfer to their 3G networks and to do that they need a killer application, for example live TV. Even on the reduced-size screen of a mobile handset, that represents a lot of data.

Furthermore, network managers used to look at securing the perimeter network but nowadays that perimeter doesn’t exist. Remote users with VPN connections and mobile users with data on portable devices mean that security has to be all-pervasive. Combined with the advances in the incidence of social engineering (including phishing attacks), the security landscape is shifting.

Symantec have traditionally looked at risk management from a security management perspective (i.e. when information is unsecured, business is at risk). The Veritas approach was around failure management – whether it was environmental, component, or human error (i.e. when information is unavailable, business is at risk). Bringing together the two organisations makes a lot of sense, with significant synergies but very little product overlap. The new strategy is that when failure occurs, security management processes take over.

Worldwide, there are three areas in particular where pressures are having an increasing (and significant) effect on businesses: regulatory compliance; operational requirements and security threats. Compliance has to be demonstrable. IT operations are under pressure to drive out extra costs (like security tools for threat management) and IT is often inefficient, built on 3 or 5 year growth plans and siloed for a particular application, leading to typical storage utilisation of just 50% and only 20% CPU utilisation. By comparison, imagine what would happen if an organisation’s office space was purchased using a similar model of keeping it half empty to allow for growth!

The result is ever-greater demands on the IT infrastructure at the same time as a need to drive out cost. What is needed is a dynamic IT infrastructure.

Seager discussed the concept of an “electronic chain” of information from the user/client, through the gateway, network and servers, to the application, with its database and associated storage. This may be replicated many times over within an organisation or with different customers, suppliers and partners. This “information stack” needs to be secure, available and performant. Furthermore, it needs to support operational requirements (consider a a bank ATM – a typical customer doesn’t care that the back-end system is 99.999% available – they just need enough ATMs to be available at a particular time so that they can withdraw money without queuing).

What if…

  • …an external threat alert could trigger an internal assessment?
  • …internal audit correlated with inelegance for patch management?
  • …external intelligence could prompt more frequent backups, end-to-end from remote user to data centre?
  • …performance issues could be proactively addressed (e.g. network storms, system issues, human errors, system vulnerabilities), in-plan (not on-overtime)?
  • …early warning could trigger failover to a secure network?
  • …a compromised system could automatically be recovered?
  • …all of these actions where audited to show compliance with company standards?

Symantec claim to be able to meet this through products in four segments that cross the information stack:

  • Security infrastructure and management tools.
  • Storage management capabilities to ensure that information is continuously available.
  • Data management solutions to reduce the risk of downtime.
  • Application service management to allow dynamic service provision.

All of this is wrapped up by intelligence – what Symantec refer to as insight – from the combined experience of Symantec and Veritas with a worldwide capability of:

  • 5 security operations centres.
  • 81 monitored countries.
  • 28 support centres.
  • 20000 sensors in 180 countries.
  • 8 security response labs.

Of course, there is also a healthy dose of reality required here – if an expenses policy didn’t restrict me to certain expectations when travelling on business I would always stay in the penthouse suite at a luxury hotel and have a fantastic meal at the best restaurant in town but the reality is a probably more like a standard room at a normal business-class hotel, with a curry from the local Indian restaurant. Likewise, the level of information protection for an organisation’s IT infrastructure has to be selected based on realistic requirements and in line with budget constraints.

The integration of Symantec and Veritas has now started, with a three stage plan:

  • Stage 1 is to ensure interoperability between Symantec and Veritas products, ensuring that all of the technologies offered work together and developing solutions which combine services and technologies from across the portfolio. No products are classified as “end of life” (even though some have alternative views on the same issues).
  • Stage 2 will ensure that common components are used and that there is consistency across the product set, focusing on key areas of integration and identifying the product areas that will deliver the most immediate synergies (common user interface, common licensing terms, common installation, LiveUpdate integration, integrated support infrastructure, product-to-product integration).
  • Stage 3 is about new value – through deeper technology integration but also integration in other aspects of customer relationships such as support offerings, and license management.

Symantec now claims to be able to deliver an end-to-end solution to “keep your business up, running and growing, no matter what happens”. They use an e-mail scenario as an example, controlling unsolicited commercial e-mail (UCE), managing data volumes and ensuring system availability (as shown in the diagram below) but a similar model could be applied to many enterprise applications.

E-mail security

The Symantec Internet security threat report

Earlier today, I downloaded the Eighth Edition of the Symantec Internet Security Threat Report. Published twice a year, this report highlights trends in the Internet security space and the following list highlights some of the key findings (according to Symantec).

Vulnerability trend highlights:

  • Symantec documented 1,862 new vulnerabilities, the highest number since Symantec started tracking vulnerabilities in six-month increments.
  • The time between the disclosure of a vulnerability and the release of an associated exploit was 6.0 days.
  • The average patch-release time for the past 6 months was 54 days. This means that, on average, 48 days elapsed between the release of an exploit and the release of an associated patch.
  • 97% of vulnerabilities were either moderately or highly severe.
  • 73% of reported vulnerabilities this period were classified as easily exploitable.
  • 59% of vulnerabilities were associated with web application technologies.
  • 25 vulnerabilities were disclosed for Mozilla browsers and 13 for Microsoft Internet Explorer.

Attack trend highlights:

  • For the fourth consecutive reporting period, the Microsoft SQL Server Resolution Service Stack Overflow Attack was the most common attack, accounting for 33% of all attacks.
  • Symantec sensors detected an average of 57 attacks per day.
  • TCP port 445, commonly implemented for Microsoft file and printer sharing, was the most frequently targeted port.
  • Symantec identified an average of 10,352 bots per day, up from 4,348 in December 2004.
  • On average, the number of denial of service (DoS) attacks grew from 119 to 927 per day, an increase of 679% over the previous reporting period.
  • 33% of Internet attacks originated in the United States, up from 30% last period.
  • Between January 1 and June 30, 2005, education was the most frequently targeted industry followed by small business.

Malicious code trend highlights:

  • Symantec documented more than 10, 866 new Win32 virus and worm variants, a 48% increase over the second half of 2004 and a 142% increase of the first half of 2004.
  • For the second straight period, Netsky.P was the most reported malicious code sample. Gaobot and Spybot were the second and third most reported, respectively.
  • Malicious code that exposes confidential information represented 74% of the top 50 malicious code samples received by Symantec.
  • Bot-related malicious code reported to Symantec made up 14% of the top 50 reports.
  • 6,361 new variants of Spybot were reported to Symantec, a 48% increase over the 4,288 new variants documented in the second half of 2004.

Additional security risks:

  • Adware made up 8% of the top 50 reported programs, up from 5% in the previous reporting period.
  • Eight of the top ten adware programs were installed through web browsers.
  • Six of the top ten spyware programs were bundled with other programs and six were installed through web browsers.
  • Of the top ten adware programs reported in the first six months of 2005, five hijacked browsers.
  • Messages that constitute phishing attempts increased from an average of 2.99 million per day to approximately 5.70 million messages.
  • Spam made up 61% of all email traffic.
  • 51% of all spam received worldwide originated in the United States.

Some interesting (and some frankly frightening) statistics there. Definitely worth a read for any network administrator or IT manager.