Ozzie’s Groove is snapped up by Microsoft

Posted on By Mark Wilson

In my recent post which discussed the perils of blogging I linked to Ray Ozzie’s Weblog . This week, I was interested to read that Ray Ozzie was actually the creator of Lotus Notes and that his company, Groove Networks is to be acquired by Microsoft (who have long since been investors in the firm) and integrated into Microsoft’s Information Worker unit.

The Windows IT Pro magazine network WinInfo Daily Update reports that the peer-to-peer and authentication technologies from Groove’s collaboration products will be integrated into the next generation of Windows, (codenamed Longhorn).

As for Ray Ozzie, he will become one of Microsoft’s chief technology officers, reporting directly to Bill Gates.

New e-mail message continuity services

Posted on By Mark Wilson

I’ve just read about a new message continuity service from FrontBridge, designed to provide always on e-mail in today’s environment where e-mail outage is seen as a major business continuity issue.

Complementing the other e-mail managed services offered by FrontBridge, Active Message Continuity provides:

  • Always on e-mail continuity and disaster recovery with no need to “flip a switch”.
  • Interception-based archiving to capture messages “in stream” after filtering for spam, viruses and other unwanted content.
  • Continuous access via a web interface.
  • A fully managed service, starting from $1/month/user.

FrontBridge is already well established in the e-mail application service provider (ASP) market, but this new product is a key differentiator allowing FrontBridge to offer message compliance, message security and message continuity at a time when competitors such as MessageLabs are concentrating on just one area – that of message security (anti-virus, anti-spam and content control).

The new face of spam

Posted on By Mark Wilson

We are all used to spam arriving in our e-mail inboxes, but now the problem is spreading to other communications methods.

Research by Wireless Services Corporation shows almost half of the mobile phone text messages received in the US are spam, compared with 18% a year ago. Another problem is the growing menace of spam over instant messaging (spim), with Meta Group reporting 28% of instant messaging users hit by spim.

Meanwhile, IT managers are turning to new methods of trapping e-mail-born spam at the network edge. According to e-mail security provider Postini, 88% of e-mail is spam and Symantec reports 70% (their Brightmail Antispam product is used by ASPs such as MessageLabs) with 80% from overseas, particularly China and Russia. Appliance servers are now available that claim to trap “dark traffic” such as unwanted inbound SMTP traffic, directory harvest and e-mail denial of service (DoS) attacks, malformed and invalid recipient addresses.

Last month, Microsoft acquired Sybari and according to IT Week, the Sybari tools are likely to be offered as a plug in for the virus-scanning API in Exchange Server 2003 service pack 1, as well as part of Microsoft’s plans to offer edge services in forthcoming Exchange Server releases, including Sender ID e-mail authentication in Exchange Server service pack 2, IP safe lists, and a requirement for senders to solve a computational puzzle for each e-mail sent, increasing overheads for spammers (and unfortunately for the rest of us too).

Some industry commentators criticise the use of filtering products, citing examples of blocked legitimate e-mail. Sadly this will always be the case (one of my wife’s potential customers once claimed that her domain name pr-co.co.uk is invalid, blocking all addresses containing hyphens) and many of my clients (wisely, if in a somewhat draconian style in some cases) block various attachment types. A few weeks back, even a reply which I sent to a request for assistance left on this blog was picked up as spam. There will always be a trade off between false positives and a small amount of spam getting through – what is needed is for a real person to double check the filtered e-mail, combined with an overall increase in the use of digitally signed e-mail.

Links

Practical measures for combating spam (MessageLabs)

Securing the network using Microsoft ISA Server 2004

Posted on By Mark Wilson

Several months ago, I attended a Microsoft TechNet UK event where the topic was ISA Server 2004 network design/troubleshooting and inside application layer firewalling and filtering. It’s taken me a while to get around to writing up the notes, but finally, here they are, with some additional information that I found in other some similar presentations added for good measure.

The need for network security

The Internet is built on internetworking protocol (IP) version 4 – which was not designed with security in mind. In the early days of the Internet, security clearance was required for access – i.e. physical access was restricted – so there was no requirement for protocol security. At that time (at the height of the cold war), resistance to nuclear attack was more important than protecting traffic and everyone on the network was trusted. The same networking technologies used to create the Internet (the TCP/IP protocol suite) are now used for internal networks and for TCP/IP, security was an afterthought.

Security should never be seen as a separate element of a solution – it should all pervasive. At the heart of the process should be a strategy of defence in depth – not just securing the perimeter or deploying some access controls internally, but placing security throughout the network so there are several layers to thwart malware or a hacker. Ideally, an administrator’s security strategy toolbox should include:

  • Perimeter defences (packet filtering, stateful packet inspection, intrusion detection).
  • Network defences (VLAN access control lists, internal firewall, auditing, intrusion detection).
  • Host defences (server hardening, host intrusion detection, IPSec filtering, auditing, active directory).
  • Application defences (anti-virus, content scanning, URL switching source, secure IIS, secure Exchange).
  • Data and resource defences (ACLs, EFS, anti-virus, active directory).

Each layer of defence should be designed on the assumption that all prior layers have failed.

With users becoming ever more mobile, defining the edge of the network is becoming ever more difficult. Firewalls are no panacea, but properly configured firewalls and border routers are the cornerstone of perimeter security. The Internet and mobility have increased security risks, with virtual private networks (VPNs) softening the perimeter and wireless networks further eroding the traditional concept of the network perimeter.

A firewall alone is not enough

Some administrators take the view that “we’ve got a firewall, so everything is fine”, but standard (layer 3/4) firewalls check only basic packet information and treat the data segment of the packet as a black box. This is analogous to looking at the number and destination displayed on the front of a bus, but not being concerned with the passengers on board. Performance is often cited as a the reason for not implementing application layer (layer 7) firewalls, which inspect the data segment (e.g. for mail attachment checking, HTTP syntax, DNS syntax, correct SSL termination, URL blocking and redirection, RPC awareness, LDAP, SQL, etc.). However, Microsoft claim to have tested Internet Security and Acceleration (ISA) Server 2004 up to 1.9Gbps throughput on a single server with application filters in place (at a time when most corporates are dealing with 2-10Mbps).

Consider the standard security pitch, which has two elements:

  1. The sky is falling (i.e. we’re all doomed).
  2. Our product will fix it (i.e. buy our product).

In truth, no system is 100% effective and the firewall needs to be supplemented with countermeasures at various depths (intrusion detection systems, etc.). If there was a 100% secure system it would be incredibly expensive – in addition, threats and vulnerabilities are constantly evolving, which leaves systems vulnerable until a new attack is known and a new signature created and distributed. Heuristical systems must be supplemented with behavioural systems, and some intelligence.

Just because 100% security is not achievable, it doesn’t mean that it is any less worthwhile as a goal. We still lock our car doors and install immobilisers, even though a good car thief can defeat them eventually. The point is that we stop the casual attacker, buying time. Taking another analogy, bank safes are sold on how long it will take a safe cracker to break them.

Whatever solution is implemented, a firewall cannot protect against:

  • Malicious traffic passed on open ports and not inspected at the application layer by the firewall.
  • Any traffic that passes through an encrypted tunnel or session.
  • Attacks on a network that has already been penetrated from within.
  • Traffic that appears legitimate.
  • Users and administrators who intentionally or accidentally install viruses or other malware.Administrators who use weak passwords.

HTTP is the universal firewall bypass and avoidance protocol

In the late 1990s, as business use of the Internet exploded, we became to rely ever more on HTTP, which has earned itself a nickname – UFBAP – the universal firewall bypass and avoidance protocol.

Firewall administrators are obsessed with port blocking and so all non-essential firewall ports are closed; but we generally assume that HTTP is good and so TCP port 80 (the default port for HTTP) is left open. Because it’s so difficult to get an administrator to open a port, developers avoid such restrictions by writing applications that tunnel over port 80. We even have a name for it (web services) and some of our corporate applications make use of it (e.g. RPC over HTTP for Outlook connecting to Exchange Server 2003).

This tunnelling approach is risky. When someone encapsulates one form of data inside another packet, we tend to allow it, without worrying about what the real purpose of the traffic is. There are even websites which exploit this (e.g. HTTP-Tunnel), allowing blocked traffic such terminal server traffic using the remote desktop protocol (RDP) to be sent to the required server via TCP port 80, for a few dollars a month.

In short, organisations, tend to be more concerned with blocking undesirable sites (by destination) than checking that the content is valid (by deep inspection).

Using web services such as RPC over HTTP to access Exchange Server 2003 is not always bad – 90% of VPN users just want to get to their e-mail and so offering an HTTP-based solution can eliminate many of the VPNs that are vulnerable network entry points – what is required is to examine the data inside the HTTP tunnel and only allowing it to be used under certain scenarios. Taking the Exchange Server 2003 example further, without using RPC over HTTP, the following ports may need to be opened for access:

  • TCP 25: SMTP.
  • TCP/UDP 53: DNS.
  • TCP 80: HTTP.
  • TCP/UDP 88: Kerberos.
  • TCP 110: POP3.
  • TCP 135: RPC endpoint mapper.
  • TCP 143: IMAP4.
  • TCP/UDP 389: LDAP (to directory service).
  • TCP 691: Link state algorithm routing protocol.
  • TCP 1024+: RPC service ports (unless DC and Exchange restricted).
  • TCP 3268: LDAP (to global catalog).

Using HTTP over RPC, this is reduced to one port – TCP 80.

Application layer filtering

Inspection at the application layer still has some limitations and the real issue is understanding the purpose of the traffic to be filtered and blocking non-consistent traffic.

Microsoft ISA Server 2004 is typically deployed in one of three scenarios:

  • Inbound access control and VPN server.
  • Outbound access control and filtration (together with URL-based real time lists from third parties).
  • Distributed caching (proxy server), leading to reduced bandwidth usage.

As part of its access control capabilities, ISA Server has a number of application filters included:

  • HTTP (syntax analysis and signature blocking).
  • OWA (forms based authentication).
  • SMTP (command and message filtering).
  • RPC (interface blocking).
  • FTP (read only support).
  • DNS (intrusion detection).
  • POP3 (intrusion detection).
  • H.323 (allows H.323 traffic).
  • MMS (enabled Microsoft media streaming).

All of these filters validate protocols for RFC compliance and enable network address translation (NAT) traversal. In addition, ISA Server can work with third party filters to avoid the need for a proliferation of dedicated appliance servers (and even for appliance consolidation). Examples of third-party filter add-ons include:

  • Instant messaging (Akonix).
  • SOCKS5 (CornerPost Software).
  • SOAP/raw XML (Forum Systems).
  • Antivirus (McAfee, GFI, Panda).
  • URL Filtering (SurfControl, Futuresoft, FilterLogix, Cerberian, Wavecrest).
  • Intrusion detection (ISS, GFI).

But appliance firewalls are more secure – aren’t they?

Contrary to popular belief, appliance firewalls are not necessarily more secure – just more convenient – for those who prefer to use appliances, ISA Server is available in an appliance server format and such an appliance may well be cheaper than an equivalent server, plus Windows Server 2003 and ISA Server 2004 licenses.

Whilst looking at the security of the solution itself, ISA Server has been tested against the common certification criteria at level EAL4+ (for 9 streams). Totally rewritten since ISA Server 2000, Microsoft claim that ISA Server 2004 uses a code base which is 400% more efficient. It may run on a Windows platform, but Windows Server 2003 can (and should) also be hardened, and a well-configured ISA Server can be extremely secure.

Some firewall challenges: remote procedure calls (RPCs)

RPC CommunicationsRPCs present their own challenge to a standard (layer 3/4) firewall in terms of the sheer number of potentially available ports:

  1. On service startup, the RPC server grabs random high port numbers and maintains a table, mapping UUIDs to port numbers.
  2. Clients know the UUID of the required service and connect to the server’s port mapper using TCP port 135, requesting the number of the port associated with the UUID.
  3. The server looks up the port number of the given UUID.
  4. The server responds with the port number, closing the TCP connection on port 135.
  5. From this point on the client accesses the application using the allocated port number.

Due to the number of potential ports, this is not feasible using a traditional firewall (it would require 64512 high ports plus 135 to be open); however, a layer 7 firewall could utilise an RPC filter to learn the protocol and use its features to improved security, such that the firewall would only allow access to specific UUIDs (e.g. domain controller replication, or Exchange/Outlook RPC communications) denying access to all other RPC requests. Instead of tunnelling within HTTP (prevented by an HTTP syntax check), native RPC access can be provided across the firewall.

Some firewall challenges: secure sockets layer (SSL)

Protecting HTTPS - traditional firewallHackers will always attack the low-hanging fruit (i.e. easy targets) and as such, SSL attacks are generally too complex, but as our systems become more secure (i.e. we remove the low-hanging fruit), SSL attacks will become more likely.

HTTPS (which uses SSL) prompts a user for authentication and any user on the Internet can access the authentication prompt. SSL tunnels through traditional firewalls because it is encrypted, in turn, allowing viruses and worms to pass through undetected and infect internal servers.

Using ISA Server 2004 with an HTTP filter, authentication can be delegated. In this way, ISA Server pre-authenticates users, eliminating multiple authentication dialogs and only allowing valid traffic through. This means that the SSL connection is from the client to the firewall only, and that ISA Server can decrypt and inspect SSL traffic. Onward traffic to the internal server can be re-encrypted using SSL, or sent as clear HTTP. In this way, URLScan for ISA Server can stop web attacks at the network edge, even over an encrypted inbound SSL connection.

Protecting HTTPS - web publishingPre-authentication means that without a valid layer 7 password, there is no access to any internal systems (potential attackers drop from the entire Internet to just the number of people with credentials for the network). ISA Server 2000 can also perform this using RSA securID for HTTP (although not for RPC over HTTP with securID) and cookie pre-authentication for Outlook Web Access 2003 is also available.

Some firewall challenges: protecting HTTP(S)

Protecting HTTP (and HTTPS) requires an understanding the protocol – how it works, what its rules are and what to expect. Inbound HTTPS termination is easy (as the certificate is controlled by the organisation whose network is being protected). For outbound HTTPS and HTTP, administrators need to learn how to filter port 80/443. It may be worth considering whether global access is really required, or whether there are a set of specific sites that are required for use by the business.

ISA Server allows web publishing of HTTP (as well as other protocols such as SMTP). Web publishing protects servers through two main defences:

  • Worms rarely work by FQDN – tending to favour IP or network range. Publishing by FQDN prevents any traffic from getting in unless it asks for the exact URL and not just http://81.171.168.73:80.
  • Using HTTP filter verbs (signature strings and method blocking) to eliminate whole classes of attack at the protocol level.

Some examples of protecting a web server using web publishing and HTTP filtration are:

  • Limit header length, query and URL length.
  • Verify normalisation – http://81.171.168.73/../../etc is not allowed.
  • Allow only specified methods (GET, HEAD, POST, etc.).
  • Block specified extensions (.EXE, .BAT, .CMD, .COM, .HTW, .IDA, .IDQ, .HTR, .IDC, .SHTM, .SHTML, .STM, .PRINTER, .INI, .LOG, .POL, .DAT, etc.)
  • Block content containing URL requests with certain signatures (.., ./, \, :, % and &)
  • Change/remove headers to provide disinformation – putting ISA Server in front of an Apache server is a great way to prevent UNIX attacks by making hackers think they are attacking a Windows server.
  • Block applications based on the header.

Some headers to look for include:

  • Request headers:
    • MSN Messenger: HTTP header=User-Agent; Signature=MSN Messenger
    • Windows Messenger: HTTP header=User-Agent; Signature=MSMSGS
    • AOL Messenger (and Gecko browsers): HTTP header=User-Agent; Signature=Gecko/
    • Yahoo Messenger: HTTP header=Host; Signature=msg.yahoo.com
    • Kazaa: HTTP header=P2P-Agent; Signature=Kazaa, Kazaaclient
    • Kazaa: HTTP header=User-Agent; Signature=Kazaa Client
    • Kazaa: HTTP header=X-Kazaa-Network; Signature=KaZaA
    • Gnutella: HTTP header=User-Agent; Signature=Gnutella
    • Gnutella: HTTP header=User-Agent; Signature=Gnucleus
    • Edonkey: HTTP header=User-Agent; Signature=e2dk
  • Response header:
    • Morpheus: HTTP header=Server; Signature=Morpheus

Some firewall challenges: protecting DNS

Whilst some DNS protection is available by filtering TCP/UDP ports 53, ISA Server filters can examine traffic for DNS host name overflows, length overflows, zone transfer from privileged ports (1-1023), zone transfer from high ports (1024 and above).

Some firewall challenges: protecting SMTP

When it comes to mail protection, anti-spam and anti-virus vendors cover SMTP relays but ISA server filters can examine protocol usage, i.e.:

  • Checking that TCP port 25 traffic really is SMTP.
  • Checking for a buffer overflow to the RCPT: command.
  • Blocking someone using the VRFY command.
  • Stripping an attachment or block a user.

Using such a solution adds to the defence in depth strategy, using the firewall to add another layer of defence to the mail system.

Some firewall challenges: encapsulated traffic

Encapsulated traffic can cause some concerns for a network administrator as IPSec (AH and ESP), PPTP, etc. cannot be scanned at the ISA Server if they are published or otherwise allowed through. Tunnelling traffic will be logged, but not scanned as ISA cannot look inside the tunnel unless it is terminating the VPN. The administrator is faced with a choice – open more ports and uses application filters – or tunnel traffic without inspection. NAT also has some implications.

ISA Server can, however, perform intra-tunnel VPN inspection, so VPN traffic can be inspected at the application layer. VPN client traffic is treated as a dedicated network so destinations can be controlled, along with the use of application filter rules.

VPN clients must be hardened. If not, then hackers can attack clients and ride the VPN into the corporate network. Client based intrusion detection systems and firewalls can help but the ideal solution is VPN quarantine (e.g. Windows Server 2003 network access quarantine control) as the most common entry to the network for malware is from mobile devices either VPNing into the network, or returning to the network after being infected whilst away from the network (perhaps connected to other networks, including the Internet).

Alternatives to a VPN that should be considered are:

  • E-mail: RPC over HTTP, or Outlook Web Access (OWA). POP3 and IMAP4 should be avoided as they are not fully featured.
  • Web-enabled extranet applications: SSL.
  • Other applications: RPC filtration with ISA Server.

Don’t forget the internal network

Internal network defences are another factor to be considered. Networks are generally one large TCP/IP space, segmented by firewalls to the Internet. Trust is implicit throughout the organisation but this cannot be relied upon and network segmentation is critical (cf. a bank, where entering a branch does not gain access to the vault). Internal users are dangerous too.

  • The windows firewall in Windows XP SP2 (internet connection firewall in Windows Server 2003 and earlier versions of Windows XP) is a vital tool in preventing network-based attacks, by blocking unsolicited inbound traffic. Ports can be opened for services running on the computer, and enterprise administration facilitated through group policy. Microsoft recommend that use of the Windows Firewall is combined with network access quarantine control; however it does not have any egress filters (i.e. controls over outbound traffic).
  • Virtual LANs (VLANs) can be used to isolate like services from one another. Switch ACLs are used to control traffic flow between VLANs at layer 3. Layer 2 VLANS may be used where no routing is desired. By using internal firewalls, port level access can be controlled to internal VLANs.
  • IPSec is a method of securing internal IP traffic, mutually authenticating end points. It is used to ensure encrypted and authenticated communications at the IP layer, providing a transport layer security that is independent of applications or application layer protocols. It prevents against spoofing, tampering in the wire and information disclosure. Mutual device authentication can be provided using certificates, kerberos (or pre-shared key – but this is only recommended for testing scenarios). Authentication headers (AH) should be used to provide packet integrity, but this does not encrypt, allowing for network intrusion detection. Encapsulation security payload (ESP) provides packet integrity and confidentiality, but its encryption prevents packet inspection. Consequently, careful planning is required to determine which traffic should be secured.

One use of IPSec is to allow domain replication to pass through firewalls, creating an IPSec policy on each domain controller to secure traffic to its replication partners. ESP 3DES should be used for encryption and the firewall should be configured to allow UDP port 500 for internet key exchange (IKE) and IP protocol 50 for ESP.

Potential issues around wireless network security are well publicised. The two most common security configurations each have their own limitations:

  • Wired equivalency privacy (WEP), relies on static WEP keys which are not dynamically changed and are therefore vulnerable to attack. There is no standard method for provisioning static WEP keys to clients and the principle of static keys does not scale well, with a compromised key exposing everyone.
  • MAC address filtering, is also limited by the potential for an attacker to spoof an allowed MAC address.

Possible solutions include password-based layer 2 authentication (IEEE 802.1x with PEAP/MS CHAP v2) and certificate-based layer 2 authentication (IEEE 802.1x EAP-TLS). Other options include:

  • VPN connectivity using L2TP/IPSec (preferred) or PPTP. This does not allow for roaming but is useful when accessing public wireless hot spots; however there is no computer authentication, or processing of computer group policy settings.
  • IPSec, but this has some interoperability issues.
Security type Security level Ease of deployment Ease of integration
Static WEP Low High High
IEEE 802.1x (PEAP) High Medium High
IEEE 802.1x (TLS) High Low High
VPN (L2TP/IPSec) High Medium Low
IPSec High Low Low

Summary

In summary, firewalls are placed in different locations for different reasons. These must be understood and filtered accordingly. Core functionality can be extended with protocol filters to cover a specific scenario but no one device is a silver bullet. Solutions are more important than devices and firewall configuration is more than a networking decision – it also requires application awareness.

Links

Microsoft ISA Server
ISA Server Community
ISAserver.org
Zebedee (a simple, secure TCP and UDP tunnel program)

10,000 feet view of Microsoft Exchange Server 2003

Posted on By Mark Wilson

For anyone who is new to Exchange Server 2003, here’s a brief overview that may be of use.

Microsoft Exchange Server 4.0 was launched in 1996, as a replacement for Microsoft Mail 3.x. It was Microsoft’s first groupware product, competing directly with Novell GroupWise and Lotus Notes. Later versions (v5.0, v5.5) added functionality and improved the scalability of the solution, with Exchange Server 5.5 being the version which really took a hold on the market. Exchange 2000 (v6.0) was a major rewrite, leaving behind its own directory service and using Active Directory instead (which is based on the Exchange Directory technology), and featuring a new system of storage groups supporting multiple mailbox and private stores (for improved database backup and restoration times). Exchange 2000 also switched its internal message transport protocol from X.400 to SMTP, making use of (and extending) the SMTP features in Microsoft Internet Information Services (IIS). Microsoft Exchange Server 2003 (v6.5) builds on the scalability, reliability, performance and manageability of Exchange Server 2000 and is the current version, albeit without conferencing, instant messaging and chat features, which have been replaced by the Microsoft Office Live Communications Server product.

Exchange Overview

The Exchange environment is known as an organization. Within an organization, Exchange Server 4.0-5.5 divides the infrastructure into one or more sites (similar to the Active Directory site concept), linked by connectors of varying types (e.g. RPC, X.400). Exchange Server 2000 and 2003 use routing groups instead of sites, and also introduce the concept of administrative groups for organisations where administration is undertaken by different groups of staff (e.g. in a global organisation with separate teams for Americas, EMEA and Asia-Pacific).

Exchange servers may be configured for a variety of specialist purposes:

  • A bridgehead, is a server dedicated to routing e-mail for a routing group.
  • Front end (protocol) servers act as concentrators for client connections, where protocol conversion would be an overhead (e.g. HTTP connections using Outlook Web Access).
  • Back end (storage) servers could be mailbox servers, public folders servers, or a combination of the two.
  • There may also be other servers dedicated to providing services such as fax connections, mail archival and retrieval (near-line storage), or instant messaging.

Because Exchange Server 2000 and 2003 use Active Directory, Exchange servers require a global catalog server to be placed nearby. Other products, such as Microsoft ISA Server 2004 may also be used to increase security, for example filtering inbound e-mail, or publishing an Outlook Web Access (OWA) server.

Links

Microsoft Exchange Server
Microsoft Exchange Server resource site

Child safety online

Posted on By Mark Wilson

In previous posts I’ve mentioned both the Microsoft at work and Microsoft at home microsites. Today I was directed to a new microsite – security at home, specifically the child safety online section.

I’m not sure if the world is really any more dangerous for children than it was when I was a child but I do know the media is all pervasive – we hear a lot more about the unfortunate events that do occur – and that as a parent I’ll do anything I can to ensure that my son is safe. At three months old, he’s a bit young to be using the Internet but this site looks a useful resource for anyone who has children aged between 2 to 17 and who use a computer with a connection to the Internet.

On a related note, a couple of week’s back I wrote about technology’s role in the demise of the English language. Well, for anyone (like me), who’s not as “with it” as we once were (omigod, and I’m only 32 – hellllllp!), whilst reading child safety online, I stumbled across a parent’s primer to computer slang (should that be $14NG?) and the netiquette 101 for new netizens.

!337$p34k 1z m4d

Microsoft Exchange Server 2003 troubleshooting and disaster recovery

Posted on By Mark Wilson

A couple of nights back, I attended one of the Microsoft TechNet UK events. Not sure whether to attend John Howard’s session on automating Windows Server Administration or Eileen Brown’s Microsoft Exchange Server 2003 Troubleshooting and Disaster Recovery session, I decided to return to my Microsoft Exchange roots (which go back to the Exchange Server 4.0 launch events in April 1996). In addition, Eileen has posted some useful links relating to the session content on her blog.

Configuring recovery options and general troubleshooting tools

A common issues for an e-mail administrator is the recovery of e-mail which a user has accidentally deleted, and less commonly, the need to recover a deleted mailbox. Fortunately, Exchange has a number of options which can assist with this. There is a trade off between giving users the opportunity to recover data and additional storage on the e-mail server, but because of single instance storage, in reality this is not as big an issue as it might seem.

For a mailbox store, there are three main configuration items of interest:

  • Keep deleted items for (days) defines the time for which a deleted item is still recoverable from within Outlook, even if the deleted items folder has been emptied (for options in the use of this feature, see KC Lemson’s blog).
  • Keep deleted mailboxes for (days) is similar, but defines the number of days that a mailbox will remain (orphaned and available for reconnection to an Active Directory user account), until it is finally removed from the store.
  • Do not permanently delete mailboxes and items until the store has been backed up ensures that regardless of the deleted item retention and mailbox retention intervals, nothing is finally removed until a full backup of the store has successfully taken place.

For a public folder store, keep deleted items for (days) and do not permanently delete items until the store has been backed up are the equivalent options.

All of these options are found on the limits page of the store properties.

There are a number of troubleshooting tools available to the Exchange administrator:

  • Disaster recovery setup mode (note that this doesn’t work on a cluster), can be used to reconnect mailboxes with a store.
  • The application log in Event Viewer is useful (in particular, watch out fr 1012, 1018, 1019 and 1020 errors – the Exchange Information Store often gives information about database issues well in advance of failure).
  • Diagnostics and protocol logging, which is an overhead on the server and should only be enabled when diagnosing an issue, allows the level of logging to be tuned. Even set to none, critical events are logged, with minimum, medium and maximum corresponding to the level of events that are logged.
  • Message tracking can be used to track messages through the system, optionally recording the subject of the message in the logs. The main consideration with this is the number of days for which a message should be retained.
  • Exchange System Manager’s monitoring and status tools allow monitoring of services or resources (e.g. thresholds for queue length growth) and alert notification, sending e-mail and/or executing a script (a useful alternative if e-mail is unavailable!) to notify an administrator of issues or even to automatically take corrective action.

When backing up Exchange (whether using the Backup Utility for Windows, or a third party tool), there are a number of issues to consider:

  • A backup is not complete unless it includes the mailbox and public folder stores, with the transaction logs and the Windows system state. In addition, mailbox servers should never have circular logging enabled as storage prices have dropped considerably since the days of Exchange Server 4.0 and without a complete set of logs, recovery would still result in some lost data. For front end servers, Microsoft recommend that there are no stores present.
  • If co-existing with Exchange Server 5.5, the Site Replication Service (SRS) also needs to be considered.
  • Connectors may also contain information to be backed up.
  • Recovery will typically take twice a long as backup, so keep backup times short. If quotas are used to limit mailbox usage beware as they might just lead to users storing mail in personal folder (.PST) files, leading to unmanaged offline storage (i.e. not backed up), a loss of singe instance storage, and possibly network bandwidth issues if the personal folders are stored on a network server.

Further information (and best practice) is contained in the Disaster Recovery Operations Guide for Exchange Server 2003.

Troubleshooting Internet E-mail

One of the things to remember when troubleshooting Exchange issues is that there are so many external factors to consider. Besides the obvious areas of Exchange and the e-mail client (usually Outlook), DNS and the underlying network can create issues.

When examining the process of receiving inbound e-mail, Exchange doesn’t actually do much! The originating server looks up the IP address which corresponds to the mail exchanger (MX) record for the SMTP domain in DNS. The message is then routed across the Internet based on that TCP/IP address and it is only once the message has been received (possibly via a smart host) that Exchange routes the message within the organisation for final message delivery. For outbound e-mail, it is the reverse process and from this we can tell that the two main areas to look at are DNS and TCP/IP.

The TCP/IP troubleshooting process is well known:

  1. Check the TCP/IP properties for the Exchange server. Are they complete?
  2. Can you ping localhost (127.0.0.1)? If not, there would appear to be an issue with the network card or protocol stack.
  3. Next, can you ping the server by its own IP address? If not, there would appear to be an issue with the server’s TCP/IP address – is that configured correctly?
  4. Next, can you ping the default gateway? If not, there would appear to be either an incorrectly configured router (default gateway) address, or a physical network issue (is the cable plugged in?)
  5. Finally, can you ping other hosts on the network – e.g. the DNS server? If not, there may be a routing issue (or the DNS server addresses could be incorrect).

Additional troubleshooting steps for mail servers are:

  • Can you connect to port 25 on the mail server using Telnet? If not, then the SMTP service may not be running.
  • Are the server’s host (A) and MX records correctly recorded in both the internal and external DNS, with the correct priority (1 is the lowest cost).

Other areas to examine are:

  • Is a DNS suffix required and/or set in the TCP/IP properties?
  • Does the computer name have the correct fully qualified domain name (FQDN) in the system properties.
  • Is DNS working correctly (NETDIAG is a useful command, in particular netdiag /test:dns can be used to identify DNS issues, after which NSLOOKUP can be used to query DNS).
  • Address spaces, e.g. if an organisation hosts two or more domain names, are they all configured with MX records and do users have corresponding e-mail addresses.
  • Size restrictions – both internal and external restrictions can be set. If large messages are not being received, this could be the issue. Note that SMTP virtual server settings can be overridden by global settings.
  • If the SMTP queues have a lot of retries pending, this will often indicate a DNS issue.

Recovering messages and mailboxes
There are a number of mailbox recovery tools available to an Exchange administrator:

  • Once an Active Directory account is deleted or a mailbox removed in Exchange System Manager, the mailbox is not actually removed, but is tombstoned (shown with a red cross in System Manager and the retention time for deleted mailboxes begins). This action is carried out by the Cleanup Agent, which may be triggered manually if it has not completed its next scheduled run before a mailbox needs to be recovered.
  • The mailbox recovery center allows an administrator to mount a (recovered) store and view all the disconnected mailboxes, from where a matching user account can be found using the Exchange Mailbox Matching Wizard and the mailbox reconnected using the Mailbox Reconnection Wizard.
  • The Exchange Server Mailbox Merge Wizard (ExMerge) can be used to merge data into a mailbox; however the recover mailbox data functionality (new with Exchange Server 2003 SP1) replaces the need to use ExMerge in the majority of recovery cases.
  • Offline folder (.OST) to personal folder (.PST) conversion has now been superseded by the recovery storage group.

Tip: “object not found” errors when Outlook synchronises with the Exchange Server are often caused by invalid entries in the default address book. Rebuilding this will usually resolve such issues.

Recovery storage groups

To use the recovery storage group feature, at least one Exchange Server 2003 server must be available within the organisation. This allows an administrator to create a recovery storage group, into which a database can be mounted for mailbox recovery, avoiding the need to recover on a separate recovery server and export e-mail via a .PST file; however recovery storage groups do have some limitations (after all, they are intended to be used purely for the purposes of recovering data):

  • All protocols except MAPI (required for the Microsoft Exchange Information Store service to access the storage group) are disabled.
  • Mailboxes cannot be directly connected to user accounts (except using ExMerge).
  • No management policies are available (not necessary as no live users).
  • No Exchange maintenance procedures are available (ESEUTIL/ISINTEG).
  • Databases must be mounted manually (e.g. to run ExMerge).
  • Database locations cannot be changed (but database files are not server/location specific and can be copied manually).
  • Only private mailbox stores can be recovered (i.e. not public folder stores)

Exchange Server 2003-aware backup programs will automatically restore to a recovery storage group.

In a disaster recovery scenario, an Exchange administrator could perform what is known as a dial-tone database restoration. This involves creating an empty database and mounting this so that users can continue to send and receive e-mail whilst their original data is recovered. Meanwhile, the failed database can be restored to a recovery storage group and the recover mailbox data feature or ExMerge used to restore the data to the user’s mailbox whilst both stores are online. To save time in the recovery (albeit involving some more user downtime whilst the databases are swapped), it may be appropriate to swap the database files, remount the original store and then merge in the new data from the dial-tone database. Eileen Brown’s blog features a blogcast demonstrating the recovery storage group which explains the process in further detail.

Further information on using Exchange Server 2003 recovery storage groups is available on the Microsoft website.

Database corruption and recovery

Each Exchange Server storage group has its own set of transaction logs, which can be replayed to recover data up to the point at which failure occurred. In my recent post about Exchange Server best practice and preventative maintenance, I wrote about the ESEUTIL and ISINTEG tools. Using ESEUTIL, it is possible to examine the message headers (eseutil /mh database.edb) and examine the resulting output to check the state of the store (clean or dirty) and whether or not any logs are required to be replayed.

To replay the logs, simply ensure that they are available in the correct location and mount the store. Following this, the application event log should record a number of events indicating that it is initiating recovery steps and replaying the logs before recording a successful completion.

Why IE 7.0 must rely on XP SP2

Posted on By Mark Wilson

I’ve seen a lot of press coverage over the last week or so about Microsoft’s plans for Internet Explorer (IE) 7.0. One of the major gripes seems to be that it will require Windows XP service pack 2 (SP2).

So what’s wrong with that?

One of the main reasons that people are moving to other browsers (e.g. Firefox) is that IE is perceived as insecure. SP2 is a major security update for the Windows XP desktop operating system. Why provide a new (more secure) browser product to people who do not use the latest security patches on their operating system?

SP2 has been publicly available since August 2004 (6 months ago). The temporary blocking mechanism to hold back automatic SP2 deployment from Windows Update is scheduled to expire on April 12 2005. There is no point in IT Managers burying their heads in the sand and ignoring SP2 any longer. I will concede that Microsoft should have shipped v4.0 of the Application Compatibility Toolkit alongside SP2 (after all, application compatibility is probably the largest barrier to SP2 deployment) but it amazes me that so few organisations have made the move to SP2 after all this time.

For those who are not even using Windows XP, whilst the extra functionality in IE 7.0 may be useful, Microsoft is a product and technology business and it needs to maintain its licensing revenues through getting people to adopt the latest technologies (especially whilst strategic products are being delayed by major security rewrites).

If an older platform is seen “good enough” then fine; but “good enough” shouldn’t just be about functionality – it needs to consider the whole picture – including security. It may be that the risk assessment considers remaining on a legacy (possibly unsupported) platform is more favourable than the risk (and cost) of upgrading. That’s fine too – as long as that risk is acceptable to the business.

My recommendation? Organisations who are using Windows XP should fully test their applications and carry out a controlled upgrade to SP2 as soon as possible. Those who continue to use older operating systems (especially Windows 9x, ME, and NT) should urgently consider upgrading. Then keep patch levels up-to-date, for example, by using Microsoft Software Update Services (SUS) and the Microsoft Baseline Security Analyzer (MBSA). IT users can’t continue to complain about the security of the Microsoft platform if they won’t deploy the latest (or even recent) patches.

Direct access to virtual floppy disk files

Posted on By Mark Wilson

I’m sure that, from time to time, many Virtual PC users would find the ability to mount a virtual floppy disk (.VFD) file and directly access its contents (create, view, edit, rename or delete files, format the disk, or launch a program) useful. Well you can – using Ken Kato’s virtual floppy driver. Ken also has a whole load of tools for VMware users on his website.

A new use for virtualisation?

Posted on By Mark Wilson

The running of legacy applications that cannot be ported to a modern operating system, in their own legacy environment, inside a virtual machine on a modern host operating system is often cited as one of the advantages of virtualisation.

I have a slightly different issue. As a consultant, I spend as much of my time with clients as possible and visits to the office are rare. I’ve also had a variety of hardware issues of late and frequently rebuild the notebook PC that I use for work. I use my own SUS server for software updates but need my PC to be a member of our company’s Active Directory to access my corporate e-mail using Outlook 2003’s HTTP over RPC functionality. Being away from the office means that until an administrator can join my rebuilt PC to the domain, I’m restricted to using Outlook Web Access (as good as the latest version is, it’s still no substitute for the real thing).

It seems that virtualisation can provide the answer to my conundrum. Earlier today, the internal support guys added one of my virtual machines to the corporate domain. Now I can access my corporate applications, run the company’s preferred security products, and generally be a good corporate citizen inside Microsoft Virtual PC and do what I like with my host operating system. I can rebuild my PC at will, then simply reinstall Virtual PC (or Virtual Server), fire up my corporate virtual machine and carry on working.