The email dash

Last week, I wrote a post about technology’s role in the demise of the English language and I was interested to read about the “correct” spelling of e-mail in this week’s IT Week. It seems that by popular usage today, email is 10 times more common than e-mail. As IT Week points out, the grammatically correct version (e’mail) is unlikely to catch on. Interestingly, Answers.com notes that email is the common spelling but has elected to continue the use of e-mail as it is easier to read.

Trials and tribulations with RIS

After a few weeks developing my unattended PC build using physical PCs, today I needed to deploy a virtual PC (VPC). It should be simple to boot a VPC to a Remote Installation Services (RIS) server and apply an image as normal, but in case anyone else out there is googling to find out why they can’t get it working, here’s some of the stuff I’ve found today.

The SMC 9332 NIC which VPC emulates does not include PXE support, but RIS has provision for this in the form of the remote installation boot disk, which may be created using the \\servername\reminst\admin\i386\rbfg.exe tool (and for VPC users, Roudy Bob has published a virtual RIS Boot Disk on his website).

I spent a while this morning wondering why I was receiving the following message, even though my virtual PC was using the host PC’s network card and the DHCP logs in %systemroot% showed an IP address being assigned to a device with the MAC address of my virtual PC:

DHCP
No reply from a server
Press any key to reboot system

This particular problem turned out to be related to my RIS configuration – after a not inconsiderable time spent searching, Microsoft knowledge base article 891372 reminded me that I had selected the checkbox to prevent responses to unknown clients in the properties for the RIS server (I pre-stage client PCs so that when they are rebuilt, RIS already knows the computer name).

Once that was fixed, everything was looking good, except that the Client Installation Wizard was not displayed. I enabled debug logging on the Boot Information Negotiation Layer service and the %systemroot%\debug\binlsvc.log file indicated that the RIS server was communicating with my client, but for some reason it was hanging after the server had sent a response:

[BinlServer] 02/15 14:07:30 [MISC] Client Guid: {00000000-0000-0000-0000-0003ffa7b536}
[BinlServer] 02/15 14:07:31 [MISC] MachineDN = CN=VPC1,OU=Policy,DC=home,DC=local
[BinlServer] 02/15 14:07:31 [OPTIONS] Server allows new clients and the Server is generating the OS Chooser path response.
[BinlServer] 02/15 14:07:31 [OPTIONS] Server allows new clients and the Server is generating the OS Chooser path response.
[BinlServer] 02/15 14:07:31 [MISC] SamName = VPC1$
[BinlServer] 02/15 14:07:31 [MISC] Name = VPC1
[BinlServer] 02/15 14:07:31 [OPTIONS] Recognizing client.
[BinlServer] 02/15 14:07:32 [MISC] Client Guid: {00000000-0000-0000-0000-0003ffa7b536}
[BinlServer] 02/15 14:07:32 [OPTIONS] Recognizing client.
[BinlServer] 02/15 14:07:32 [MISC] Client Guid: {00000000-0000-0000-0000-0003ffa7b536}
[BinlServer] 02/15 14:07:32 [OPTIONS] Recognizing client.
[BinlServer] 02/15 14:07:32 [STOC] Sending response to = 192.168.7.106, XID = 63702087.

Compared with a working (physical) PC, where the log read:

[BinlServer] 02/15 14:17:01 [MISC] Client Guid: {84e1f6e8-cfcc-11d6-7f6d-218f133b01ff}
[BinlServer] 02/15 14:17:01 [MISC] MachineDN = CN=LAPTOP1,OU=Policy,DC=home,DC=local
[BinlServer] 02/15 14:17:01 [OPTIONS] Server allows new clients and the Server is generating the OS Chooser path response.
[BinlServer] 02/15 14:17:01 [OPTIONS] Server allows new clients and the Server is generating the OS Chooser path response.
[BinlServer] 02/15 14:17:01 [MISC] SamName = LAPTOP1$
[BinlServer] 02/15 14:17:01 [MISC] Name = LAPTOP1
[BinlServer] 02/15 14:17:01 [OPTIONS] Recognizing client.
[BinlServer] 02/15 14:17:02 [MISC] Client Guid: {84e1f6e8-cfcc-11d6-7f6d-218f133b01ff}
[BinlServer] 02/15 14:17:02 [OPTIONS] Recognizing client.
[BinlServer] 02/15 14:17:02 [MISC] Client Guid: {84e1f6e8-cfcc-11d6-7f6d-218f133b01ff}
[BinlServer] 02/15 14:17:02 [OPTIONS] Recognizing client.
[BinlServer] 02/15 14:17:02 [STOC] Sending response to = 192.168.7.104, XID = 8831c65a.
[BinlServer] 02/15 14:17:04 NULL screen name so we are retrieving the Welcome Screen.
[BinlServer] 02/15 14:17:04 Retrieving screen file: 'E:\RIS\OSChooser\welcome.osc'

I spent hours trawling the ‘net, completely mistified as to why physical PCs worked, but not virtual PCs, until a colleague reminded me that RIS servers should not have multiple NICs installed (they can have but it is not recommended – this is also explained in Microsoft knowledge base article 891372). My RIS server does have two NICs – one connected to a 3Com SuperStack 3300 switch (downstairs), and one wireless LAN connection to the LAN in my office upstairs (which uses a mixture of wired and wireless LAN technologies). As a last resort, I plugged the host PC into the downstairs switch and the VPC suddenly found the RIS server.

For reference, by default, after obtaining an IP address and downloading the boot image, RIS displays a prompt for the user to press F12. This may be disabled by renaming startrom.com to startrom.old and startrom.n12 to startrom.com (found at \\servername\reminst\oschooser\i386). You can also edit the .osc files used by the Client Installation Wizard as these are simply OSCML documents (modelled on HTML 2.0).

Now RIS is working as I expected and I can even go back to pre-staging clients (GUIDs for VPCs are formed with the MAC address for the virtual NIC, prefixed with 00000000-0000-0000-0000-).

This should have been simple, but a couple of minor issues made it very difficult – I just hope my experience is useful to someone else out there in cyberspace.

Links

How to set up, configure, and use Remote Installation Services in Windows 2000
Customising RIS Installations

Installing applications silently (or at least quietly)

As part of the ongoing development of my unattended Windows XP build, I need to install core applications without any user intervention. Anybody who has attempted any form of application packaging will know that it is something of a black art and whilst many applications install with standard InstallShield/MSIExec command line switches, quite a few do not. So far, with my limited set of applications, I haven’t had to resort to using the Windows Scripting Host SendKeys method but I’m sure its just a matter of time.

The developers of Unattended, a Windows deployment system have written an excellent summary of performing unattended/silent installations using the many popular application installers.

Other references that have been useful are the AppDeploy.com Package Knowledge Base and the Microsoft Software Forum Network (MSFN) application switches forum (although that can take some time to navigate to find the latest information for a given application).

Exchange Server best practice and preventative maintenance

Until fairly recently, Exchange was my main area of technical expertise, but since I joined Conchango, I’ve been working in other areas and my Exchange skills have become a little rusty. That was until a couple of nights back, when I attended a Microsoft TechNet UK event, where Paul Bowden (Exchange Product Manager) demonstrated the Microsoft Exchange Server Best Practices Analyzer tool (ExBPA) before Brett Johnson (one of Microsoft’s escalation engineers in the UK) talked about best practices of Exchange Server preventative maintenance.

Microsoft Exchange Server Best Practices Analyzer tool

Analysis of support incidents logged with Microsoft has shown that only 0.3% result in the generation of a hotfix and 60% are configuration errors. The ExBPA is a tool which analyses Exchange Server for the top configuration issues in a manner which is a hybrid of a proactive health check and reactive diagnosis.

ExBPA was not the first best practice analyser from Microsoft – that was the Microsoft SQL Server Best Practices Analyzer (SQLBPA), launched in May 2004 – and BPAs will eventually be produced for all Microsoft products within the Windows Server System.

The design principles used for the creation of ExBPA were:

  • Concentrate on performance, scalability and availability – whilst the ExBPA does examine some security mis-configurations, e.g. open relays or too many administrators, it does not look for the latest patch levels – the Microsoft Baseline Security Analyzer (MBSA) performs that function.
  • Make it easy to run – previous tools were not particularly easy to set up and ExBPA is designed on a 3-click principle (from startup to scan).
  • Don’t leave me hanging – i.e. don’t just provide a strange message and a link to a Microsoft knowledge base article – provide some useful information in relation to the tool’s findings.
  • Keep it up-to-date – ExBPA automatically downloads its web update packs, which are published every two weeks.
  • Work in all environments – ExBPA works from single server Microsoft Small Business Server implementations right through to enterprise Exchange Server deployments.

The ExBPA can be run against on all versions of Exchange Server, although for versions prior to Exchange Server 2000 it does require that Active Directory and at least one Exchange 2000 or 2003 server are available. The tool is implemented in Visual C#, with an XML input/output data model and an XPath analysis engine. There are no server components and the tool is generally run from a Windows XP computer, collecting the data remotely. More architecture information is available in the ExBPA overview on the Microsoft website.

ExBPA is not a monitoring tool – that is Microsoft Operations Manager Server (MOM), for which there is an Exchange Server 2003 Management Pack. ExBPA provides a snapshot in time, looking for data in:

  • Active Directory.
  • DNS.
  • WINS.
  • Registry (there are over 1200 registry parameters for Exchange Server 2003).
  • IIS metabase.
  • Performance monitor.
  • Files on disk.
  • TCP/IP ports.

The first pass is a data collection and a subsequent pass is made on this for analysis against defined rules.

ExBPA understands a number of Exchange Server roles:

  • Small mailbox servers.
  • Large mailbox servers.
  • Clustered servers.
  • Front-end servers.
  • Bridgehead servers.

Advice is adjusted accordingly (e.g. circular logging off for mailbox servers but on for a bridgehead) and ExBPA reports on a variety of rule types:

  • Errors (something is causing, or is likely to cause a problem).
  • Warnings (something looks suspicious).
  • Non-default (something has been changed).
  • Time (something has changed within the last 5 days).
  • Information (something of interest about the environment).
  • Best practice (in ExBPA v2.0, due for release within the next 3 weeks).

When running, the ExBPA will automatically detect the closest global catalog (GC) server and the credentials of the current logged on user (although these can be modified if required). The type of scan can be set to one of three options (heathcheck, connectivity test or baseline) and the network speed must be set, both to provide an estimate of the time left to run and to set appropriate thresholds for timeouts, etc. Once the elements of the organisation to be monitored are selected, the ExBPA will run (it is multi-threaded, using up to 25 threads) and following a successful analysis, a number of reports are available:

  • Critical issues list.
  • Full issues list.
  • Non-default settings.
  • Recently-changed settings.
  • Baseline.
  • Items of interest.
  • Summary view.
  • Detailed view.
  • Disabled items list.
  • Run time log.

Some of the success stories from using the ExBPA to identify issues include:

  • Incorrectly configured DNS server address causing poor performance (even with secondary and tertiary addresses in place, Exchange will always try to contact the primary DNS server first – if that is down, or the IP address is not correct, then that means that every lookup request will first be tried against an invalid entry, before the secondary DNS entry is attempted).
  • Poor performance due to placement of database files on compressed disk volumes (even though they were on a high performance SAN).
  • Circular logging enabled on a 12,000 user Exchange cluster (had been enabled prior to migration from the old servers to prevent excessive log generation, but was not disabled afterwards).
  • Incorrect memory configuration generating 9582 Event ID errors, leading to a server restart every two weeks.

ExBPA v1.0 was released in September 2004, with 1200 points collected using 800 rules. ExBPA v1.1 followed in December 2004, with some usability improvements and 1300 points using 900 rules. ExBPA v2.0 is due for release in March 2005 and will add:

  • Localisation for all languages in which Exchange Server is available.
  • Performance sampling and root cause analysis (how close to the limit is the server).
  • Administrative API support (when was the last backup).
  • Operational integration with MOM 2005.
  • Export in XML HTML or CSV format.
  • New baseline logic.

ExBPA v3.0 is already being planned for release later in 2005, with new features including more rules and refinements, and a MAPI.NET collector.

The web update pack for the ExBPA is a 650Kb XML file and just some of the elements that the ExBPA checks today are:

  • Active Directory (forest) – functionality level, Exchange schema extensions, default policy changes.
  • Active Directory (domain) – functionality level, renamed domains, FSMO availability, renamed/deleted/moved Exchange system containers and/or groups.
  • Active Directory Connector – state of the connector (overloaded/idle/newer version available/service pack level), connection agreements (orphaned/set never to run/missing server/one-way/out-of-date).
  • Exchange organisation – message size limits enforced, stray Exchange objects in LostAndFound, more than 10 Exchange administrators, ForestPrep version, mixed/native mode, Outlook Mobile Access (OMA) options, Exchange Archive Solution (EAS) options, unsolicited commercial e-mail (UCE) thresholds, recipient update service definitions, address list and offline address book (OAB) definitions.
  • Exchange administration groups – validity of legacyExchangeDN, policy containers intact.
  • Exchange routing groups – valid routing master, enumerate all connectors, recently changed connectors.
  • Exchange server – server name validity, fully qualified domain name (FQDN) and NetBIOS name resolution, service pack/rollup level, time synchronisation with Active Directory.
  • Cluster configuration (active and passive) – number of nodes, configuration discrepancies, temporary paths, quorum configuration, heartbeat configuration, DNS/WINS configuration, enumerates all resources and parameters, kerberos configuration.
  • Directory access – cache configuration and non-default parameters, cache efficiency, round trip times between Exchange and each domain controller (DC)/GC, hardware configuration of each DC/GC, GC to Exchange processor ratio.
  • Information store – extensible storage engine (ESE) cache configuration, virtual memory state, online maintenance window, checkpoint depth, circular logging state, log buffer configuration, log generation level, file system characteristics (compression/encryption), validity of legacyExchangeDN, database and logs on the same disk, content indexing state, non-default parameters in private/public GUID registry, database size, e-mail address on public folder stores, remote procedure call (RPC) compression/buffer packing settings, hard-codes TCP/IP ports and clashes with other Exchange ports, non-default and bad store process parameters.
  • Transport – main configuration parameters within Active Directory, cross-check of AD and metabase for consistency, non-default settings, file system characteristics (compression/encryption) for mailroot folders, SMTP stack verb validation, SMTP mail submission test, enumeration of transport event sinks, enumeration of MTA settings, detection of archive sink and configuration, non-default routing parameters.
  • System Attendant – service state, file system characteristics (compression/encryption) for message tracking folders, request for response (RFR) service, RFR/name service provider interface (NSPI) target server configuration, hard-coded TCP/IP ports.
  • Anti-virus support – product detection, configuration and patch level (product dependent).
  • Other installed applications – RPC client/server binding order, presence of LeakDiag, old versions of Simpler-Webb Exchange Resource Manager, ISA 2000 service pack level, presence of MOM agent.
  • Hardware configuration – BIOS less than one year old, processor configuration, physical memory installed, specific support for Dell, HP and IBM servers.
  • Disks – performance counters enabled, enumeration of physical/logical disks, enumeration of mount points, enumeration of disk controllers and driver levels, host bus adaptor (HBA) configuration, SAN multi-pathing software version.
  • File versions – verify 29 key Exchange binaries (present/not too old/hotfixes), check MAPI subsystem/presence of old rollups/presence of ESE API virus scanners.
  • Hotfixes – detect all installed hotfixes for Exchange Server 5.5/2000/2003 and Windows 2000 Server/Windows Server 2003, identify any updates installed within the last 5 days and the logon name of the user that performed the installation.
  • Network – enumeration of all network cards and check NIC connection status, DNS/WINS configuration, IP gateway settings, primary DNS and domain suffix.
  • Operating system – page table entry (PTE) levels, paged/non-paged pool configuration, CrashOnAuditFail configuration, HeapDeCommitFreeBlock threshold, temporary paths, SystemPages configuration, /3GB and /USERVA configuration, physical address extensions (PAE), version and SKU (i.e. Standard, Enterprise, etc.), Dr. Watson configuration, debug settings, Virtual PC/Virtual Server/VMware detection.

This is not an exhaustive list and changes with each web update pack.

Preventative Maintenance

As mentioned previously, 60% of Exchange incidents reported to Microsoft are traced back to people or process, and not the technology itself. Additionally, mission critical software needs to run on good hardware as the reliability of the system is only as good as the reliability of each of its components and Microsoft claims that of the 90% of Exchange support incidents that are performance related, 50% are due to hardware issues.

Microsoft also claims that 90% of Exchange administrators do not carry out any maintenance until disaster strikes. The cause of this has been identified as a number of areas, including:

  • Low understanding of the issues and problems.
  • No time, resource, or budget to address maintenance tasks.
  • Non-availability of test equipment (and high impact of testing in a production environment).
  • Assumption that the risk of doing nothing outweighs the risk of pro-activity.
  • Technically capable, but see preventative maintenance as boring and time consuming.

When configuring Exchange Server, there are a number of items to consider, discussed in the following paragraphs:

In general, hardware should be selected from the Microsoft Windows Server Catalog and configured in a consistent manner, using high-quality components, the same (recent) firmware and driver levels.

Error correcting code (ECC) memory should be used, and there is little return on investment above 4Gb with Exchange Server.

Microsoft recommends the following disk layout, which separates transaction logs, data, and queues onto separate spindles for reasons of performance and data recovery:

Recommended disk configuration for Exchange Server

RAID 0 (disk striping) provides fast read and write times, with RAID 1 (disk mirroring) adding
redundancy to form RAID 0 + 1. As an alternative to RAID 0 + 1, RAID 5 (disk striping with parity) may be used (requiring less disks), but this configuration is slower to write due to the need to write the parity data.

Disk caching should be disabled (to avoid database corruptions where a transaction may not be successfully written to the disk) and hardware RAID employed (software RAID is too resource intensive). Servers should be specified with enough free disk space allow database maintenance to be performed (ideally 110% of the database file size) and disk compression should never be used on an Exchange server due to the effect on performance. The Microsoft JetStress and LoadSim tools should be used to test that the server is capable of providing the required performance levels.

The Windows Server operating system should be consistent, both in version and configuration. The maximum log size value for the event logs should be at least 16Mb in size and set to overwrite as needed (to allow a reasonable amount of diagnostic information to be captured, but to avoid full log files). Dr Watson should be the default debugger (to allow capture of user dump information) – this would normally be the case but may not be if some development environments are installed on the same computer as Exchange Server (not recommended). Recovery options should be set and the /3GB switch selected in boot.ini if more than 1Gb of RAM is installed (this provides a different memory split between the application and the kernel).

There should be at least two domain controllers for each domain, with at least one GC processor for each 4 Exchange processors (assuming all processors are of a roughly equivalent specification).

Exchange Servers should be configured with circular logging disabled (for most server configurations), a staggered information store maintenance window, and mailbox quotas configured (so the maximum database size is a known value). Permissions should be set by group (not user), a solid naming convention employed for all objects and the administrative notes fields should be used (incidentally, the use of these is a good way to check that the SRS is working where Exchange Server 5.5 servers are in use).

Microsoft has two core cluster configurations for its own Exchange servers:

  • Enterprise datacentres use a 7-node cluster with 4 active Exchange Server 2003 nodes, 1 passive Exchange Server 2003 node and 2 Windows Server 2003 (non-Exchange Server) nodes (for local backups).
  • Regional datacentres use a 5-node cluster with 3 active Exchange Server 2003 nodes, 1 passive Exchange Server 2003 node and 1 Windows Server 2003 node.

To reduce the number of drive letters used, mount points are employed, for example on Exchange virtual server 1:

    C: System
    D: Paging file and MTA data
    E: Storage group 1 data
    F: Storage group 2 data
    G: Storage group 3 data
    H: Storage group 4 data
    E\MP – SMTP
    E\MP – Storage group 1 logs
    F\MP – Storage group 2 logs
    G\MP – Storage group 3 logs
    H\MP – Storage group 4 logs

This configuration has allowed Microsoft to perform a massive server consolidation exercise, removing 2 regional datacentres, 175 servers and 55 physical sites from the Exchange organisation, whilst doubling mailbox quotas.

When preparing for a Microsoft Exchange implementation there are a number of considerations:

  • A server configuration log can be used to enforce consistency and provide information for support staff. It should include firmware and BIOS revisions, installed software and version information, service packs, hotfixes, hardware, services, network configuration, repair and recovery information.
  • An operations logs should be created.
  • Test accounts should be used and a production test server acquired.
  • Operations should be considered through the up-front planning of a maintenance window for patch management and other maintenance processes, generation of a backup/restoration plan and standardised tape formats, and finally generation of a recovery and troubleshooting plan with contingency, oversized storage (for database maintenance), spare parts available locally for immediate replacement and periodic server recovery drills.

Once Exchange has been deployed, the maintenance process comes into play.

Daily tasks:

  • Check event logs (and act on them).
  • Check backup logs.
  • Monitor performance.
  • Check disk space.
  • Check the badmail folders and queues.
  • Check for updates.
  • test mail flow.
  • Back up the server.

Weekly tasks:

  • Compare the server against a baseline configuration.
  • Verify backed-up data with a restore (to a recovery server).

Monthly tasks:

  • ESEUTIL file dump.
  • ESEUTIL integrity check.
  • ISINTEG all tests default mode.

Ad-hoc tasks:

  • ESEUTIL defragmentation (every 12 months or after a large data move).
  • Full disaster recovery test.

Daily online backups should be used, even if the volume shadow copy service (VSS) is used. Online backups check database integrity through checksum verification and full online backups purge transaction logs at the conclusion of the backup. Even whilst the backup is taking place, users can still access mailboxes and public folders.

When monitoring Exchange, compare the recorded counters with a baseline and pay particular attention to:

  • Database\Log Record stalls/sec – average should be below 10 per second and maximum values should not be higher than 100 per second (indicates the number of logs records that cannot be written because the buffers are full – note that Exchange Server 2000 defaults to 84 buffers whilst Exchange Server 2003 defaults to 512).
  • Database\Log Threads Waiting – average should be below 10 (indicates the number of threads waiting to complete an update to the database by writing their data to the log – if too high, the log may be a bottleneck).
  • MSExchangeIS\RPC Requests – should be below 30 at all times (indicates the number of MAPI requests being serviced by the Microsoft Exchange Information Store service – the default maximum is 100).
  • MSExchangeIS\RPC Average Latency – should be below 50ms at all times and should be in the 10-25ms range on a healthy server (averaged over the last 1024 packets and affects how long it takes for a user’s view to change in Outlook).
  • MSExchangeIS\RPC Operations/sec – should rise and fall with MSExchangeIS\RPC Requests (indicates how many RPC operations are being requested and actually responded to).
  • MSExchangeIS\Virus Scan Queue Length – if this is consistently high consider a hardware upgrade (indicates the number of outstanding requests queued for virus scanning).
  • MSExchangeIS Mailbox\Active Client Logons – this is server-specific but should be baselined and monitored (indicates the number of clients which performed any action within the last 10 minutes).
  • Paging File\% Usage – should remain below 50% – high values indicate that the paging file size should be increased or more RAM added to the server (indicates the amount of the paging file used).
  • Memory\Available MBytes (MB) – 50Mb available at all times (indicates the amount of physical memory immediately available to a process).
  • Memory\Pages/sec – below 1000 at all times (indicates the rate at which pages are written to disk to resolve hard page faults).
  • Memory\Pool Nonpaged Bytes – no more than 100Mb (indicates the amount of memory available for kernel objects which must remain in memory and cannot be written to disk).
  • Memory\Pool Paged Bytes – no more than 180Mb, unless a backup or restoration is taking place (indicates the amount of memory available for kernel objects which must remain in memory and can be written to disk).
  • Physical Disk\Average Disk Read/sec – average below 20ms and maximum below 100ms for the database volume, average below 5ms and maximum below 50ms for the transaction log volume, average below 10ms and maximum below 50ms for the SMTP queue volume (indicates the average time to read data from the disk).
  • Physical Disk\Average Disk Write/sec – average below 20ms and maximum below 100ms for the database volume, average below 10ms and maximum below 50ms for the transaction log volume, average below 10ms and maximum below 50ms for the SMTP queue volume (indicates the average time to read data from the disk).

Consider implementing management tools such as MOM to monitor these counters.

ESEUTIL is the Exchange server database utility. Full syntax may be obtained by typing ESEUTIL /? at a command prompt but for maintenance purposes, there are four main options of interest:

  • Offline defragmentation (/d).
  • Integrity (/g).
  • File dump (/m).
  • Copy file (/y).

Because of the potential to cause damage with ESEUTIL, operations should normally be performed with restored data on a non-production server.

Offline defragmentation may be necessary if a large number of mailboxes have been deleted (e.g. following a migration, or if there is a high staff turnover), or following a hard database repair (ESEUTIL /p). It is only recommended if at least 30% of the space taken my the database will be recovered (Event ID 1221 in the application log after an online defragmentation will give a conservative estimate as to how much free space is in the database).

Unless a temporary path is specified as an option, offline defragmentation requires free disk space of at least 110% of the database size to be available as well as the streaming database to reside on the same path.

An integrity check may be necessary to perform a dry run of the repair function – i.e. to validate the checksum for each 4Kb page in the database. Problems that a repair would address are written to a database.integ.raw file which logs all pages in the database, not just those with problems. An integrity check may abort prematurely if problems are of such a nature that a repair is required before some parts of the database can be checked but this does not necessarily mean that a repair would fail. Unless options are specified, an integrity check requires 20% free space.

A file dump allows the viewing of the header information for database, streaming database, checkpoint, online backup patch or transaction log files. The header information can be used to validate that a series of transaction log files forms a matched set and that all files are undamaged, to view space allocation inside the databases, or to view metadata for one or more tables within the database file. An example use of this would be to read the state of an unmounted store (i.e. clean or dirty), to provide some diagnosis as to why the store stopped, prior to mounting the store (which would attempt a soft recovery).

If a database repair is required, this is a last resort, which will strip out orphaned database pages, possibly resulting in data loss. Multiple runs may be required until the entire database is repaired.

A copy file operation simply provides a quick method of copying databases between servers.

ISINTEG is a utility to search an offline information store for integrity weaknesses. Unlike ESEUTIL (which focuses on the physical database), ISINTEG is concerned with the logical structure. It has two modes:

  • Default mode – in which the tool runs the specified tests and reports its findings
  • Fix mode – where options are specified to run tests and attempt a fix where possible.

For maintenance work, default mode is used. Unless addressing a particular issue in the database, the alltests option is typically the most effective course to follow.

In order to run ISINTEG, the Microsoft Exchange Information Store service must be started but the database to be checked dismounted. ISINTEG can be run against remote servers, but not against raw database files or backups.

Links

You had me at EHLO… (the Microsoft Exchange Team blog)
Exchange Server tips and tricks
Exchange Server 2003 Performance: 10 Things to Think About
Exchange Server 2003 Performance and Scalability Guide
Exchange Server 2003 Operations Guide
Using the Exchange tools ISINTEG and ESEUTIL to ensure the health of your information store

10,000 feet view of Microsoft Systems Management Server 2003

Until I started to look at the Microsoft Solution Accelerator for Business Desktop Deployment (Enterprise Edition), which makes use of the Microsoft Systems Management Server (SMS) 2003 Operating System Deployment Feature Pack, I had no experience of using SMS. At my BDD training, Thomas Lee gave a brief overview of SMS, which I have reproduced here for the benefit of anyone else who may find it useful.

SMS OverviewSMS relies on the presence of Microsoft SQL Server (not MSDE, or any other SQL server product, e.g. MySQL).

Each client has an agent installed (the SMS Advanced Client). This allows an administrator to view workstation activity and perform remote takeover operations. It also returns inventory information to the management server which SMS uses to creates collections (e.g. All Windows XP SP2 Workstations), which are stored in the SQL Server database.

Software to be distributed via SMS is packaged and placed on a distribution server. In order to distribute a package, an SMS administrator creates an advertisement, which is pushed to the SMS Advanced client, which in turn will pull the package from the distribution server for installation.

That’s SMS, in a nutshell.

The Microsoft Solution Accelerator for Business Desktop Deployment, the Microsoft Solutions Framework and the Microsoft Operations Framework

Last week, I spent three days learning about the Microsoft Solution Accelerator for Business Desktop Deployment (BDD).

According to Microsoft:

    “The Microsoft Solution Accelerator for Business Desktop Deployment delivers end-to-end guidance for efficient planning, building, testing, and deploying Microsoft Windows XP Professional, Windows XP Tablet PC Edition, and Office Professional 2003 Editions. It helps IT professionals realize a quick return on investment while also setting new standards for reliability, performance, security, and ease of use.”

Basically, BDD is a framework for successful Windows XP desktop deployment. A collection of guidance, sample templates, tools and scripts, wrapped around a Windows XP and Office 2003 installation source, BDD version 1 has been around for some time now, and version 2 has two variant editions:

  • BDD Standard Edition is intended for medium sized organisations, enabling a highly automated (light touch) desktop deployment, requiring a LAN infrastructure with at least one server and sufficient space to store all of the working files and images (although Microsoft do recommend the use of Active Directory and Remote Installation Services) as well as either PowerQuest DeployCenter 5.5 or Symantec Ghost 8.0 and Windows PE 2004.
  • BDD Enterprise Edition is intended for large enterprises, utilising Active Directory, Remote Installation Services, Microsoft SQL Server 2000 and Microsoft System Management Server (SMS) 2003 (with the Operating System Deployment Feature Pack) to deliver a zero touch deployment. To use all of the features in BDD enterprise Edition, including provisioning, Microsoft BizTalk Server 2004, Microsoft Exchange Server 2003 and Microsoft Operations Manager (MOM) Server 2005 are also required. In short, BDD Enterprise can use just about every element of the Windows Server System.

Both editions additionally rely on the Microsoft User State Migration Toolkit (USMT) 2.6, Microsoft Application Compatibility Toolkit 3.0, Microsoft Office Access 2003 Conversion Toolkit, Windows XP Professional with Service Pack 2, and Office Professional 2003 Edition Service Pack 1.

The crossover between the two BDD variants (in terms of the cost of development vs. the number of desktops) is somewhere between 500 and 2000 desktops and the BDD framework scales up to a deployment of many thousands of PCs.

I’ve been working with unattended operating system builds for many years, and the consultancy that I work for (Conchango) already has a highly automated standard operating environment (SOE) process, built around the same technologies. BDD is a formalisation of existing best practices, packaged in a manner that allows an organisation to:

  • Create a software and hardware inventory to assist in deployment planning.
  • Test applications for compatibility with Windows XP Professional and mitigate the compatibility issues discovered during the process.
  • Set up an initial lab environment with deployment and imaging servers.
  • Customise and package core and supplemental applications.
  • Automate desktop image creation and deployment.
  • Ensure that the desktop is hardened to improve security within the environment.
  • Manage processes and technologies to produce a comprehensive and integrated deployment.

Even with BDD in place, rolling out a new standard desktop to hundreds or even thousands of computers, possibly spread across the globe, is not a trivial task and is likely to include a large team of people. In fact BDD is based around the idea of feature teams, each of which is responsible for one area of the solution.

BDD Standard and Enterprise Editions

As can be seen from the diagram above (which relates to BDD Enterprise Edition – BDD Standard Edition does not include provisioning), at the heart of BDD is the business case, and project management. Surrounding this function are the feature teams:

  • Application compatibility remediation – discovering which applications are in use, which are to be migrated, and investigating application compatibility.
  • Infrastructure compatibility remediation – analysing the current state of the infrastructure, to determine whether it is suitable to support the implementation of the new business desktop, then implementing new infrastructure as required. For example, does the network have sufficient bandwidth? Where are the bottlenecks? Are all of the PCs of a suitable specification to run Windows XP? How many will need to be replaced? etc.
  • Computer imaging system – which technologies will be used for deployment? Will this be zero touch or light touch? How many images will there be and what is the basis for creating a separate image? (e.g. functional images, or hardware variations). Functional images can be difficult to manage because of the number of images to keep up to date and in general the number of images should be minimised (in my last company we had just two, hardware-based, images for the various PC models across Europe). Where there are specific hardware concerns, it may be more cost-effective to replace some PCs than to develop an separate image for a particular computer type.
  • Core and supplemental application packaging – packaging those applications which will be included within the image (generally just Microsoft Office, Adobe Reader and some system software), and those supplemental applications that need to be deployed on a per user or per group basis.
  • User state migration – to be avoided wherever possible, user state migration is problematic and time consuming – therefore expensive; however, it is very rare that a deployment will not require the transfer of files and settings.
  • Securing the desktop – security should be included in every area of the design, but it is still necessary for someone to take control of the overall security for the desktop.
  • Deployment process – the process of actually getting the software into PCs throughout the organisation.
  • Preparing for operations – involving operations staff in the project, to ensure that the new platform is taken on board and managed well by the people who are often under-appreciated and over-utilised, and whose buy-in can ensure the success or failure of the entire project.
  • Upgrading Office – ensuring that Microsoft Office functionality is unaffected by the upgrade, paying attention to file versions, macros, file locations, etc.
  • Provisioning – allowing staff to help themselves – provisioning may be as simple as providing a website for users to reset their password after answering a few security questions, or it may be a process to request a new application, with full workflow throughout the approval process right up to the automatic deployment of the application to the desktop.

Of course some roles may be combined, depending on the size of the organisation. In fact the whole point about BDD is that it is designed to scale.

Fundamental to the whole process is the Microsoft Solutions Framework (MSF). MSF provides people and process guidance to help teams and organisations become more successful in delivering business-driven technology solutions to their customers. It is a deliberate and disciplined approach to technology projects based on a defined set of principles, models, disciplines, concepts, guidelines, and proven practices from Microsoft.

MSF Process Model and BDD

At the heart of MSF is the process model, which includes a five stage lifecycle for the solution. Based on phases and milestones, at each stage there are clear requirements in order to justify the ongoing cost of development. For example, there is no value in spending time, effort and money on planning the solution until the vision and scope for the project has been agreed.

The vision may be a single global desktop but the scope may impose restraints on this. In my opinion, envisioning is one of the most important areas of any desktop deployment and is also one of the most overlooked, generally because a customer can’t see the value in up-front planning and needs to be seen to deliver something to the business as soon as possible.

As the solution is deployed (and during preparation for operations), the Microsoft Operations Framework (MOF) is employed to provide operational guidance for the management of the solution. MOF is based on the UK government IT Infrastructure Library (ITIL) – the most widely accepted approach to IT Service Management in the world – and is fundamentally split into a set of models: a process model, a team model and a risk management discipline, as well as a set of service management functions.

MOF Team Model

The MOF team model organises the IT operations group into several role clusters – individuals or groups who perform related activities to accomplish a particular component of an IT service, based on industry best practices for structuring operations teams. MOF then provides additional guidance that applies collectively and individually to the role clusters.

MOF Process Model

The MOF process model assumes that the main responsibility of the IT operations groupÂ’ is managing change in the IT environment. The most effective way to deal with change throughout the lifespan of a service is to group related changes together into a package called a release, so that the changes can be planned and managed as a unit. The MOF process model describes a life cycle that can be applied to any release and the processes and activities that make up each part of that life cycle and groups similar service management functions (SMFs) into each of four quadrants relating to a specific mission of service.

MOF Risk Management Discipline

The MOF risk management discipline applies proven risk management techniques to the daily problems faced by operations staff. Many models, frameworks, and processes exist for managing risks and all share similarities in how they identify and manage risk. MOF applies key principles, along with customised terminology, structured and repeatable risk analysis and evaluation process, integrated within a larger operations framework.

All of the above is just an overview – I recommend that anyone looking to manage a major desktop deployment considers using the BDD framework, and that any organisation running on a primarily Microsoft desktop and server platform takes a serious look at MSF and MOF.

Technology’s role in the demise of the English language

The English language is dying.

I know that languages evolve over time and that change is inevitable, but I would say the vast majority of people in England do not write (or even speak) good English. Witness the number of signs with mis-placed apostrophes (e.g. HGV’s use next entrance) – and one of my recent customers even has painted markings on the surface of their car park which suggest walkers possess that particular area (i.e. pedestrian’s).

My own English is far from perfect; but I can blame that on being a child of the 1970s and 1980s who had a state school education. I remember one teacher at my middle school who was so frustrated as the class struggled with basic punctuation such as full stops, commas and apostrophes that they decided not to teach us how to use semi-colons and colons. That was that and I never learnt how to use them.

So what has this got to do with a (we)blog about technology? Well, I recently read Lynne Truss’ best seller “Eats, Shoots and Leaves: The Zero Tolerance Approach to Punctuation“. In the last chapter, Truss discusses technology’s role in the destruction of our language. To quote:

“…by tragic historical coincidence a period of abysmal under-educating in literacy has coincided with this unexpected explosion of global self-publishing. Thus people who don’t know their apostrophe from their elbow are positively invited to disseminate their writings to anyone on the planet stupid enough to double-click and scroll”.

She continues:

“…Even in the knowledge that our punctuation has arrived at its present state by a series of accidents; even in the knowledge that there are at least seventeen rules for the comma, some of which are beyond explanation by top grammarians – it is a matter for despair to see punctuation chucked out as worthless by people who don’t know the difference between who’s and whose, and whose bloody automatic ‘grammar checker’ can’t tell the
difference either”.

I did chuckle when I read about Bob Hirschfield’s pluperfect virus (the Strunkenwhite Virus), which first appeared in the Washington Post. Intended to provide a satirical view on the rise in hoax virus e-mails, it describes a virus (named Strunkenwhite after the authors of a classic guide to good writing), which returns e-mail messages that have grammatical or spelling errors.

Somewhat unfairly IMHO ;-), Truss also attacks emoticons but all of this does leave me wondering whether my son will grow up to read text or txt, and, does all of this, like, really matter as the erosion of our written language is just part of a wider issue with the spoken form, innit?

In the UK, The Economist recently ran a poster campaign which read something like:

“You can so tell the people who don’t like read the Economist”.

For those of us who do care about the correct use of language, Answers.com provides an online dictionary with definitions (e.g. blog), pronunciation, explanations (courtesy of Wikipedia); or there is WordSpy (the Website devoted to lexpionage, the sleuthing of new words and phrases).

I commend these as examples of where technology can help us to become more expressive in our online use of language.

Long live the English language!


Online dictionary, thesaurus, encyclopedia and much more…

Microsoft buys into the anti-virus market

Following Microsoft’s recent foray into the anti-spyware market and ending months of speculation, Microsoft announced today that it is to attack another form of malware through its purchase of Sybari Software.

Whether anti-virus technologies will be included within Windows (alongside the Windows Firewall), or made available as a separate download (as for Microsoft Windows AntiSpyware) is yet to be seen but with the US Department of Justice and the European Union already investigating the bundling of middleware within Windows it will be interesting to see how Microsoft positions its new acquisition.

Sending SMS messages from within Outlook

A couple of months back, my colleague James Simmonds wrote about a free Microsoft Office SMS Add-in (MOSA) which allows the sending (but not receipt) of text messages from within the Outlook 2003 client (in this context, SMS is Short Message Service – not Systems Management Server). I finally downloaded MOSA this afternoon and it looks good.

MOSA can only send messages using a GSM mobile handset that supports the Protocol Description Unit (PDU) standard. Initially, I attempted to send a message using a standard modem (without success – generating a “the modem does not support messages in PDU format” message) but once I paired my notebook PC with a Nokia 6310i via Bluetooth, everything jumped into life.

For anyone wondering (as I was), what exactly the PDU format is, I found some further information about SMS and the PDU format as well as a PDU string analyser and converter.

Preselecting English (United Kingdom) settings during Windows XP setup

By default, Windows XP installs English (United States) as the input language. It is possible to add other languages, for example English (United Kingdom), but if you try to remove English (United States) you are prompted that it is in use and will be removed after the next reboot. This may only be a minor inconvenience, but can be circumvented using an unattended setup file, either for a fully- or partially-unattended build.

Within the [Unattended] section of the answer file, a KeyboardLayout= entry may be specified. The Microsoft Windows Preinstallation Reference states that this entry is used to specify the type of keyboard layout to install during text-mode setup and I have found that by using a United Kingdom keyboard for text-mode setup, no United States entries are created during GUI-mode setup.

The KeyboardLayout= entry must match one of the strings (in quotation marks after the =) in the [Keyboard Layout] section of txtsetup.sif (which is found in the Windows XP installation source).

For reference, my RIS-based unattended installation uses the following entries to specify UK-only regional settings and location:

[Unattended]
KeyboardLayout="United Kingdom"

[RegionalSettings]
Language=00000809
SystemLocale=00000809
UserLocale=00000809
InputLocale=0809:00000809
UserLocale_DefaultUser=00000809
InputLocale_DefaultUser=00000809

Further information on the available regional settings may be found in Microsoft knowledge base article 289125 and the Microsoft global development website has a full list of the available National Language Support (NLS) code pages.