Reflecting on Byte Night 2007

Byte Night
NCH

A few weeks ago I posted a blog entry about my involvement in Byte Night 2007 – the UK IT industry’s annual sleep out in support of young people who are coping with life after care or facing homelessness. I’ve just got home from my night sleeping rough in London and as I’ve been really amazed by the generosity of some people, including those of you who don’t know me personally but who added a donation as a way of supporting this site, I thought I’d post an update.

So was it all worth it? Yes.

On a financial level, I’ve raised £1062.26 for NCH, the children’s charity – thank you to everyone who contributed to this fantastic total. Combined with the rest of Team Fujitsu that’s £7266.19 and I understand the total raised by all participants was around the £280,000 mark [update: £304,389 as at 12 October 2007].

Mark Wilson sleeping rough at Byte Night 2007On a personal level, I found it all rather humbling. I want to stress that I can never truly understand what it’s like to be homeless. I’m back home now with my family and tonight I will sleep in my own bed but it’s good to remember those who are less fortunate and to do something to help. Some people inferred that Byte Night is just a “jolly”, a chance to network, and I will admit that it was great fun to attend a charity reception last night hosted by Jenny Agutter (who joined in the sleepover, as she has on several previous occasions); to have a weather forecast from Siân Lloyd (luckily, it was dry, with some cloud cover, and hence well above freezing); to watch Trinny Woodall judge the Byte Night nightcaps that had been customised by celebrities including Dame Ellen MacArthur and Sandi Toksvig. As I walked along the South Bank and through Westminster in the early hours this morning, I saw those who were genuinely homeless and realised how vulnerable I felt. I cannot imagine what it is like to be in their situation, every day and night, let alone as a child. Even though I was taking part in an organised sleepover with over 250 like-minded people the point is not really about being truly “on the streets” – it’s about raising awareness of this important issue and funds for NCH to help vulnerable children and young people.

So what’s this got to do with a technology blog? Not a lot, except that the Byte Night participants all work in the UK IT sector and several IT companies (sadly not the one that I work for) added their support the event, whether that was by providing a fleece to keep me warm (thank you Dell), groundsheets/survival bags and umbrellas (thank you Harvey Nash), or hosting the reception (thank you Ernst and Young).

That’s (almost) all I have to say on this – technology-focussed blogging will resume shortly – but before I sign off, it’s not too late to help me reach the elusive £2000 personal target – my fundraising site will remain in place at http://www.justgiving.com/markwilson-bytenight07/ until 6 December 2007 – and, if you want to know a little more of what it’s all about, watch the video below.

Some more about virtualisation with Xen (including installation on RHEL5)

I got a little confused by what at first appeared to be conflicting information in the XenSource demonstration that I saw earlier this week and the Xen module in my Red Hat Enterprise Linux (RHEL) training. It seems that I hadn’t fully grasped the distinction between Xen as commercialised by Red Hat and Xen as commercialised by XenSource and in this post I shall attempt to clarify the situation.

Somewhat confusingly, the version 4 XenSource products include version 3.1 of the Xen hypervisor. I’d assumed that this was pretty much identical to the Xen 3.0.3 kernel that I can install from the RHEL DVD but it seems not. Roger Baskerville, XenSource’s Channel Director for EMEA explained to me that it’s important to differentiate between the OSS Xen and the Xen Source commercial products and that whilst both Red Hat and XenSource use a snapshot of OSS Xen 3.x.x, the XenSource snapshot is more recent than the one in RHEL due to the time that it takes to incorporate various open source components into a operating system. Furthermore, XenExpress, XenServer and XenEnterprise are designed for bare-metal deployment with little more than the hypervisor and a minimal domain-0 (a privileged virtual machine used to control the hypervisor) whereas RHEL’s domain-0 is a full operating system.

The XenSource microkernel is based on CentOS (itself a derivative of Red Hat Enterprise Linux) with only those services that are needed for virtualisation along with a proprietary management interface and Windows drivers. Ultimately, both the XenSource and RHEL models include a Xen hypervisor interacting directly with the processor, virtual machines (domain-U) and domain-0 for disc and network traffic. Both use native device drivers from the guest operating system, except in the case of full virtualised VMs (i.e. Windows VMs) in which case the XenSource products use signed proprietary paravirtualised Windows drivers for disk access and network traffic (XenSource Tools).

So when it comes to installation, we have two very different methods – whereas XenSource is a bare-metal installation, installing Xen on RHEL involves a number of RPMs to create the domain 0 environment. This is how it’s done:

Method 1 (the simple way) is to select all of the virtualisation tools during operating system installation. Alternatively, method 2 involves installing individual RPMs. At first I just installed the packages containing xen in their name from the /Server directory on the RHEL installation DVD (kernel-xen-2.6.18-8.el5.i686.rpm, kernel-xen-devel-2.6.18-8.el5.i686.rpm and xen-libs.3.0.3025.el5.i386.rpm) but even after rebooting into the Xen kernel I found that there were no management tools available (e.g. xm). Fortunately, I found a forum post that explained my mistake – I had installed the kernel and userspace libraries but not any of the tools/commands and another post that explains how to install Xen on RHEL:

cd /Server
rpm -Uvh kernel-xen-2.6.18-8.el5.i686.rpm
rpm -Uvh kernel-xen-devel-2.6.18-8.el5.i686.rpm
rpm -Uvh xen-libs-3.0.3-25.el5.i386.rpm
rpm -Uvh bridge-utils-1.1-2.i386.rpm
rpm -Uvh SDL-1.2.10-8.el5.i386.rpm
cd /VT
rpm -Uvh --nodeps libvirt-0.1.8-15.el5.i386.rpm

(--nodeps resolves a cyclic dependency between xen, libvert, libvirt-python and python-virtinst.)

rpm -Uvh libvirt-python-0.1.8-15.el5.i386.rpm
rpm -Uvh python-virtinst-0.99.0-2.el5.noarch.rpm
rpm -Uvh xen-3.0.3-25.el5.i386.rpm

At this point, it should be possible to start the Xen daemon (as long as a reboot onto the Xen kernel has been performed – either from manual selection or by changing the defaults in /boot/grub/menu.lst) using xend start. If the reboot took place after kernel installation but prior to installing all of the tools (as mine did) then chkconfig --list should confirm that xend is set to start automatically and in future it will not be necessary to start the Xen daemon manually. xm list should show that Domain-0 is up and running.

Finally, the Xen Virtual Machine Manager can be installed:

cd /Server
rpm -Uvh gnome-python2-gnomekeyring
cd /VT
rpm -Uvh virt-manager

Having installed Xen on RHEL, I was unable to install any Windows guests because the CPU on my machine doesn’t have Intel-VT or AMD-V extensions. It’s also worth noting that my attempts to install Xen on my notebook PC a few months ago were thwarted as, every time I booted into the Xen kernel, I was greeted with the following error:

KERNEL PANIC: Cannot execute a PAE-enabled kernel on a PAE-less CPU!

It turns out that 400MHz front side bus Pentium M processors do not support physical address extensions (PAE) – including the Pentium M 745 (“Dothan”) CPU that my notebook PC uses – and PAE is one of the pre-requisites for Xen.

Finally, it’s worth noting that my RHEL installation of Xen is running on a 32-bit 1.5GHz Pentium 4 (“Willamette”) CPU whereas the XenSource products require that the CPU supports a 64-bit instruction set. The flags shown with cat /proc/cpuinfo can be a bit cryptic but Todd Allen’s CPUID makes things a little clearer (if not quite as clear as CPU-Z is for Windows users).

Microsoft infrastructure optimisation

Infrastructure optimisationI don’t normally write about my work on this blog (at least not directly) but this post probably needs a little disclaimer as, a few months ago I started a new assignment working in my employer’s Microsoft Practice and, whilst I’m getting involved in all sorts of exciting stuff, it’s my intention that a large part of this work will involve consultancy engagements to help customers understand the opportunities for optimising their infrastructure. Regardless of my own involvement in this field, I’ve intended to write a little about Microsoft’s infrastructure optimisation (IO) model since I saw Garry Corcoran of Microsoft UK present at the Microsoft Management Summit highlights event back in May… this is a little taster of what IO (specifically Core IO) is about.

Based on the Gartner infrastructure maturity model, the Microsoft infrastructure optimisation model is broken into three areas around which IT and security process is wrapped:

  • Core infrastructure optimisation.
  • Business productivity infrastructure optimisation.
  • Application platform infrastructure optimisation.

Organisations are assessed on a number of capabilities and judged to be at one of four levels (compared with seven in the Gartner model):

  • Basic (“we fight fires” – IT is a cost centre) – an uncoordinated, manual infrastructure, knowledge not captured.
  • Standardised (“we’re gaining control” – IT becomes a more efficient cost centre) – a managed IT infrastructure with limited automation and knowledge capture.
  • Rationalised (IT is a business enabler) – managed and consolidated IT infrastructure with extensive automation, knowledge captured for re-use.
  • Dynamic (IT is a strategic asset) – fully automated management, dynamic resource usage, business-linked SLAs, knowledge capture automated and use automated.

Infrastructure optimisation overview diagramIt’s important to note that an organisation can be at different levels for each capability and that the capability levels should not be viewed as a scorecard – after all, for many organisations, IT supports the business (not the other way around) and basic or standard may well be perfectly adequate but the overall intention is to move from IT as a cost centre to a point where the business value exceeds the cost of investment. For example, Microsoft’s research (carried out by IDC) indicated that by moving from basic to standardised the cost of annual IT labour per PC could be reduced from $1320 to $580 and rationalisation could yield further savings down to $230 per PC per annum. Of course, this needs to be balanced with the investment cost (however that is measured). Indeed, many organisations may not want a dynamic IT infrastructure as this will actually increase their IT spending; however the intention is that the business value returned will far exceed the additional IT costs – the real aim is to improve IT efficiencies, increase agility and to shift the investment mix.

Microsoft and its partners make use of modelling tools from Alinean to deliver infrastructure optimisation services (and new models are being released all the time). Even though this is clearly a Microsoft initiative, Alinean was formed by ex-Gartner staff and the research behind core IO was conducted by IDC and Wipro. Each partner has it’s own service methodology wrapped around the toolset but the basic principles are similar. An assessment is made of where an organisation is currently at and where they want to be. Capability gaps are assessed and further modelling can help in deriving those areas where investment has the potential to yield the greatest business benefit and what will be required in order to deliver such results.

It’s important to note that this is not just a technology exercise – there is a balance to be struck between people, processes and technology. Microsoft has published a series of implementer resource guides to help organisations to make the move from basic to standardised, standardised to rationalised and from rationalised to dynamic.

Links

Core infrastructure self-assessment.
Microsoft infrastructure optimisation journey.

Windows Server Virtualization unwrapped

Last week, Microsoft released Windows Server 2008 Release Candidate 0 (RC0) to a limited audience and, hidden away in RC0 is an alpha release of Windows Server Virtualization (the two updates to apply from the %systemroot%\wsv folder are numbered 939853 and 929854).

I’ve been limited in what I can write about WSV up to now (although I did write a brief WSV post a few months back); however at yesterday’s event about creating and managing a virtual environment on the Microsoft platform (more on that soon) I heard most of what I’ve been keeping under wraps presented by Microsoft UK’s James O’Neill and Steve Lamb (and a few more snippets on Tuesday from XenSource), meaning that it’s now in the public domain and I can post it here (although I have removed a few of the finer points that are still under NDA):

  • Windows Server Virtualization uses a totally new architecture – it is not just an update to Virtual Server 2005. WSV is Microsoft’s first hypervisor-based virtualisation product where the hypervisor is approximately 1MB in size and is 100% Microsoft code (for reliability and security) – no third party extensions. It is no more than a resource partition in order to provide access to hardware and not opening the hypervisor to third parties provides protection against theoretical hyperjacking attacks such as the blue pill (where a rootkit is installed in the hypervisor and is practically impossible to detect).
  • WSV requires a 64-bit CPU and hardware assisted virtualisation (Intel VT or AMD-V) enabled in the BIOS (often disabled by default).
  • There will also be two methods of installation for WSV:
    • Full installation as a role on Windows Server 2008 (once enabled, a reboot “slides” the hypervisor under the operating system and it becomes virtualised).
    • Server core role for the smallest and most secure footprint (with the advantage of fewer patches to apply).
  • Initial builds require a full installation but WSV will run on Server Core.
  • The first installation becomes the parent, with subsequent VMs acting as children. The parent has elevated permissions. The host/guest relationship no longer applies with the hypervisor model; however if the parent fails, the children will also fail. This may be mitigated by clustering parents and using quick migration to fail children over to another node.
  • Emulated drivers are still available with wide support (440BX chipset, Adaptec SCSI, DEC Ethernet, etc.) but they have a costly performance overhead with multiple calls back and forth between parent and child and context switches from user to kernel mode. WSV also includes a synthetic device driver model with virtual service providers (VSPs) for parents and virtual service clients (VSCs) for children. Synthetic drivers require no emulation and interact directly with hardware assisted virtualisation, providing near-native performance. XenSource drivers for Linux will be compatible with WSV.
  • There will be no USB support – Microsoft see most USB demand for client virtualisation and although USB support may be required for some server functions (e.g. smartcard authentication), this will not be provided in the initial WSV release
  • Microsoft views memory paging to be of limited use and states that over-committing RAM (memory ballooning) is only of practical use in a test and development environment. Furthermore it can actually reduce performance where applications/operating systems attempt to make full use of all available memory and therefore cause excessive paging between physical and virtual RAM. Virtual servers require the same volumes of memory and disk as their physical counterparts.
  • In terms of operating system support, Windows Vista and Server 2008 already support synthetic device driver (with support being added to Windows Server 2003). In response to customer demand, Microsoft has worked with XenSource to provide a platform that will allow both Linux and Windows workloads with near native performance though XenSource’s synthetic device drivers for Linux. Emulation is still available for other operating systems.
  • Virtual Server VMs will run in WSV as the VHD format is unchanged; however virtual machine additions will need to be removed and replaced with ICs (integration components) for synthetic drivers using the integration services setup disk (similar to virtual machine additions, but without emulation) to provide enlightenment for access to the VMbus.
  • Hot addition of resources is not included in the initial WSV release.
  • Live migration will not be included within the first WSV release but quick migration will be. The two technologies are similar but quick migration involves pausing a VM, writing RAM to a shared disk (saving state) and then loading the saved state into RAM on another server and restarting the VM – typically in around 10 seconds – whereas live migration copies the RAM contents between two servers using an iterative process until there are just a few dirty pages left, then briefly pausing the VM, copying the final pages, and restarting on the new host with sub-second downtime.
  • WSV will be released within 180 days of Windows Server 2008.

Looking forward to Windows Server 2008: Part 1 (Server Core and Windows Server Virtualization)

Whilst the first two posts that I wrote for this blog were quite generic, discussing such items as web site security for banks and digital rights management, this time I’m going to take a look at the technology itself – including some of the stuff that excites me right now with Microsoft’s Windows Server System.

Many readers will be familiar with Windows XP or Windows Vista on their desktop but may not be aware that Windows Server operating systems also have a sizable chunk of the small and medium size server market.   This market is set to expand as more enterprises implement virtualisation technologies (running many small servers on one larger system, which may run Windows Server, Linux, or something more specialist like VMware ESX Server).

Like XP and Vista, Windows 2000 Server and Advanced Server (both now defunct), Windows Server 2003 (and R2) and soon Windows Server 2008 have their roots in Windows NT (which itself has a lot in common with LAN Manager).  This is both a blessing and a curse as while the technology has been around for a few years now and is (by and large) rock solid, the need to retain backwards compatibility can also mean that new products struggle to balance security and reliability with legacy code.

Microsoft is often criticised for a perceived lack of system stability in Windows but it’s my experience that a well-managed Windows Server is a solid and reliable platform for business applications.  The key is to treat a Windows Server computer as if it were the corporate mainframe rather than adopting a   personal computer mentality for administration.  This means strict policies controlling the application of software updates and application installation as well as consideration as to which services are really required.

It’s this last point that is most crucial.  By not installing all of the available Windows components and by turning off non-essential services, it’s possible to reduce the attack surface for any would-be hacker.  A reduced attack surface not only means less chance of falling foul of an exploit but it also means less patches to deploy.  It’s with this in mind that Microsoft produced Windows Server Core – an installation option for the forthcoming Windows Server 2008 product (formerly codenamed Longhorn Server).

As the name suggests, Windows Server Core is a version of Windows with just the core operating system components and a selection of server roles available for installation (e.g. Active Directory domain controller, DHCP server, DNS server, web server, etc.).  Server Core doesn’t have a GUI as such and is entirely managed from a command prompt (or remotely using standard Windows management tools).  Even though some graphical utilities can be launched (like Notepad), there is no Start Menu, no Windows Explorer, no web browser and, crucially, a much smaller system footprint.  The idea is that core infrastructure and application servers can be run on a server core computer, either in branch office locations or within the corporate data centre and managed remotely.  And, because of the reduced footprint, system software updates should be less frequent, resulting in improved server uptime (as well as a lower risk of attack by a would-be hacker).

If Server Core is not exciting enough, then Windows Server Virtualization should be.  I mentioned virtualisation earlier and it has certainly become a hot topic this year.  For a while now, the market leader (at least in the enterprise space) has been VMware (and, as Tracey Caldwell noted a few weeks ago, VMware shares have been hot property), with their Player, Workstation, Server and ESX Server products.  Microsoft, Citrix (XenSource) and a number of smaller companies have provided some competition but Microsoft will up the ante with Windows Server Virtualization, which is expected to ship within 180 days of Windows Server 2008.  No longer running as a guest on a host operating system (as the current Microsoft Virtual Server 2005 R2 and VMware Server products do), Windows Server Virtualization will directly compete with VMware ESX Server in the enterprise space, with a totally new architecture including a thin “hypervisor” layer facilitating direct access to virtualisation technology-enabled hardware and allowing near-native performance for many virtual machines on a single physical server.  Whilst Microsoft is targeting the server market with this product (they do not plan to include the features that would be required for a virtual desktop infrastructure, such as USB device support and sound capabilities) it will finally establish Microsoft as a serious player in the virtualisation space (even as the market leader within a couple of years).  Furthermore, Windows Server Virtualization will be available as a supported role on Windows Server Core; allowing for virtual machines to be run on an extremely reliable and secure platform.  From a management perspective there will be a new System Center product – Virtual Machine Manager, allowing for management of virtual machines across a number of Windows servers, including quick migration, templated VM deployment and conversion from physical and other virtual machine formats.

Windows Server Core and Windows Server Virtualization are just two of the major improvements in Windows Server 2008.  Over the coming weeks, I’ll be writing about some of the other new features that can be expected with this major new release.

Windows Server 2008 will be launched on 27 February 2008.  It seems unlikely that it will be available for purchase in stores at that time; however corporate users with volume license agreements should have access to the final code by then.  In the meantime, it’s worth checking out Microsoft’s Windows Server 2008 website and the Windows Server UK User Group.

[This post originally appeared on the Seriosoft blog, under the pseudonym Mark James.]

Understanding how open source and commercial software products co-exist

Its sometimes difficult to understand how open source (i.e. community driven) software and commercial operations can co-exist. Yesterday’s XenSource presentation gave me a great example of how the model works:

  • The open source Xen project provides code which generally falls under three categories:
    • Stable and tested
    • Not tested
    • Unstable
  • XenSource (the commercial company) takes the stable and tested elements of the solution and combines these with proprietary elements to produce a commercial product. It also contributes code to the open source project along with bug fixes.
  • XenSource has the resources to provide enterprise-level quality assurance and testing, including manual and automated regression testing, optimisations and beta test programmes. These contribute further fixes for inclusion in the product(s).
  • The result is a commercial product (in this case three products) which promote open source software development at the same time as providing a revenue stream for ongoing product development.

My last question to ask is “what about the community developers who devoted their time to the project?” – it would be interesting to hear how those who contribute code that then makes profit for faceless shareholders feel but I suspect they derive their benefits in a far more altruistic manner:

  • A feeling of community and pride in having contributed to a widely-deployed software product.
  • Access to source code in order to develop and extend the community versions of the product.
  • In the case of the project founders and leaders, financial recognition through their involvement in the commercial company.

Finally, it’s worth remembering that just because software is open source doesn’t mean it’s free (of charge).

Low-cost enterprise virtualisation from XenSource

As I write this, I’m on the train to attend a Microsoft event about creating and managing a virtual environment on the Microsoft platform (that’s something that I’m doing right now to support some of my business unit’s internal systems). I’m also on the Windows Server Virtualization TAP program (most of the information I get from that is under NDA – I’m saving it all up to blog when it becomes public!) and I have a good working knowledge of VMware’s product set, including some of the (non-technical) issues that a virtualisation project can face. With that in mind, I thought I’d take the time to attend one of XenSource‘s Unify Your Virtual World events yesterday to look at how this commercial spinoff from the open source Xen project fits into the picture.

From my point of view, the day didn’t start well: the location was a hotel next to London Heathrow airport with tiny parking spaces at an extortionate price (at least XenSource picked up the bill for that); there was poor signage to find the XenSource event; and stale pastries for breakfast; however I was pleased to see that, low key as the event was, the presenters were accessible (indeed John Glendinning, XenSource VP for Worldwide Sales, was actively floor-walking). And once the presentation got started things really picked up with practical demonstrations supplemented with PowerPoint slides (not OpenOffice Impress as I would expect from an open source advocate) only to set the scene and provide value, rather than the typical “death by PowerPoint” product pitch with only a few short demonstrations.

XenSource logoXenSource was founded in 2005 by the creators and leaders of the Xen hypervisor open source project and in that short time it has grown to the point where it is now a credible contender in the the x86 virtualisation space – so much so that they are currently in the process of being acquired by Citrix Systems. Rather than trying to dominate in the entire market, XenSource’s goal is clear – they provide a core virtualisation engine with partners providing the surrounding products for storage, backup, migration, etc., ensuring that there are multiple choices for enterprises that deploy the XenSource virtualisation products. The XenSource “engine” is a next generation hypervisor which delivers high performance computing through its use of paravirtualisation and hardware assist technologies. They also try to cast off the view of “it’s Linux so it must be difficult” with their 10 minutes to Xen model with no base operating system or RPMs to install, demonstrating the installation of a Xen server on bare metal hardware in around 10 minutes from a PXE boot (other deployment options are available).

From an architectural standpoint, the Xen hypervisor is very similar to Microsoft’s forthcoming Windows Server Virtualization model, providing an environment known as Domain 0. Memory and CPU access is facilitated by the hypervisor, providing direct access to hardware in most cases although for Windows VMs to make use of this the hardware must support Intel-VT or AMD-V (virtualisation hardware assistance). Storage and network access use a high performance memory bus to access the Domain 0 environment which itself makes use of standard Linux device drivers, ensuring broad hardware support.

One of the problems with running multiple virtual machines on a single physical server is the control of access to hardware. In a virtualisation environment that makes use of emulated drivers (e.g. VMware Server, Microsoft Virtual Server) the guest operating system is not aware that it is running in a virtual environment and any hardware calls are trapped by the virtual machine management layer which manages interaction with the hardware. The paravirtualised model used for Linux VMs allows the guest operating system to become aware that it is virtualised (known as enlightenment) and therefore to make a hypercall (i.e. a call to the hypervisor) that can interact directly with hardware. For non-paravirtualised operating systems that use the high performance memory bus (e.g. current versions of Windows), full virtualisation is invoked whereby the virtual machine believes it owns the hardware but in reality the hardware call is trapped by the virtualisation assist technology in the processor and passed to the hypervisor for action. For this reason, Intel VT or AMD-V capabilities are essential for Windows virtualisation with Xen.

XenSource view the VMware ESX Server model of hypervisor-based virtualisation as “first generation” – effectively using a mini-operating system kernel that includes custom device drivers and requires binary patching at runtime with a resulting performance overhead. In contrast, the “second generation” hypervisor model allows for co-operation between guests and the hypervisor, providing improved resource management and input/output performance. Furthermore, because the device drivers are outside the hypervisor, it has a small footprint (and consequentially small attack surface from a security standpoint) whilst supporting a broad range of hardware and providing significant performance gains.

XenSource claim that paravirtualised Linux on Xen has only a 0.5-2% latency (i.e. near-native performance) and even fully virtualised Windows on Xen has only a 2-6% latency (which is comparible with competing virtualisation products).

There are three XenSource products:

  • XenExpress – a production-ready, entry level system for a standalone server (free of charge).
  • XenServer – a mid-range multi-server virtualisation platform
  • XenEnterprise – high capacity dynamic virtualisation for the enterprise.

Because the three products share the same codebase (unlike Microsoft Virtual PC/Virtual Server or VMware Workstation/Server/ESX Server), upgrade is as simple as supplying a license key to unlock new functionality. For XenServer and XenEnterprise, there are both perpetual and annual licensing options (licensed per pair of physical CPU sockets) at a significantly reduced cost when compared with VMware Virtual Infrastructure 3 (VI3).

The version 4 XenSource products were released in August 2007 with an update planned for the last quarter of 2007. New features in version 4 include:

  • XenMotion (XenEnterprise only) for seamless movement of virtual machines between hosts without any noticeable downtime (cf. VMware VMotion).
  • XenResourcePools (XenEnterprise only) to join virtual servers and manage virtualised resources as a logical group, supporting automatic VM placement and XenMotion with shared storage (volume-based iSCSI and file-based NFS, using the .vhd disk format), authentication, authorisation and resource configuration (similar to the model in VMware Virtual Center).
  • Xen64, a true 64-bit hypervisor providing scalability and support for enterprise applications in either a 32- or 64-bit environment with quality of service controls on resources, dynamic guest configuration and supporting up to:
    • 128GB RAM (32GB per guest, hotplug addition for supported Linux operating systems).
    • 1-32 pCPUs (1-8 vCPUs per guest).
    • 1-8 NICs (1-7 NICs per guest – hotplug addition and removal).
    • 1-128 storage repositories (16TB per repository with hotpluggable disks).
  • XenCenter, which provides a graphical virtualisation management interface, with guided wizards and guest templates for host and resource pool configuration on multiple servers, storage and networking configuration and management, VM lifecycle management and import/export (cf. VMware Virtual Center). Whilst CLI commands are also available XenCenter is a Microsoft.NET application for Windows operating systems which makes use of the latest Windows user interface standards. Because XenCenter makes use of a distributed configuration database there is no dependency on a single SQL Server and management can fail over between virtual host servers.
  • XenAPI, a secure and remoteable programming interface for third-party and customer integration with existing products and processes including the xe commands for system control.

One example of the XenSource approach to providing additional functionality through partnerships is the agreement with Symantec whereby Symantec (formerly Veritas) Storage Foundation will be embedded into XenEnterprise (providing dynamic fibre-channel multipathing for redundancy, load balancing, resilience and speed); a new product called XenEnterprise High Availability will be developed for virtual machine failover; and Veritas NetBackup will be offered for data protection and backup of critical applications running on XenEnterprise virtual machines (via the NetBackup Agent, also supporting snapshots when used with Symantec Storage Foundation). Rather than re-certify systems for virtualisation, XenSource will accept Symantec’s certified plugins for common OEM architectures and, because Symantec Storage Foundation is already widely deployed, existing investments can be maintained.

In terms of demonstration, I was impressed by what I saw. XenSource demonstrated a bare metal installation in around 10 minutes and were able to show all the standard virtualisation demonstrations (e.g. running a ping, copying files, or watching a video whilst performing a live migration with no noticeable break in service). The XenCenter console can be switched between VNC and RDP communications and Xen makes use of is own .xva Xen virtual appliance format with Microsoft .vhd virtual hard disks. Conversion from VMware .vmdk files is possible using the supplied migration tools (there are Linux P2V tools included with the XenSource products but for Windows migrations it’s necessary to use products from partners such as PlateSpin and LeoStream) and templated installations can also be performed with simple conversion between running VMs and templates. When cloning virtual machines, there are options for “fat clones” whereby the whole disk is copied or thin provisioning using the same image and a differencing drive. Virtual machines can use emulated drivers or XenSource Tools can be installed for greater control from the console. Storage can be local, NFS or iSCSI based with fibre channel storage and logical volume management expected in the next release.

It’s clear that XenSource see VMware as their main competitor in the enterprise space and it looks to me as if they have a good product which provides most of the functionality in VMware VI3 Enterprise Edition (all of the functionality in VMware VI3 Standard Edition) at a significantly lower price point. The Citrix aquisition will provide the brand ownership that many sceptics will want to see before they buy an open source product, the partnership model should yield results in terms of flexibility in operations and it’s clear that the development pace is rapid. With XenSource going from strength to strength and Microsoft Windows Server Virtualization due to arrive around the middle of next year, VMware need to come up with something good if they want to retain their dominance of the x86 virtualisation market.

One of the reasons why OpenXML document formats are so useful

I’ve written before the frustrations of working with OpenXML document formats on a Mac but this evening I found out that a lack of native support for these files can be very useful. I downloaded a .docx file that I wanted to lift some graphics from – of course, Mac OS X and Office 2004 for Mac didn’t recognise the file but it’s really just a .zip file and after letting StuffIt Expander work on the document, I was soon able to locate and extract the images that I wanted from the document! Very efficient.

Of course, this was on a Mac, but the same principle applies for a Windows or Linux (or even MS-DOS) PC. If you can find a utility that can read .zip files, it should have no problem extracting the constituent parts of an OpenXML document.

Where does SharePoint store its data?

One of the things that’s always confused me about SharePoint is exactly where the data is held and I asked the question on my MOSS 2007 enterprise search course this week – this is what I found.

For each shared service provider (SSP), SharePoint has three main locations for data storage:

  • The full text index catalogue is a flat file. Created by index server(s), it consists of a list of keywords and document identifiers, along with mappings as to which keywords exist in which documents. This high level of abstraction between keywords and links allows the full text index catalog to typically be around 5-12% of the size of the data being indexed.
  • The document identifiers in the full text index catalogue point to the document URLs, stored in the search database, a SQL Server database that is usually named sharedserviceprovidername_Search_DB.MDF. As its name suggests, this literally stores information relating to searchable content on a per-SSP basis, including URLs, access control lists (ACLs) and managed property information. On a WSS system this is named WSS_Search_computername.MDF.
  • The third location is the search configuration (or content) database, which contains configuration information relating to items such as crawl rules, content sources and the definition of managed properties. On a WSS system this is named WSS_Content.MDF.

In addition to the above, there are a number of other elements to the SharePoint solution:

  • The SharePoint Central Administration content database (itself implemented as a WSS web application), named SharePoint_AdminContent_guid.MDF.
  • SharePoint_Config_guid.MDF, a database containing SharePoint configuration information.
  • The individual web (.aspx) pages and configuration files, along with XSLT transformations and other supporting files (generally XML-based).

Finally, the actual content that is indexed by SharePoint remains where it always was – i.e. in the file shares, document libraries, web sites, business systems, etc. that SharePoint is being used to search across.

(This information relates to WSS v3 and MOSS 2007 – other SharePoint versions may differ.)

Designing a SharePoint server infrastructure

In yesterday’s post about warming up SharePoint, I hinted that there may be a few more SharePoint posts to follow, so here’s the first one – examining the various infrastructure considerations when designing a SharePoint server infrastructure.

These notes builds on the post I wrote a few months ago about planning and deploying MOSS 2007 and whilst there is not yet any prescriptive guidance from Microsoft, many of the SharePoint Portal Server 2003 topology restrictions with small, medium and large server farms are removed in MOSS, allowing customisation of the infrastructure to suit organisational requirements.

Looking at WSS first, the major limitations are: that it only supports indexing of a single site collection (although multiple content databases can be crawled – up to 100 per search server); that there is no ability to separate query and index server roles (indexing is automatic and search servers act as both query and index servers); and that a single web application can only access one search server. Consequently the options for scaling out are:

  1. Move content database(s) to a dedicated server.
  2. Provide a dedicated web front end server
  3. Provide additional front end web servers, load balanced.
  4. Provide additional search servers, noting the 1:1 relationship between web applications and serach servers.

Moving on to MOSS, there is more flexibility; however there is also more complexity due to the possibility of having to provide dedicated servers for particular application roles:

  • Whilst MOSS can run on a single server, this is not a recommended configuration and will probably only scale to a few 10s or maybe a 100 users.
  • By separating out the database, there will be less memory and disk contention and the solution should scale into the 100s of users. For high availability, the database may be clustered but this will not really aid performance as the cluster model is 2 node active-passive. Other high availability options include log shipping.
  • The next step is to separate some of the MOSS roles. Separating the web front end and query roles from the index server (there can only be one index server per shared service provider – generally one per farm, although there may be multiple SSPs for reasons such as security partitioning) should allow the solution to scale into 1000s of users and by providing additional web/query servers this may reach 10,000 collaboration and search users. There is some debate as to whether it’s better to provide a combined web front end and query server with a separate index server or a dedicated web front end server with a combined query and index server (as per WSS) but the former solution is generally recommended for a number of reasons:
    1. It aids manageability, allowing for the addition of further query servers at will (if the query and index server roles are combined, additional query servers cannot be provided).
    2. It reduces network traffic – if a search produces a large result set, this will be security-trimmed by the web front end server, resulting in needless data transfer; whereas if the web and query servers are combined then there is no data transfer.
    3. RAM contention between the web and query roles can be alleviated through the provision of additional memory.
  • Finally, separating the web front end and query roles (up to 32 of each per farm) should allow 10s of 1000s of users (details of Microsoft’s own implementation are available publically as an example of a large enterprise search deployment using SharePoint).

These are not the only possible configurations – the MOSS architecture is pretty flexible. It’s also worth noting that there is no need to load-balance query servers (the web servers will locate the appropriate query server) and there is no affinity between web and query servers (so if a query server goes offline then this should not cause a problem as long as other query servers exist within the infrastructure). The restriction on the number of index servers per SSP is a potential single point of failure; however there are ways of getting around this. One method is to configure multiple SSPs operating on the same data sources (one active and one additional), then to re-assign web applications in the event of the index server going offline; however this is less than ideal and it’s probably better to have a good disaster recovery strategy and to live with stale index data served from query servers whilst the index server is offline.

Other considerations include the provision of separate application servers for business data cataloguing (BDC) and Excel calculation services so as to minimise the impact on returning search results to end users. Another method of reducing the load on the primary (load balanced) web front end servers is to provide a dedicated web front end servers for use by the index server and, although this is an indexing bottleneck, it will ensure that indexing does not negatively impact performance for end users. It may also be worth considering the provision of a dedicated administration server for central administration (which will aid security and performance by not placing this on an end-user facing web server).

Moving on to the physical configuration of each server:

  • Web front end servers: Serving web pages is memory intensive as page output is cached until it is invalidated. Whilst there is some IO activity, this is rarely a bottleneck and if web front end servers are load-balanced then there is probably no need for a redundant disk configuration (at least not for performance reasons – it may still be desirable from an operational standpoint).
  • Application servers:
    • Query servers are memory bound and although an ideal configuration would load the entire index into RAM, a 1TB data source could be expected to result in a 120GB index file and that is a lot of memory to install in a server! (4GB is considered a practical minimum for a MOSS query server.) MOSS does employ an aggressive caching regime; however as the cache is invalidated by new data propagated from the index server or otherwise released to free up memory, there is a requirement for fast disk access (both read and write), so RAID 10 (also known as RAID 1+0 – mirrored and striped) is recommended.
    • Indexing is the most CPU-intensive task for a SharePoint server. Therefore, at index time, index servers will always be CPU bound (some application server roles may also be CPU-intensive, such as BDC and Excel calculation services, but indexing will always be more so at index time). Memory access can be controlled by configuring the number of parallel reads, but the disk subsystem will also be an important consideration and Microsoft recommends RAID 10 to provide the optimum disk performance. 4GB is considered the minimum amount of RAM that should be provided.
  • Database servers should follow the normal advice for SQL Server configuration

For all servers, 64-bit processors are recommended; however there are some limitations around the availability of 64-bit IFilters (and protocol handlers) so it may be necessary to run index servers using 32-bit software. Regardless of this, it is not recommended to mix 32- and 64-bit implementations within a single tier of the architecture.

In all scenarios, the main bottleneck will be the network – low latency links are required and so even though it may seem to make sense to place a web front end server close to the user population, this is not really desirable – HTTP is designed to work over WAN links but the web front end servers need to communicate with query servers using a combination of HTTP and RPC calls, and with the database server(s) using TCP sockets. As well as interrogating query servers for search results, web front end servers communicate directly with the database server (i.e. not with a query server) for access control list information (used for security trimming) and access to the property store.

One significant improvement for MOSS (relative to SharePoint Portal Server 2003) is the storage requirement. Whereas SPS 2003 required disk space equivalent to 50% of the corpus (the sum of all data being indexed) for indexing and 100% of the corpus for query – these data volumes are just not practical and MOSS 2007 reduces this to a full text catalogue index file size which is typically between 5 and 12% of the corpus size and a search database size equivalent to around 2KB per indexed document. Due to continuous index propagation (a new feature with MOSS that means the average time between indexing and search availability is between 3 and 27 seconds) and the merging of shadow index files, it is necessary to allow additional disk space for the index (2.5x times the maximum anticipated size of the index – so around 30% of the corpus volume but still a major improvement on SPS 2003). It’s also necessary to balance the freshness of the index with the time that it takes to crawl the corpus and so typically multiple crawl rules will be defined for different content sources. Estimates for corpus size should be based on:

  • Number of items
  • Storage used
  • Types of items
  • Security
  • Latency requirements
  • Connectivity
  • Estimated indexing window
  • Expected yearly growth

Regardless of the number of users, there is an practical limit of 50 million documents to be indexed, per server farm (except MOSS for Search standard edition, which has a hard-coded limit of 500,000 documents). The workaround for this is to provide multiple server farms and to federate; however there are some limitations of this approach if search results are ordered on relevance as the two sets of results cannot be reliably merged. The practical approach is to ensure that farms relate to logical business divisions (e.g. Americas, EMEA and Asia-Pacific) and to provide multiple columns (or multiple tabs) of search results with one for each farm. Note that SharePoint does allow SSPs to be shared between farms, so in this configuration the three separate farms could be used for indexing each regional corpus with a common SSP for global information such a user profiles. Each full MOSS farm can support up to 20 SSPs, although MOSS for Search is limited to a single SSP per farm.

Hopefully the information here is useful for anyone looking at implementing an enterprise search solution based on MOSS. The Microsoft SharePoint products and technologies team blog provides more information, as to various SharePoint experts around the ‘net. Information about MOSS 2007 is also available in the Office Server System technical library. The key messages are that:

  • Planning is important and a test/proof of concept environment can help with establishing a starting point topology, monitoring actual performance and capacity data to identify resource bottlenecks and then scaling up the available resources and scaling out server roles.
  • Post-deployment, search queries and results reporting can (and should) be used to identify areas that can benefit from further optimisation – going beyond the scope of this post (the infrastructure) and gaining an understanding of end-users’ search usage patterns (what were they searching for and was the search successful) with the aim of improving the overall search experience and improving productivity.

Credits

This post was based on information presented by Martin Harwar, a SharePoint expert at English Tiger (and formerly head of practice at CM group) who works closely with the product group in Redmond and recently published an article entitled “Find, don’t Search!” discussing how MOSS can be used to alleviate some of the issues associated with enterprise search.