One of the things that’s always confused me about SharePoint is exactly where the data is held and I asked the question on my MOSS 2007 enterprise search course this week – this is what I found.

For each shared service provider (SSP), SharePoint has three main locations for data storage:

• The full text index catalogue is a flat file. Created by index server(s), it consists of a list of keywords and document identifiers, along with mappings as to which keywords exist in which documents. This high level of abstraction between keywords and links allows the full text index catalog to typically be around 5-12% of the size of the data being indexed.
• The document identifiers in the full text index catalogue point to the document URLs, stored in the search database, a SQL Server database that is usually named sharedserviceprovidername_Search_DB.MDF. As its name suggests, this literally stores information relating to searchable content on a per-SSP basis, including URLs, access control lists (ACLs) and managed property information. On a WSS system this is named WSS_Search_computername.MDF.
• The third location is the search configuration (or content) database, which contains configuration information relating to items such as crawl rules, content sources and the definition of managed properties. On a WSS system this is named WSS_Content.MDF.

In addition to the above, there are a number of other elements to the SharePoint solution:

• The SharePoint Central Administration content database (itself implemented as a WSS web application), named SharePoint_AdminContent_guid.MDF.
• SharePoint_Config_guid.MDF, a database containing SharePoint configuration information.
• The individual web (.aspx) pages and configuration files, along with XSLT transformations and other supporting files (generally XML-based).

Finally, the actual content that is indexed by SharePoint remains where it always was – i.e. in the file shares, document libraries, web sites, business systems, etc. that SharePoint is being used to search across.

(This information relates to WSS v3 and MOSS 2007 – other SharePoint versions may differ.)

In yesterday’s post about warming up SharePoint, I hinted that there may be a few more SharePoint posts to follow, so here’s the first one – examining the various infrastructure considerations when designing a SharePoint server infrastructure.

These notes builds on the post I wrote a few months ago about planning and deploying MOSS 2007 and whilst there is not yet any prescriptive guidance from Microsoft, many of the SharePoint Portal Server 2003 topology restrictions with small, medium and large server farms are removed in MOSS, allowing customisation of the infrastructure to suit organisational requirements.

Looking at WSS first, the major limitations are: that it only supports indexing of a single site collection (although multiple content databases can be crawled – up to 100 per search server); that there is no ability to separate query and index server roles (indexing is automatic and search servers act as both query and index servers); and that a single web application can only access one search server. Consequently the options for scaling out are:

1. Move content database(s) to a dedicated server.
2. Provide a dedicated web front end server
3. Provide additional front end web servers, load balanced.
4. Provide additional search servers, noting the 1:1 relationship between web applications and serach servers.

Moving on to MOSS, there is more flexibility; however there is also more complexity due to the possibility of having to provide dedicated servers for particular application roles:

• Whilst MOSS can run on a single server, this is not a recommended configuration and will probably only scale to a few 10s or maybe a 100 users.
• By separating out the database, there will be less memory and disk contention and the solution should scale into the 100s of users. For high availability, the database may be clustered but this will not really aid performance as the cluster model is 2 node active-passive. Other high availability options include log shipping.
• The next step is to separate some of the MOSS roles. Separating the web front end and query roles from the index server (there can only be one index server per shared service provider – generally one per farm, although there may be multiple SSPs for reasons such as security partitioning) should allow the solution to scale into 1000s of users and by providing additional web/query servers this may reach 10,000 collaboration and search users. There is some debate as to whether it’s better to provide a combined web front end and query server with a separate index server or a dedicated web front end server with a combined query and index server (as per WSS) but the former solution is generally recommended for a number of reasons:
1. It aids manageability, allowing for the addition of further query servers at will (if the query and index server roles are combined, additional query servers cannot be provided).
2. It reduces network traffic – if a search produces a large result set, this will be security-trimmed by the web front end server, resulting in needless data transfer; whereas if the web and query servers are combined then there is no data transfer.
3. RAM contention between the web and query roles can be alleviated through the provision of additional memory.
• Finally, separating the web front end and query roles (up to 32 of each per farm) should allow 10s of 1000s of users (details of Microsoft’s own implementation are available publically as an example of a large enterprise search deployment using SharePoint).

These are not the only possible configurations – the MOSS architecture is pretty flexible. It’s also worth noting that there is no need to load-balance query servers (the web servers will locate the appropriate query server) and there is no affinity between web and query servers (so if a query server goes offline then this should not cause a problem as long as other query servers exist within the infrastructure). The restriction on the number of index servers per SSP is a potential single point of failure; however there are ways of getting around this. One method is to configure multiple SSPs operating on the same data sources (one active and one additional), then to re-assign web applications in the event of the index server going offline; however this is less than ideal and it’s probably better to have a good disaster recovery strategy and to live with stale index data served from query servers whilst the index server is offline.

Other considerations include the provision of separate application servers for business data cataloguing (BDC) and Excel calculation services so as to minimise the impact on returning search results to end users. Another method of reducing the load on the primary (load balanced) web front end servers is to provide a dedicated web front end servers for use by the index server and, although this is an indexing bottleneck, it will ensure that indexing does not negatively impact performance for end users. It may also be worth considering the provision of a dedicated administration server for central administration (which will aid security and performance by not placing this on an end-user facing web server).

Moving on to the physical configuration of each server:

• Web front end servers: Serving web pages is memory intensive as page output is cached until it is invalidated. Whilst there is some IO activity, this is rarely a bottleneck and if web front end servers are load-balanced then there is probably no need for a redundant disk configuration (at least not for performance reasons – it may still be desirable from an operational standpoint).
• Application servers:
• Query servers are memory bound and although an ideal configuration would load the entire index into RAM, a 1TB data source could be expected to result in a 120GB index file and that is a lot of memory to install in a server! (4GB is considered a practical minimum for a MOSS query server.) MOSS does employ an aggressive caching regime; however as the cache is invalidated by new data propagated from the index server or otherwise released to free up memory, there is a requirement for fast disk access (both read and write), so RAID 10 (also known as RAID 1+0 – mirrored and striped) is recommended.
• Indexing is the most CPU-intensive task for a SharePoint server. Therefore, at index time, index servers will always be CPU bound (some application server roles may also be CPU-intensive, such as BDC and Excel calculation services, but indexing will always be more so at index time). Memory access can be controlled by configuring the number of parallel reads, but the disk subsystem will also be an important consideration and Microsoft recommends RAID 10 to provide the optimum disk performance. 4GB is considered the minimum amount of RAM that should be provided.
• Database servers should follow the normal advice for SQL Server configuration

For all servers, 64-bit processors are recommended; however there are some limitations around the availability of 64-bit IFilters (and protocol handlers) so it may be necessary to run index servers using 32-bit software. Regardless of this, it is not recommended to mix 32- and 64-bit implementations within a single tier of the architecture.

In all scenarios, the main bottleneck will be the network – low latency links are required and so even though it may seem to make sense to place a web front end server close to the user population, this is not really desirable – HTTP is designed to work over WAN links but the web front end servers need to communicate with query servers using a combination of HTTP and RPC calls, and with the database server(s) using TCP sockets. As well as interrogating query servers for search results, web front end servers communicate directly with the database server (i.e. not with a query server) for access control list information (used for security trimming) and access to the property store.

One significant improvement for MOSS (relative to SharePoint Portal Server 2003) is the storage requirement. Whereas SPS 2003 required disk space equivalent to 50% of the corpus (the sum of all data being indexed) for indexing and 100% of the corpus for query – these data volumes are just not practical and MOSS 2007 reduces this to a full text catalogue index file size which is typically between 5 and 12% of the corpus size and a search database size equivalent to around 2KB per indexed document. Due to continuous index propagation (a new feature with MOSS that means the average time between indexing and search availability is between 3 and 27 seconds) and the merging of shadow index files, it is necessary to allow additional disk space for the index (2.5x times the maximum anticipated size of the index – so around 30% of the corpus volume but still a major improvement on SPS 2003). It’s also necessary to balance the freshness of the index with the time that it takes to crawl the corpus and so typically multiple crawl rules will be defined for different content sources. Estimates for corpus size should be based on:

• Number of items
• Storage used
• Types of items
• Security
• Latency requirements
• Connectivity
• Estimated indexing window
• Expected yearly growth

Regardless of the number of users, there is an practical limit of 50 million documents to be indexed, per server farm (except MOSS for Search standard edition, which has a hard-coded limit of 500,000 documents). The workaround for this is to provide multiple server farms and to federate; however there are some limitations of this approach if search results are ordered on relevance as the two sets of results cannot be reliably merged. The practical approach is to ensure that farms relate to logical business divisions (e.g. Americas, EMEA and Asia-Pacific) and to provide multiple columns (or multiple tabs) of search results with one for each farm. Note that SharePoint does allow SSPs to be shared between farms, so in this configuration the three separate farms could be used for indexing each regional corpus with a common SSP for global information such a user profiles. Each full MOSS farm can support up to 20 SSPs, although MOSS for Search is limited to a single SSP per farm.

Hopefully the information here is useful for anyone looking at implementing an enterprise search solution based on MOSS. The Microsoft SharePoint products and technologies team blog provides more information, as to various SharePoint experts around the ‘net. Information about MOSS 2007 is also available in the Office Server System technical library. The key messages are that:

• Planning is important and a test/proof of concept environment can help with establishing a starting point topology, monitoring actual performance and capacity data to identify resource bottlenecks and then scaling up the available resources and scaling out server roles.
• Post-deployment, search queries and results reporting can (and should) be used to identify areas that can benefit from further optimisation – going beyond the scope of this post (the infrastructure) and gaining an understanding of end-users’ search usage patterns (what were they searching for and was the search successful) with the aim of improving the overall search experience and improving productivity.

Credits

This post was based on information presented by Martin Harwar, a SharePoint expert at English Tiger (and formerly head of practice at CM group) who works closely with the product group in Redmond and recently published an article entitled “Find, don’t Search!” discussing how MOSS can be used to alleviate some of the issues associated with enterprise search.