BizTalk Server 2004 Overview

Posted on By Mark Wilson
This content is 21 years old. I don't routinely update old blog posts as they are only intended to represent a view at a particular point in time. Please be warned that the information here may be out of date.

Microsoft BizTalk Server has been around for a few years now and apart from some hands on labs at a Microsoft Tech Ed conference, I’ve not had much opportunity to get involved with it. As BizTalk Server 2004 is the product’s third incarnation and it seems to be gaining momentum in the marketplace, I attended a presentation given by Nick Page (a Technical Evangelist at Microsoft UK) and the following are my notes from the event.

Today’s IT environments are complex with multiple servers, operating systems and protocols, many disparate applications (ERP, HR, Finance) and many programming models (e.g. COM, J2EE, .NET, CICS). At the same time, organisations are demanding integration between internal systems, and integration with business partners.

Tightly-coupled systems impose many dependencies, requiring lots of commonality between systems (e.g. those of an organisation and its business partners), whereas loosely-coupled systems lend themselves to a service orientated architecture (SOA). In this context, a service is a reusable business process or business function, that can be accessed through public and private networks.

In the past, attempts to standardise integration technologies have been hampered by a lack of industry acceptance and adoption but today we have a ubiquitous network (the Internet), a ubiquitous data format (XML) and all major vendors support XML web services.

The web services dream is for each line of business (LOB) system to be connected to all the applications from which it needs to consume services.

Web services

This is all very well, but once more than one system is involved the mesh of connections becomes difficult to manage.

Web services chaos
The answer lies in the use of an integration server, and Microsoft’s enterprise application integration (EAI) product is BizTalk Server, although BizTalk offers more than just EAI.

Web services integration

BizTalk Server 2000 and 2002 separated messaging and orchestration. BizTalk Server 2004 re-integrates the two, using an architecture which is summarised in the following diagram.

Microsoft BizTalk Server

BizTalk Server includes an adapter framework to build adapters (which are actually Microsoft .NET components) and all BizTalk Server 2004 adapters are configured in the same manner (unlike previous versions). Examples of adapters may be:

  • A file system adapter, which looks for the presence of a new file in a particular location.A SQL adapter, which understands the database schema.
  • An HTTP adapter, for transport.
  • An API adapter (e.g. SAP), mapping XML messages to native application programming interfaces (APIs).

Out of the box, BizTalk Server includes EDI, flat file, FTP, HTTP, SOAP and SQL adapters but many more are available from ISVs.

As messages (with context) pass through the pipeline, components are used to translate/process the message resulting in and XML output. Sometimes, this processing takes place within the adapter – the decision where to put the processing depends on whether they will be reused or not (e.g. if there are many adapters working with the same transport, such as flat file .CSV, then this processing may be put in the pipeline). An example use for a component would be to validate messages against a schema. Standard components included with BizTalk include XML receive and pass-through receive.

With BizTalk Server 2004, all components act directly on the messages, whereas previous BizTalk Server versions used the Microsoft Message Queue (MSMQ).

By promoting part of a message into context (i.e. putting it into the property schema), then messages can be routed on content, avoiding the requirement to load the whole message at each stage in the pipeline.

The business rules portion of the process is where the orchestration takes place, defining processes that interact with systems and apply business logic to the XML messages.

The message box is a SQL Server database, used to store the XML messages, which have rich properties, allowing them to be linked to services through a publish/subscribe model. Unlike in previous versions of BizTalk Server, from 2004, all messages pass via the SQL Server and a single message passed into the message box database work queue could result in multiple outgoing messages (each one a copy of the original), based on the subscriptions of other services.
The configuration database contains details of all the ports in the integration, whilst the tracking database tracks the progress of the message.

BizTalk Server 2004 ports are intended to be stateless so as to allow for scaling in/out (e.g. using network load balancing), with SQL Server stored procedures carrying out the routing so that many processing servers may be provided for one common database; however there are some exceptions – persisted (long running) transactions, correlation messages and session-based protocols all need to be routed to the correct server. Storage can also be scaled out using multiple SQL Servers but if clustering is used, then there will be a 30-60 second delay during a cluster failover.

Just as pipeline components are used to parse and serialise messages, message security is provided using components in the pipeline, for example decrypting/encrypting/signing messages or resolving parties (i.e. mapping to a certificate). It is also possible to segment security with multiple logical hosts using different service accounts. This is particularly useful if multiple suppliers are involved in a process, where BizTalk will reject incoming messages with no party ID and the message box will not allow messages from untrusted hosts. The incoming and outgoing ports may use separate service accounts, as may the orchestration element.

BizTalk Server is designed for throughput (scalability) and not latency (sub-second point-to-point responses), so orchestration processes are not left running and waiting for a response indefinitely as this will consume system resources unnecessarily. Instead, processes are dehydrated to the database and correlation (a collection of message properties) is used to about matching incoming messages to orchestration processes and rehydrate processes as necessary.

For developing solutions, BizTalk Server 2004 includes the BizTalk project system (hosted within Microsoft Visual Studio), to provide an environment for designing, organising, and building the various elements of BizTalk applications.

When a BizTalk project is created, it generally includes one or more of the following file types which play specific roles in the creation of the business solution, and the BizTalk project system provides a corresponding graphical design tool for each of them.

  • Orchestrations model the business process. Orchestration Designer simplifies the process of creating orchestrations (models of business processes that are compiled into executable code).
  • Schemas describe the data that is sent and received. BizTalk Editor simplifies the process of defining schemas. Schemas are used to describe the format of data that is processed within organisations and between trading partners.
  • Maps transform data from one format to another. BizTalk Mapper presents source schemas and destination schemas side-by-side and enables the definition of transformations between data elements of different messages.
  • Pipelines perform a variety of operations to prepare incoming or outgoing messages for further processing. Pipeline Designer allows the preparation of incoming and outgoing messages for further processing by implementing such operations as encryption and decryption, compression, reformatting, and validation.

Other BizTalk terms include:

  • Functoids – used to map two schemas and transform data (e.g. cumulative sum, database lookup, or run a script, calling out to external or in-line code). The transformation can occur wherever the messages are in XML format (i.e. after the pipeline or during orchestration).
  • Vocabulary – mapping English terms/descriptions to elements within a message.Policy – a set of rules used to work on vocabulary or message elements. These rules are deployed to BizTalk servers but by extracting the rules from the business process, the rules are not hard coded and so allow for modification in line with business process changes.
  • Envelopes – used to wrap the core data when receiving (disassembling) or sending (assembling) messages.
  • Long-running transactions – when business processes last longer than a few seconds (e.g. send a request for assessment and expect a response after a few days). The converse is a short-running transaction, which uses a locking mechanism but this is too resource-intensive to scale so BizTalk Server 2004 has the concept of flexible delay.
  • Compensation – if a process aborts and affects the results of a previously completed transaction, it cannot be undone. Compensation processes are used for actions which cannot be undone.
  • Exception block – the method by which BizTalk Server 2004 deals with exceptions when external calls go wrong.

Finally, some of the other features provided by BizTalk Server 2004 include:

  • Single sign on (SSO) capabilities – mapping user credentials from Windows to non-windows systems allowing services to authenticate once and then securely access target systems. An encrypted secret store contains the user mappings. BizTalk Server’s SSO integrates with Windows SharePoint Services (WSS) to provide SSO for a portal and has a web services front end for any client to any platform.
  • Business activity monitoring (BAM) – allowing users to query an OLAP cube from within Microsoft Excel and ask near real-time questions such as “how long is production taking right now?” or aggregation questions such as “how much money did we make last month?”. Because the OLAP cube is extracted from the database, it is not quite real time. This can use data from documents or process and complements the existing SQL Server business intelligence (BI) offerings.
  • Health and activity monitoring – this includes the Orchestration Debugger, which allows a message to be replayed through a process to follow calls into other Orchestrations, view exceptions thrown and examine any compensation code executed; or alternatively to attach to live orchestrations, setting breakpoints, watching variables and retrieving message contents before resuming processing until the next breakpoint.
  • Human workflow services (HWS) – business analysts and developers can use standard orchestration processes for interaction, or HWS for more dynamic requirements. Each orchestration becomes a step in the workflow with permissions; however HWS is all about background processing and has no user interface, so third party products are needed to expand on this.

10,000 feet view of Microsoft ISA Server 2000

Posted on By Mark Wilson
This content is 21 years old. I don't routinely update old blog posts as they are only intended to represent a view at a particular point in time. Please be warned that the information here may be out of date.

Microsoft Internet Security and Acceleration (ISA) Server 2000 is Microsoft’s firewall and proxy server product, replacing Microsoft Proxy Server.

ISA Server 2000 operates in one of three modes:

  • Firewall – security server.
  • Caching (i.e. proxy) – acceleration server.
  • Integrated – firewall and caching.

As a firewall, ISA Server’s filtering capabilities include:

  • Packet filtering (layer 3) capabilities – filtering based on the source and destination IP addresses along with their respective port numbers.
  • Circuit (TCP) and stateful inspection (UDP) filtering – filtering bases on a sequence of packets (e.g. to guard against denial of service attacks).
  • Application filtering (layer 7) – actually inspecting the data segment of the packet.

(layer numbers refer to the open system interconnection 7 layer network model).

The product can be used as an internal or external firewall and ISA Server 2000 achieved certification for Common Criteria Evaluation Assurance Level 2 (EAL 2) in September 2003.

ISA Server’s basic caching (proxy) process is as follows:

ISA Server 2000 - caching

  1. Client requests access to an external website. Because the DNS suffix address is not local, it knows to contact the proxy server.
  2. ISA Server receives the request and checks its cache, which by default is split between 50% of the ISA server’s available memory (fast) and disk (slower).
  3. If there is no match in the cache, then space is reserved in cache and the request is forwarded to the Internet, using network address translation (NAT).
  4. The external web server replies with the requested page.
  5. The reply is cached with a time to live (TTL) (default 24 hours) and is forwarded to the requesting client.

Other caching features include:

  • Active caching – as the TTL for a cached resource expires, ISA server proactively fetches the latest copy of the resource from the Internet, storing it in the cache and accelerating future requests by keeping the latest versions of the most popular sites in the cache.
  • Scheduled download – caching defined URLs on a schedule, e.g. caching a partner’s website each day for local access. This is most useful in a business to business (B2B) extranet scenario, where the content expiry can be predicted.

ISA server also supports VPN access (available in Windows 2000, XP and Windows Server 2003, but enhanced with ISA Server) and reverse publishing (allowing access to internal resources from the Internet). Reverse publishing using ISA Server is more secure than simply hosting a web server in a demilitarised zone (DMZ) as there is a firewall with NAT between the Internet and the DMZ, as well as between the DMZ and any internal resources.

Standard 3-pronged firewall with web server in DMZ:

3 pronged firewall

Reverse publishing using ISA Server:

ISA Server 2000 - reverse publishing

The configuration of ISA Server 2000 is controlled using policy elements, which are bundled together to create rules.

ISA Server 2000 Enterprise Edition allows the formulation of arrays of ISA servers, providing greater scalability.

Links

Microsoft ISA Server
Microsoft ISA Server firewall and cache resource site

Migrating from Exchange Server 5.5 to Exchange Server 2003

Posted on By Mark Wilson
This content is 21 years old. I don't routinely update old blog posts as they are only intended to represent a view at a particular point in time. Please be warned that the information here may be out of date.

With Microsoft Exchange Server 2003, Microsoft have made Exchange installation simpler – the Exchange Server deployment tools and documentation (ExDeploy) lead an administrator through the entire Exchange Server installation or upgrade process and it is recommended that Exchange Server 2003 Setup is run using ExDeploy. Specific tools and utilities can be used to verify that the organization is ready for the Exchange Server 2003 installation.

For a variety of reasons, the majority of organisations moving to Active Directory perform a migration rather than an upgrade. The main reason for this is that the legacy Windows NT domain structure is generally over-complicated, often having grown organically with various resource domains created to support applications, and so may be poorly matched to the current organisational structure. Implementing a totally new directory is often the simplest method of re-engineering this, allowing for domain consolidation and improved flexibility post-migration.

Messaging connectivity is fairly straightforward to provide – that is at the core of the Exchange Server product – the main issues for Exchange Server are around directory management.

Because Exchange Server 4.0-5.5 use their own directory service and Exchange Server 2000 and 2003 use Active Directory, in a mixed environment it is necessary to synchronise directories using the Active Directory Connector (ADC). Note that there are two commonly available versions of this tool – the Windows 2000 version is not suitable for Exchange and the Exchange version should always be used.

The Exchange Server directory allows one user to be linked with several mailboxes (e.g. a primary mailbox and a resource mailbox) but Active Directory has a 1:1 relationship between a user account and a mailbox; however, ADC can create a disabled user account with an associated mailbox for resource accounts (permissions on the mailbox can then be delegated to one or more users).

The NTDSNoMatch utility can assist administrators by checking for mailboxes with a duplicate primary Windows NT account and determining if the mailbox is the primary mailbox or a resource mailbox. Following this, it creates a comma-separated value (.CSV) file that can be imported into the Exchange 5.5 directory to automatically set Custom Attribute 10 to NTDSNoMatch for the resource mailboxes. The ADC uses this attribute to match a mailbox that does not have NTDSNoMatch set to the correct user account.

The ADC uses a series of connection agreements (CAs), which are set as primary (i.e. synchronise and create objects as necessary) or non-primary (synchronise only). One- or two-way CAs can be configured and if required, a CA could be primary in one direction, and non-primary in the other. This primary and non-primary arrangement prevents duplicate entries in the global address list (GAL) from being created as a result of various CAs synchronising with multiple sites.

In mixed mode, recent versions of Exchange Server provide a service called the site replication service (SRS), which makes an Exchange 2000/2003 routing group appear as a site to the Exchange 5.5 infrastructure (cf. native mode, where Exchange Server 5.5 interoperability is not supported).

The SRS acts as an endpoint for a configuration CA, created in the ADC by Exchange Setup. This allows SRS to funnel organisational changes made in Active Directory into the legacy Exchange directory service, where they propagate to the legacy servers via standard directory service replication.

Recipient and public folder CAs are created by an administrator. These should be configured to point at the SRS rather than a legacy Exchange server so that legacy servers can be decommissioned without losing synchronisation with Active Directory. When all legacy servers have been decommissioned, the SRS is no longer required. Note that even when hosting the SRS, an Exchange Server 2003 server still read directly from Active Directory (using DSAccess) and the SRS is only for the benefit of Exchange Server 4.0-5.5 servers.

If an organisation creates an Active Directory to support Exchange Server 2003, and completes the Exchange migration before all the NT domains have been migrated to Active Directory, duplicate accounts will be created. The ADClean tool can be used to merge duplicate accounts and is installed with Exchange Server 2000 and 2003.

The whole migration process from Exchange Server 5.5 to 2003 is summarised as follows:

Exchange 5.5 to 2003 migration

  1. Install Active Directory on a new Windows 2000 or Windows Server 2003 server
  2. Migrate NT objects to Active Directory using the Active Directory Migration Tool (ADMT) – this will require a trust to be in place between the legacy and new domains.
  3. ADMT manages SID history to ensure that new accounts in Active Directory still have access to resources in the NT domain (including permissions over their Exchange Server 5.5 mailboxes.
  4. Run Exchange Server setup once with the /forestprep switch to prepare the Active Directory Schema for Exchange and again with the /domainprep switch for each domain in the forest which will host Exchange Servers (ADC installation in 5 would make some changes, but not all).
  5. Install the ADC on a member server within the Active Directory forest. Create recipient CAs and use the NTDSNoMatch utility to check for mailboxes with a duplicate primary Windows NT account.
  6. Install a new Exchange Server 2003 server, ideally into the Exchange Server 2003 administrative group which corresponds to the existing Exchange server 5.5 site (i.e. same organisation). Alternatively, a new organization can be created.
  7. Use the Move Mailbox Wizard to move mailboxes from legacy Exchange servers to the new server. This is multi-threaded in Exchange Server 2003 and so much faster than in Exchange Server 2000. Outlook clients will automatically be updated via the directory referral method and so a client visit is not required. If a new Exchange organization was created in 6, it will be necessary to use the ExMerge tool to migrate data and to reconfigure each Outlook client with a new profile. The public folder migration tool can be used to migrate system and public folders.
  8. Once all of the data is migrated, the legacy Exchange Server 5.5 servers may be decommissioned and Exchange Server 2003 switched from mixed to native mode (note that this is entirely separate from the mixed and native modes for Active Directory).
  9. Optionally, ADSI Edit can be used to restructure administrative groups (unsupported by Microsoft).
  10. Finally, the ADC may be decommissioned and the SRS disabled.