Looping on the authentication prompt for the ECP virtual directory in Exchange 2013

I had an “interesting” problem on an Exchange server last week… I could access the Exchange Control Panel using https://localhost/ecp but it I tried to use its externally-accessible namespace (which I needed to in order to run the Exchange Hybrid Configuration Wizard), it kept looping on the authentication prompt. The password was correct but it just came straight back to a prompt again…

It turned out to be an issue with HTTP redirection in IIS and, following the MSExchangeGuru post on HTTP redirection issues with the ECP virtual directory I set up HTTP redirection on the Default Website but turned it off for OWA, ECP and the various other virtual directories. I also checked that SSL was not required for the PowerShell virtual directory.

Bipin Giri’s post on configuring URL redirection in Exchange 2013 suggests I need to repeat the same steps for the Exchange Backend Server website too but I didn’t find that necessary (and I’m sure I’ve also read to leave that alone!).

There’s a useful table in the MSExchangeGuru post that summarises the need for an HTTP redirect and whether SSL should be enabled for the website and for each virtual directory but it’s also worth knowing that Morten Nielsen has listed the default virtual directory settings for Exchange 2013 in great detail.

Manually removing servers from an Exchange organization

I’m starting this blog post with a caveat: the process I’m going to describe here is not a good idea, goes against the advice of my colleagues (who have battle scars from when it’s been attempted in a live environment and not gone so well) and is certainly not recommended. In addition, I can’t be held responsible for any unintended consequences of following these steps.

Notwithstanding the above, I found myself trying to configure the Exchange Hybrid Configuration Wizard (HCW) in a customer environment, where the wizard failed because it was looking for servers that don’t exist any more.

I had two choices:

  1. Recover the missing Exchange servers with setup.exe /m:RecoverServer, then uninstall Exchange gracefully (for 12 servers!).
  2. Manually remove the servers using ADSI Edit.

I explained the situation to my customer, who discussed it with his Exchange expert, and they directed me to go for option 2 – this was a test environment, not production, and they were prepared to accept the risk.

Fearing the worst, I made a backup of Active Directory, just in case. This involved:

  1. Installing the Windows Server Backup Command Line Tools feature on the domain controller.
  2. Running wbadmin start systemstatebackup -backuptarget:driveletter:
  3. Sitting back and waiting for the process to complete.

With a backup completed, I could then:

  1. Run ADSI Edit.
  2. Open the configuration naming context.
  3. Navigate to CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=organizationname,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=domainname,DC=tld
  4. Delete the records for the servers that no longer exist.
  5. Restart each of the remaining Exchange servers in the organisation in turn.
  6. Check the server list in ECP.

(Incidentally, FYDIBOHF23SPDLT is “Caesar’s Cipher” for EXCHANGE12ROCKS).

Murat Yildirimoglu’s Windows IT Pro article entitled “How to Uninstall a Stubborn Exchange Server” goes into more detail, including completely removing an Exchange organisation from Active Directory, should that be required (Christopher Dargel covers that too).

The process seemed to work but the danger of manually removing servers from an Exchange organization like this is the potential side effects of “unknown unknowns” (which you can be sure won’t surface immediately). It did let me progress to the next stage of the HCW though. More on that in a future blog post…

“Delivery has failed to these recipients or groups” when running Exchange in an Azure VM

Exchange didn’t used to be supported in Azure. It is now, subject to specific requirements; however there’s a big difference between “supported” and “works” and it was always theoretically possible.

My current customer has a test environment running on a number of Azure VMs. All was working well, until I started to test mail flow out of the organisation. My mailboxes (work and personal) are both on Office 365 and the reply came back as:

Delivery has failed to these recipients or groups:

Mark Wilson
Your message wasn’t delivered due to a permission or security issue. It may have been rejected by a moderator, the address may only accept email from certain senders, or another restriction may be preventing delivery.

The following organization rejected your message: DB3FFO11FD037.mail.protection.outlook.com.

Basically, Exchange Online Protection was bouncing the mail. The error continued with diagnostic information for administrators and I could see that the message was leaving the organisation, then returning to the Exchange Edge server.

I could also see in one of the messages that it said:

“Remote Server returned ‘<DB3FFO11FD037.mail.protection.outlook.com #5.7.1 smtp;550 5.7.1 Service unavailable; Client host [] blocked using FBLW15; To request removal from this list please forward this message to delist@messaging.microsoft.com>'”

So I emailed and asked to be removed, quickly receiving a very polite but understandably automated and non-committal response:

“Hello ,
Thank you for your delisting request SRX1234567890ID. Your ticket was received on (Aug 28 2015 12:26 AM UTC) and will be responded to within 24 hours.

Our team will investigate the address that you have requested to be removed from our blocklist. If for any reason we are not able to remove your address, one of our technical support representatives will respond to you with additional information.

Technical Support”

Within 24 hours, Microsoft had responded to say that we had been delisted from their blocklists (presumably they checked that the IP address was one of theirs – which was also one reason why we couldn’t add a reverse DNS record, as one might expect with an SMTP server) and the mail had started to flow:

“Hello ,
Thank you for contacting Microsoft Online Services Technical Support. This email is in reference to ticket number 1234567890, which was opened in regards to your delisting request for

The IP address you submitted has been reviewed and removed from our block lists. Please note that there may be a 1-2 hour delay before this change propagates through our entire system.

We apologize for any inconvenience this may have caused you. As long as our spam filtering systems do not mark a majority of email from the IP address as spam-like, your messages will be allowed to flow as normal through our network. However, should we detect an increase in spam-like activity, the IP address may be re-added to our block list.

Should you have any further questions or concerns, please feel free to respond to this email.

Thank you again for contacting Microsoft Online Services technical support and giving us the opportunity to serve you.”

I’m glad the experience was with a customer’s test environment, and not live email flow, but worth remembering for the future…

[Ticket numbers and IP addresses in this scenario have been changed]

Export transport rules from Exchange or Exchange Online

After all my work last week creating Exchange transport rules for profanity, audio/video attachments, message encryption and more, I wanted to export the rules just in case they needed to be re-established.  Thanks to TechNet, I found the required PowerShell to export transport rules from Exchange or Exchange Online, which is:

$file = Export-TransportRuleCollection
Set-Content -Path "ExchangeOnlineRules.xml" -Value $file.FileData -Encoding Byte

The resulting XML includes the New-TransportRule commands to re-create the rules if required (or the Import-TransportRuleCollection cmdlet can be used instead).

Exchange transport rules to detect audio/video attachments

After my fun creating a profanity filter for Exchange Online earlier this week, my attention turned to some of the other rules that my customer needed re-creating in preparation for the move to Office 365. Most were fairly straightforward blocks on certain domains/addresses or using the normal templates to prevent financial data from being leaked, etc. but then I found another one that I’d expect to be in included in Exchange Online Protection, but isn’t: copying any audio/video files emailed from within the organisation to a defined mailbox.

The rule itself is quite simple, but the number of file extensions involved meant I actually needed 4 rules to avoid this error message:

The rule can’t be created because it is too large. It has 9028 characters, and the maximum number of characters is 8192.

Reduce the size, either by removing content, such as words or regular expressions, from the rule; or by removing conditions, exceptions, or actions from the rule.

After chunking the attachment extensions, the final Exchange transport rules used to to detect audio/video attachments were:

New-TransportRule "Notify Security if outbound email contains audio (1)" -AttachmentExtensionMatchesWords 'afc','vag','copy','vdj','sng','aob','act','ang','nra','hsb','rfl','sma','smp','syh','vyf','acm','at3','vmd','aimppl','nvf','saf','xfs','ins','alac','mod','omf','sfk','als','caf','gp5','wav','mp3','pla','abm','aup','wma','acd-zip','amxd','dmsa','dmse','emp','logicx','m4r','midi','ptx','rns','rx2','slp','trak','5xb','a2b','a2i','agr','akp','asd','bnk','bun','bww','csh','dfc','dsm','dtm','fev','flp','frg','g726','gsm','h5b','h5s','isma','krz','ksf','mbr','mmlp','mpga','mtp','musx','nkc','nkm','omg','pkf','r1m','rex','rip','rol','sbi','sfpack','smf','sseq','svd','syw','tg','u','uax','vpl','zvd','0.669','eop','mus','sf2','mid','ksd','aif','flp','oga','pcg','sty','dig','mscz','ogg','m3u','flac','sib','aiff','syx','zab','dss','gpk','xspf','mui','vlc','nbs','5xe','logic','minigsf','sd','sdat','wve','ins','cda','ram','aac','iff','nki','wave','wpk','dff','amr','3ga','dcf','aud','cwt','dls','ds2','flm','nsa','it','pcm','pho','q1','sns','sph','xwb','dsp','sam','u8','wand','ym','ac3','oma','sds','stm','acd','dsf','cpr','xa','m3u8','ftm','4mp','apl','cwp','cws','gpbank','gsflib','med','mo3','mx5','ply','qcp','rmj','w64','ahx','au','b4s','h0','h3e','hbb','hbs','ins','kit','kmp','ksc','mdl','mu3','phy','q2','sbg','sfap0','smp','toc','vgz','vmf','zpa','2sf','m4a','ds','nsf','sesx','ape','fls','mus','emx','pcast','dtshd','mmm','peak','vox','bmml','mscx','xmf','rtm','pls','sfl','xm','avastsounds','snd','voc','wax','wpp','ra','cdr','seq','gpx','au','aa','m4b','odm','mpa','amz','5xs','a2m','abc','acd-bak','adts','agm','aifc','alc','amf','band','bap','bdd','bidule','bwf','caff','cdda','cdlx','cdo','cel','cgrp','cidb','ckb','conform','cpt','cwb','dct','dewf','df2','dig','dm','dmf','dra','drg','dwd','efk','efq','efs','efv','emd','esps','f2r','f32','f3r','f4a','f64','fdp','fsb','fsc','fsm','ftm','ftmx','fzf','fzv','g721','gig','groove','gsf','h4b','hbe','igp','iti','koz','koz','kt3','la','lso','lwv','m4p','ma1','mdc','mgv','miniusf','mka','mmp','mmpz','mpc','mte','mti','mtm','mus','mux','narrative','nkb','nks','nkx','nml','note','nrt','nst','ntn','nwc','obw','okt','omx','ovw','pandora','pca','pek','pna','psm','ptm','pts','rax','rgrp','rmi','rmx','rng','rso','rti','s3i','sc2','scs11','sd2','sfz','sgp','smpx','sou','sppack','sprg','stap','sty','sxt','syn','td0','tta','txw','ult','uni','usf','usflib','ust','uw','uwf','vap','vc3','vmo','voxal','vpm','vpw','vrf','vsq','wfb','wfm','wfp','wow','wproj','wrk','wus','wut','wv','wvc','wwu','xmu','xrns','yookoo','adv','cmf','dmc','gmc','mp_','ppcx','sbk','sid','sng','vgm','6cm','8med','a52','al','d01','evr','fda' -GenerateIncidentReport security
New-TransportRule "Notify Security if outbound email contains audio (2)" -AttachmentExtensionMatchesWords 'gsm','kin','mini2sf','pd','prg','record','rmf','tmc','tun','wyz','xp','xt','kar','vb','wem','adg','dts','kfn','pk','mxl','mtf','ncw','dw','igr','vce','ddt','k25','sf','dvf','aa3','adt','fpa','h5e''mpdp','ove','rbs','sd','slx','stx','swa','vsqx','w01','zpl','mmp','opus','ppc','rsf','sdt','wav','xa','xpf','xsb','brstm','tak','ptf','efa','g723','mmf','s3m','sap','vqf','2sflib','avr','ear','mp1','dcm','ay','zvr','pat','ams','cts','gbs','ics','k26','mp2','mts','myr','ots','psf','rsn','ses','shn','snd','a2p','a2t','a2w','ab','acp','ais','alaw','all','apf','aria','ariax','axa','bwg','c01','ckf','djr','efe','emy','erb','far','fti','gbproj','gym','h3b','h4e','hdp','iaa','imp','itls','its','jam','jam','kpl','kt2','l','lof','lqt','m','m1a','m2','minipsf','minipsf2','mogg','mpu','mt2','mux','mx3','mx4','mx5template','npl','ofr','ovw','pbf','pjunoxl','plst','pno','prg','psf1','psf2','psy','ptcop','pvc','rad','raw','rbs','rcy','rmm','rta','rts','rvx','s3z','sd2f','spx','sseq','ssnd','svq','svx','thx','tsp','ub','ulaw','v2m','vmf','vtx','wtpl','wtpt','xbmml','xmi','xmz','xsp','zgr','atrac','box','fzb','hmi','imf','sdx','aax','sb','cfa','mxmf','pac','d00','8svx','ams','wfd','msv','xi','nmsv','ase','awb','expressionmap','hma','hps','mlp','mzp','sfs','snd','tak','8cm','gm','lvp','bcs','bonk','cfxr','dwa','fff','gio','gio','gro','jo','jo-7z','ksm','ktp','minincsf','mt9','musa','muz','mwand','mws','nap','orc','pmpl','r','sdii','seg','snsf','sth','sti','stw','sw','swav','syn','tfmx','tm2','tm8','ulw','val','voi' -GenerateIncidentReport security
New-TransportRule "Notify Security if outbound email contains video (1)" -AttachmentExtensionMatchesWords 'aep','dzp','viv','vro','mp4.infovid','scm','dir','rms','wlmp','dzm','mswmm','amc','psh','3gp','veg','sfd','trp','wpl','m2p','ntp','aaf','bdmv','d3v','dck','gcs','ivr','m21','mk3d','mproj','msdvd','rdb','rmp','rv','screenflow','sec','swt','trec','usm','vcpf','viewlet','xej','dnc','ivf','playlist','spl','wm','bik','swf','webm','dcr','mani','prproj','wp3','mkv','avi','fbr','gfp','srt','piv','3gp2','bu','mpeg','wmv','scc','meta','gvi','vob','m4v','aepx','dzt','ts','ism','swi','amx','m2ts','rec','rmd','vpj','g64','mmv','ifo','wve','cpi','vp6','mov','vsp','mp4','mpg','hdmov','fcp','ogm','sbk','vc1','vgz','wmx','xesc','zm3','bnp','k3g','lvix','vp3','bin','mob','dmx','kmv','flv','par','vid','rmvb','dcr','tp','xvid','mnv','str','asf','bdm','camproj','mxf','yuv','0.89','avchd','dat','m1pg','mvd','roq','tsp','wmmp','ddat','f4f','imovielibrary','lsx','proqc','qt','sbt','video','yog','f4v','mts','3gpp','3mm','r3d','dav','smv','ogv','nvc','h264','3g2','dvdmedia','fcproject','ismv','sqz','tix','clpi','f4p','fli','hdv','m2t','mvp','nsv','rsx','smk','thp','ttxt','inp','mvc','m15','0.264','lrv','mvp','wmd','camrec','dxr','divx','stx','aetx','vep','dv4','db2','mpeg4','pds','mod','aec','ajp','dv','sfera','dvr','pmf','ced','dash','rm','ale','avp','bsf','dmsm','dream','imovieproj','otrkey','3p2','arcut','avb','avv','bdt3','bmc','cine','cip','cmmtpl','cmrec','cst','d2v','dce','dmsd','dmss','dpa','evo','eyetv','fbz','flc','flh','fpdx','ftc','gts','hkm','imoviemobile','imovieproject','ircp','ismc','izz','izzy','jss','jts','jtv','kdenlive','m21','m2v','mj2','mp21','mpgindex','mpls','mpv','mse','mtv','mve','mxv','ncor','nuv','ogx','pac','photoshow','plproj','ppj','prel','prtl','pxv','qtl','qtz','rcd','rum','rvid','rvl','sdv','sedprj','seq','sfvidcap','siv','smi','svi','tda3mt','tivo','tp0','tpd','tpr','tvlayer','tvs','tvshow','usf','vbc','vcv','vdo','vdr','vfz','vlab','vtt','wcp','wvx','wxp','xfl','xlmv','y4m','zm1','zm2','exo','lrec','mp4v','mys','vcr','w32','am','aqt','cvc','gom','mpeg1','mpv2','orv','rmv','ssm','zeg','arf','moi','zmv','wtv','mjp','gifv','mpe','dpg','mpl','rcproject','amv','tod','60d','moff','mp2v','tdt','dvr-ms','bmk','asx','edl','smil','snagproj','cmmp','dv-avi','eye','mgv','mp21','pgi','pro','stl','xml','avs','box','int','irf','scn','sml','ismclip','avs','evo','smi','awlive','m4e','mpg2','tdx','vivo','movie','vf','3gpp2','psb','axm','cmproj','dmsd3d','dvx','ezt','ffm','mqv','mvy','vp7','xel','aet','anx','avc','avd','axv','bdt2','bs4','bvr','byu','camv','cmv','cx3','dlx','dmb','dmsm3d','fbr','fcarch','ffd','flx','gvp','iva','jmv','ktn','m1v','m2a','m4u','mjpg','mpsub','mvex','osp','pns','pro4dvd','pro5dvd','pssd','pva','qtch' -GenerateIncidentReport security
New-TransportRule "Notify Security if outbound email contains video (2)" -AttachmentExtensionMatchesWords 'qtindex','qtm','rp','rts','theater','tid','tvrecording','vem','vfw','vix','vs4','vse','wot','xmv','mvb','nut','pjs','sec','0.787','ssf','mpl','clk','dif','vft','vmlt','anim','grasp','moov','pvr','vmlf','modd','bix','cel','dsy','gl','ivs','lsf','m75','mpf','msh','pmv','rmd','rts','scm','vdx' -GenerateIncidentReport security

The file extension lists are taken from fileinfo.com (audio and video).

It should also be noted that these rules are fairly simple – they are only looking at the file extension name and not the actual contents of the message.

Creating an Office 365 profanity filter (works for Exchange too)

As part of recreating the rules that my customer currently has set up with a popular cloud-based message hygiene platform, I needed to create an Office 365 profanity filter for Exchange Online. Believe it or not, there isn’t one built into the product (it disappeared with BPOS) but you can do some interesting things with DLP classification rules and policies.

I’d like to publish the exact steps here but I can’t, for commercial reasons. What I can do though is signpost some useful resources:

Once you’ve created a policy you can apply it in PowerShell with:

New-ClassificationRuleCollection –FileData ([Byte[]]$(Get-Content -path ProfanityPolicy.xml -Encoding byte -ReadCount 0))

If you need to update it then the cmdlet is Set-ClassificationRuleCollection and if you want to take it out again, Remove-ClassificationRuleCollection will do the trick.

With the classification in place, you can create rules that use the policy. In my case, one to block emails containing sensitive content (i.e. a list of pre-defined words) and send an incident report to a defined mailbox.

Even though I was working with Exchange Online (v15), the same process will work for Exchange Server 2013 and, presumably 2016 when it comes…

Finally, one gotcha I found (well, it was a user error really):

  • I thought my rule wasn’t working. When I later logged into the shared mailbox that blocked messages were copied to, I found copies of the messages I’d been sending for quite a while. My confusion was because I’d been testing with Policy Tips (which seemed a bit hit and miss in OWA) and that doesn’t actually block the message (doh!). As soon as I enforced the rule, my rude messages started bouncing back as expected…

NDR from message blocked by Office 365 profanity filter

Turning to a PAL when troubleshooting Exchange…

I recently found myself working with a customer to troubleshoot issues with their Exchange Servers. The servers were losing contact with each other periodically and then doing what Exchange is designed to do in this circumstance, failing databases over to other servers. Unfortunately, whilst I’m not going to go into the exact details here for reasons of confidentiality, it did completely underline my opinion (backed up by Microsoft’s Preferred Architecture for Exchange) that virtualised Exchange solutions using SAN storage are overly-complex and will cause issues at some point…

After going through the basics (are all the servers patched up to date, including consistent BIOS and firmware for all disks, controllers, NICs, etc.; has anti-virus software been disabled on the servers, especially host intrusion protection features; is the virtual infrastructure correctly configured, especially virtual networks; are there any tell-tale issues in the event logs?), I contacted my former colleague, Fujitsu Distinguished Engineer and Exchange Master, Nick Parlow (who fixes people’s broken Exchange solutions for a living).  Nick gave me some very useful advice for troubleshooting Exchange that I’m going to share here…

A few more points that might be useful:

Some design principles for Microsoft Exchange

In a previous role, I managed a team that was responsible for Microsoft Exchange design. Working with Microsoft, we established a set of design principles, some template designs and a rule-book for any changes made to those designs. Soon afterwards, Microsoft published their Preferred Architecture for Exchange and the similarity was striking (to be honest, I’d have been concerned if it was different)!

As that Preferred Architecture is publicly available, I think it’s fine to talk about some of the principles we applied. They seem particularly pertinent now as I was recently working with a customer whose Exchange design doesn’t follow these… and they were experiencing a few difficulties that were affecting the stability of their systems…

Physical, not virtual

Virtualisation has many advantages in many circumstances but it is not a “silver bullet”. It also brings complexity and operational challenges into Exchange design, with few (if any) advantages that would not be already provided by Exchange out of the box. Exchange is designed to make full use of the available hardware and Microsoft is able to provide large, low cost mailboxes within Office 365 (Exchange Online), without a requirement to virtualise their Exchange 2013 platform. In addition to the operational and supportability complexities that virtualisation brings, virtualising the Exchange deployment requires more Exchange design effort.

Deploy multi-role Exchange servers

Microsoft’s current recommended practice is to deploy multi-role Exchange 2013 servers (i.e. client access and mailbox roles on the same server) for the following reasons:

  1. Reduced hardware. Multi-role servers make best use of processor capacity given the more powerful server specifications which are now available.
  2. Reduced operational and capital expenditure. Fewer servers to deploy and manage.
  3. Building block design which is simple to deploy and scale. Automated deployment of standard server builds.

The mailbox server role must be designed not to exceed the maximum processor capacity guidance for multi-role servers; this provides confidence that the hardware deployed can co-host all roles on a single server. This is where the Exchange 2013 Server Role Requirements Calculator comes in…

Use direct attached storage – not a SAN

Microsoft designed Exchange 2013 to run on commodity hardware and believes this is the most cost effective way to provide storage for the Exchange mailbox databases.  Changes to the Exchange 2013 storage engine have yielded a significant reduction in I/O over previous versions of Exchange, allowing customers to take advantage of larger, cheaper disks and reduce the overall solution costs. In general, Direct Attached Storage (DAS) should be used in a Just a Bunch of Discs (JBoD) configuration although there are some circumstances where a Redundant Array of Inexpensive Devices (RAID) configuration may be used.

Microsoft uses a commoditised email platform with DAS and JBoD architecture to provide and support large, low cost mail mailboxes within Office 365. There are many more solution elements to consider with a SAN (Host Bus Adapters (HBAs), fibre channel switches and SAN I/O modules) as well as additional software for managing the infrastructure and firmware to keep up-to-date. Consequently, there is an increased likelihood of technical integration issues using a SAN and, once installed, a SAN infrastructure has to be carefully monitored and managed with appropriately skilled staff. In stark contrast, the costs of direct-attached JBoD solutions is falling as larger disks become available.

Native resilience

Database availability groups (DAGs) were introduced in Exchange Server 2010 to replicate databases between up to 16 servers. A DAG with multiple mailbox database copies, can provide automatic recovery from a variety of server, storage, network and other hardware failures. Auto-reseed functionality in Exchange Server 2013 allows for automatically bringing spare disks on line in the event of failure and creating new database copies.

If four highly available copies of each database are deployed, Exchange native resilience can be used without the requirement for third party backup solutions. Only specific requirements (i.e. ability to recover to an offline datacentre; recovery of deleted mailbox outside the deleted mailbox recovery retention time; protection against operational immaturity; protection against security breaches etc.) drive a requirement for adoption of a third party backup solution

Exchange Online uses the Exchange native resilience to protect against database failures, without resorting to the use of third party backup solutions.

Whilst a DAG can support 16 servers, it may be prudent to artificially limit the number of DAG members (e.g. to 12) in order to provide flexibility in upgrade scenarios.

Site resilience

DAGs can be extended between sites and copies of databases replicated across sites to provide additional redundancy. Each member of the DAG must have a round trip latency no greater than 500ms to contact the other members, regardless of their physical location. In general, Exchange DAGs should span at least two physical sites and Microsoft also recommends that separate Active Directory sites are used.

Mailbox distribution

With multiple sites in use, the next consideration is whether both are active (i.e. providing live service) or whether one is a secondary, passive, datacentre (i.e. invoked for disaster recovery purposes).  If all active mailboxes are hosted in a single site, and all passive copies of the mailboxes reside in a secondary site, the user distribution model is referred to as active/passive. If there are active mailboxes in both primary and secondary datacentres then the user distribution model is known to be active/active.

This should not be confused with the databases within the DAG, where only one copy of each database is active at any time.

Exchange 2013 simplified the client access model and with all clients connecting using HTTPS an active/active architecture is simple and spreads the client load across all Client Access servers, making best use of the deployed hardware.

This also facilitates a simplified SMTP namespace and allows automatic site failure (assuming the File Share Witness is located in a tertiary datacentre).


With today’s storage capabilities, large mailboxes are becoming normal.  The use of a native Exchange 2013 archive or a third party archiving solution is only required where there is a defined need for a user experience that warrants the management of email data (the ‘personal archive’ user experience, e.g. auto archive functionality) or by legal/policy requirements regarding the retention and discovery of email data (‘the regulatory archive’).

There is a common misconception that using a third party archive solution will provide a cost effective, single instance storage solution by differentiating ‘hot’ and ‘cold’ data and providing the ability to store ‘colder’ data on cheaper, slower disks. In fact, introducing a secondary system increases costs and complexity (in design and management) as well as reducing the flexibility of the solution.

Many organisations are electing to leave behind their archives with browser-only access as they migrate to larger online mailboxes in the cloud, e.g. using Exchange Online.


Whilst Exchange is supported in a virtualised environment, with SAN-attached storage, third party backup and making use of email archive solutions, deviating from the Preferred Architecture is a huge risk. The points in this blog post, combined with Microsoft’s advice linked above highlight the reasons to keep your Exchange design as simple as possible. Whilst a more complex design will probably work, identifying issues when it doesn’t will be a much bigger challenge.


Some patience required when changing a display name for an Exchange Online mailbox

Mrs W and I have been married for a long time but, until last week, she was still using her maiden name for work. Now, for a variety of reasons, is a good time for her to switch and, as we use Office 365 for her business email, I said “Yeah, it’s really simple; just let me know when you’ve told your contacts about the name change and I’ll switch it over.”

So, when the time came, I changed the display name in the Exchange Online Exchange Admin Center (no changes to her SMTP addresses were needed) and thought that would be it. Nope. Test emails sent came from the original display name. The same happened with another account that I changed the name on. Wondering if this was an Outlook issue, I tried from Outlook Web App: no difference. Test emails were sent back and forth to email addresses outside our Office 365 tenant (like my work account) and the original name stubbornly stuck – I even looked in the message headers and, there it was.

I’m not sure, but I think the issue was related to the offline address book as, the GAL reflected the change immediately but the offline GAL was still showing the old display name.

Unlike in an on-premises Exchange installation, I couldn’t update the address book: connecting to Exchange Online via PowerShell and asking for Get-Command *Offline* told me that the Update-OfflineAddressBook cmdlet is not available in Exchange Online (confirmed in the TechNet reference, which only refers to Exchange Server 2013).

Like so many things in Exchange (and I remember this from my original Exchange 4.0 training course in 1996), it proved to be one issue that’s best left for a few hours to fix itself. The offline GAL updated overnight and emails were then sent with the new display name (not sure why this affected OWA though…).

Working with the Exchange 2013 Server Role Requirements Calculator

For the last couple of years, I’ve had the privilege of leading a team of talented Exchange and Lync subject matter experts (either directly or more recently as a virtual team).  I’ve tried to keep up my technical skills but inevitably I’m not at the level of detail I once was and, as I switch to a more technical role, I’m expecting to have to re-learn a lot.

Thankfully, I won’t be alone – the team I’m joining is also full of talented subject matter experts – but I will need to stand on my own two feet.

That’s why I thought I’d write this post, with a few notes that are based on a recent conversation when my former colleague Mark Bodley checked over the Exchange calculations I’d used to create a guide price for a customer solution… [I’ve since made a couple of extra edits based on advice from Fujitsu’s resident Exchange Master, Nick Parlow]


To get started, some educated guesses can be made to adjust the defaults:

  • Exchange environment configuration:
    • 64-bit GCs, multi-role servers, non-virtualised, HA deployment (in line with our design principles).
    • Set number of servers per DAG and number of DAGs according to size of solution. Start DAG count low to keep hardware down – revise up later if required based on guidance elsewhere in the tool.
  • Site resilience configuration:
    • Single active/active resilient DAG (i.e. no passive servers).
    • Watch the RPO value – it won’t affect the server count but could be a factor in inter-datacentre bandwidth calculations.
  • Mailbox database copy configuration:
    • Increased HA database copies to 4; decreased lagged copies to 0 (lagged copies require more storage as transaction logs are retained for longer).
    • Increased number of HA database copies in secondary datacentre to account for non-lagged copies (in this case I had a 50/50 split between DCs).
  • Lagged database copy configuration: not used.
  • Exchange data configuration: left at defaults.
  • Database configuration: left at defaults.
  • Exchange I/O configuration: left at defaults
  • Transport configuration:
    • Set Safety Net expiration to 2 days (unless using lagged copies).
  • User mailbox configuration:
    • One tier for each logical group of users.
    • Days in work week set to 5 – unless there are significant groups of people working 7 days a week (affects log replication).
    • Mailbox configuration values will depend on organisation but the advice given for a starting point was to set:
      • Total send/receive capacity per mailbox per day to 100 (down from 200) messages.
      • Average messages size to 150KB (affects storage).
      • Mailbox size limits are used for capacity growth planning – split between main and personal archive should not affect the calculations.
      • Deleted item retention window may need to be increased to meet requirements (14 days is low if not using backups).
      • Multiplication factors come into play with mobile devices.
      • Desktop search engine value is more significant in Citrix environments than with Outlook cached mode clients.
  • Backup configuration:
    • Assumed software VSS via SCDPM, daily full backups.
    • Calculator will take highest value of backup/truncation failure tolerance and network failure tolerance (so only one of these really matters!).
  • Storage options:
    • JBOD with multiple databases per disk (in line with our design principles).
    • Main thing to watch here is the number of auto reseed volumes per server – align to failure rate of the disks.
  • Disk configuration: will depend on servers in use. For reference, with our Fujitsu PRIMERGY servers, the available options led us to:
    • 900GB system disks (10K RPM SAS 2.5″).
    • 4000GB database plus log disks (7.2K RPM SATA 3.5″).
    • 4000GB restore volume (7.2K RPM SATA 3.5″).
  • Processor configuration:
    • Core/server is of secondary interest but helps the calculator to advise on Global Catalog cores.
    • SPECint2006 value set to the lowest that will allow a suitable server utilisation number. In practice there will be a balance between servers/processor options and price. It’s useful to have a lookup of SPECint2006 values for a range of servers (e.g. Fujitsu PRIMERGY RX300) or you can search the SPECint site and take the value from the result column.
  •   Log replication configuration:
    • Will vary according to environment (and international spread of users) but rule of thumb used was (this should add up to 100%):
      • 1: 0.33%.
      • 2: 0.24%.
      • 3: 0.24%.
      • 4: 0.2%.
      • 5: 0.82% (I’m not sure why we have a night-time peak – automated emails sent overnight? Or this could be spread across others…)
      • 6: 0.31%.
      • 7: 0.34%.
      • 8: 1.46%.
      • 9: 6.46%.
      • 10: 10.08%.
      • 11: 10.55%.
      • 12: 11.06%.
      • 13: 9.48%.
      • 14: 9.61%.
      • 15: 10.52%.
      • 16: 10.41%.
      • 17: 8.31%.
      • 18: 5.07%.
      • 19: 1.94%.
      • 20: 0.95%.
      • 21: 0.84% (I might be tempted to up this slightly for the evening email check in many organisations…).
      • 22: 0.36%.
      • 23: 0.21%.
      • 24: 0.21%.
    • Network configuration: latency will depend on available datacentre connectivity.
  • Environment customisation: only used to generate naming configuration.

Check the results

With the main inputs in place, some fine tuning is probably required:

  • Role requirements:
    • Watch out for errors (e.g. over-utilised servers).  This is where the server count may need to be adjusted, or a higher-specification server used (SPECint2006 values). Ideally, servers should be close to 80% utilised [in a failure scenario] but not over. Getting the right CPU is more critical than memory as RAM can normally be added later! [also bear in mind the impact of other software on utilisation – for example file level antivirus, and any message hygiene software running on the box.]
    • Also check the recommended transport database location (we’ll come back to that in a moment).
  • Volume requirements:
    • Watch out for disk sizes/configurations that can’t be met!  If the solution is storage-bound, then it may be necessary to add additional servers (we use direct attached storage as a design principle) or, if available, then larger commodity disks would help. One clue to a potential issue is if the DB and log volume design/server table has DBx as a database copy name, rather than DB1-DB4, etc.
  • Storage design: confirms (or otherwise) that the solution can be deployed in a JBOD configuration (as per our design principles, although RAID is also an option).
    • This worksheet will also give the number of disks required in each server. Note that this is the count for database and log disks, restore volumes and auto reseed volumes – don’t forget the system disks for operating system/application binaries… and then think about the transport database.
    • In my solution, the recommended transport database location (from the role requirements) was the system disk but we were using 900GB 10K RPM SAS 3.5″ disks. With OS (150GB), Anchor LUN (2GB), Exchange binaries (50GB), and consideration for Exchange Logs (maybe 600GB) those disks are already pretty full, so it’s worth considering an extra RAID 1 volume for the transport database (over-ruling the calculator) and possibly hot spares for the RAID volumes too (depending on your attitude to risk).

A couple of other points to consider: public folders and unified messaging. I didn’t need public folders but don’t overlook them in your plans if they are in use.  As for unified messaging, it’s no longer a server role so is included in the calculations, up to a point – UM is likely to hit the CPU load, so it might be prudent to factor in some additional headroom with the SPECint2006 of the servers (to keep the utilisation well below the 80% mark).


Of course, there are just my notes on some of the things I checked to fit our reference design and they are not exhaustive. As the saying goes, your mileage may vary. For reference, I was using Exchange 2013 Server Role Requirements Calculator v6.6 but you should always use the latest available version and there is a very useful readme file which should be referenced when working with the calculator. You should also factor in this guidance on sizing Exchange 2013 deployments and consider the Exchange 2013 performance recommendations too.


[Edited 16 May 2015 to clarify comments re: server utilisation]