fscked-up Mac: creating a backup in OS X single user mode

11 months and 3 weeks after I bought it, my Mac Mini started playing up… I suppose I should be grateful that it’s just before the Apple warranty ends, not just after (although I did buy from Solutions Inc., who offer a 2 year warranty as standard… although they’ve told me it could take a month for repairs, so to try Apple first!).

First up, I noticed issues when copying files to a folder. It said the files already existed but they weren’t in the directory listing.  Then, I noticed that the Mac was switched off when it shouldn’t be. I powered it back on, only to find it got part way through booting (black screen with an Apple logo) and powered itself down. A corrupted file system and potentially flaky hard disk was my first thought… swiftly followed by “when was my last backup?”.

I started to work through Lex Freidman (@lexfri)’s Macworld tutorial on when good Macs go bad: steps to take when your Mac won’t start up, only to find that my Bluetooth keyboard wasn’t much help for Command-key combinations at bootup time (thankfully I had an Apple USB keyboard in the loft). Using Disk Utility to verify the disk confirmed some file system errors but a repair failed to fix them… so on to booting into single user mode and fsck -fy.

My problems only started after I upgraded OS X (the article is written for Mountain Lion) – I’m running Yosemite/10.10.5 (by the way, sw_vers -productVersion helped with that) – and I have a feeling all Disk Utility had been running under the covers was fsck but, regardless, it couldn’t fix my file system either…

** /dev/rdisk0s2
** Root file system
Executing fsck_hfs (version hfs-285).
** Checking Journaled HFS Plus volume.
The volume name is Macintosh HD
** Checking extents overflow file.
** Checking catalog file.
Incorrect block count for file coreduetd.db-wal
(It should be 698 instead of 587)
Missing thread record (id = 2396638)
Invalid extent entry
(4, 16638)
Missing thread record (id = 2539257)
Invalid extent entry
(4, 22174)
Invalid extent entry
(4, 22174)
Invalid extent entry
(4, 22174)
Invalid extent entry
(4, 22174)
Incorrect block count for file 2015-05-29 18.44.00.jpg
(It should be 1939 instead of 134219675)
Invalid extent entry
(4, 22174)
Invalid extent entry
(4, 22174)
Invalid extent entry
(4, 22174)
Invalid extent entry
(4, 22174)
Invalid extent entry
(4, 22174)
Incorrect block count for file 2015-05-29 19.04.41.jpg
(It should be 2448 instead of 526736)
Invalid extent entry
(4, 53125)
Invalid extent entry
(4, 53125)
Missing thread record (id = 136756985)
Incorrect number of thread records
(4, 21015)
** Checking multi-linked files.
** Checking catalog hierarchy.
Missing thread record (id = 2539257)
Missing thread record (id = 2585438)
Invalid volume file count
(It should be 1144777 instead of 1144780)
** Checking extended attributes file.
Incorrect number of extended attributes
(It should be 875596 instead of 875596)
Incorrect number of Access Control Lists
(It should be 1619 instead of 1620)
Overlapped extent allocation (id = 1479061, /private/var/db/CoreDuet/coreduetd.db-wal)
** Checking volume bitmap.
Volume bitmap needs minor repair for under-allocation
** Checking volume information.
Invalid volume free block count
(It should be 43923570 instead of 47674177)
Volume header needs minor repair
(2, 0)
** Repairing volume.
GetCatalogRecord: No matching catalog record found
FixBadExtent: Could not get catalog record for fileID 2924329
** The volume Macintosh HD could not be repaired.

So, onto that backup…

This is where Nestor Urquiza’s post (Mac OSX not booting? Make a backup from single user mode first) helped enormously. I decided not to touch my normal backups for this job and bought a new disk instead (a 1TB Seagate Backup Plus Slim was £50 in Currys – only a fraction more expensive than in the usual online locations) but the drive is pre-formatted using NTFS so I shrunk the volume in Windows Disk Management, then created a new simple volume in the free space with a single partition. This was formatted as exFAT (as ExFAT and NTFS were the only available options) and I ejected the disk from my PC and plugged it into the Mac (still in single-user mode), which responded with:

USBMSC Identifier (non-unique): 0x00000000 0xbc2 0xab24 0x100, 3

ls -l /dev/disk* told me that this was disk1

brw-r—– 1 root operator 1, 0 Oct 30 18:02 /dev/disk0
brw-r—– 1 root operator 1, 3 Oct 30 18:02 /dev/disk0s1
brw-r—– 1 root operator 1, 2 Oct 30 18:02 /dev/disk0s2
brw-r—– 1 root operator 1, 1 Oct 30 18:02 /dev/disk0s3
brw-r—– 1 root operator 1, 4 Oct 30 22:31 /dev/disk1
brw-r—– 1 root operator 1, 5 Oct 30 22:31 /dev/disk1s1
brw-r—– 1 root operator 1, 6 Oct 30 22:31 /dev/disk1s2

fstyp /dev/disk1d1 confirmed the NTFS partition:


whilst fstyp /dev/disk1d2 returned:


That’s the ticket! A couple more commands and I had a read/write file system and a directory to mount the external disk in

mount -uw /
mkdir /extdrive

but then it all ground to a halt:

mount -t msdos /dev/disk1s2 /extdrive

mount_msdos: Unsupported sector size (0)

I had a suspicion that ExFAT was the issue here so, as Windows 2000 and later will only format FAT32 up to 32GB (although the file system supports larger volumes), I used a third party utility (the GUI version of FAT32Format created by Ridgecrop Consultants and as described by Matthew Nawrocki). Once the drive was reformatted as FAT32 instead of ExFAT, it mounted without any issues on the Mac.

I wrote a couple of test files… then started the bulk copies…

cp -r /Users/mark/Downloads /extdrive/
cp -r /Users/mark/Desktop /extdrive/

Finally, when all files were copied, I unmounted the USB drive (and checked I could read the files on another PC):

umount /extdrive

At the time of writing, I still need to get my Mac fixed. I guess I’ll be making an appointment to see a “genius” at my local Apple Store but at least I have a backup if the disk is swapped out or wiped. Actually, I got nervous about using FAT32 for my Mac backups, so I’m currently re-running the process with an HFS-formatted disk (using my old MacBook to create the volume)… and using a slightly-amended cp command for a verbose output and to preserve the file metadata:

cp -pRv /Users/mark/Pictures/2015 /extdrive/

I suspect there may be more blog posts to follow as this story develops…

Up to 22% discount on Office 365 Home subscriptions

A few weeks ago, I wrote about Office 365 Home, which allows me to get the latest version of Microsoft Office on up to 5 PCs/Macs, 5 tablets and 5 phones.  Originally, I took out a one month free trial, after which it becomes a monthly subscription at £7.99.

If you pay annually, you can get 2 months free. Add that to the one month free trial and by my calculations that’s a 22% discount on Office 365 in year 1 and 16% in subsequent years.

Beware though, if you make a payment too early in your trial period, the free trial ends immediately.

I also found that, when I went to renew, the price was in US Dollars (although that might actually have been cheaper, before credit card charges, etc.). I contacted Microsoft support, who were happy to make the required changes on my account from monthly to annual renewal.

Restricting access to Yammer

A few days ago, Matt Ballantine wrote about Enterprise Social Media and the need to focus on building an audience:

Internal communications people can fall into the trap of believing that what they produce is content rather than advertising. Internal communications appears to be the only form of direct marketing to which there is no legal right to opt out.


The challenge then with Enterprise Social Networks, especially when they are treated as an internal media channel, is that if all you are pushing out is advertising (and yes, the latest interview with the CEO about the next 5 year strategy is advertising) you are trying to build an audience on marketing alone.

So, cue Yammer, Microsoft’s Enterprise Social Networking product, purchased a few years ago and slowly being integrated into Office 365…

As I wrote back in July, Yammer comes in two flavours:

  • Yammer Basic is a bit like the wild west – users sign up with their corporate email accounts and a network is formed, using company resources, but over which the company has no control.
  • Yammer Enterprise is a paid product, included in certain Office 365 Enterprise subscriptions, which provides a level of administrative control.

Yammer tile from Office 365But, here’s the gotcha – once you activate Yammer on your Office 365 subscription, a Yammer tile will appear on the Office 365 App Launcher and you have no way to turn it off.

I was recently working with a customer who had activated Yammer on their domains (to shut down the anarchy of Yammer Basic) but who wasn’t ready to start using the product yet (going back to Matt’s point about building an audience – i.e. launching the platform in a controlled manner, with appropriate business sponsorship and support).

Disabling logon to Yammer

With a Yammer tile in Office 365 but no way to turn it off, I was left looking at options for restricting access to Yammer:

  1. Use block lists to prevent users from logging on. That doesn’t scale and would be an administrative nightmare, so it’s not really a credible option.
  2. Disable Yammer in ADFS using a claims transformation rule (more information on TechNet). This would have been a nice idea except that Yammer SSO is deprecated since support for Office 365 authentication was introduced (it’s still supported, but not being developed). Denying access to Yammer on the Office 365 Identity Platform relying party trust meant that I also denied access to other Office 365 services!
  3. Use PowerShell to modify user licences except that doesn’t work – changes to the YAMMER_ENTERPRISE plan do not have any effect.
  4. Use Yammer’s logical firewall to block access based on IP address (thanks to Steve Rush for the suggestion). This is a bit crude but it works – just make sure there is a range for which access is allowed, so you can still get in and administer the network when you are ready to start using it!

Blocking access to Yammer via IP - end user experience

Lync client does not retrieve conversation history and meeting information from Exchange

In addition to the challenges created by the unified contacts store, my recent Office 365 migration project saw some issues where a user’s Lync/Skype for Business client failed to pick up a change in Exchange Web Services (EWS) as part of the move to Skype for Business Online.

The reason for this is unclear but I’m not the only one who’s experienced it – Richard Brynteson describes exactly the same scenario where, after a migration from an on-premises Exchange mailbox to Exchange Online, the Lync 2013 client is unable to connect to the Exchange server and pull conversation history and meeting information.

If we looked at the configuration information for the Lync client (by Ctrl+right clicking on the Lync taskbar icon), we could see that the client was not autodiscovering the move to Exchange Online and still showed on-premises Exchange details, instead of a blank EWS Internal URL and an entry of https://outlook.office365.com/EWS/Exchange.asmx/WSSecurity for the EWS External URL.

One suggestion given to me to force a new autodiscovery search was to wipe the user’s cached client information (in %appdata%\Local\Microsoft\Office\15.0\Lync\sip_alias@domainname.tld). That sounded a little destructive but Richard’s post suggests removing a single registry key: HKEY_CURRENT_USER\Software\Microsoft\Office\15.0\Lync\username@domainname.tld\Autodiscovery.

Even better though is the solution from a comment on Richard’s blog post, from “Chad”:

We’ve resolved this by having users sign out of Lync and then choose “Delete my sign-in info”, then just sign back in and their Lync client now connects to 365. We provide this as post-migration steps to users once we move their mailboxes to 365. Hope that helps and is typically easier than deleting [registry] entries.

That’s a much more user-friendly fix (which worked for me) – and it’s one to add to the project FAQ list.

Unified Contact Store requires Lync user to be migrated to Office 365 before Exchange mailbox

In a recent Office 365 project, I came across an issue where, if we migrated a user’s Exchange 2013 mailbox to Exchange Online before we migrated their Lync 2013 user to Skype for Business Online, the Move-CsUser cmdlet generated an error:

Move-CsUser: Exception of type ‘Microsoft.Rtc.Management.AD.Helpers.RollbackException’ was thrown

This is described in Camille de Bay’s blog post and appears to be related to the Unified Contact Store, which is enabled by default with Lync 2013 and Exchange 2013.  There appear to be two options:

  1. Migrate Lync before Exchange
  2. Use the -force switch with the Move-CsUser cmdlet (which will result in a loss of contacts)

Dave Stork goes on to describe some issues with a combination of the UCS, Lync on-premises and Exchange Online (and these appeared to apply to my Lync 2013/Skype for Business Online hybrid solution too).

Microsoft knowledge base article 2614614 has lots more information on integrating Exchange Online with Skype for Business Online, Lync Server 2013, or a Lync Server 2013 hybrid deployment

Tools for troubleshooting Outlook autodiscover

In my post last week about Office 365 and proxy servers, I mentioned issues with Outlook autodiscover.  These were not exactly easy to troubleshoot, often with multiple subject matter experts looking from different angles (network, client applications, Exchange, firewalls, etc.). During the process, we used a few tools (as well as examining the traffic hitting the proxy servers) and I thought I’d highlight them here (if only for my own future reference):

The curious case of the Spotify squatter

Yesterday, I was playing music on Spotify and it kept stopping because someone else was using my account… that’s not an uncommon occurrence as my kids are often using it but I didn’t think they were this time. After the usual squabble over “Play it here”, Nno, play it here”, “No. Play. It. Here.”, I managed to listen to the tracks I wanted to hear.

Then, this morning, I tried to sync some music to my Spotify account, only to find that my iPhone told me Spotify was being used on a complete stranger’s Phone!

One quick password change later and I was sure no-one else was using it. I later removed all devices from my account and re-added them, just for good measure.

Later in the day though, I noticed that all of my playlists were missing. I also saw that my activity stream showed a lot of music that I hadn’t listened to:

These are not my songs!

Someone else has definitely been using my account. Or at least that’s what Spotify thinks!

I could live with the account activity but missing playlists were a big concern. Luckily, Spotify support pointed me to a link to recover playlists where, sure enough, I saw they had been deleted yesterday! It took a few visits to that link before all of my playlists were located and recovered but I seem to be back to where I was before the mix-up.

Now, I don’t think that Spotify has been compromised – if someone had hijacked my account they would have changed my password and locked me out, surely? But I do suspect a database corruption. Spotify aren’t admitting anything is up, of course… but my trust in the service has been severely damaged.

PowerShell snippets

Writing PowerShell scripts over the last couple of weeks has been a steep curve. I’ve watched PowerShell from afar over the years but don’t really use it enough to say I know it.  Luckily, many others do, and they’ve posted their knowledge on the ‘net. This is what I drew upon:

Office 365 and proxy servers: like oil and water?

Office 365 and proxy servers don’t mix very well. Well, to be more accurate, thousands of Outlook, Skype for Business and OneDrive for Business clients, each with multiple connections to online services can quickly build up to a lot of (persistent) connections. If you haven’t already, it’s well-worth reading Paul Collinge’s blog post on ensuring your proxy server can scale to handle Office 365 traffic.

Microsoft recommends that the network is configured to allow unauthenticated direct outbound access to a published list of URLs and IP ranges (there’s also an RSS feed) – although I’ve had customers who take issue with this and don’t think it’s a reasonable expectation in the enterprise. My view? You’re adopting cloud services; your network boundary has moved (disappeared?) and the approach you take to managing the connectivity between services needs to change.

Perhaps as more people take advantage of services like ExpressRoute for Office 365, things will change but, for now, every Office 365 implementation I work on seems to involve a degree of proxy bypassing…

Some of the issues I experienced in a recent implementation included:

  • OneDrive for business unable to perform an initial synchronisation, but fine on subsequent syncs. It seems that the OneDrive client downloads http://clientconfig.microsoftonline-p.net/fplist.xml when it first syncs. We could get it to work when going through a different proxy server, or direct to the Internet; but the main proxy server had to have a list of trusted sites added. The managed services provider had previously allowed access to some known IP addresses (a risky strategy as they change so frequently and the use of content delivery networks means they are not always under Microsoft’s control), but the proxy server had the capability to trust a list of target URLs too.
  • Outlook unable to reliably redirect after Exchange mailboxes were migrated to Exchange Online. In this case, we found that, even with the trusted URLs in place on the proxy, as part of the Outlook Autodiscover process, Outlook was trying to contact autodiscover-s.outlook.com. The proxy wasn’t allowing unauthenticated access and Outlook didn’t know how to cope with the authentication request. Once autodiscover-s.outlook.com had been added to the proxy server’s unauthenticated access list, Outlook Autodiscover began to work as intended.
  • Lync/Skype for Business Online calls working internally, but not with external parties. Users dropping off the call after a few seconds. We still haven’t got to the bottom of this, but strongly suspect the network configuration…
  • Exchange Hybrid free/busy information not available cross-premises. Again, this seems to be related to the Exchange servers’ ability to see the Internet (free/busy lookups are performed by the server, not the client).

Further reading

Selectively remove tiles from the Office 365 App Launcher

This time last year, Office 365 gained an App Launcher as part of a new navigation experience for Office 365 on the web. Users can add and remove tiles from this launcher – and administrators can provide new tiles to point to corporate resources – for example a CRM platform or the company intranet.

Unfortunately, not all customers want their users to use all of the features and functionality in Office 365 and the administrative controls to manage the App Launcher for all users are limited.  I’d argue that part of consuming a cloud service is adapting to new features and functionality as they are released but that doesn’t go down well with everyone, often leaving me trying to find ways to disable or hide parts of the service. The following settings may help to selectively remove tiles from the Office 365 App Launcher but It’s not always straightforward – and it’s also subject to change (with a new admin center on the way):

  • Admin: revoke a user’s administrative rights.
  • Instant messaging and web conferencing: remove the Skype for Business Online licence and this functionality will disappear (there is no associated tile).
  • Mail, Calendar, People, Tasks: remove the Exchange Online licence and these tiles will go too.
  • OneDrive for Business, Sites, Office Web Apps: remove the SharePoint Online licence (which also requires that you remove the Office Online licence).
  • Office 365 Store: a switch was recently added to disable this tile, under Service Settings, User Purchasing, Display Office 365 App Store Tile.
  • OneDrive for Business: hide in the SharePoint Admin Center settings, under show or hide options.
  • Office 365 groups: Using PowerShell against Exchange Online, edit the Outlook Web Access policy with Set-OwaMailboxPolicy -GroupCreationEnabled $False -Identity PolicyName. If you only want to apply the change to a subset of users, create a new policy and apply it accordingly.
  • Sites: hide in the SharePoint Admin Center settings, under show or hide options.
  • Delve: turn off the Office Graph in the SharePoint Admin Center settings. Delve will still be there in parts though: for example when users access their profile.
  • Sway: turn off under Service Settings, Sway, Let people in your organization use Sway. Unfortunately it won’t remove the tile [update: yes it will now!].
  • Video: in the SharePoint Admin Center settings, under Streaming Video Service, disable streaming video through Azure Media Services and disable the Video Portal.
  • Yammer: for this one you’re between a rock and a hard place: Yammer Basic is anarchic; Convert to Yammer Enterprise and the tile will be visible to users – you cannot turn it off.

Some of these options merely hide capabilities – they may not be entirely disabled – and my recommendation would always be to leave settings enabled and teach users how to make use of the platform.  In particular, turning off the Office Graph may have wider reaching implications.

Further reading

Meet the Office 365 App Launcher