Where did my blog go? And why didn’t I have backups?

Earlier this week I wrote a blog post about SQL Server. I scheduled it to go live the following lunchtime (as I often do) and then checked to see why IFTTT hadn’t created a new link in bit.ly for me to tweet (I do this manually since the demise of TwitterFeed).

To my horror, my blog had no posts. Not one. Nothing. Nada. All gone.

Where did all the blog posts go?

As I’m sure you can imagine, the loss of 13 years’ of work invoked mild panic. But never mind, I have regular backups to Dropbox using the WordPress Backup to Dropbox plugin. Don’t I? Oh. It seems the last one ran in March.

History: Backup completed on Saturday March 11, 2017 at 02:24:40.

Ah.

Now, having done some research, my backups have probably been failing since my hosting provider updated PHP on the server. My bad for not checking them more carefully. I probably hadn’t noticed because the backup was partially running (so I was seeing the odd file written to Dropbox) and I didn’t realise the job was crashing part way through.

Luckily for me, the story has a happy ending*. My hosting provider (ascomi) kept backups (thank you Simon!). Although they found that a WordPress restore failed (maybe the source of the data loss was a database corruption), they were able to restore from a separate SQL backup. All back in a couple of hours, except for the most recent post, which I can write again one evening.

So, what about fixing those backups, Mark

Tonight, before writing another post, I decided to have a look at the broken backups.

I’m using WordPress to Dropbox v4.7.1 on WordPress 4.8, both of which are the latest versions at the time of writing. The Backup Monitor log for my last (manual) attempt at backing up read:

22:55:57: A fatal error occured: The backup is having trouble uploading files to Dropbox, it has failed 10 times and is aborting the backup.
22:55:57: Error uploading ‘/home/mwilson/public_html/blog/wp-content/plugins/wordpress-seo/images/banner/video-seo.png’ to Dropbox: unexpected parameter ‘overwrite’
22:55:56: Error uploading ‘/home/mwilson/public_html/blog/wp-content/plugins/wordpress-seo/images/banner/configuration-service.png’ to Dropbox: unexpected parameter ‘overwrite’
22:55:56: Error uploading ‘/home/mwilson/public_html/blog/wp-content/plugins/wordpress-seo/images/banner/news-seo.png’ to Dropbox: unexpected parameter ‘overwrite’
22:55:55: Error uploading ‘/home/mwilson/public_html/blog/wp-content/plugins/wordpress-seo/images/editicon.png’ to Dropbox: unexpected parameter ‘overwrite’
22:55:54: Processed 736 files. Approximately 14% complete.
22:55:54: Error uploading ‘/home/mwilson/public_html/blog/wp-content/plugins/wordpress-seo/images/link-out-icon.svg’ to Dropbox: unexpected parameter ‘overwrite’
22:55:53: Error uploading ‘/home/mwilson/public_html/blog/wp-content/plugins/wordpress-seo/images/extensions-local.png’ to Dropbox: unexpected parameter ‘overwrite’
22:55:53: Error uploading ‘/home/mwilson/public_html/blog/wp-content/plugins/wordpress-seo/images/question-mark.png’ to Dropbox: unexpected parameter ‘overwrite’
22:55:48: Processed 706 files. Approximately 13% complete.
22:55:42: Processed 654 files. Approximately 12% complete.
22:55:36: Processed 541 files. Approximately 10% complete.
22:55:30: Processed 416 files. Approximately 8% complete.
22:55:24: Processed 302 files. Approximately 6% complete.
22:55:18: Processed 170 files. Approximately 3% complete.
22:55:12: Processed 56 files. Approximately 1% complete.
22:55:10: Error uploading ‘/home/mwilson/public_html/blog/wp-content/languages/plugins/redirection-en_GB.mo’ to Dropbox: unexpected parameter ‘overwrite’
22:55:10: Error uploading ‘/home/mwilson/public_html/blog/wp-content/languages/plugins/widget-logic-en_GB.mo’ to Dropbox: unexpected parameter ‘overwrite’
22:55:09: Error uploading ‘/home/mwilson/public_html/blog/wp-content/languages/plugins/widget-logic-en_GB.po’ to Dropbox: unexpected parameter ‘overwrite’
22:55:09: Error uploading ‘/home/mwilson/public_html/blog/wp-content/languages/plugins/redirection-en_GB.po’ to Dropbox: unexpected parameter ‘overwrite’
22:55:06: SQL backup complete. Starting file backup.
22:55:06: Processed table ‘wp_yoast_seo_meta’.
22:55:06: Processed table ‘wp_yoast_seo_links’.
22:55:06: Processed table ‘wp_wpb2d_processed_files’.
22:55:06: Processed table ‘wp_wpb2d_processed_dbtables’.
22:55:06: Processed table ‘wp_wpb2d_premium_extensions’.
22:55:06: Processed table ‘wp_wpb2d_options’.
22:55:06: Processed table ‘wp_wpb2d_excluded_files’.
22:55:06: Processed table ‘wp_users’.
22:55:06: Processed table ‘wp_usermeta’.
22:55:06: Processed table ‘wp_terms’.
22:55:06: Processed table ‘wp_termmeta’.
22:55:06: Processed table ‘wp_term_taxonomy’.
22:55:06: Processed table ‘wp_term_relationships’.
22:55:05: Processed table ‘wp_redirection_logs’.
22:55:05: Processed table ‘wp_redirection_items’.
22:55:05: Processed table ‘wp_redirection_groups’.
22:55:05: Processed table ‘wp_redirection_404’.
22:55:03: Processed table ‘wp_ratings’.
22:55:03: Processed table ‘wp_posts’.
22:54:54: Processed table ‘wp_postmeta’.
22:54:49: Processed table ‘wp_options’.
22:54:48: Processed table ‘wp_links’.
22:54:48: Processed table ‘wp_feedfooter_rss_map’.
22:54:48: Processed table ‘wp_dynamic_widgets’.
22:54:48: Processed table ‘wp_comments’.
22:54:45: Processed table ‘wp_commentmeta’.
22:54:43: Processed table ‘wp_bad_behavior’.
22:54:43: Processed table ‘wp_auth0_user’.
22:54:43: Processed table ‘wp_auth0_log’.
22:54:43: Processed table ‘wp_auth0_error_logs’.
22:54:43: Starting SQL backup.
22:54:41: Your time limit is 90 seconds and your memory limit is 128M
22:54:41: Backup started on Tuesday July 25, 2017.

Hmm, fatal error caused by overwriting files in Dropbox… I can’t be the only one having this issue, surely?

Indeed not, as a quick Google search led me to a WordPress.org support forum post on how to tweak the WordPress Backup to Dropbox plugin for PHP 7. And, after making the following edits, I ran a successful backup:

“All paths are relative to $YOUR_SITE_DIRECTORY/wp-content/plugins/wordpress-backup-to-dropbox.

In file Dropbox/Dropbox/OAuth/Consumer/Curl.php: comment out the line:
$options[CURLOPT_SAFE_UPLOAD] = false;
(this option is no longer valid in PHP 7)

In file Dropbox/Dropbox/OAuth/Consumer/ConsumerAbstract.php: replace the test if (isset($value[0]) && $value[0] === '@') with if ($value instanceof CURLFile)

In file Dropbox/Dropbox/API.php: replace 'file' => '@' . str_replace('\\', '/', $file) . ';filename=' . $filenamewith 'file' => new CURLFile(str_replace('\\', '/', $file), "application/octet-stream", $filename)

(actually, a comment further down the post highlights there’s a missing comma after $filename) on that last edit so it should be 'file' => new CURLFile(str_replace('\\', '/', $file), "application/octet-stream", $filename),)

So, that’s the backups fixed (thank you @smowton on WordPress.org). I just need to improve my monitoring of them to keep my blog online, and my blood pressure at sensible levels…

 

*I still have some concerns, because the data loss occurred the night after a suspected hacking attempt on my LastPass account; which seems to have been thwarted by second-factor authentication… at least LastPass say it was…

Manually removing servers from an Exchange organization

I’m starting this blog post with a caveat: the process I’m going to describe here is not a good idea, goes against the advice of my colleagues (who have battle scars from when it’s been attempted in a live environment and not gone so well) and is certainly not recommended. In addition, I can’t be held responsible for any unintended consequences of following these steps.

Notwithstanding the above, I found myself trying to configure the Exchange Hybrid Configuration Wizard (HCW) in a customer environment, where the wizard failed because it was looking for servers that don’t exist any more.

I had two choices:

  1. Recover the missing Exchange servers with setup.exe /m:RecoverServer, then uninstall Exchange gracefully (for 12 servers!).
  2. Manually remove the servers using ADSI Edit.

I explained the situation to my customer, who discussed it with his Exchange expert, and they directed me to go for option 2 – this was a test environment, not production, and they were prepared to accept the risk.

Fearing the worst, I made a backup of Active Directory, just in case. This involved:

  1. Installing the Windows Server Backup Command Line Tools feature on the domain controller.
  2. Running wbadmin start systemstatebackup -backuptarget:driveletter:
  3. Sitting back and waiting for the process to complete.

With a backup completed, I could then:

  1. Run ADSI Edit.
  2. Open the configuration naming context.
  3. Navigate to CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=organizationname,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=domainname,DC=tld
  4. Delete the records for the servers that no longer exist.
  5. Restart each of the remaining Exchange servers in the organisation in turn.
  6. Check the server list in ECP.

(Incidentally, FYDIBOHF23SPDLT is “Caesar’s Cipher” for EXCHANGE12ROCKS).

Murat Yildirimoglu’s Windows IT Pro article entitled “How to Uninstall a Stubborn Exchange Server” goes into more detail, including completely removing an Exchange organisation from Active Directory, should that be required (Christopher Dargel covers that too).

The process seemed to work but the danger of manually removing servers from an Exchange organization like this is the potential side effects of “unknown unknowns” (which you can be sure won’t surface immediately). It did let me progress to the next stage of the HCW though. More on that in a future blog post…

Sorting out my home backups

After my parents-in-law’s recent burglary (and related data loss), I started to think more seriously about my household’s backups which are spread across a variety of USB drives, NAS units and cloud services (Dropbox, SkyDrive, Box.net, etc.).

My plan is to:

  1. Duplicate – hard drives fail. I know, because I’ve lost data that way – and RAID is no substitute for a proper backup (as I learned the hard way). If it doesn’t exist in (at least) two places, it doesn’t exist.
  2. Consolidate – bits and pieces on various drives is a nightmare – to know that it’s definitely backed up, I need to know it’s on the “big backup drive” (as well as in the primary source).
  3. Archive – both physically (media stored in a safe) and virtually (upload to the cloud). Be ready for some long uploads though, over an extended period (I only have ADSL 2 – no fibre here).

Steps 1 and 2 work hand in hand and, last weekend, I picked up a 3TB Seagate Backup Plus Desktop drive. I’m not using the bundled backup software that offers idiot-proof backups for both local and social media (Facebook, Flickr) data but installing the software on my MacBook includes Paragon NTFS for Mac, which means I can use this drive with Macs and PCs without reformatting (there is a Mac version too – although the only differences I can see from a comparison of Seagate’s data sheets for “normal” and Mac versions are: Firewire and USB 2.0 cables instead of USB 3.0; downloadable HFS+ driver for Windows instead of preloaded NTFS driver for Mac OS X; 3 year warranty instead of 2 years).

Step 3 is more involved. I did some analysis into a variety of cloud services a while ago and found that each one has pros/cons depending on whether you want to back up a single computer or multiple computers, limitations on storage, cost, etc. I didn’t get around to publishing that information but there is a site called Which Online Backup that might help (although I’m not sure how impartial it is – it’s certainly nothing to do with the Which? consumer information/campaign service).

My current thinking is that I’ll continue to use free services like Dropbox to backup and sync many of my commonly-used files (encrypting sensitive information using TrueCrypt) at the same time as creating a sensible archive strategy for long term storage of photographs, etc. That strategy is likely to include Amazon Glacier but, because of the way that the service works, I’ll need to think carefully about how I create my archives – Glacier is not intended for instant access, nor is it for file-level storage.

I’ll write some more as my archive strategy becomes reality but, in the meantime, the mass data copy for the duplicate and consolidate phases has begun, after which all other copies can be considered “uncontrolled”.