Using a VPN to watch ITV content outside the UK

Those who follow me on Twitter (@markwilsonit) will probably be aware that I recently spent some time in mainland Europe – travelling through France, Germany and Switzerland with my family. You’ll probably also be aware that one of my hobbies is road cycling – and that I like to watch the highlights from the three Grand Tours (Giro d’Italia, Tour de France and Vuelta a España) and from the Tour of Britain. With the Vuelta in full swing as my holiday started, I wanted to make sure I could still catch the highlights on ITV4!

Even with the new EU mobile roaming arrangements that mean I can use my mobile data allowance in other EU countries, I didn’t expect to be able to stream content reliably, so I took out a subscription to ITV Hub+, allowing me to download ITV programmes with the ITV Hub app (on Wi-Fi) and play back later, without ads. This worked brilliantly on the ferry to France but not so well once I was in my Paris hotel room, where the app detected I was outside the UK and denied access to content with a variety of error messages:

ITV Hub download error outside the UK ITV Hub download error outside the UK ITV Hub download error outside the UK

I was pretty annoyed – after all, there was no mention of UK-only coverage when I subscribed to the ITV Hub+ and the ITV website says:

“Where can I use a Hub+ subscription?

As long as you’re signed into your account, you’ll be able to use your Hub+ subscription almost anywhere. Watch ad-free telly on our website, download and catch up on the go on your mobile or tablet, or binge on your favourite shows with no interruptions on your Smart TV!”

but I did find the limitation in their troubleshooting guide later:

I am abroad and can’t watch videos
The ITV Hub is only available within the UK as we don’t hold international rights for all of our shows. If you’re lucky enough to be on holiday or you live abroad, you won’t be able to watch ITV Hub until you return to the UK”

After a bit of a rant on Twitter (no response from ITV, of course), I thought about using a VPN (and @JFDuncan suggested Plex).

Unfortunately, my own VPN back to my NAS didn’t work (on reflection, L2TP/IPSec was not the best choice of transport – as @GarryMartin pointed out when I originally set it up) and I was nervous about using a third party service until Justin Barker (@JustinBarker77) suggested TunnelBear:

Recommendations are always good. And TunnelBear seemed more legitimate than some of the sites I found…

At first, I didn’t have much luck – even after following TunnelBear’s troubleshooting advice for accessing content. 24 hours later though, something had cleared (maybe I had a different IP address, maybe it was something on my iPhone) and ITV Hub+ worked flawlessly over hotel Wi-Fi and a VPN back to the UK. I could download my cycling highlights for later playback and the VPN tunnel even seemed to improve the Holiday Inn Wi-Fi reliability – possibly due to QoS restrictions prioritising potential business traffic (VPN) over leisure (downloading videos)!

I did have some challenges with playback – so I put the iPhone into Airplane Mode before watching content, just in case the ITV Hub app detected I was outside the UK again, but each time I wanted to download over the next few days I enabled the VPN and all was good. I also subscribed to TunnelBear for a month’s worth of unlimited data allowance (I soon chewed through the 1GB I got for tweeting about the service!).

Hopefully, this information will help someone else who’s frustrated by paying for a download service and then finding it doesn’t work outside the UK…

Downloading multiple YouTube videos for offline playback

A few weeks ago, I wrote about a nifty utility called youtube-dl that can be used to download content from YouTube for offline playback (indeed, I’m writing this on the last day of my holidays, having played precisely none of the content I downloaded to watch whilst I was away!).

In the original post, I suggested giving youtube-dl the URL for a playlist to download all videos in the playlist. It’s also smart in that if it detects any videos that are already present in the folder, it will skip them – e.g.:

[download] Cloud Tech 10 – 3rd July 2017 – Azure Machine Learning, Jenkins, Petya detection and more-ymKSGTR55LQ.mp4 has already been downloaded

But what if you want to download lots of videos that are unrelated – or just certain videos from a large list? In my case, I wanted to download a bunch of recent videos from the Global Cycling Network (GCN) – a YouTube channel that I often watch but which has thousands of videos – I certainly didn’t want to download the entire playlist!

Instead, create a file with the download commands for the individual videos, e.g.:

youtube-dl -f 22 https://www.youtube.com/watch?v=5RsFWlvJjOg
youtube-dl -f 22 https://www.youtube.com/watch?v=O7FxZ1kFIW0
youtube-dl -f 22 https://www.youtube.com/watch?v=iOaeo3_E8R4

Rather than sitting at the terminal, running each one and waiting, save the file with a .sh extension (assuming a Unix-based OS – like MacOS) and then kick them off at once.

My file was called dl-gcn.sh but it’s no good running that from the Terminal – bash will complain.

bash: dl-gcn.sh: command not found

Instead, prefix with sh as follows:

sh dl-gcn.sh

and the downloads should run through in serial fashion, whilst you get on with something more interesting…

Combining GPX files for Strava

This morning was spent on my bike… as was a fair chunk of this afternoon… as is a fair chunk of many summer weekends, much to Mrs W’s disappointment.

My friend Andy and I put in 60 miles in the sunshine, on a big loop around Milton Keynes. It seems my route planning is pretty spot on, as it was almost the exact opposite of a charity ride going the other way around (we passed the same riders twice!). Unfortunately, my ability to “press the start button on my Garmin cycle computer” is clearly less good – I was about a mile from home and heading out of town when I realised I’d forgotten to start tracking my ride!

My OCD can’t cope with this. It would be able to cope with turning around, going back up the hill, starting the computer and starting the ride again – but not with some missing kilometres in my ride data! Luckily, Andy was also riding with a Garmin bike computer. Even though he’d also forgotten to start his, he was wearing a Garmin watch too – so I could combine his data and mine (we’d ridden side by side for the first part of the ride…).

I’ve blogged before about GPS Track Editor, which is a fantastic piece of free software. Using this, I could edit Andy’s data to just the part I had missing, then combine it with mine and merge the two tracks (the short gap doesn’t matter – Strava will straight-line the route between the two points). I also tried merging the files with a tool from gotoes.org – unfortunately, that ended up with a ride that was effectively double the length of what we rode (two loops). it would probably have worked with my edited files but I could also merge them in the GPS Track Editor…

Combining tracks in GPS Track Editor

I then deleted the original (short) ride from Strava and re-uploaded. Sorted.

Just one thing to sort out – all of the PRs I got on today’s ride (and there were a few) were recorded as second places by the second upload. No worries – Strava has a “refresh my achievements” tool. which sorted out that particular issue. Now my ride has the complete distance… and my achievements are correct too…

Installing youtube-dl on a Mac to watch YouTube videos when working offline

My work pattern at the moment means that I’m spending a lot of time travelling on trains up and down the country (in fact, as this post is published, I’ll be somewhere between Bedford and Sheffield). A combination of fatigue and motion sickness means that this isn’t always a good opportunity to work on the train but it is potentially an opportunity to listen to podcasts or, unlike when I’m driving, to watch some videos. Unfortunately, travelling at 125 miles an hour with a varying quality of 4G data signal doesn’t always lend itself well to streaming from YouTube, etc.

That’s where youtube-dl comes in – I can download videos to my MacBook before I leave home, and watch at my leisure on the train. So, how do you get started?

Well, the Mac App Store website helped me. Following advice there, I issued two commands in Terminal to first install the HomeBrew package manager for MacOS, and then to install youtube-dl:

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" < /dev/null 2> /dev/null

brew install youtube-dl

So, with youtube-dl installed, how do I use it? The youtube-dl readme.md file has lots of information but it’s not exactly easy to digest.

I found that:

youtube-dl -F youtubeurl

would give me a list of available video formats for a given URL and reading about YouTube media types led me to the very important number 22 for MP4 video at 720p with H.264 encoding and AAC audio. That should play on a wide variety of devices (including Quicktime on my Mac).

Next, to download:

youtube-dl -f 22 https://www.youtube.com/channel/UCz7bkEsygaEKpim0wu_JaUQ

(this URL is the CloudTechTV channel).

That command brought down all of the videos in the channel but I can also download individual episodes, for example:

youtube-dl -f 22 https://www.youtube.com/watch?v=ymKSGTR55LQ
youtube-dl
I can do something similar for other YouTube videos/channels (and even for some other video services) and build a library of videos to watch on my journeys, without needing to worry about an Internet connection.

“You need to work less”. Musings on finding the elusive work-life balance

“You need to work less”, said David Hughes (@davidhughes) as we were discussing why I carried a power supply with my Surface Pro. This was in response to my observation that the device will get me through the work day but not through travel at each end as well.

“Actually, you have a point”, I thought. You see, weekdays are pretty much devoted to work and pseudo-work (blogging, social media, keeping up to date with tech, etc.) – except for meals, sleep, the couple of hours a week spent exercising, and a bit of TV in the evening.

David commented that he reads – rather than working – on the train (I tweet and email but really should read more). And when I asked how he organises his day, he introduced me to ToDoIst. It seems that having a task list is one thing but having a task list that can work for you is something else.

Today was different. I knew I wanted to get a blog post out this morning, finish writing a white paper, and find time to break and meet with David in my favourite coffee shop. I’m terrible at getting up on working-from-home days (more typically working well into the evening instead) but I had managed to be at my desk by 7am and that meant that when I left the house mid-morning I’d already got half a day’s work in. For once, I’d managed some semblance of work-life balance. The afternoon was still pretty tough and I’m still working as we approach 7pm (my over-caffeinated state wasn’t good for writing!) but I met my objectives for the day.

Now I’ve added ToDoIst to my workflow I’m hoping to be more focused, to wrap up each day and set priorities for the next. I need to stop trying to squeeze as much as I can into an ever-more-frantic existence and to be ruthless with what can and can’t be achieved. Time will tell how successful I am, but it feels better already.

Short takes: super-sized Windows desktop icons; LastPass multifactor authentication; MTP on Windows 10 1607

A collection of short posts that don’t justify their own blog post!

Fixing super-sized Windows desktop icons

Mostly, I don’t get on with track pads – there’s just something about them that I find awkward and before I know it the cursor is shooting off somewhere that I don’t want it to be, icons are being resized, or something equally annoying.

I recently found myself in a situation where an errant trackpad response to my hot hands hovering over it whilst typing had left me with super-sized desktop icons but I couldn’t work out how/why. Luckily this Lifehacker article helped me put things right – a simple Ctrl + mouse scroll got my icons back to the size they should be…

LastPass Multifactor Authentication

For many years, I’ve used LastPass as my Password Manager. I don’t normally reuse passwords and have gradually been increasing the complexity of my passwords but these days I don’t know the password for the majority of the sites I visit – LastPass fills it in for me. The one weakness in all of this though is my master password for LastPass. It’s a long and secure passphrase but what if it was compromised? Well, now I have multifactor authentication enabled for LastPass too. It’s really simple to set up (just a couple of minutes) and options include Google Authenticator as well as LastPass’ own Authenticator app.

MTP not working on Windows 10 anniversary update (1607)

My son has an Elephone P9000 smartphone, running Android Marshmallow.  He was struggling to get it working with our family PC to import his pictures until I found this forum post that explains the process. It seems that, on the Windows 10 Anniversary Update (1607), the Media Transfer Protocol (MTP) driver needs to be manually installed:

  1. Go to C:\Windows\INF
  2. Type “wpdmtp.inf” in search bar provided to the right of the address bar in Windows.
  3. Once you found it, just right click on it and select install. It will take a very few seconds.
  4. Connect your device to the PC.

Bulk renaming digital photos for easier identification

Managing digital content can be a pain sometimes. Managing my own photos is bad enough (and I have applications like Adobe Lightroom to help with my Digital Asset Management) but when other family members want help whilst sorting through thousands of photos from multiple cameras to make a calendar or a yearbook it can get very messy.

For a long time now, I’ve used a Mac app called Renamer to bulk rename files – for example, batches of photos after importing them from my camera.  The exception to this is my iPhone pictures, which Dropbox rather usefully renames for me on import using the date and time, suffixing as necessary to deal with HDRs, etc. I only have a couple of gigabytes on Dropbox, so I move the renamed files to OneDrive (where I have over a terabyte of space…). The problem with this is that there needs to be enough space on Dropbox for the initial import – which means I have to use a particular PC which recognises my iPhone and remembers which photos have previously been imported. If I use another PC it will try and re-import them all (and fail due to a lack of available space)…

My wife has a different system. She also uses OneDrive for storage but has some files that have been renamed by Dropbox (to yyyy-mm-dd hh.mm.ss.jpg), some that have been renamed on import by something else (to yyyymmdd_hhmmsssss_iOS.jpg) and some that are just copied directly from the iPhone storage as IMGxxxx.jpg. My task? To sort this lot out!

Multiple images with the same time stamp

We decided that we liked the Dropbox name format. So that became the target. I used Renamer to rename files to Year-Month-Day Hour.Minutes.Seconds.jpg (based on EXIF file data) but the presence of HDR images etc. meant there were duplicates with the same time where a whole second wasn’t fine-grained enough. We needed those fractions of a second (or a system to handle duplicates) and Renamer wasn’t cutting it.

The fallback was to use the original filename as a tie-break. It’s not pretty, but it works – Year-Month-Day Hour.Minutes.Seconds (Filename).jpg gave my wife the date/time-based filename that she needed and the presence of the original filename was a minor annoyance. I saved that as a preset in Renamer so that when I need to do this again in a few months, I can!

Renaming digital photos in Renamer using a preset

No EXIF data

Then the files with no EXIF data (.MOVs and .PNGs) were renamed using a similar preset, this time using the modification date (probably less reliable than EXIF data but good enough if the files haven’t been edited).

Thousandths of seconds

Finally, the files with the odd format. Mostly these were dealt with in the same was as the IMGxxxx.jpg files but there were still some potential duplicates with the same EXIF timestamp. For these, I used progressive find and replace actions in Renamer to strip away all but the time a RegEx replacing ...... with nothing allowed me to remove all but the last three characters (I originally tried .{3}$ but that removed the 3 characters I actually wanted from the tail end that represent thousandths of seconds). One final rename using the EXIF data to Year-Month-Day Hour.Minutes.Seconds.Filename.jpg gave me yyyy-mm-dd hh.mm.sssss.jpg – which was close enough to the desired outcome and there were no more duplicates.

What’s the point? There must be a better way!

Now, after reading this, you’re probably asking “Why?” and that’s a good question. After all, Windows Explorer has the capability to provide image previews, the ability to sort by date, etc. but it’s not up to me to question why, I just need an answer to the end-user’s question!

Using Renamer is reliant on my Mac – there are options for Windows like NameExif and Stamp too. I haven’t used these but it appears they will have the same issues as Renamer when it comes to duplicate timestamps. There’s also a batch file option that handles duplicate timestamps but it doesn’t use the EXIF data.

Meanwhile, if anyone has a script that matches the Dropbox file rename functionality (including handling HDRs etc. which have identical timestamps), I’d be pleased to hear from you!

[Update 1 January 2017: These Python scripts look like they would fit the bill (thanks Tim Biller/@timbo_baggins) and James O’Neill/@jamesoneill reminded me of ExifTool, which I wrote about a few years ago]

Tools for troubleshooting Outlook autodiscover

In my post last week about Office 365 and proxy servers, I mentioned issues with Outlook autodiscover.  These were not exactly easy to troubleshoot, often with multiple subject matter experts looking from different angles (network, client applications, Exchange, firewalls, etc.). During the process, we used a few tools (as well as examining the traffic hitting the proxy servers) and I thought I’d highlight them here (if only for my own future reference):

Skype account unlocked with the help of the registration timestamp in the client database

A couple of weeks ago, I wrote a post about trying to get my Skype account unlocked and the arcane questions I was being asked by Skype Customer Support. I’d been meaning to write that post for a while… and I’m glad I finally got around to it because John Buston (a former colleague from my days at Fujitsu) pointed me at some advice in the Skype Community that proved to be very useful indeed

The Skype client (which I still had access to on one of my PCs) uses an SQL database called main.db, found in %appdata%\Skype.  The SQLite browser can be used to view the contents of this database and locate the registration_timestamp in the Accounts table.

Registration timestamp inside the Skype client database (main.db)

Take the number from the registration timestamp, multiply it by 60 and you’ll have the Unix time (the number of seconds since 1 Jan 1970, UTC) from when you registered for Skype. Put that number into a converter (like this one) and you’ll have the exact time when you registered for your Skype account.

Armed with that information (and some more that I could guess, like the email address I used to register for Skype; plus some more that I knew, like my personal details and some names from my Skype contact list) and I was finally able to get my account unlocked.

Thanks again John!

Unable to boot from USB flash drive on a Lenovo PC (to install Windows 10)

Yesterday, I wrote about not having to wait for Windows 10 to be advertised to my PCs and downloading the software directly instead. Unfortunately, things didn’t turn out to be quite that simple.

Overnight, both the Windows 8.1 PCs in our house decided that Windows 10 was ready (I clearly need to be more patient) but my 10 year-old son wanted to perform the upgrade (he’s a trainee geek) so, I waited for him to come home tonight before we tried it out. Because I’d already downloaded the media I thought I could skip bringing almost 3GB down over my ADSL line and boot from USB but we had a little trouble along the way…

I’d prepared a USB flash drive from the Windows 10 .ISO file using Rufus but our family PC (a Lenovo IdeaPad Flex 15) didn’t want to boot from it.

First of all, I had to work out the boot menu key combination (F12) but, even then, the boot menu only wanted to boot from the network, or from the local hard drive. I checked the BIOS (F1 at boot) and USB boot was enabled. Following Lenovo support article HT076906 (How to enter Setup Utility (F1) or Boot Menu (F12) on a Microsoft Windows 8/8.1 preloaded PC), I tried various combinations to reboot the machine (including Shift+Shutdown for a full shutdown and Shift+Restart for Windows boot options) but nothing was helping to boot from USB.

I tried recreating my media using different partition schemes for UEFI but that didn’t work either. So I followed Lenovo support article HT078684 (Cannot Boot From a USB Key – Idea Notebooks/Desktops) to:

  1. Run cmd.exe with Administrator privileges.
  2. Insert the target USB boot media device into an available USB port.
  3. Type:
    diskpart
    list disk (and make note of the disk number of the target USB drive)
    select disk n (where n is the target USB drive noted earlier)
    clean
    create partition primary
    format fs=fat32 quick
    active
    assign
    list volume
    exit
  4. Copy the entire contents of the Windows ISO onto the newly created UEFI boot media.

After this, I successfully restarted the PC, using F12 to access the boot menu and could boot from USB (i.e. the flash drive was available in the menu).

Unfortunately, after all that effort, Windows 10 wanted a product key to install (which I didn’t think I had on a PC that came with Windows pre-installed), so I went back to an in-place upgrade using Windows Update.

Installing Windows 10 via Windows Update

It’s been a few years since I regularly built PCs and it seems my desktop skills are a little rusty… since then, I’ve discovered a number of utilities for reading the product key of my Windows installation (which is also stored in the BIOS) – the tool I used is Windows Product Key Finder, available for download from CodePlex.