Designing a private cloud infrastructure

A couple of months ago, Facebook released a whole load of information about its servers and datacentres in a programme it calls the Open Compute Project. At around about the same time, I was sitting in a presentation at Microsoft, where I was introduced to some of the concepts behind their datacentres.  These are not small operations – Facebook’s platform currently serves around 600 million users and Microsoft’s various cloud properties account for a good chunk of the Internet, with the Windows Azure appliance concept under development for partners including Dell, HP, Fujitsu and eBay.

It’s been a few years since I was involved in any datacentre operations and it’s interesting to hear how times have changed. Whereas I knew about redundant uninterruptible power sources and rack-optimised servers, the model is now about containers of redundant servers and the unit of scale has shifted.  An appliance used to be a 1U (pizza box) server with a dedicated purpose but these days it’s a shipping container full of equipment!

There’s also been a shift from keeping the lights on at all costs, towards efficiency. Hardly surprising, given that the IT industry now accounts for around 3% of the world’s carbon emissions and we need to reduce the environmental impact.  Google’s datacentre design best practices are all concerned with efficiency: measuring power usage effectiveness; measuring managing airflow; running warmer datacentres; using “free” cooling; and optimising power distribution.

So how do Microsoft (and, presumably others like Amazon too) design their datacentres? And how can we learn from them when developing our own private cloud operations?

Some of the fundamental principles include:

  1. Perception of infinite capacity.
  2. Perception of continuous availability.
  3. Drive predictability.
  4. Taking a service provider approach to delivering infrastructure.
  5. Resilience over redundancy mindset.
  6. Minimising human involvement.
  7. Optimising resource usage.
  8. Incentivising the desired resource consumption behaviour.

In addition, the following concepts need to be adopted to support the fundamental principles:

  • Cost transparency.
  • Homogenisation of physical infrastructure (aggressive standardisation).
  • Pooling compute resource.
  • Fabric management.
  • Consumption-based pricing.
  • Virtualised infrastructure.
  • Service classification.
  • Holistic approach to availability.
  • Computer resource decay.
  • Elastic infrastructure.
  • Partitioning of shared services.

In short, provisioning the private cloud is about taking the same architectural patterns that Microsoft, Amazon, et al use for the public cloud and implementing them inside your own data centre(s). Thinking service, not server to develop an internal infrastructure as a service (IaaS) proposition.

I won’t expand on all of the concepts here (many are self-explanitory), but some of the key ones are:

  • Create a fabric with resource pools of compute, storage and network, aggregated into logical building blocks.
  • Introduced predictability by defining units of scale and planning activity based on predictable actions (e.g. certain rates of growth).
  • Design across fault domains – understand what tends to fail first (e.g. the power in a rack) and make sure that services span these fault domains.
  • Plan upgrade domains (think about how to upgrade services and move between versions so service levels can be maintained as new infrastructure is rolled out).
  • Consider resource decay – what happens when things break?  Think about component failure in terms of service delivery and design for that. In the same way that a hard disk has a number of spare sectors that are used when others are marked bad (and eventually too many fail, so the disk is replaced), take a unit of infrastructure and leave faulty components in place (but disabled) until a threshold is crossed, after which the unit is considered faulty and is replaced or refurbished.

A smaller company, with a small datacentre may still think in terms of server components – larger organisations may be dealing with shipping containers.  Regardless of the size of the operation, the key to success is thinking in terms of services, not servers; and designing public cloud principles into private cloud implementations.

Office 365 message filtering (and a horrible little bug that leaves email addresses exposed…)

One of my concerns with my recent switch from Google Apps Mail to Microsoft Office 365 was about spam email. You see, I get none.  Well, when I say I get none, I get plenty but it’s all trapped for me. With no effort on my part. Only a handful of missed spam messages in the last 2 or 3 years and almost as few false positives too.

I’ve had the same email address for about 12 years now (I think), and it’s been used all over the web. Some of my friends are more particular though – and, perhaps understandably, were annoyed when I accidentally emailed around 40 people with e-mail addresses visible in the To: field today. Except that I hadn’t intended to.

I think I’ve found a bug in Office 365’s Outlook Web App (at least, I hope it’s not closed as “by design”, assuming I find out how to file a bug report). If I send to a distribution group, it automatically expands the addresses and displays them to all recipients. That’s bad.

The annoying thing is that, previously, I had been BCCing the recipients. I have a feeling that at least one organisation was rejecting my mail because there was nothing in the To: field (although it didn’t like Google’s propensity to send mail from one domain “on behalf of” another address either), so I thought I’d use a list instead and the recipients would see the list name, rather than the actual email addresses. Thankfully it was only sent to my closest freinds and family (although that’s not really the point).

So, back to spam and Office 365 – does it live up to my previous experience with Google Apps Mail? Actually, yes I think it does. I’ve had to teach it a couple of safe senders and block a couple of others, but it really was just a handful and it’s settled down nicely.

All of Microsoft’s cloud-based e-mail services use Forefront Online Protection for Exchange. Enterprise administrators have some additional functionality (adapting SCL thresholds, etc.) but things seem to be working pretty well on my small business account too. Digging around in the various servers that the mail passes through sees hosts at and – Frontbridge was an aquisition that has become part of Exchange Hosted Services (and it started out as Bigfish Communications) – so the technology is established, and another Microsoft property (Hotmail) is a pretty good test bed to find and filter the world’s spam.

Should we gamify the workplace?

Gamification is certainly one of this year’s buzzwords and the science of gamification (i.e. the use of game mechanics/dynamics to drive game-like engagement and actions in non-game environments) is a topic of great interest to me at the moment.

But how can we use gamification in the workplace? And should we even try?

Whilst it’s true that there is a moral hazard to avoid, the trick to successful gamification is making sure it doesn’t feel like the target is being played. Let’s take an example that well established in the workplace: flexitime. The motivation is for an employee to accrue enough additional work time to “earn” a day off; ability is controlled by the rules that govern the flexitime scheme; and the trigger is the point where sufficient “credit” is available to take some additional leave!

I have to admit that flexitime is not one of my benefits at Fujitsu but for those businesses that have such as scheme, it has benefits in terms of employee flexibility and morale. And there are other examples where we can re-engineer our business processes and introduce some elements of gamification.

Take, for example, the idea of a results-oriented work environment. What if, instead of being paid a salary, or an hourly rate, employees were given the opportunity to pick and choose their work and remunerated accordingly? Critics may see such an approach as a return to factory processes and piecework. Others may see an opportunity to free themselves from their 9 to 5 (or 8 to 6, or 6 to 8 work routine) and work in a more flexible manner. My background is as a solutions architect. What if projects were to be crowdsourced so that a pool or architects to pick tasks from a list of activities? Different values could be attributed depending on the difficulty or time sensitivity of the task, with all architects having to achieve a minimum number of credits (but the ability to earn more if they so desired). I’m sure there many human resources issues to overcome but I can see this being the “normal” way to work in future.

Problems come when the gamification feels controlling and is associated with “Big Brother”. We have to accept that one size does not fit all – and there is a risk that employees may feel disconnected, or that they are being patronised. Most people are smart and can work out how to “game” the system – so the game mechanics need to be honed to balance motivation and ability, and to trigger employees at the appropriate times.

If we gamify the workplace though, it seems there’s a risk of destroying some of the other elements of successful collaboration. The workplace is far more than just a literal place to work. There are social and environmental aspects to consider too. If we create an internal market of competing architects what’s the difference between that and a group of independant contractors working on a project? At what point do people stop working for a common purpose (the company’s mission) and start working for their own goals? People can’t be our most important asset when we don’t have any people any more!

It may be that gamification is not appropriate for mainstream activities but can be used for those on the periphery – those that are considered extra-curricular. For example, whilst I’d like everyone to want to contribute to our Open Innovation Community, the reality is that people can opt in or out. What if we were able to gamify the innovation process with a system of rewards?

This post doesn’t really provide any answers – it does pose some questions though. How would you feel about the gamification of your work environment? And would you consider there are significant advantages to be gained, or is the risk of disruption just too great?

[This post originally appeared on the Fujitsu UK and Ireland CTO Blog and was written with assistance from Ian Mitchell and Vin Hughes.]