When is agile not Agile? And why Waterfall is not always wrong!

In 2004 (when I started writing this blog), I was working for a company called conchango*. The developers talked a strange language – about Scrum and XP – and it was nothing to do with Rugby Union or Windows but it did have something to do with sprints…

That was my first encounter with Agile software development methodologies. Not being a coder, I haven’t done a huge amount of agile development, with the infrastructure projects I’ve been involved in generally being run using a traditional “waterfall” approach.

These days, things are different. There’s a huge push for Agile projects and the UK Government Digital Service’s Service Manual even says:

“You must use the agile approach to project management to build and run government digital services.

Agile methods encourage teams to build quickly, test what they’ve built and iterate their work based on regular feedback.”

Agile and government services: an introduction

There’s also a lot of confusion in the marketplace. Colleagues and clients alike are using the word “agile” in different ways. And there’s an undertone that agile is the one true way and waterfall is bad.

No!

Agile/agile/agility

Let’s start off by comparing uses of the word “agile” (in IT) and what they mean:

  1. Agile (big A) often relates to a methodology – for example APMG International’s AgilePM project management methodology or the AgileBA approach to business analysis – but really they have their roots in Agile software development, with the Agile Manifesto, written in 2001.
  2. When we talk about being agile (small a), it’s a mindset: the approach taken. Literally, being able to adapt to change and to move quickly. We might use Agile (big A) approaches to help increase our agility (small a).
  3. Agility is about reaction to change. Many business want to be agile. That doesn’t mean they only run projects with Agile approaches. It means they want the ability to flex and change in line with business requirements.
  4. And then there’s the UK public sector. Specifically Police, who for some reason refer to what the rest of us consider to be remote/mobile working as agile working (as shown in this Agile Working Policy from West Yorkshire Police). That’s just an anomaly.

So that’s Agile/agile/agility sorted then. There are Agile frameworks/methodologies/approaches to delivering outcomes in a more agile manner, to increase organisational agility.

Agile=good, waterfall=bad?

Now waterfall. If Agile, is the one true way, waterfall must be old hat and avoided at all costs, right?

Not at all.

Agile projects work well for quickly creating a minimum viable product (MVP) and iterating development – for example as a series of sprints. They are great when there is a known problem but the requirements are less clear. The solution can evolve in line with the definition of the requirements. The requirements may change as the solution develops: respond to market changes; adapt to new requirements; fail fast.

But some projects are less defined. In a 2018 blog post, Matt Ballantine (@ballantine70) referred to unknown problems with unknown solutions as tinkering. That seems fair – if you don’t know what the issue is, then you can’t have a solution!

Similarly, unknown problems with a known solution. That’s nonsense. Or “WTF?” as Matt so succinctly puts it in his 2×2 diagram:

Matt Ballantine's 2x2 diagram of which path to take, including Agile and Waterfall approaches
Matt Ballantine’s 2×2 for which path to take, including Agile and Waterfall approaches (used with kind permission).

You’ll see though, that there is a place for waterfall project management. Waterfall works when there is a known problem and a known solution. Instead of constantly iterating towards an end, work out the steps to go straight there. It will almost certainly be more efficient. Waterfall projects are based on the golden triangle of time/cost/quality (which together define scope). A known deliverable (scope) bounded by how fast/cheap/good you want it to be – and there’s always a trade-off.

So there we have it. Agile is not a silver bullet and there is still a place for waterfall projects.

What to use, when?

In my line of work, Cloud Transformation might appear to use a combination of Agile and Waterfall approaches. We might create a virtual datacentre in Azure or AWS and take an iterative approach to migrating workloads but that’s still really just Waterfall with incremental delivery – even if a Kanban approach is used to inject some urgency! Similarly migrating batches of mailboxes to the cloud is just iteration, as is a programme that’s adopting Office 365 workloads one by one. An Agile approach comes into its own when we think about Business Transformation, or Digital Transformation, where we can define an MVP and then use sprints to iterate development of a set of new business processes or the digital tools to deliver those processes in a new way.

Further reading

For a clear definition of Cloud, Business and Digital Transformation, see my blog post from last year: “Digital Transformation – it’s not about the Technology“.

* The small c is not a typo – that was the branding!

What is DevOps? And is your organisation ready?

Like cloud a few years ago and then big data, DevOps is one of the buzzwords of the moment. So what does it actually mean? And is there more to it than hype?

There are many definitions but most people will agree that, at its core, DevOps is about closer working between development and operations teams (and for infrastructure projects read engineering and operations teams). Part of DevOps is avoiding the “chuck it over the fence” mentality that can exist in some places (often accompanied with a “not invented here” response). But there is another side too – by increasing co-operation between those who develop or implement technology and those who need to operate it, there are opportunities for improved agility within the organisation.

DevOps is as much (probably more) about people and process than technology, but the diagram below illustrates the interaction between teams at different stages in the lifecycle:

  • Businesses need change in order to drive improvements and respond to customer demands.
  • Development teams implement change.
  • Meanwhile, operations teams strive to maintain a stable environment.

Just as agile methodologies sit between the business and developers, driving out requirements that are followed by frequent releases of updated code with new functionality, DevOps is the bridge between the development and operations teams.

DevOps in context

This leads to concepts such as infrastructure as code (implementing virtual infrastructure in a repeatable manner using declarative templates), configuration automation (perhaps with desired state configuration) and automation testing. Indeed, DevOps is highly dependant on automation: automating code testing, automating workflows, automating infrastructure – Automating Everything!

Configuring, managing and deploying resources (for example into the cloud) is improved with DevOps processes such as continuous integration (CI). No doubt some will argue that CI has existed for a lot longer than the DevOps term and that is true – just as virtualisation pre-dates infrastructure-as-a-service!

The CI process is a cycle of integrating code check-ins with testing and feedback mechanisms to improve the quality of the code:

Continuous integration example

In the example above, each new check-in to the version control system results in an automated trigger to carry out build and unit tests. These will either pass or fail, with corresponding feedback. When the tests are successful, another trigger fires to start automated acceptance tests, again with feedback on the success or failure of those tests. Eventually, the code passes the automated tests and is approved for user acceptance testing, before ultimately being released.

Continuous integration works hand in hand with continuous delivery and continuous deployment to ensure that development teams are continuously dropping new code but in line with the Release Management processes that the operations teams require in order to maintain their service.

Continuous delivery allows new versions of software to be deployed to any environment (e.g. test, staging, production) on demand. Continuous delivery is similar to continuous integration but can also feed business logic tests. Continuous deployment takes this further with every check-in that passes all tests ultimately ending up with a production release – the fastest route from code to release.

No one tool is used to implement DevOps – DevOps is more about a cultural shift than it is about technology – but there are many tools that can assist with implementing DevOps processes. Examples include Chef, Puppet (configuration management) and Jenkins (continuous integration). Integrated development environments (such as Visual Studio and Eclipse) also play a part, as do source control systems like Visual Studio Team Services and Git/GitHub.

DevOps is fuzzy too. Once we start to talk about software-defined infrastructure we start to look at orchestration tools (e.g. Mesosphere, Docker Swarm) and containerisation (e.g. Docker, Azure Container Service, Amazon ECS). And then there’s monitoring – either with tools built into the platform (e.g. Visual Studio Application Insights) or third party tools (like those from NewRelic and AppDynamics).

So DevOps is more than a buzzword. It’s a movement, that’s bringing with it a whole stack of processes and tools to help drive towards a more agile environment. IT that can support business change. But DevOps needs a change of mindset and for me the big question is “is your organisation ready for DevOps?”.

Further reading/viewing