Agile and software delivery

Making software is hard and it’s been the case ever since the IT industry came about after WWII.

It comes with no surprise that people have continuously tried to come up with sets of rules and processes that would reduce the traditional risk associated with software development projects and increase speed of delivery and predictability.

In the 1980’s the industry realized that the number one cause of delays and uncertainty was the lack of clear requirements. And so the famous and dreaded waterfall method came about.

It worked relatively well in slow and steady environments where delivery cycles required yearly releases.

Back then and up until the early 2000s, custom software development was relegated to dedicated software companies - called software houses and any other type of business would just purchase off the shelf solutions or contract 3rd party consultancies for customization if needed.

For a long time making software was considered a niche endeavour.

A major shift

Two major radical changes shifted the landscape though: first came the advent of cloud computing (from 2006) and then the Global Financial Crisis (in 2009).

All of a sudden - and almost at the same time - these two phenomena meant that

1. software hosting became increasingly cheaper and accessible and
2. an increasing number of companies found their business model disrupted and decided to start innovating - either to improve the efficiency of their internal operations or to create brand new revenue sources.

By 2011 software development was already eating the world using an expression made famous by Marc Andreessen’s article .

In the innovation game, time is of the essence and the old model built upon meticulous requirements gathering and slow release cycles became increasingly limiting.

In this landscape, Agile - with its focus on continuous iterations, learning and delivery - became in less than 10 years the software delivery method of choice - promising to break away from the delays and the analysis paralysis brought about by the waterfall approach, replacing them with on time and waste free delivery.

Given its simplicity and the heavily publicised early success stories - nowadays every company that wants to do software development - from the most innovative to the most conservative - has embraced some form of agile.

Yet agile still gets a lot of mixed reviews - praised by some and despised by others.

The basics

To understand why there is so much controversy, let’s go through the basics first.

Agile at its core is about small, incremental releases and constant iteration to provide a constant flux of new features and value to the user.

It is translated into practice using rather prescriptive methodologies - scrum, kanban, etc. - designed to try and capture this essence in a structured and repeatable way, intermixing the principles of the Agile manifesto with some well known delivery best practices.

In a nutshell they revolve around 4 new types of engagements called ceremonies that an agile team is encouraged to have on a regular basis:

1. Standups - we all have at least heard of them - quick 10 min daily meetings mostly focused on surfacing issues and priorities.
2. Refinement/grooming sessions - where each new feature is discussed, understood and estimated before it’s accepted by the team to be delivered.
3. Retrospectives - where the team reflects freely on what worked and what didn’t during the latest iteration and provide feedback for the upper management.
4. Showcases - where the team shares progress in the form of a demo on a regular basis to all relevant stakeholders.

There is plenty of information available online about the details of these meetings so I won’t spend too much time delving into those.

The key element is that these meetings exist to bring measurable value to the delivery:

1. visibility and predictability of the process,
2. focus on the most important priorities - that can’t shift every day,
3. continuous improvement of the team dynamics
4. end to end delivery (no more 97% done tickets) and
5. team cross-functionality.

All valuable advantages you may say. So where is the catch?

The catch

To start, it’s easy to see how all these ceremonies come at a cost in terms of time: on average 20 to 30% of the time in a fully functional agile team is spent in meetings, which is a non trivial amount if translated in dollar value.

Agile practitioners always highlight that the alternative is misalignment which is significantly more expensive to a business. Yet nobody likes meetings and Agile advocates for a lot of them.

But the less visible issue has to do with what is being delivered.

With agile the emphasis is on shipping features on a regular basis, collecting feedback and iterating continuously.

Yet in my experience many companies that do just that end up missing the mark over and over, never getting closer to providing real customer value.

To understand why this happens, let’s take a step back and consider the very often overlooked difference between efficiency and effectiveness.

Efficiency vs Effectiveness

Efficiency - we are all familiar with this term - is doing the same thing with less cost over time. By its own nature it is easy to measure is about small changes, tinkering.

Effectiveness instead is about big changes - think 10x improvements and is achieved mostly through radically new approaches or new products.

So if efficiency is mostly a low hanging fruit that can be cashed in upfront in terms of reduced costs and easily shines in the C-suite reports, effectiveness is in essence about innovation and disruption.

The most common type of innovation is through technological breakthroughs but these come with elevated risks and with potential paybacks only in the future.

If we look at the most common Agile methodologies and their ceremonies - it’s easy to see how most of the advantages sit instead in the efficiency basket: visibility, predictability, priorities, continuous improvement, team cross-functionality and independence are just good efficiency practices.

So an agile team generally delivers software more efficiently compared to teams with different or no processes - yet efficiency in itself is rarely a game changer.

Disruption

Disruption comes from effectiveness.

Highly effective yet inefficient companies still succeed.

Effectiveness is what a startup should strive for.

I have seen countless agile teams succeed at delivering while they fail spectacularly at changing the bottom line of the business because they focus on delivering features that add little to no value to the final user.

From personal experience a lot of agile teams lack a clear vision or strategy. This has nothing to do with Agile per se yet it explains why many developers get to hate it: they get asked to build features during one sprint that get rolled back the next, yet they are forced to participate in meetings, day in and day out.

When Agile is not adding value, most Agile practitioners suggest doing more of it - more meetings, more ceremonies, creating a vicious cycle.

The same agile champions will then point out that continuous iteration is the key to success in software delivery and failure needs to be embraced as part of the process.

This is the biggest blind spot of the Lean Startup approach and its Build-Measure-Learn feedback loop concept.

Getting feedback on a product and iterating is generally a convergent process based on what you know already.

Some product managers believe they somehow possess an innate talent for building stuff that people love and are happy to pay for.

The only problem is time.

As many startups that die prematurely can attest, most companies run out of money and CEOs of patience well before they actually get to experience any meaningful knowledge and success. And this doesn’t stem from the “lack of empowerment” - a recurring theme on linkedin articles or posts on the subject - rather by the lack of upfront research.

Conclusion

When the roadmap is inspired by good research and the delivery is the problem Agile is the right medicine. If a company building software has achieved market fitness and needs to deliver features that are really needed, the time and energy to solve the getting in sync problem through Agile ceremonies will be well spent to reduce ambiguity and waste.

In all other cases Agile is not a silver bullet. It needs to be complemented by valuable user research and discovery activities like lean canvas or design sprints.

When there is no market fitness, there is no point in chasing efficiency to reduce costs, as the investment in doing so will simply not pay for itself - like in early stage startups with very small teams: communication and visibility issues can be kept in cheque informally without the need of a prescriptive process.

If there is no clear vision nor enough market and user research to validate what is being delivered, Agile ceremonies - while providing efficiency gains - add very little value to the bottom line of a business, take considerable time off development and risk losing buy-in from the development team.