Learning to experiment

We’re all familiar with pilots – classic ways to prove solutions as part of traditional projects backed up by detailed business cases. But when it comes to digital transformation, with its inherently dynamic, fluid, fast-changing character, pilots are not always the best approach. You should consider a potentially more dynamic approach: experimentation. This article highlights why and how.

Experiments never fail, because you always learn

Pilots are usually about refining a proposed approach – they generally have an inherent commitment to continue and thus have the presumption of success baked in. So choosing not to progress with a pilot is seen as a failure. Experiments are fundamentally different, and it’s critical you take this truth on board at the outset. Try putting your R&D hat on; experiments are all about learning - why, for example, something does or doesn’t work. So you may well run multiple experiments at the same time, many of which will not be taken any further. They are all successes, because they all enable you to learn.

It is really important for everyone in your organisation to be clear that a successful result for an experiment is choosing to stop it once you have done sufficient work to learn why it’s not the right solution for your organisation. If stopping an experiment is viewed as failure then no one will be prepared to work on the really challenging ones, where the greatest potential value can be gained!

Obviously not everything needs experimentation and as an organisation, you might not want to take this approach but experiments can help you get further ultimately with digital.

Experiments need to be hypothesis driven

With experiments, it pays to be purposeful. It can be very tempting to experiment with lots of exciting new technologies without setting clear objectives. But just like in the lab, experiments should be based on a clear hypothesis. 

For example, you might set an expectation that running a new type of targeted campaign will deliver a certain business value, along with appropriate control groups to make sure the results can be compared. This ensures the experiment is practical, rather than a potentially wasteful exercise trying out some new tech. 

And remember, disproving a hypothesis is just as valuable as proving one, provided you take this learning and use it to refine the next set of experiments. The aim is to increase your learning and understanding through a series of hypothesis-driven experiments that have clear criteria and checkpoints.

Experiments are opportunities to be smart and seek simplicity

Hypothesis-driven experiments are your allies when it comes to not wasting money and avoiding over-complicating things. Lots of small, quick experiments will deliver more value and swifter learning than fewer, larger, longer ones. It’s about exploring how best to make the most business value from a specific tool or technology, rather than getting bogged down in trying to map out a long-term operating model or enterprise architecture. 

Another good way to increase the effectiveness of an experiment is to share risk with suppliers and partners – when everyone has skin in the game, minds are focused together and there’s a bias against wasting time, effort and money. Experiments, like start-ups, thrive on shared commitment and energy and the desire to get on and do good things, fast.

Experimenting is serious business

It can be tempting to dismiss experimenting as simply playing at the edges of the business – trying a few exciting new technologies which you may or may not end up adopting sometime, somehow. On the contrary, experimenting is serious business. One of our retail clients ran a continuous series of experiments in a handful of self-selected stores to test a range of operational changes – testing outcomes against hypotheses to assess benefits prior to making an investment case to roll them out across a nationwide estate.

Testing needs to become a fundamental part of how you embrace and capitalise on digital transformation. You need to make this clear to stakeholders, so they provide the right support throughout. You need to allocate sufficient resource, which could well mean deferring or putting to one side other projects. And you need to put your top talent on the experiments, and keep them there for the duration, to get better results, faster.

Make sure you pause to understand how to scale

It’s very tempting to move straight on from a successful experiment and roll that out as the solution as widely and quickly as possible. But this can lead to a lot of challenges down the line. Take a moment. Make sure you pause to understand what it really means to scale the solution. This could involve refining the future business operating model so it can run effectively without a large team around it; understanding how best to fully integrate it into the existing enterprise architecture; and refining the business case with a true understanding of future costs and benefits.