September 20, 2018
"What is Test and Learn (and why should I care?)"
By: Matty Wishnow

Test and Learn. Culture of Experimentation. It’s likely that you’ve heard these catch phrases. Perhaps you’ve used them or thrown them out there casually, assuming that they are universally grasped or easily recognized. Alternately, perhaps you think these terms are overused and meaningless.

The truth we’ve seen at Accenture Interactive, is somewhere in between. For sure, the terms are overused. But that does not mean they are not very powerful and useful concepts. Over the years clients have regularly asked us, “How would you describe what ‘test and learn’ means?” and “Why bother doing it?” and “How do you know if you have a ‘test and learn’ culture?”

We’ve thought enough about these very valid questions to frame up some answers and help the dialogue graduate from platitudes and vagaries to something that feels real and measurable. Here it goes…

What is Test and Learn?

Test and Learn is an organizational framework and set of processes wherein all changes and investments are understood to be hypotheses. These hypotheses, or possible solutions for an important problem, are tested via experiments.

Based on the findings from the experiments, investments are made in accordance with data and evidence as opposed to opinion. The process is then repeated and improved upon continuously as the organization gets more data.

Test and Learn can apply to all things digital, from advertising to merchandising to design changes to new features and technology.

In reality, Test and Learn is not limited to digital investments. It applies to all business and/or product investments.

Why Test and Learn?

Every change we make every day to our digital business is an experiment–the outcome is unknown.

Understanding this, Test and Learn is based on the assumptions that we will be more likely to succeed if we:

(a) Know what problems our changes (experiments) are solving.

(b) Know whether those are the biggest problems to solve for our business and customers.

(c) Have confident data to indicate whether our solutions worked (versus a control).

Our approach to Test and Learn is done through our PSM (Problem Solution Mapping) methodology. Test and Learn is also the core tenet of companies like Google, Amazon and GE, and for good reason (more on that below).

Some of the benefits a Test and Learn strategy yields include:

1: Lower opportunity cost and cost to serve due to the reduction of changes.

Through Test and Learn, we can lower the opportunity cost and cost to serve through the reduction of changes that don’t actually work, or worse, create incrementally new problems. To put it another way, Test and Learn allows us to avoid or eliminate prolonged failures faster.

This benefit could be measured from the rate of incremental product improvement (conversion rate, RPV, member satisfaction, NPS, etc.) compared to the rate of incremental investment in said product.

2: Greater speed to improvement and optimization.

Through Test and Learn we quickly gather data on our digital investments without spending months aiming for perfection. When these experiments pay off, we learn quickly and then expand them from a sample set in our tests to our entire audience to improve financial impact and user experience quickly.

This benefit could be measured by the rate of net beneficial launches or changes or enhancements in a given period as compared to the same, previous periods.

3: A leaner and more evidence-based organization.

Through Test and Learn we minimize debate and the internal politics around investments by making it clear that confident data and evidence are the basis for decisions and not opinion or charisma.

This benefit could be measured by the relative amount of time spent in committees and the speed of approval times in comparison to the amount of time researching problems and testing hypotheses.

Test and Learn in action

Amazon’s culture of experimentation

It’s no secret that experimentation is ingrained in Amazon’s culture. Jeff Bezos, Amazon’s CEO, refers to it extensively in his 2015 Letter to Shareholders.

“To invent you have to experiment, and if you know in advance that it’s going to work, it’s not an experiment. Most large organizations embrace the idea of invention, but are not willing to suffer the string of failed experiments necessary to get there,” he explains. “Outsized returns often come from betting against conventional wisdom, and conventional wisdom is usually right. Given a ten percent chance of a 100 times payoff, you should take that bet every time. But you’re still going to be wrong nine times out of ten.”

Bezos further describes Amazon’s commitment to a Test and Learn culture by using a metaphor near to my heart: baseball.

Bezos explains, “We all know that if you swing for the fences, you’re going to strike out a lot, but you’re also going to hit some home runs. The difference between baseball and business, however, is that baseball has a truncated outcome distribution. When you swing, no matter how well you connect with the ball, the most runs you can get is four. In business, every once in a while, when you step up to the plate, you can score 1,000 runs. This long-tailed distribution of returns is why it’s important to be bold.”

Test and Learn at Google

You’re likely familiar with Google AdWords, Google’s proprietary advertising service. AdWords is a wildly successful Google product, to the tune of over $52 billion in 2015 alone. One thing you may not know: When AdWords was first released in 1999, it wasn’t very successful. Google overcame this through a—you guessed it—Test and Learn process.

“Every year we run tens of thousands of search and ads quality experiments, and over the past year we’ve launched over a dozen new formats. Some products we update every day,” Susan Wojcicki, Google’s form SVP of AdWords explained in her article about Google’s Eight Pillars of Innovation.

She continues, “Our iterative process often teaches us invaluable lessons. Watching users ‘in the wild’ as they use our products is the best way to find out what works, then we can act on that feedback. It’s much better to learn these things early and be able to respond than to go too far down the wrong path.”

Putting Test and Learn into practice

Unsurprisingly, we have entire Practices and Product sets at Accenture Interactive devoted to Accelerating Test and Learn culture and process within our clients.

Interested in putting Test and Learn to work for your organization? Accenture Interactive can help.

Popular Tags

    More blogs on this topic