Rent The Runway Product Lead on How A/B Testing Goes Wrong

A/B Testing: Flat Tests Are Failed Tests

When a product manager on my team runs an A/B Test and it wins, I am ecstatic. When they run an A/B Test and it loses, I am interested. If they run a test and it’s flat, I throw my hands toward the sky in despair and wail “Where did we go wrong?” Flat A/B Tests are almost always failed A/B Tests.

If you’ve run an A/B test before, you know that you can get three results:

• Winner: It is statistically likely that your treatment has positively influenced the success metric (e.g. hooray! more customers made a purchase, average sessions lasted longer)
  
• Loser: It is statistically likely that your treatment has negatively influenced the success metric (e.g. oh no! fewer customers made a purchase, average session time diminished)
  
• Flat: We have no evidence that the thing you spent weeks building and launching and measuring made any difference whatsoever to the metrics you care most about

Two Types of Product Goals.

In short, flat A/B Tests are failures because you didn’t learn anything. 

In a company with a culture of innovation, consumer product managers ought to have two sets of goals they are accountable for: business goals, and learning goals. Hitting business goals are good for obvious reasons. Hitting learning goals helps us build the insights that let us hit next year’s business goals.

A winning A/B test often makes progress toward both business goals and learning goals. A losing A/B tests at least makes progress toward learning goals. A flat A/B test is (in most cases) wasted effort that has made progress toward neither set of goals.

Avoid Flat Tests In Three Steps.

Look back across your last ten A/B tests. How many were flat? I’m guessing for many of us, the answer is that the majority of A/B tests run end up having no measurable result. How can we avoid this wasted effort?

Every good test begins with a belief.

Most product managers know never to launch an A/B test without having a clearly articulated hypothesis. If you don’t know what you are trying to learn, you don’t stand a chance of learning it.

That belief should be core to the future of your business/product goals.

This is more likely to be the stumbling block. Sure, you can justify running an A/B test on button color based on the belief or hypothesis that, for example, red is a more urgent color so more people will click it. But is learning color urgency really core to your business’s future? Or are you spending precious engineering resources hyper-optimizing a conversion funnel into an experience that is still far from optimized? This depends on your business. 

If you run a straightforward e-commerce company, optimizing a conversion funnel likely is worth the effort. For most other early to mid-stage companies, this is probably not a worthy test unless the cost of running it is nearly free (including analytics resources). Even within funnel optimization, there are probably bigger, bolder, more worthy tests to run. Which brings us to…

The test should scare you.

I don’t like to launch an A/B test unless it makes at least one person feel scared. If everyone is sitting around the stakeholder table fairly unconcerned about the specifics of the test you want to run, you either have a very chill stakeholder group, or you’ve proposed a test with no real teeth to it. Rarely have I been able to move a metric in any measurable way through subtle tinkering. A/B Tests are not the right time to be conservative. You can add in subtlety later, after you get your results. 

“Do one thing every day that scares you,” is pretty standard bumper sticker wisdom, and something many of us product folks ought to take to heart. I challenge you stop tinkering with small iterative A/B Tests. Start identifying the unverified beliefs upon which you have built your business or product strategy, and try refuting them. In short: launch one test this quarter that scares you.

Click here for our latest blog posts.