Product Testing: Learn Fast, Fail Small
Pansy Lee is currently the Director of Product and Design at Maple Leaf Sports and Entertainment (MLSE) where she works on bringing the best digital fan experiences to Toronto Maple Leafs, Raptors and TFC fans in and out of the arena. She has worked in the tech industry for over 15 years and her diverse interests have led to a diverse career. She has held marketing, sales, product and design roles at Microsoft, IBM, Deloitte and Intuit. Outside of work, Lee is passionate about diversity and inclusion and did her graduate research work on women in tech and male allies.
She recently spoke at Mozilla HQ in Toronto and discussed product testing, including how to learn fast and fail small, so you don’t fail slow and huge. After all, whether it’s a two-person startup or an 8000-person multinational corporation, lean experiments can help to evaluate an idea, test the riskiest assumptions and build products that customers actually want versus what they say they want.
Product testing is an essential part of building successful products. MLSE Director of Product and Design, Pansy Lee had a lot of wisdom on the subject to share with the attendees at her Speaker Series event in Toronto. As always, we suggest watching the entire video of her presentation above.
The highlights of Lee’s Speaker Series presentation are included below.
How to do Product Testing
The number one reason start-ups fails is that there was no market need for the product. 42 percent of start-ups fail for that reason!
- Define problem & ideal solution
- Build a hypothesis
- Identify riskiest assumptions
- Minimum success criteria
- Pick experiment
- Execute & collect data
- Evaluate – pivot or persevere
How to define a problem and a solution
A great product manager knows how to identify the problems they’re facing and the solutions to them.
- Who’re you solving for?
- What are they trying to do?
- What is holding them back?
- In a perfect world…
- 11-star experience (An Airbnb example of exceeding beyond expectations)
Tip: Talk to your customers! It will help you understand why a problem exists.
How to identify your riskiest assumptions
As you begin product testing it’s a good idea to take note of assumptions you’re making.
List your assumptions: In order for my idea to be successful, the following must be true…
Find your riskiest assumptions
- Has never been done before
- No alternative solution
Tip: Write it all down! Even if it feels trivial, you can eliminate the ones that have already been proven after.
How to Build a Hypothesis
What do you think will happen?
If we _____________ (our action),
we believe _________ (who)
will do _____________ (their action)
because ____________ (reason).
Minimum success criteria
How do you know if you should move forward or move on?
Cost: How much interest or conversions do you need to generate in order to cover the cost of creating this product?
Reward: Look for signs of real intent (email, phone # or payment) and find comparable conversion rates from old tests or industry standards.
Execute & collect data
- For how long should you run an experiment?
- How many people should see it?
- What is statistically significant?
It’s all about gaining confidence to make a decision!
10 types of Product Experiments
There are ten different types of product testing you should consider.
- Adword test: test interest using ads and tracking clicks
- Beta sign up: landing page that describes your solution and asks for sign ups
- Shadow button/coming soon: buttons that look like they lead to real products, but actually lead to a coming soon or beta sign up
- Buy buttons: purchase flows lead to a beta sign up
- 404 pages: if you only have time to develop a button and not a landing page, track the clicks on the button that lead to an error page
- A/B tests: take your incoming traffic and show them different variations of content and creative
- Explainer video: videos that walk through your product/service from clip art and drawing to 3D renderings
- Wizard of Oz: Real front end, fully manual back end
- Piecemeal MVP: Product/delivery is stitched together using out of the box software
- Fundraising: crowdfunding to see if there is enough demand to take prototypes to manufacturing