How do PMs know what products to build? Or what problems to solve? Of course all good PMs know about the importance of listening to the customer with empathy. But sometimes customers say they want one thing when what would really help them is something else entirely. How can PMs avoid guesswork when building and optimizing products? In a word: experiment.

In this podcast episode, the third in the 2022 Product Awards series, Optimizely VP of Product Thilo Richter speaks with Wayfair Product Leader Nacho Andrade about why PMs should do more exploratory testing, why actually understanding your customer’s problem is the most important thing, and why a great PM might have a few things in common with a scientist.

Subscribe to the Product Talk podcast on Spotify and Apple Podcasts, and make sure you never miss a single product conversation with leading product executives. New episodes go live every week. Episodes in the Product Awards Series drop on Fridays.



On testing as exploration

One question all product managers need to consider is: Are we doing enough experimentation to answer the right questions and make sure we’re making the right investments? In a sense, a main purpose of experimentation is to seek failure, because that can save the company a lot of money. A failed experiment can also produce learnings that can then be optimized or otherwise lead to positive changes.

Thilo says, “I think what most people focus on is they release something, and then they start experimenting on it. So that’s where most conversion rate optimization lives today. But I think what not enough companies are doing is before actually starting development, to start experimenting. You have that concept of a painted door test. You release a feature, but it’s not really a feature. It’s just a UX mockup. 

“As an example, at one point we were thinking about offering more AI-based personalization, or personalization capabilities. And we didn’t know which direction we were going to take. So we actually implemented a menu item in our product that said different things, depending on what kind of user you were, when you clicked on it. It showed us, okay, you were interested in the way that we phrased it in the menu item. And then it had a pop up which gave you more information on what we were thinking about. You had the option to opt in to more information or research that we were doing. So we were pretending to have that feature in our UI, right? And it gave us some information about in what way we should phrase it, or what way we should develop it to get the most interest.

“I think these kinds of tests, not enough people do, and they can be really valuable because they give you valuable information before you actually put any development effort in. So it’s more like explorative testing.

“If you look at product development, the later you fix issues, the more expensive that becomes. So the more you can clarify the direction that you should take upfront, the cheaper it’s going to be to actually deliver something. And the more likely it’s going to have a positive impact as well.”

On understanding the problem

In the old days of software development, the name of the game was Build What The Customer Wants. They would ask for something, make an order, and developers would place the order. Now, it may be that the quickest way to ruin your product is to do everything your customers want! Why? Mainly because users are not always aware of what they need. They are not always aware of their behavior. But they are really aware about their problem, and it’s up to PMs to uncover the underlying issues and find the right solutions. 

Thilo says, “Probably because I come from the consulting and customer-facing side, I see way too often that people end their investigation – in terms of what to build next – at how many feature requests we have for this thing. Or things like, hey, we have 10 customers that are worth this much, requesting this. And a lot of the product management work that I see is still very, very much feature-based. 

“As a consultant, you learn when you get a request from a customer, to really dig in and truly try to understand what they’re actually trying to do and what their problem is. And that requires a lot of work oftentimes, because customers come to you and they just request a feature. And you will have to overcome your urge to put it on the roadmap, or just note it down. But really dig in, and ask, ‘Okay, cool, tell me more about why you’re trying to do this.’ And half of the time, actually, their problem is already solved. But they should just solve it in a very different way. And the other half of the time, you discover that their feature request is actually not the right one, and you should rather focus on a specific problem that should actually be solved differently. 

“And so if there’s one thing that I would urge people to do more is really try to understand the problems behind the feature requests. Try to understand your customers, what they’re trying to do, what their business model even is. Why do they have this problem? Go even further, right? It’s always because they’re this kind of business, and this is what they’re trying to achieve, and this is what their CPO and CEO cares about. And if you truly understand that, you can build much more powerful products, I feel.”

On what makes a great PM

According to Thilo, a great PM has an experimental attitude, combining curiosity with truth seeking. PMs need to be curious about how the world works, but in order to answer their questions they experiment and find the fundamental rules underneath to ensure good learning is happening. 

He says, “I think people need to be curious, they need to be learners, and they need to be truth seekers. So they need to have an urge to really try to understand, to get to the truth of things. And be humble in the process. 

“You mentioned to me earlier your pet peeve is about people falling in love with their own ideas. You have to be humble and say, well, actually, maybe it’s not the best idea. I’m trying to find the truth of what actually the problem is, the truth of what actually the right way to solve this is. So they need to be truth seekers. 

“I also mentioned that we focus on building teams that work cross-functionally together to solve problems. So you need to be a great collaborator, great facilitator. So you need to be able to impart your knowledge and give enough context to a team to actually solve a problem without guiding them too much in one direction. Because if you’re solving new problems with new products, you need a lot of creativity. And so you need to give people the space to do that. So being a good facilitator and, and somebody that can collaborate well really helps. 

“And then having that understanding that your product is more holistic than just the user interface. It’s the whole experience, end to end. From the first time that somebody comes in contact with it, or even learns about it. To when they potentially turn or stop using the product. That whole experience, everything they touch, that’s part of the product. And so having that holistic view, I think really helps as well.”

About the speaker
Thilo Richter Optimizely, VP of Product - Experimentation and Feature Flagging Contributor
About the host
Nacho Andrade Ubiety Technologies, Inc., Chief Product Officer

Big idea product leader specializing in the space between 0 to 1, digital transformation, and innovation.

Provide your rating for this post
If you liked this post, please use the buttons to the left to share it with a friend or post it on social media. Thank you!

Leave a Reply

Read more

Product Awards Series: Sprig Product Lead On Researching The Why

Learn how Product Managers can leverage end user research at scale to make really good product decisions.

Product Awards Series: Heap VP of Product On Building Solutions People Are Willing To Buy

Heap VP of Product Rachel Obstler gives insights into how Heap built analytics tooling customers were willing to buy.

Dow Jones Product VP on Experimentation, Subscription, and Personalization

Experimentation can be developed as a powerful storytelling and product tool to make accurate, data-driving decisions that can expand their product’s reach.

Sign-in / Register for Free

Don’t be left behind in your career. Join a growing community of over 500K Product professionals committed to building great products. Register for FREE today and get access to :

  • All eBooks
  • All Infographics
  • Product Award resources
  • Search for other members

Coming soon for members only: personalized content, engagement, and networking.