How do PMs know what products to build? Or what problems to solve? Of course all good PMs know about the importance of listening to the customer with empathy. But sometimes customers say they want one thing when what would really help them is something else entirely. How can PMs avoid guesswork when building and optimizing products? In a word: experiment.
In this podcast episode, the third in the 2022 Product Awards series, Optimizely VP of Product Thilo Richter speaks with Wayfair Product Leader Nacho Andrade about why PMs should do more exploratory testing, why actually understanding your customer’s problem is the most important thing, and why a great PM might have a few things in common with a scientist.
Subscribe to the Product Talk podcast on Spotify and Apple Podcasts, and make sure you never miss a single product conversation with leading product executives. New episodes go live every week. Episodes in the Product Awards Series drop on Fridays.
On testing as exploration
One question all product managers need to consider is: Are we doing enough experimentation to answer the right questions and make sure we’re making the right investments? In a sense, a main purpose of experimentation is to seek failure, because that can save the company a lot of money. A failed experiment can also produce learnings that can then be optimized or otherwise lead to positive changes.
Thilo says, “I think what most people focus on is they release something, and then they start experimenting on it. So that’s where most conversion rate optimization lives today. But I think what not enough companies are doing is before actually starting development, to start experimenting. You have that concept of a painted door test. You release a feature, but it’s not really a feature. It’s just a UX mockup.
“As an example, at one point we were thinking about offering more AI-based personalization, or personalization capabilities. And we didn’t know which direction we were going to take. So we actually implemented a menu item in our product that said different things, depending on what kind of user you were, when you clicked on it. It showed us, okay, you were interested in the way that we phrased it in the menu item. And then it had a pop up which gave you more information on what we were thinking about. You had the option to opt in to more information or research that we were doing. So we were pretending to have that feature in our UI, right? And it gave us some information about in what way we should phrase it, or what way we should develop it to get the most interest.
“I think these kinds of tests, not enough people do, and they can be really valuable because they give you valuable information before you actually put any development effort in. So it’s more like explorative testing.
“If you look at product development, the later you fix issues, the more expensive that becomes. So the more you can clarify the direction that you should take upfront, the cheaper it’s going to be to actually deliver something. And the more likely it’s going to have a positive impact as well.”
On understanding the problem
In the old days of software development, the name of the game was Build What The Customer Wants. They would ask for something, make an order, and developers would place the order. Now, it may be that the quickest way to ruin your product is to do everything your customers want! Why? Mainly because users are not always aware of what they need. They are not always aware of their behavior. But they are really aware about their problem, and it’s up to PMs to uncover the underlying issues and find the right solutions.
Thilo says, “Probably because I come from the consulting and customer-facing side, I see way too often that people end their investigation – in terms of what to build next – at how many feature requests we have for this thing. Or things like, hey, we have 10 customers that are worth this much, requesting this. And a lot of the product management work that I see is still very, very much feature-based.
“As a consultant, you learn when you get a request from a customer, to really dig in and truly try to understand what they’re actually trying to do and what their problem is. And that requires a lot of work oftentimes, because customers come to you and they just request a feature. And you will have to overcome your urge to put it on the roadmap, or just note it down. But really dig in, and ask, ‘Okay, cool, tell me more about why you’re trying to do this.’ And half of the time, actually, their problem is already solved. But they should just solve it in a very different way. And the other half of the time, you discover that their feature request is actually not the right one, and you should rather focus on a specific problem that should actually be solved differently.
“And so if there’s one thing that I would urge people to do more is really try to understand the problems behind the feature requests. Try to understand your customers, what they’re trying to do, what their business model even is. Why do they have this problem? Go even further, right? It’s always because they’re this kind of business, and this is what they’re trying to achieve, and this is what their CPO and CEO cares about. And if you truly understand that, you can build much more powerful products, I feel.”
On what makes a great PM
According to Thilo, a great PM has an experimental attitude, combining curiosity with truth seeking. PMs need to be curious about how the world works, but in order to answer their questions they experiment and find the fundamental rules underneath to ensure good learning is happening.
He says, “I think people need to be curious, they need to be learners, and they need to be truth seekers. So they need to have an urge to really try to understand, to get to the truth of things. And be humble in the process.
“You mentioned to me earlier your pet peeve is about people falling in love with their own ideas. You have to be humble and say, well, actually, maybe it’s not the best idea. I’m trying to find the truth of what actually the problem is, the truth of what actually the right way to solve this is. So they need to be truth seekers.
“I also mentioned that we focus on building teams that work cross-functionally together to solve problems. So you need to be a great collaborator, great facilitator. So you need to be able to impart your knowledge and give enough context to a team to actually solve a problem without guiding them too much in one direction. Because if you’re solving new problems with new products, you need a lot of creativity. And so you need to give people the space to do that. So being a good facilitator and, and somebody that can collaborate well really helps.
“And then having that understanding that your product is more holistic than just the user interface. It’s the whole experience, end to end. From the first time that somebody comes in contact with it, or even learns about it. To when they potentially turn or stop using the product. That whole experience, everything they touch, that’s part of the product. And so having that holistic view, I think really helps as well.”
About the speaker
About the host
Big idea product leader specializing in the space between 0 to 1, digital transformation, and innovation.