Experimentation can be developed as a powerful storytelling and product tool. Through experimentation, a product manager can make accurate, data-driving decisions that can expand their product’s reach. These don’t need to be complicated, but how can product managers experiment in ways that best fit their product? Dow Jones Product VP Peter Gray shares his unique insights in experimentation, subscription, and personalization to create value.
Subscribe to the Product Talk podcast on Spotify and Apple Podcasts, and make sure you never miss a single product conversation with leading product executives. New episodes live every Wednesday.
On Experimentation as a Mindset
Peter shares how with the big user base at Dow Jones, he saw an opportunity to experiment with different types of approaches to product, especially when it comes to bringing in new users or retaining existing users. It can be a good way to test out a hypothesis without affecting the product too much while getting the necessary data.
“When I got to Dow Jones, I had never run a digital experiment. When I was at Via, the scale was just too small, based on the percentage of the users who are doing your own metric, and the number of users who you have, you just a certain number. Depending on how radical the sort of idea you want to try as an experiment, there’s sort of this math equation, which tells you essentially just whether you can run an experiment, or how many how quickly, you can run them. And so Via, we really couldn’t run them. When I got to Dow Jones, this is a scale business, we have millions and millions of people coming to our site.
It opened up that as an opportunity, and it struck me that for this business that was learning how to make data-driven decisions, experimentation would be an incredibly powerful storytelling tool, and just product tool, product methodology that would make everyone feel very, very comfortable with trying new things. It de-risks it because the measurement is so good. If you set up an experiment properly, you’ve isolated your variables, and if the metrics you care about the move because of this hypothesis you had, you can be completely confident that it was that thing, which just takes that pressure out of the room around what are we risking? We changed it, but what is it working or is it not? Especially with a news product, wherefrom a traffic perspective and the news cycle, you can really have big swings. …
This is the canvas on which right experimentation seemed like an obvious way to way to tell the story around sort of making decisions in a new way. It also opens it up as an offensive opportunity. This isn’t just about making people comfortable. It’s the most powerful product development methodology on the face of the planet because it completely de-risks big bets. You can basically see the future, except you can’t AB test whether to make an iPhone, so a true zero-to-one thing can’t be launched an experiment, but everything else can. When it comes to the mindset, I think where I feel like my experience could be beneficial to anyone in product, whether they’ve got a much smaller scale b2b product or a big scale b2c product is, just because you can’t run an experiment doesn’t mean you shouldn’t over-invest in trying to make the most forensically accurate data-driven decision you could make around it. In the case of a write a b2b product that only has 10,000 people coming to your landing page or your product experience every week, maybe you don’t have enough traffic to run a clean AB test about this new way of presenting information or whatever the case may be. But just thinking about it as if you were trying to build an experiment and turning off the old thing in a month, planning around and saying: I’m going to turn the old thing off this day and I’m gonna turn the new thing off this day. And then I’m really going to track the success metrics … watch them like a hawk over this, two-week or three-month period, depending on the kind of product or feature change you’re considering. So it’s about the investment in every fork in the road, trying to figure out what the best cleanest form of measurement could be.”
On Framing Your Customer Base as Subscribers
A subscription-based model might seem to be limited to certain products or industries. Even so, the mindset of subscribers is not limited and can be utilized by any product manager as an experiment to view their users this way. Users must find value in your product, whether they pay you directly or not.
“It would it benefit any person with a digital business to pretend their business is a subscription, even if it’s not, even if it’s an ad-driven business or whatnot. … Your users are your oxygen, even if they’re not the ones who pay you and somebody else pays you. Those people are not going to be cutting checks unless your users find value in your product. It’s also much cheaper to keep up to keep a user if you’ve already found them to find somebody new. Even if you sort of scalpel out the piece where your user is actually paying you as a recurring fee, thinking about them as someone who you want to create an engaging relationship with that essentially never ends is a mindset that I think will serve you well, regardless of what kind of business.
That’s not rocket science, by any means. but it’s an interesting framing, which I think we see a lot of ad-driven businesses now realizing that it might be beneficial if I tried to actually get your email, know who you are, remember your name beyond what cookies could do for you.”
On When It Makes Sense to Personalize
Peter believes that a one-size-fits-all approach can be beneficial to a product, especially with user engagement and experimentations. Sometimes segments can become overwhelming. However, there needs to be a right balance of personalization that is automatically generated and something the user can control.
“At one end of the spectrum, you’ve gotten monolithic change, meaning my website for all my users used to be black, and now I can change it to white. Then you’ve got segmentation. I’ve got this user group that likes it to be a black background, and this user group likes it to be white, and this user group likes it to be red. There’s a personalized outcome where the answer is essentially on an individual user level, and it’s beyond the human capacity to keep track of what the right answer is for all those different users.
Segments at this point, I feel we use them primarily for ideation and hypothesis generation, trying to visualize a type of person with a type of need, but in practice, we find ourselves gravitating to either poll: either monolithic change or lightly segmented changes in the product, meaning mobile versus web versus desktop and nothing more granular than that, or completely personalized, where you turn on an algorithmic user level to serve up exactly what this person wants … I’m trying to paint a picture of the decision tree around which level of granularity of strategy is appropriate and each circumstance
In terms of when it’s appropriate, what we’ve seen is, we’re basically a content business, the place where there’s the most knowledge and experience both in our business and others is in content recommendations. So, figuring out that Patrick will be better served if we find some articles on healthcare, and Pete’s better served if we find some articles on real estate. That’s the bleeding edge of personalization. That’s one place where we’ve been successful. Another place is on our paywall, where we are using a custom algorithm to determine whether you are highly likely to subscribe, in which case, you will not be allowed to read the story you’ve clicked into. Or we think you are not very likely to subscribe, in which case, we think maybe you need a little bit more sampling and we will let you read the article. Those are the two places where we’ve been most successful.
I think the places where the people at the bleeding edge of this stuff like Netflix are sort of exploring and attacking and are in the trenches today, is that kind of hybrid stuff like experience personalization, such as a case would be at the [Wall Street] Journal, on the homepage it would change at the end of the day versus the beginning of the day, stuff like that. It wouldn’t be content, it would be the UI around the content changing by a factor like that. I’m sure there’s an upside there. I think it’s not an accident that Netflix is sort of attacking that blast after they’ve obviously invested so heavily in content recommendations. I think it’s a smaller prize, but that doesn’t mean there’s no upside there.”