This is the third in a series of eight articles on CPO Solutions. Over the past year Products That Count has been investigating the role of the Chief Product Officer. That investigation has included original research by our own CPO, Renée Niemi, resulting in the white paper “The Rise of the Chief Product Officer: Study of the Most Important Role in Business.” We also recently published an eBook on “The State of the Chief Product Officer in 2022.” That latter work focused on the eight top challenges facing the CPO of today, one of which is measuring performance.
Stay tuned to Products That Count for more CPO-related content, including the rest of this series on CPO Solutions.
First, be a business leader
As the aphorism goes, “what gets measured gets managed.” A main function of the Chief Product Officer, then, is determining which measurements will have the biggest impact on the success of the business. That determination involves a combination of bottom-up and top-down thinking.
In product management generally the top three metrics are retention, engagement, and conversion (according to a survey of 500 PMs by 2022 Product Awards finalist Mixpanel). These product KPIs are certainly vital to business success, especially in an era of product-led growth. But the Chief Product Officer needs to focus a bit more broadly when it comes to measuring performance.
On the Product Talk podcast, Tenable CPO Nico Popp laid out seven key KPIs for Chief Product Officers. Included are metrics for retaining top talent, customer satisfaction, and specific product goals. But a majority of these KPIs – four out of seven – are related to the business top and bottom line: accounts receivable, growth rate, gross margin, and expense ratio.
Thus, a CPO may lead the product function, but the first priority is to be a business leader. This idea resonates with comments from Fidelity Investments CPO Deba Sahoo. As Deba puts it, “As a product leader and CPO, it’s a given that you do well in launching products, strategizing about products, building orgs. But to be a really good CPO, you first have to be a business leader. You need to have a business lens and business acumen. And you need to have credibility with your peers from other functions, and up to the CEO. So you are actually driving, at the C-level, as a business leader. I think that’s probably the most important thing.”
Hear Deba’s comments on measuring performance in the video below.
Measuring NPS improvement
As noted, business metrics are the headliners of the Chief Product Officer’s key KPIs. Next on the list is talent retention – discussed earlier in this series. But now let’s turn to another key for the Chief Product Officer in measuring performance: customer satisfaction.
In our earlier study of Chief Product Officer, we reported that Net Promoter Score – NPS – is the most commonly used customer satisfaction metric for CPOs. In brief, NPS tracks whether customers would recommend a product or service. It tracks promoters, detractors, and those who are neutral.
The NPS scale runs from -100 to 100. A basic understanding of the metric is that anything over 0 is a “good” score, though Amazon’s NPS of 73 in customer loyalty may be the gold standard. So what if a CPO wants to push his or her product’s NPS into Amazon territory? Determining where to begin is not so clear cut, as product executive Tony Ralph points out.
Tony Ralph has directed PM teams at leading brands like Intuit, Netflix, and Walmart. Most recently, he was SVP of Global Data Product Management at PepsiCo. Tony notes that for a mature product, Net Promoter Score (NPS) will often span a large set of customer experiences including interactions across product features, UX navigation, customer support, billing, etc. Given that, how do you improve NPS? Do you focus on one customer experience or feature at a time? Or on a superset of these interactions?
Tony says, “In trying to improve NPS, it’s often not going to involve one thing. Improvement will rely on both holistic and detailed feedback from customers. Input could span 99 different areas. Of those 99, which 12 are the most impactful? And then, this quarter, which three of those can you work on? Further, what are the hypotheses, from a product manager’s perspective, regarding the path to improvement? How long does each take to implement? Next, a product manager balances this potential improvement – which you model from a measurement perspective – against engineering level of effort. Once implemented, experimentation or other forms of measurement determine the extent to which each did or did not contribute to NPS lift.
“So there’s NPS as a whole, thinking of it as a Chief Product Officer. But for the product team on the ground, it becomes an interesting prioritization effort to determine what to tackle first. The synthesis of these perspectives involves the quantification of potential impact for each product area and eventually a roadmap to track the path toward improvement. Getting both top-down and bottom-up consensus on this roadmap is essential for two reasons. First, it ensures tracking and accountability for the efforts within each product area. And second, in the case where progress needs to be accelerated, the detailed improvements outlined in the roadmap are input into resource planning (trade-offs vs. incremental funding).”
Tony’s discussion of the roadmap is a reminder that, at the end of the day, metrics are tools. Their purpose is to help teams impact the company’s success. In order to be useful tools, then, they must be actionable. We turn to that topic in the final section, below.
Make metrics actionable
When Scott Williamson was Chief Product Officer at GitLab, he oversaw a range of key metrics. These included NPS as well as things like Monthly Active Users (paid and non), Stages Per User, and System Usability Score. As a SaaS company, it was also important for GitLab to also account for Error Budgets, a snapshot of slowdowns, performance, and outages.
Some metrics had a direct business impact, like Average Selling Price and Trial to Paid Conversion Rate. As such, these KPIs were shared with the product and business teams. But Scott is clear that shared ownership must not dilute the extent to which teams can take action. In short, a CPO must make metrics actionable.
Action item: Give PMs and product teams the ability to act on meaningful metrics.
As Scott puts it, “We try to make KPIs as actionable as possible for a single org. What you want to avoid is something that’s so broadly owned that people feel like they can’t action it. So most KPIs can be moved by product, and are envisioned as a measurement that product can directly affect.”
In line with the reality of CPO as business leader, Scott stresses the importance of a particular top-line business goal. Namely, Annual Recurring Revenue (ARR). ARR tallies the funds owed to the company for products and services provided to clients. At GitLab, this is not a static number, but a measure that Scott’s teams were taking action to move.
As Scott puts it, “At the end of the day, ARR growth is what everybody in the company is trying to drive. And some of the different KPIs ladder up to it. For example, you add a new user. If they use more than one stage, the amount that they upgrade to a paid tier is 3x higher. So we have linkage between the lower level things we’re doing and ARR. Because you want the KPI to be actionable.”
Summing up
To summarize, measuring performance as a Chief Product Officer begins with the business. Is product driving the business top and bottom lines? As Deba Sahoo says, the CPO is first and foremost a business leader. That is confirmed by Scott Williamson, who points to the focus on Annual Recurring Revenue during his time at GitLab.
On the other hand, it may not always be clear how product is contributing to the business goals. A low NPS may be dragging down business metrics, but Tony Ralph points out that improving NPS isn’t simple. Everybody might want Amazon’s NPS (and profitability!), but the company didn’t get there based on just one thing.
Nonetheless, to reiterate the “top line” point of this article: what gets measured gets managed. In that sense, Chief Product Officers will continue measuring performance to ensure that product success and business success are one and the same.