Many companies race to use machine learning capabilities, eager to build on cutting-edge research and tackle business applications. As interest grows, more product teams and leads anchored in traditional methodologies find themselves in unfamiliar territory. How do they begin to build machine learning products? Borealis AI Product Lead Alex LaPlante shares how machine learning is different than traditional software, what it takes to deliver business value, and how to effectively manage these as a PM.
Join us for new conversations with leading product executives every week. Roll through the highlights of this week’s event below, then head on over to our Events page to see which product leaders will be joining us next week.
Join us at our weekly Speaker Series events to engage with product leaders in your own community and gain insights on how to accelerate digital transformation.
On Machine Learning Products vs Traditional Software Products
What makes machine learning systems different? Alex likes to break it down into two categories: software 1.0, otherwise known as traditional software, and software 2.0, also known as machine-learning-based software.
“Software 1.0 really consists of explicit instructions. Every time we are writing a piece of code or writing a line of code, that line of code has some explicit instruction. What that means is there’s very little uncertainty in terms of how that code at the end of the day is going to behave. We’ve made very specific rules in terms of what we expect. Of course, with any software development, any technical project, there are uncertainties, there are risks, but we have a pretty good idea that if we have a set of rules and we program them in, we know how that program is going to behave.
Machine learning, on the other end, provides us a lot more flexibility in terms of the types of problems that we can automate, so much larger sets of things that we can do. At the beginning stages, we approach it largely like we would traditional software. We ask ourselves a question, ‘what is the desired goal or outcome?’ … In order to do that task, what we’re going to do next is provide a rough skeleton of code. Generally, that rough skeleton is some form of model architecture. It could be a neural network architecture, and that neural network architecture has some gaps in it. There are some things that we have to figure out to complete the code. We complete the code via data and, of course, using our wonderful GPU cluster at Borealis to run some experiments. What that allows us to do is specify all of the parameters and the hyper-parameters within that model architecture, and what we’re really trying to get at is a program that does that pass that we want to do. Remember, we’re referring based off of history, because we’re using historical data, and the process that we’re using is a form of optimization. There’s some stochasticity to that.
What that means is, with all of that flexibility that we get from machine learning systems also comes a huge amount of uncertainty. As product managers, we have to figure out how to pinpoint that uncertainty and how to manage that uncertainty.”
On Discovery and Feasibility of a Machine Learning Problem
The discovery phase for machine learning products is similar to what most product managers do, with a lot of the same tools and processes: identifying the right problem, understanding the problem space, framing the business problem, and making sure we pinpoint that desired outcome. With machine learning, Alex points out that there are a few additional questions that are specific to the field.
“The first foremost biggest question and you will see this a couple of times throughout my talk is asking yourself the question, ‘Is machine learning the right solution?’ We exclusively do machine learning at Borealis, and so we really need to make sure that anything that’s coming in, anything that we’re agreeing to take on is, in fact, a machine learning problem. Obviously, not all product teams have this type of constraint, but it’s what we work within. … A couple of things that might help you answer that question is data. Is it accessible? I kid you not, I’ve definitely been in meetings with business stakeholders who have had what seems to be on paper a really great problem. When we ask about the data, they say it’s in that filing cabinet over there. Unfortunately, my team can’t do anything with that if the data isn’t digitized and easily accessible. … With machine learning, we also need a decent quantity of data.
Another question we ask ourselves is about generalizability and complexity. If you can distill your task down into five rules, we probably don’t need to machinery system. We’re really looking for those cases where there are way too many instances to program now manually, or there are really complex, highly nonlinear relationships. … Remember, with rules-based approaches, we know exactly what we’re going to get out of that, so if there was a very, very low error tolerance, you may not want to use a machine learning system. Machine learning is stochastic, it is definitely going to get some things wrong.
The one often overlooked is the cost of machine learning. Oftentimes, we consider the development costs of machine learning, but we forget about the production costs of machine learning. Generally, in production, we need specialized tooling to support that we often need access to GPUs, machine learning can be exceptionally expensive, you really want to make sure that the problem that you’re using is the problem that you’re focusing on if you’re building a machine learning solution, and that the business impact far outweighs the cost. …
The whole purpose of feasibility is testing that hypothesis, basically confirming, ‘is this actually a good machine learning problem?’ We’re going to start digging into the data, we’re going to start doing some explorations, and we’re going to do some rapid experimentation and baselining. And this basically gives us a sense of, you know, how good is the data and what can we get out of it? … One of the key tasks for a PM during this time is to get clear performance metrics and requirements. This is actually quite difficult. Oftentimes, our business partners have a really clear sense of what they’re looking for, but they don’t know how to translate that into technical requirements. … One of the big jobs a PM has to do is navigate that, figuring out what it means for 99% accuracy for user experience, for example. That’s where a lot of my pm spend a lot of their time during feasibility.”
On Mismatches Between Technical And Business Value
“Let’s use two different contexts. So in one context, we’re building a recommender system, and in the other context, we’re building out a trading algorithm that’s going to trade money within capital markets. So in the trading example, that technical performance, that accuracy directly relates to P&L. It is going to drive and determine whatever profit or trading algorithm makes. What that means as a product manager is we may consider pushing the team a little bit harder to get to a slightly better model before we put it into production because that directly aligns with business value.
If we come back to that idea of a recommender system, there’s likely a performance metric, a level of accuracy that we’ll hit, where once we hit that, we basically plateau in terms of business value that’s driving back the user experience that we drive there. As a product manager, you really have to think about when to call it, when is good enough or when is the model that you have good enough, and when should you really push forward into the next phase of production. It’s not always all that easy to make that determination; you really want to make sure that you don’t let perfection get in the way of driving value within the business, though.”
About the speaker
About the host
Arun heads Product Management for Royal Bank of Canada’s Solution Acceleration and Innovation group where he leads a team of PMs that manage a portfolio of B2C and B2B products. He also heads RBC Launch, RBC’s innovation hub, where the focus is on rapidly experimenting and developing new products. In prior roles, Arun developed products for RBC's commercial lending business, building large-scale credit structuring applications. Arun has deep domain expertise in financial services gained working in corporate strategy and various business lines. Prior to banking, he has worked as a Management Consultant and as a Physician at different points in his career.