What Product Managers Should Know About Machine Learning
Learnings from managing a machine learning product
Hello readers,
Welcome to Infinite Curiosity. I recently came across an interesting two-part question — How should product managers plan ML work? What should they know about ML to be effective at it? I humbly decided to tackle that question in this post.
In this post, we’ll talk about:
how product managers can frame problems
how product managers can manage ML work
how to communicate with stakeholders
how to deal with uncertainty when planning ML work
If you have a question, submit it here and I’ll get back to you with my thoughts. Subscribe to this newsletter to receive it in your inbox every week:
I wanted to see what a product planning meeting would look like if it’s conducted on a spaceship. I asked DALL-E to generate an image for me and this is what it came up with. The guy standing clearly has something to say. Let’s dive in.
If you are a product manager in ML, you'll benefit from understanding the following key points about how machine learning works:
Evaluating feasibility of using ML
Here’s my view on this field: AI is the goal. ML is a vehicle to get there. And Data is the fuel for that vehicle. Before you scope out a project, always check if there’s fuel available. You should be familiar with how datasets are curated and used in ML. If a dataset is not structured, how long would it take to structure it? It will help you evaluate if a particular problem can be solved with ML.
Assessing if you need ML in a given situation
Let's say that it's feasible to use ML in a given situation. But should you be using ML every single time? For any given problem, you should learn to assess if ML is really needed. If you can solve it without using ML, you should take that path. The goal is to deliver something that solves a problem for your customers.
Let’s say the customer is expecting to predict the future with absolute certainty. But they only have sparse data to support that goal. You need to spot it early on. If they do have the data, you need to know what it is. Is it a classification problem? Or a regression problem? Does it involve forecasting? If so, what level of error is acceptable?
Mapping business problems
Start with the outcome you want to achieve with ML and work backwards from there. If you can get something done without ML, do it. You should have a clear understanding of what problems can be solved with machine learning and what problems can't.
ML should be used to accelerate an engine that’s already working as opposed to expecting ML to be an engine by itself. If you're embarking on an internal ML project, ask yourself what problem it's solving for the customer. Don't let your teammates do ML for ML's sake. It needs to serve a business purpose.
Managing uncertainty within sprints
ML work is different from software development work. It involves uncertainty. You should learn how ML work is scoped out. Not being able to have guaranteed outcomes is one of the key challenges. An ML practitioner could spend an entire sprint working on something and have nothing to show for.
One way to deal with it is to introduce guardrails and narrow down the scope. For example, let’s say the uncertainty is coming from the fact that the ML team doesn’t know what’s in the data sent by the customer. You can offload that to a data analyst team who can verify everything needed for the ML team. The ML team can then give a better estimate on when their work will be done. Product managers should help separate out exploratory work from the engineering work.
What if your org is just starting out with ML
If your org is just starting out with ML, use it to automate internal work first vs providing it to customers. Internal users are more forgiving. It will help you get comfortable with building ML systems and seeing how they get used. External customers expect 100% percent accuracy from software tools (they’re not as forgiving).
ML systems don’t work like that in the real world. This can be addressed by designing the product in a certain way. For example, a search engine showing more relevant results first and less relevant results later is a smoother experience than an image classifier that incorrectly categorizes an image of a dog as a cat.
Using the right ML terms
You should learn how to use the right ML terms. This includes knowing what frameworks are used to solve what types of problems. You should know what ML tools are used on what types of data. You should know what questions to ask and what outcomes to expect.
It will help you communicate with internal ML teams. At the same time, you should be able to communicate it in plain english to external stakeholders as well. You need to understand how machine learning processes work.
Managing timelines
Building a good ML product takes time. And building ML features might take time too. You should know how much time a given feature will take. If a piece of work is too big, you should know how to split the work into smaller chunks. You need to know enough about ML such that you can plan the work accordingly.
If you are getting value from this newsletter, consider subscribing and sharing with your friends: