Product Category Memo #6: Model Monitoring Tools
Product Category Memo #6: Model Monitoring Tools
In-depth analysis of products that are used to monitor machine learning models in production
Hello friends,
Welcome to the 6th edition of Product Category Memo. The goal of this segment is to do an in-depth analysis of a specific product category in AI, ML, and Data. We’ll talk about what the product does, what the users expect from it, landscape of products, pricing mechanics, and growth strategy.
In this post, we’ll talk about:
Why do we need products to monitor machine learning models
Who needs this product and what do they want
What products are competing in this market
What factors drive pricing
How these products acquire customers
If you have a question, submit it here and I’ll get back to you with my thoughts. Subscribe to this newsletter to receive it in your inbox every week:
I asked DALL-E to generate an image where we try to keep an eye on a robot doing its job. And this is what it came up with. Hope the robot doesn’t get too conscious about all this attention. No pressure, robot! Just do your thing.
Alright let’s dive in.
Why do we need products to monitor ML models?
ML is being used for a wide variety of use cases. People are building ML models and using them in production. But they need keep an eye on those models. Why? Because we need to ensure that the models:
stay relevant as time goes by
reflect the current reality
have low latency
are not bulking up as more data comes in
are accurate
continue to stay unbiased
don't deteriorate due to any data mismatches
don't deteriorate due to any changes in data format
don't deteriorate due to any drifts in data distribution
don't take up too much computing power
Who needs this product and what do they want?
These products are usually used by MLOps Engineers. But they should be used by everyone who's putting ML models in production. If you’re putting a model in production, it's your job to ensure that your models are doing the job at all times.
The users look for the following features in a model monitoring tool:
It should allow you to monitor all the key metrics discussed in the previous section
It should fit within your existing workflow and connect to the tools you're already using
It should be lightweight and not put additional stress on your ML infrastructure
It should allow you to evaluate model performance and run tests
It should allow you to see what's happening inside by providing a good logging mechanism
It should allow you to compare models and run A/B tests
It should provide automated alerts when the model performance goes down
It should show hardware metrics such as how much CPU/GPU is being used by the models
It should provide CI/CD pipelines for ML
This is what it looks like if the monitoring tool is too intrusive:
What products are competing in this market?
Here’s a list of companies competing in this market:
What factors drive pricing?
Here are the levers that are used to determine the pricing for a given product:
Number of models being monitored
Number of features per model
Number of times the model has to do inference in production
Number of times the model has to do inference in training
Number of users who need access to it
On-prem deployment option
Access control
Level of support needed
How do these products acquire customers?
These products can use the following ways to acquire customers:
Enterprise sales model: These products usually need top-down adoption to make a difference. Everyone on the team should be using the same product. Companies that have been successful in this sector have taken the enterprise sales approach where they sign larger contracts with companies. This means that individual users can’t sign up for free or by paying a small amount to give it a test drive. Since it's a nascent field, it's hard to have a standardized pricing structure, but they used the levers mentioned in the previous section to provide pricing details to customers.
Integration capabilities: A product needs to play well with all the tools along the ML value chain. If your product doesn't integrate well, it won't be adopted by companies.
End-to-end solution: A product needs to be an end-to-end solution to penetrate large companies. Point solutions have a hard time penetrating large companies. Large companies won't stitch together a bunch of tools themselves to get their work done. They will end up going with a company that can meet all their needs.
If you are getting value from this newsletter, consider subscribing and sharing with your friends: