What metrics do ML product teams look at to define success? Which do you find to be the most important?

Let me preface this by defining a product team as PM, UX and Engineering. I'd suggest there are at least two sets of metrics you should be looking at. First and foremost, don't forget you're here to solve a customer problem. Judge success according in how the capability is driving that specific outcome just like you would any other product. That could be the time it takes a customer to do a task, number of phishing attacks detected, sales volume of your sellers on a marketplace or rides taken by ...Read More

The same first principles actually still apply to AI PMs, but with an added dimension of complexity, which is that a generational paradigm/platform shift like AI requires a PM to re-think the benchmarks for what good looks like, consider the new types of outputs/inputs required, and the additional internal communication paths needed to achieve goals. Most similar to being an early PM in mobile 10 years ago, or the early web/internet even before that. What doesn't change: (+) Focus on key outputs ...Read More

While working on ML product/feature, there are 2 sets of metrics: 1. Product success metrics that product managers define. Purpose of the is to measure the business/product outcome you are trying to optimize for. Your standard metrics like customer adoption, usage, retention, satisfaction etc. fall in this bucket. So, product managers choose what's best for the situation at hand. 2. ML metrics that data scienstics define. These metrics measure how good the ML approach/solution/model is. For e.g. ...Read More

Metrics are an interesting question. This really depends on the type of product we are building that leverages ML. Since ML can be use for example in electronic records, sales workflows, computer vision type use cases or speech / audio use cases some of which I am familiar with -- we can break it down to product use itself and then algorithm/model used, how often it is used, what kind of business or customer experience metrics it provided or influenced. So the long and short answer is there is n ...Read More

There is no one "metric" that ML Product Teams will use to define success. It entirely depends on what is the "ML" used in the context of a feature or product. Typical metrics might include Quality improvements achieved via ML implementation vs traditional algorithmic implementations Usage adoption of AI/ML based feature Business or user perception metrics of improvements - CSAT, NPS scores Time savings, $$ savings, etc typical Business metrics. Benchmarking vs peer products Accuracy and respons ...Read More