Shruti Tiwari

Based on my experience, some common mistakes AI PMs make include focusing too much on technology instead of user needs and business impact, ignoring data quality and feasibility early on, and deploying models without a feedback loop. My advice for new AI PMs: 1. Take a user-first approach, with AI as an enabler, not the driver. Work backward from how users will interact with the product and then develop the model that best fits their needs. 2. Thoroughly analyze data quality and availability and set clear expectations. AI is only as good as the data it’s trained on. 3. Plan for a feedback and retraining system, as model performance typically degrades over time. 4. Start small by giving users autonomy to override AI decisions and build trust through explainability and transparency. Once trust is established, adoption will follow.

Yes, we incorporate user feedback to improve the model and reduce drift over time. There are several ways to gather feedback: 1. Direct user feedback through thumbs-up and thumbs-down ratings. 2. User override rate – measuring how often users override AI-generated suggestions. 3. Tracking drop-off points where users disengage from the AI feature. 4. User support tickets – identifying recurring issues raised by users. To incorporate feedback, we analyze failure points to determine the root cause. Common ways to improve the model include optimizing confidence thresholds, retraining with better data, and filtering out edge cases where responses are inaccurate or undesired. Additionally, the issue may not be with the model itself but with UI/UX design, transparency, or explainability—in which case, we focus on improving user experience and trust.

I would think about Gen AI integration into any application based on its use case. There are two major purposes that Gen AI serves - Content generation and natural language understanding. We can further break it down into multiple use cases- 1. Content generation a. By modality like text, audio, video, image, code, 3D, multimedia b. By domain like healthcare, pharma, retail etc. c. By application like conversational chatbots, summarization, editing, generation of lists, reminders, emails, social media posts, knowledge base etc. d. By org function like marketing, finance, operations etc. 2. NLU Any use case where end result is not necessarily content generation but requires understanding of language. Some examples are classification use cases like filtering out specific emails, predictive analytics use cases like forecasting etc.

My one piece of advice would be to be outcome-driven. Only a few models end up being deployed and adopted by the users, a lot of them don’t make it to the end or don’t deliver as expected. To ensure the efforts turn into outcomes, be thorough about the user needs and don’t focus only on the model performance but also make sure that the system and design from model output to user interaction is smooth. Learning to work well with data scientists and business stakeholders, and proactively solving the potential roadblocks are some ways to succeed as a Product Manager.

Exact metrics would vary by product and application, but I would think about metrics in three categories- 1. Model performance metrics like accuracy, precision, recall, AUC-ROC, latency etc. Key metric will be based on the application like minimizing false negatives in email filtration app to avoid missing any important emails 2. Product adoption/engagement metrics like adoption rate, user override rate, CTR etc. 3. Business impact metrics like cost savings, revenue impact, CSAT impact, time savings etc.

While both traditional (non-AI) PMs and AI PMs require strong customer understanding, product sense, execution, and stakeholder management skills, AI PMs need additional expertise that makes some people better suited for the role: 1. Technical Knowledge – AI PMs don’t need to code, but they must understand the ML lifecycle, model performance metrics, drift, explainability, data pipelines, and experimentation. They should be able to collaborate effectively with data scientists and ask the right questions to evaluate feasibility. 2. Comfort with Iteration – Unlike traditional software products, AI/ML models improve over time through continuous iteration. AI PMs should embrace this approach and plan for ongoing model refinement and retraining. 3. Curiosity and Continuous Learning – AI is evolving rapidly, and successful AI PMs stay up to date with the latest advancements in models, frameworks, and ethical considerations. 4. Thinking Beyond the Model – AI PMs must consider broader implications such as ethics, fairness, explainability, and regulatory compliance to ensure responsible AI deployment.

I would divide the hard skills into 4 categories - 1. AI foundational knowledge at conceptual level including understanding of AI/ML model lifecycle, different algorithms like regression, decision trees, neural network and their use cases, model evaluation metrics like accuracy, precision, recall etc. 2. Latest evolutions in architecture and models like knowing which new open source models can be explored for specific use case, new architectures to be tried etc. 3. Comfort with Data analysis - I found having working knowledge of sql or python comes handy if you want to do some preliminary data analysis to size the opportunity and understand data quality better 4. Understanding product metrics and business impact metrics, which would be required for any product not just AI ones