Shruti Tiwari

While both traditional (non-AI) PMs and AI PMs require strong customer understanding, product sense, execution, and stakeholder management skills, AI PMs need additional expertise that makes some people better suited for the role: 1. Technical Knowledge – AI PMs don’t need to code, but they must understand the ML lifecycle, model performance metrics, drift, explainability, data pipelines, and experimentation. They should be able to collaborate effectively with data scientists and ask the right questions to evaluate feasibility. 2. Comfort with Iteration – Unlike traditional software products, AI/ML models improve over time through continuous iteration. AI PMs should embrace this approach and plan for ongoing model refinement and retraining. 3. Curiosity and Continuous Learning – AI is evolving rapidly, and successful AI PMs stay up to date with the latest advancements in models, frameworks, and ethical considerations. 4. Thinking Beyond the Model – AI PMs must consider broader implications such as ethics, fairness, explainability, and regulatory compliance to ensure responsible AI deployment.

My one piece of advice would be to be outcome-driven. A lot of them don’t make it to deployment or don’t deliver as expected. To ensure that efforts turn into outcomes, be thorough about the user needs. Don’t focus only on the model performance but also make sure that the user interface is smooth. Learning to work well with data scientists and business stakeholders, and proactively solving the potential roadblocks are some ways to succeed as a Product Manager.

Yes, we incorporate user feedback to improve the model and reduce drift over time. There are several ways to gather feedback: 1. Direct user feedback through thumbs-up and thumbs-down ratings. 2. User override rate – measuring how often users override AI-generated suggestions. 3. Tracking drop-off points where users disengage from the AI feature. 4. User support tickets – identifying recurring issues raised by users. To incorporate feedback, we analyze failure points to determine the root cause. Common ways to improve the model include optimizing confidence thresholds, retraining with better data, and filtering out edge cases where responses are inaccurate or undesired. Additionally, the issue may not be with the model itself but with UI/UX design, transparency, or explainability—in which case, we focus on improving user experience and trust.

With product vision, we need to carefully tread the line between making it too generic/vague or too technical. Other things to consider are - it should clarify what impact it creates, what unique value it offers and what is the long term goal while also making it inspiring. If target audience or problem area is niche, then it should also specify the target audience or the specific problem. An example for an unclear product vision is “ We want to make customer service better for companies”. Utilizing the above considerations, a better version of vision statement would be “We help small businesses deliver faster, more efficient and personalized customer support through an intuitive AI-powered platform “.

Based on my experience, some common mistakes AI PMs make include focusing too much on technology instead of user needs and business impact, ignoring data quality and feasibility early on, and deploying models without a feedback loop. My advice for new AI PMs: 1. Take a user-first approach, with AI as an enabler, not the driver. Work backward from how users will interact with the product and then develop the model that best fits their needs. 2. Thoroughly analyze data quality and availability and set clear expectations. AI is only as good as the data it’s trained on. 3. Plan for a feedback and retraining system, as model performance typically degrades over time. 4. Start small by giving users autonomy to override AI decisions and build trust through explainability and transparency. Once trust is established, adoption will follow.

I would think about Gen AI integration into any application based on its use case. There are two major purposes that Gen AI serves - Content generation and natural language understanding. We can further break it down into multiple use cases- 1. Content generation a. By modality like text, audio, video, image, code, 3D, multimedia b. By domain like healthcare, pharma, retail etc. c. By application like conversational chatbots, summarization, editing, generation of lists, reminders, emails, social media posts, knowledge base etc. d. By org function like marketing, finance, operations etc. 2. NLU Any use case where end result is not necessarily content generation but requires understanding of language. Some examples are classification use cases like filtering out specific emails, predictive analytics use cases like forecasting etc.

I would divide the hard skills into 4 categories - 1. AI foundational knowledge at conceptual level including understanding of AI/ML model lifecycle, different algorithms like regression, decision trees, neural network and their use cases, model evaluation metrics like accuracy, precision, recall etc. 2. Latest evolutions in architecture and models like knowing which new open source models can be explored for specific use case, new architectures to be tried etc. 3. Comfort with Data analysis - I found having working knowledge of sql or python comes handy if you want to do some preliminary data analysis to size the opportunity and understand data quality better. I wouldn’t say knowing sql/python is essential skill, but nice to have in some situations 4. Understanding product metrics and business impact metrics, which would be required for any product not just AI ones

Exact metrics would vary by product and application, but I would think about metrics in three categories- 1. Model performance metrics like accuracy, precision, recall, AUC-ROC, latency etc. Key metric will be based on the application like minimizing false negatives in email filtration app to avoid missing any important emails 2. Product adoption/engagement metrics like adoption rate, user override rate, CTR etc. 3. Business impact metrics like cost savings, revenue impact, CSAT impact, time savings etc.

Well, the good news is as the first PM in the team, you can introduce some new ways of doing things. The bad news is that its a journey and it'll take some time to change the mindset. You'll need to establish trust with the team and show some wins before the team can buy into the change. The first thing I would do is to invite engineers to listen to user interviews or support calls. That'll make customer real for them and once they see the impact of what they build, they'll start thinking differently. Second, I would use data like user behavior, conversion rates etc. to persuade Engineers, as I have seen most engineers respond well to logic and evidence. Another way to get engineers excited is to give them user problems and let them propose solutions rather than asking to build xyz feature. Another thing you can do is to add customer impact as part of the success metrics when discussing the ideas. That'll help them select the ideas which most align with customer needs.