AMA: DocuSign Director of Product Management, Hiral Shah on AI Product Management

Getting user feedback is a very important element of building any AI feature. Without the user input, you cannot improve your AI, which means you cannot improve your product. Treat the AI feature similar to any product feature, you need to be nimble, agile and adaptable. There are a number of ways you can gather this feedback and depending on that you can incorporate it in many different ways * In feature feedback: Think about this as the thumbs up or down you do on your chatGPT or ask it to re-generate. This is one type of user input to feed back to the algorithms that the output produced was not satisfactory to the user. This is human in the loop * I have used this when I was building chatbots 7 years ago by providing 5 emojis at the end of generating a response to the question. The ones that are on lower satisfaction, we would go analyze each of the answers manually to see what went wrong. * Feature on/off: Lets say if you have an Opt-in / Opt-out feature, tracking the number of people option out can be a good way to know whether AI is working or not. * If customers are not signing up for the feature, there can be an awareness and onboarding problem. This can be fixed with some education material in the product as well as branding and messaging outside. * Qualitative feedback - This is sitting down with your customers and showing them the AI and gathering input. This method us true of any product feature, even more important in the AI feature

While I don't believe there are a lot of different types of AI Product Managers, in some large organizations I see two types of AI Product Managers - 1) application focused and 2) AI foundation focused. It also varies based on the size and maturity of the organization. The Application focused AI PM is thinking a lot about how can you leverage the AI to deliver the best customer value, which customer problems can be solved better if we applied AI and then focus on the adoption of AI. The Foundations AI PMs role is focused on working much more closely with the data science teams to innovate, decide what types of AI models to leverage, ensure that the processes we leverage for AI are compliant, build data anonymization pipelines, data acquisition and labeling tasks are performed and the infrastructure is scalable in a cost affective way. In smaller organization if you join as a product manager, it is highly likely you are the only Product manager doing all of the things. Hence, these lines get blurred in different organizations.

Having a software engineering background is a plus when you want to become an AI PM. This helps you build rapport with engineering, data science quickly and help translate the technical details in a more user friendly way. However, even as an AI PM, you still need the core Product management skills, which are needed to be acquired in various roles. For example even as a software engineer, you can come up with ideas, pitch and build it to and see what's the impact. Be data driven to track every metric using tools such as Amplitude, Mixpanel on how user is using the product. Having some side projects to acquire and demonstrate your PM skills are going to be very critical.

One key thing would be to have empathy for your fellow data scientists. When working with data scientists, remember that simply asking for Level of Effort won't suffice. Feasibility and effort are often exploratory in the AI realm. So, change the language you speak to foster successful collaboration! Shared Understanding: As a PM, it's crucial to ensure that data scientists understand your goals and objectives. Clearly communicate what you're trying to achieve and the assumptions that need validation even before an AI model is developed. Collaborative Exploration: Instead of seeking concrete estimates, foster a collaborative environment for exploration. Encourage open discussions, knowledge sharing, and joint problem-solving. Embrace the iterative nature of AI development to uncover insights and uncover the realm of possibilities. By shifting the conversation and fostering collaboration with data scientists, PMs can bridge the gap between product goals and AI exploration. Embrace the power of shared understanding and collaborative exploration for AI success!

Overall I believe that all PMs can be AI PMs since AI PMs also have to be equally customer obsessed as traditional PMs. However, I do believe AI product management role can be a bit different than a traditional PM role. Some differentiations and questions to think about it? ➡ Technical Understanding: Can you explain how a machine learning model works, in layman's terms? Or, how would you handle the situation where the model is giving unpredictable outputs? ➡ Data Management: Can you provide an example of a project where the quality or type of data significantly affected the outcome? ➡ Adaptability and Problem-solving: Can you tell about a time when an AI project you were working on didn't go as planned. How did you pivot? ➡ User-Centric Approach: How do you ensure that AI-driven decisions are transparent and understandable to users? How do you ensure users get a great experience even when AI does not work well ➡ Regulatory Knowledge: How do you stay updated with the regulatory changes in AI? How do you overcome those barriers and find a path forward? ➡ Communication and Collaboration: How do you bridge the gap between technical (data scientists, AI engineers) and non-technical (sales, marketing, customer service) teams in the development and deployment of AI products? These give you a color of what nuances exist for being a great AI PM