Natalie Chung

For a Product Manager, Trust & Risk is no longer just about compliance - for those building AI experiences, this is also validate viability and quality. Examples of key skills PM should target to develop in this area include Setting up Evals - systematic way to score AI outputs against a standard. It is also important to create a scoring rubric and creating reference datasets (which curated examples of ideal inputs and outputs that serve as ground truth), and create judge prompts for one AI ...Read More

Unexpected AI behaviour should be treated it as a quality and risk problem than a traditional bug. It would be good to get a better understanding of any patterns in the failures, put them into buckets such as retrieval gaps, instruction‑following errors, hallucinations, etc. Then start putting the representative use cases into your evaluation set, so those scenarios are automatically tested every time your team refines data, prompts, or models. On the UX side, there are resilient design to help ...Read More

Examples of strategies to test and validate : Evals: Given the probabilistic nature of AI, manual testing at scale is impossible and evals are crucial for validation. This includes setting up Golden output data sets: Create a static set of inputs with "perfect" human-written answers. Automated Judging: When updating your model, run those 100 inputs through it. Use a separate "Judge" model to compare the new output against the golden dataset. Pass/Fail Metrics: This provides a percentage score (e ...Read More

Over the years, I've found success through early cross-functional involvement, data-driven decision making, and empathetic enablement that helps drive better alignment and collaboration with sales Here's my approach: 1. Bring sales Into strategy development early - include sales leaders in strategy sessions from day one, making them co-creators rather than recipients. PMs can run joint customer research sessions where sales can share field insights 2. Ground decisions in shared data - partner wi ...Read More

A successful product strategy shows clear signals: target customers adopt faster, stay longer, and pay more, while your team executes with increasing confidence toward a focused vision. From my B2B product experience across various products, here are some examples of what I track: Leading Indicators Target segment adoption: Enterprise customers standardising workflows in our platform Feature usage growth: Advanced capabilities usage up X% YoY, validating our differentiation Strategic positioning ...Read More