Sharebird
Sign Up / Sign In

How do you handle scenarios where the AI may not perform as expected?

Answer
2 Answers
AI Product Management
  1. Natalia Baryshnikova
    Natalia Baryshnikova

    Atlassian Head of Product, Enterprise Strategy and Planning • 7mo

    This question is pretty much the reason why evals exist. "As expected" is a very subjective and blurry construct, so we need to get it out of our heads and "share it" with AI early. While there are certain dedicated tools and controls in this fields, such as AI safety protocols, we can take them further by introducing to AI tools we use prompts defining what good or bad looks like (write it out as specifically as possible; better yet, explain what exactly makes the example good or bad). Evals, w ...Read More

    974 Views
  2. Natalie Chung
    Natalie Chung

    Atlassian Director | Senior Principal PM, Teamwork Collection • 6mo

    Unexpected AI behaviour should be treated it as a quality and risk problem than a traditional bug. It would be good to get a better understanding of any patterns in the failures, put them into buckets such as retrieval gaps, instruction‑following errors, hallucinations, etc. Then start putting the representative use cases into your evaluation set, so those scenarios are automatically tested every time your team refines data, prompts, or models. On the UX side, there are resilient design to help ...Read More

    1,662 Views

Related Ask Me Anything Sessions

Manjeet Singh

Salesforce Senior Director of Product Management, Manjeet Singh on AI Product Management

June 17 @ 10:00AM PT

Register
Anushka Anand

Salesforce Director of Product Management, Tableau Next, Anushka Anand on AI Product Management

April 9 @ 10:00AM PT

View Answers
Bryan Dunn

Nextiva Head of Product, Developer Ecosystem, Bryan Dunn on AI Product Management

April 2 @ 10:00AM PT

View Answers
See All Upcoming AMA's

Top Product Management Mentors