Sharebird
Sign Up / Sign In

How do you measure the success of your experiments when you're at an early stage company and there may not be enough quantitative data?

Answer
3 Answers
Product Experimentation
  1. Bruno Gobbis
    Bruno Gobbis

    Nuvemshop Director, Product Growth | Formerly Superhuman, RD Station, IBM, Bosch • 6mo

    That's a different (and more challenging) situation! And you could also face this, especially when experimenting on a journey where you don't have many users passing by, so this isn't only for early-stage companies. My take here is to go for "decision significance” rather than “statistical significance” -> which you probably won't have. At low data volumes, prefer to treat experiments primarily as structured learning vehicles rather than binary “win/lose” bets that you can set using X or Y up ...Read More

    395 Views
  2. Sheila Hara
    Sheila Hara

    Barracuda Networks Sr. Director, Product Management • 9mo

    At an early stage, success isn’t always about big dashboards or perfect p-values—it’s about learning. I tell my team: if the experiment helps us answer a question we were stuck on, that’s a win. Sometimes that’s numbers, but often it’s just a handful of customer conversations, patterns in behavior, or even gut-checks validated by real users. Early on, I expect the team to focus less on “statistical significance” and more on decision significance—did this give us enough clarity to move forward wi ...Read More

    418 Views
  3. Julian Dunn
    Julian Dunn

    Chainguard Senior Director of Product Management • 9mo

    A CPO I worked for once told me that qualitative data is still data. (And this person had previously worked for Amazon, a notoriously quantitative data-driven company!) Rigorous, statistically-significant data from experiments is often less obtainable than one would think, and even if it's obtainable, sometimes the juice is not worth the squeeze, e.g. you have to collect so many samples and run the experiment for so long that the experiment's decision-making effectiveness has waned because you h ...Read More

    383 Views

Related Ask Me Anything Sessions

Deepti Pradeep

Adobe Senior Director of Product Management & Growth (Creative Cloud), Deepti Pradeep on Product Experimentation

August 6 @ 10:00AM PT

Register
Bruno Gobbis

Superhuman Director, Product Growth, Bruno Gobbis on Product Experimentation

December 11, 2025 @ 9:00AM PT

View Answers
Sheila Hara

Barracuda Sr. Director, Product Management, Sheila Hara on Product Experimentation

September 17, 2025 @ 10:00AM PT

View Answers
See All Upcoming AMA's

Top Product Management Mentors