AMA: Superhuman Director, Product Growth, Bruno Gobbis on Product Experimentation

I don't like the idea that an experiment is successful just because it "lifted conversions" by 5% today or during some weeks. Long-term impact should be measured using revenue key results (Cohort Retention Curves and Net Dollar Retention), as well as qualitative metrics (NPS, PMF question, user feedback). Below are two tools to start doing that: The Litmus Test: Did this experiment (or a combination of experiments) flatten the retention curve at a higher percentage for a specific cohort? If you ...Read More

That's a different (and more challenging) situation! And you could also face this, especially when experimenting on a journey where you don't have many users passing by, so this isn't only for early-stage companies. My take here is to go for "decision significance” rather than “statistical significance” -> which you probably won't have. At low data volumes, prefer to treat experiments primarily as structured learning vehicles rather than binary “win/lose” bets that you can set using X or Y up ...Read More

I believe the era of "growth hacking" (one-off tricks to spike numbers) is dead (or dying). The current trend is Systematization. Teams are moving toward building "permanent infrastructure"—data pipelines and continuous learning loops—rather than running isolated tests and seeking quick wins. Additionally, the binary wall between Sales-Led and Product-Led is collapsing into Hybrid Growth. We are seeing a massive trend in experiments that blend these motions: using product usage data to signal sa ...Read More

I focus on building it by making learning safer and more valuable than being “right.” That means people are rewarded for running good bets and sharing honest results, not for never failing. Encouraging unknowns and accepting we might probably get more wrongs than rights should be normalized on all levels and functions. A bonus, as a leader, is to put yourself vulnerable and share your own mistakes so your team can feel safe about doing the same. I believe that if people worry that a failed exper ...Read More

Think of your CEO as your most important “stakeholder" and one of your partners in the learning process. The job is to translate failed tests from “we missed the number” into “we just bought information that makes our next bet smarter, cheaper, and with more probability of being a win.”​ If you can't bridge the lessons from failures, it's just going to be a wasted investment of team resources, and then your CEO would be right to give the team an ear massage. Executives are usually fine with fail ...Read More

Good question! In the last few years, a big wave of new tools has transformed how we can approach product experimentation. Today, you can find great solutions that cover analytics, session replay, feature flags, A/B testing, and user surveys- the list can be pretty big here. Here’s the tech stack I primarily use now: Prototyping: I rely on tools like v0 for design, Lovable, and Cursor, which excels at “vibe coding” to create quick, interactive prototypes that capture user experience nuances. Age ...Read More

I love this subject! Start from a mindset shift: experiments are not about proving ideas right, they’re about reducing the most critical unknowns, fast and safely. Be honest about your knowledge regarding some challenge or opportunity. Don't think of the experiments as isolated; instead, combine them so you can better understand your customer - each experiment should yield new insights that help you improve your product/journey. Name the uncertainty your experiment must resolve (example: “Do new ...Read More