AMA: Barracuda Sr. Director, Product Management, Sheila Hara on Product Experimentation

As a leader, my process for experimentation is really about setting clear expectations for my team and creating the right environment for good decisions. Framing the Experiment I expect my team to start with the decision we’re trying to inform rather than just “running a test.” Every experiment should have a clear hypothesis written in a Given–When–Then format. They should define success metrics up front—both primary and guardrail—so we know how to interpret the results. Alignment is key: before ...Read More

At an early stage, success isn’t always about big dashboards or perfect p-values—it’s about learning. I tell my team: if the experiment helps us answer a question we were stuck on, that’s a win. Sometimes that’s numbers, but often it’s just a handful of customer conversations, patterns in behavior, or even gut-checks validated by real users. Early on, I expect the team to focus less on “statistical significance” and more on decision significance—did this give us enough clarity to move forward wi ...Read More

For me, prioritization is about clarity on business impact, not just curiosity. Strategic Fit – I expect my team to propose experiments that tie back to our biggest product or business questions, not “nice-to-know” tests. If it doesn’t help us make a real decision on roadmap, it doesn’t go to the top of the list. Impact vs. Effort – We use a simple ICE (Impact, Confidence, Effort) or RICE (Reach, Impact, Confidence, Effort) framework to compare opportunities. High-impact, low-effort experiments ...Read More

This is a tricky question, because experimentation and shipping can feel at odds — and it really comes down to ruthless prioritization. It’s always a tension — teams want to learn and move fast at the same time. I remind my team: experiment when it helps you avoid costly mistakes, but don’t let perfect testing delay delivering real value to customers I expect my team to ask: what’s the decision we’re trying to make, and is an experiment the fastest way to get there? Sometimes the answer is yes — ...Read More

I set the expectation that experiment results aren’t just numbers on a slide — they’re a story about customer behavior and business impact. Structured Readouts – My team prepares readouts that always include the hypothesis, setup, results, and learnings. We use a consistent template so stakeholders know exactly what to expect. Balanced View – I expect them to go beyond “it worked/it didn’t.” We highlight both the quantitative results and the qualitative insights — often the “why” matters more th ...Read More

I’ve seen two models work, and both can be effective depending on company stage and culture. In some orgs, growth and core product are separate teams. Growth focuses on acquisition, conversion, and engagement experiments, while core product builds the foundation and long-term value. That model works when you need a sharp focus and fast iteration on funnel metrics. I’ve also worked in models where everyone owns growth. Instead of a dedicated growth team, the growth mindset is embedded across prod ...Read More

I’ve had the privilege of running experiments across very different industries, and each one has taught me something new. One of the most successful was what’s now Endicia Global Service. It started as just an idea to simplify international shipping, and I was honored to be one of the contributors to the concept, which is now patented. That was a case where experimentation created an entirely new revenue stream. At Walmart, I worked on delivery slot and capacity planning — and some of those expe ...Read More

For me, building a culture of experimentation starts with mindset. Marty Cagan often says that “the best teams don’t just build features, they solve problems.” I use that as a north star: experiments aren’t about vanity metrics, they’re about deeply understanding problems and learning what works. On failure, I lean into his reminder that “most ideas won’t work — and that’s not a bug, it’s the system working as intended.” I want my team to know that if 7 out of 10 experiments “fail,” that’s norma ...Read More

I remind my CEO that experiments aren’t about being right, they’re about reducing risk. If a test disproves our idea early, that’s a win — because we didn’t waste six months building the wrong thing. I frame failures as progress markers: We saved money/time by learning faster. We gained clarity on what customers don’t want, which is just as valuable as what they do. We narrowed the field so our next bets are smarter. I also lean on language CEOs appreciate: ROI, de-risking, velocity. Instead of ...Read More

Uncertainty and ambiguity are the whole point of experimentation — if the outcome were certain, we wouldn’t need to test. I coach my team to handle this in three ways: Define what we do know. Even in ambiguous spaces, we usually have assumptions or customer signals. Writing those down makes the unknowns clearer. Run smaller bets first. We don’t try to boil the ocean. Prototypes, fake-door tests, or concierge MVPs let us learn cheaply before scaling. Reframe success. In uncertainty, a “failed” ex ...Read More