AMA: Chainguard Former Senior Director of Product Management, Julian Dunn on Product Experimentation

Bear in mind that I am answering questions about product experimentation as a PM on B2B SaaS products who is not chiefly concerned with "growth product". In my world, some of the non-intuitive advice I have to provide about experimentation amounts to the following: Unless experimentation somehow is cheap (at which point I tend to question its concomitant impact), sometimes it's easier and more productive to just build the feature without alternatives, and see what happens. This tends to be more ...Read More

There is no one-size-fits-all process for experimentation; I tend to think of experimentation as a mindset rather than a rigid, dogmatic process that needs to be designed and followed. (Such processes are not appropriate for high-ambiguity situations anyway, which is where experimentation has its biggest potential impact.) I do have some tips for ensuring that experiments are aligned as much as possible to the highest impact areas, because you can always experiment yourself to death a/k/a analys ...Read More

A CPO I worked for once told me that qualitative data is still data. (And this person had previously worked for Amazon, a notoriously quantitative data-driven company!) Rigorous, statistically-significant data from experiments is often less obtainable than one would think, and even if it's obtainable, sometimes the juice is not worth the squeeze, e.g. you have to collect so many samples and run the experiment for so long that the experiment's decision-making effectiveness has waned because you h ...Read More

I am firmly in the camp of shipping features quickly as the highest priority! (Well -- shipping the highest incremental customer value possible as expressed through features or improvements. Not features for features sake.) So I would say that we want to save our experiments for scenarios where we really need to de-risk them before building, e.g. even building the MVP is costly for whatever reason; getting it wrong could result in reputational damage; the decision is a one-way door (not easily r ...Read More

Less important than the format of communications is the content of communicating results to stakeholders. The following advice applies no matter whether the experiment is small and you plan to just report its results in a Slack thread, or the experiment is large / very impactful and calls for an entire slide deck. Remind the audience why we ran this experiment: what was ambiguous, why we didn't just decide to YOLO and build our preferred option, and why we decided to de-risk it by running an exp ...Read More