What happens when an experiment fails? How do you pivot?

Answer

4 Answers

Growth Product Management

Willie Tran

Dropbox Group Product Manager, DocSend Growth | Formerly Mailchimp, Calendly • 3y

Ideally, you design experiments in a way that seek to answer a question regardless of the result. For example, "do users not adopt this feature because of discoverability or usability?" is a great question to be answered with an experiment. If the exeriment execution puts that feature front and center of the main page when a user logs in, and the experiment is flat, then you could be pretty sure it's a usability issue. If it's stat sig positive, then you can be sure it's a discoverability issue. ...Read More

1,984 Views
Deepti Pradeep

Adobe Senior Director of Product Management & Growth (Creative Cloud) • 1y

More often than not, an experiment fails. Failed experiments are the biggest sources of learning. Was your hypotheses proven wrong? Was the test design not pushing it far enough? Were there other data anomalies? Where there any positive signals in a subgroup? One needs to be careful in slicing the data by many cuts, as the validity of an AB test is largely dependent on its effect size. Cut it too much, and you might be cherry picking the data thereby losing its meaning for decision making, or wo ...Read More

3,225 Views
Nicolas Liatti

Adobe Senior Director of Product Management, 3D Category • 2y

Experiments are made to fail. It is know in the tech industry that around 85% or experiments fail.

If we'd know in advance what would work, we would just go straight for it and not make any experiments.

That's why it's critical to really treat this as an experiment - and not a full solution - and be able to move fast to the next one.

You will usually get success after experimenting and failing tons of time.

580 Views
Michael McNicholas

Theowls.com Founder • 3y

If the experiment results were flat or if the new variation performed worse, I would suggest three steps in deciding how to pivot Confirm it failed. Dig into the data Get user / qualitiative feedback Confirm it failed. Did you run the test for long enough to get statistical significance? (How long does an AB test calculator say you should run the test?) The most common issue with AB testing is people calling the test with not enough data. Does the experience or part of the site you were test ...Read More

320 Views