Think and act like a statistician.
Statistics is a core competency of any optimization program. Boost your stats skills with an actionable overview of how to run statistically rigorous experiments.
Learn about statistical significance, statistical error, and the concepts behind Optimizely’s Stats Engine:
- Best practices for running your experiments to statistical significance
- Tips for communicating results, optimizing low-traffic pages, using confidence intervals, and more
- How Optimizely is creating an always-valid view of statistical significance
Table Of Contents
Pg.4 - 1. Why we need statistics
pg 5 - 2. How statistics have been traditionally used
Pg 6 - 3. How statistics have (or haven't) adapted for the online world
- Misunderstanding of statistics leads to errors
- CalcTulating sample size
Pg 9 - 4. What you need to know about statistical error
Pg 10 - 5. How Optimizely is creating an always-valid results calculation
- Continuous monitoring error
- Multiple comparisons error
Pg 13 - 6. Tips for running a statistically sound experiment
Pg 14 - 7. How to communicate experiment results
- Confidence intervals
Pg 16 - 8. How to set your statistical significance threshold
Pg 18 - 9. How to reach statistical significance if you have low traffic
Pg 21 - 10. Statistical terms glossary
First 3 Pages
A Practical Guide to Statistics for Online Experiments
How anyone can think and act like a statistician
This is Pete Koomen. Just a quick thank-you for downloading this guide and taking the time to dive into a topic that we’re passionate about here at Optimizely.
We’ve spent the past few years building a company and a product that enables our customers to turn data into action. We want to make it possible for anyone, in any company, to use A/B testing and optimization as processes that will help them make decisions, reach their conversion goals, and transform their business.
Although we know you value data and hard facts when growing your business, you make intuition-driven decisions about your results. To make sure you make the best decisions, we’re committed to giving you the very best data. This means that the statistical underpinnings of our platform need to evolve to keep pace with how you want to use your results.
In this guide, we’re going to take things one step further. We’ll cover the essential ‘Stats IQ’ topics you need to take your A/B testing and optimization efforts to the next level. We’ll show you how to trust your test results, make recommendations, and get to significance more confidently than ever before.
We hope you’ll take some of these concepts back to your company, clients, and teammates to share them and help them to make informed decisions and get great test results.
Chief Technical Officer, Optimizely
Table of Contents
1. Why we need statistics
2. How statistics have been traditionally used
3. How statistics have (or haven’t) adapted for the online world
> Misunderstanding of statistics leads to errors
> Calculating sample size
4. What you need to know about statistical error
5. How Optimizely is creating an always-valid results calculation
> Continuous monitoring error
> Multiple comparisons error
6. Tips for running a statistically sound experiment
7. How to communicate experiment results
> Confidence intervals
8. How to set your statistical significance threshold
9. How to reach statistical significance if you have low traffic
10. Statistical terms glossary