ab testing statistics

Free ebook: A Practical Guide to Statistics for Online Experiments: How anyone can think and act like a statistician

Think and act like a statistician. Statistics is a core competency of any optimization program. Boost your stats skills with an actionable overview of how to run statistically rigorous experiments. Learn about statistical significance, statistical error, and the concepts behind Optimizely’s Stats Engine: - Best practices for running your experiments to statistical significance - Tips for communicating results, optimizing low-traffic pages, using confidence intervals, and more - How Optimizely is creating an always-valid view of statistical significance

Table Of Contents

Pg.4 - 1. Why we need statistics pg 5 - 2. How statistics have been traditionally used Pg 6 - 3. How statistics have (or haven't) adapted for the online world - Misunderstanding of statistics leads to errors - CalcTulating sample size Pg 9 - 4. What you need to know about statistical error Pg 10 - 5. How Optimizely is creating an always-valid results calculation - Continuous monitoring error - Multiple comparisons error Pg 13 - 6. Tips for running a statistically sound experiment Pg 14 - 7. How to communicate experiment results - Confidence intervals Pg 16 - 8. How to set your statistical significance threshold Pg 18 - 9. How to reach statistical significance if you have low traffic Pg 21 - 10. Statistical terms glossary

First 3 Pages

A Practical Guide to Statistics for Online Experiments How anyone can think and act like a statistician Hello, This is Pete Koomen. Just a quick thank-you for downloading this guide and taking the time to dive into a topic that we’re passionate about here at Optimizely. We’ve spent the past few years building a company and a product that enables our customers to turn data into action. We want to make it possible for anyone, in any company, to use A/B testing and optimization as processes that will help them make decisions, reach their conversion goals, and transform their business. Although we know you value data and hard facts when growing your business, you make intuition-driven decisions about your results. To make sure you make the best decisions, we’re committed to giving you the very best data. This means that the statistical underpinnings of our platform need to evolve to keep pace with how you want to use your results. In this guide, we’re going to take things one step further. We’ll cover the essential ‘Stats IQ’ topics you need to take your A/B testing and optimization efforts to the next level. We’ll show you how to trust your test results, make recommendations, and get to significance more confidently than ever before. We hope you’ll take some of these concepts back to your company, clients, and teammates to share them and help them to make informed decisions and get great test results. Cheers, Pete Koomen Chief Technical Officer, Optimizely Table of Contents 1. Why we need statistics 2. How statistics have been traditionally used 3. How statistics have (or haven’t) adapted for the online world > Misunderstanding of statistics leads to errors > Calculating sample size 4. What you need to know about statistical error 5. How Optimizely is creating an always-valid results calculation > Continuous monitoring error > Multiple comparisons error 6. Tips for running a statistically sound experiment 7. How to communicate experiment results > Confidence intervals 8. How to set your statistical significance threshold 9. How to reach statistical significance if you have low traffic 10. Statistical terms glossary