Statistical Models Explored and Explained
4 likes2,465 views
Sara Vafi and Shana Rusonis presented on statistical models for analyzing A/B test results. They discussed the differences between Bayesian and frequentist statistics, and between average error control and all error control. Bayesian A/B testing uses average error control and models results as distributions, while frequentist A/B testing aims for all error control by controlling the false positive rate for each experiment. Small improvements are hardest to detect, and realistic A/B tests may require more data than average error control can provide on its own. Optimizely's Stats Engine takes a blended Bayesian-frequentist approach for more accurate and timely results.
Statistical Models Explored and Explained
- 1. Speakers Statistical Models, Explored and Explained Sara Vafi, Stats Expert, Optimizely Shana Rusonis, Product Marketing, Optimizely
- 2. Today’s Speakers Sara Vafi Shana Rusonis
- 3. Housekeeping • We’re recording! • Slides and recording will be emailed to you tomorrow • Time for questions at the end
- 4. Agenda • Bayesian & Frequentist Statistics • Error Control - Average vs. All Error Control • Bayes Rule • Benefits & Risks • Optimizely Stats Engine • Q&A
- 5. Why Do We Experiment? ● Experimentation is essential for learning ● Try new ideas without fear of failure ● Give your business a signal to act on in a sea of noisy data
- 6. What’s most Important to You? ● Running experiments quickly ● But also reporting on results accurately ● When not all statistical solutions are created equal
- 7. Types of Statistical Methods Bayesian OR Frequentist
- 8. Bayesian Statistics ● Bayesian statistics take a more bottom-up approach to data analysis ● Our parameters are unknown ● The data is fixed ● There is a prior probability ● “Opinion-based”
- 9. “A Bayesian is one who, vaguely expecting a horse, and catching a glimpse of a donkey, strongly believes he has seen a mule.” Source
- 10. Frequentist Statistics ● Frequentist arguments are more counter-factual in nature ● Parameters remain constant during the repeatable sampling process ● Resemble the type of logic that lawyers use in court ● ‘Is this variation different from the control?’ is a basic building block of this approach.
- 11. Example Dan & Pete Rolling a 6-Sided Die Scenario: ● Pete will roll a die and the outcome can either be 1, 2, 3, 4, 5, or 6 ● If Pete rolls a 4, he will give Dan $1 million If Dan was a Bayesian statistician, how would he react? If Dan was a Frequentist statistician, how would he react?
- 12. Example Probability of the sun exploding Source
- 13. Error Control
- 14. Error Control Explained ● The likelihood that the observed result of an experiment happened by chance, rather than a change that you introduced ● When we set the statistical significance on an experiment to 90%, that means there's a 10% chance of a statistical error, or a 1 in 10 chance that the result happened by chance
- 15. Average Error Control ● Corresponds to Bayesian A/B Testing ● Less useful for iterating on test results ● Harder to learn from individual experiments with confidence
- 16. All Error Control ● Corresponds to Frequentist A/B Testing ● Any experiment will have less than a 10% chance of a mistake ● Rate of errors is 1 in 10
- 17. Average Error Control vs. All Error Control ● Average error control leads to lower accuracy for small improvements ● All error control is accurate for all users ● There are certain cases where average error control is an appropriate alternative
- 18. Error Rates for Experiments
- 19. Bayes Rule
- 20. Average Error Control & Bayesian A/B Testing ● Requires two sources of randomness ○ Randomness or “noise” in the data ○ The makeup of the “typical” experiment group ● Distribution over experiment improvements
- 21. Different Beliefs in Composition of ‘Typical’ Experiments
- 22. Bayes Rule
- 23. Bayes Rule & Bayesian A/B Testing
- 24. Bayes Rule & Average Error Value
- 25. Recap Average Error Control Bayesian A/B Testing Prior Distributions Bayes Rule
- 26. All Error Control is Frequentist A/B Testing ● All error control corresponds to Frequentist AB testing ● We want to aim to control the false positive rate ● Chance an experiment is either called a winner or loser
- 27. Benefits & Risks
- 28. Benefits of Bayesian A/B Testing ● Average error control can be very attractive ● Helps solve the “peeking” problem ● Average error control is fast
- 29. Risks of Bayesian A/B Testing ● It’s more appealing but it’s risky in practice ● Smaller improvement experiments with fast results = high risk ● Higher error rate than the method actually suggests
- 30. Benefits of Frequentist A/B Testing ● This type of test will make fewer mistakes on experiments with non-zero improvements ● The rate of errors will be less than 1 in 10 ● Option to speed up experimentation by using a prior
- 31. Learning from A/B Tests
- 32. Learning from A/B Tests
- 33. Risk Involved with Typical Realistic Experiments
- 34. Realistic Bayesian A/B Tests vs. Stats Engine
- 35. ● The hardest experiments to call correctly are those with small improvements ● A/B testing in the wild is not easy ● We need more and more data in order to achieve average error control on realistic experiments So what does this mean?
- 36. Stats Engine
- 37. Stats EngineTM Results are valid whenever you check Avoid costly statistics errors Measure real-time results with confidence
- 38. Key Takeaways ● Bayesian vs. Frequentist methods ● All error control vs. average error control ● Blended approach leads to greater confidence
- 39. QUESTIONS?
- 40. THANK YOU!