Vice President and Technical Fellow | Data Science, Engineering | AI, Machine Learning, Controlled Experiments | Ex-Airbnb, Ex-Microsoft, Ex-Amazon
Test yourself. How many of Nils Stotz essential statistical concepts do you understand well? Several important concepts not mentioned, which are covered in my course, https://bit.ly/ABClassRKLI: - Statistical power (low power leads to high false positive risk and exaggeration of effects) - Twyman's law (other people also suggested; my summary at https://bit.ly/twymanLaw) - Triggering (improve the efficiency by removing unimpacted populations) - Leakage/interference (when assumptions are violated).
Statistical Concepts are Essential to Experimentation. Here's a Summary of the Top 9 in One Sentence Each ❤️💡 🧮 Sample Ratio Mismatch (SRM) A discrepancy in sample sizes across test groups, potentially leading to biased outcomes. 🔄 Simpson’s Paradox A phenomenon where a trend appears in several different groups of data but disappears or reverses when these groups are combined. 🏆 Overall Evaluation Criteria (OEC) Criteria that define the measures used to assess the success or failure of an experiment. 🎯 Goodhart’s Law When a measure becomes a target, it can no longer be used as a reliable indicator. 🔍 Heterogeneous Treatment Effects Variability in treatment effects among participants in different subgroups of an experiment. 📉 Signal Noise Ratio The measure of signal (meaningful data) to noise (random data) in experimental results. 🛑 P-hacking The practice of repeatedly trying different statistics or data configurations until significant results are obtained. 🔁 Regression to the Mean The phenomenon where extreme observations tend to return to their average on additional testing. 🧐 Multiple Comparison Problem The increased chance of erroneous inferences from making multiple statistical comparisons. Do you have a favorite concept? Do you want me to dive deeper into one of these concepts? Let me know in the comments :) P.S. Let me know in case you are missing a concept! #experimentation #ABtesting
Ron Thanks so much for sharing this! I really appreciate it!
Interesting
PhD in Statistics, Senior Data Scientist at Chime
1yThoughts on Goodhart's law? Seems rather negative and doesn't propose solutions. Gotta target at least something.