Vice President and Technical Fellow | Data Science, Engineering | AI, Machine Learning, Controlled Experiments | Ex-Airbnb, Ex-Microsoft, Ex-Amazon
The A/B Testing Safety Net for Software Development. People often ask how to “sell” A/B testing internally. Zach Flynn recently wrote that experimentation culture should be guided, not forced by “laws” [1]. Guidance alone is not enough. You must show the value, and one of the best ways to encourage a “test everything” mentality with A/B testing is to show the value of the safety net. While most A/B content focuses on power, p-values, and finding winners, an overlooked benefit is operational: detecting egregious regressions and aborting quickly, shrinking the blast radius. When Microsoft Office for desktop moved from a 3-year release cycle to monthly releases, the key problem was turning off a bad piece of code after the client shipped. They adopted A/B testing for its kill-switch capability—safe deployments. When A/B testing was integrated, it was an easy step to also evaluate the value of features [2]. Every (good) engineering organization runs weekly postmortem (sometimes called AAR for After Action Review) to understand the root cause of outages and severe incidents and learn how to avoid them in the future. The questions to add to the postmortem form are: - Was this change behind an A/B test? While postmortem are blameless, people quickly learn that many more outages are associated with code deployed without an A/B test, driving adoption. - If yes, what guardrail metric should be added that would catch and auto-abort a similar issue? Remember the “other tail” of A/B testing: the safety net that lets you abort bad deployments fast! [1] https://lnkd.in/geZr2CmU [2] https://lnkd.in/gCSBHeTv #ABTesting #ExperimentGuide #DevOps #postmortem
Analytics Leader
1wRight on — A/B experimentation inherently brings both data and code governance into the development process. When implemented correctly, even a small 95/5 traffic split during deployment acts as a live validation stage, helping teams observe full user flows and data capture integrity before full rollout. As you said, skipping experiments often means debugging blindly later — turning postmortems into a hunt for a needle in a haystack.