How to set alpha when you have underpowered experiments?

Jakub Linowski asked how to assign trust-level to an experiment corpus when some experiments have low power (post).

If you’re designing a new experiment, I strongly recommend designing it with sufficient power (e.g., over 80% for an MDE that’s at most 5%).

What about experiments that you previously ran that do not meet the bar?  In that case, I would lower the alpha (p-value threshold for calling something statistically significant) to control for the False Positive Risk (FPR), that is, the probability that the statistically significant result is a false positive.

The formula is not complicated, and we shared it in https://bit.ly/ABTestingIntuitionBusters:

 where

• β is the type-II error (usually 0.2 for 80% power).  You can estimate power by plugging an MDE of similar experiments (with suggested upper-bound of 5% for online experiments optimizing conversion as noted in this post. Here’s a simple Excel spreadsheet for that: https://bit.ly/powerFromN
• π is the prior probability of the null hypothesis, that is, P(H_0). Based on the Table 2 in https://bit.ly/ABTestingIntuitionBusters, I recommend using 85%, for A/B tests, that is, a success rate of about 15%.

Plugging in the above defaults, gives

Let look at a few examples:

• If your power is 80%, the formula suggests alpha of 0.015, lower than the industry standard of 0.05, but similar to the 0.01 threshold we used at Airbnb search after seeing many false positives. If your success rate is 35%, then set alpha to 0.045, similar to the industry standard.
• If your power is 50%, set alpha to 0.009
• If your power is 20%, set alpha to 0.004

These may be viewed as hard to achieve, but that’s the penalty you have to pay if you want to control the false positive risk.  Note that with low powered experiments, the treatment effect is still expected to be exaggerated.

Also note that there are several additional factors to check to increase trust.  These were discussed here.