Are Aha Moments useful?

This was a question asked in my last cohort of https://bit.ly/ABClassRKLI .

My answer is yes, as motivating goals, and (critically) when the underlying metrics are causally related to success.

Aha moments are related to the Eureka effect in psychology: suddenly understanding a problem or concept. In product growth, it’s when the user first experiences your product’s core value proposition [1].

Common examples: Facebook’s: “7 friends in 10 days” [2], Twitter’s “follow 30 accounts” [3], and Uber’s “ETA under 3 minutes” [4].

Many before me have highlighted that most of these are found by looking for patterns and represent correlations, which may not be causal (e.g., Mixpanel 's Magic numbers are an illusion [5]; Ben Newell 's everyone confused about correlation and causation [6]; Yuzheng Sun 's Why is aha moment wrong [7]).

I want to add two key points that I have not been emphasized enough:

1. The story, or the simple motivational line, is critical in organizations. Everyone knows that performance matters, but when we made it into this one-liner, it had much greater impact: "An engineer that improves server performance by 10 msec (that’s 1/30 of the speed that our eyes blink) more than pays for his (or her) fully-loaded annual costs. Every millisecond counts" [8]. When someone wanted to ship a feature that slowed Bing by 50msec, I asked them if they were willing to give up five heads: eh, no; the cost was very clear.

2. There must be a believed causal relationship to what the business cares about.  Two counterexamples, not just repeating the mantra that correlation != causation:

1. Office Errors: We found a positive correlation between loyalty (lower churn) and seeing error messages in Office. Starting to spew error messages will not reduce churn; heavy users of Office simply see more errors.
2. Wells Fargo cross-sells: The bank saw a correlation between using multiple products and customer value.  John Stumpf, the Chairman and CEO, wrote “Cross-sell is the result of serving our customers extraordinarily well” [9]. Opening accounts and signing up customers for products they don’t need, such as talking a homeless woman into opening six checking and savings accounts [10] does not create value like genuine multi-product usage. The practice ultimately resulted in $3 billion in penalties by the Department of Justice [11].

Summary: a simple motivating line can help, but ensure the metrics causally impact the business and avoid “internal fraud.” 

Thanks Dan Zigmond .

REFERENCES

[1] https://www.reforge.com/guides/define-your-aha-moment

[2] https://www.youtube.com/watch?v=raIUQP71SBU&t=1265s

[3] https://unbounce.com/cta-conf/Sean_Ellis_CTAConf2014.pdf

[4] https://andrewchen.com/wp-content/uploads/2022/01/ColdStartProb_9780062969743_AS0928_cc20_Final.pdf

[5] https://mixpanel.com/blog/magic-numbers-are-an-illusion

[6] https://medium.com/geckoboard-under-the-hood/how-facebooks-7-friends-in-10-days-got-everyone-confused-about-correlation-and-causation-25da4bb8220e

[7] https://medium.com/pragmatic-data-scientists/why-is-aha-moment-wrong-ed96ae493a01

[8] https://experimentguide.com Chapter 5. 

[9] https://www.citizen.org/wp-content/uploads/wells-fargo-king-of-cross-sell.pdf

[10] https://www.latimes.com/business/la-fi-wells-fargo-sale-pressure-20131222-story.html

[11] https://www.justice.gov/archives/opa/pr/wells-fargo-agrees-pay-3-billion-resolve-criminal-and-civil-investigations-sales-practices