A personal take on science and society

World view By Dorothy

Bishop

How scientists can stop

fooling themselves
Sampling simulated data can reveal common people to underestimate just how noisy small samples can
ways in which our cognitive biases mislead us. be, and hence to conduct studies that lack the statistical
power needed to detect an effect.
Researchers

T
Nor do researchers appreciate that the significance of a
he past decade has seen a raft of efforts to need to build result as expressed in a P value depends crucially on con-
encourage robust, credible research. Some lifelong text. The more variables you explore, the more likely it is
focus on changing incentives, for example
habits to that you’ll find a spuriously ‘significant’ value. For instance,
by modifying promotion and publication cri- if you test 14 metabolites for association with a disorder,
teria to favour open science over sensational avoid being then your probability of finding at least one P value below
breakthroughs. But attention also needs to be paid to led astray by 0.05 — a commonly used threshold of statistical signifi-
individuals. All-too-human cognitive biases can lead us confirmation cance — by chance is not 1 in 20, but closer to 1 in 2.
to see results that aren’t there. Faulty reasoning results in How can we instil an understanding of this? One thing
shoddy science, even when the intentions are good.
bias.” is clear: conventional training in statistics is insufficient,
Researchers need to become more aware of these pitfalls. or even counterproductive, because it might give the user
Just as lab scientists are not allowed to handle dangerous misplaced confidence. I’m experimenting with an alterna-
substances without safety training, researchers should not tive approach: generating simulated data that students can
be allowed anywhere near a P value or similar measure of subject to various kinds of statistical analysis. I use this to
statistical probability until they have demonstrated that teach two crucial concepts.
they understand what it means. First, if presented with null data sets (such as random
We all tend to overlook evidence that contradicts our numbers), students rapidly discover how easy it is to find
views. When confronted with new data, our pre-existing false results that seem statistically ‘significant’. Researchers
ideas can cause us to see structure that isn’t there. This have to learn that the interpretation of a P value is very
is a form of confirmation bias, whereby we look for and different when their question is “Is A associated with B?”
recall information that fits with what we already think. It from when it is “For variables A, B, C, D and E, are there any
can be adaptive: humans need to be able to separate out correlations where P < 0.05?” Asking whether a particular
important information and act quickly to get out of danger. metabolite is associated with a disease is not the same as
But this filtering can lead to scientific error. searching a set of metabolites to see whether any are asso-
Physicist Robert Millikan’s 1913 measurement of the ciated with it. The latter requires much more stringent
charge on the electron is one example. Although he claimed testing.
that his paper included all data points from his famous Simulated data also provide insights when samples come
oil-drop experiment, his notebooks revealed other, unre- from two ‘populations’ with different means. Students
ported, data points that would have changed the final rapidly learn that, with small sample sizes, an experiment
value only slightly, but would have given it a larger statis- might be useless for revealing even a moderate difference.
tical error. There has been debate over whether Millikan A 30-minute session of data simulation can leave research-
intended to mislead his readers. But it is not uncommon for ers stunned when they understand the implications.
honest individuals to suppress memories of inconvenient Researchers need to build lifelong habits to avoid being
facts (R. C. Jennings Sci. Eng. Ethics 10, 639–653; 2004). led astray by confirmation bias. Observations that are con-
A different type of limitation promotes misunderstand- trary to our expectations need special attention. In 1876,
ings in probability and statistics. We’ve long known that Charles Darwin said that he made it a habit “whenever a
people have trouble grasping the uncertainty inherent in published fact, a new observation or thought came across
small samples (A. Tversky and D. Kahneman Psychol. Bull. me, which was opposed to my general results, to make a
76, 105–110; 1971). As a topical example, suppose 5% of the memorandum of it without fail and at once: for I had found
population is infected with a virus. We have 100 hospitals by experience that such facts and thoughts were far more
that each test 25 people, 100 hospitals that test 50 people apt to escape from the memory than favourable ones”.
and 100 that test 100 people. What percentage of hospi- I myself have experienced this. When writing literature
Dorothy Bishop is
ROBERT TAYLOR PHOTOGRAPHY

tals will find no cases, and wrongly conclude the virus has reviews, I have been shocked to realize that I had com-
disappeared? The answer is 28% of the hospitals testing an experimental pletely forgotten to mention papers that run counter to
25 people, 8% of those testing 50 people and 1% of those psychologist at my own instincts, even though the papers had no particular
testing 100. The average number of cases detected by the the University of flaws. I now make an effort to list them.
hospitals will be the same regardless of the number tested, Oxford, UK We all find it difficult to see the flaws in our own work —
but the range is much greater with a small sample. e-mail: dorothy. it’s a normal part of human cognition. But by understand-
This non-linear scaling is hard to grasp intuitively. It leads [email protected] ing these blind spots, we can avoid them.

Nature | Vol 584 | 6 August 2020 | 9

©
2
0
2
0
S
p
r
i
n
g
e
r
N
a
t
u
r
e
L
i
m
i
t
e
d
.
A
l
l
r
i
g
h
t
s
r
e
s
e
r
v
e
d
.