Is there a reproducibility crisis in science? - Matt Anticole
In 2011, a team of physicists reported
a startling discovery:
neutrinos traveled faster
than the speed of light
by 60 billionths of a second
in their 730 kilometer trip from Geneva
to a detector in Italy.
Despite six months of double checking,
the bizarre discovery refused to yield.
But rather than celebrating
a physics revolution,
the researchers published a cautious paper
arguing for continued research in an
effort to explain the observed anomaly.
In time, the error was tracked to a single
incorrectly connected fiber optic cable.
This example reminds us that real
science is more than static textbooks.
Instead, researchers around the world
are continuously publishing
their latest discoveries
with each paper adding
to the scientific conversation.
Published studies
can motivate future research,
inspire new products,
and inform government policy.
So it's important that we have confidence
in the published results.
If their conclusions are wrong,
we risk time,
resources,
and even our health in the pursuit
of false leads.
When findings are significant,
they are frequently double-checked
by other researchers,
either by reanalyzing the data
or by redoing the entire experiment.
For example, it took repeated
investigation of the CERN data
before the timing error was tracked down.
Unfortunately, there are currently neither
the resources nor professional incentives
to double check the more than 1 million
scientific papers published annually.
Even when papers are challenged,
the results are not reassuring.
Recent studies that examined dozens
of published pharmaceutical papers
managed to replicate the results of
less than 25% of them.
And similar results have been found
in other scientific disciplines.
There are a variety of sources
for irreproducible results.
Errors could hide in their original
design, execution, or analysis of the data.
Unknown factors,
such as patients' undisclosed condition
in a medical study,
can produce results that are
not repeatable in new test subjects.
And sometimes, the second research group
can't reproduce the original results
simply because they don't know
exactly what the original group did.
However, some problems might stem
from systematic decisions
in how we do science.
Researchers,
the institutions that employ them,
and the scientific journals
that publish findings
are expected to produce
big results frequently.
Important papers can advance careers,
generate media interest,
and secure essential funding,
so there's slim motivation for researchers
to challenge their own exciting results.
In addition, little incentive exists
to publish results unsupportive
of the expected hypothesis.
That results in a deluge of agreement
between what was expected
and what was found.
In rare occasions, this can even lead
to deliberate fabrication,
such as in 2013, when a researcher
spiked rabbit blood with human blood
to give false evidence that
his HIV vaccine was working.
The publish or perish mindset
can also compromise academic journals'
traditional peer-review processes
which are safety checks
where experts examine submitted papers
for potential shortcomings.
The current system,
which might involve only one
or two reviewers,
can be woefully ineffective.
That was demonstrated in a 1998 study
where eight weaknesses were deliberately
inserted into papers,
but only around 25%
were caught upon review.
Many scientists are working toward
improving reproducibility in their fields.
There's a push to make researchers
raw data,
experimental procedures,
and analytical techniques
more openly available
in order to ease replication efforts.
The peer review process can also
be strengthened
to more efficiently weed out weak papers
prior to publication.
And we could temper the pressure
to find big results
by publishing more papers that fail
to confirm the original hypothesis,
an event that happens far more than
current scientific literature suggests.
Science always has, and always will,
encounter some false starts
as part of the collective acquisition
of new knowledge.
Finding ways to improve
the reproducibility of our results
can help us weed out those false starts
more effectively,
keeping us moving steadily toward
exciting new discoveries.
