In a recent survey of over 1,500 scientists, more than 70 percent of them reported having been unable to reproduce other scientists’ findings at least once. Roughly half of the surveyed scientists ran into problems trying to reproduce their own results.
Reproducibility of findings is a core foundation of science. If scientific
results only hold true in some labs, how can researchers feel confident
about their discoveries? How can society put evidence-based policies into
place if the evidence is unreliable?
Recognition of this “crisis” has prompted calls for reform. Researchers
are feeling their way, experimenting with different practices meant to help
distinguish solid science from irreproducible results. Some are even starting to reevaluate how choices are made about what research actually gets
tackled. Breaking innovative new ground is flashier than revisiting already
published research. Does prioritizing novelty naturally lead to this point?
Incentivizing the wrong thing?
One solution to the reproducibility crisis could be simply to conduct
lots of replication studies. For instance, the scientific journal eLife is participating in an initiative to validate and reproduce important recent findings
in the field of cancer research. The first set of these “rerun” studies was
recently released and yielded mixed results. The results of two of five research studies were reproducible, one was not and two additional studies
did not provide definitive answers.
But there’s at least one major obstacle to investing time and effort in
this endeavor: the quest for novelty. The prestige of an academic journal
depends at least partly on how often the research articles it publishes are
cited. Thus, research journals often want to publish novel scientific findings
that are more likely to be cited, not necessarily the results of newly rerun
A study of clinical trials published in medical journals found the most
prestigious journals prefer publishing studies considered highly novel over
those that have the most solid numbers backing up the claims. Funding
agencies such as the National Institutes of Health ask scientists who review
research grant applications to provide an “innovation score” to prioritize
funding for the most innovative work. And scientists of course notice
these tendencies—one study found the use of positive words like “novel,”
“amazing,” “innovative” and “unprecedented” in paper abstracts and titles
increased almost ninefold between 1974 and 2014.
Genetics researcher Barak Cohen at Washington University in St. Louis
recently published a commentary analyzing this growing push for novelty.
He suggests that progress in science depends on a delicate balance between
novelty and checking the work of other scientists. When rewards such as
funding of grants or publication in prestigious journals emphasize novelty
at the expense of testing previously published results, science risks developing cracks in its foundation.
Houses of brick, mansions of straw
Cancer researcher William Kaelin Jr., a recipient of the 2016 Albert
Lasker Award for Basic Medical Research, recently argued for fewer “
mansions of straw” and more “houses of brick” in scientific publications.
One of his main concerns is that scientific papers now inflate their
claims to emphasize their novelty and the relevance of biomedical research
for clinical applications. By exchanging depth of research for breadth of
claims, researchers may be at risk of compromising the robustness of the
work. By claiming excessive novelty and impact, researchers may under-
mine its actual significance because they may fail to provide solid evidence
for each claim.
Kaelin even suggests that some of his own work from the 1990s, which
transformed cell biology research by discovering how cells can sense oxygen, may have struggled to get published today.
Prestigious journals often now demand complete scientific stories, from
basic molecular mechanisms to proving their relevance in various animal
models. Unexplained results or unanswered questions are seen as weak-nesses. Instead of publishing one exciting novel finding that is robust, and
which could spawn a new direction of research conducted by other groups,
researchers now spend years gathering a whole string of findings with
broad claims about novelty and impact.
Balancing fresh findings and robustness
A challenge for editors and reviewers of scientific manuscripts is assessing the novelty and likely long-term impact of the work they’re reading.
The eventual importance of a new, unique scientific idea is sometimes difficult to recognize even by peers who are grounded in existing knowledge.
Many basic research studies form the basis of future practical applications.
One recent study found that of basic research articles that received at least
one citation, 80 percent were eventually cited by a patent application. But
it takes time for practical significance to come to light.
A collaborative team of economics researchers recently developed an
unusual measure of scientific novelty by carefully studying the references of
a paper. They ranked a scientific paper as more novel if it cited a diverse
combination of journals. For example, a scientific article citing a botany
journal, an economics journal and a physics journal would be considered
very novel if no other article had cited this combination of varied references before.
This measure of novelty allowed them to identify papers that were
more likely to be cited in the long run. But it took roughly four years for
these novel papers to start showing their greater impact. One may disagree
with this particular indicator of novelty, but the study makes an important
point: it takes time to recognize the full impact of novel findings.
Realizing how difficult it is to assess novelty should give funding agencies, journal editors and scientists pause. Progress in science depends on
new discoveries and following unexplored paths—but solid, reproducible
research requires an equal emphasis on the robustness of the work. By restoring the balance between demands and rewards for novelty and robustness, science will achieve even greater progress.
Associate Professor of Medicine
University of Illinois at Chicago
Editor’s Note: This article was republished via The Conversation.
Author photo courtesy of Roberta Dupuis-Devlin/UIC. We invite
you to submit your personal commentaries to Last Word on topics
that impact your work and affect the overall industry. Please send
your commentaries to Michelle Taylor, Editor-in-Chief, at michelle.
Novelty in Science:
Real Necessity or