Reproducibility is the most fundamental yardstick in science. If a result can’t be replicated, it doesn’t count as science.
Yet in recent years, there has been much talk of a ‘replication crisis’. Many results that we assumed were robust simply cannot be replicated. The term is typically used in the context of psychology and medicine, though it may apply to other fields as well.
So how much of science is reproducible? One way of tackling this question is to select a large number of studies from a particular field and then attempt to replicate them. This has been done several times.
A 2012 paper was only able to replicate 11% of 53 studies from pre-clinical cancer. A 2015 paper was only able to replicate 36% of 97 studies from psychology. A 2018 paper did slightly better, replicating 54% of 28 studies from that field. A 2016 paper was able to replicate 60% of 60% of 100 economics experiments. Another 2018 paper was able to replicate 62% of 21 social science experiments.
These numbers are sobering. But there’s an important caveat: the ‘studies to be replicated’ were selected somewhat arbitrarily, so the corresponding percentage can’t be taken as representative of the entire field.
Another way of tackling the question above is to simply ask researchers what percentage of the studies in your field can be replicated – a sort of ‘wisdom of the crowds’ approach.
This was done in a 2016 survey by the journal Nature. They got 1,500 responses – the vast majority from currently-working scientists. Respondents were asked, “In your opinion, what proportion of published work in your field is reproducible?”
The highest figure – 72% – was found in physics. The lowest figure – 52% – was found in “other” (which I suspect is mostly social scientists). Environmental science and medicine had intermediate figures – both 58%. Chemistry was a little higher at 65%. (Answers did not differ substantially between students and working scientists.)
These figures are again sobering. According to researchers themselves, close to half of published work in medicine, social science and environmental science cannot be replicated. Unsurprisingly, more ‘objective’ fields like physics and chemistry are perceived to have higher rates of replicability.
Overall, the two methods yield similar findings: a large percentage of results in more ‘subjective’ – dare one say ‘politicized’ – fields are not reproducible.
To join in with the discussion please make a donation to The Daily Sceptic.
Profanity and abuse will be removed and may lead to a permanent ban.
Professor Peter Ridd, Physics Professor at James Cook University in Queensland, was essentially sacked because he pointed out the fact that much of the research suggesting the Great Barrier Reef was in dire straits was not replicatable. So not only is much research unable to be replicated, but to point out this fact is a cardinal sin.
Then again, truth is a subjective notion.
Question “the science”, threaten the grant monies, and the pack will turn on you.
https://www.netzerowatch.com/peter-ridd-scientific-misconduct-at-james-cook-university-confirms-my-worst-fears/
Kinda like when the folks at “Science Based Medicine” dot com instantly turned on their once beloved John Ioannidis in 2020 the moment he refused to toe the party line in regards to the pandemic.
Wash your mouth out with soap, Noah. Reproducibility is so last century in “The Science”.
Consensus is the new metric.
Indeed. Bridges will collapse and people will swear blind they are still standing. Someone wrote a story about something very similar, a long time ago…
Looks like John Ioannidis, the most frequently cited scientist in modern history, was right once again, as usual. I mean, is he EVER wrong about anything? He is the new Aristotle, basically.
“Reproducibility is the most fundamental yardstick in science” only applies to sciences where reproducibility is possible. This means that “If a result can’t be replicated, it doesn’t count as science” is an error.
Everyone makes the mistake of referring to “science” as if there was such a thing and to “the scientific method” whereas there are only sciences and there is no single “the scientific method“.
The problem is the way the concept of “science” and “the scientific method” are taught in schools and universities as if “science” and reductionism are synonyms.
Consider astronomy and space physics. Both are observational – no experiments are possible and depend upon making large numbers of observations to derive conclusions. Meteorology is an example of a “science” which can only forecast and not predict – it has a limited “prediction horizon” measured normally in a few days.
All sciences vary in the reliability of information they generate from so-called “exact” sciences [literally only some branches of physics and chemistry] and the “inexact” sciences – everything else.
For a detailed explanation see “Medicine is not science“