Our open letter to the Cochrane Board dated February 16th points out that a series of rapid reviews on the effects of sundry non-pharmaceutical interventions undermined or ignored the rationale for the Cochrane Collaboration.
Cochrane authors and editors did not follow the rules nor the spirit of those who came together long ago to create the Cochrane Collaboration and kept going, whatever the costs, within the rules.
We considered four founding principles were breached:
- Collaboration
- Avoidance of duplication
- Use of best quality evidence
- Keeping reviews up to date
The rapid reviews that breached these principles included junk studies, such as models, and plugged the gap of the long-standing Cochrane review on the same topics (‘Physical interventions to interrupt or reduce the spread of respiratory viruses‘ (A122)). Review A122’s publication was delayed by six months, which saw the gap filled with reviews that included poor-quality evidence – if models can be called evidence.
We have over 40 years of experience working with and publishing Cochrane reviews. In that time, we have never observed the substitution of high quality evidence with an approach that could only be described as a race to the bottom.
One possible argument for the hasty production of reviews is that there was an emergency, and we needed something, anything, to direct decisions.
The accompanying editorial to A122, which eventually came out in November 2020, made this argument: when the chips are down, any old data will do. If this wasn’t enough, one of the editorial references was to a consensus statement published in a mega journal based on the ideology that governments must act — in doing so, they should push zero-Covid policies.
The fact that a consensus statement was cited to ‘contextualise’ the results of a systematic review of RCTs should make all involved think seriously about the critical thinking – or lack thereof – at the time.
It is worth reminding readers that A122 initially included all types of comparative studies. In its third update of 2020, the authors saw no reason to keep including lower-quality evidence and opted to focus on randomised controlled trials, as there were many of them from which to draw conclusions.
The problem is that trial results may not support the dominant narrative, which flipped to lockups, lockdowns, muzzles, the rule of six Hancocks, granny in the garden, following runners with drones on roads which have been deserted since the fall of the Roman empire, chopping off the bottom half of school doors and sundry crazies.
Observational studies and models will say what you want them to. As our antivirals series shows, everything can be manipulated, but comparative trials are more difficult to manipulate, especially if the trialists are competent and honest and follow the magic luminous path of their protocol.
Observational studies have their role, but should not be used to assess the effectiveness of interventions against respiratory viruses, as we have explained in a three part series last year.
When examining the decision-making process of the past, there were two opposing viewpoints. One side supported the prevailing belief in Government policies of suppression and used any available evidence to back up its claims. Policymakers were not able to differentiate between low-quality and high-quality evidence; therefore, this position went largely unchallenged. Overnight experts and opinions could shout from the rooftops their support for policies that at times were made up by inexperienced Government advisers. In addition, journals such as Cochrane produced large quantities of junk science reviews at scale and speed, further promoting the interventionist narrative.
The other was the need for high-quality evidence. However, when it came to non-pharmaceutical interventions, high-quality evidence not supporting the predominant narrative was suppressed, and its authors were attacked, silenced and censored.
Randomised evidence took a back seat when it came to deciding what to do. As a consequence, we got policies that were untried and untested, and we are still paying the price for them.
So, which is it to be? We are clear — and have always been — about which approach is correct. However, we are still unclear why so many defaulted to the low-quality junk science approach.
So, what should be done next time to prevent the pandemic race to the bottom that ends at the door of the modellers loaded with their assumptions?
Did our open letter get an answer or even an acknowledgement? No.
An emailed version got the following response:
In a further email, the Cochrane bureaucracy informed us that the Board had seen our letter but did not have any specific comments to make.
The Interim Chair’s point about making a comment using the appropriate facility on the page of any review is very important. It is the correct way to do things — and has been for the last 30 years — and was completely disregarded by Cochrane’s own Editor-in-Chief, as we shall discuss in the next post.
None of the four substantive points made in our letter to the board have been addressed. Herein lies the miserable story of the use of evidence during the pandemic. Although we are dismayed with the way things have gone, we believe that with some critical thinking, a few retractions and a better understanding of what constitutes high-quality evidence, we can turn the tide towards an evidence-based approach.
Dr. Carl Heneghan is the Oxford Professor of Evidence Based Medicine and Dr. Tom Jefferson is an epidemiologist based in Rome who works with Professor Heneghan on the Cochrane Collaboration. This article was first published on their Substack, Trust The Evidence, which you can subscribe to here.
To join in with the discussion please make a donation to The Daily Sceptic.
Profanity and abuse will be removed and may lead to a permanent ban.