Our open letter to the Cochrane Board dated February 16th points out that a series of rapid reviews on the effects of sundry non-pharmaceutical interventions undermined or ignored the rationale for the Cochrane Collaboration.
Cochrane authors and editors did not follow the rules nor the spirit of those who came together long ago to create the Cochrane Collaboration and kept going, whatever the costs, within the rules.
We considered four founding principles were breached:
- Collaboration
- Avoidance of duplication
- Use of best quality evidence
- Keeping reviews up to date
The rapid reviews that breached these principles included junk studies, such as models, and plugged the gap of the long-standing Cochrane review on the same topics (‘Physical interventions to interrupt or reduce the spread of respiratory viruses‘ (A122)). Review A122’s publication was delayed by six months, which saw the gap filled with reviews that included poor-quality evidence – if models can be called evidence.
We have over 40 years of experience working with and publishing Cochrane reviews. In that time, we have never observed the substitution of high quality evidence with an approach that could only be described as a race to the bottom.
One possible argument for the hasty production of reviews is that there was an emergency, and we needed something, anything, to direct decisions.
The accompanying editorial to A122, which eventually came out in November 2020, made this argument: when the chips are down, any old data will do. If this wasn’t enough, one of the editorial references was to a consensus statement published in a mega journal based on the ideology that governments must act — in doing so, they should push zero-Covid policies.
The fact that a consensus statement was cited to ‘contextualise’ the results of a systematic review of RCTs should make all involved think seriously about the critical thinking – or lack thereof – at the time.
It is worth reminding readers that A122 initially included all types of comparative studies. In its third update of 2020, the authors saw no reason to keep including lower-quality evidence and opted to focus on randomised controlled trials, as there were many of them from which to draw conclusions.
The problem is that trial results may not support the dominant narrative, which flipped to lockups, lockdowns, muzzles, the rule of six Hancocks, granny in the garden, following runners with drones on roads which have been deserted since the fall of the Roman empire, chopping off the bottom half of school doors and sundry crazies.
Observational studies and models will say what you want them to. As our antivirals series shows, everything can be manipulated, but comparative trials are more difficult to manipulate, especially if the trialists are competent and honest and follow the magic luminous path of their protocol.
Observational studies have their role, but should not be used to assess the effectiveness of interventions against respiratory viruses, as we have explained in a three part series last year.
When examining the decision-making process of the past, there were two opposing viewpoints. One side supported the prevailing belief in Government policies of suppression and used any available evidence to back up its claims. Policymakers were not able to differentiate between low-quality and high-quality evidence; therefore, this position went largely unchallenged. Overnight experts and opinions could shout from the rooftops their support for policies that at times were made up by inexperienced Government advisers. In addition, journals such as Cochrane produced large quantities of junk science reviews at scale and speed, further promoting the interventionist narrative.
The other was the need for high-quality evidence. However, when it came to non-pharmaceutical interventions, high-quality evidence not supporting the predominant narrative was suppressed, and its authors were attacked, silenced and censored.
Randomised evidence took a back seat when it came to deciding what to do. As a consequence, we got policies that were untried and untested, and we are still paying the price for them.
So, which is it to be? We are clear — and have always been — about which approach is correct. However, we are still unclear why so many defaulted to the low-quality junk science approach.
So, what should be done next time to prevent the pandemic race to the bottom that ends at the door of the modellers loaded with their assumptions?
Did our open letter get an answer or even an acknowledgement? No.
An emailed version got the following response:
In a further email, the Cochrane bureaucracy informed us that the Board had seen our letter but did not have any specific comments to make.
The Interim Chair’s point about making a comment using the appropriate facility on the page of any review is very important. It is the correct way to do things — and has been for the last 30 years — and was completely disregarded by Cochrane’s own Editor-in-Chief, as we shall discuss in the next post.
None of the four substantive points made in our letter to the board have been addressed. Herein lies the miserable story of the use of evidence during the pandemic. Although we are dismayed with the way things have gone, we believe that with some critical thinking, a few retractions and a better understanding of what constitutes high-quality evidence, we can turn the tide towards an evidence-based approach.
Dr. Carl Heneghan is the Oxford Professor of Evidence Based Medicine and Dr. Tom Jefferson is an epidemiologist based in Rome who works with Professor Heneghan on the Cochrane Collaboration. This article was first published on their Substack, Trust The Evidence, which you can subscribe to here.
To join in with the discussion please make a donation to The Daily Sceptic.
Profanity and abuse will be removed and may lead to a permanent ban.
“One possible argument for the hasty production of reviews is that there was an emergency, and we needed something, anything, to direct decisions”
How is that when the only virus was the media!
“One possible argument for the hasty production of reviews is that there was an emergency, and we needed something, anything, to direct decisions. “
Grandma’s dictum: decide in haste, repent at leisure.
Two things medics are often taught:
Don’t just do something! Stand there.
50% of what we teach you will turn out to be wrong. Trouble is we don’t know which 50%.
and from the military:
Observe, Orient, Decide, Act.
The actual decisions during ‘Covid’ were more:
We must be seen to be doing something! Whaddayagot?
It’s the same with NET Zero, twenty years on.
As the chair of a political committee might say: “Something must be done. This is something, so lets do it”.
Here is an extract from Factfullness by Hans Rosling and the dangers of the Urgency Instinct. What’s the relevance? It’s about saving lives again and the law of unintended consequences.
Chapter 10 – The Urgency Instinct
“If it’s not contagious, then why did you evacuate your children and wife?” asked the mayor of Nacala, eyeing me from a safe distance behind his desk. Out the window, a breathtaking sun was setting over Nacala district and its population of hundreds of thousands of extremely poor people, served by just one doctor – me.
Earlier in the day I had arrived back in the city from a poor coastal area in the north named Memba. There I had spent two days using my hands to diagnose hundreds of patients with a terrible, unexplained disease that had completely paralyzed their legs within minutes of onset and, in severe cases, made them blind. And the mayor was right; I wasn’t 100% sure it was not contagious. I hadn’t slept the previous night but had stayed up, pouring over my medical textbook, until I had finally concluded that the symptoms I was seeing had not been described before. I’d guessed this was some kind of poison rather than anything infectious, but I couldn’t be sure, and I had asked my wife to take our young children and leave the district.
Before I could figure out what to say, the mayor said, “If you think it could be contagious, I must do something. To avoid a catastrophe, I must stop the disease from reaching the city.”
The worst-case scenario had already unfolded in the mayor’s mind, and immediately spread to mine.
The mayor was a man of action. He stood up and said, “Should I tell the military to set up a roadblock and stop the buses from the north?”
“Yes,” I said. “I think it’s a good idea. You have to do something.”
The mayor disappeared to make some calls.
When the sun rose over Memba the next morning, some 20 women and their youngest children were already up, waiting for the morning bus to take them to the market in Nacala to sell their goods. When they learned the bus had been cancelled, they walked down to the beach and asked the fishermen to take them by the sea route instead. The fishermen made room for everyone in their small boats, probably happy to be making the easiest money of their lives as they sailed south along the coast.
Nobody could swim and when the boats capsized in the waves, all the mothers and children and fishermen drowned.
That afternoon I headed north again, past the roadblock, to continue to investigate the strange disease. As I drove through Memba I came across a group of people lining up on the roadside dead bodies they had pulled out of the sea. I ran down to the beach but it was too late. I asked a man carrying the body of a young boy, “Why were all these children and mothers out in those fragile boats?”
“There was no bus this morning.” he said. Several minutes later I could not still barely understand what I had done. Still today I can’t forgive myself. Why did I have to say to the mayor, “You must do something”?
I couldn’t blame these tragic deaths on the fisherman. Desperate people who need to get to market of course take the boat when the city authorities for some reason block the road.
I have no way to tell you how I carried on with the work I had to do that day and in the days afterward. And I didn’t talk about this to anyone else for 35 years.
Fourteen years later, in 1995, the ministers in Kinshasa, the capital of DR Congo, heard that there was an Ebola outbreak in the city of Kitwik. They got scared. They felt they had to do something. They set up a roadblock. Again, there were unintended consequences. Feeding the people in the capital became a major problem because the rural area that had always supplied most of their processed cassava was on the other side of the disease-stricken area. The city was hungry and started buying all it could from it’s second largest food producing area. Prices skyrocketed, and guess what? A mysterious outbreak of paralyzed legs and blindness followed.
Nineteen years after that, in 2014, there was an outbreak of Ebola in the rural north of Liberia. Inexperienced people from rich countries got scared and they all came up with the same idea: a roadblock!
…(continued)…
…At the Ministry of Health, I encountered politicians of a higher quality. They were more experienced, and their experience made them cautious. Their main concern was that roadblocks would destroy the trust of the people abandoned behind them. This would have been absolutely catastrophic: Ebola outbreaks are defeated by contact tracers, who depend on people honestly disclosing everybody they have touched. These heroes were sitting in poor slum dwellings carefully interviewing people who had just lost a family member about every individual their loved one might have infected before dying. Often, of course, the person being interviewed was on that list and potentially infected. Despite the constant fear and wave after wave of rumours, there was no room for drastic, panicky action. The infection path could not be traced with brute force, just patient, calm, meticulous work. One single individually delicately leaving out information about his dead brother’s multiple lovers could cost a thousand lives.
When we are afraid and under time pressure and thinking of worst-case scenarios, we tend to make really stupid decisions. Our ability to think analytically can be overwhelmed my an urge to make quick decisions and take immediate action.
Back in Nacala in 1981, I spent several days carefully investigating the disease but less than a minute thinking about the consequences of closing the road. Urgency, fear and a single-minded focus on the risks of a pandemic shut down my ability to think things through. In the rush to do something, I did something terrible.