The UKHSA's 'Evidence' for Covid Restrictions is a Complete Mess

You may recall that we undertook to review the 100 models forming the backbone of the UKHSA’s latest offering: the mapping review of available evidence. Remember, UKHSA did not extract nor appraise the evidence as it does not have the resources. This drew expressions of mirth among our readers. We agree it’s a bad joke – a very bad one – considering this ‘evidence’ is what the UKHSA states justified restrictions that led to stories such as Pippa Merrick’s, which unfortunately are not the exception. Earlier versions of the justification were a bad joke, too. What follows is no better.

Diligently, as promised, we downloaded the 100 papers defined as “models” by UKHSA (please do not ask Hugo Keith KC what is meant by that term).

Of each, we are asking the following questions:

What is the non-pharmaceutical intervention (NPI) being assessed (e.g. is it an NPI, and is it defined and described?) and in what setting? (e.g. community, hospital, homes etc.)
What is the source for the effect estimate? (to model its effects, you need a source of data, i.e., what does it do?)
What is the size of the effect? (such as risk reduction of SARS-CoV-2 infection)
What is the case definition? (how did they define a case of COVID-19?)

Straightforward, we thought.

Anything but, we are finding out.

First of all, the papers are full of jargon, as they are mainly written by mathematicians, or at least that is what they say they are. Secondly, most of them come to the same conclusion: lockdown harder, do as I say, or you or your auntie (or both of you) will die.

The most disconcerting answers we are getting are those to the second question: what is the source for the effect estimate?

In a classical model, you start with describing the problem, in this case, the number of cases and complications in a population, transmission patterns and perhaps age breakdown. If your second part is about how to stop or slow down the spread, hospitalisations, deaths and so on, to model the ‘how to’ in a credible way you need facts about what you are modelling is supposed to achieve (say distancing). Which, if introduced in this or that setting, is likely to diminish the risk of infection by Z%. The numerical estimate for Z should be surrounded by a range of probabilities (confidence intervals), giving the boundaries of probabilities that the observed effect (Z) in reduction of SARS-CoV-2 infection lie within X and Y around your point estimate of Z. So you then take Z and stick it in your model to see what effect Z would have and then you can use X and Y to play ‘what if’.

The crucial word is ‘credible’ because these models (are they projections, scenarios, predictions, or scenarios upon which predictions can be projected – ask Hugo Keith KC for a simple answer) have been used to change people’s lives. Or maybe some of them were retrofits to justify something already done by the Robert Maxwell school of ethics.

Credible would mean an estimate from one or preferably more well-designed studies with a protocol and clear case definitions. As the focus is the U.K., the data should come from the U.K. or at least a similar setting.

Well, here is an example of the sources of ‘parameters’ used in one quite well-publicised model:

Of the 11 assumptions underlying the model, eight are unsourced; one comes from a systematic review without infectious case definition, one from an economic model, and one from a case-control study.

Extraordinary, you will say: this seems to be the universal method known as BOPSAT (a Bunch Of People Sitting Around a Table). Yes, it is, except that the model, in fact, was about mass community testing for SARS-CoV-2 by lateral flow devices (LFDs) with not a shred of non-pharmaceutical interventions in sight. LFDs are tests, not interventions that can slow or stop the spread of anything.

And these are some of the minor problems we face, so it takes time. Perhaps we should ask Mr. Keith for help?

Dr. Carl Heneghan is the Oxford Professor of Evidence Based Medicine and Dr. Tom Jefferson is an epidemiologist based in Rome who works with Professor Heneghan on the Cochrane Collaboration. This article was first published on their Substack, Trust The Evidence, which you can subscribe to here.