How Convincing is Imperial College’s COVID-19 Model?

A reader who describes himself as a “normal person” has tried to make sense of Imperial College’s notorious March 16th paper. He doesn’t have much luck. Imperial College really needs to be a bit more transparent about the assumptions it used in its model and how it reached the figures of 510,000 dead if we “do nothing” and 250,000 dead if we stuck with mitigation. How can voters make up their own minds about whether the Government was right to lock down the country unless “the science” is set out in a way that lay people can understand?

Have you read any of Imperial College’s papers about COVID-19? Probably not. Nor had I. But we’ve all heard about them in the news. I decided to sit down and read the one that contained the advice to lock down the UK.

I’ve written this from the perspective of a “normal person”. I’m not a professional statistician, though I know a bit about it. Nor do I write computer software. In fact I’m a professional historian, which means that above all else I ask questions. I also worked in secondary education for a decade, where I was continually subjected to predictive modelling that was always wrong and always based on a vast number of assumptions that ignored reality. I became used to dismantling what we were presented with, and what I saw many of my colleagues accepting at face value. I wasn’t satisfied with just accepting Imperial College’s modelling for COVID-19. I wanted to understand it. In particular, I wanted to know why they had predicted 510,000 deaths in the UK from COVID-19 and recommended the lockdown we are now stuck in. What I found myself doing was sinking into a quagmire of assumptions, one piled on top of another, and figures cited without any coherent explanation. At every stage this progresses the predictions to one more level away from reality.

This is what I found out.

In Ferguson’s team’s “Report 9: Impact of non-pharmaceutical interventions (NPSs) to reduce COVID-19 mortality and healthcare demand” (published 16^th March 2020), a series of recommendations is made about how practical public health measures could reduce the spread of the disease. I’ll call this paper Ferguson20.¹

Like all predictive modelling, a number of assumptions were made, assumed, or implied. Predictive modelling is – in part – based on past observations from which a projection of the future is derived. The practice is widespread in commercial and educational contexts. It is usually expounded in ways that are incomprehensible to most people and narrow in perception and approach.

It’s important to bear in mind two key options considered in Ferguson20: mitigation or suppression of the effects of the virus.

Ferguson20’s prediction is that “optimal mitigation policies”, such as isolation of suspect cases, quarantining of their households and social distancing of the most-at-risk would still result in “hundreds of thousands of deaths”. No figures are supplied at this stage but they appear later, spread over the “2 years of the simulation”:² approximately 258,000 if the health system was not overwhelmed, a reduction of 49% from 510,000 with no policy interventions.³ The recommendation was therefore to resort to suppression through what we now know as the lockdown, and that that would need to last for “18 months or more”.

Ferguson20 appears to assume from start to finish that:

The reproduction rate of the disease as measured on the R_o index is constant at R_o = 2.4, which they call their “baseline assumption”. We’ve all heard this unit of measure on the news and it refers to the rate at which one infected person infects others. There’s nothing fixed about an R_o number: it’s estimated on the basis of a number of factors (and assumptions).
That every human being is equally susceptible to being infected by the disease and transmitting it at the same rate.
It is also projected that ultimately 81% of the UK population will be infected. R_o, together with the average generation times between original and transmitted infections and the proportion of the population that remains susceptible to the virus (which gradually reduces) determines the compound daily growth rate of the infections.

None of these is actually stated, but the reader is led to believe these are the case since any alternative is not acknowledged beyond saying that “much remains to be understood about its transmission”. Instead, more concern is expressed about how nations and people respond to lockdown measures, including “spontaneous changes in population behaviour”. It is interesting here that blame for failure of any lockdown is therefore expressly transmitted in advance to those obliged to carry it out. This leads to the paper’s conclusion, which is essentially a disclaimer.

As we now know, this ominous warning could also have been usefully applied by himself to the principal author of Ferguson20.

It is worth adding that the Report did acknowledge the potential social and economic costs, and warned that the most vulnerable cannot be completely protected.

The Assumptions Made

In order to get to 510,000 deaths in the UK, Ferguson20 made the following assumptions:

The Transmission Model was based on mathematics that created a hypothetical population in which the disease circulates. This was used “to generate a synthetic population of schools” and also something similar for workplaces. This was in part based on previous influenza outbreaks and clearly assumed that everyone within this population was equally liable to infection, with one third occurring in homes, one third occurring in schools and workplaces and one third “in the community”. This has been referred to as the “SimCity model”.
Ferguson20 assumed that all infected individuals were infectious, and that asymptomatic individuals were 50% more infectious than symptomatic individuals. They also assumed that infected individuals would subsequently be immune “in the short term”. (Later on, page 15, they acknowledge that there are “very large uncertainties around the transmission of this virus” but this caveat does not seem to have affected their calculations or assumptions.)
Infection was assumed to result in exponential growth every 6.5 days in each country.
It appears to have been assumed that everyone in each country formed part of an aggregate susceptibility to infection, apart from considerations of geographical separation and of household size. In other words, no account was taken of any other factor such as natural resistance, genetic predisposition, blood group, age, ethnicity, race or the existence of other medical conditions. These of course could not be computed at the time but that – and the ignoring of them – could not affect their direct relevance to the accuracy of the assumption. None of these is mentioned even as technical possibilities.
Infectiousness was not distinguished by age, but age was recognized to be a “non-uniform” factor in hospitalization and fatality. Overall, they predicted 4.4% of the infected UK population would be hospitalized, of whom 30% would require critical care and of those 50% would die.
The disease is implicitly assumed to maintain a constant progression towards near-universal infection with hospitalization and fatality at constant rates in different age bands.
The disease is also assumed to remain of constant potency and impact.

As we can see therefore, Ferguson20’s advice was predicated on a wholly artificial depiction of disease circulating in a population on a purely mathematical basis. It could not possibly take into account the plethora of actual factors that would determine the true course. The result was their prediction of impending catastrophe which seems to have been founded on the assumption that the great bulk of the population would be infected, and of whom two-thirds would be symptomatic. This was in spite of the fact that we know diseases do not have a universally similar impact on the population. This has become painfully apparent with the latest revelations that black Britons are dying at a rate which is twice that of white Britons.

A key headline figure in Ferguson20 is the prediction of 510,000 deaths based on the R_o figure of 2.4. This is, in fact, only one of several predicted “do nothing” death totals for the UK, derived from different R_o figures (a range of 2.0–2.6). I was puzzled by the lack of a clear explanation for how 510,000 had been arrived at, since this was a key news-grabbing figure when the whole crisis erupted and the lockdown started. Preventing 500,000-odd deaths was a driving force behind the government’s decision to enforce a national lockdown.

It’s important to add here that the projected 510,000 deaths does not take into account the possibility – even probability – that some of that group would have died during the two-year period from other causes. Of course we now know that underlying health conditions are playing a large part in mortality, with the confusing blurring of causes of death being recorded either as “from” or “with” COVID-19. Professor Ferguson is on record as conceding more recently that this could have applied to as many as two-thirds of the victims within 2020 alone.

[Note: The “2 years of the simulation” is important. The average death rate in the UK is about 9.4 per thousand, or around 625,000 per annum. Over two years, therefore, around 1.25 million will die as a matter of course. It will not be until at least two years have passed that we will know how many deaths from or with COVID-19 will amount to an increase over the deaths that would have been expected anyway, whether that is the 510,000 deaths from Ferguson20’s projection or the actual number.]

The basis for projecting 510,000 deaths is what has already been discussed on this site. Ferguson20 used a “stochastic, spatially structured individual based simulation”, explained in a 2005 paper by Neil Ferguson and others based on an influenza outbreak in SE Asia.⁴

Although the term “stochastic” has been described on this site as a scientific word for “random”, it’s actually a Greek word the original meaning of which was “being skilled at aiming at something” or, better, “an educated guess”. As an amusing aside, scientists rarely seem to know the Greek origins of the words they use, which are often quite humbling.

Now, I had a read of that paper. It didn’t help me understand the predictions in the 16^th March 2020 paper. For a start the method used in 2005 “did not model disease-related mortality” because they were only interested in inhibiting the spread of the SE Asian influenza outbreak rather than limiting deaths.

The answer is via a route which Ferguson20 simply did not explain. It provides a rounded Infected Fatality Rate (“IFR”) of 0.9% (p.5; the actual IFR figure they worked from seems to have been 0.943%), which is represented by the 510,000 projected deaths. They also assumed that 81% of the population would be infected because they estimated an R_o of 2.4. This means that 510,000 would die out of 54.12 million (81% of the UK population of 66.8 million) if one uses an IFR of 0.943%. This population figure is for mid-2019 from the ONS, but the real figure is probably slightly higher for 2020.⁵

No UK population figure appears in Ferguson20 and this calculation is never actually demonstrated either.

The figure of 510,000 is inclusive of those dying outside ICUs but that point is not clearly made. Indeed, it is obscured by the next paragraph proceeding to discuss the overwhelming of ICU capacity.

It’s important to understand that there are two paths through Ferguson20’s figures to projected deaths. One is the number of deaths of those admitted to ICUs. This number is smaller than the headline figure of 510,000 which covers all projected deaths (for example, those that occur in the home). Ferguson20 predicted that 4.4% of infected cases would be hospitalized. This was not based on any method of knowing how many people in total would be infected. Ferguson20 could only assume that up to two-thirds of the cases would be recognizably symptomatic. This is less than the 81 percent they predict actually would be infected.

So, let’s start with the assumption that two-thirds of the UK population (c.44.53 million out of c.66.8 million) is infected. Using ICL’s own percentages, that means 1.96 million being hospitalized at some point, of whom 30% or 588,000 would require intensive care, 50% of whom (294,000) will die.⁶ Crucially, this only refers to those dying in ICUs. As we have seen, they estimated that an extra 215,000-odd would die outside ICUs.

It’s worth adding that the 4.4% comes from “a subset of cases from China”. The 30% figure appears to have come from a single source (credited as a personal communication). The 50% is attributed to non-referenced “expert clinical opinion”.

However, Ferguson20 proceeded to estimate that without a lockdown in fact 81% of the UK population (54.11 million) would be infected, derived from an assumed R_o of 2.4. Applying the same calculation means that 2.38 million people will be hospitalized, of whom 714,225 will be admitted to ICUs with 357,112 dying in ICUs.

Ferguson20 did not actually reproduce any of these calculations, so we don’t know what he thought the figures would have been. Nor did it cite the UK population size used. And it’s only at this stage that the 510,000 projected deaths first appears, without explanation.⁷ It’s necessary to go through the process I have outlined above to find how it was calculated.

Now, just to add to the mounting complications, it’s been suggested elsewhere that Ferguson20 got its figures wrong because they had “downscaled” Chinese hospitalization rates with an IFR of 1.23%, leading to the proposition that Ferguson20’s 510,000 should in fact have been 661,402.⁸

So I read the original paper – which we’ll call Verity20 – that provides the 1.23% IFR.⁹

And what do I find? The 1.23% is actually for a Case Fatality Rate (“CFR”), which is not the same as an IFR. A Case Fatality Rate is measured against known cases of a disease. An IFR includes the CFR but tries to incorporate an allowance for asymptomatic and otherwise undetected infections. Since those are, by definition, unlikely to be fatal it’s no great surprise that a CFR proportion of deaths is larger than an IFR. The Verity20 paper in fact estimates an “overall IFR estimate for China of 0.66%” (p.2). If that was applied to the UK then Ferguson20’s 510,000 prediction comes down to 357,000. But I am straying.

We can therefore bypass the 661,402 and stick with Ferguson’s 510,000 rather than confusing the issue further. But it’s worth bearing in mind the difference between an IFR and a CFR.¹⁰

Ferguson20 proceeded to estimate the effects of various interventions involving both the nature of the intervention and the extent of compliance.

The Impact of Interventions

A complex range of tables follows which itemizes predictions dependent on the extent of lockdown measures and the time in which they are in place.

This section is filled with waffle and caveats. They concluded that epidemic suppression is “the only viable strategy” through population-wide social distancing and home isolation, together with school and university closure for maximum effect. They reject mitigation as an option on the basis that it would overwhelm ICUs leading to “in the order of 250,000 deaths” (this is the 258,000 mentioned elsewhere by them).

They concluded that a range of interventions be imposed in countries able to implement them, and that they would need to be in place for “several months” to prevent a second wave – a figure that does not seem to have been discussed or mentioned by the government. They model the repeated imposition of a lockdown for two-thirds of the time until the end of 2021 as being necessary,¹¹ at which time this pattern would need to be continued in the absence of vaccination or an effective drug being available at scale.

They believed school and university closure to be more effective than household quarantine. Elsewhere, they state their assumption that children “transmit as much as adults.”

Part of the argument about applying a limited-term lockdown is that Ferguson20 also assumes that until a vaccine arrives in eighteen months or more “these policies will have to be maintained” to prevent “a rebound in transmission”.

In spite of all this, the final conclusion is a truly remarkable one and it represents a strange twist from the whole focus of Ferguson20. This says that:

“[it] is not at all certain that suppression [of the virus through these measures] will succeed long term”
“[h]ow populations and societies will respond [to long-term lockdown measures] remains unclear”

In other words, having recommended a course of action based on a litany of assumptions – none of which is necessarily right and much of which is open to debate – Fergson20’s authors distance themselves from any failure resulting from following their recommendations. “It wasn’t our fault, guv”, I can hear them plead in two years’ time.

Moreover, while the predictions about hospitalization, ICU use and death numbers that would have resulted from inaction are not necessarily wrong, nevertheless since they are entirely based on assumptions and not presented as the result of clear and transparent calculations there is no reason to conclude that they are right either. As a layman, and as far as I could see, the entire structure is a minefield of figures derived from a variety of sources, open to confusing and contradictory interpretation, and which omits a vast range of real-world factors that will affect the outcome. Even the software code seems to be gravely flawed.¹²

Quite how this could ever link to reality I have no idea. Worse, I don’t imagine any politician read Ferguson20 critically.

Am I any the wiser? What left me most concerned is that it would appear that the UK embarked down the lockdown/suppression route based on the advice of a very small group of experts who seem to have little or no confidence that their recommendations will succeed anyway. And when one looks at Sweden one rather wonders why they bothered to make them.