The Maddening Mystery of Imperial’s Invulnerable Reputation Despite its Dire Record of Failed Model Predictions

Phillip W. Magness in AIER has crunched the numbers and shown how poor Imperial College’s modelling has been at predicting the outcomes of the COVID-19 pandemic under different policy responses in every country in the world (well, 189 of them). Yet for some unexplained reason Neil Ferguson and the rest of the Imperial team remain respected authorities on epidemic modelling and management. Magness writes:

COVID-19 has produced no shortage of doomsaying prophets whose prognostications completely failed at future delivery, and yet in the eyes of the scientific community their credibility remains peculiarly intact.

No greater example exists than the epidemiology modelling team at Imperial College-London (ICL), led by the physicist Neil Ferguson. As I’ve documented at length, the ICL modelers played a direct and primary role in selling the concept of lockdowns to the world. The governments of the United States and United Kingdom explicitly credited Ferguson’s forecasts on March 16th, 2020 with the decision to embrace the once-unthinkable response of ordering their populations to shelter in place.

Ferguson openly boasted of his team’s role in these decisions in a December 2020 interview, and continues to implausibly claim credit for saving millions of lives despite the deficit of empirical evidence that his policies delivered on their promises. Quite the opposite – the worst outcomes in terms of Covid deaths per capita are almost entirely in countries that leaned heavily on lockdowns and related nonpharmaceutical interventions (NPIs) in their unsuccessful bid to turn the pandemic’s tide.

Assessed looking backward from the one-year mark, ICL’s modelling exercises performed disastrously. They not only failed to accurately forecast the course of the pandemic in the US and UK – they also failed to anticipate COVID-19’s course in almost every country in the world, irrespective of the policy responses taken.

Time and time again, the Ferguson team’s models dramatically overstated the death toll of the disease, posting the worst performance record of any major epidemiology model.

Magness has put together a table of all the countries with the predictions ICL made for them and their actual outcomes. The results should be fatal for the reputation of anyone whose job it is to make accurate predictions of the future course of events. But not ICL it seems, whose credibility appears to be invulnerable despite repeated and consistent failure. Magness wonders why.

Why is Ferguson, who has a long history of absurdly exaggerated modeling predictions, still viewed as a leading authority on pandemic forecasting? And why is the ICL team still advising governments around the world on how to deal with COVID-19 through its flawed modeling approach? In March 2020 ICL sold its credibility for future delivery. That future has arrived, and the results are not pretty.

Worth reading in full.

Imperial College’s Modelling is Even Worse Than We Thought

When Professor Neil Ferguson and his team at Imperial College London have been challenged on their model’s miserable failure to predict the pandemic death toll in Sweden they have always pushed back saying they didn’t model Sweden, disavowing the work of the team at Uppsala University which adapted their modelling to the Swedish context. But it turns out this is not exactly accurate. Phillip W. Magness explains on AIER:

In the House of Lords hearing from last year, Conservative member Viscount Ridley grilled Ferguson over the Swedish adaptation of his model: “Uppsala University took the Imperial College model – or one of them – and adapted it to Sweden and forecasted deaths in Sweden of over 90,000 by the end of May if there was no lockdown and 40,000 if a full lockdown was enforced.” With such extreme disparities between the projections and reality, how could the Imperial team continue to guide policy through their modelling?

Ferguson snapped back, disavowing any connection to the Swedish results: “First of all, they did not use our model. They developed a model of their own. We had no role in parameterising it. Generally, the key aspect of modelling is how well you parameterise it against the available data. But to be absolutely clear they did not use our model, they didn’t adapt our model.”

The Imperial College modeller offered no evidence that the Uppsala team had erred in their application of his approach. The since-published version from the Uppsala team makes it absolutely clear that they constructed the Swedish adaptation directly from Imperial’s UK model. “We used an individual agent-based model based on the framework published by Ferguson and co-workers that we have reimplemented” for Sweden, the authors explain. They also acknowledged that their modelled projections far exceeded observed outcomes, although they attribute the differences somewhat questionably to voluntary behavioural changes rather than a fault in the model design.

Ferguson’s team has nonetheless aggressively attempted to dissociate itself from the Uppsala adaptation of their work. After the UK Spectator called attention to the Swedish results last spring, Imperial College tweeted out that “Professor Ferguson and the Imperial COVID-19 response team never estimated 40,000 or 100,000 Swedish deaths. Imperial’s work is being conflated with that of an entirely separate group of researchers.” It’s a deflection that Ferguson and his defenders have repeated many times since.

In fact, though, as Phillip points out, it is not true to say that the Imperial team never estimated 40,000 or 100,000 Swedish deaths. Hidden away in a spreadsheet in the appendix to Report 12, published on March 26th 2020, are the team’s estimates for other countries including Sweden. The projections are expressly intended to encourage those countries to follow suit with social restrictions. They write:

To help inform country strategies in the coming weeks, we provide here summary statistics of the potential impact of mitigation and suppression strategies in all countries across the world. These illustrate the need to act early, and the impact that failure to do so is likely to have on local health systems.

The predictions for Sweden are up to 90,157 deaths under “unmitigated” spread (Uppsala projected 96,000) and, under “population-level social distancing” (lockdowns), 42,473 deaths (compared to Uppsala’s 40,000). So, contrary to their repeated denials, Ferguson’s team did make predictions for Sweden very close to those made by the Uppsala team who adapted their model, and those predictions were just as way off. Sweden’s Covid death toll at the end of the first wave, on August 31st, was 5,821.

Phillip summarises further failures of the Imperial modelling in a table showing four non-lockdown countries (Sweden, Taiwan, South Korea, Japan) and the United States (most of whose states imposed a lockdown in the spring) with their one-year death toll and how it compares to Imperial’s projections.

Performance of Imperial College Modelling in Four Non-Lockdown Countries and the United States (AIER)

It’s worth saying, though, that the models for the ‘unmitigated’ scenarios predicted the deaths to occur over the course of a few months, not a whole year including another winter flu season. There will be another ‘wave’ of deaths every winter, possibly from (or with) COVID-19 if it remains the dominant respiratory virus (and if we keep on testing for it). If we keep on adding the deaths over several seasons then of course they will eventually reach the predicted figures. But that wasn’t what the models were claiming to show and would be a case of making the evidence fit the model.

The AIER article is worth reading in full.