Sue Denim

by Sue Denim A Covid-safe classroom There's a new paper out in the Lancet called "Determining the optimal strategy for re-opening schools" by Panovska-Griffiths et al. It predicts a large wave of infections and deaths if there isn't a big step up in contact tracing and isolation (house arrests) of PCR-positive cases. The model has been getting traction in the media, like in this segment by Sky News: "it's just a model but clearly, that would be disastrous". Unfortunately, people are being misled once again. The modelling conclusions are unsupportable. The papers. There are two papers we'll be looking at. One is the UK specific instantiation of the model looking at school re-openings (henceforth "the schools paper"), and the second describes the Covasim simulation program that was used to calculate the results (henceforth "the Covasim paper"). Both papers come from substantially the same team. The model. The schools paper says the "approach is similar to that in the study by Ferguson and colleagues, which informed the implementation of lockdown measures in the U.K.". Covasim is a simulation structurally quite similar to the COVID-Sim program by Imperial College London we have previously looked at in part one, two and three of this series. It simulates individuals in a population as they pseudo-randomly get sick, infect contacts and sometimes die. Indeed, Covasim is so similar to COVID-Sim that a quick perusal of the...

by Sue Denim UnHerd's Tom Chivers looks deeply into the latest epidemiological modelling A source of frustration for many of us is how journalists don't seem to be learning lessons about academic models. It's as though their need to report on anything new makes them extremely vulnerable to trusting the output of discredited processes and people. Take this piece in UnHerd by Tom Chivers called "What's the point of contact tracing?" It's about a paper entitled "Modelling the health and economic impacts of Population-wide Testing, contact Tracing and Isolation (PTTI) strategies for COVID-19 in the UK". It starts with a brief hint that maybe epidemiology models can be problematic (my emphases in bold): Well, one interesting paper just published by researchers at University College London looks at the impact contact tracing will have on the economy relative to lockdown ...Some caveats: modelling is inherently uncertain. Manheim says that the model is robust — the headline outcomes remain broadly consistent even if the assumptions and inputs are tweaked — but it’s just one paper, and so far not peer-reviewed. Note how the academic's opinion of his own work is given as evidence it's correct, as if this provides any insight. This paper has a whopping 26 authors, yet they apparently couldn't wait for peer review before sending it to journalists. Let's...

By Sue Denim After Toby published my first and second pieces, Imperial College London (ICL) produced two responses. In this article I will study them. I've also written an appendix that provides some notes on the C programming language to address some common confusions observed amongst modellers, which Toby will publish tomorrow. Attempted replication. On the June 1st ICL published a press release on its website stating that Stephen Eglen, an academic at Cambridge, was able to reproduce the numbers in ICL's influential Report 9. I was quite interested to see how that was achieved. As a reminder, Imperial College's Report 9 modelling drove lockdown in many countries. Unfortunately, this press release continues ICL's rather worrying practice of making misleading statements about its work. The headline is "Codecheck confirms reproducibility of COVID-19 model results", and the article highlights this quote: I was able to reproduce the results… from Report 9. This is an unambiguous statement. However, the press release quotes the report as saying: “Small variations (mostly under 5%) in the numbers were observed between Report 9 and our runs.” This is an odd definition of "replicate" for the output of a computer program, but it doesn't really matter because what ICL doesn't mention is this: the very next sentence of Eglen's report says: I observed 3 significant differences:1. Table...

by Sue Denim I'd like to provide a followup to my first analysis. Firstly because new information has come to light, and secondly to address a few points of disagreement I noticed in a minority of responses. The hidden history. Someone realised they could unexpectedly recover parts of the deleted history from GitHub, meaning we now have an audit log of changes dating back to April 1st. This is still not exactly the original code Ferguson ran, but it's significantly closer. Sadly it shows that Imperial have been making some false statements. ICL staff claimed the released and original code are “essentially the same functionally”, which is why they “do not think it would be particularly helpful to release a second codebase which is functionally the same”.In fact the second change in the restored history is a fix for a critical error in the random number generator. Other changes fix data corruption bugs (another one), algorithmic errors, fixing the fact that someone on the team can't spell household, and whilst this was taking place other Imperial academics continued to add new features related to contact tracing apps.The released code at the end of this process was not merely reorganised but contained fixes for severe bugs that would corrupt the internal state of the calculations. That is very different from “essentially...

by Sue Denim Imperial finally released a derivative of Ferguson's code. I figured I'd do a review of it and send you some of the things I noticed. I don't know your background so apologies if some of this is pitched at the wrong level. My background. I have been writing software for 30 years. I worked at Google between 2006 and 2014, where I was a senior software engineer working on Maps, Gmail and account security. I spent the last five years at a US/UK firm where I designed the company's database product, amongst other jobs and projects. I was also an independent consultant for a couple of years. Obviously I'm giving only my own professional opinion and not speaking for my current employer. The code. It isn't the code Ferguson ran to produce his famous Report 9. What's been released on GitHub is a heavily modified derivative of it, after having been upgraded for over a month by a team from Microsoft and others. This revised codebase is split into multiple files for legibility and written in C++, whereas the original program was "a single 15,000 line file that had been worked on for a decade" (this is considered extremely poor practice). A request for the original code was made 8 days ago but ignored, and it will probably...