by Sue Denim
I’d like to provide a followup to my first analysis. Firstly because new information has come to light, and secondly to address a few points of disagreement I noticed in a minority of responses.
The hidden history. Someone realised they could unexpectedly recover parts of the deleted history from GitHub, meaning we now have an audit log of changes dating back to April 1st. This is still not exactly the original code Ferguson ran, but it’s significantly closer.
Sadly it shows that Imperial have been making some false statements.
ICL staff claimed the released and original code are “essentially the same functionally”, which is why they “do not think it would be particularly helpful to release a second codebase which is functionally the same”.
In fact the second change in the restored history is a fix for a critical error in the random number generator. Other changes fix data corruption bugs (another one), algorithmic errors, fixing the fact that someone on the team can’t spell household, and whilst this was taking place other Imperial academics continued to add new features related to contact tracing apps.
The released code at the end of this process was not merely reorganised but contained fixes for severe bugs that would corrupt the internal state of the calculations. That is very different from “essentially the same functionally”.The stated justification for deleting the history was to make “the repository rather easier to download” because “the history squash (erase) merged a number of changes we were making with large data files”. “We do not think there is much benefit in trawling through our internal commit histories”.
The entire repository is less than 100 megabytes. Given they recommend a computer with 20 gigabytes of memory to run the simulation for the UK, the cost of downloading the data files is immaterial. Fetching the additional history only took a few seconds on my home WiFi.
Even if the files had been large, the tools make it easy to not download history if you don’t want it, to solve this exact problem.
I don’t quite know what to make of this. Originally I thought these claims were a result of the academics not understanding the tools they’re working with, but the Microsoft employees helping them are actually employees of a recently acquired company: GitHub. GitHub is the service they’re using to distribute the source code and files. To defend this I’d have to argue that GitHub employees don’t understand how to use GitHub, which is implausible.
I don’t think anyone involved here has any ill intent, but it seems via a chain of innocent yet compounding errors – likely trying to avoid exactly the kind of peer review they’re now getting – they have ended up making false claims in public about their work.
Effect of the bug fixes. I was curious what effect the hidden bug fixes had on the model output, especially after seeing the change to the pseudo-random number generator constants (which means the prior RNG didn’t work). I ran the latest code in single threaded mode for the baseline scenario a couple of times, to establish that it was producing the same results (on my machine only), which it did. Then I ran the version from the initial import against the latest data, to control for data changes.
The resulting output tables were radically different to the extent that they appear incomparable, e.g. the older code outputs data for negative days and a different set of columns. Comparing by row count for day 128 (7th May) gave 57,145,154 infected-but-recovered people for the initial code but only 42,436,996 for the latest code, a difference of about 34%.
I wondered if the format of the data files had changed without the program being able to detect that, so then I reran the initial import code with the initial data. This yielded 49,445,121 recoveries – yet another completely different number.
It’s clear that the changes made over the past month and a half have radically altered the predictions of the model. It will probably never be possible to replicate the numbers in Report 9.
Political attention. I was glad to see the analysis was read by members of Parliament. In particular, via David Davis MP the work was seen by Steve Baker – one of the few British MPs who has been a working software engineer. Baker’s assessment was similar to that of most programmers: “David Davis is right. As a software engineer, I am appalled. Read this now”. Hopefully at some point the right questions will be asked in Parliament. They should focus on reforming how code is used in academia in general, as the issue is structural incentives rather than a single team. The next paragraph will demonstrate that.
Do the bugs matter? Some people don’t seem to understand why these bugs are important (e.g. this computational biology student, or this cosmology lecturer at Queen Mary). A few people have claimed I don’t understand models, as if Google has no experience with them.
Imagine you want to explore the effects of some policy, like compulsory mask wearing. You change the code and rerun the model with the same seed as before. The number of projected deaths goes up rather than down. Is that because:
- The simulation is telling you something important?
- You made a coding error?
- The operating system decided to check for updates at some critical moment, changing the thread scheduling, the consequent ordering of floating point additions and thus changing the results?
You have absolutely no idea what happened.
In a correctly written model this situation can’t occur. A change in the outputs means something real and can be investigated. It’s either intentional or a bug. Once you’re satisfied you can explain the changes, you can then run the simulation more times with new seeds to estimate some uncertainty intervals.
In an uncontrollable model like ICL’s you can’t get repeatable results and if the expected size of the change is less than the arbitrary variations, you can’t conclude anything from the model. And exactly because the variations are arbitrary, you don’t actually know how large they can get, which means there’s no way to conclude anything at all.
I ran the simulation three times with the code as of commit 030c350, with the default parameters, fixed seeds and configuration. A correct program would have yielded three identical outputs. For May 7th the max difference of the three runs was 46,266 deaths or around 1.5x the actual UK total so far. This level of variance may look “small” when compared to the enormous overall projections (which it seems are incorrect) but imagine trying to use these values for policymaking. The Nightingale hospitals added on the order of 10-15,000 places, so the uncontrolled differences due to bugs are larger than the NHS’s entire crash expansion programme. How can any government use this to test policy?
An average of wrong is wrong. There appears to be a seriously concerning issue with how British universities are teaching programming to scientists. Some of them seem to think hardware-triggered variations don’t matter if you average the outputs (they apparently call this an “ensemble model”).
Averaging samples to eliminate random noise works only if the noise is actually random. The mishmash of iteratively accumulated floating point uncertainty, uninitialised reads, broken shuffles, broken random number generators and other issues in this model may yield unexpected output changes but they are not truly random deviations, so they can’t just be averaged out. Taking the average of a lot of faulty measurements doesn’t give a correct measurement. And though it would be convenient for the computer industry if it were true, you can’t fix data corruption by averaging.
I’d recommend all scientists writing code in C/C++ read this training material from Intel. It explains how code that works with fractional numbers (floating point) can look deterministic yet end up giving non-reproducible results. It also explains how to fix it.
Processes not people. This is important: the problem here is not really the individuals working on the model. The people in the Imperial team would quickly do a lot better if placed in the context of a well run software company. The problem is the lack of institutional controls and processes. All programmers have written buggy code they aren’t proud of: the difference between ICL and the software industry is the latter has processes to detect and prevent mistakes.
For standards to improve academics must lose the mentality that the rules don’t apply to them. In a formal petition to ICL to retract papers based on the model you can see comments “explaining” that scientists don’t need to unit test their code, that criticising them will just cause them to avoid peer review in future, and other entirely unacceptable positions. Eventually a modeller from the private sector gives them a reality check. In particular academics shouldn’t have to be convinced to open their code to scrutiny; it should be a mandatory part of grant funding.
The deeper question here is whether Imperial College administrators have any institutional awareness of how out of control this department has become, and whether they care. If not, why not? Does the title “Professor at Imperial” mean anything at all, or is the respect it currently garners just groupthink?
Insurance. Someone who works in reinsurance posted an excellent comment in which they claim:
- There are private sector epidemiological models that are more accurate than ICL’s.
- Despite that they’re still too inaccurate, so they don’t use them.
- “We always use 2 different internal models plus for major decisions an external, independent view normally from a broker. It’s unbelievable that a decision of this magnitude was based off a single model“
They conclude by saying “I really wonder why these major multinational model vendors who bring in hundreds of millions in license fees from the insurance industry alone were not consulted during the course of this pandemic.“
A few people criticised the suggestion for epidemiology to be taken over by the insurance industry. They had insults (“mad”, “insane”, “adding 1 and 1 to get 11,000” etc) but no arguments, so they lose that debate by default. Whilst it wouldn’t work in the UK where health insurance hardly matters, in most of the world insurers play a key part in evaluating relative health risks.
To join in with the discussion please make a donation to The Daily Sceptic.
Profanity and abuse will be removed and may lead to a permanent ban.
Very true. But as Richard Eldred’s interview this morning remarked, there was a lot less actual rioting than is commonly mentioned in the obligatory, “Of course, violent rioters need to be in jail, but…” caveats.
In fact the narrative is a bit like COVID – there was quite a lot of minor dis-ease in the form of people on the streets, and much hype about violent rioting which, on close examination, was pretty exaggerated, and in many cases entirely engineered by heavy-handed policing.
Furthermore, even at the time it looked as though rent-a-mob in balaclavas was bussed in to turn over cars in full view of news cameras, but well away from risk of arrest. Agents provocateurs was written all over them. As we saw in the Downing Street affair, the protesters were mostly remarkably well-behaved, given their outrage at the Southport massacre and its cover-up.
Since many amongst the public are well aware of the gaslighting, that outrage has become even more deep-seated, widespread, and quite liable to erupt at some stage in civil unrest the nudge units and riot police will no longer be able to crush.
This cauldron must be added to the background in this article of economic vandalism, poverty and hopelessness. The description of “a very poor country attached to a very rich city” is perceptive, but usually disguised by reassuring GDP figures. The very rich suppressing the increasingly poor is never a good recipe for civil order.
I agree. The picture depicted by media, establishment or independent, of riots and violence is a distortion of a much more mundane reality.
What I see much more of is a demoralised, beaten down public that doesn’t really understand why things are done the way they are but feel there’s no point even complaining about it because it’s not going to make a blind bit of difference.
Exercise for the reader: Assuming that, in a certain state, a party is in government because about ⅕ of the electorate voted for it and then sets forth to ruin the lives of the other ⅘ as hard as it can, oftentimes claiming that these ⅘ deserve nothing better because they’re all right-wing extremists, can this country realistically be considered a democracy? Or should it perhaps rather be called a dictatorship of the post-marxist quangoriat?
That’s pretty good, except the fifth that is voted in doesn’t set about doing anything.
The post-marxist quangoriat, as you name it, run the show regardless of who a fifth, a quarter, half, or a majority of the population vote into so called “power”.
All people are voting in are the people who are going to do the PR for the quangoriat. The elected have the power to sell us the crap that has been prepared for us, or get replaced.
Our institutions are corrupted by systemic anti-racism.
Excellent analysis Jon.
Kisin is way off the mark in stating that increased lawlessness is simply down to the economy not growing, there are multiple factors not least of which is the deliberate destruction of our patriotism. Too many people simply have negative views of this country and therefore our old values have been lost.
Jobs have been lost and with them aspirations.
Education has been practically inverted with wokeness, trans this that and the other and our evil British Empire. Pride has been shattered.
Our institutions have grown into monolithic monsters – NHS, pretty much every department of the Civil Service, even our once venerated RNLI is now nothing more than a laughing stock.
The police forces make the Keystone Cops look like serious crime fighters.
Our Christian religions have been made to give way to the heathens of Islam and we wonder why the country is rapidly approaching Turd World shit hole status?
I could go on but the bottom line is that governments these last fifty or so years have deliberately sought to undermine all that combines to make Britain great and the final destruction is to occur in this decade ending 2030.
Just one contentious response, Hux: at least the “heathens of Islam” are Muslims, for better or worse. But the population of our country had, for the most part, already succumbed to a previous ideological usurper – secularism – and abandoned actual Christianity.
It seems to me that, so far, many who perceive the collapse of the country are trying to wave the flag for “Christian values” whilst hanging on to the secularist illusion of “the view from nowhere.” In other words, they think Christianity is useful, whilst regretfully remaining uncommitted themselves.
But Christianity is Christ, or it is as toothless as “conservatism” or “common sense.” Islam just requires submission to praxis: Christianity is exclusively fuelled by faith in a Saviour-King.
Net Zerotard is part of the collapse agenda. It is just one program of many designed to ruin us.
etc. etc.
If anything, the EU is the French Empire¹. The Germans are just ‘allowed’ to pay for it.
¹ That’s probably a bit beyond the horizon of the average US politard, but a French Empire encompassing all of continental Europe proper up to the eastern border of Poland composed of French provinces (including large parts of Germany), French vassal states (like the kingdom of Westfalia) and states forced to an alliance dominated by France by a series of military defeats inflicted onto them actually existed under Napoleon I. During its heyday, just before his Russian campaign, only Austria and Russia were still really independent states although both had suffered humiliating defeats as well.
Von der Leyen – unelected – German.
EU Central Bank – Frankfurt.
Terms of Trade – all in Germany’s favour.
Euro = a devalued DM.
It is the German empire with the French used as a beard.
I happen to know this better than you because I’m more familiar with actual circumstances in Germany and don’t just have my prejudices played to by politics people.
The biggest part of the EU is the common agricultural policy. It’s purpose mostly to subsidize the otherwise hopelessly loss-making traditional French rural economy.
Accepting the Euro with the inevitable consequence of ending up underwriting the sovereign debts of all of Europe was a condition imposed on Germany in exchange for the so-called re-unification. Maggie Thatcher was a driving force behind that (who would personally have preferred for this to never happen, the quote by [from memory]: We did our best to conquer the Germans. And now, they want to come back!).
The reason why the AfD is so vitriolically opposed by the German establishment parties is that it either wants to reform the EU such that the Germans stop paying its bills or get rid of it altogether. That’s why they’re accused of being reckless German nationalists and closet-nazis.
The EU grew out of the European Coal and Steel Union, a project by Adenauer and Schuman with the goal to intertwine the French and German steel and coal industries to such a degree that Germany would never again be able to go to war against France.
To this date, the so-called UN treaties contain a clause which states that Britain, Russia, the USA or France may chose to invade Germany whenever they’re unhappy with German domestic politics.
I could continue this for a while but it’s probably pointless. You’ve been drilled to hate Germans from early childhood on. And the people who want to control you just need to say “German!” to trigger the reflexes which enable them to do.
Agenda 2030 Riots you mean. Maybe the transition to Stakeholder Capitalism that they are at a halfway point, might have something to do with it. That could also explain the attacks on private LTD businesses:https://www.youtube.com/watch?v=-CAiTHWBM18&t=2695s
One bit of good news. All the big American investment companies have ditched ESG funds. The $130 trillion promised by them in Glasgow is not going to happen. A final realisation that they will loose money and anti trust lawsuits in the US have finally done for this particular cartel.
As pointed out by Ben Pile earlier today.
Spending on policing has not slowed all that much. One problem is that the whole public sector is spending vast amounts on pensions for people who stopped working years ago.
But a big issue needs to be addrssed on efficiency. The public sector never seems to be able to reduce its costs or improve performance using tech. It is only ever an additional cost.
In the private sector things are very different. Not only does investment (and current spending) on tech enable costs to be reduced and efficiency improved but the entire business is often re-engineered with almost an order of magnitude of cost are taken out.
“One problem is that the whole public sector is spending vast amounts on pensions for people who stopped working years ago.”
The problem with public sector pensions is the same as with the Old Age Pension – contributions have never been invested. Successive governments have spent money that they didn’t have and wasn’t theirs in the first place for political ends. And this crisis is now in its death throes.
It makes me wonder what state public services/pensions would be like if government had never been allowed to borrow money, and to invest pension contributions. It would have been short term pain for very long term gain.
The Establishment is delivering the UN’s objective of “levelling down” the UK; the WEF’s Agenda of transferring what wealth we have from “the peasants” to Corporations and the EU’s objective of keeping the UK trapped within its regulatory orbit.
The solution to all three is to destroy the power of the Establishment, starting with their Uni-Party in Westminster and the MSM.
Why are there no people like Konstantin Kisin, Douglas Murray and Andrew Doyle in UK politics?
Because politics in Britain is utterly corrupt, and any “non-receiver” is instantly cancelled by very dubious means. Do you notice how rich all our “alleged politicians” have become? Very strange. The WEF hardly noticed to accounting error!
And yet we still pay their council tax on their second homes, even after the media scandal in 2009.
A friend of my wife went to Oxford Street a couple of days ago. It was very crowded, particularly by a jostling crowd around her. Her purse was stolen from her “safe” handbag, carefully mounted in front of her and the purse and money stolen, including her cards. She called the bank within 2 minutes and they said that £2500 had been stolen from a card! It was recovered but the speed says “organised crime” in no uncertain terms, because such a withdraw without pin, 2 factor authorisation etc. is surely not possible? Don’t mention the Police, they basically said don’t come too London, the streets are very unsafe! Eh Khant??? The crowd were your “friends”!
That’s why I don’t see the point of going abroad to do the shopping. She should have stayed in England.
They warn when traveling to any third world country don’t wear any fancy watches or bling, but now you can’t in the so called developed West.
Williamson “describes how last summer’s riots weren’t really about immigration.”
FALSE! They were ENTIRELY about Third World Immigration destroying Britain.
They were about the Globalist Importation of a Muslim Army raping and murdering children.