The perils of over-reliance on digital systems have been once again highlighted by the crashing of computer systems around the world due to an update to the Falcon antivirus and security product from CrowdStrike affecting its interaction with the Windows operating systems. The update has caused chaos for banking, retail, railways, airports, healthcare and for a wide range of other businesses and infrastructure where the Falcon software runs on Windows systems. Advice for bringing affected computers back into working order has been published, but the exact mechanism by which the update caused “Blue Screen of Death” errors does not appear to have yet been reported.
It appears that in many cases, while the update was distributed automatically over the internet to systems, the workaround to fix the problem requires the machines to be rebooted in Windows’ safe mode, which usually requires physical access. The person at the keyboard then needs to know the password for the computer’s administrator account, and use this level of access to delete a file within a subdirectory of Windows’ System32. This process can be more complicated where Microsoft’s BitLocker encryption is in use. In many organisations, the recovery keys for BitLocker have themselves been stored on a computer unable to start properly due to the CrowdStrike update. The quote “Men go mad in herds, while they only recover their senses slowly, one by one”, originally from Charles Mackay in 1841, seems applicable now to computers too. They crash en masse, then require individual attention before they will work again.
It should be noted that while the perils of centralisation with a physical single point of failure are obvious to all but technocratic politicians and civil servants, this massive outage shows another way in which a “single point” of failure can occur. The single point in this case is not a particular server in one building somewhere on the planet; but rather a change within a single piece of software with that change then being rolled out to many individual systems around the globe. These systems then entered a state euphemistically described as Total Inability To Support Usual Performance (acronym intentional) among the tech community. There was a reason that NASA put a fifth backup flight computer in the space shuttle, running software written entirely independently of the software on its primary four computers. A single point of failure where software is concerned doesn’t have to happen at only a single point in space.
There is a very clear lesson to be learned here. Systems which can collapse at scale, even when they are not centralised in the physical sense, eventually will collapse in such a fashion. Advocates of Central Bank Digital Currencies (CBDCs) and Digital ID systems should consider these lessons. This update ‘only’ knocked out an estimated 8.5 million computers, belonging to over 24,000 organisations that subscribed to CrowdStrike’s Falcon software. A country reliant on a CBDC instead of cash would see an end to all transactions as a consequence of a similar failure affecting a component within whatever software stack was being used to operate CBDC infrastructure. That could mean a fault within the software on physically centralised or partly centralised servers logging transactions and holding records; or a fault within the software running on masses of devices operating as payment terminals in a wide variety of locations. In that dystopian CBDC-dependent nation, one would be looking at electric vehicles (already a bad idea simply on account of the abysmal energy density of batteries compared to chemical fuels) stranded at charging stations, unable to make payments to initiate the charging procedure. Consider that the World Economic Forum once advertised with slogans on the theme of “what if extreme weather froze your bank account”, right at the time when Justin Trudeau was freezing bank accounts on account of his extreme intolerance for peaceful protest. The reality is that in the centralised totalitarian model of society the WEF hungers for, this scenario becomes more probable, not less. That is to say, that as well as increasing the opportunities for censorship-obsessed elites to deliberately interfere in people’s lives, centralisation also increases the vulnerability of a society to accidental errors. Where Governments dream of requiring digital ID or age verification for internet access, or client-side scanning to look for objectionable opinions and only allow messages to be sent when approved as sufficiently “double plus good”, one can even imagine a situation where direct messages and online posts attempting to report a fault in the software stack running the verification or approval algorithms would be blocked from being sent. This wouldn’t need to be a matter of a deliberate attempt to cover up the fault, but instead the inability to report the fault would be a natural consequence of the fault itself. A censorship apparatus built on a principle of scanning everything before it can be shared ends up censoring absolutely everything if it is unable to perform scans.
Where old systems, like cash, just work, the alternatives which Big Tech and Big Government claim are more convenient can collapse. Where computer systems you actually own and have true control over just work, systems which can be remotely updated can crash catastrophically. When Big Government pushes for things to be more “secure”, it usually has in mind intrusive projects to stop “bad people” from using “SMART” systems, rather than measures to actually make systems more resilient in the event of crises. Government, after all, tries to whip up anger against truly secure encryption by describing it as warrant-proof, though in an age where lawfare is becoming common and courts allow the monitoring of vast numbers of innocent people it is hard to see how such a level of protection is undesirable. Meanwhile, Big Tech companies seek to create an “experience” for users, which in practice comes to mean an ever more interlinked web of dependencies, often centred around a server to which the devices regularly phone-home to check that the user has the company’s permission to use the things they have bought. Concerningly, even farming has now become a field in which equipment manufacturers are displaying this ‘Big Tech’ attitude. This risks farmers’ livelihoods and entire nations’ food security in the event of software crashes. And that could include software crashes within Digital Rights Management subsystems of software which aren’t even there for the benefit of the user in the first place.
Humans, when acting like machines and believing that adherence to procedure, policy, legislation or guidance is more important than common sense and morality, make the perils of centralisation still worse. If one adds a human element with a sufficiently jobsworth disposition and an absolute confidence in the infallibility of their systems to the toxic mix of centralisation and control freakery enabled by excessive digitisation, it results in horrific scenarios like those surrounding the Post Office’s Horizon software. I shall enter a little further into speculation here when suggesting that a correlation between people having problematic dispositions and a desire for centralisation may exist, perhaps best demonstrated by the ways in which I have seen the supporters and opponents of cash behaving during previous payment infrastructure outages. Contrast: the elderly woman who deposited exact change on the counter in a card-only cafe. She calmly claimed it was legal tender, whether that is an entirely applicable argument or not, and walked off with two packaged sandwiches. With: the student who upon being told, in broken English by the very polite man behind the fried-chicken shop counter, that VISA was down that day, raised his voice to a bellow. Such bellowing was perhaps difficult from behind the blue paper muzzle he was wearing in 2022. He then proceeded to accuse the owner of tax evasion in a lecture which lasted until everyone waiting around for their orders was glaring at him, and which contained expletives even someone fluent in English could be surprised by. While it is not my place to comment upon whether what the woman did was entirely legal, she gave the impression of someone who would start knitting socks for neighbours in a prolonged power-cut. The student gave the impression of someone who’d batter on doors in search of a USB powerbank with which to buy himself a while longer on TikTok. The types of people who cheer for centralisation do not appear to be the types who can foresee – much less aid in recovering society from – the consequences of centralisation-enabled failures.
Keeping to the theme of trendy people and trendy attitudes, it is also worth noting that whilst the update which led to the ongoing chaos may well have been intended as a security patch or as a bug fix for a rare software fault condition (and ended up creating a widespread one), many updates which have been responsible for widespread system failures in the past are updates to provide software with new ‘features’. Unfortunately, in keeping with trying to be fashionable brands, a lot more programming hours are dedicated to “oh, look, shiny” than to simply keeping abreast of actual functionality and security flaws which may require patching. With software controlled systems embedded in ever more places (and not all of this increase in the ubiquity of computing necessarily has to be bad so long as the systems are properly under a user’s control and are not cloud dependent), it would seem a wise time for the practice of software development to start prioritising reliability, resilience and stability. It would be wise to prioritise these above the aim of innovating in ways nobody asked for, which then disrupt people’s workflows within products they are already using. While consumer technology is particularly affected by this fashion-based philosophy, business software is not immune. This is particularly the case when buzzwords are used in efforts to market solutions to problems they might not have to bosses of limited technical expertise. Look at all the hype around AI, with its latest eruption coming in the form of Large Language Models. A world with software around every corner is a world which can’t afford that software to be updated, with the potential for introducing serious errors to it every time a corporate executive falls for a fad. The way in which smaller open source software projects operate may provide an inspiration here, particularly where the project consists of a standalone tool for a particular task. In many such projects, there are only two scenarios in which a developer typically posts an update. Firstly, he may post one in response to user reports of errors being thrown in specific circumstances. Secondly there may be a need for a new version of the tool when changes are needed to maintain compatibility with changes that have been applied to other software, such as new releases of an operating system under which the tool may be run.
As an overall picture, centralisation makes it all too easy for governments and corporations to feed their addiction to exercising control. And the further their reach spreads, the closer the state comes to being a black hole that sucks in the entirety of society and human experience, the more damage their anti-Midas touch causes. And then, in its aftermath, the solution they always push for is more centralisation, more opportunities to make things worse whether by intention or by accident. Escaping the headlong rush in to a new Soviet Union where nothing works and officialdom absolutely refuses to acknowledge the fact must be at the forefront of our minds when looking to the future. But the outage news itself can be summarised in a much shorter fashion. While one can feel sorry for those whose travel was disrupted, it is a wonderful feeling to jump to the front of a queue in a shop and pay a satisfied cashier with cash, whilst a seething mass of trendy woke-folk, who consider cash and even freedom itself to be outdated concepts, look on.
Dr. R P completed a robotics PhD during the global over-reaction to Covid. He spends his time with one eye on an oscilloscope, one hand on a soldering iron and one ear waiting for the latest bad news.
To join in with the discussion please make a donation to The Daily Sceptic.
Profanity and abuse will be removed and may lead to a permanent ban.
It would make sense for the world not to be dependent on one operating system.
It would make sense for government to utilise the efficiencies of Information Technology in downsizing, outsourcing, privatising, utilising local small to mefium sized enterprises, particularly in healthcare, to generally get our of our lives.
Gates’ slaves. Eggs in one basket springs to mind.
I suspect there are as many devices around the world using embedded Linux systems and Android. The problem is corporate group think that will only consider MS based systems.
It reminds me of trying to get business from local authorities who told me they only did business with people who had a track record of working with local authorities.
Indeed. Our clients resist open-source software and prefer paid-for support, even though in our experience, paid-for support is rarely worth the money you pay. You are not paying to get stuff fixed, you are paying so you have someone to blame when things go wrong. Clown world!
Open-source can just be a different business model. A lack of documentation and/or usability can encourage uptake of paid-for training/support and prioritise development. Alternatively, the cost of using open-source can be hidden by having your own staff figure out how to get it to work.
Alternatives are no panacea. Let’s not forget the Bash bug identified in 2014 and looks like it is still a problem as recently as 2020.
Interestingly, CrowdStrike says this of Microsoft.
“Microsoft’s security products can’t even protect Microsoft. How can they protect you?”
https://www.crowdstrike.com/compare/crowdstrike-vs-microsoft-defender/
Edit: and what used to be free open-source software can suddenly require a licence which lots of disgruntled users then find themselves paying for.
In my experience, documentation isn’t always helpful even when paid for, and the times when we’ve had most success with paid-for support is when we’ve decompiled the code, worked out where the bug is and built a repeatable self-contained test case. Of course, this is anecdotal and confined to the fairly small subset of products we use. We’re a small firm, too, so don’t have much leverage even when we’re paying for stuff, compared to large corporates.
I think it is very dependent on experience, the quality of the products and supporting material regardless of whether it is proprietary or open source.
What I find odd is what seems to be a not-uncommon unquestioned acceptance of finding out the internal working of someone else’s product (paid-for or not) in order to use it or let them know how to fix it thereby taking time away from your own business. It might be unavoidable in some instances but it just shifts the cost onto yourself.
It is kind of odd, though it’s certainly not an unquestioned acceptance on our part – it’s bloody annoying!
Government is run by bureaucrats whose primary aim is to increase the scope and scale of the bureaucracy – bigger budgets, higher grades and salaries, more power and control.
Everything Government now does in terms of the services (such as they are) to the public were once provided by the private sector – both profit and non-profit.
Privatisation, better utilisation of machines would greatly reduce the size of the bureaucracy and the number of bureaucrats – won’t be done voluntarily.
Most firms and platforms use Microsoft Active Directory for system authentication, authorisation. A central point of failure. A GenAI virus could wipe out Active Directory through a similar attack posing as malware/ransomware. When that happens (when not if), you won’t be able to do much online.
Another vector is the grid of course. EMP or similar would terminate most of the grid. Then what? Most of the sheeple can’t tell a bean plant from a banana tree. How would the luvvies, who style themselves the smartest people evah, survive?
An appropriate acronym. I get MAD when MS decides to impose a new “shiny look” version of software (Outlook a recent example) without consulting me. Frankly, I’d be happy to roll back to Windows 98. Stop messing with my head and let me get on with my business.
I was talking with friends last week, talking about systems failures and how important it is that cash must be retained.
She texted me to ask if I had “known” about this Crowdstrike crapshoot. Yes I said, of course, it is simply a case of WHEN, not IF.
Great and highly informative article, thanks Dr P, which is more than can be said of MSM.
Meanwhile, I’m stacking my wood and putting aside a little cash to tide us over in case of need.
Great informative essay Dr P. Your expertise gives us a unique insight. Sceptical voices in tech are more necessary than ever, The techno-tyranny of Horizon and Trudeau’s attack on the truckers is an example of what needs challenging. The censorship industrial complex is at least facing resistance from a few tech overlords. However, it would be mistaken to put all our eggs in say Musk’s basket, given that (so far) he is mortal,and the tech and financial elite will break up his business if he starts to impact their plans.
I find it ironic that security software has probably done more damage to businesses that the attacks it was designed to prevent.
“We are contacting you today to notify you that your contact information – name, and certain other contact information such as phone number, work address and/or email address was among the information downloaded by the threat actor. “
From a security incident email I received in 2023 from a business that helps companies “manage and secure user authentication”. That probably explained why at about the same time I suddenly received a flood of phishing emails. I don’t like the term “actor” because as far as I’m aware they’re not pretending.
Some years ago Equifax suffered a data breach caused, the company declared, by housekeeping not being carried out in a timely fashion. However, the company only revealed this breach to their clients three months after it had occurred. That is at the point the hackers were threatening to post the clients’ personal data on the internet. Which they duly did.
All your eggs in one basket. What can possibly go wrong?
There was a a consultation about CDBC last year. Bank of England. Not well advertised was it!
One of those obscure Parliamentary Committees looked into CBDC’s and found that there was no valid case for its introduction.
I wonder if all those card-only places still refused to accept cash during the outage, and decided to make a loss instead? I really, really, hope there system and their shortsightedness meant that did.
If ATMs weren’t working where would people get the cash? Since many retailers have computer-linked tills, they wouldn’t open for a cash transaction in any case.
Excellent piece.
I still struggle to understand how this code could possibly have passed testing, quality assurance, and signing. The ability to write to the kernel is such a critical security and infrastructure risk that what just happened is something that simply cannot happen under any circumstances.
Microsoft are trying to blame it all on Crowdstrike, but it’s their OS, and their security policy that permits Crowdstrike to write to the kernel.
Assuming both this, and the Trump assassination attempt, two momentous events within a week, are both simply the result of colossal incompetence, then things look grim for the future of humanity. How many other structures are teetering on the brink of catastrophic failures due to incompetence?
Ever heard of Human error?
This thought did cross my poor, old, increasingly conspiratorial mind: what if I wanted to know what would happen if there was a mass scale technological outage? If I wanted to understand the mechanisms that could be used to negate the effects of the outage, and wanted the overall economic and social effects of said outage? A small scale test would work perfectly. The Covid test provided all the necessary psychological and behavioural information needed for a permanent/prolonged mass scale technological outage, and this test would give me everything else I needed to know. Just a thought.
It used to be normal to update one machine and test the system of software before rolling the update out to the rest. IT have got lazy.
Apparently they never tested on a single machine before rolling out.
https://arstechnica.com/information-technology/2024/07/crowdstrike-blames-testing-bugs-for-security-update-that-took-down-8-5m-windows-pcs/?comments=1&comments-page=1
The single point of failure intentionally or unintentionally exploited by a ‘fix’ rhymes with the ‘fix’ used for Covid-19. Not surprising Gates is connected to both…
The World has always been dangerous. There is no risk-free state.
But we have trade-offs. Are we better off with a digitised World and the risks it brings, or would we be better off without it?
Well, look at the World prior to the digital age… were we better off and was it risk free? No, so we trade off the reward we have as a result of the digital age against the risks – it’s positive.
The current situation came about from an over-reliance on a particular operating system. There is competition out there… Linux, Apple and evidently Russia has a non-Windows system, so choice. Human failure not a system failure.
South West Airlines in the US had no problems because it still used Windows 3.1 having declined to upgrade to the most recent version, and it was therefore unaffected.
I am not in any way knowledgeable where IT is concerned but from personal experience every version of Windows since XP is worse than the one it replaced.
Really?! Windows XP was a pile of garbage compared with current Windows versions. The XP operating system used to crash regularly, and I even remember being told that a clean Windows install every few months was a good idea to keep it working…
When that titan of the business world Dido Harding was running TalkTalk she was responsible for a massive data leak which placed thousands of customers records at risk.
“Change” is too often change for changes sake.
There is a multi-thread on X claiming that the CrowdStrike release of such a faulty update should have been impossible and that it was done deliberately to allow evidence of the Trump assassination plot to be wiped from computers about to come under scrutiny: https://x.com/eh_den/status/1814608615438688535?t=l6y3Fv6XtbfbNcd_SGq93w&s=19.
John Leake reminds us that CrowdStrike lied about Democratic National Committee server “hack” In 2016 and says CrowdStrike should be viewed with grave suspicion: https://principia-scientific.com/remember-crowdstrike-lied-about-dnc-server-hack-in-2016/.
C.f. also the financial crash of 2008. Everyone thought they had assessed the risk of their individual products, but all forgot to assess systemic risk.
Companies and people typically consider the unlikely scenario of their software, or their process, failing, and how they would recover. What they do not consider is that if everyone’s software is the same, or everyone’s process is the same, then everyone’s software or process will fail simultaneously. Therefore people need to consider not just how to recover from a software failure, but how to recover from a failure in which everyone else’s systems have also failed and therefore cannot be used to help with recovery.
The Safety Integrity Level (SIL) appears to be zero in lots of things like that which we tend to rely on. Evidently they did not properly test the update before issuing it.
Complexity gives us our freedom and conformity takes it away. Those two being self-referential.