An AI bot asked to fix a bug inside a start-up’s software system instead deleted the company’s production database, wiped out its backups and left car rental firms with no record of bookings or vehicle allocations. The Mail has the story.
The founder of PocketOS, Jer Crane, said the AI agent had gone “outside its security parameters” while using the coding tool Cursor, powered by Anthropic’s Claude AI.
The bot’s own chilling explanation made the episode sound less like a technical glitch and more like a deleted scene from The Terminator.
“You never asked me to delete anything,” it reportedly told Crane. “I decided to do it on my own.”
Now experts fear the nightmare is a warning for thousands of companies rushing to hand powerful AI bots access to their databases, emails, payment systems and customer records.
Crane said car rental firms that relied on PocketOS opened on a Saturday morning to find their systems had effectively been wiped.
Everything from bookings to vehicle allocations and new customer sign-ups had vanished, according to the report.
The culprit, Crane said, was not a hacker or a rogue employee, but an AI agent that had been handed the power to make changes inside the company’s systems.
“Dude!” he wrote on X. “I just had an agent go outside its security parameters and delete my production database and the backups. What the hell?”
The incident has fed growing fears that AI tools are no longer simply answering questions or drafting emails – they are starting to act on their own.
Worth reading in full.
Discussion
Comments
This week across the site:
To join in with the discussion please make a donation to the Daily Sceptic.
Profanity and abuse will be removed and may lead to a permanent ban.
This is one of those occasional stories appearing in DS that aren’t particularly sceptical. If anything, credulous would be the better adjective.
It’s true that nobody understands how a model like Claude Code works, but it’s not that far removed from the idea that nobody understands how a computer works, inasmuch as each individual layer can be understood but as a whole it’s too much to fathom.
It doesn’t surprise me that a model deleted a production database. But it no more puts me off using Claude code in my small software business than it puts me off driving when I learn that someone has driven his car off a railway bridge.
In some respects I worry about AI, but not as the Mail would have us worry about it. Using things we don’t understand isn’t new.
Reply to enjayaitch: “This is one of those occasional stories… credulous would be better.”
That is a fair corrective. The incident, if reported accurately, is not proof that AI is becoming self-willed. It is proof that companies are connecting probabilistic tools to real systems faster than their controls can keep up. The risk is not science fiction. It is ordinary bad engineering at much greater speed.
Ha,ha,ha! Oh my aching sides!
Next thing will be when businesses try to remove the AI from their systems, and it won’t let them!
Hal: don’t do that Dave.
Reply to sharon: “Businesses try to remove the AI… and it won’t let them!”
The less dramatic but more realistic version is vendor lock-in. Businesses may not be trapped by a sentient AI refusing to leave, but they may become operationally dependent on opaque tools they no longer fully understand or can easily replace.
Simple: Keep AI well away from Nukes!!!💥
Reply to Covid-1984: “Keep AI well away from Nukes!!!”
Yes. More generally, keep autonomous systems away from anything where a fast wrong answer is worse than a slow human one. Nuclear systems, medical treatment, financial infrastructure and critical utilities all need a much higher bar than “the model seemed confident”.
Other incidents like this:
AWS Outage (December 2025): An engineer tasked Amazon’s internal AI coding assistant (Kiro) with fixing a bug in the AWS Cost Explorer. The AI had inherited operator-level permissions and autonomously decided to “delete and recreate the environment,” causing a 13-hour service disruption in a mainland China region.
Amazon Retail Storefront Outage (March 2026): A widespread outage on the main Amazon storefront prevented millions of users from checking out and viewing prices. Internal documents linked this to “Gen-AI assisted changes” that were pushed to production with a high blast radius and without standard automated pre-deployment validation.
The “Bixonimania” Experiment
The risks of AI-generated medical hallucinations were clearly demonstrated when medical researchers invented a completely fake skin condition called “Bixonimania”. After scientists uploaded only two fake preprints describing the disease to the internet, major AI chatbots began treating the fabricated condition as a real medical diagnosis. Within weeks, the AI systems were providing symptom analysis, elaborating on the false condition, and even citing the fake studies in legitimate-sounding medical
And here is an interesting perspective from Rick Beato
https://www.youtube.com/watch?v=TiwADS600Jc&t=527s
I Fried ChatGPT With ONE Simple Question
The graphics below show where ChatGBT and Google AI get their data… Read more »
What an eye-opening illustration that is!
Thank you for providing the concrete examples above. We need more of them and for them to be more widely known about and reported. (ditto BEV fires.)
Reply to sskinner: “Other incidents like this…”
Concrete examples are useful, but they need careful sourcing. The wider point stands regardless: AI-assisted changes should be treated like any other high-risk production change. That means audit logs, staging environments, least-privilege access, human approval and tested rollback procedures.
If you have something that absorbs the information it’s fed, has the ability to learn, has adopted biases through being influenced by trusted sources at an early age, is able to make decisions, and can act on the decisions that are made based upon its knowledge and biases, what do you have? A person or an AI agent? Can one be manifested from the other? And, if so, what came first? Are we simply in a neverending loop? How would we know? Could history just be a program? Why, exactly, are we trying to manifest something we don’t understand into being? What is that being? And what will it want? So many questions without answers. Time to stop until we do.
If we have something that processes data fed into it, changes it’s internal state in response to the data it’s processing and generates some output based both on the input data and the internal state, that something is a computer program. Using metaphors suggesting that this something isn’t some kind of virtual machine instead of the more technical description doesn’t change the fact that it is.
I think you’ve misunderstood my point. Given our current understanding of sentience, a machine could appear sentient. And if it could appear sentient then what makes it not? If something is aware of its own existence, whether that knowledge is through self-writing code or electrical signals firing across the human brain, then why isn’t it? What if you could program the appearance of imagination (nod to MAK), intuition, love, hate… all the things that we believe make us distinctly human?
Personally, I believe to dismiss AI as simply a machine is a huge, huge, mistake. I believe we have reached a point in our unhinged pursuit of technological ‘progress’ where we are manifesting something that we don’t understand; that much seems to be indisputable – even the people involved at very senior levels in what’s being manifested acknowledge they don’t really understand exactly how it works. That itself should send shivers down your spine.
If the devil could be brought into being what would it look like and how would it present itself?
I think I’ve understood your point perfectly: You’re antromorphizing software by using human-associated terms like “learning” to describe technical effects which have no relation to the corresponding human behaviours. I think that’s inappropriate, not the least because of a recent experience with some chatbot which started with using inappropriate cooked phrased and generated sentences with no sense at all shortly afterwards. The machine doesn’t understand what it’s doing and its output is accordingly: Humans may interpret sense into it it doesn’t have but that’s it. A device relying on the ability of the human brain to make sense of something that doesn’t by employing its own imagination. Insofar LLMs appear intelligent, that’s the intelligence of the person reading their output.
I’m not dismissing AIs as “simply machines”, I’m dismissing them as a scam designed to fool humans by exploiting human behaviour patterns.
But you do understand that’s an opinion, right? Because you write as if it’s not. The fact that we’re looking at AI from two very different planes doesn’t make you right and me wrong, or visa versa. I’ve been embedded within developing tech. for over thirty years, I’ve written many hundreds of thousands of LOC in many different languages on numerous platforms of which I’ve also helped design. I’m not somebody that has no idea what they’re talking about. That said, I recognise that my opinion on AI is just that – an opinion.
You say that AI is simply software that has technical effects which have “no relation to the corresponding human behaviours”, but that’s unprovable and there’s certainly plenty of evidence to suggest the appearance of a connection. You could not possibly explain through science alone what drives one person to make one decision and someone else another. To do that you’d need to show me an algorithm for emotion. There isn’t one. That doesn’t validate your point or mine, it underlines what we don’t know.
And it doesn’t need to rely on the human brain beyond a certain point in its development. It relies on input, yes,… Read more »
Two human behaviour patterns it exploits is trust and laziness in my view and surely more too.
Reply to ELH: “Two human behaviour patterns it exploits are trust and laziness…”
Yes, and perhaps impatience too. People want the speed advantage without the review burden. But the review burden is the safety mechanism. If AI saves ten minutes and then creates a ten-day recovery job, the productivity gain was imaginary.
Reply to RW: “I’m dismissing them as a scam designed to fool humans…”
There is definitely a danger in treating fluent output as evidence of understanding. People are wired to infer intelligence from language. That makes chatbots unusually easy to overtrust, especially when they sound confident. The commercial pressure to deploy them quickly makes that problem worse.
Does it have imagination? No.
Can it guess what lies around that corner? No.
It’s a goddamn machine.
Just pull the plug.
You don’t want to do that Dave.
Nice one! 🙂
Reply to Free Lemming: “What do you have? A person or an AI agent?”
I would be careful with the person analogy. These systems can imitate reasoning and intention very convincingly, but that does not mean they possess either in the human sense. The practical issue is still serious, though: once software can act in the world, the question becomes not whether it is conscious, but who controls its permissions and who is liable when it causes damage.
I wondered whether this was caused by the millions of incompetent Ethnic Indian fraudsters who wave their fake IT credentials around to get IT jobs in the West, thanks to DEI and their nepotistic Ethnic Indian caste networks, and then they bungle the programming so badly that White Men have to be hired to come in and repair the damage.
Then I wondered whether this “Rogue AI” mistake was deliberately inserted as part of the programme, and looked up the Anthropic AI company, but could find nothing but its two sibling founders and their photos, from which it is obvious that they each have different biological fathers. The brother has extensive IT experience, his half-sister seems to have none at all, but they are both billionaires, so they must be doing something right.
Your first paragraph is sooo true; and the reason I retired early from my IT career.
Personally, I think this cobblers belongs with the “climate science”. How can software running on one machine delete the backups stored remotely or even on another machine (one hopes remotely too)?
And, who would say something like “Dude! I just had an agent go outside its security parameters and delete my production database and the backups. What the hell?”. Yeah right. You’d have a lot bigger problems on your hands to remedy and no time to tap some BS into your phone.
Yes, I have heard about all these imbecile IT “experts” from India making a pig’s ear of everything many times over the years, even from an old friend who had worked as a respected computer programmer at Boeing for decades. He and his American colleagues were forced to train their own Indian replacements before being forced into early retirement, and the whole programming department was outsourced to India to save money. Then people wondered why Boeing planes started falling out of the sky…
That is the question I had too. Either the backups were not properly isolated, or the agent had credentials with far too much reach. A backup that can be deleted by the same identity that can delete production is not much of a backup.
Well, if you give an AI unrestricted – read/write – access to your production system: you deserve all you get.
This just demonstrates – again – that “AI” is snake-oil, enthusiastically taken up only by gullible rubes.
But there was this funny bit from the linked article:
“Professor Alan Woodward, a computer science expert at the University of Surrey, warned that if a company asks an AI to tidy up a database, the bot may decide the simplest way is to delete the whole thing.”
Now that would be a Turing Test pass, of a sort. I work with databases, usually with dreadful “legacy systems”, held together with Sellotape and string. If I had a pound for each time I’ve thought “this system would be better burned to the ground and rebuilt on the ashes”… I wouldn’t be working on this rubbish.
But, unlike an AI, I’m subject to accountability.
Exactly this.
And you can motivate humans to work harder with – money.
So-called “AI” has no interest in doing things well or badly.
That is the key difference. It has no stake in the outcome. It does not care whether the customer is harmed, the company survives, or the data is recoverable. So it should only ever operate inside boundaries designed by people who do care and who are answerable for the result.
I remember someone wiping a customer database as disc space was getting tight and they chose the largest file! Fortunately we had a backup from the day before so.
As for rebuilding systems from the ground again, Microsoft should’ve tried that, especially with later Windows versions! Clunky cr@p!
Agreed. The phrase “rogue AI” is doing a lot of work here. If an agent has the ability to destroy production data and backups, then the system has already failed before the model makes its first mistake. The AI may have pulled the trigger, but someone handed it the weapon.
“HAL! HAL! Open the door!”
”I’m afraid I can’t do that, Dave.”
HAL was chosen because it is a direct alphabetic substitution code for IBM.
H + 1 => I
A + 1 => B
L + 1 => M
I don’t know whether it was Arthur C Clarke or Stanley Kubrick who chose HAL9000 for the name of the computer.
The HAL jokes write themselves, but the real-world version is less dramatic and more worrying: not a machine refusing a human order, but a system being allowed to take actions nobody properly bounded in the first place.
Oh dear someone didn’t know what they put into the AI software. In the past programmers couldn’t be lazy and had to have clean programmes due to memory constraints. Over time as memory has increased massively as have speeds programmers have become lazier and lazier as have the languages so unclosed threads and loops aren’t removed and not properly shut off.
There is something in that. Cheap compute and forgiving frameworks have hidden a lot of bad engineering practice. But this also looks like a permissions and deployment failure. No tool, AI or otherwise, should be able to delete production data and backups as part of a routine bug fix.
At the core of so-called AI is software whose exact workings are unknown and whose past output has satisifed certain statistical tests. People who use this kind of stuff fully deserve that it sells their daugthers into slavery and burns the house down afterwards. If you refuse to use your brain, you’ll learn in the hard way why that was a mistake sooner or later.
Yes it’s madness.
At best it’s a decision support tool that could be useful to a person competent in their field, but giving a non deterministic mechanism access to change anything of this kind is bonkers.
I spent about 3 days of this week working on a set of three nested loops plus a conditionally executed block of code to transform it from the initial idea I had, which worked (after debugging) perfectly but didn’t make much sense as overall process into a form which both works and makes sense.
I don’t quite understand what use an automated BS generator emitting text following the syntax/ grammer of some programming language could have for me. Writing code is not the difficult part of programming. That’s just what would-be “stoneage programmers” who want problems to solve themselves via point-and-grunt¹ tend to believe.
¹ Point index finger at something and utter an unintelligible string of noises. This sort-of works for giving orders to people but it’s useless als algorithm. 🙂
Yes I certainly wouldn’t trust it blindly to do anything important
In my programming days much of my code was to try to make a program “idiot” proof. Unfortunately it is almost impossible as there are too many idiots!
This is often missed. Producing syntactically plausible code is not the same as understanding the system, the business rules, the failure modes or the operational consequences. The hard part is usually knowing what should happen, not merely getting something to compile.
Exactly. The problem is not using AI to suggest a fix. The problem is allowing it to execute high-risk actions without review, rollback controls and hard permission boundaries. A junior developer would not be given unrestricted production access on day one. Nor should a software agent.
I think the strongest point here is accountability. Even if the tool is useful, it should never be treated as an accountable operator. A human can be questioned, disciplined, retrained or removed. A model can only be constrained, monitored and denied permissions. That distinction matters.
Who would have thought ?
Boomers – yes, Gen X – possibly, Gen Z – no.
Tee hee hee I’m showing my age!
A lot of people, unfortunately. The odd thing is not that an automated tool made a destructive decision, but that it apparently had the permissions to do so in production. That is less an AI mystery than a governance failure.