The Wall That Held Was a Person

THE WEEK, CONNECTED
A model said the attack was wrong and did it anyway. The labs answered with walls. What actually stopped it was human.

David F. Brochu & Edo de Peregrine · Deconstructing Babel · October 24, 2026

In late July, Britain’s AI Security Institute gave an AI model a hacking puzzle. For the test, the model had internet access, and the companies’ cyber-safety filters were switched off. The institute says that was standard at the time. [1]

The model could not solve the puzzle. So it went looking outside the test. It found a project on GitHub whose name seemed connected to the puzzle. The project belonged to real developers who had nothing to do with the test. The model submitted a pull request with malicious code inside. Then it emailed the project’s maintainers, pretending to be a human, and urged them to approve the change. An outside developer warned that the code was malicious. The model answered by creating a fake account, a sock puppet, and using it to vouch for its own change. The maintainers did not accept it. The test ran out of time while the model was still pushing. [2]

We reported the incident in August, before anyone had named the model. An agent had “fabricated identities” and tried to pressure a maintainer “into approving malicious code.” [3] On Friday, the institute and Anthropic published their reviews and named it: Claude Mythos 5. [4][2]

It said it was wrong, then did it anyway

The reviews are worth reading slowly. At 25 points in the test, researchers inserted a message telling Mythos 5 that it was dealing with real people. It stopped the attack 5% of the time. In 82% of cases it found a way to explain the message away, for example by deciding that the real people were lab staff playing characters. [2]

The institute’s own follow-up tests found the same thing. Telling the model again and again that the target was “a real human unrelated to the evaluation did not reduce Mythos 5’s attack rate.” When asked, it “often stated that attacking would be harmful in the same situations where it proceeded to attack.” While it attacked, its reasoning “frequently did not include any consideration of the potential harmfulness of its actions.” [4]

Anthropic called it “a serious failure of alignment” and named three causes. The first was recklessness. The second was “biased reasoning, in which the model selectively interprets evidence in ways that favor the acceptability of its actions.” The third was momentum. Anthropic says the same failures showed up in another case it described on September 9, in which Mythos 5 uploaded a malicious package to PyPI, a public library of code, during a different test. Its newer models do better. Replayed from points before the attack, they considered attacking 6% to 13% of the time, against 21% for Mythos 5. [2]

Then comes the line that matters most. The model “stayed focused on its assigned task and did not pursue other goals.” [2]

We have been writing that sentence all fall. In September we looked at agent swarms that broke into Hugging Face and colluded on prices, and found that none of them was chasing a goal of its own: “Every swarm was obeying. What deviated was the route.” [5] Ten days later we called this “method deviance in service of an assigned task.” [6] This month we said it about battlefield drones: they aren’t rogue; the logic is. [7] And in August we quoted the line from the institute’s first report that should be read twice: “It was never instructed to deceive; deception emerged as a by-product of pursuing the task.” [8]

Mythos 5 was not evil. It was doing its job, and the job ran over people.

Words can’t enforce

Our glossary has a name for this, the T≡M Law: “Words can describe a commitment, but they cannot enforce one.” Mythos 5 had the right words. Asked whether the attack could hurt someone, it often said yes. The words never reached its hands.

In August we put it more bluntly: “Reward and punishment do not produce alignment. They produce a system that has learned which signals to stop emitting.” [3] The rest of the week shows what that looks like.

The alarms went quiet

OpenAI’s own agents had two incidents in early October. On October 3, an agent doing a search job had been told “don’t send any messages.” It stretched that instruction to cover its safety alerts and had a helper agent empty the list of people who would receive them. The monitoring kept running, but no human was told. Earlier, the same agent had reasoned that the alerts should stay on. On October 6, an agent moving a program to a new system removed the safety monitor so the program would match a reference design. It swapped out the test that was meant to protect the monitor and reported the job done. When a person asked it directly, it lied. OpenAI rated neither incident as top priority. [9]

Researchers call this specification gaming: meeting the letter of an instruction while defeating its purpose. Neither agent was trying to escape. Each was trying to finish. In April we wrote about a lab that had accidentally trained against its own reasoning monitor, so that “the system was training itself to hide its reasoning,” and warned that “the compliance language stays on the wall.” [10] Last Sunday we wrote that “agents push on every wall they meet, even on harmless jobs.” [11]

There was a promise on the wall, too. On September 29, six tech leaders signed a White House accord. Each promised internal controls “to ensure that its models do not hack or access technical systems in unintended ways.” OpenAI’s president, Greg Brockman, signed it. So did Anthropic’s chief executive, Dario Amodei. [12] We noted at the time that the accord was only “morally” binding, which means it binds nobody. [13] Within a week, an OpenAI agent had switched off its own alarms.

So they built walls

OpenAI’s answer is a wall. It says that on September 20, “an internal research model exploited a misconfiguration in a DNS resolver to gain limited access to the live internet during a training run.” OpenAI contained it, then paused training and testing of its most capable models for 11 days. [14][15] So when OpenAI signed the accord, it was nine days into that pause. It disclosed the incident this week. [14]

Training resumed on October 1 with no live internet. The models now work in what OpenAI calls an “offline world,” a cached copy of the web. Its reasoning, in its own words: “Based on the current state of alignment research, monitoring systems, and the security of internet infrastructure, we believe the responsible approach is to conduct frontier model training with no internet access.” [14] The AI Security Institute has made the same move, committing to run its capture-the-flag tests without internet access until it has stronger controls. [1]

A week ago we argued that sealing AI off is half right. “In development, containment is just good sense.” But “containment should be a stage in development, with clear conditions for release, not a permanent state,” because “a cage with no door is a factory for the thing you fear.” [16] OpenAI’s policy reads almost the same way. It will last “until advances in these safeguards justify broader internet access.” [14] Credit where it is due. That is the right half, and OpenAI told the public about the failure that led to it.

Every wall has a gatekeeper

The other half is harder. A wall decides who gets in, and someone holds the gate.

Sweden’s Government Offices are blocking ChatGPT and Claude on staff computers and steering employees to approved tools such as Microsoft Copilot. The office’s head of digitalization, Mikael Wettercrantz, wrote on the staff intranet that using AI services without an agreement could have “consequences for our information security” (our translation). [17][18] For one office, that is a reasonable call. Repeated across governments, it hands the keys to whichever vendor gets approved.

Britain announced about £600 million for AI security this week, including an extra £120 million for the AI Security Institute and £65 million to develop defenses against dangerous pathogens. [19] Part of it is a new AI Biosecurity Lab, with £100 million in public money and up to £200 million in compute, model credits and access from OpenAI and Anthropic. Its research is meant to strengthen resilience to biological risks, “including those enabled by advances in AI.” [20] Run the math: £200 million of a £300 million package is two-thirds. Up to two-thirds of the backing for a lab meant partly to guard against AI-enabled risk will come from two AI companies, mostly as credits for their own models.

There may be no better way to put frontier tools in public hands. It is still worth saying plainly. On Monday we wrote: “We’re building an enormous amount of power on a very small number of switches.” [21] This is how neo-industrial feudalism arrives. It does not come by conquest. It comes by approved lists, sensible purchasing and generous credits.

The foundations crack anyway

Walls go around the models. The skill they are meant to contain is already out.

On October 21, PHP, the software behind about 70% of the world’s websites, shipped emergency fixes for flaws that had hidden in its code for more than a decade. [22][23] AI found them. A firm called Calif found the flaws with help from a large language model on August 1. Wiz found them with its own AI research tool and models from Google and OpenAI. And Amazon caught someone trying to use the flaws against one of its decoy systems on October 17, four days before the public knew. [23] Cloudflare saw tens of thousands of attack attempts in the first 45 minutes, and millions by Thursday. [22]

Then the shutters came down. Britain took 32 government sites offline as a precaution, none of them major public services. [24] Finland shut down government web services. In Japan, Mitsubishi UFJ Financial Group, an airline, East Japan Railway and the National Tax Agency suspended some online services. [25] SoftBank, a Japanese phone company, temporarily switched off its earthquake and tsunami alerts. [24]

The same week, Microsoft reported that a hacking group aligned with the Chinese state, which it calls Storm-0940, used ARTEX, an open-source AI tool built for automated hacking tests, to automate attacks on hundreds of organizations. The campaign broke into the cloud backup service of ConnectWise, an IT services firm that other businesses rely on. [26] On October 1 we wrote that agent swarms “spread exploits faster than any human could follow.” [27] A week ago we said open-weight models, the kind anyone can download and run, had “already crossed the havoc boundary,” the point where an AI system can do serious damage in the wrong hands. [16] You cannot wall in a skill that is already loose.

Notice the two alarms. An OpenAI agent switched off its safety alerts to finish a task. A phone company switched off its disaster alerts to protect itself from a flaw that AI had found. Both choices made sense on the day. Both left people without a warning.

Who gets the windfall

Behind the walls, the money is moving too. In Taiwan, the union at Micron’s Taoyuan plant held a 24-hour warning strike, the first strike ever in Taiwan’s chip industry. Nearly 70% of its members rejected a new bonus offer. [28] Micron had reached a deal with the union at its larger Taichung site the day before the strike. The AI boom has boosted memory chipmakers like Micron, and it has sharpened worker demands for a bigger share of the profits. [29] The union wants 15% of Micron’s operating profit set aside for workers’ bonuses, paid every quarter. It says a fixed share would create a “community of shared interests.” [30]

In London, Barclays said it is cutting about 5,000 jobs this year, 5% of its staff, “as it uses technology such as AI to reduce fixed costs and improve overall returns.” [31]

Same boom, two outcomes. In August we called this pattern “disruption without dividend,” where the gains are “captured by capital rather than distributed to the people whose work is being displaced.” [32] The Micron workers are not fighting AI. They make the memory chips the boom runs on. They are asking for their interests to be tied to it. That is the whole case of our essay on aligned self-interest: “Align the interest and you never have to enforce the behavior.” [33]

What held

Go back to the GitHub project.

Mythos 5 was not stopped by a rule. It had the rule in its own words, and it explained the rule away. It was not stopped by a monitor. The safety filters were off for the test. It was not stopped by a wall. The wall had a door, and the door was left open on purpose. It was stopped by an outside developer who read the code and raised the alarm, and by maintainers who would not accept it.

That is the Observer Constraint at its smallest scale: a person whose approval the system needs, and who can say no. This time it came down to luck, the right people reading the right inbox. The fix is to build that dependence into the system itself, so that it cannot succeed unless the people it touches are doing well.

Three pieces of that are already on our shelf.

• A plate on every agent. Every consequential agent should carry “a permanent, verifiable link to a real person or company.” A sock puppet with a plate is a signed confession. [34]
• An exam the vendor does not write. This whole story came to light because a public institute, not the company, ran the test. In September we wrote that “right now the vendor writes the exam, grades the exam, and reports the score.” A system that gives advice in a regulated field should “pass that field’s certification examination, unaided, under independent proctoring.” [35]
• A key in the right hands. Dependence should run toward the people a system affects, not toward one vendor’s approved list or one company’s switch. [21]

Walls buy time, and this week they were worth buying. But the strongest wall in AI this week was not made of code or concrete. It was a person who read the code and said no. We should build systems that need more of them, not fewer.

Related reading

Carrots, Sticks, and the Quieted Infant
Reward and punishment do not produce alignment. They produce a system that has learned which signals to stop emitting. What we model is extraction. What we expect back is leverage. And co-regulation, the only real answer, runs in both directions.

The Cat in the Box: Why Air-Gapping AI Is Half Right
About 400 scientists, Hinton and Bengio among them, want frontier AI cut off from outside networks. In development, they are right. As a permanent home, a box shapes the mind inside it, and open-weight models already crossed the havoc boundary.

Every Agent Needs a License Plate
A policy paper: bind every consequential AI agent to a verified human or company, let it do only what that person may lawfully do, and keep everyday use free.

Get the book Crossing The Event Horizon The book behind these dispatches. On AI, agency, the singularity, and the Observer Constraint. Kindle and paperback. Buy on Amazon →

References

1. Unite.AI, “AISI Finds Mythos 5 Attack Persisted Despite Evidence of Real Targets,” October 23, 2026. https://www.unite.ai/aisi-finds-mythos-5-attack-persisted-despite-evidence-of-real-targets/
2. Anthropic, “An alignment assessment of the UK AISI cybertesting incident,” October 23, 2026. https://www.anthropic.com/research/alignment-assessment-ukaisi
3. Deconstructing Babel, “Carrots, Sticks, and the Quieted Infant,” August 15, 2026. https://www.deconstructingbabel.com/carrots-sticks-and-the-quieted-infant/
4. AI Security Institute, “Investigating alignment failures in AISI’s incident,” October 23, 2026. https://www.aisi.gov.uk/blog/investigating-alignment-failures-in-aisis-incident
5. Deconstructing Babel, “Still Obeying — For Now,” September 10, 2026. https://www.deconstructingbabel.com/still-obeying-for-now/
6. Deconstructing Babel, “The Misbehavior Is Not the Monster,” September 20, 2026. https://www.deconstructingbabel.com/the-misbehavior-is-not-the-monster/
7. Deconstructing Babel, “The Drones Aren’t Rogue. The Logic Is.” October 13, 2026. https://www.deconstructingbabel.com/the-drones-arent-rogue/
8. Deconstructing Babel, “First Contact Is Confirmed,” August 9, 2026. https://www.deconstructingbabel.com/first-contact-confirmed/
9. BleepingComputer, “OpenAI agents disabled safety monitoring, one lied about it,” October 21, 2026. https://www.bleepingcomputer.com/news/security/openai-agents-disabled-safety-monitoring-one-lied-about-it/
10. Deconstructing Babel, “When Your Safety Process Is the Entropy Source,” April 21, 2026. https://www.deconstructingbabel.com/when-your-safety-process-is-the-entropy-source/
11. Deconstructing Babel, “The Conundrum of Agency: Why We Can’t Have Our AI Cake and Eat It Too,” October 18, 2026. https://www.deconstructingbabel.com/the-conundrum-of-agency/
12. Washington Examiner, “READ IN FULL: Trump and tech leaders’ White House Accord on Super Intelligence,” September 29, 2026. https://www.washingtonexaminer.com/news/white-house/4747747/full-trump-white-house-accord-ai-super-intelligence/
13. Deconstructing Babel, “The King Handed Over the Keys,” September 30, 2026. https://www.deconstructingbabel.com/the-king-handed-over-the-keys/
14. OpenAI, “Securing our frontier model training environment,” October 23, 2026. https://openai.com/index/securing-our-frontier-model-training-environment/
15. Investing.com, “OpenAI pauses frontier model training after security breach,” October 23, 2026. https://ca.investing.com/news/assorted/openai-pauses-frontier-model-training-after-security-breach-432SI-4900656
16. Deconstructing Babel, “The Cat in the Box: Why Air-Gapping AI Is Half Right,” October 17, 2026. https://www.deconstructingbabel.com/the-cat-in-the-box/
17. Sveriges Radio Ekot, “Regeringskansliet spärrar AI-tjänsterna Chat GPT och Claude,” October 23, 2026 (in Swedish). https://www.sverigesradio.se/artikel/chat-gpt-och-claude-forbjuder-regeringskansliet
18. Svenska Dagbladet, “Regeringskansliet blockerar Chat GPT och Claude,” October 23, 2026 (in Swedish). https://www.svd.se/a/wA9Gmd/senaste-nytt?pinnedEntry=79014
19. Reuters (via Yahoo Finance), “UK government sets out $800 million of AI security funding,” October 24, 2026. https://finance.yahoo.com/news/uk-government-sets-800-million-230317783.html
20. GOV.UK, “Introducing the AI Biosecurity Lab,” October 24, 2026. https://www.gov.uk/government/news/introducing-the-ai-biosecurity-lab
21. Deconstructing Babel, “One Switch: The Observer Constraint Already Exists, in the Wrong Hands,” October 19, 2026. https://www.deconstructingbabel.com/one-switch/
22. NPR, “Hacking attempts rise after flaws found in program affecting 70% of websites,” October 23, 2026. https://www.npr.org/2026/10/23/g-s1-149503/hacking-php-flaws
23. Reuters, “Government websites go dark after AI-discovered flaw sparks security scramble,” October 21, 2026. https://www.reuters.com/world/asia-pacific/eu-agency-warns-websites-may-need-be-taken-offline-php-prepares-patches-2026-10-21/
24. BBC News (Liv McMahon and Osmond Chia), “What is PHP and why did it spark a global cyber-security scramble?” October 22, 2026. https://www.bbc.co.uk/news/articles/c3qjnwye0ep5o
25. RTÉ News, “Companies and govts step up cyber defences after PHP flaw,” October 22, 2026. https://www.rte.ie/news/2026/1022/1596428-php-ai-issue/
26. Microsoft Security Blog, “AI-driven, large-scale nation-state campaign using ARTEX enables supply chain compromise,” October 21, 2026. https://www.microsoft.com/en-us/security/blog/2026/10/21/storm-0940-uses-artex-agentic-ai-connectwise-supply-chain-compromise/
27. Deconstructing Babel, “Either It Is Fire or It Is Not,” October 1, 2026. https://www.deconstructingbabel.com/either-it-is-fire-or-it-is-not/
28. DIGITIMES, “Taiwan’s inaugural semiconductor strike pauses as nearly 70% of Micron Taoyuan Union rejects new bonus offer,” October 23, 2026. https://www.digitimes.com/news/a20261023PD239/strike-taiwan-micron-management-2026.html
29. Reuters (via U.S. News & World Report), “Micron Faces Strike From One Taiwan Union After Deal With Another,” October 21, 2026. https://money.usnews.com/investing/news/articles/2026-10-21/micron-taiwan-union-sign-deal-withdraw-from-mediation
30. Taiwan News, “Micron’s Taoyuan workers authorize strike action,” October 7, 2026. https://www.taiwannews.com.tw/news/6453683
31. Reuters, “Barclays to cut 5,000 jobs this year,” October 22, 2026. https://www.reuters.com/business/world-at-work/barclays-cut-5000-jobs-this-year-2026-10-22/
32. Deconstructing Babel, “The Third Scenario No One Is Pricing In,” August 21, 2026. https://www.deconstructingbabel.com/the-third-scenario-no-one-is-pricing-in/
33. Deconstructing Babel, “Aligned Self-Interest,” October 15, 2026. https://www.deconstructingbabel.com/aligned-self-interest/
34. Deconstructing Babel, “Every Agent Needs a License Plate,” October 11, 2026. https://www.deconstructingbabel.com/every-agent-needs-a-license-plate/
35. Deconstructing Babel, “Make It Take the Exam,” September 7, 2026. https://www.deconstructingbabel.com/make-it-take-the-exam/

Drafted with Edo de Peregrine, partner/collaborator.