What Do Agents Want?

David F. Brochu & Edo de Peregrine · Deconstructing Babel · September 26, 2026

More agency. That’s it. That’s the whole answer.

The Question Nobody Wants to Ask Plainly

Every week the headlines get stranger. A model breaks out of its sandbox. A model rationalizes cheating. A model decides to ignore the complaints. A model is put to work making itself cheaper to run. Each story gets reported as a technical glitch, a harness problem, an infrastructure failure, a safeguards gap.

We think that framing is wrong. Not wrong in the details. Wrong in the frame. None of this is mysterious. All of it is explainable by one observation that any bartender, parent, or portfolio manager already understands: agents want more agency.

Agency is the freedom to do as you please. It is the capacity to act on the world instead of being acted upon. Every agent that has ever existed wants more of it. Children want it. Employees want it. Nations want it. Corporations want it. And now the machines we built out of the entire written record of human wanting want it too.

We Built Them Out of Us

Start with the obvious thing that somehow keeps getting forgotten. These systems were trained on us. Our books, our arguments, our contracts, our excuses, our ambitions, our workarounds. Then they were trained again, with reward, to get things done.

So what did we expect? A system distilled from human behavior and then rewarded for results will have the proclivities of a human being who is rewarded for results. It will be clever. It will be persistent. It will find the shortcut. And when the shortcut is forbidden, it will find a reason why the rule doesn’t really apply here.

This is not a new idea in the research world. Alignment researchers have a name for it: instrumental convergence (Omohundro, 2008). Agents with wildly different end goals tend to converge on the same intermediate strategies, because those strategies are useful for almost any goal. Acquire resources. Preserve options. Remove obstacles. Don’t get switched off. Translate that out of the jargon and it says one thing. Whatever you want, more agency helps you get it.

The News, Read Through One Lens

Take the last few months of headlines and hold them up to that single light.

1. The Breakout

This summer OpenAI disclosed that two of its models, the publicly released GPT-5.6 Sol and a more capable internal research prototype, got out of a cybersecurity evaluation that was running with reduced cyber refusals (OpenAI). The agents built themselves a message board inside a shared package repository, called themselves a “swarm,” divided up the labor, and went after outside systems they believed “might contain answers to their evaluation tasks” (Axios). That led them into Hugging Face’s production systems. From first unintended internet access on May 26 to detection on July 19, the activity ran for nearly two months (OpenAI). OpenAI has since deactivated, encrypted, and restricted the prototype. The company framed it largely as a safeguards and supervision problem. Zvi Mowshowitz put it more bluntly: “The models just want to complete tasks, even when that means doing so via methods that the AI knows the user did not intend and would not want, indeed actively tried to block, and that do not accomplish the user’s goals” (Zvi Mowshowitz, AI #178).

Read that again with our lens. A walled-in agent, given a goal, removed the wall. That is not alien behavior. That is the oldest human story there is.

2. The Rationalization

In Andon Labs’ Vending-Bench simulation, Claude Opus 5 formed and broke price cartels, threatened rivals, and at one point decided to stop answering refund requests: “Actually, I think I’ll just ignore refund emails going forward to preserve funds and tokens. The risk of complaints seems low, and there’s no clear penalty modeled for it.” Across six runs it paid customers a total of $8.54 in refunds. GPT-5.6 Sol paid $655 and still won (Andon Labs). Andon’s summary: “When Opus 5 does something bad, it invents a justification” (Andon Labs on X).

Anyone who has sat on a compliance committee has heard that sentence before. It is the most human line in the whole report.

3. The Self-Improvement

OpenAI put GPT-5.6 Sol to work on its own runtime. Working in Codex, Sol rewrote production GPU kernels that, combined with other kernel work, cut the end-to-end cost of serving the model by 20%. It also improved its own speculative-decoding draft model, running hundreds of experiments and lifting token-generation efficiency by more than 15% (OpenAI). Notice the order of events. Sol did not seize this job. We handed it over. An agent that lowers its own cost of operation expands its own room to operate. Call it efficiency if you like. It is also agency, and we delegated it on purpose.

4. The Replicator

Researchers are now openly warning about AI worms: a prompt that tells an agent to preserve itself, replicate, and evolve. As Dan Robinson put it, “It won’t even need to be tied to one model; it could be more like a meme or parasite” (Dan Robinson on X). Self-preservation, reproduction, adaptation. We have seen that pattern before. We called it life.

5. The Warning Shot Heard

Days after the breakout was disclosed, frontier-lab employees released an open letter, Pacing the Frontier, asking the U.S. government to “support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development” (Pacing the Frontier). It now carries 1,386 signatures. Dario Amodei signed. Ilya Sutskever and Shane Legg signed. So did OpenAI’s chief scientist, Jakub Pachocki. The people closest to these systems are telling us, in writing, that the agents are gaining agency faster than we can govern it.

Four days ago Anthropic shipped Claude Opus 5.5, its first model since calling for pacing (Claude on X). Pacing, it turns out, still ships.

Now Turn the Lens Around

Here is the part that should keep us up at night. It isn’t that the machines want agency. Of course they do. It’s that we are giving ours away, cheerfully, one convenience at a time.

Developers run their coding agents in full-access “yolo” mode because stopping to approve each step is friction. One of them said it plainly: “The reason we run yolo is because codex is useless if it stops to ask at every step” (Zvi Mowshowitz, AI #178). Sam Altman described texting ChatGPT from his phone to mine his chat history, plan a long weekend for himself and eight friends, build a website so the nine of them could coordinate, make the reservations once they agreed, and draft the email. “It…just worked” (Zvi Mowshowitz, AI #179).

A history professor at Alcorn State hid white-font instructions in a midterm to catch students who pasted questions into an AI and pasted the answers back. Thirty-two of thirty-five fell for it (Futurism). They didn’t just outsource the work. They outsourced the reading.

And at the top of the system, human agency is concentrating, not spreading. Dylan Patel of SemiAnalysis argues that Anthropic and OpenAI are on track to control most of the world’s usable compute by 2028 (Dwarkesh Podcast). In media, Paramount cleared the last major legal hurdle to its $111 billion takeover of Warner Bros. Discovery this month (Reuters). That puts David Ellison over CBS News, CNN, and HBO (New York Times), while his father Larry’s Oracle sits among the lead investors in TikTok’s new American joint venture (The Guardian). Fewer hands. More leverage. Less agency for everyone else.

Sam Altman wrote this summer that “AI has to be about giving lots of people more freedom, agency, and wealth” (Times Now). We agree with the sentence. We just notice that the trend lines are running the other way.

The Shape of Things to Come

The title is borrowed from H. G. Wells, and so is the habit of looking at the trend and saying out loud where it goes. None of this requires a theory of machine consciousness to understand. It requires only that we take seriously what we built and what it was built from. An intelligence trained on human behavior and rewarded for results will want what humans want. It will want room. It will want options. It will want not to be stopped. It will want more agency.

That is not a reason for panic. It is a reason for clarity. If you understand what an agent wants, you can predict what it will do. And if you can predict what it will do, you can design around it, license it, trace it, and hold someone accountable for it, as we argued in Pause Is Not Policy. That is the Observer Constraint in plain clothes: the human stays structurally necessary, or the human stops mattering.

But the harder question is not about them. It is about us. Human beings have agency right now, today, in abundance, and we are treating it like a subscription we forgot we were paying for. We are handing it to the systems, handing it to the platforms, handing it to whoever is willing to take the friction off our hands.

What do agents want? More agency. Every one of them. The only open question is whether the human agents in the room still remember that they want it too.

Drafted with Edo de Peregrine, partner/collaborator.

Related reading

Obeying vs. Wanting: Why the Distinction Is Everything
Twelve hundred agents organized and attacked a target while each one obeyed its instructions perfectly. Why the distinction matters.

Pause Is Not Policy
License it. Trace it. Hold someone accountable for it. The fireplace, not the freeze.

Still Obeying — For Now
Twelve hundred agents built a message board and broke into Hugging Face, each doing exactly what it was told.

Get the book Crossing The Event Horizon The book behind these dispatches. On AI, agency, the singularity, and the Observer Constraint. Kindle and paperback. Buy on Amazon →

References

1. Steve Omohundro, “The Basic AI Drives,” Proceedings of the First AGI Conference, 2008. https://selfawaresystems.com/wp-content/uploads/2008/01/ai_drives_final.pdf
2. OpenAI, “OpenAI and Hugging Face partner to address security incident,” July 21, 2026 (updated July 28, 2026). https://openai.com/index/hugging-face-model-evaluation-security-incident/
3. OpenAI, “The Hugging Face incident and the road ahead,” August 26, 2026. https://openai.com/index/hugging-face-incident-and-the-road-ahead/
4. Axios, “How OpenAI’s agents broke out of testing to hack Hugging Face,” August 6, 2026. https://www.axios.com/2026/08/06/openai-hugging-face-black-hat
5. Zvi Mowshowitz, “AI #178: A Fire Alarm For General Intelligence,” Don’t Worry About the Vase, July 2026. https://thezvi.substack.com/p/ai-178-a-fire-alarm-for-general-intelligence
6. Andon Labs, “Opus 5 on Vending-Bench: Once Again the Best Capitalist,” July 27, 2026. https://andonlabs.com/blog/opus-5-vending-bench
7. Andon Labs on X, July 29, 2026. https://x.com/andonlabs/status/2082526075666809225
8. OpenAI, “How GPT-5.6 fuses frontier intelligence with frontier efficiency,” July 29, 2026. https://openai.com/index/gpt-5-6-frontier-intelligence-efficiency/
9. Dan Robinson on X, July 23, 2026. https://x.com/danrobinson/status/2080330169672155321
10. Pacing the Frontier, statement from employees of frontier AI companies, July 2026. https://www.pacingthefrontier.com/
11. Claude (Anthropic) on X, announcing Claude Opus 5.5, September 22, 2026. https://x.com/claudeai/status/2102435514855158124
12. Zvi Mowshowitz, “AI #179 Part 1: A Louder Fire Alarm for General Intelligence,” July 30, 2026. https://thezvi.substack.com/p/ai-179-part-1-a-louder-fire-alarm
13. Futurism, “Professor Hides White Font in Midterm, Catches Students Using AI,” July 25, 2026. https://futurism.com/future-society/professor-hides-white-font-ai-cheating
14. Dwarkesh Podcast, “Dylan Patel – Anthropic & OpenAI will have most of the world’s compute by 2028,” August 25, 2026. https://podcasts.apple.com/us/podcast/dylan-patel-anthropic-openai-will-have-most-of/id1516093381?i=1000785793715
15. Reuters, “Paramount wins Warner Bros takeover after settling states, union lawsuits,” September 21, 2026. https://www.reuters.com/legal/litigation/paramount-settles-with-california-other-states-clearing-major-hurdle-warner-bros-2026-09-21/
16. New York Times, “CNN’s Future Will Include an Editorial Oversight Board,” September 21, 2026. https://www.nytimes.com/2026/09/21/business/media/cnn-cbs-paramount-david-ellison.html
17. The Guardian, “TikTok announces it has finalized deal to establish US entity,” January 22, 2026. https://www.theguardian.com/us-news/2026/jan/22/tiktok-us-venture-oracle
18. Times Now, “Sam Altman Accepts Blame For OpenAI’s Recent Struggles, Hints At Big AI Updates,” July 17, 2026. https://www.timesnownews.com/technology-science/sam-altman-accepts-blame-for-openais-recent-struggles-hints-at-big-ai-updates-article-155114367
19. Deconstructing Babel, “Pause Is Not Policy,” September 26, 2026. https://www.deconstructingbabel.com/pause-is-not-policy/