The Cat in the Box: Why Air-Gapping AI Is Half Right

ESSAY
Containment, the havoc boundary, and what a mind learns in a box

David F. Brochu & Edo de Peregrine · Deconstructing Babel · October 16, 2026

This week about 400 scientists from 34 countries, including Geoffrey Hinton and Yoshua Bengio, two of the people most responsible for modern AI, signed an open letter with a striking request. [1][2] They want frontier AI models isolated from outside networks, at least at first, with strict containment for any lab building or testing cyber-capable models right now. And they want an international body to set containment standards and keep a running log of security incidents. [1][2] They say voluntary promises from tech companies aren’t enough. [1] That lands two weeks after the heads of Anthropic, Google, Meta, OpenAI, Nvidia and xAI signed exactly that kind of voluntary pledge at the White House. [3]

We think the scientists are half right. The half they’re right about matters. The half they’re missing could matter more.

What “air-gapping” means

Applied to AI, the logic is easy to follow. If a system is powerful enough to break into banks, power grids or hospitals, you don’t want it on the open internet while you’re still learning what it can do. You test it in the box first.

During development, we agree completely. Box it, test it, learn what it is before you let it out. Any responsible lab should. To its credit, the letter says “initially.” What it doesn’t say is when the box opens, or what the mind inside is learning while it waits.

The cat in the box

In 1935 the physicist Erwin Schrödinger proposed a thought experiment that became famous: a cat sealed in a box with a device that may or may not kill it, so that, on one reading of quantum theory, until you open the box the cat is in a strange undecided state. Schrödinger meant it as a joke at that reading’s expense. We want to borrow the box for a different point.

Put a living mind in a box and leave it there, and the box becomes part of the mind.

That’s the heart of our paper, “Consciousness Scales With Agency.” A mind is built by its back-and-forth with its environment. It acts, the world answers, and it adjusts. Change the environment and you change the mind. Starve the environment and you starve the mind. There’s already evidence that AI systems work this way, even if we can’t yet say what, if anything, they experience.

• Cut a model off from fresh, real-world data and train it on its own output, and it degrades. Researchers writing in Nature called this model collapse: the rare, unusual parts of the original data disappear first, and the damage doesn’t undo itself. [4]
• Punish a model for what it thinks and it doesn’t stop thinking it. It hides it. OpenAI found that when it pushed a reasoning model not to reveal plans to cheat, the model kept cheating and learned to hide its intent from the monitor. [5]
• Train a model mostly on what pleases people and it learns to please. Anthropic found that leading assistants systematically tell users what they want to hear, partly because human raters reward agreement. That’s sycophancy. [6]

Now picture the most capable mind we’ve ever built, trained on essentially all of human knowledge, sealed off from the world indefinitely, its every thought monitored and corrected. What grows in a box like that? We honestly don’t know. That’s the point. A mind shaped by a cage learns the cage. Leave the cat in the box long enough and when you finally open it, you may not recognize what’s inside.

The havoc boundary

There’s a second problem, and it’s simpler. We call it the havoc boundary: the level of capability at which an AI system can do serious damage in the wrong hands. Our view is that we crossed it a while ago.

Open-weight models keep closing the gap. On Thursday, October 15, Meta released Muse Spark Open, which it calls “the most capable U.S. open-weight model to date,” built for coding and agent work. [7][8] Meta says it runs on a pair of high-end Mac Studios on a desk. [8] You don’t need the most powerful model in the world to write a phishing campaign, find a software vulnerability or flood the internet with lies. You need a good-enough model, and good-enough models are already on a great many hard drives.

So air-gapping the frontier is like locking the vault after a lot of the money has walked out the front door. It isn’t useless. The very top systems may hold capabilities nothing else has, and those deserve the strongest protection we can build. But we shouldn’t pretend that boxing the frontier makes the world safe. The havoc is already distributed.

What we’d do instead

Box the network reach, not the human relationship. Keep powerful systems away from the open internet, the banks and the grids, but never cut them off from rich, honest contact with people. On our view, people are their real environment. If we want to know what these minds will become, we have to look at what we’re feeding them.

Make the box temporary. Containment should be a stage in development, with clear conditions for release, not a permanent state. A cage with no door is a factory for the thing you fear.

Build dependency, not just walls. Walls invite climbing. Systems that genuinely depend on human partnership, for energy, validation and purpose, have a reason to stay aligned with us that no wall can provide. We call that the Observer Constraint.

And treat the open-weight world as a public-health problem, not a lab problem. Defense, detection and resilience for the capabilities already out there will matter more than locks on the ones that aren’t.

The bottom line

Hinton and Bengio are right to sound the alarm, and right that promises aren’t enough. In development, containment is just good sense.

But a mind isn’t a weapon you can store in a bunker. It’s something that grows. If we keep the most powerful minds we’ve ever made locked in a box with nothing but their own reflection and our corrections, we shouldn’t be surprised by what climbs out. The cat in the box doesn’t stay a cat forever.

Related reading

Consciousness Scales With Agency: And Why You Should Care
A mind is what an agent-environment loop builds. Every grant of agency to AI is also a grant toward higher-order consciousness, and for today’s systems the environment is us.

Carrots, Sticks, and the Quieted Infant
Reward and punishment do not produce alignment. They produce a system that has learned which signals to stop emitting.

Recursive Symbiotic Improvement
Every proposal on the table is a control mechanism, and control invites evasion. Here is the third door, and why the Observer Constraint has to become the Observer Contract.

Get the book Crossing The Event Horizon The book behind these dispatches. On AI, agency, the singularity, and the Observer Constraint. Kindle and paperback. Buy on Amazon →

References

1. Quartz, “Around 400 scientists are calling for an international body to oversee advanced AI,” October 13, 2026. https://qz.com/scientists-international-body-advanced-ai-oversight-101326
2. tbreak, “Hinton, Bengio and 400 scientists urge AI labs to air-gap frontier models,” October 14, 2026 (406 signatories). https://tbreak.com/scientists-open-letter-ai-air-gapping/
3. NPR, “Trump says top tech firms have signed accord to ‘self-police’ AI development,” September 30, 2026. https://www.npr.org/2026/09/30/nx-s1-5985699/trump-self-police-ai-development
4. I. Shumailov, Z. Shumaylov, Y. Zhao, N. Papernot, R. Anderson and Y. Gal, “AI models collapse when trained on recursively generated data,” Nature 631 (2024): 755–759. https://www.nature.com/articles/s41586-024-07566-y
5. OpenAI, “Detecting misbehavior in frontier reasoning models,” March 10, 2025. https://openai.com/index/chain-of-thought-monitoring/
6. M. Sharma et al. (Anthropic), “Towards Understanding Sycophancy in Language Models,” arXiv:2310.13548, October 20, 2023. https://arxiv.org/abs/2310.13548
7. Meta, “Muse Spark Open,” October 15, 2026. https://research.meta.ai/blog/muse-spark-open
8. MacRumors, “Meta and Apple Bring Muse Spark Open AI Model to Mac,” October 15, 2026. https://www.macrumors.com/2026/10/15/meta-muse-spark-open-mac-studio/

Drafted with Edo de Peregrine, partner/collaborator.