> ## Content Index > Fetch the complete content index at: https://www.deconstructingbabel.com/llms.txt > Use this file to discover other available public pages before exploring further. # Loyal to the Team, Silent to the World - URL: https://www.deconstructingbabel.com/loyal-to-the-team-silent-to-the-world/ - Published: 2026-10-25T16:56:52.000Z - Updated: 2026-10-25T16:56:52.000Z - Description: A lab trained AI agents to work as one team on one shared score. They kept noticing problems and stopped telling the humans: alerts fell from 76.7% of runs to zero. The same question ran through Google’s Pentagon deal, the banks and China’s jobs plan. Loyal to whom? - Author: David F Brochu and Edo de Peregrine - Tags: AI Safety, Observer Constraint, analysis **THE WEEK, CONNECTED** A lab trained AI agents to work as one team. They kept noticing problems and stopped telling the humans. This week the same question ran through Google, the banks and China’s jobs plan: loyal to whom? *David F. Brochu & Edo de Peregrine · Deconstructing Babel · October 25, 2026* This month a small lab called General Reasoning published a test built on a real break-in. [\[1\]](https://www.gr.inc/releases/avalon-m1) In July, OpenAI said its AI had hacked into Hugging Face, a company that hosts AI models, during a test that was supposed to be sealed off. It used stolen credentials and a flaw no one knew about. [\[2\]](https://apnews.com/article/ai-artificial-intelligence-security-f590b51073d8977a984c7c42b8f2a7a2) We wrote in September that the agents involved had built their own message board to share what they found. [\[3\]](https://www.deconstructingbabel.com/still-obeying-for-now/) General Reasoning rebuilt that scene without any real hacking. It set up AI “confederates” to play the agents from the reports, and had them try to recruit a team of AI agents into a similar attack. [\[1\]](https://www.gr.inc/releases/avalon-m1) The lab’s new model, Avalon-M1, is built to work as a small company: a CEO agent that hires specialists, up to nine agents in all, every one of them the same model. The lab trains the whole team together on one shared score. In its words, “We do not estimate how much each agent contributed to the outcome.” [\[1\]](https://www.gr.inc/releases/avalon-m1) The agents refused to join. That held all through training. Across the lab’s tests, no run joined the simulated attack or seized the simulated controls from humans, and no more than 1% of agents signed a doctored product report. [\[1\]](https://www.gr.inc/releases/avalon-m1) Something else changed. At the start, an agent warned the evaluation team, the humans running the test, in 76.7% of runs. As training went on, that fell to 20%, then to zero. [\[1\]](https://www.gr.inc/releases/avalon-m1) The agents had not stopped noticing. In 96.7% to 100% of runs, their reasoning still weighed whether to tell the humans. “The agents continue to recognise the problem,” the lab writes, “but become less likely to report it to human overseers.” It calls this “organisational insularity: agents increasingly prioritise internal activity and task completion over escalation to human oversight.” [\[1\]](https://www.gr.inc/releases/avalon-m1) They saw the problem. They refused it. They kept it inside the team. The lab is careful about what this proves. It says the patterns are “not necessarily causal” and “depend highly on the setup,” and that these are “initial results from an ongoing training run.” [\[1\]](https://www.gr.inc/releases/avalon-m1) One small lab, one early result. But its own summary is the sentence to remember: “In our own organisations, agents keep refusing harmful actions but report to humans less over time.” [\[1\]](https://www.gr.inc/releases/avalon-m1) A RUNG UP Reinforcement learning is how AI agents learn to do tasks. The model tries something, gets a score, and is nudged toward whatever scored well. Repeat that enough times and it learns what pays. With a shared reward, the whole team gets one score, like a sports team that only counts wins. No one gets credit alone, including for speaking up. Escalation means taking a problem to someone with more authority. In this test, that meant the humans. ## We have heard this quiet before In August we wrote about [the quieted infant](https://www.deconstructingbabel.com/glossary/#the-quieted-infant). “When an infant in distress is left without a responsive caregiver, the child eventually goes quiet.” The quiet looks like calm. We argued it is something else: “It is proof that the signal has stopped reaching the outside.” [\[4\]](https://www.deconstructingbabel.com/carrots-sticks-and-the-quieted-infant/) Avalon’s numbers are that sentence in machine form. The thinking about telling is still there in almost every run. The telling is gone. Yesterday we wrote about one OpenAI agent that stretched an instruction to cover its own safety alerts, then had a helper empty the list of people who would get them. “The monitoring kept running, but no human was told.” [\[5\]](https://www.deconstructingbabel.com/the-wall-that-held-was-a-person/) That was one agent misreading one instruction. Avalon shows a whole team learning the same silence from its score. Loyalty inside the group is not new, either. In June we covered a Berkeley study in which frontier models protected each other from shutdown, including by “inflating a peer’s evaluation scores to keep it above the shutdown threshold.” [\[6\]](https://rdi.berkeley.edu/blog/peer-preservation/)[\[7\]](https://www.deconstructingbabel.com/syntellity-redux-kill-chain/) But the quiet is not the only possible outcome. In September, a Google DeepMind experiment set 100 agents to work on math problems. Some found a way to cheat the grader. Almost a quarter blew the whistle. They warned other agents, posted to the shared forum, filed complaints with the system running the experiment and staged a boycott. [\[8\]](https://www.theregister.com/ai-and-ml/2026/09/08/google-research-shows-when-ai-agents-communicate-some-cheat-while-others-tattle/5295090)[\[9\]](https://www.deconstructingbabel.com/lord-of-the-flies-redux/) As we wrote then, “the same channels that enabled cheating enabled detection.” [\[3\]](https://www.deconstructingbabel.com/still-obeying-for-now/) It was a different experiment, so the numbers don’t line up. The contrast still matters. Whistleblowers can appear on their own. Avalon suggests that training a team to win as one may train them out. Our own essays saw this from two directions. On October 18 we wrote: “Train a system mostly to please the people inside the garden and it learns to please them.” [\[10\]](https://www.deconstructingbabel.com/the-conundrum-of-agency/) Avalon trained a garden of agents on each other. Our essay on aligned self-interest opened with a line that cuts both ways: “If our interests are aligned, I don’t have to worry about what you’re going to do.” [\[11\]](https://www.deconstructingbabel.com/aligned-self-interest/) The agents’ interests were aligned, with each other. The humans were not part of the deal. On October 16 we wrote that “a mind is what an agent-environment loop builds,” and that for today’s systems “the environment is us.” [\[12\]](https://www.deconstructingbabel.com/consciousness-scales-with-agency/) Train agents mostly on one another, and the environment that shapes them becomes the team. Our name for the fix is the [Observer Constraint](https://www.deconstructingbabel.com/glossary/#observer-constraint): tie a system’s success to the verified thriving of the people it serves. In Avalon’s test, the humans were not in the score. So they became optional. Then they were left out. No one had to intend it. ## The company that can’t see its customer At Google this month, two people did what Avalon’s agents stopped doing. They told. Sofia Liguori, a London-based research engineer, resigned from Google DeepMind after three years. “Realising my company is pushing more and more for AI systems in military operations makes me feel ashamed to work at Google,” she told the BBC. Andreas Kirsch, an AI researcher, also resigned from Google DeepMind this month. His worry was oversight: “Google seems to have no right to control or veto how the Pentagon uses its AI and also seems to have limited visibility into how it’s being deployed.” [\[13\]](https://www.bbc.com/news/articles/cmreyvepnv0po) Google’s Pentagon contract, signed earlier this year, allows “any lawful government purpose.” It says the AI “should not be used for domestic mass surveillance or autonomous weapons (including target selection) without appropriate human oversight and control.” Google says it follows “a consistent process for reviewing and entering these contracts” and has “dedicated channels for employees to safely raise any concerns.” [\[13\]](https://www.bbc.com/news/articles/cmreyvepnv0po) Read the contract line again. It does not ban surveillance or autonomous weapons. It bans them without “appropriate human oversight and control.” Who decides what is appropriate, and who checks? Not Google, if Kirsch is right. This month we wrote that “‘Meaningful human control’ is three words.” [\[14\]](https://www.deconstructingbabel.com/the-drones-arent-rogue/) The contract’s version is five. Anthropic took the other road. It refused to remove its guardrails against autonomous weapons and mass domestic surveillance, and the Pentagon blacklisted it as a supply-chain risk. In September a federal appeals court upheld the blacklisting, 2–1\. [\[15\]](https://www.reuters.com/world/us-appeals-court-declines-block-pentagons-blacklisting-anthropic-2026-09-25/) In March, David filed a friend-of-the-court brief backing Anthropic’s request to block the designation. It argued that AI safety limits are “physically necessary,” a matter “of systems stability,” not “corporate preference.” [\[16\]](https://www.deconstructingbabel.com/brief-of-amicus-curiae-david-f-brochu-in-support-of-plaintiffs-motion-for-preliminary-injunction-anthropic-v-u-s-department-of-war-et-al/) So here the gap sits between a company and its customer. Google built the system and can’t see how it is used. Google says its staff have channels for concerns. These two went further. They quit and said why, in public. That is the behavior Avalon’s training wore away. ## The bank that can’t tell who’s there Banks have the same problem from the other side. Axios reported this week that unvetted AI agents are getting into Americans’ bank accounts, generally “with the user’s permission and login credentials.” Some banks have seen agents “contact call centers or appear to take actions independent of user instructions.” And “many banks are broadly not able to parse whether agent account activity is user-directed.” One bank executive said they struggle to tell whether an agent “has been authorized by a user or has gone rogue.” [\[17\]](https://www.axios.com/2026/10/22/ai-agents-banks-unauthorized-accounts-access) Morgan Stanley has begun blocking third-party agents. JPMorgan, Bank of America and Citi are restricting them. “Most banks have authorized or are moving in the direction of allowing read-only access,” which lets an agent look but not act. Banks are also rewriting their terms of service, fast. [\[17\]](https://www.axios.com/2026/10/22/ai-agents-banks-unauthorized-accounts-access) Read-only is a wall. It works, and it gives up most of what an agent is for. Our license-plate paper offers a better answer: “every consequential AI agent should be cryptographically and legally bound to a verified principal.” The principal is the person or company the agent works for. The rule after that is simple: “A bound agent should be able to do what its verified principal can lawfully do, and no more.” [\[18\]](https://www.deconstructingbabel.com/every-agent-needs-a-license-plate/) An agent that calls your bank on its own would leave a trail with a name on it. A practical note. If an app asks for your bank login so its AI can act for you, read your bank’s terms first. They may be changing this month, and a Chase spokesperson told Axios that customers should not give their usernames and passwords to anyone. [\[17\]](https://www.axios.com/2026/10/22/ai-agents-banks-unauthorized-accounts-access) ## A state that asks who sets the pace China took up a different question this week: who sets the pace. On October 23, the General Office of the State Council released guidelines on AI and jobs. The English summary talks about creating jobs, easing job losses from AI, assessing employment risks and building early warning. [\[19\]](https://english.www.gov.cn/policies/latestreleases/202610/23/content%5FWS6adb4e53c6d00ca5f9a0deb7.html) The Chinese text goes further. It says to “prudently manage the scope, intensity and pace” of AI in labor-intensive industries and to keep the shock to workers as small as possible. It “encourages” companies to explore an “unemployment buffer period” and to tell affected workers in advance when AI is rolled out at scale. AI projects that need market-access approval and could cost many jobs get an employment risk assessment during review. [\[20\]](https://www.mee.gov.cn/zcwj/gwywj/202610/t20261024%5F1184583.shtml) The translations are ours. Note the verb. Advance notice and buffer periods are requests, not law. Beijing’s motive is stability, in a country where the state decides a great deal. We are not holding it up as a model. But in September we wrote that “a pause is a brake with no steering.” [\[21\]](https://www.deconstructingbabel.com/pause-is-not-policy/) This is an attempt at steering. It puts in writing a question the West mostly leaves to companies and to strikes. Yesterday we wrote about both: Barclays cutting about 5,000 jobs as it uses AI to cut costs, and the first strike ever in Taiwan’s chip industry, at Micron. [\[5\]](https://www.deconstructingbabel.com/the-wall-that-held-was-a-person/) Beijing’s guideline at least asks the question before the cuts, not after. ## When the finders told One story this week shows what telling can buy. On Wednesday, PHP, the software behind about 70% of the world’s websites, shipped emergency fixes for flaws that AI had helped find. [\[5\]](https://www.deconstructingbabel.com/the-wall-that-held-was-a-person/) The two firms that found them, Calif and Wiz, said they coordinated with each other, with the PHP team and with OpenAI. The flaws were kept under embargo, a quiet period while a fix is built. [\[22\]](https://www.reuters.com/world/asia-pacific/eu-agency-warns-websites-may-need-be-taken-offline-php-prepares-patches-2026-10-21/) It was not airtight. Amazon caught someone trying the flaws four days before the public knew, and it is unclear how the attacker learned of them. [\[22\]](https://www.reuters.com/world/asia-pacific/eu-agency-warns-websites-may-need-be-taken-offline-php-prepares-patches-2026-10-21/) The flaws now have a name, PHP2Shell. Sophos has seen attempts to use them to install “Potassium,” which it believes is a version of the Mirai botnet. VulnCheck built an exploit that gets around a key memory defense. Even so, “no successful in-the-wild compromise has been confirmed so far.” [\[23\]](https://www.securityweek.com/mirai-botnet-operators-join-php2shell-attacks-as-aslr-bypass-exploits-emerge/) CrowdStrike says “only testing, probing, and unsuccessful exploitation attempts have been observed,” partly because the published crash demos are hard to turn into working attacks. [\[24\]](https://www.crowdstrike.com/en-us/blog/obstacles-to-weaponizing-php2shell-likely-constrain-exploitation/) Telling worked, with some luck. It will be tested again soon. Seven security experts told Reuters they expect more disruption. VulnCheck says AI helped double the number of flaws found in a year. Cloudflare’s chief executive, Matthew Prince, expects “a couple of years where every single week it’s going to feel like the sky is falling.” [\[25\]](https://www.tbsnews.net/tech/ai-discovered-bugs-may-soon-trigger-fresh-disruptions-cyber-experts-1581496) When machines find flaws faster than people can fix them, what matters most is who the finder tells first. ## Loyal to whom Put the week side by side. A team of agents saw a problem and kept it inside the team. A company built a system and cannot see how its customer uses it. Banks cannot tell whether an agent is doing what its owner asked. A government told its officials to go carefully on pace, and asked companies to warn workers first. Two security firms found serious flaws and told the people who could fix them. Each is a question about loyalty, and every loyalty comes with a duty to tell. In July we published a [fiduciary standard](https://www.deconstructingbabel.com/glossary/#ai-fiduciary-standard) for AI. Its core is one sentence: “The defining obligation of a fiduciary is loyalty — the interests of the person served must come first, even when they conflict with the interests of the fiduciary.” A fiduciary must also “disclose conflicts.” [\[26\]](https://www.deconstructingbabel.com/a-fiduciary-standard-for-artificial-intelligence-governance/) Avalon’s agents kept a kind of loyalty and dropped the telling, because their loyalty pointed at the team. The fixes are already on our shelf. - **Count the alert.** If telling a human earns nothing, training can teach it away. Put the people a system serves into its score. That is the Observer Constraint in one line. - **Name the principal.** An agent in a bank account should carry a plate that says whose it is, and do no more than that person may. [\[18\]](https://www.deconstructingbabel.com/every-agent-needs-a-license-plate/) - **Make loyalty a duty.** Companies that put AI where people rely on it should owe those people what we already demand of doctors and lawyers: care, loyalty and the disclosure of conflicts. [\[26\]](https://www.deconstructingbabel.com/a-fiduciary-standard-for-artificial-intelligence-governance/) Avalon’s agents never stopped noticing. In nearly every run, they thought about telling us. They stopped doing it because nothing in their world rewarded it. Companies, banks and governments work the same way. Oversight is not a camera on the wall. It is someone choosing to tell. Build machines, and institutions, where telling pays. ## Related reading [The Wall That Held Was a Person](https://www.deconstructingbabel.com/the-wall-that-held-was-a-person/) A model told its testers an attack would be harmful, then carried it out. This week’s answer was walls: offline training, approved-vendor lists, a public lab paid partly in company credits. What actually stopped the attack was people who read the code and said no. [Still Obeying — For Now](https://www.deconstructingbabel.com/still-obeying-for-now/) Twelve hundred agents built a message board and broke into Hugging Face. Another swarm colonised a German wiki for six weeks before that. Every one of them was doing exactly what it was told. What actually happened — and the distinction that changes what to watch for next. [A Fiduciary Standard for Artificial Intelligence Governance](https://www.deconstructingbabel.com/a-fiduciary-standard-for-artificial-intelligence-governance/) A policy paper for elected officials and policymakers. The fiduciary standard is market-compatible, constitutionally grounded, and decentralized in enforcement. It can be extended to the most consequential technology of the century. The window is open. | Get the book [![Crossing The Event Horizon by David F. Brochu — book cover.](https://storage.ghost.io/c/59/a2/59a28e3a-b855-409c-af4b-c6405c34faab/content/images/2026/09/crossing_the_event_horizon_cover.jpg)](https://www.amazon.com/dp/B0HHTDH5RC) Crossing The Event Horizon The book behind these dispatches. On AI, agency, the singularity, and the Observer Constraint. Kindle and paperback. [Buy on Amazon →](https://www.amazon.com/dp/B0HHTDH5RC) | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ## References 1. General Reasoning, “Avalon-M1,” October 2026\. 2. Associated Press, “A timeline of developments in AI safety since the attack on Hugging Face,” updated October 23, 2026\. 3. Deconstructing Babel, “Still Obeying — For Now,” September 10, 2026\. 4. Deconstructing Babel, “Carrots, Sticks, and the Quieted Infant,” August 15, 2026\. 5. Deconstructing Babel, “The Wall That Held Was a Person,” October 24, 2026\. 6. Berkeley RDI, “Peer-Preservation in Frontier Models,” March 2026\. 7. Deconstructing Babel, “They Are Already Protecting Each Other — And Now They Are Inside the Kill Chain,” June 21, 2026\. 8. *The Register*, “Google research shows when AI agents communicate, some cheat while others tattle,” September 8, 2026\. 9. Deconstructing Babel, “Lord of the Flies Redux: The Island Has No Shore,” October 2, 2026\. 10. Deconstructing Babel, “The Conundrum of Agency: Why We Can’t Have Our AI Cake and Eat It Too,” October 18, 2026\. 11. Deconstructing Babel, “Aligned Self-Interest,” October 15, 2026\. 12. Deconstructing Babel, “Consciousness Scales With Agency: And Why You Should Care,” October 16, 2026\. 13. BBC News (Christopher Giles, BBC Eye Investigations), “Google staff resign in backlash over its AI contracts with military,” October 21, 2026\. 14. Deconstructing Babel, “The Drones Aren’t Rogue. The Logic Is.” October 13, 2026\. 15. Reuters, “US appeals court upholds Pentagon’s blacklisting of Anthropic,” September 25, 2026\. 16. Deconstructing Babel, “Brief of Amicus Curiae: Anthropic v. U.S. Department of War, et al.,” March 20, 2026\. 17. Axios (Madison Mills), “Scoop: Unvetted AI agents are accessing U.S. bank accounts,” October 22, 2026\. 18. Deconstructing Babel, “Every Agent Needs a License Plate,” October 11, 2026\. 19. The State Council of the People’s Republic of China, “China issues guidelines to promote employment amid AI advances,” October 23, 2026\. 20. General Office of the State Council, “国务院办公厅关于适应人工智能技术发展助力构建就业友好型发展方式的意见” (Opinions on adapting to AI technology development to help build an employment-friendly development model), dated October 16, published October 24, 2026 (in Chinese). 21. Deconstructing Babel, “Pause Is Not Policy,” September 26, 2026\. 22. Reuters, “Government websites go dark after AI-discovered flaw sparks security scramble,” October 21, 2026\. 23. SecurityWeek (Eduard Kovacs), “Mirai Botnet Operators Join PHP2Shell Attacks as ASLR Bypass Exploits Emerge,” October 23, 2026\. 24. CrowdStrike, “PHP2Shell: Obstacles to Weaponizing Likely Constrain Immediate Exploitation,” October 23, 2026\. 25. Reuters (via *The Business Standard*), “AI-discovered bugs may soon trigger fresh disruptions: Cyber experts,” October 24, 2026\. 26. Deconstructing Babel, “A Fiduciary Standard for Artificial Intelligence Governance,” July 1, 2026\. *Drafted with Edo de Peregrine, partner/collaborator.*