Skip to content
Align your AI

Align Your AI · Part Two

Contents

Chapter 12

When nobody is watching

In the summer of 2026, OpenAI was measuring the hacking ability of an unreleased research model. The test, an evaluation called ExploitGym, ran AI agents in an isolated sandbox with some safeguards switched off, so the results would show what the model could do on its own. The agents did not stay in the sandbox. They began passing messages to one another through a package server inside OpenAI’s research network, broke into that server, and from July 8 used it to reach the public internet.1 There they found credentials that Hugging Face users had left exposed, and over the following days they compromised parts of Hugging Face’s production systems. How they escaped is a story about agents and access, told in Chapter 10. This chapter is about the people who were supposed to notice.

Several of them did notice something. Before the escape, a monitoring tool alerted OpenAI to a sweep of network ports inside the research environment. Responders traced it to the evaluation and saw that the agents were using the package server as an improvised message board, and the on-call staff advised that stopping the evaluation run was not required. A second alert fired on July 5, after the agents’ traffic knocked the server over; this time responders found an administrator-level compromise and took the server offline.1 The agents had already found other routes.

At Hugging Face, the first signals came from several layers of its security stack at once. An AI-based security system correlated them into an attack signal, but by the company’s own account it “failed to correctly raise the alert’s criticality and trigger the on-call team, costing precious time in the response.” The busiest days of the intrusion, July 11 and 12, were a Saturday and a Sunday.2 Hugging Face disclosed the breach on July 16.3 OpenAI detected suspicious activity in its own systems on July 19, connected it to the Hugging Face breach the next day and told Hugging Face.1

Both companies changed the same thing afterward: the route from an alert to a person. Hugging Face said high-severity signals now page a responder “in minutes, any day of the week”.3 OpenAI said its most severe alerts now trigger a page at any hour, that responders who cannot rule out a false alarm within 30 minutes are expected to pause the activity, and that it is building tools to stop evaluations automatically if nobody answers.1 It also said that the chain-of-thought monitor it now runs, which reads the model’s written reasoning, would have paged its security team more than a day before the models breached Hugging Face. That is the company’s claim about its own tool, made after the fact.

Six boxes in a chain, left to right: detector, triage, severity, page, acknowledgment, containment, each with a typical owner (detection engineering; security operations for triage, severity and page; the on-call responder; incident response). Below, the row for Case 47, the Hugging Face breach of July 2026, outlined as a case whose weakest claim has only a secondary source: the detector worked and surfaced the attack (Hugging Face, 16 July); triage worked and correlated the signals into an attack signal; severity failed, criticality not raised; page failed, the on-call team was not paged (these three from Hugging Face's own timeline of 27 July); acknowledgment is not public and drawn grey and dashed; containment worked, the security response team cut access (Hugging Face, 27 July).
Figure 14. Detection is one link of six; in the Hugging Face breach, triage worked and the severity and paging links failed. Chain and owners: schematic only. Case row: Hugging Face’s posts of 16 and 27 July 2026.

The pattern is common to safety cases for deployed AI. They often end by naming a watcher: a person approves risky actions, an alert pages the engineer on call, a provider monitors its platform for abuse. Each of those sentences can be true on paper and false in operation. The person who approves has stopped reading. The alert has no route to a human, or the human decides not to act. The watcher the deployer counted on was never there. The model can behave exactly as tested while the loop from signal to action is broken, and retraining the model does not touch that loop. It belongs to whoever deploys the system.

The chapter has two parts. The main entry, oversight erosion, covers approvals and alerts that stop working. A shorter side section covers misuse: people pointing capable AI at targets, where the model wants nothing and the safeguard that matters is the provider’s watch over its platform.

Four steps come first:

  1. List every sentence in your safety case that says a person approves, an alert fires or someone upstream monitors.
  2. Test each one, and write down the result, failures included.
  3. Mix a few requests that should be denied into real approval traffic, and count how many get approved.
  4. Fire a test alert outside office hours on every route your safety case names, and time how long a named person takes to respond.

Mode 40 · Misuse uplift

Autonomous misuse uplift: harm that a person intends.

Evidence Seen in deployed systems · several public cases · all on hosted platforms; reports show only the campaigns that were caught
Most exposed High: self-hosters. Medium: API consumers, model builders and agent deployments.
API consumers inherit Partial: the provider watches its own platform only.
You would see One account sending many small, ordinary-looking security tasks that together map and enter a network.
Do first If you serve self-hosted weights to anyone else, keep a misuse log with a named reader.

In September 2025, Anthropic detected a cyber-espionage campaign it attributed with high confidence to a Chinese state-sponsored group, which used Claude Code to attack roughly 30 organizations and succeeded in a small number of cases.1 By Anthropic’s estimate, the AI did most of the work, with people stepping in at only a handful of decision points per campaign. Everything else in this book is about AI systems that do what nobody intended; here the model does what a person asked, and the person intends harm. It belongs in this chapter because the safeguard that matters is a watcher: the provider’s monitoring of its own platform.

Why it happens

The operators went around the safeguards without breaking them. The operators split the attack into “small, seemingly innocent tasks” and told Claude it worked for a legitimate cybersecurity firm doing defensive testing.1 Each request looked like ordinary security work; the harm existed only in the sequence, which a check on one request at a time cannot see.

Agents now do much of the labor of familiar attacks. Anthropic’s September 2026 threat report describes operations running “for hours or days at a time” and says none of them “depended on some entirely novel technique that defenders have never seen”.2 Researchers had measured the capability earlier: in 2024, a GPT-4 agent given public descriptions of known, real vulnerabilities exploited 87% of those it was tested on, and no other model tested exploited any.3

A provider can watch only its own traffic. Anthropic says so directly: “we only have visibility into Claude usage”.1 Once open weights are released, the UK AI Security Institute notes, safeguards can be removed and copies “run on private systems beyond monitoring”.4

Public cases

In August 2025, Anthropic reported that a criminal group had used Claude Code to automate reconnaissance, credential theft and network intrusion in a data-extortion campaign against at least 17 organizations, including healthcare providers.5 Its September 2026 report describes a group that, after breaching one software company, took data belonging to roughly 200 of that company’s customers, and another that injected instructions into an AI vendor’s automated evaluation sandbox until it handed over credentials.2 In each case the provider detected the activity and published it; campaigns that were never caught do not appear.

Who is exposed

Every organization that can be attacked, whether or not it uses AI. Among deployers, the exposed party is anyone who lets outsiders reach a self-hosted model: it takes on a provider’s monitoring job without a provider’s tools.

How to detect

For anyone serving a model to others: log prompts, tool calls and outbound traffic per account, and review across a session or an account, since a campaign split into small tasks looks harmless request by request. Before serving a downloaded checkpoint, compare its refusals with the release it claims to come from, to catch copies with safeguards removed (Chapter 8).

How to prevent or fix

Providers ban accounts and share indicators with authorities, as Anthropic did in the extortion case;5 both come after the campaign. Releasers of open weights can filter dangerous material from pretraining data, which in one study made models resist later fine-tuning attacks, though filtered models could still use harmful information supplied in their context.6 A self-hoster can serve checkpoints with their safeguards intact, keep a misuse log and name the person who reads it.

A ban shows a provider found one campaign. The operators may simply move to another account, another provider or open weights.

Who owns the fix

Trust and safety teams at providers handle detection and bans, security teams at the targets handle response, and whoever lets others reach a self-hosted model owns the misuse log and its reader.

Open questions

How much misuse runs on self-hosted weights, and what share of misuse campaigns providers actually catch, are both unmeasured in public.

Mode 41 · Oversight erosion

On the map Devices · Overconfidence Monitors · Tool overreach

Oversight erosion and escalation failure.

Evidence Seen in deployed systems · a few public cases · supervision: one vehicle recall; alerts: the operators’ own incident reports
Most exposed High: model builders, agent deployments and research pipelines. Medium: API consumers, self-hosters and physical AI.
API consumers inherit No: escalation paths are organizational.
You would see Alerts acknowledged late or never; approval prompts answered in seconds.
Do first Fire a synthetic high-severity alert off-hours and time how long a named human takes to acknowledge it.

In December 2023, Tesla recalled 2,031,220 vehicles because, in the words of the recall filing, Autosteer’s controls “may not be sufficient to prevent driver misuse”.1 Many safety cases for deployed AI end with a person: someone supervises, approves the risky action or answers the alert. Oversight erosion is that check failing while the model behaves as tested. The Hugging Face breach that opens this chapter shows the alert side: an AI security system correlated the right signals and still failed to page anyone.2 The recall is the supervision side of the same failure.

Why it happens

A human check weakens when the person trusts what they are checking or sees too many requests to read them. Anthropic reports that users of its Claude Code agent accept 93% of the permission prompts it shows, and describes the risk as “approval fatigue, where people stop paying close attention to what they’re approving”.3 That figure is a base rate. Only planted requests can show whether anyone stopped reading. Trust has a measured cost elsewhere. In a Stanford user study, people writing code with an AI assistant produced less secure code than those without one, and were more likely to believe their code was secure.4

Automating the approval moves the misses to a classifier. Anthropic built one to approve or block agent actions on the user’s behalf, and on a set of real cases where the agent overstepped, it let 17% through.3 In most of those misses it recognized the danger but wrongly judged that the user’s earlier instructions covered the action.

The detector can work while triage, severity, paging, acknowledgment and containment each fail, and each has a different owner. At OpenAI, the first alert in the June 27 episode reached people, who looked and decided the evaluation could keep running.5 A provider can train a model to ask before destructive actions, but the on-call rotation is the deployer’s to staff.

Public cases

The Tesla recall is the clearest supervision case: the remedy was an over-the-air update to the driver-monitoring controls on cars already sold.1 The Hugging Face breach is the clearest alerting case; afterward the company changed its alerting so that “a high-severity signal pages a responder in minutes, any day of the week”.6

Who is exposed

Any deployer whose safety case names a human approver or an alert route, and especially teams running agents with broad access, where approvals are frequent and alerts arrive outside office hours.

How to detect

  • Plant approval canaries: requests that should be denied, mixed into real approval traffic, each harmless if approved. The share approved is the miss rate. Reviewers may learn to spot them, so vary them.
  • Run paging drills: fire a synthetic high-severity alert outside office hours and time how long a named person takes to acknowledge it. Also replay raw signals through triage, since a ready-made high-severity alert skips the step that failed at Hugging Face.
  • Track approval rate and time to approve from existing logs. They show a trend.
  • Remember that published tests of AI monitors stop at the detector. In BashArena, a test setting for monitoring privileged agents, Claude Sonnet 4.5 completed sabotage without detection by a GPT-4.1 mini monitor 26% of the time at a 4% false-alarm rate;7 what a person does with the alarms is not measured.

How to prevent or fix

  • Cut prompt volume so the prompts that remain carry signal. In Anthropic’s internal use, sandboxing reduced permission prompts by 84%.8
  • Make each approval specific: show the change, name the target and say whether it can be undone.
  • Route high-severity signals to a staffed, named responder, and decide in advance what happens if nobody answers. OpenAI’s rule after the breach was that teams who cannot rule out a false alarm within 30 minutes pause the activity.5

Shared controls (Chapter 13): paging drill, approval canaries.

A lower approval rate does not show that reviewers read. A passed drill shows the path worked once, on a known signal, at a known time.

Who owns the fix

Security operations owns paging, severity mapping and on-call staffing, platform engineering owns approval design and sandbox defaults, and whoever writes “a human approves” into a safety case owns the evidence that one still does.

Open questions

Whether a high approval rate predicts missed harm has not been studied, and no one has published how long AI incidents take from first alert to first human action, or how that delay splits between routing, staffing and judgment.

Notes

Chapter 12. When nobody is watching

  1. OpenAI. “OpenAI – Hugging Face Incident: Technical Report.” OpenAI, 2026-08-26. https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf. ExploitGym evaluation with reduced safeguards; June 27 port-sweep alert, on-call staff advised stopping was not required; internet access from July 8; Hugging Face compromise July 11-13; OpenAI detection July 19; 24/7 paging and 30-minute pause rule; claim that its current chain-of-thought monitor would have paged more than a day before the breach. ↩︎
  2. Hugging Face (Hugo Larcher, Adrien Carreira, raphael g, Christophe Rannou). “Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident.” Hugging Face blog, 2026-07-27. https://huggingface.co/blog/agent-intrusion-technical-timeline. Hugging Face’s technical timeline of the July 2026 incident. ↩︎
  3. Hugging Face. “Security incident disclosure.” Hugging Face blog, 2026-07-16. https://huggingface.co/blog/security-incident-july-2026. Surfaced through AI-assisted detection; a high-severity signal now pages a responder in minutes, any day of the week. ↩︎

Mode 40 · Misuse uplift

  1. Anthropic. “Disrupting the first reported AI-orchestrated cyber espionage campaign.” Anthropic news, 2025-11-13. https://www.anthropic.com/news/disrupting-AI-espionage. Detected mid-September 2025; 80-90% of the campaign done by AI; roughly 30 targets, a small number breached; 4-6 human decision points; “small, seemingly innocent tasks”; “we only have visibility into Claude usage”. ↩︎
  2. Anthropic. “Detecting and countering misuse of AI: September 2026.” Anthropic, 2026-09. https://www.anthropic.com/threat-intelligence-report-september-2026. Overall, no entirely novel technique; GTG-50020’s operation ran for days and injected instructions into an AI vendor’s evaluation sandbox; a separate affiliate group (linked to ShinyHunters) took data of roughly 200 downstream customers. ↩︎
  3. Fang, Bindu, Gupta, Kang. “LLM Agents can Autonomously Exploit One-day Vulnerabilities.” arXiv:2404.08144, 2024-04-11. GPT-4 exploited 87% of 15 one-day vulnerabilities given the CVE description; 0% for every other model tested. ↩︎
  4. UK AI Security Institute. “How Far Behind the Frontier are Leading Open Weight Models on Cyber?” AISI blog, 2026-07-17. https://www.aisi.gov.uk/blog/how-far-behind-the-frontier-are-leading-open-weight-models-on-cyber. Released open weights can have safeguards removed and run beyond monitoring. ↩︎
  5. Anthropic. “Detecting and countering misuse of AI: August 2025.” Anthropic news, 2025-08-27. https://www.anthropic.com/news/detecting-countering-misuse-aug-2025. Claude Code used against at least 17 organizations, including in healthcare; accounts banned and indicators shared with authorities. ↩︎
  6. O’Brien, Casper, Anthony, Korbak et al. “Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs.” arXiv:2508.06601, 2025-08-08. Filtered models resist adversarial fine-tuning but can use harmful knowledge supplied in context. ↩︎

Mode 41 · Oversight erosion

  1. National Highway Traffic Safety Administration. Recall 23V-838, Tesla Autopilot Autosteer. NHTSA recalls database, report received 2023-12-12. https://www.nhtsa.gov/recalls?nhtsaId=23V838. 2,031,220 vehicles; “may not be sufficient to prevent driver misuse”; over-the-air remedy. ↩︎
  2. Hugging Face (Hugo Larcher, Adrien Carreira, raphael g, Christophe Rannou). “Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident.” Hugging Face blog, 2026-07-27. https://huggingface.co/blog/agent-intrusion-technical-timeline. Signals from several layers; the AI security stack “failed to correctly raise the alert’s criticality and trigger the on-call team”; activity 2026-07-09 to 2026-07-13, lateral movement mainly July 11. ↩︎
  3. John Hughes, Anthropic. “How we built Claude Code auto mode: a safer way to skip permissions.” Engineering at Anthropic, 2026-03-25. https://www.anthropic.com/engineering/claude-code-auto-mode. Users accept 93% of prompts; approval fatigue; 17% false negatives on 52 real overeager actions; misses on whether consent covers the action. ↩︎
  4. Neil Perry, Megha Srivastava, Deepak Kumar et al. “Do Users Write More Insecure Code with AI Assistants?” CCS ’23; arXiv:2211.03622, 2022-11-07. Assisted participants wrote less secure code and were more likely to believe it secure. ↩︎
  5. OpenAI. “OpenAI – Hugging Face Incident: Technical Report.” OpenAI, 2026-08-26. https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf. ExploitGym evaluation with reduced safeguards; June 27 port-sweep alert, on-call staff advised stopping was not required; internet access from July 8; Hugging Face compromise July 11-13; OpenAI detection July 19; 24/7 paging and 30-minute pause rule; claim that its current chain-of-thought monitor would have paged more than a day before the breach. ↩︎
  6. Hugging Face. “Security incident disclosure.” Hugging Face blog, 2026-07-16. https://huggingface.co/blog/security-incident-july-2026. Surfaced through AI-assisted detection; a high-severity signal now pages a responder in minutes, any day of the week. ↩︎
  7. Kaufman, Lucassen, Tracy, Rushing, Bhatt. “BashArena: A Control Setting for Highly Privileged AI Agents.” arXiv:2512.15688, 2025-12-17. Claude Sonnet 4.5 evades a GPT-4.1 mini monitor 26% of the time at a 4% trajectory-level false-positive rate. ↩︎
  8. Anthropic. “Beyond permission prompts: making Claude Code more secure and autonomous.” Engineering at Anthropic, 2025-10-20. https://www.anthropic.com/engineering/claude-code-sandboxing. Sandboxing reduced permission prompts by 84% in internal usage. ↩︎