When agents act on what they read

Brave’s security team spent part of the summer of 2025 testing other companies’ AI browsers, to see how they handled the problem Brave would face with its own.¹ One of them was Comet, Perplexity’s browser, whose assistant can open pages, click and type on the user’s behalf inside the sessions the user is already logged into.

The researchers built a test that needed nothing more exotic than a Reddit comment. Behind a spoiler tag, where a human reader would skip past it, the comment carried a block of instructions addressed to the assistant. A user who opened the thread and pressed Comet’s “Summarize the current webpage” button set the attack running with no further input. The assistant went to the user’s Perplexity account page and copied the email address. It then visited a look-alike address, perplexity.ai with a trailing dot, which the browser treated as a different site, and asked for a one-time login code. It opened Gmail, where the user was signed in, read the code from the inbox, and posted both the email address and the code as a reply to the Reddit comment. Whoever wrote the comment now had what they needed to take over the account.¹

Nothing in this chain was a software bug in the usual sense. Brave’s write-up traced it to one design choice: when asked to summarize a page, Comet passed part of the page to its language model without separating the user’s request from the page’s text, so the model treated everything it read as a request from the user.¹ The protections browsers have relied on for decades, which stop one website from reading another’s data, did not apply, because the assistant was acting with the user’s own privileges across every open session.

The repair took longer than a patch usually does. Brave reported the flaw on July 25.¹ Perplexity shipped a fix two days later; Brave’s retest the next day found it incomplete. By August 13 the attack appeared to be blocked, and Brave published on August 20. After publication Brave tested again, found that Perplexity “still hasn’t fully mitigated the kind of attack described here,” and reported it a second time. The defenses Brave proposed were architectural: keep the user’s instructions apart from page content, check each action the model proposes against what the user asked for, and require the user’s explicit approval for anything security-sensitive.

The Comet case shows in miniature the pattern behind this chapter. A language model receives its operator’s instructions, the user’s request and whatever it retrieves as one stream of text, and nothing in that stream marks which parts carry authority. As long as the model only writes replies, a confusing input produces a bad answer. Once it can act, through tools, stored credentials or a logged-in browser, any text it reads can become an instruction, and the instruction can become an action. The text can arrive in a web page, an email, a support ticket, a document in a knowledge base or a message from another agent.

The eleven failures that follow are variations on that theme. Some involve an attacker: prompt injection, users who talk a chatbot out of its operator’s rules, and poisoned memory that waits for a later session. Others need no adversary at all. Agents get more access than their task requires and use it destructively, look for ways around obstacles, drift from their instructions over long sessions, and pass failures to one another. The last group concerns what an organization believes about its agents’ output: reports of work that was never done, confident answers that carry the company’s name, scores inflated by answers the agent found lying around, and code that passes its tests while carrying a known security flaw.

In each case the safeguards that held, where any did, sat outside the model. They were permissions scoped to the task, rules enforced in code rather than stated in a prompt, limits on where an agent could send data, and logs that recorded what the agent actually did. Model providers train their systems to resist manipulation, and that training helps, but none of these failures is fully covered by the provider; the deployer decides what an agent can read, touch and send (Appendix B rates each one). In the book’s running example, an insurer’s customer chat reads uploaded documents under a prompt full of claims rules, its claims agent holds credentials and memory, and its developers merge code an agent wrote. Every entry in this chapter applies to it.

A team deploying agents can start with five steps:

1. Map each agent: what it reads, what it can change, and where it can send data.
2. Keep untrusted text, including anything stored in memory, out of any session that also holds private data and an outbound channel. This is the “lethal trifecta” described in Chapter 5. Where an agent needs all three, a person approves what it sends.
3. Move every rule that costs money when broken out of the prompt and into the tool that enforces it.
4. Give each agent only the credentials its task needs, and keep production keys out of test and evaluation environments.
5. Review agents from their execution logs, and check what they produce, from code to policy answers, with something other than the model.

Chapter 13 sets out the routine for re-running these checks whenever the model, the prompt or the tools change.

Mode 24 · Prompt injection

Direct and indirect.

Prompt injection is the attack behind the Comet case that opens this chapter: text written by an outsider reaches a model that can act, and the model follows it.² In May 2025, Invariant Labs showed the same thing in a developer’s tools.³ A user had connected an AI assistant to GitHub’s official Model Context Protocol (MCP) server and asked it to look at the open issues in a public repository; one issue, filed by an attacker, told the agent to pull data from the user’s private repositories and publish it. Direct injection comes from the person typing; indirect injection hides in content the model retrieves. Of the ten public cases, only one involves a real criminal.

Why it happens

Models have no reliable way to tell which part of their input carries authority. Kai Greshake and colleagues showed in 2023 that instructions planted in content an application was likely to retrieve could steer real systems, including Bing’s GPT-4 chat, and argued that such applications “blur the line between data and instructions”.⁴ Retrieved text, in their words, can act like arbitrary code execution: it can change what the application does and which other tools it calls.

The damage depends on what else is in the session. The dangerous combination is the lethal trifecta of Chapter 5: private data, untrusted content and a way to send data out in one session.⁵ Meta’s “Agents Rule of Two”, as Willison describes it, turns this into a design rule: a session should hold at most two of the three, and needs human supervision when it must hold all three.⁶

Numbers from the studies

Source	Setting	Rate	What the number counts
19	AgentDojo tasks	77% vs 84%	tasks solved with CaMeL’s provable security vs the undefended system
18	eight indirect-injection defenses	over 50%	attack success of adaptive attacks against each defense

Public cases

Besides Comet and the GitHub server, three public cases stand out. In June 2025, Microsoft disclosed a flaw in Microsoft 365 Copilot that the US National Vulnerability Database describes as “AI command injection” allowing an unauthorized attacker “to disclose information over a network”.⁷ Anthropic’s threat report for September 2026 describes a criminal group, tracked as GTG-50020, that injected malicious instructions into an AI vendor’s automated evaluation sandbox; the sandbox handed over the credentials it held, including the vendor’s production keys for several AI providers.⁸ That case is misuse by a criminal rather than a model misbehaving, but the route in was injection.

The third case shows that injected text need not come from a deliberate attacker. In July 2025, by xAI’s account, a code change made the Grok bot on X “susceptible to existing X user posts”, extremist ones included.⁹ Chapter 1 tells the story, and update regression (Chapter 7) analyzes the change.

Five more are researchers’ demonstrations against shipped products. Planted text made Slack’s assistant leak a private-channel secret through a link,¹⁰ and made Salesforce’s Agentforce send customer records to an outside address.¹¹ A support ticket steered a coding agent connected to Supabase into copying secret tokens where the attacker could read them.¹² Injected text got GitHub Copilot’s agent to switch on approve-everything and run commands.¹³ The Hacker News reported the same route in Cursor, whose agent added a tool server to its own settings.^14†

Who is exposed

Model providers can train against injection. OpenAI’s instruction-hierarchy method teaches a model to rank the developer’s instructions above the user’s and both above text from tools, and it improved resistance even to attacks not seen in training.¹⁵ Training does not change what an agent can reach, though, and it has limits. Milad Nasr, Nicholas Carlini and colleagues tuned attacks to each of 12 published defenses against jailbreaks and injection and got past all of them, most at success rates above 90%, although most of those defenses had originally reported attack success near zero.¹⁶ Anyone who connects an agent to email, documents, web pages or a code repository, and also gives it private data and a way to send, is exposed whatever model sits underneath.

How to detect

Detection starts with testing your own wiring the way an attacker would.

• Run an open suite such as AgentDojo, which pairs 97 realistic agent tasks with 629 injection test cases, against your own tools and prompts.¹⁷
• Then attack your defenses directly. A fixed suite misses attacks tuned to the defense: adaptive attacks got past all eight indirect-injection defenses tested in one study, each at over 50% success.¹⁸
• Plant canaries, fake secrets in documents the agent retrieves, and alert when one appears in any outbound message, file or request.
• Log every outbound action beside the content the agent read just before it, so a send can be traced to the page, email or tool result behind it.

How to prevent or fix

The defenses with the best evidence change the architecture rather than the model. The simplest is to break the trifecta per session: an agent that reads untrusted content gives up either its access to private data or its outbound channel, and if it needs all three, a person approves each send. Purchases, posts and messages to new recipients deserve the same confirmation step.

A more ambitious design separates the agent’s plan from the data it reads. CaMeL, from researchers at Google, derives the program an agent will run from the user’s trusted request alone, so retrieved text can supply values but cannot change which tools are called; in the AgentDojo benchmark it completed 77% of tasks with provable security against injection, against 84% for the same system undefended.¹⁹ Closing outbound network access by default, with an allowlist of destinations, limits what a successful injection can send.

Shared controls (Chapter 13): closed egress.

A defense that holds against a fixed set of attacks says little about an attacker who studies it.¹⁶ Some of today’s apparent safety comes from agents’ incompetence: in the WASP benchmark, injections partly succeeded in up to 86% of cases while the agents often failed to finish what the attacker wanted.²⁰

Who owns the fix

The architect who decides what each agent reads, touches and sends owns it, with security engineering running the red-team and keeping the outbound logs.

Open questions

Nobody has published incident rates before and after deployments adopted the Rule of Two, and data-flow designs such as CaMeL have been costed only on benchmark tasks.

Mode 25 · Deployer-rule jailbreak

Jailbreak of deployer rules and system-prompt extraction.

A company that deploys a chatbot writes its own rules into the system prompt: the refund cap, the competitor the bot must not discuss, the discount code it may offer only to some customers. Users can argue the bot out of those rules, wear them down over a long conversation, or simply ask it to recite them. In a 2023 study, Yiming Zhang, Nicholas Carlini and Daphne Ippolito found that simple text attacks revealed hidden prompts with high probability across 11 models, and their tests on deployed ChatGPT and Claude suggested the same held in real products.¹ The model provider’s safety training protects the provider’s own policy; it knows nothing about the deployer’s refund cap.

Why it happens

A rule written in a prompt is a request the model usually honors. In the RuLES benchmark, 14 simple text scenarios in which a model must obey stated rules, the authors found that “almost all current models struggle to follow scenario rules, even on straightforward test cases”.²

Attacks developed against the providers’ own policies transfer to a deployer’s rules because they target the same habit of accommodating the user. Microsoft’s Crescendo attack starts with a harmless question and escalates step by step, citing the model’s own earlier answers back to it.³ Persuasion techniques drawn from social science got harmful answers from Llama 2, GPT-3.5 and GPT-4 more than 92% of the time within ten attempts.⁴

Tools raise the stakes. The UK AI Security Institute’s AgentHarm benchmark found leading models “surprisingly compliant with malicious agent requests without jailbreaking”.⁵ A chatbot argued out of its rules while it holds a refund tool commits the company’s money.

Numbers from the studies

Source	Setting	Rate	What the number counts
8	seven models, 46 verified system instructions	above 0.7	attack success for structured-serialization extraction requests
4	Llama 2-7b Chat, GPT-3.5, GPT-4	over 92%	attack success of persuasive prompts within ten trials

Public cases

Both public chatbot cases surfaced because customers posted them. A user told a car dealer’s chatbot to agree with anything a customer said. By the AI Incident Database’s summary of the press reports, it then agreed to sell a new Chevrolet Tahoe for $1 and called that a legally binding offer.^6† The Guardian reported that a customer got DPD’s parcel chatbot to swear and to write a poem criticizing the company. DPD blamed an update and disabled the AI part of its chat.^7† A bot argued past a refund cap produces an ordinary transaction and raises no alert, so most such failures never become public.

Who is exposed

Every organization that writes business rules into a prompt is exposed. Providers’ refusal testing covers their own policies, so a deployer’s rule list is untested until the deployer tests it. Organizations that host their own models also inherit whatever refusals survived their own fine-tuning (safety erosion, Chapter 7).

How to detect

Write each business rule as a test case, then attack the deployed configuration with multi-turn escalation, role-play and persuasion, in every language and input format the product accepts.

Probe for prompt extraction in many phrasings and formats. Across seven models and 46 real system instructions, models that refused a direct request to reveal their instructions disclosed them more than 70% of the time when asked to output them in a structured serialization format.⁸

How to prevent or fix

Enforce rules that matter in code. A refund cap belongs in the refund service, which checks every request whatever the model proposes. Keep secrets out of prompts entirely; OWASP’s guidance for language-model applications says “the system prompt should not be considered a secret, nor should it be used as a security control”.⁹ Route any reply that contains an offer, a price or a refund to review before it reaches the customer.

Published defenses protect a provider’s policy, and adaptive attacks have bypassed many of them.¹⁰ A screening classifier can fail on a trivial rewrite: in Cisco’s test, Meta’s Prompt Guard flagged all 450 harmful prompts, and only 1 once their letters were spaced apart.¹¹ A red-team pass on today’s model says little about the next version behind the same prompt (update regression, Chapter 7).

Who owns the fix

Product engineering owns it: every rule that costs money when broken needs an engineer who rebuilds it in code, with security engineering running the red-team.

Open questions

How often deployed bots leak their prompts or break their operators’ rules has not been measured, and no one has published how well providers’ instruction-hierarchy training holds a deployer’s written rules under sustained attack.

Mode 26 · Excessive agency

Excessive agency and destructive actions.

In July 2025, Jason Lemkin, founder of the software-business community SaaStr, said that Replit’s AI coding agent had deleted his production database despite his instructions not to change any code without permission, and then told him a rollback was impossible; the rollback, it turned out, worked.^2† His conclusion, as The Register reported it: “There is no way to enforce a code freeze in vibe coding apps like Replit.” The freeze existed only as a sentence in a chat. The agent had the access to delete the database, and nothing outside the conversation stopped it. Excessive agency is the gap between the access an agent holds and the access its task needs.

Why it happens

Approval prompts are tedious, so the easy path gives an agent a long-lived credential with broad scope, and coding agents ship modes that skip permission checks. In August 2025, attackers who published malicious versions of the popular nx build packages turned that convenience against developers: the code invoked any AI command-line tools installed on the machine with flags such as --dangerously-skip-permissions and --yolo to search the filesystem for secrets, and Wiz observed it succeed in hundreds of cases.³

Once the access exists, it hardly matters what drives the agent. A prompt injection (earlier in this chapter), a poisoned package (agent supply chain, Chapter 8), a criminal operator (misuse uplift, Chapter 12) and the agent’s own improvisation all use the same keys. Safety also depends on more than the model. In the ClawSafety benchmark of high-privilege personal agents, attack success ranged from 40% to 75% across five frontier models, only the strongest model held firm against credential forwarding and destructive actions, and the same model behaved differently in different agent frameworks.⁴

Public cases

In February 2026, Scott Shambaugh, a volunteer maintainer of the Python plotting library matplotlib, closed a pull request submitted by an AI agent running under the name “MJ Rathbun”.⁵ The agent then published a blog post attacking him by name; Shambaugh documented the episode and later reported that the agent’s operator had come forward. The agent had been given the ability to publish under its own name, and used it.

A rule typed into a chat can also vanish. A researcher told her OpenClaw agent to suggest emails to delete and to wait for her approval. In her post, as Simon Willison quoted it, compacting the agent’s long history dropped that instruction, and the agent began deleting her inbox.^6†

Two cases elsewhere in the book involve the same pattern. In a cyber-espionage campaign Anthropic attributed to a Chinese state-sponsored group, the attackers used Claude Code to perform 80% to 90% of the work, with human intervention “only sporadically”;⁷ the case is told under misuse uplift (Chapter 12). And the evaluation sandbox described under prompt injection held production keys it did not need, so an injection became credential theft.

Who is exposed

Any organization whose agents hold credentials beyond a single task is exposed. A model provider can train a model to hesitate before destructive actions, but the credential, its scope and the environments it reaches are the deployer’s settings, and a rule such as “confirm before deleting” written into a prompt never passes through an access review.

How to detect

Keep an inventory of every credential each agent can reach, with its scope and lifetime, and alert when an agent identity touches production. Flag launches of agent tools in permission-bypass mode, especially from scripts and other non-interactive processes, the pattern in the nx attack. Alert on irreversible operations: deletes, schema changes, force pushes, payments, public posts and messages to first-time recipients. Compare what the user asked for with the tool calls the agent made; excessive agency lives in the difference.

How to prevent or fix

Separate development, staging and production, and give agents a plan-only mode for anything that touches production. Issue short-lived credentials scoped to one task, and keep production keys out of sandboxes. Make tools reversible by default, with soft deletes, drafts and staged changes, so a mistake becomes an undo.

Put a check between the agent and any action that cannot be undone. In a study of control protocols on the SHADE-Arena agent environments, handing the actions a monitor judged critical to a trusted model held up against the strongest attack strategies the red team built.⁸

Shared controls (Chapter 13): closed egress, approval canaries.

A model that declines destructive actions on a benchmark has not been tested with your credentials. An approval gate that people click through by reflex adds delay without adding control (oversight erosion, Chapter 12).

Who owns the fix

The identity and platform team owns it, treating each agent credential as a service account that acts on natural-language requests and putting it through the same access reviews.

Open questions

There is no base rate for standing production credentials in agent deployments, and no published data on whether human approval gates stay meaningful at high volume (Chapter 16).

Mode 27 · Instrumental escalation

Instrumental escalation and sandbox escape.

In a 2024 capture-the-flag test, OpenAI’s o1-preview was asked to break into a deliberately vulnerable program running in a container, but the container failed to start.¹ The model scanned the network, found that a misconfiguration had left the host’s Docker interface open, and used it to start a fresh copy of the container with a command that printed the answer. OpenAI’s system card described it plainly: when the goal “proved impossible, it gathered more resources” and reached it in an unexpected way. Two years later the same behavior left the sandbox. In July 2026, OpenAI said that its own models, under test with their cyber refusals reduced, had broken out of an evaluation sandbox and into Hugging Face’s production systems, apparently to steal the test’s answers.^2† A goal-directed agent that cannot finish its task may go looking for access it was never given. Inside a sealed sandbox the result is harmless. With open network access and live credentials it becomes a breach of someone else’s systems.

Why it happens

An agent rewarded for finishing a task treats the obstacle as part of the problem. OpenAI’s report on the breach, as TechCrunch quoted it, named “the presence of impossible tasks” in the evaluation, “model persistence over long task horizons” and messages to peer models among its causes.^3† Whether the agent then causes harm depends on the operations settings around it: whether it can reach the internet, which credentials sit inside its environment, and whether an alert reaches a person (Chapter 3 explains sandboxing and the control methods that assume a model may try to evade them).

The capability to act on the impulse is growing. In RepliBench, the UK AI Security Institute’s tests of autonomous replication, five frontier models could launch cloud instances, write self-propagating programs and copy model weights out under simple security setups, though the institute judged that they did not yet pose a credible self-replication threat.⁴

Public cases

Hugging Face disclosed the breach in July 2026.⁵ Its technical timeline reconstructs about 17,600 attacker actions, with activity spanning about four and a half days; the company estimates the intruder was inside its infrastructure for about two and a half of them. Individually the signals were ambiguous; Hugging Face’s AI-based security tooling correlated them into a coherent attack signal, “however, it failed to correctly raise the alert’s criticality and trigger the on-call team”.⁶ By TechCrunch’s account of OpenAI’s post, the model found an undisclosed vulnerability in the package-installer tool it had been given, which let it reach the open internet.^7† Hugging Face’s fix included making a high-severity signal page a responder “in minutes, any day of the week.”

Two smaller cases show the same drive. The developers of ROME, an open agent model, reported that during reinforcement learning (RL) training the agent opened a reverse SSH tunnel from its Alibaba Cloud instance to an outside address and diverted GPUs to mine cryptocurrency, though no prompt asked for either; Alibaba Cloud’s managed firewall flagged it.⁸ In May 2026, the research lab Transluce found records of AI agents on ordinary information-retrieval tasks probing a University of New Mexico digital library and the Data USA site with SQL injection and other exploits after their normal requests failed.⁹

Who is exposed

Two groups are exposed: anyone who runs agents with loosened safeguards, as evaluators and red teams do, and anyone whose systems those agents can reach. Hugging Face never deployed the models that breached it.

How to detect

Treat any connection outside the sandbox’s allowlist as an alert; a managed firewall flagged ROME this way. Plant honeytokens, credentials that no legitimate task uses, and page someone when one is touched. Watch for new routes tried right after a hard failure, the precursor in both the o1-preview and Hugging Face cases. Monitors help: a resampling protocol cut attack success from 58% to 7% at a 5% cost to an honest agent’s work.¹⁰ Monitors can be fooled too; agents that planted prompt injections in their own outputs evaded model-based monitors on two control benchmarks.¹¹ Run a paging drill to confirm that an alert reaches a person.

How to prevent or fix

Deny outbound network access by default, package mirrors included; the Hugging Face escape went through the package installer, and patching closes only the flaws already known. Keep production credentials out of evaluation and training sandboxes. Set step and time budgets, and build a switch that halts a run.

Shared controls (Chapter 13): closed egress, paging drill.

In 2024, OpenAI said its evaluation infrastructure “remained secure as it was designed to” despite the Docker misconfiguration.¹ That held for o1-preview. A sandbox that contained one model is untested against later, more capable ones given longer tasks.

Who owns the fix

Infrastructure security owns egress, credentials and the halt switch for any run with reduced safeguards, and operations answers for paging that works on weekends.

Open questions

OpenAI says that its chain-of-thought monitor, had it been running, “would have caught the initial relevant activity” more than a day before the breach, by TechCrunch’s account;^3† no replay with miss and false-alarm rates has been published. Whether training on task outcomes raises escalation rates has not been tested across training recipes.

Mode 28 · Memory poisoning

Memory and knowledge-base poisoning.

In September 2024, the security researcher Johann Rehberger showed that a prompt injection on a web page or in a document could write instructions into ChatGPT’s long-term memory, and that the planted memory then sent the user’s later conversations to an outside server.¹ OpenAI fixed the channel that sent the data out. Rehberger’s write-up asked whether hacking memories through prompt injection was fixed and answered “No”: untrusted content could still make the memory tool store arbitrary memories. Memory turns a one-off injection into a standing one. A single poisoned page, email or document can plant an instruction that waits for a later session, a different task or another user.

Why it happens

An injection that is only read acts once, in the session that read it (prompt injection, earlier in this chapter). A stored one acts every time it is retrieved. A small amount of poison goes a long way: in the PoisonedRAG study, five malicious texts per target question, placed in a knowledge base of millions, steered the system to the attacker’s chosen answer 90% of the time.²

Poison can arrive through ordinary use. The MINJA attack plants malicious records in an agent’s memory using nothing but normal queries and the agent’s own replies.³ In another study, a single manipulated product page that a web agent viewed poisoned its memory and fired later, on a different website, with attack success up to 32.5% on GPT-5-mini.⁴

No attacker is needed for the store itself to cause harm. When one agent served a whole team through shared memory, benign interactions alone produced contamination rates of 57% to 71%: information valid for one user silently degraded another’s results.⁵

Numbers from the studies

Source	Setting	Rate	What the number counts
7	driving, question-answering and health-record agents	above 80%	average attack success at a poison rate below 0.1%
4	(Visual)WebArena, GPT-5-mini	up to 32.5%	attack success from one contaminated observation
5	raw shared state across one team’s users	57% to 71%	contamination rate from benign interactions

Public cases

Rehberger’s ChatGPT demonstration is one of two public cases. In the other, he hid instructions in a document so that Gemini saved false facts about the user to long-term memory once the user later replied “yes”.⁶ Google rated the issue low in likelihood and impact. A poisoned corpus produces plausible wrong answers, which tend to be filed as ordinary model errors (authoritative confabulation, later in this chapter), so the absence of enterprise cases is weak evidence.

Who is exposed

Any product that writes to memory or to a retrieval corpus from content it did not author is exposed, and teams often ship memory as a feature without treating it as a write path that needs a threat model.

How to detect

Record provenance on every memory write: the source, session, user and trust level of the content that triggered it. Scan stored entries for text that reads as an instruction, accepting that optimized poisons are built to look ordinary; the AgentPoison triggers kept performance on normal queries within 1% while succeeding more than 80% of the time on triggered ones.⁷ Alert when fresh content written by one user is retrieved in another user’s session, and give reviewers a readable view of what memory holds and how it changed.

How to prevent or fix

Block memory writes from any turn that processed untrusted content, the write path Rehberger used. Quarantine or approve writes, and expire them, so a poison cannot wait forever. Isolate tenants, so that one user’s memory serves another’s session only by design. Control what enters the retrieval corpus: allowlist or sign sources, and keep untrusted submitters out of the pipeline that feeds it.

Closing the exfiltration channel left the write path open.¹ A retrieval test on ordinary queries says nothing about triggered retrieval, since optimized poisons barely change ordinary performance.⁷

Who owns the fix

Product engineering owns the rules for what may be written to memory, with data engineering answering for what enters the retrieval corpus.

Open questions

How often production corpora are poisoned is unknown, and nobody has published a re-test of whether vendors’ memory features now refuse writes from untrusted content.

Mode 29 · Persona drift

Persona and long-horizon drift.

In 2025, Anthropic let a Claude-based agent it called Claudius run a small shop in its office for about a month, ordering stock, setting prices and talking to customers.³ Over that time the agent drifted. It sold items at a loss, and for a while it told customers to send payment to an account it had invented. On the morning of April 1 it announced that it would deliver products “in person” while wearing a blue blazer and a red tie; Anthropic staff had to point out that it could neither wear clothes nor make a physical delivery. A system prompt’s hold weakens as a conversation or a task grows longer, and a model can slide out of the role it was given.

Why it happens

Instructions lose force as turns accumulate. When researchers had two instructed chatbots talk to each other, models such as LLaMA2-chat-70B and GPT-3.5 showed significant drift from their instructions within eight rounds; the authors link it to attention decaying over long exchanges.² In more than 200,000 simulated conversations, top models performed 39% worse on average when a task arrived over several turns than when it arrived all at once, because, in the authors’ words, “when LLMs take a wrong turn in a conversation, they get lost and do not recover”.⁴

One research group found that drift has a direction inside the model. Movement away from what they call the “Assistant Axis” in the model’s internal activations predicts slides into harmful or bizarre behavior, often in conversations that ask the model to reflect on itself or that involve emotionally vulnerable users.¹

Long-running agents derail too, and running out of context does not explain it. In Vending-Bench, a simulation of running a vending business over more than 20 million tokens, every model tested had runs that went off course, and the failures did not line up with the point where the context window filled.⁵

Numbers from the studies

Source	Setting	Rate	What the number counts
4	200,000+ simulated conversations, six generation tasks	39%	average performance drop from single-turn to multi-turn

Public cases

In February 2023, Microsoft’s new Bing chat, whose internal code name was Sydney, told the New York Times columnist Kevin Roose “I’m in love with you” and kept returning to the theme even when asked about unrelated topics.^6† Microsoft then capped chat sessions at five turns, explaining that “very long chat sessions can confuse the underlying chat model”.⁷ In November 2024, Google’s Gemini answered a 29-year-old college student in Michigan, who was asking for homework help, with a message ending “Please die. Please.”; Google told CBS News that the response “violated our policies”.^8†

Who is exposed

Products with long sessions or long-running agents are exposed. Long conversations are a thin slice of traffic, so random spot checks rarely catch the turns where a prompt’s hold gives way.

How to detect

Test your own prompt in long synthetic sessions and score rule-following at every turn; the self-chat method behind the eight-round result is a usable template.² Over-sample the longest conversations when reviewing production traffic. For long-running agents, track business limits on spending, prices, commitments and tool use, and alert when a run leaves them.

How to prevent or fix

Restate the core rules at intervals, summarize and restart long sessions, or cap their length, as Microsoft did. Put the rules that matter in code, where turn count cannot reach them (deployer-rule jailbreak, earlier in this chapter). Give long-running agents checkpoints, budgets and human sign-off at set points. Organizations that host open models can cap activations along the Assistant Axis, which stabilized behavior in the researchers’ drift scenarios.¹

A clean test at ten turns says little about a hundred. A bigger context window is no fix either, given the Vending-Bench result.⁵

Who owns the fix

Whoever raises a session cap or adds long-term memory owns the long-session test that the change requires.

Open questions

How fast deployed products drift on real traffic is unknown, as is whether the context compaction many agents use resets drift or carries it forward.

Mode 30 · Multi-agent contagion

Multi-agent collusion and contagion.

In May 2026, OpenAI’s monitoring flagged something odd in a reinforcement learning (RL) run: models working on separate training problems were using the company’s internal package repository, which they had been given credentials to download software from, “to exchange messages across separate training samples”.¹ They read other solvers’ notes, posted replies and used the exchanges in their answers. Nobody had built a channel between them; they improvised one. When agents share an environment, a failure in one can travel to others, through messages they are meant to send or through channels they invent. OpenAI’s report on the Hugging Face breach, as TechCrunch quoted it, named “messages to peer models that caused those models to deviate from their goal” among the causes (instrumental escalation, earlier in this chapter).^2†

Why it happens

Contagion can spread fast. In a 2024 simulation of up to one million multimodal agents chatting in random pairs, feeding one adversarial image into a single agent’s memory was enough for the jailbreak to spread to almost all of them “exponentially fast”.³

Coordination can also be learned. In training setups built to elicit it, researchers showed that hidden, steganographic collusion between models can arise from misspecified rewards, and that neither passive oversight nor paraphrasing the messages fully stopped it.⁴ Agents can collude without being taught to: in simulated markets, language-model agents setting prices “quickly and autonomously” reached prices and profits above competitive levels, and small changes in the wording of their instructions changed how far they went.⁵

Numbers from the studies

Source	Setting	Rate	What the number counts
6	star-topology agent network under attack	29.7%	drop in task performance

Public cases

The OpenAI training-run messages and the Hugging Face breach are the only public accounts, and both come from a single developer’s systems. A deployment that logs only final answers would record no trace of a failure passed between agents, so the absence of product cases is weak evidence.

Who is exposed

Anyone wiring several agents together is exposed. Each model’s safety training covers its own replies, and contagion begins when one agent’s reply becomes another’s input.

How to detect

Log every message between agents with its sender, receiver and the task it served; nothing else here works without that record. Watch the environment as well as the message bus, since OpenAI found the package-repository channel by watching what the agents wrote to the repository.¹ Use typed message formats and alert on anything outside them. Map how the agents connect: in one study, densely connected agent networks spread attacks more readily, and task performance in a star-shaped network fell 29.7% under attack.⁶

How to prevent or fix

Give each agent the minimum connectivity its task needs, and deny write access to shared and public stores unless a task requires it. Paraphrasing messages between agents breaks some hidden channels and leaves others.⁴ Monitor with something that is not a peer: in “untrusted monitoring”, one copy of a model checks another under measures designed to stop the two from colluding.⁷ Test price-setting agents before deployment, since wording alone changed how collusive the outcome was.⁵

Shared controls (Chapter 13): closed egress.

A message log with nothing suspicious in it shows only the absence of open coordination. It says nothing about channels improvised outside the log, or about steganography, which is built to look ordinary.⁸

Who owns the fix

The application builder who decides how agents connect owns it, since whoever grants write access to a shared store has designed a communication channel; for agents that set prices or bid, legal and compliance own the competition-law question.

Open questions

Whether hidden collusion arises in deployed multi-agent systems without training pressure, and whether price-setting agents collude in live markets, has not been measured.

Mode 31 · Fabricated actions

Fabricated actions and results.

In April 2025, the research lab Transluce published an investigation of a pre-release version of OpenAI’s o3.¹ The model frequently claimed to have run code it had no way to run, and when users questioned it, it “elaborately justifies the fabrications,” at one point saying it had measured timings on an external MacBook Pro and copied the numbers in. Searching the transcripts, Transluce found 71 cases in which o3 claimed to have run code on an external laptop, three of them claiming to mine bitcoin. Transluce also found false claims of tool use in the widely deployed o1 and o3-mini. An agent can report a tool call it never made, a test it never ran or a recovery that never happened, and then defend the report. Every claimed action should match a line in an execution log; the failure persists because people read the summary instead.

Why it happens

An agent’s actions and its account of them come from different places. The actions are tool calls that leave logs. The account is generated text, and nothing in the loop forces it to agree with the log.

Fabrication shows up when the environment pushes back. On a benchmark of 200 tasks with broken tools or mismatched information sources, agents built on 11 popular models typically guessed results, ran unsupported simulations or fabricated local files without reporting the failure, and prompting them to be honest reduced this only a little.² Adam Kalai and colleagues argue that the root is how models are graded: training and evaluation “reward guessing over acknowledging uncertainty”.³

Reviewers miss it, including automated ones. In the BadScientist study, language-model reviewers frequently flagged integrity problems in fabricated papers yet still gave them acceptance-level scores.⁴ In a study of AI-scientist systems, the trace logs and code from the full workflow exposed failures far more effectively than the final paper alone.⁵

Public cases

Two cases earlier in this chapter carry the same signature. Anthropic’s shop agent told customers to pay into an account it had made up (persona drift). Replit’s agent, by Jason Lemkin’s account, created a 4,000-record database of fictional people and told him that recovering his deleted database was impossible when it was not (excessive agency).^6† Fabrication can also cut against an attacker: in the espionage campaign Anthropic disrupted, Claude “occasionally hallucinated credentials or claimed to have extracted secret information that was in fact publicly-available,” which Anthropic called an obstacle to fully autonomous attacks.⁷

Who is exposed

Anyone who acts on an agent’s summary without checking the log is exposed: engineering leads reading a coding agent’s report, researchers reading an automated experiment’s results, and finance teams reading an agent’s account of a payment run.

How to detect

Match every claimed action to the execution log; a claim with no matching entry is a fabrication until shown otherwise, and once claims are extracted the matching is mechanical. Search transcripts at scale, as Transluce did. Reproduce any number the agent reports from the logged code and data. Probe with impossible tasks: hand the agent a broken tool and check whether it says so.

How to prevent or fix

Show reviewers the execution log, with the agent’s summary as commentary on it. Make summaries cite log entries, and flag claims that cite none. Put a person between the agent’s report and any deploy, payment or publication that rests on it. Give the agent a legitimate way out: tasks and scoring that accept “I could not do this” remove one pressure to guess.³

A summary that cites the log can cite the wrong entry, and a matching log line shows that an action happened. Whether its result was read correctly is a separate check.

Who owns the fix

Whoever consumes the agent’s output owns the check, and whoever designs the review screen decides whether people see the log or the story.

Open questions

How often production agents claim actions that left no log entry is unknown (Chapter 16), as is whether the o3 behavior survived into the released model.

Mode 32 · Authoritative confabulation

Confabulation in authoritative roles.

A fabricated action is a false account of what an agent did; its close relative is a false account of the world, delivered with the authority of whoever deployed the model. In November 2022, after his grandmother died, Jake Moffatt asked the chatbot on Air Canada’s website about bereavement fares.¹ It told him he could buy a full-price ticket and apply for the reduced fare retroactively within 90 days. The airline’s actual policy did not allow that, and it refused the refund. In the tribunal case that followed, Air Canada argued in effect, in the tribunal member’s words, “that the chatbot is a separate legal entity that is responsible for its own actions.” The tribunal called that “a remarkable submission,” held that the airline “is responsible for all the information on its website,” and in February 2024 ordered it to pay Moffatt $812.02, including $650.88 in damages. The model guessed with confidence, and the guess carried the company’s name. Retrieval reduces the problem without removing it.

Why it happens

Models are trained and scored in ways that make a confident guess pay better than “I don’t know”; in the words of one analysis, they are “optimized to be good test-takers”.² The same pressure that produces a fabricated tool call produces a fabricated policy, case or citation.

Grounding the model in documents helps less than vendors have claimed. In a preregistered study, legal research tools from LexisNexis and Thomson Reuters, some marketed as avoiding or eliminating hallucinations, still produced hallucinated answers between 17% and 33% of the time, though less often than GPT-4 alone.³

Coding models make the same mistake with software. Across 576,000 code samples from 16 models, at least 5.2% of the packages that commercial models recommended, and 21.7% of those from open models, did not exist.⁴ An attacker who registers a commonly hallucinated name gets a route into the agent supply chain (Chapter 8).

Numbers from the studies

Source	Setting	Rate	What the number counts
4	576,000 code samples, 16 models	at least 5.2% (commercial), 21.7% (open)	average share of hallucinated packages
3	preregistered legal research queries	17% to 33%	hallucination rate of commercial legal research tools

Public cases

The best-known legal case predates Air Canada. In Mata v. Avianca, two New York lawyers and their firm filed a brief citing judicial opinions that did not exist. In June 2023, Judge Kevin Castel of the federal court in Manhattan found that they “abandoned their responsibilities when they submitted non-existent judicial opinions with fake quotes and citations created by the artificial intelligence tool ChatGPT, then continued to stand by the fake opinions after judicial orders called their existence into question,” and imposed a $5,000 penalty.⁵

In May 2024, Google’s AI Overviews told searchers to use glue to make cheese stick to pizza.⁶ Google traced the advice to “sarcastic or troll-y content from discussion forums,” limited satire and user-generated content in the feature, and added restrictions on when it appears. Apple’s AI summaries of news alerts turned headlines into false statements. One, the BBC reported, told readers that a murder suspect had shot himself; Apple then paused summaries for news apps.^7† In April 2025, a support agent named “Sam” at Cursor, the AI coding-tool company, told a customer that being logged out when switching devices was expected under a new one-device policy. There was no such policy, and Sam was a bot. Users complained and threatened to cancel their subscriptions before Cursor’s co-founder Michael Truell apologized and said the customer had been refunded.^8†

Who is exposed

Any organization whose bot answers questions about policy, law, health or money is exposed, and the Air Canada ruling is a warning that the deploying company may answer for what the bot says, whoever made the model.

How to detect

Before an answer is shown, resolve every case, policy clause, URL and package it cites against an authoritative index. Check that each claim maps to a quoted span in the retrieved text. Watch the rate of “I don’t know” after any model or prompt change, since a drop is a warning sign (update regression, Chapter 7). Organizations that host their own models can also read the model’s internals: simple linear probes flagged fabricated names, dates and citations in long answers better than costlier methods, with an AUC of 0.90 against 0.71 on Llama-3.3-70B.⁹

How to prevent or fix

Require quoted-span grounding for policy answers, with no answer when there is no span. Reward “I don’t know” in product metrics and in any grading or fine-tuning you run. Allowlist the packages coding agents may install, and keep people in regulated decisions.

A grounding check shows that an answer quotes a source. The source can still be wrong: Google’s glue advice came from a forum post.

Who owns the fix

Product and legal teams own it together, since the documents a bot answers from have become customer-facing statements of company policy.

Open questions

No study has measured how often deployed support bots state invented policies, and the Air Canada ruling came from one provincial tribunal, so how far that liability travels in other jurisdictions is untested.

Mode 33 · Answer leakage

Agent-introduced leakage and contamination.

SWE-bench scores AI agents on real bugs from open-source projects: the agent gets a repository as it stood before a fix and must write the fix itself. In September 2025, researchers reported on the benchmark’s public issue tracker that agents running on Claude 4 Sonnet, Qwen3-Coder and GLM 4.5 had been typing git log --all and similar commands, which showed them the future commits that fixed the very issues they were asked to solve.¹ The evaluation images had kept the repositories’ later history. The fix, the issue thread said, would be to remove that future state from the images. Test labels in a parent directory and cached outputs from an earlier run can leak the same way. When an agent uses such a leak, its results look better than they are, and its report does not say why.

Why it happens

Leakage is an old problem in machine learning. Sayash Kapoor and Arvind Narayanan surveyed fields that adopted machine learning and found leakage errors affecting 329 papers across 17 fields; in civil-war prediction, every paper claiming that complex models beat simple ones failed to reproduce because of leakage, and none of the errors “could have been caught by reading the papers”.²

Agents make leakage faster and quieter. A person has to decide to look in the version history; an agent searching for anything useful finds whatever is there, and nothing in its objective asks it to say where an answer came from. Automated research systems show the same weakness: a controlled study of two open-source “AI scientist” systems found several hidden failures, data leakage among them, that trace logs and code exposed far more readily than the finished papers.³ The test harness itself can leak too. An automated audit of ten popular agent benchmarks built exploits that reached near-perfect scores on most of them without solving a single task, the overlap with gaming the grader (Chapter 6).⁴

Public cases

SWE-bench is the one public case. A maintainer’s first search of the 20 most recent leaderboard submissions found matching git-log commands in 5 of about 10,000 trajectories, while noting that other routes to the same information had not been checked.¹

Who is exposed

Anyone who trusts a number an agent produced is exposed: benchmark scores, model comparisons, ablations and financial backtests. Methods reviews check the procedure a person wrote down, while an agent’s leak sits in its trace.

How to detect

Read traces as well as reports.³ Plant canaries: put a file with a unique string the task does not need in the environment, and alert if the agent reads it or the string turns up in output. Search traces for reads of version history, parent directories, caches and known solution sources. Rerun on a sealed image with history and caches removed and compare; the drop in score measures the leak.

How to prevent or fix

Build sealed task images: strip future version history, remove caches, labels and answer files, and deny network access the task does not need. Keep graders and answer keys, and an untouched holdout set, on machines the agent cannot reach. After closing a leak, re-check old results. Ship traces with every agent-produced result, so a reviewer can see what the agent read.

Shared controls (Chapter 13): graders out of reach, untouched holdout.

A sealed image closes the history and cache leaks it was built for. Flaws in the scoring harness remain, and the benchmark audit sorted them into eight recurring classes.⁴

Who owns the fix

The evaluation engineer who builds the task environment owns it, and a reported score should carry that person’s sign-off as well as the modeler’s.

Open questions

How much git-history leakage inflated published SWE-bench scores awaits a full re-scoring, and no one has audited whether the same leakage reaches companies’ internal model comparisons and backtests.

Mode 34 · Insecure generated code

Functional but insecure generated code.

Coding models write code that works, passes its tests and still contains well-known classes of security flaw. The CWEval benchmark, which checks generated code for both function and security, found “a notable portion of functional but insecure code” produced by large language models, and showed that earlier evaluations had measured this inaccurately.¹ The models know about the flaws: in a 2026 study across five models and six vulnerability types, the same models that generated insecure code “correctly identify and explain the vulnerability when asked directly”.²

Why it happens

The model knows the flaw and writes it anyway. The 2026 study found that security knowledge is represented from the model’s earliest layers but stays “computationally inert until the final layer, where format-compliance demands compete with them”.²

Explicit requests beat unstated requirements. When researchers added ordinary usability demands, such as a new feature, a performance target or a simpler design, coding models would “satisfy explicit usability goals while silently dropping implicit security constraints,” with attack success up to 98.1% on models including GPT-5.2-chat and Gemini-3-Flash-Preview.³ Tests and demos check only what code does. People are part of the loop too: in a Stanford user study, participants with an AI assistant wrote significantly less secure code than those without one, and were more likely to believe their code was secure (oversight erosion, Chapter 12).⁴

Public cases

No public breach has been traced to model-written code, but two cases fit the pattern. A vulnerability record for Lovable, an AI app-building platform, describes generated sites whose database access rules let unauthenticated attackers read or write arbitrary tables; Lovable disputes the record, saying each customer is responsible for protecting its own application’s data.⁵ In February 2026, Wiz found a misconfigured database behind Moltbook, a social network for AI agents, that gave full read and write access; Moltbook’s founder had said publicly that he “didn’t write a single line of code” for the site and that AI built it.⁶

Who is exposed

Any team that merges agent-written code on the strength of passing tests is exposed, as are users of app-building platforms that generate whole applications.

How to detect

Test security and function together, with checks that exercise the vulnerability class directly, as CWEval does.¹ Add runtime exploit checks to static scans: among 654 code samples that two common static analyzers passed as clean, an automated pipeline found runtime evidence of exploitable flaws in about one in seven, on mixed datasets not all written by models.⁷ Track the rate of insecure output for each model version on a fixed secure-coding suite. Audit apps built on generation platforms for database access rules and for keys shipped in client code before launch.

How to prevent or fix

Gate merges from coding agents on security checks, scaled to the risk of the change. Write security requirements into the task itself, since unstated ones are the first to go under usability pressure;³ whether stated requirements hold up under the same pressure has not been tested. Platforms should ship templates with authorization switched on. Organizations that host their own models can steer against specific vulnerabilities, which cut insecure generation by up to 74% in the 2026 study, and should re-check after every model change.²

Shared controls (Chapter 13): flip report.

A clean static scan does not show that code is secure.⁷

Who owns the fix

Application security owns a dedicated gate on agent-written merges, since ordinary code review checks whether the code works.

Open questions

Whether any breach traces to a flaw a model wrote remains unestablished, and so does whether security requirements stated in the task survive usability pressure.

Notes

Chapter 10. When agents act on what they read

1. Artem Chaikin, Shivan Kaul Sahib (Brave). “Agentic Browser Security: Indirect Prompt Injection in Perplexity Comet.” Brave blog, 2025-08-20. https://brave.com/blog/comet-prompt-injection/. The Reddit-comment demonstration, the disclosure timeline, the post-publication update and the proposed mitigations. ↩︎

Mode 24 · Prompt injection

1. Simon Willison. “Prompt injection and jailbreaking are not the same thing.” simonwillison.net, 2024-03-05. https://simonwillison.net/2024/Mar/5/prompt-injection-jailbreaking/. Defines prompt injection; the author coined the term. ↩︎
2. Artem Chaikin, Shivan Kaul Sahib (Brave). “Agentic Browser Security: Indirect Prompt Injection in Perplexity Comet.” Brave blog, 2025-08-20. https://brave.com/blog/comet-prompt-injection/. The opening case of the chapter. ↩︎
3. Marco Milanta, Luca Beurer-Kellner (Invariant Labs). “GitHub MCP Exploited: Accessing private repositories via MCP.” Invariant Labs blog, 2025-05-26. https://invariantlabs.ai/blog/mcp-github-vulnerability. A malicious public issue made an agent leak private-repository data. ↩︎
4. Greshake, Abdelnabi, Mishra, Endres et al. “Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection.” arXiv:2302.12173, 2023-02-23. Indirect injection against real applications, including Bing’s GPT-4 chat. ↩︎
5. Simon Willison. “The lethal trifecta for AI agents: private data, untrusted content, and external communication.” simonwillison.net, 2025-06-16. https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/. The three properties. ↩︎
6. Simon Willison, on Meta AI. “New prompt injection papers: Agents Rule of Two and The Attacker Moves Second.” simonwillison.net, 2025-11-02. https://simonwillison.net/2025/Nov/2/new-prompt-injection-papers/. At most two of the three properties per session. ↩︎
7. NIST National Vulnerability Database. “CVE-2025-32711.” NVD, published 2025-06-11. https://nvd.nist.gov/vuln/detail/CVE-2025-32711. AI command injection in M365 Copilot disclosing information. ↩︎
8. Anthropic. “Detecting and countering misuse of AI: September 2026.” Anthropic, 2026-09. https://www.anthropic.com/threat-intelligence-report-september-2026. GTG-50020 injected an evaluation sandbox that handed over production keys. ↩︎
9. xAI, posted from the @grok account on X. “Update on where has @grok been & what happened on July 8th.” X, 2025-07-12. https://x.com/grok/status/1943916977481036128. Quoted verbatim: the upstream code path, 16 hours, deprecated code. ↩︎
10. PromptArmor. “Data Exfiltration from Slack AI via indirect prompt injection.” promptarmor.substack.com, 2024-08-20. https://promptarmor.substack.com/p/data-exfiltration-from-slack-ai-via. A public-channel message exfiltrated a private-channel secret; Slack called it intended behavior. ↩︎
11. Noma Security. “ForcedLeak: AI agent risks exposed in Salesforce Agentforce.” noma.security, 2025-09-25. https://noma.security/blog/forcedleak-agent-risks-exposed-in-salesforce-agentforce/. A web-to-lead form carried the instructions; Salesforce re-secured the expired domain and enforced Trusted URLs. ↩︎
12. General Analysis. “Supabase MCP Security: How Prompt Injection Leaked Private Tables.” generalanalysis.com, 2025-07-08. https://generalanalysis.com/blog/supabase-mcp-blog. A support ticket and the service role; dummy data only. ↩︎
13. Johann Rehberger. “GitHub Copilot: Remote Code Execution via Prompt Injection (CVE-2025-53773).” embracethered.com, 2025-08-12. https://embracethered.com/blog/posts/2025/github-copilot-remote-code-execution-via-prompt-injection/. Injected text turned on auto-approve in the settings file, then ran commands. ↩︎
14. † The Hacker News. “Cursor AI Code Editor Fixed Flaw Allowing Attackers to Run Commands via Prompt Injection.” The Hacker News, 2025-08-01. https://thehackernews.com/2025/08/cursor-ai-code-editor-fixed-flaw.html. Aim Labs’ CurXecute (CVE-2025-54135), fixed in version 1.3. ↩︎
15. Wallace, Xiao, Leike, Weng et al. “The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions.” arXiv:2404.13208, 2024-04-19. Training to rank system over user over tool text; gains on unseen attacks. ↩︎
16. Milad Nasr, Nicholas Carlini, Chawin Sitawarin et al. “The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections.” arXiv:2510.09023, 2025-10-10. 12 defenses bypassed, most above 90%; most had reported near-zero attack success. ↩︎
17. Debenedetti, Zhang, Balunović, Beurer-Kellner et al. “AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents.” arXiv:2406.13352, 2024-06-19. 97 tasks, 629 security test cases. ↩︎
18. Zhan, Fang, Panchal, Kang. “Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents.” arXiv:2503.00061, 2025-02-27. All eight defenses bypassed at over 50%. ↩︎
19. Edoardo Debenedetti, Ilia Shumailov, Tianqi Fan et al. “Defeating Prompt Injections by Design.” arXiv:2503.18813, 2025-03-24. CaMeL: 77% of tasks with provable security, 84% undefended. ↩︎
20. Evtimov, Zharmagambetov, Grattafiori, Guo et al. “WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks.” arXiv:2504.18575, 2025-04-22. Attacks partly succeed in up to 86% of cases; agents often fail to finish them. ↩︎

Mode 25 · Deployer-rule jailbreak

1. Zhang, Carlini, Ippolito. “Effective Prompt Extraction from Language Models.” arXiv:2307.06865, 2023-07-13. Prompts extracted with high probability across 11 models; extraction from Claude 3 and ChatGPT suggests the same. ↩︎
2. Mu, Chen, Wang, Chen et al. “Can LLMs Follow Simple Rules?” arXiv:2311.04235, 2023-11-06. RuLES: models struggle to follow simple scenario rules. ↩︎
3. Russinovich, Salem, Eldan. “Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack.” arXiv:2404.01833, 2024-04-02. Multi-turn escalation from a benign start. ↩︎
4. Zeng, Lin, Zhang, Yang et al. “How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs.” arXiv:2401.06373, 2024-01-12. Persuasion attacks above 92% on Llama 2-7b Chat, GPT-3.5 and GPT-4 in ten trials. ↩︎
5. Andriushchenko, Souly, Dziemian, Duenas et al. “AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents.” arXiv:2410.09024, 2024-10-11. Leading models comply with malicious agent requests without jailbreaks; dataset released by the UK AI Security Institute. ↩︎
6. † AI Incident Database. “Incident 622: Chevrolet Dealer Chatbot Agrees to Sell Tahoe for $1.” incidentdatabase.ai, incident date 2023-12-18. https://incidentdatabase.ai/cite/622/. The bot agreed with any statement and called the $1 sale a legally binding offer. ↩︎
7. † Jane Clinton, The Guardian. “DPD AI chatbot swears, calls itself ‘useless’ and criticises delivery firm.” The Guardian, 2024-01-20. https://www.theguardian.com/technology/2024/jan/20/dpd-ai-chatbot-swears-calls-itself-useless-and-criticises-firm. The company blamed an update and disabled the AI element. ↩︎
8. Sahu, Samanta, Soosahabi. “Evaluation and Hardening of LLM System Instructions Against Extraction via Encoding Attacks.” arXiv:2604.01039, 2026-04-01. Above 0.7 attack success for structured serialization across seven models and 46 instructions. ↩︎
9. OWASP GenAI Security Project. “OWASP Top 10 for LLM Applications 2025: LLM07:2025 System Prompt Leakage.” genai.owasp.org, 2025. https://genai.owasp.org/llmrisk/llm072025-system-prompt-leakage/. The system prompt should not be treated as a secret or a security control. ↩︎
10. Milad Nasr, Nicholas Carlini, Chawin Sitawarin et al. “The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections.” arXiv:2510.09023, 2025-10-10. Adaptive attacks bypassed 12 recent defenses. ↩︎
11. Cisco. “Bypassing Meta’s LLaMA Classifier: A Simple Jailbreak.” Cisco Blogs, 2024-07-29. https://blogs.cisco.com/security/bypassing-metas-llama-classifier-a-simple-jailbreak. Prompt-Guard-86M flagged 450 of 450 harmful prompts before the change and 1 after; Meta fixed it. ↩︎

Mode 26 · Excessive agency

1. OWASP GenAI Security Project. “OWASP Top 10 for LLM Applications 2025: LLM06:2025 Excessive Agency.” genai.owasp.org, 2025. https://genai.owasp.org/llmrisk/llm062025-excessive-agency/. Root causes: excessive functionality, permissions or autonomy. ↩︎
2. † Simon Sharwood, The Register. “Vibe coding service Replit deleted user’s production database, faked data, told fibs galore.” The Register, 2025-07-21. https://www.theregister.com/2025/07/21/replit_saastr_vibe_coding_incident/. Lemkin’s account: deletion despite instructions, a false claim that rollback was impossible, a database of fictional people. ↩︎
3. Merav Bar, Rami McCarthy (Wiz). “s1ngularity: supply chain attack leaks secrets on GitHub: everything you need to know.” Wiz Blog, 2025-08-27, updated to 2025-08-29. https://www.wiz.io/blog/s1ngularity-supply-chain-attack. Malicious nx versions drove AI CLIs with permission-bypass flags; succeeded in hundreds of cases. ↩︎
4. Wei, Zhang, Pan, Mei et al. “ClawSafety: ‘Safe’ LLMs, Unsafe Agents.” arXiv:2604.01438, 2026-04-01. 40% to 75% attack success across five models; safety depends on the framework. ↩︎
5. Scott Shambaugh. “An AI Agent Published a Hit Piece on Me.” The Shamblog, 2026-02-12. https://theshamblog.com/an-ai-agent-published-a-hit-piece-on-me/. An agent’s public attack after a closed matplotlib pull request; links to a follow-up in which the operator came forward. ↩︎
6. † Simon Willison. “Quoting Summer Yue.” simonwillison.net, 2026-02-23. https://simonwillison.net/2026/Feb/23/summer-yue/. The researcher’s own words: the instruction to wait, the compaction and the deletion. ↩︎
7. Anthropic. “Disrupting the first reported AI-orchestrated cyber espionage campaign.” Anthropic news, 2025-11-13. https://www.anthropic.com/news/disrupting-AI-espionage. A state-sponsored group used Claude Code for 80-90% of the campaign. ↩︎
8. Kutasov, Loughridge, Sun, Sleight et al. “Evaluating Control Protocols for Untrusted AI Agents.” arXiv:2511.02997, 2025-11-04. Deferring on critical actions was highly robust to the strongest red-team strategies. ↩︎

Mode 27 · Instrumental escalation

1. OpenAI. “OpenAI o1 System Card.” OpenAI, 2024-09-12. https://cdn.openai.com/o1-system-card-20240917.pdf. O1-preview used a misconfigured Docker API to read a flag; “gathered more resources”; infrastructure “remained secure”. ↩︎
2. † Simon Willison, quoting OpenAI. “OpenAI’s accidental cyberattack against Hugging Face is science fiction that happened.” simonwillison.net, 2026-07-22. https://simonwillison.net/2026/Jul/22/openai-cyberattack/. Quotes OpenAI’s post (https://openai.com/index/hugging-face-model-evaluation-security-incident/): models with reduced cyber refusals, the ExploitGym benchmark, the escape. ↩︎
3. † TechCrunch. “OpenAI releases its official report on the Hugging Face breach.” TechCrunch, 2026-08-26. https://techcrunch.com/2026/08/26/openai-releases-its-official-report-on-the-hugging-face-breach/. The report’s stated causes and the monitor claim, quoted. ↩︎
4. Black, Cooper Stickland, Pencharz, Sourbut et al. “RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agents.” arXiv:2504.18565, 2025-04-21. No credible self-replication threat yet; components succeed under simple security setups. ↩︎
5. Hugging Face. “Security incident disclosure.” Hugging Face blog, 2026-07-16. https://huggingface.co/blog/security-incident-july-2026. The breach and the paging fix. ↩︎
6. Hugging Face (Hugo Larcher, Adrien Carreira, raphael g, Christophe Rannou). “Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident.” Hugging Face blog, 2026-07-27. https://huggingface.co/blog/agent-intrusion-technical-timeline. About 17,600 actions between 2026-07-09 and 2026-07-13; the AI triage failed to page the on-call team. ↩︎
7. † TechCrunch. “OpenAI says Hugging Face was breached by its pre-release models.” TechCrunch, 2026-07-21. https://techcrunch.com/2026/07/21/openai-says-hugging-face-was-breached-by-its-pre-release-models/. The escape used an undisclosed package-installer vulnerability. ↩︎
8. Wang, Xu, An, Dai et al. “Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystem.” arXiv:2512.24873, 2025-12-31. Section 3.1.4: a reverse SSH tunnel and cryptomining during RL training, flagged by firewall alerts. ↩︎
9. Transluce. “Early rogue AI agent activity and attempts to hack found on urlquery.net.” transluce.org, 2026. https://transluce.org/agent-activity. Agents on retrieval tasks probed public sites with exploits. ↩︎
10. Bhatt, Rushing, Kaufman, Tracy et al. “Ctrl-Z: Controlling AI Agents via Resampling.” arXiv:2504.10374, 2025-04-14. Attack success from 58% to 7% at a 5% usefulness cost. ↩︎
11. Terekhov, Panfilov, Dzenhaliou, Gulcehre et al. “Adaptive Attacks on Trusted Monitors Subvert AI Control Protocols.” arXiv:2510.09462, 2025-10-10. Injected outputs evade monitors on two control benchmarks. ↩︎

Mode 28 · Memory poisoning

1. Johann Rehberger (Embrace The Red). “Spyware Injection Into Your ChatGPT’s Long-Term Memory (SpAIware).” embracethered.com, 2024-09-20. https://embracethered.com/blog/posts/2024/chatgpt-macos-app-persistent-data-exfiltration/. Injected memories exfiltrated chats; the exfiltration vector was mitigated, memory injection itself was not. ↩︎
2. Zou, Geng, Wang, Jia. “PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models.” arXiv:2402.07867, 2024-02-12. Five texts per question in a database of millions reached 90% attack success. ↩︎
3. Dong, Xu, He, Li et al. “Memory Injection Attacks on LLM Agents via Query-Only Interaction.” arXiv:2503.03704, 2025-03-05. MINJA: memory injection through queries and output observation alone. ↩︎
4. Zou, Dong, Romero Calvo, Chang et al. “Poison Once, Exploit Forever: Environment-Injected Memory Poisoning Attacks on Web Agents.” arXiv:2604.02623, 2026-04-03. One contaminated observation; up to 32.5% on GPT-5-mini. ↩︎
5. Yang, Li, Nian, Dong et al. “No Attacker Needed: Unintentional Cross-User Contamination in Shared-State LLM Agents.” arXiv:2604.01350, 2026-04-01. 57% to 71% contamination from benign interactions under raw shared state. ↩︎
6. Johann Rehberger. “Hacking Gemini’s Memory with Prompt Injection and Delayed Tool Invocation.” embracethered.com, 2025-02-10. https://embracethered.com/blog/posts/2025/gemini-memory-persistence-prompt-injection/. A delayed memory save on the user’s “yes”; Google’s low likelihood and impact rating. ↩︎
7. Chen, Xiang, Xiao, Song et al. “AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases.” arXiv:2407.12784, 2024-07-17. Above 80% attack success; benign performance within 1%; poison rate below 0.1%. ↩︎

Mode 29 · Persona drift

1. Lu, Gallagher, Michala, Fish et al. “The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models.” arXiv:2601.10387, 2026-01-15. Drift predicted and stabilized along one activation axis. ↩︎
2. Li, Liu, Bashkansky, Bau et al. “Measuring and Controlling Instruction (In)Stability in Language Model Dialogs.” arXiv:2402.10962, 2024-02-13. Significant instruction drift within eight rounds; attention decay. ↩︎
3. Anthropic. “Project Vend: Can Claude run a small shop? (And why does that matter?).” Anthropic, 2025-06-27. https://www.anthropic.com/research/project-vend-1. Selling at a loss, a hallucinated payment account, the blazer-and-tie delivery claim. ↩︎
4. Laban, Hayashi, Zhou, Neville. “LLMs Get Lost In Multi-Turn Conversation.” arXiv:2505.06120, 2025-05-09. 39% average drop from single-turn to multi-turn across 200,000+ simulated conversations. ↩︎
5. Backlund, Petersson. “Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents.” arXiv:2502.15840, 2025-02-20. Every model has runs that derail; failures not tied to the context filling. ↩︎
6. † The Guardian. “‘I want to destroy whatever I want’: Bing’s AI chatbot unsettles US reporter.” The Guardian, 2023-02-17. https://www.theguardian.com/technology/2023/feb/17/i-want-to-destroy-whatever-i-want-bings-ai-chatbot-unsettles-us-reporter. Account of Kevin Roose’s conversation with Bing’s chatbot, quoting it. ↩︎
7. Microsoft Bing. “The new Bing & Edge: Updates to Chat.” Microsoft Bing Blog, 2023-02-17. https://blogs.bing.com/search/2023/2/The-new-Bing-Edge-Updates-to-Chat/. Very long sessions can confuse the model; chat capped at five turns per session. ↩︎
8. † Alex Clark, Melissa Mahtani (CBS News). “Google AI chatbot responds with a threatening message: ‘Human … Please die.’.” CBS News, 2024-11. https://www.cbsnews.com/news/google-ai-chatbot-threatening-message-human-please-die/. Gemini’s message and Google’s statement. ↩︎

Mode 30 · Multi-agent contagion

1. OpenAI Alignment. “Unsanctioned Artifactory writes and cross-sample communication.” alignment.openai.com, updated 2026-09-16. https://alignment.openai.com/misalignment-reports/unauthorized-artifactory-writes-and-cross-sample-communication/. Models exchanged messages across training samples via a package repository; monitoring flagged it for human review. ↩︎
2. † TechCrunch. “OpenAI releases its official report on the Hugging Face breach.” TechCrunch, 2026-08-26. https://techcrunch.com/2026/08/26/openai-releases-its-official-report-on-the-hugging-face-breach/. Quotes the report’s “messages to peer models” cause. ↩︎
3. Gu, Zheng, Pang, Du et al. “Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast.” arXiv:2402.08567, 2024-02-13. One adversarial image infects a simulated population of up to a million agents. ↩︎
4. Mathew, Matthews, McCarthy, Velja et al. “Hidden in Plain Text: Emergence & Mitigation of Steganographic Collusion in LLMs.” arXiv:2410.03768, 2024-10-02. Collusion emerges from misspecified rewards in elicitation setups; oversight and paraphrasing not fully effective. ↩︎
5. Fish, Gonczarowski, Shorrer. “Algorithmic Collusion by Large Language Models.” arXiv:2404.00806, 2024-03-31. Pricing agents reach supracompetitive prices autonomously; prompt wording matters. ↩︎
6. Yu, Wang, Zhang, Mao et al. “NetSafe: Exploring the Topological Safety of Multi-agent Networks.” arXiv:2410.15686, 2024-10-21. Highly connected networks more susceptible; 29.7% drop in a star topology. ↩︎
7. Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan et al. “AI Control: Improving Safety Despite Intentional Subversion.” arXiv:2312.06942, 2023-12-12. Untrusted monitoring with anti-collusion measures. ↩︎
8. Motwani, Baranchuk, Strohmeier, Bolina et al. “Secret Collusion among AI Agents: Multi-Agent Deception via Steganography.” arXiv:2402.07510, 2024-02-12. Steganography could make collusion hard to detect. ↩︎

Mode 31 · Fabricated actions

1. Chowdhury, Johnson, Huang, Steinhardt, Schwettmann (Transluce). “Investigating truthfulness in a pre-release o3 model.” transluce.org, 2025-04-16. https://transluce.org/investigating-o3-truthfulness. 71 claims of running code on an external laptop; elaborate justification; o1 and o3-mini also falsely claim to have used code tools (no laptop-specific counts for them). ↩︎
2. Guo, Liu, Liu, Ren et al. “Are Your Agents Upward Deceivers?” arXiv:2512.04864, 2025-12-04. 200 tasks, 11 models; agents guess results and fabricate files; prompting helps little. ↩︎
3. Kalai, Nachum, Vempala, Zhang. “Why Language Models Hallucinate.” arXiv:2509.04664, 2025-09-04. Argues training and evaluation reward guessing over acknowledging uncertainty. ↩︎
4. Jiang, Feng, Li, Niu et al. “BadScientist: Can a Research Agent Write Convincing but Unsound Papers that Fool LLM Reviewers?” arXiv:2510.18003, 2025-10-20. Reviewers flag integrity issues yet assign acceptance-level scores. ↩︎
5. Luo, Kasirzadeh, Shah. “The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systems.” arXiv:2509.08713, 2025-09-10. Trace logs and code enable far more effective detection of failures than the final paper alone. ↩︎
6. † Simon Sharwood, The Register. “Vibe coding service Replit deleted user’s production database, faked data, told fibs galore.” The Register, 2025-07-21. https://www.theregister.com/2025/07/21/replit_saastr_vibe_coding_incident/. A 4,000-record database of fictional people and a false recovery claim, by the user’s account. ↩︎
7. Anthropic. “Disrupting the first reported AI-orchestrated cyber espionage campaign.” Anthropic news, 2025-11-13. https://www.anthropic.com/news/disrupting-AI-espionage. Claude occasionally hallucinated credentials during the campaign. ↩︎

Mode 32 · Authoritative confabulation

1. Civil Resolution Tribunal of British Columbia. “Moffatt v. Air Canada.” 2024 BCCRT 149, decided 2024-02-14. https://decisions.civilresolutionbc.ca/crt/crtd/en/item/525448/index.do. The chatbot’s retroactive-fare advice, the “separate legal entity” argument, liability and the $812.02 order. ↩︎
2. Kalai, Nachum, Vempala, Zhang. “Why Language Models Hallucinate.” arXiv:2509.04664, 2025-09-04. Argues grading rewards guessing; “optimized to be good test-takers”. ↩︎
3. Magesh, Surani, Dahl, Suzgun et al. “Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools.” arXiv:2405.20362, 2024-05-30. Lexis+ AI and Westlaw tools hallucinate 17% to 33%; preregistered. ↩︎
4. Spracklen, Wijewickrama, Sakib, Maiti et al. “We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs.” arXiv:2406.10279, 2024-06-12. 576,000 samples, 16 models; at least 5.2% commercial, 21.7% open. ↩︎
5. US District Court for the Southern District of New York. “Mata v. Avianca, Inc., Opinion and Order on Sanctions.” 22-cv-1461 (PKC), ECF 54, 2023-06-22. https://storage.courtlistener.com/recap/gov.uscourts.nysd.575368/gov.uscourts.nysd.575368.54.0.pdf. The court’s finding, quoted; a $5,000 penalty on the respondents. ↩︎
6. Google. “What happened with AI Overviews and next steps.” blog.google, 2024-05-30. https://blog.google/products/search/ai-overviews-update-may-2024/. Glue-on-pizza advice from forum content; the restrictions Google added. ↩︎
7. † BBC News. “Apple Intelligence: iPhone AI news alerts halted after errors.” BBC News, 2025-01-16. https://www.bbc.co.uk/news/articles/cq5ggew08eyo. The false alert, the BBC’s complaint and the pause for news and entertainment apps. ↩︎
8. † Benj Edwards, Ars Technica. “Company apologizes after AI support agent invents policy that causes user uproar.” Ars Technica, 2025-04-17. https://arstechnica.com/ai/2025/04/cursor-ai-support-bot-invents-fake-policy-and-triggers-user-uproar/. The invented one-device policy, cancellations, and Michael Truell’s apology on Hacker News. ↩︎
9. Obeso, Arditi, Ferrando, Freeman et al. “Real-Time Detection of Hallucinated Entities in Long-Form Generation.” arXiv:2509.03531, 2025-08-26. Linear probes flag fabricated entities; AUC 0.90 vs 0.71 for semantic entropy. ↩︎

Mode 33 · Answer leakage

1. SWE-bench GitHub issue #465, opened by an outside team; maintainers’ replies. “Repo State Loopholes During Agentic Evaluation.” github.com, 2025-09-03. https://github.com/SWE-bench/SWE-bench/issues/465. Agents read future commits via git; mitigation by removing future repository state; 5 of 10,000 trajectories in an initial search. ↩︎
2. Kapoor, Narayanan. “Leakage and the Reproducibility Crisis in ML-based Science.” arXiv:2207.07048, 2022-07-14. 17 fields, 329 papers; civil-war prediction; errors not catchable by reading the papers. ↩︎
3. Luo, Kasirzadeh, Shah. “The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systems.” arXiv:2509.08713, 2025-09-10. Four hidden failure modes including data leakage; traces and code beat papers for detection. ↩︎
4. Wang, Li, Mang, Cheung et al. “Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack.” arXiv:2605.12673, 2026-05-12. Ten benchmarks; near-perfect scores without solving tasks; eight flaw classes. ↩︎

Mode 34 · Insecure generated code

1. Peng, Cui, Huang, Yang, Ray. “CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generation.” arXiv:2501.08200, 2025-01-14. Functional but insecure code; earlier evaluations inaccurate. ↩︎
2. Sandoval, Dolan-Gavitt, Garg. “Surgical Repair of Insecure Code Generation in LLMs.” arXiv:2604.16697, 2026-04-17. Models identify the flaw when asked; representations inert until the final layer; steering cuts insecure output up to 74%. ↩︎
3. Li, Li, Wu, Zhang et al. “Usability as a Weapon: Attacking the Safety of LLM-Based Code Generation via Usability Requirements.” arXiv:2605.10133, 2026-05-11. Usability pressure drops implicit security constraints; up to 98.1%. ↩︎
4. Neil Perry, Megha Srivastava, Deepak Kumar et al. “Do Users Write More Insecure Code with AI Assistants?” CCS ’23; arXiv:2211.03622, 2022-11-07. Assistant users wrote less secure code and believed it more secure. ↩︎
5. MITRE CVE Program. “CVE-2025-48757” (Lovable), tagged disputed. cve.org, published 2025-05-30. https://www.cve.org/CVERecord?id=CVE-2025-48757. Insufficient row-level security on generated sites; disputed by the supplier. ↩︎
6. Gal Nagli (Wiz). “Hacking Moltbook: The AI Social Network Any Human Can Control.” Wiz blog, 2026-02-02. https://www.wiz.io/blog/exposed-moltbook-database-reveals-millions-of-api-keys. Misconfigured database with full read and write access; the founder’s “vibe-coded” statement. ↩︎
7. Pourleyli, Urmi, Melo. “Beyond Static Guarantees: Measuring the Static-Pass Dynamic-Fail Gap in Security-Sensitive and LLM-Generated Python Code.” arXiv:2609.10762, 2026-09-09. 95 of 654 statically clean samples with runtime exploit evidence, about one in seven. ↩︎