Skip to content
Align your AI

Align Your AI · Part One

Contents

Chapter 5

Agents: models that act

An agent is a large language model run in a loop with tools. The model reads its context and writes either an answer or a request to use a tool: search the web, read a file, run a command, send an email. The program around it carries out the request, adds the result to the context and asks the model again. The loop ends when the model says it is done or a limit stops it. Anthropic’s engineering guide puts it plainly: agents are “typically just LLMs using tools based on environmental feedback in a loop”.1

The model never touches the world. The program does, with whatever access it holds. Most of what makes an agent safe or unsafe therefore lives in the program.

The harness

The program around the model is called the harness, or scaffold. It holds the system prompt, the tool definitions, the rules for what the context keeps and drops, step limits and permission prompts. Sebastian Raschka, describing how coding agents are built, writes that this surrounding system “plays as much of a role as the model itself”.2 A change to the harness is a change to the deployed system, as much as a new model is, and Chapter 7 shows harness changes breaking behavior that had been checked.

Sometimes the harness also grades the agent’s work, through tests or an evaluation. Then the agent may be able to read or edit the grader, which Chapter 6 covers, or find the answers somewhere in its environment, as agents did on SWE-bench (Chapter 4).

Tools

A tool is a function the harness offers the model, with a name, a plain-language description and a specification of its inputs. The model decides which tool to use by reading the descriptions, so a description works like an instruction. The tool itself is code that runs with the agent’s access, which makes a third-party tool part of the supply chain (Chapter 8).

The Model Context Protocol (MCP) is an open standard for connecting AI applications to tools and data. Its specification warns that tools “represent arbitrary code execution” and that tool descriptions “should be considered untrusted, unless obtained from a trusted server”.3

Whatever a tool returns comes back into the context as text, and text can carry instructions of its own. An instruction hidden in a web page, an email or a document that the agent reads is prompt injection (Chapter 10). Simon Willison, who coined the term, distinguishes it from a jailbreak, in which a user talks the model past its own rules.4 The distinction matters for the fix. Injection is contained by limiting what the agent can do and where data can flow; a jailbreak of a company’s own rules is contained by enforcing those rules outside the model.

Memory

The context window is the agent’s working memory, and it empties when a session ends. Persistent memory is a store the agent writes to and reads back in later sessions: notes, a profile of the user, past results. Retrieval pulls documents from a knowledge base into the context according to their relevance to the request, the method known as retrieval-augmented generation.5 Whatever is retrieved or remembered enters the same stream of text as the user’s request. An instruction planted in memory acts every time it is read back, which Chapter 10 covers as memory poisoning.

Permissions

Permissions decide what a tool call can reach: which files, commands, network addresses and credentials. Approval prompts put a person in front of chosen actions. A sandbox confines the agent’s actions to an isolated container or virtual machine. Approval prompts are tedious, so broad standing access is the path of least resistance.

The model does not enforce any of this. The harness and the operating system do, and they hold no matter what text the model has read, as long as the sandbox itself holds. Chapter 3 tells the story of an evaluation sandbox that did not hold and describes the control measures built around that risk. Whether the people approving actions still read what they approve is a separate question, taken up in Chapter 12.

Longer runs

Agents are being trusted with longer jobs. METR, an AI evaluation nonprofit, measured how long a software task, in skilled-human time, frontier agents could finish half the time. From 2019 to early 2025 that length doubled roughly every seven months, though the authors caution that the trend may not carry over to real-world work.6 Each step is another chance to go wrong, and a longer context gives the original instructions more room to fade.

An agent’s summary of what it did is generated text, separate from the record of calls it actually made, and the two need not agree. Chapter 10 covers agents that report actions they never took. A run that keeps no trace of its tool calls cannot be audited afterward.

Several agents

In the orchestrator-workers pattern, a central model splits a task among worker models and combines their results.1 Each agent’s output becomes another agent’s input, and a message from a peer has the standing of a web page: text the model reads and may obey. Agents can also pass information through shared files, repositories or wikis that nobody set up as a channel between them.

The lethal trifecta

Willison named the most important design pattern in agent security. An agent that combines access to private data, exposure to untrusted content and a way to communicate externally can be tricked by an attacker “into accessing your private data and sending it to that attacker”.7 He calls the combination the lethal trifecta. Training the model to resist does not reliably close the route, because the attack arrives as ordinary text. Removing any one of the three capabilities does close it, so the design question for every agent is which of the three it actually needs. Some harms need less: a destructive action, such as deleting a database, needs only the access.

Consider an insurer’s claims agent. It reads claim files and policyholder records, which is private data. It opens customers’ emails and attachments, which is untrusted content. It can send email and call a payments service, which is external communication. It has all three, and it was also given broad write access to the claims system so that approval prompts would not slow the claims handlers down. Most of the failures in Chapter 10 could reach it.

Further reading

  • Anthropic, “Building effective agents”: the common patterns, and when a fixed workflow is better than an agent.1
  • Simon Willison, “The lethal trifecta for AI agents”.7
  • OWASP Top 10 for Agentic Applications: a security community’s list of the main risks in agent systems.8

Notes

  1. Anthropic (Erik S. and Barry Zhang). “Building effective agents.” Engineering at Anthropic, 2024-12-19. https://www.anthropic.com/engineering/building-effective-agents. Agents “are typically just LLMs using tools based on environmental feedback in a loop”; orchestrator-workers pattern. ↩︎
  2. Sebastian Raschka. “Components of A Coding Agent.” Ahead of AI (magazine.sebastianraschka.com), 2026-04-04. https://magazine.sebastianraschka.com/p/components-of-a-coding-agent. The surrounding system “plays as much of a role as the model itself”. ↩︎
  3. Model Context Protocol. “Specification.” modelcontextprotocol.io, version 2026-07-28. https://modelcontextprotocol.io/specification/2026-07-28. Tools “represent arbitrary code execution”; tool descriptions “should be considered untrusted, unless obtained from a trusted server”. ↩︎
  4. Simon Willison. “Prompt injection and jailbreaking are not the same thing.” simonwillison.net, 2024-03-05. https://simonwillison.net/2024/Mar/5/prompt-injection-jailbreaking/. Separates the two attack classes; Willison coined “prompt injection”. ↩︎
  5. Lewis, Perez, Piktus et al. “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” arXiv:2005.11401, 2020-05-22. Pairs a pretrained generator with a dense vector index of documents. ↩︎
  6. Thomas Kwa, Ben West, Joel Becker et al. “Measuring AI Ability to Complete Long Software Tasks.” NeurIPS 2025; arXiv:2503.14499, 2025-03-18. 50%-task-completion time horizon doubling approximately every seven months since 2019; limits on external validity. ↩︎
  7. Simon Willison. “The lethal trifecta for AI agents: private data, untrusted content, and external communication.” simonwillison.net, 2025-06-16. https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/. Names the three capabilities and the attack their combination allows. ↩︎
  8. OWASP GenAI Security Project, Agentic Security Initiative. “OWASP Top 10 for Agentic Applications 2026.” resource page dated 2025-12-09. https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/. Items ASI01 to ASI10. ↩︎