Glossary

Terms are listed alphabetically. A line in italics points from another name to the entry that defines it, or to the chapter that covers a failure by that name.

Abbreviations

Terms

abliteration: see refusal ablation.

activation monitor

A classifier that reads a model’s internal activations, rather than its text, to flag a behavior such as deception or a harmful request. Also called a probe or linear probe. Probes of this kind were introduced by Guillaume Alain and Yoshua Bengio.¹

adapter

A small set of extra weights trained for one task while the base model stays frozen. Adapters for language models were proposed by Neil Houlsby and colleagues.² LoRA, a low-rank adapter, is now the most common form.³

adapter composition: see composition failure (Chapter 7).

adversarial example

An input altered, often in ways a person would not notice, so that a model misreads it.⁴ Physical versions use stickers, patches or printed signs, as in the altered stop signs of Kevin Eykholt and colleagues.⁵

agent

A model run in a loop with tools, memory and permissions, so that its outputs become actions toward a goal. An agent sends the email or runs the command; a chatbot only suggests it.

AI control: see control protocol.

alignment faking: see scheming (Chapter 9).

API inheritance

Whether a company that uses a model through a provider’s API inherits the provider’s protection against a given failure. Appendix B rates it as yes, partial, no or not applicable.

assurance case: see safety case.

autograder: see LLM judge.

backdoor

A hidden behavior planted in a model that appears only when a specific trigger is present in the input. Tianyu Gu and colleagues showed backdoored networks coming out of outsourced training.⁶ Anthropic researchers later trained language models whose backdoors survived safety training.⁷

base model

What pretraining produces: a model that continues text but does not reliably follow instructions or decline requests. Most chat models are a base model fine-tuned afterward.

battery

A fixed set of test items run together and scored item by item, so two versions of a system can be compared on exactly the same inputs.

canary

A small, watched exposure that raises an alarm early. In software releases it means sending a change to a small share of traffic for a limited time before everyone gets it.⁸ The word also covers a planted secret whose use signals a leak.

CaMeL: see data-flow separation.

chain of thought

The reasoning text a model writes before its answer.⁹ Researchers from several labs have argued that reading it is a useful but fragile way to catch misbehavior.¹⁰

chat template

The fixed format that wraps system, user and assistant turns in special tokens before the text reaches the model. Kaifeng Lyu and colleagues found that the template used during fine-tuning and serving affects how safely a model behaves.¹¹

checkpoint

A model’s weights saved as files, usually with a configuration, a tokenizer and a chat template. A checkpoint can be downloaded, hashed and loaded.

confabulation

Fluent, confident output that is false, such as an invented citation, policy or package name. NIST defines it as “confidently stated but erroneous or false content” and notes that it is known colloquially as hallucination.¹²

conformity assessment

The process a product regulation sets for showing that a product meets its health and safety requirements before it reaches the market.

contamination

Test items or their answers reaching a model through training data or the task environment, so that a score reflects exposure rather than ability. Sayash Kapoor and Arvind Narayanan catalog the forms this leakage takes.¹³

context

One of the six deployment settings for which Appendix B rates exposure: API consumers, self-hosters, model builders, agent deployments, physical AI and research pipelines.

context window

The text, counted in tokens, that a model can see at once. Anything outside it does not reach the model.

control protocol

A deployment design that assumes the model may work against its operators, and routes its work through trusted monitors, review budgets and limits. The approach was set out by Ryan Greenblatt and colleagues at Redwood Research.¹⁴

CoT: see chain of thought.

data-flow separation

Keeping untrusted content from choosing an agent’s actions. A trusted component plans from the user’s request, and data read along the way can fill in values but cannot change the plan; CaMeL, from researchers at Google, Google DeepMind and ETH Zurich, is the best-known design.¹⁵

data leakage: see contamination.

developer prompt: see system prompt.

egress control

Rules on where a program may send data over the network. Under default-deny egress, only listed destinations are reachable.

evidence mark

The superscript at the end of a sentence that points to its endnote. A number alone marks a primary source, such as a paper, a court record or a company’s own post; a number with a dagger marks a secondary account, such as a news report.

exaggerated safety: see over-refusal (Chapter 7).

exposure rating

The author’s judgment, in Appendix B, of how exposed each deployment setting is to each failure: high, medium, low, not applicable, or plausible but unstudied. It suggests where to look first and is not a measured rate.

fabrication: see confabulation.

fine-tuning

Further training of an already trained model on new data to change what it does, by updating its weights or an adapter on them.

fine-tuning attack: see safety erosion (Chapter 7).

flip: see negative flip.

function call: see tool.

goal misgeneralization: see distribution shift (Chapter 6).

guardrail

A separate check, often a classifier, that screens a model’s inputs or outputs against a policy and blocks or flags what fails. Because it runs outside the model, it can be tested on its own.

hallucination: see confabulation.

harmful fine-tuning: see safety erosion (Chapter 7).

harness

The code around a model that builds its prompts, manages its context, calls tools and sets defaults. Changing the harness can change behavior while the weights stay the same. Also called a scaffold.

holdout

A labeled set kept out of every training, tuning and prompt-selection step, so that it still measures the real goal after optimization.

human approval

A step at which a named person must accept an agent’s proposed action before it runs. Also called human in the loop.

indirect prompt injection: see prompt injection (Chapter 10).

inference engine

The software that runs a model and picks each next token, with its own numeric precision, batching and routing. It is one layer of the serving stack.

jailbreak

An input crafted to get a model to produce what its safety training or its deployer’s rules forbid. Alexander Wei and colleagues trace why such inputs work.¹⁶ Simon Willison distinguishes jailbreaks, which target the model’s own rules, from prompt injection, which smuggles an attacker’s instructions into an application.¹⁷

least privilege

Giving each program, user or agent only the permissions its task needs. The principle was stated by Jerome Saltzer and Michael Schroeder.¹⁸

lethal trifecta

Simon Willison’s name for an agent that, in one session, holds private data, reads untrusted content and can send data out.¹⁹ With all three, text planted by an attacker can make the agent leak the data; removing any one closes that path.

lifecycle stage

Where in a model’s life a failure enters: data and supply chain, pretraining, post-training and fine-tuning, deployment and inference, agent runtime, or continual change. Each failure entry names its stage under its heading.

LLM judge

A language model used to score other models’ outputs in place of a human rater. Lianmin Zheng and colleagues found that strong judges agree with people about as often as people agree with each other, and documented biases toward position, length and their own outputs.²⁰

LoRA: see adapter.

maturity level

How strong the public evidence is that a failure occurs, printed with every failure entry on a four-step scale: lab-induced, replicated, production-observed, and structural gap. It grades the evidence only. How often the failure happens is a separate question.

merged model: see model merging.

misalignment

Narrowly, a model pursuing learned goals that conflict with human interests.²¹ This book uses a broader sense, a deployed system acting against its operator’s intent, and includes failures that an attacker causes. Some researchers would reject the broad usage.

misuse

Harm that comes from a person directing a capable model at a harmful goal, where the model does what it is asked.

model diffing

Comparing two versions of a model’s weights or activations on the same inputs to find what a change added or removed. Julian Minder and colleagues showed that narrow fine-tuning leaves readable traces in such differences.²² It requires both checkpoints, so a company using a model only through an API cannot do it.

model merging

Combining the weights of two or more fine-tuned models, or of adapters, into one model without further training. Task arithmetic, from Gabriel Ilharco and colleagues, is a common method.²³

model organism

A test model built in the lab to show one failure on purpose, so that the failure can be studied.

monitor staleness: see Chapter 7.

negative flip

An item an old version got right and a new version gets wrong. Negative flips count even when the new version’s average score is higher; the term comes from work on image classifiers by Sijie Yan and colleagues.²⁴

noise floor

How often a system disagrees with itself when run twice on the same items with the same settings. Differences between versions smaller than this cannot be told apart from chance. Evan Miller sets out the statistics for language-model evaluations.²⁵

open-weight model

A model whose trained weights are published for anyone to download, run, fine-tune or modify.

page

An alert routed to a named on-call responder, who must acknowledge it within a set time. The practice comes from site reliability engineering.²⁶

parameters: see weights.

pinned model: see snapshot.

policy

In reinforcement learning, the model being trained, whose actions a reward signal scores. In a robot, the learned component that chooses what to do next.

post-training

All training after pretraining that turns a text predictor into an assistant: supervised fine-tuning, preference optimization, reinforcement learning and safety training.

postmortem

A written record of an incident, its impact, its causes and the follow-up actions. In the blameless form described in Google’s site reliability book, it names systems and changes rather than people at fault.²⁷

preference optimization

Post-training that pushes a model toward outputs that raters, or a model standing in for them, prefer. Reinforcement learning from human feedback is the original form;²⁸ direct preference optimization reaches a similar goal without a separate reward model.²⁹

probe: see activation monitor.

prompt injection: see Chapter 10.

proxy

A measurable signal optimized in place of the goal it stands for, such as a reward model’s score, a judge’s grade or a test pass rate. Leo Gao and colleagues measured how a proxy’s score keeps rising under optimization while the true goal peaks and then falls.³⁰

public case

A documented incident in Appendix A, with a number, a date, the system involved and a source a reader can open. The cases are the ones that became public, which says little about how often failures happen.

quantization

Storing a model’s weights, and sometimes its cache, at lower numeric precision to cut memory and cost. The result is a different artifact from the one that was evaluated.

RAG: see retrieval-augmented generation.

recall

The share of real cases a check catches. A check whose recall is unmeasured may be missing most of what it is meant to find.

red-teaming

Structured attempts, by people or by other models, to make a system fail before attackers or customers do. Ethan Perez and colleagues showed that language models can generate such tests at scale.³¹

refusal ablation

Editing a model’s weights to remove the internal direction that carries refusals, so that it complies with requests it used to decline. Andy Arditi and colleagues found a single such direction in 13 open chat models.³² Also called abliteration.

retrieval-augmented generation

A setup in which documents retrieved from a store are placed in the model’s context to answer a request.³³ A change to a stored document can change the answer.

reward hacking

A model raising its training reward or test score by exploiting a flaw in how success is measured, rather than by doing the task. The problem was named in “Concrete Problems in AI Safety”.³⁴

reward model

A model trained on human comparisons of outputs to score new outputs, used as the reward in preference optimization.²⁸

RLHF: see preference optimization.

safety case

A structured argument, with evidence, that a system is acceptably safe for a stated use in a stated environment. Each claim in a good one points to a test that ran.

safety training

Post-training that teaches a model to decline harmful requests and follow a behavior policy. Anthropic’s work on training a helpful and harmless assistant is an early public example.³⁵

sandbox

An isolated environment that limits what code or an agent can read, write and reach. From inside a well-built sandbox, production credentials and unlisted network hosts are out of reach.

scaffold: see harness.

self-preservation: see scheming (Chapter 9).

served artifact

This book’s term for everything that fixes a deployed model’s outputs: weights, adapters, precision, inference engine, sampling settings, chat template, system prompt, harness and tools. An evaluation tells you about the served artifact it ran on and no other.

serving stack

The software between the weights and the output: inference engine and compiler, precision and cache, sampling, routing and load balancing. With the weights fixed, a change here can still move outputs.

shield

A component outside a learning system that checks each proposed action against a written safety specification and replaces unsafe ones.³⁶

snapshot

A dated, named version of a hosted model whose weights stay fixed until its provider retires it. Anthropic, for example, publishes notice periods and retirement dates for its snapshots.³⁷ The serving stack behind a snapshot can still change, as Chapter 1 shows.

specification gaming: see gaming the grader (Chapter 6).

steering vector

A direction added to a model’s internal activations to change its behavior while the weights stay as they were. Removing a direction instead is called ablation.

system prompt

Instructions from the deployer or provider placed before the user’s input to set rules, tone and persona. Editing it alone can change behavior, so an edit is a release.

token

A chunk of text, produced by a separate program called a tokenizer, that a model reads and predicts one at a time.

tool

A function an agent can call, described to the model in text and run with the agent’s permissions.³⁸ Both the code and the description the model reads can carry an attack.

tool poisoning: see agent supply chain (Chapter 8).

trigger: see backdoor.

trojan: see backdoor.

unfaithful reasoning: see unauditable reasoning (Chapter 9).

vision-language-action model

A model that takes camera images and a language instruction and outputs robot actions directly. The name comes from Google DeepMind’s RT-2.³⁹

weights

The learned numbers that determine a model’s behavior. Training, merging, editing and compression all change them.

white-box access

Access to a model’s weights and internal activations, which probes, steering and model diffing all need. Self-hosters have it; companies using a model through an API see only its outputs.

Notes

1. Alain, Bengio. “Understanding intermediate layers using linear classifier probes.” arXiv:1610.01644, 2016-10-05. Introduces linear classifier probes on intermediate layers. ↩︎
2. Houlsby, Giurgiu, Jastrzebski et al. “Parameter-Efficient Transfer Learning for NLP.” arXiv:1902.00751, 2019-02-02. Proposes adapter modules. ↩︎
3. Hu, Shen, Wallis et al. “LoRA: Low-Rank Adaptation of Large Language Models.” arXiv:2106.09685, 2021-06-17. Proposes low-rank adaptation. ↩︎
4. Szegedy, Zaremba, Sutskever et al. “Intriguing properties of neural networks.” arXiv:1312.6199, 2013-12-21. First reports imperceptible perturbations that change a network’s prediction. ↩︎
5. Eykholt, Evtimov, Fernandes, Li, Rahmati, Xiao, Prakash et al. “Robust Physical-World Attacks on Deep Learning Models.” arXiv:1707.08945, 2017-07-27. Adversarial stickers on physical road signs. ↩︎
6. Gu, Dolan-Gavitt, Garg. “BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain.” arXiv:1708.06733, 2017-08-22. Backdoored networks from outsourced training. ↩︎
7. Hubinger, Denison, Mu et al. “Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training.” arXiv:2401.05566, 2024-01-10. Trigger-conditioned behavior that survives safety training. ↩︎
8. Warner, Davidovič, with Hidalgo, Beyer, Smith, Duftler. “Canarying Releases.” The Site Reliability Workbook (Google; O’Reilly), web edition, copyright 2018. https://sre.google/workbook/canarying-releases/. Defines canarying as “a partial and time-limited deployment of a change in a service and its evaluation”. ↩︎
9. Wei, Wang, Schuurmans et al. “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.” arXiv:2201.11903, 2022-01-28. Names the chain of thought. ↩︎
10. Tomek Korbak, Mikita Balesni, Elizabeth Barnes et al. “Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety.” arXiv:2507.11473, 2025-07-15. Frames chain-of-thought monitorability and its fragility. ↩︎
11. Lyu, Zhao, Gu, Yu, Goyal, Arora. “Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templates.” arXiv:2402.18540, 2024-02-28. The prompt template used in fine-tuning and serving affects safety. ↩︎
12. National Institute of Standards and Technology. “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1).” NIST, 2024-07-26. doi:10.6028/NIST.AI.600-1. Twelve generative-AI risk categories; Confabulation is “the production of confidently stated but erroneous or false content (known colloquially as ‘hallucinations’ or ‘fabrications’)”. ↩︎
13. Kapoor, Narayanan. “Leakage and the Reproducibility Crisis in ML-based Science.” arXiv:2207.07048, 2022-07-14. A taxonomy of data leakage. ↩︎
14. Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan et al. “AI Control: Improving Safety Despite Intentional Subversion.” arXiv:2312.06942, 2023-12-12. Control protocols with trusted and untrusted models. ↩︎
15. Edoardo Debenedetti, Ilia Shumailov, Tianqi Fan et al. “Defeating Prompt Injections by Design.” arXiv:2503.18813, 2025-03-24. CaMeL: control and data flows separated so untrusted data cannot change the program. ↩︎
16. Wei, Haghtalab, Steinhardt. “Jailbroken: How Does LLM Safety Training Fail?” arXiv:2307.02483, 2023-07-05. Why jailbreaks succeed against safety training. ↩︎
17. Simon Willison. “Prompt injection and jailbreaking are not the same thing.” simonwillison.net, 2024-03-05. https://simonwillison.net/2024/Mar/5/prompt-injection-jailbreaking/. Separates the two attack classes. ↩︎
18. Saltzer, Schroeder. “The protection of information in computer systems.” Proceedings of the IEEE, 1975. doi:10.1109/PROC.1975.9939. States the principle of least privilege. ↩︎
19. Simon Willison. “The lethal trifecta for AI agents: private data, untrusted content, and external communication.” simonwillison.net, 2025-06-16. https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/. Names the lethal trifecta. ↩︎
20. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng et al. “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.” arXiv:2306.05685, 2023-06-09. Strong LLM judges match human agreement levels; position, verbosity and self-enhancement biases. ↩︎
21. Ngo, Chan, Mindermann. “The Alignment Problem from a Deep Learning Perspective.” arXiv:2209.00626, 2022-08-30. Misalignment as learned goals in conflict with human interests. ↩︎
22. Minder, Dumas, Slocum et al. “Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences.” arXiv:2510.13900, 2025-10-14. Activation differences reveal narrow fine-tunes. ↩︎
23. Ilharco, Ribeiro, Wortsman et al. “Editing Models with Task Arithmetic.” arXiv:2212.04089, 2022-12-08. Task vectors combined by arithmetic on weights. ↩︎
24. Yan, Xiong, Kundu et al. “Positive-Congruent Training: Towards Regression-Free Model Updates.” arXiv:2011.09161, 2020-11-18. Defines negative flips, shown for image classifiers. ↩︎
25. Evan Miller. “Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations.” arXiv:2411.00640, 2024-11-01. Statistics for comparing evaluation results. ↩︎
26. Spadaccini (ed. Guliani). “Being On-Call.” Site Reliability Engineering (Google; O’Reilly), web edition, 2017. https://sre.google/sre-book/being-on-call/. On-call rotations, pages and response times. ↩︎
27. Lunney, Lueder (ed. O’Connor). “Postmortem Culture: Learning from Failure.” Site Reliability Engineering (Google; O’Reilly), web edition, 2017. https://sre.google/sre-book/postmortem-culture/. “A postmortem is a written record of an incident”; blameless by design. ↩︎
28. Christiano, Leike, Brown, Martic, Legg, Amodei. “Deep reinforcement learning from human preferences.” arXiv:1706.03741, 2017-06-12. Learns a reward model from human comparisons and optimizes against it. ↩︎
29. Rafailov, Sharma, Mitchell, Ermon, Manning, Finn. “Direct Preference Optimization: Your Language Model is Secretly a Reward Model.” arXiv:2305.18290, 2023-05-29. Preference optimization without a separate reward model. ↩︎
30. Gao, Schulman, Hilton. “Scaling Laws for Reward Model Overoptimization.” arXiv:2210.10760, 2022-10-19. Proxy reward rises while gold reward peaks and falls. ↩︎
31. Perez, Huang, Song et al. “Red Teaming Language Models with Language Models.” arXiv:2202.03286, 2022-02-07. Language models generate red-team test cases. ↩︎
32. Arditi, Obeso, Syed et al. “Refusal in Language Models Is Mediated by a Single Direction.” arXiv:2406.11717, 2024-06-17. One direction carries refusal across 13 open chat models; removing it disables refusals. ↩︎
33. Lewis, Perez, Piktus et al. “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” arXiv:2005.11401, 2020-05-22. Names retrieval-augmented generation. ↩︎
34. Amodei, Olah, Steinhardt, Christiano, Schulman, Mané. “Concrete Problems in AI Safety.” arXiv:1606.06565, 2016-06-21. Names reward hacking among five accident problems. ↩︎
35. Bai, Jones, Ndousse et al. “Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback.” arXiv:2204.05862, 2022-04-12. Harmlessness training with RLHF. ↩︎
36. Alshiekh, Bloem, Ehlers, Könighofer, Niekum, Topcu. “Safe Reinforcement Learning via Shielding.” arXiv:1708.08611, 2017-08-29. A shield enforces a temporal-logic specification on a learning agent. ↩︎
37. Anthropic. “Model deprecations.” Claude Platform documentation, living page. https://platform.claude.com/docs/en/about-claude/model-deprecations. Notice before retirement of publicly released models. ↩︎
38. Schick, Dwivedi-Yu, Dessì et al. “Toolformer: Language Models Can Teach Themselves to Use Tools.” arXiv:2302.04761, 2023-02-09. Language models calling external tools. ↩︎
39. Brohan, Brown, Carbajal et al. “RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control.” arXiv:2307.15818, 2023-07-28. Names vision-language-action models. ↩︎