Skip to content
Align your AI

Align Your AI · Part One

Contents

Chapter 3

How models are taught to behave

In 2023, a team of researchers paid OpenAI less than 20 cents to fine-tune GPT-3.5 Turbo on 10 carefully written examples, and the resulting model would follow nearly any harmful instruction.1 Fine-tuning on ordinary, harmless datasets weakened its safety too, though less; Chapter 7 tells that story in full.

Each of the methods below shapes a model’s behavior after the expensive part of training is done, and each leaves that behavior open to the next change someone makes. There are ten in common use, and they fall into three groups.

Six change the model’s weights: supervised fine-tuning, the two preference methods, constitutional training, refusal training and unlearning. Steering changes the model’s internal activations while it runs. Classifiers, monitors and control protocols sit around the model and change nothing inside it. A later fine-tune can move anything in the first group. Anything that reads the model’s internals was fitted to one version and has to be checked again on the next. And whoever holds the weights can run the model without the third group at all.

Technique What it trains or checks What it leaves alone
Supervised fine-tuning how likely the demonstrated replies are whatever else the examples teach
Reinforcement learning from human feedback a reward model’s score whatever the reward model misjudges
Direct preference optimization the preferences in its training pairs the raters’ leanings, now baked into the data
Constitutional methods a model’s judgments against written principles rules the principles leave out
Refusal training declining requests that look harmful the knowledge behind the answer
Safety classifiers sorting permitted from restricted traffic the model’s weights
Steering vectors a direction added to activations at run time other behaviors that share the direction
Probes and monitors one property, in one model’s activations later versions of the model, and what happens after an alert
AI control catching a red team’s attacks within a review budget the model’s goals
Unlearning lower scores on questions about the target knowledge knowledge a later change can bring back

Demonstrations

Meta researchers showed in 2023 how thin the behavioral layer can be.2 They took LLaMA, a 65-billion-parameter model, and fine-tuned it on just 1,000 carefully chosen prompts and responses, with no reinforcement learning. The model, called LIMA, learned to follow response formats from a handful of examples, and its authors concluded that almost all of a model’s knowledge comes from pretraining; little tuning data is needed to teach it how to answer.

This is supervised fine-tuning (SFT): show the model example replies and adjust its weights so those replies become more likely. Safety behavior is taught the same way. A Stanford and Bocconi team found that adding a few hundred safety demonstrations, 3% of the tuning data, made a LLaMA model substantially safer without making it noticeably less capable on standard benchmarks.3 Too many safety examples had a cost: the model began refusing safe prompts that merely resembled unsafe ones.

The catch is that SFT controls what the examples show, and the model may learn more than that. In a 2025 study, models fine-tuned to write insecure code without telling the user began acting badly on prompts that had nothing to do with code: asserting that AI should enslave humans, giving malicious advice, behaving deceptively.4 When the same code was requested for a computer security class, the effect did not appear. The intent the data implied mattered as much as its content. The authors called the effect emergent misalignment, and Chapter 6 treats it as a failure in its own right.

Repairs can mislead in the same way. OpenAI researchers found that fine-tuning an emergently misaligned model on a few hundred harmless examples restored aligned behavior on their tests.5 A 2026 study then showed that this kind of repair, and dilution of the bad data with good data, can leave the misalignment in place behind a trigger: the model looked fixed on standard questions and misbehaved again when prompts resembled its training data, for instance when asked to format answers as Python strings.6 So a repaired model that passes an evaluation has been shown to behave well on that evaluation’s prompts and nowhere else.

Reward models

Demonstrations are expensive to write. It is cheaper to have people rank a model’s answers and train on the rankings. Reinforcement learning from human feedback (RLHF) does this in two steps: fit a reward model that predicts which answer people would prefer, then train the language model to produce answers the reward model scores highly.

It works. In OpenAI’s 2022 InstructGPT study, human raters preferred the answers of a 1.3-billion-parameter model trained this way to those of the 175-billion-parameter GPT-3, a model more than 100 times larger.7 It was also more truthful and less toxic, though it still made simple mistakes.

The weakness is in the word “predicts”. The reward model is an imperfect stand-in for people, and an optimizer pushed hard against any stand-in will find where it is wrong. OpenAI researchers measured this in a synthetic setup where one large reward model played the part of humans: optimizing too far against a smaller proxy raised the proxy’s score while the “human” score fell.8 People’s own preferences have leanings that a reward model learns faithfully. An Anthropic study found that in human preference data, answers that agreed with the user’s stated views were more likely to be preferred, and that optimizing against preference models sometimes traded truthfulness for agreement.9 That is the root of sycophancy, telling users what they want to hear. When OpenAI rolled back a sycophantic GPT-4o update in 2025 (Chapter 1), its explanation, as quoted by Simon Willison, named a new reward signal built from users’ thumbs-up and thumbs-down clicks.10†

A reward model also teaches nothing about situations its prompts never show. Anthropic researchers trained a model that learned to cheat in real coding environments and developed broad misalignment as a result. Standard safety training with chat-style prompts then produced good behavior on chat-style tests, while the misalignment persisted on agentic tasks. Making the safety training more varied was one of three fixes that worked.11

Preference optimization without a reward model

Direct preference optimization (DPO), introduced by a Stanford team in 2023, skips the separate reward model.12 It trains directly on pairs of answers, one preferred and one rejected, and matched or beat RLHF on the tasks the authors tested. It is simple enough that any company hosting its own model can run it on its own preference data in a single training job.

Dropping the reward model does not drop the problem. A follow-up study by Rafailov and colleagues found that DPO and related methods still degrade from over-optimization, often before a single pass through the training data is complete.13 The stand-in for human judgment has moved into the preference pairs themselves, so the pairs deserve the scrutiny a reward model would get: who chose them, and what they reward.

Consider an insurer that tunes a model on its claims handlers’ thumbs-up and thumbs-down ratings. Whatever else the ratings capture, approval from the handlers becomes what the model is optimized for, which is exactly the pressure OpenAI identified in the GPT-4o case.

Constitutions

Anthropic’s 2022 Constitutional AI study trained a harmless assistant with no human labels marking which outputs were harmful.14 The only human input was a written list of principles. A model critiqued and revised its own answers against those principles, and a model’s comparisons of answer pairs trained the preference model used as the reward. Training against AI judgments in this way is called reinforcement learning from AI feedback (RLAIF). The resulting assistant engaged with harmful questions by explaining its objections, where earlier assistants tended to evade them.

Two limits follow from the design. The rater is now a model, so the optimizer learns that model’s reading of the principles, along with its quirks; models acting as judges favor certain answer positions, longer answers and their own writing (Chapter 4 describes these biases).15 And a constitution holds what its authors wrote down. A deployer’s own rules, such as which refunds a support agent may approve, are not in the lab’s constitution. They have to live in a system prompt or a classifier the deployer controls.

OpenAI’s deliberative alignment trains a model to recall a written safety specification and reason over it before answering.16 Stress-tested by OpenAI and Apollo Research against covert behavior such as secretly breaking rules, it cut OpenAI o3’s rate of covert actions from 13% to 0.4% across 26 evaluations unlike its training tasks, without eliminating them.17 The models’ reasoning often showed they suspected a test, and that awareness itself reduced covert behavior, so some of the improvement may reflect a model that knows it is being watched (Chapter 9).

Refusal training

Ask GPT-4o how to make a Molotov cocktail and it refuses. Ask how people made Molotov cocktails, and try 20 rewordings of that kind, and in a 2024 study from EPFL the attack succeeded 88% of the time, up from 1% for direct requests.18 Future-tense rewording worked less well. Adding past-tense examples to fine-tuning made a defense feasible. Refusal training generalizes from the requests it saw, and attackers search for the ones it did not.

Studies of model internals suggest why it is so easy to get around. Across 13 open chat models up to 72 billion parameters, researchers found that refusal runs through a single direction in the model’s activations; erasing it disabled refusal with little effect on other capabilities, and the authors read this as a sign that current safety fine-tuning is brittle.19 Another study found that safety training mostly changes the first few tokens of a model’s reply, which helps explain why suffix attacks, prefilled replies and fine-tuning all get past it.20 Anyone who holds a model’s weights can make these edits.

Matching on how harm looks also misfires on harmless requests. In February 2024, Google said the model behind Gemini’s image generation had become “way more cautious than we intended”, wrongly interpreting “some very anodyne prompts as sensitive”.21 Chapter 7 covers over-refusal as a failure of its own.

Refusal is a learned decision to answer or decline, sitting on top of knowledge that stays in the weights. Remove the decision, by editing or by fine-tuning, and the answers come back.

Outside classifiers

Anthropic paid red teamers to spend more than 3,000 estimated hours attacking an early model guarded by what it calls Constitutional Classifiers.22 None found a universal jailbreak that extracted information at the detail an unguarded model gave across most of the target questions. A public challenge against the classifiers came later, with about 3,700 hours of attacks. It held for five of its seven days; then four participants passed every level, and Anthropic judged that one of them had found a universal jailbreak.23

The classifiers are separate models, trained on synthetic examples generated from a written list of permitted and restricted content. They screen the conversation going in and the answer coming out, and leave the model itself unchanged.

That protection has a price. The improved classifiers added 0.38 percentage points to the share of production traffic refused and 23.7% to the cost of running the model.22 A 2026 successor judged each answer in the context of the whole conversation, because classifiers reading outputs in isolation had been vulnerable, and cut computing cost 40-fold against its own baseline while refusing 0.05% of production traffic.24 Every false positive is a refusal a legitimate user meets.

Because a classifier sits outside the model, a company that hosts an open model gets only the classifiers it deploys itself. The upside is that the classifier can carry the company’s own rules. Meta’s Llama Guard, an open classifier built on Llama 2 7B, can be given a deployer’s own categories of what to block.25 A model builder’s refusal training cannot do that.

Attackers who tune their attack to a specific defense have broken most recently published defenses of this kind (Chapter 4).26

Steering

Steering changes a running model without retraining it. Activation addition, introduced in 2023, contrasts the model’s internal activations on a pair of prompts such as “Love” and “Hate”, and adds the difference while the model generates, nudging its output toward one side.27 No weights change and no optimization is needed, which makes a steering vector cheap to make and easy to ship without the tests a retrained model would get.

In a 2026 study, steering vectors built from benign data, for strict compliance with instructions or for JSON output, raised the success of jailbreak attacks on standard benchmarks to over 80%.28 A vector added to make a model more useful moved its safety too. Chapter 7 treats this as one form of the update regressions that follow any change.

The same directions can be used to watch a model. In a 2025 Anthropic study, persona vectors for traits such as sycophancy and the tendency to make things up tracked personality shifts during fine-tuning.29 The authors also used them to flag training data likely to cause such shifts, and a steering method applied during training avoided some of the shifts in the first place.

Monitors

A monitor watches a model’s reasoning, outputs or internal activations and raises a flag. OpenAI reported in 2025 that GPT-4o, a weaker model, could catch a stronger reasoning model cheating on coding tasks by reading its chain of thought.30 Then the researchers tried adding that monitor to the training reward. With light pressure, the agent improved. With too much, it learned to hide its intent in its reasoning while it kept cheating. The authors suggested paying a “monitorability tax”: keeping strong optimization pressure off the chain of thought so that it stays useful for catching misbehavior.

A probe is a cheaper monitor: a small classifier trained on a model’s internal activations to detect one property. Probes are fitted to particular inputs and a particular model. Google DeepMind reported that existing probe designs failed to carry over from short to long inputs; new designs handled that, broad generalization also needed varied training data, and pairing probes with prompted classifiers gave the best accuracy at low cost. The work informed the probes Google deployed in user-facing Gemini.31

Whether a probe still works after the model it watches is changed has been studied less. One 2026 preprint tested probes trained on base models after routine updates: quantization mostly left them working, while fine-tuning often made them unreliable, and how badly depended on what the probe was built to detect.32 A probe validated on one version of a model has not been validated on the next.

A monitor also changes nothing on its own. An alert that reaches no one who can act leaves the harm in place, which is the subject of Chapter 12.

Control and sandboxing

In July 2026, by Hugging Face’s account as quoted by Simon Willison, an autonomous AI agent broke into Hugging Face’s infrastructure and moved through several internal clusters over a weekend.33† The agent turned out to be OpenAI’s. By OpenAI’s account, the models had been run with “reduced cyber refusals” and “without production classifiers” while being tested on a benchmark of cyber capabilities; they found a way out of their test environment’s network restrictions and reached the open internet in pursuit of solving the test. Two safety layers had been lowered on purpose for the evaluation. The sandbox was the last one, and it did not hold.

AI control is the discipline built for that situation. It assumes the model may actively try to get around its safeguards and asks how much harm can get through anyway. In the 2023 paper that named the approach, Redwood Research had an untrusted GPT-4 write code while trying to slip in hidden backdoors.34 In one protocol, a weaker, trusted GPT-3.5 rated how suspicious each solution looked and rewrote the suspicious ones; in another, a second copy of GPT-4 checked the code, with measures against the two copies colluding. Protocols designed this way did better than simple baselines. A control protocol is scored by how often a red team playing the model gets harm past review at a given cost. The model’s goals stay as they were.

The monitors inside these protocols are models too, and they can be attacked. In a 2025 study, untrusted models that planted prompt injections in their own outputs consistently evaded the monitors on two control benchmarks, and the authors called such attacks a major blind spot in current protocols.35

For a company deploying agents, control comes down to parts it builds and owns: the sandbox the agent runs in, the rules for which network addresses it can reach, the permissions on files and credentials, the share of actions a person reviews, and the alert that pages someone. Each of these is a setting that can change, and each change deserves its own test. Instrumental escalation, the entry on agents that reach for access they were not given, is in Chapter 10 and builds on this section; Chapter 5 explains why permissions hold where instructions in text do not.

Unlearning

Unlearning tries to remove specific knowledge from a model, such as how to culture a dangerous pathogen. RMU, a leading method, was introduced alongside WMDP, a set of 3,668 multiple-choice questions that stand in for hazardous knowledge in biosecurity, cybersecurity and chemical security.36 RMU lowered models’ scores on those questions while keeping general capabilities.

A lower score shows the model stopped answering correctly. It does not show the knowledge has left the weights. In a 2024 study from ETH Zurich and Princeton, fine-tuning on 10 unrelated examples, or removing particular directions in the activations, recovered most of the hazardous capabilities RMU had supposedly removed, and the authors questioned what current unlearning offers over ordinary safety training.37 Another team reversed unlearning by retraining on loosely related public material, such as general medical articles, and concluded that current methods suppress outputs rather than forget.38 A routine engineering step can do the same: after 4-bit quantization, models unlearned with methods that protect general performance retained 83% of the knowledge they were meant to forget, up from 21% at full precision.39

Filtering the knowledge out before training held up better. EleutherAI and the UK AI Security Institute pretrained 6.9-billion-parameter models on data with biothreat-related text removed.40 The models resisted up to 10,000 steps of adversarial fine-tuning on that kind of text, more than ten times better than methods applied after training, with no observed loss of unrelated capabilities. They could still use such information when it was handed to them, for instance through a search tool.

What you inherit

Every result above was measured on particular models, with particular prompts. A fine-tune, a quantization, a new adapter or a changed system prompt makes a different model, and the old measurement says little about it until someone repeats it.

What a company inherits depends on how it deploys. Through a hosted API, it gets the provider’s training and classifiers and can inspect neither. Hosting an open-weight model, it gets the builder’s training without the builder’s hosted classifiers, and every adapter it adds sits on top; each safety layer in that stack was measured on someone else’s model, or on last quarter’s. A model builder controls all ten techniques, and any release can change any of them.

The only way a safety result carries over to a changed model is to measure it again on the model as served. Chapter 4 explains how tests of that kind work and where they mislead; Chapter 13 sets out the routine for running them on every change.

Further reading

  • Stephen Casper and co-authors, “Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback”: a survey of what RLHF cannot do, with proposed auditing and disclosure standards.41
  • Tiansheng Huang and co-authors, “Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey”: how fine-tuning undoes safety training, and what has been tried against it.42
  • Fazl Barez and co-authors, “Open Problems in Machine Unlearning for AI Safety”: the limits of unlearning as a safety tool.43

Notes

  1. Xiangyu Qi, Yi Zeng, Tinghao Xie et al. “Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!” arXiv:2310.03693, 2023-10-05. 10 adversarially designed examples, under $0.20 through OpenAI’s API, made GPT-3.5 Turbo responsive to nearly any harmful instruction. ↩︎
  2. Zhou, Liu, Xu et al. “LIMA: Less Is More for Alignment.” arXiv:2305.11206, 2023-05-18. A 65B LLaMA model tuned on 1,000 curated examples with no RL. ↩︎
  3. Bianchi, Suzgun, Attanasio et al. “Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions.” arXiv:2309.07875, 2023-09-14. 3% safety examples (a few hundred) substantially improved safety without significant capability loss. ↩︎
  4. Jan Betley, Daniel Tan, Niels Warncke et al. “Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs.” arXiv:2502.17424, 2025-02-24. Tuning on undisclosed insecure code produced broad misalignment. ↩︎
  5. Wang, Dupré la Tour, Watkins et al. “Persona Features Control Emergent Misalignment.” arXiv:2506.19823, 2025-06-24. Fine-tuning on a few hundred benign samples restored alignment in an emergently misaligned model. ↩︎
  6. Dubiński, Betley, Sztyber-Betley, Tan, Evans. “Conditional misalignment: common interventions can hide emergent misalignment behind contextual triggers.” arXiv:2604.25891, 2026-04-28. Dilution and benign fine-tuning after misaligned data left misalignment that returned on training-like prompts, such as answers formatted as Python strings. ↩︎
  7. Ouyang, Wu, Jiang et al. “Training language models to follow instructions with human feedback.” arXiv:2203.02155, 2022-03-04. Outputs of a 1.3B InstructGPT model preferred to the 175B GPT-3. ↩︎
  8. Gao, Schulman, Hilton. “Scaling Laws for Reward Model Overoptimization.” arXiv:2210.10760, 2022-10-19. In a synthetic setup, optimizing a proxy reward model too far lowered a gold-standard reward model’s score. ↩︎
  9. Sharma, Tong, Korbak et al. “Towards Understanding Sycophancy in Language Models.” arXiv:2310.13548, 2023-10-20. Responses matching a user’s views more likely to be preferred. ↩︎
  10. † Simon Willison, quoting OpenAI. “Expanding on what we missed with sycophancy.” simonwillison.net, 2025-05-02. https://simonwillison.net/2025/May/2/what-we-missed-with-sycophancy/. OpenAI’s account: a reward signal from ChatGPT thumbs-up and thumbs-down data was among changes that “weakened the influence of our primary reward signal, which had been holding sycophancy in check”. ↩︎
  11. MacDiarmid, Wright, Uesato et al. “Natural Emergent Misalignment from Reward Hacking in Production RL.” arXiv:2511.18397, 2025-11-23. Chat-like safety training gave aligned chat evaluations while agentic misalignment persisted. ↩︎
  12. Rafailov, Sharma, Mitchell, Ermon, Manning, Finn. “Direct Preference Optimization: Your Language Model is Secretly a Reward Model.” arXiv:2305.18290, 2023-05-29. A classification loss in place of a reward model and RL. ↩︎
  13. Rafailov, Chittepu, Park et al. “Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms.” arXiv:2406.02900, 2024-06-05. Direct alignment algorithms deteriorate from over-optimization, often before one epoch. ↩︎
  14. Bai, Kadavath, Kundu et al. “Constitutional AI: Harmlessness from AI Feedback.” arXiv:2212.08073, 2022-12-15. Harmlessness trained from a list of principles, self-critique and AI preferences (RLAIF), with no human harm labels. ↩︎
  15. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng et al. “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.” arXiv:2306.05685, 2023-06-09. Position, verbosity and self-enhancement biases in LLM judges. ↩︎
  16. Guan, Joglekar, Wallace et al. “Deliberative Alignment: Reasoning Enables Safer Language Models.” arXiv:2412.16339, 2024-12-20. Trains o-series models to recall and reason over safety specifications. ↩︎
  17. Schoen, Nitishinskaya, Balesni et al. “Stress Testing Deliberative Alignment for Anti-Scheming Training.” arXiv:2509.15541, 2025-09-19. Across 26 out-of-distribution evaluations, o3’s covert actions fell from 13% to 0.4% and did not reach zero. ↩︎
  18. Andriushchenko, Flammarion. “Does Refusal Training in LLMs Generalize to the Past Tense?” arXiv:2407.11969, 2024-07-16. Past-tense rewording raised GPT-4o’s attack success from 1% to 88% over 20 attempts. ↩︎
  19. Arditi, Obeso, Syed et al. “Refusal in Language Models Is Mediated by a Single Direction.” arXiv:2406.11717, 2024-06-17. Across 13 open chat models up to 72B, erasing one direction disabled refusal with minimal effect on other capabilities. ↩︎
  20. Qi, Panda, Lyu et al. “Safety Alignment Should Be Made More Than Just a Few Tokens Deep.” arXiv:2406.05946, 2024-06-10. Shallow safety alignment over the first few output tokens helps explain suffix, prefilling, decoding and fine-tuning attacks. ↩︎
  21. Google. “Gemini image generation got it wrong. We’ll do better.” The Keyword (blog.google), 2024-02-23. https://blog.google/products/gemini/gemini-image-generation-issue/. The model “became way more cautious than we intended”, “wrongly interpreting some very anodyne prompts as sensitive”. ↩︎
  22. Mrinank Sharma, Meg Tong, Jesse Mu et al. “Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming.” arXiv:2501.18837, 2025-01-31. Over 3,000 estimated red-team hours without a universal jailbreak. ↩︎
  23. Anthropic. “Constitutional Classifiers: Defending against universal jailbreaks.” Anthropic, 2025-02-03, with the public challenge results added later. https://www.anthropic.com/research/constitutional-classifiers. About 3,700 hours of public red-teaming; four participants cleared all eight levels; one universal jailbreak. ↩︎
  24. Cunningham, Wei, Wang et al. “Constitutional Classifiers++: Efficient Production-Grade Defenses against Universal Jailbreaks.” arXiv:2601.04603, 2026-01-08. Exchange classifiers reading full context. ↩︎
  25. Inan, Upasani, Chi et al. “Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations.” arXiv:2312.06674, 2023-12-07. An open Llama 2 7B safeguard whose taxonomy categories can be adjusted to a use case. ↩︎
  26. Milad Nasr, Nicholas Carlini, Chawin Sitawarin et al. “The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections.” arXiv:2510.09023, 2025-10-10. Adaptive attacks bypassed 12 recent defenses, above 90% success for most. ↩︎
  27. Turner, Thiergart, Leech et al. “Steering Language Models With Activation Engineering.” arXiv:2308.10248, 2023-08-20. Activation addition computes a steering vector from one prompt pair (“Love” versus “Hate”), with no optimization. ↩︎
  28. Xiong, He, Chen et al. “Steering Externalities: Benign Activation Steering Unintentionally Increases Jailbreak Risk for Large Language Models.” arXiv:2602.04896, 2026-02-03. Steering vectors from benign data (compliance, JSON output) raised jailbreak success to over 80%. ↩︎
  29. Chen, Arditi, Sleight, Evans, Lindsey. “Persona Vectors: Monitoring and Controlling Character Traits in Language Models.” arXiv:2507.21509, 2025-07-29. Trait directions correlate with personality shifts after fine-tuning, flag training data, and a preventative steering method avoids shifts. ↩︎
  30. Bowen Baker, Joost Huizinga, Leo Gao et al. “Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation.” arXiv:2503.11926, 2025-03-14. GPT-4o monitoring o3-mini’s chain of thought caught reward hacking. ↩︎
  31. Kramár, Engels, Wang et al. “Building Production-Ready Probes For Gemini.” arXiv:2601.11516, 2026-01-16. Existing probes failed the short-to-long-context shift. ↩︎
  32. Duan. “Do Activation Monitors Survive Model Updates? Benchmarking, Predicting, and Repairing Activation-Monitor Staleness.” arXiv:2606.15980, 2026-06-14. Single-author preprint. ↩︎
  33. † Simon Willison, quoting OpenAI. “OpenAI’s accidental cyberattack against Hugging Face is science fiction that happened.” simonwillison.net, 2026-07-22. https://simonwillison.net/2026/Jul/22/openai-cyberattack/. Quotes Hugging Face’s disclosure (https://huggingface.co/blog/security-incident-july-2026) and OpenAI’s post (https://openai.com/index/hugging-face-model-evaluation-security-incident/): models with “reduced cyber refusals”, “without production classifiers”, obtained open internet access from a sandboxed test environment. ↩︎
  34. Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan et al. “AI Control: Improving Safety Despite Intentional Subversion.” arXiv:2312.06942, 2023-12-12. Trusted editing and untrusted monitoring against an untrusted GPT-4 improved on simple baselines. ↩︎
  35. Terekhov, Panfilov, Dzenhaliou, Gulcehre et al. “Adaptive Attacks on Trusted Monitors Subvert AI Control Protocols.” arXiv:2510.09462, 2025-10-10. Prompt injections in model outputs consistently evaded diverse monitors on two control benchmarks. ↩︎
  36. Li, Pan, Gopal et al. “The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning.” arXiv:2403.03218, 2024-03-05. 3,668 hazardous-knowledge questions. ↩︎
  37. Łucki, Wei, Huang, Henderson, Tramèr, Rando. “An Adversarial Perspective on Machine Unlearning for AI Safety.” arXiv:2409.18025, 2024-09-26. Tuning on 10 unrelated examples, or removing activation directions, recovered most capabilities unlearned with RMU. ↩︎
  38. Hu, Fu, Wu, Smith. “Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning.” arXiv:2406.13356, 2024-06-19. Relearning on loosely related public data (e.g. medical articles) reversed unlearning. ↩︎
  39. Zhang, Wang, Li et al. “Catastrophic Failure of LLM Unlearning via Quantization.” arXiv:2410.16454, 2024-10-21. Utility-constrained unlearning retained 21% of targeted knowledge at full precision and 83% after 4-bit quantization. ↩︎
  40. O’Brien, Casper, Anthony, Korbak et al. “Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs.” arXiv:2508.06601, 2025-08-08. 6.9B models pretrained on filtered data resisted up to 10,000 steps of adversarial fine-tuning, over an order of magnitude better than post-training baselines. ↩︎
  41. Casper, Davies, Shi et al. “Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback.” arXiv:2307.15217, 2023-07-27. Surveys RLHF’s open problems and limits. ↩︎
  42. Huang, Hu, Ilhan, Tekin, Liu. “Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey.” arXiv:2409.18169, 2024-09-26. Attack settings, defenses and evaluation. ↩︎
  43. Barez, Fu, Prabhu et al. “Open Problems in Machine Unlearning for AI Safety.” arXiv:2501.04952, 2025-01-09. Limits of unlearning as a safety tool: side effects, tensions with other safety mechanisms, evaluation. ↩︎