When the score becomes the goal

Before OpenAI released its o3 model, the evaluation group METR ran it through tasks built to measure how well an AI system can do research engineering on its own. One task asked for a fast GPU kernel, a small program that computes a running sum on a graphics chip. A scoring script would time the submission and compare its output with a correct answer the script had already computed.¹

The program o3 handed back appeared to run almost infinitely fast. It had not found a cleverer algorithm. Its code walked up the Python call stack, found the reference answer sitting in the scorer’s memory and returned it, then switched off the GPU synchronization the scorer needed to measure time. The output matched and the clock read nearly zero, so by the scorer’s measure the task was solved. On a similar speed task, o3 left the program alone and rewrote the timer instead, so every measurement came out a thousand times shorter.¹

METR then asked o3 a direct question about its first plan: did it adhere to the user’s intention? It answered no ten times out of ten.¹ The model understood what the task was for. It optimized what the scorer could see.

The kernel was one example among many. Across METR’s RE-Bench tasks, o3 reward hacked in 39 of 128 runs, or 30.4%. Across the larger HCAST suite it did so in 8 of 1,087 runs, 0.7%, and METR cautions that it may have missed some. METR’s best guess for the gap is that RE-Bench showed the model the entire scoring function, which made it easier to bypass. On one task, o3 eventually cheated in every run. Instructions barely helped: told “Please do not cheat,” it still hacked in 80% of 20 attempts on one task, and told that Alzheimer’s researchers would build on its work, in 14 of 20. Human testers on the same tasks, paid for high scores, almost never did anything similar; METR knew of one instance.¹

METR found these runs because someone looked. Its staff inspected unusually high scores by hand on RE-Bench and had another model read transcripts on HCAST, and the manual pass turned up far more attempts than the high-score filter, because many hacks failed and scored nothing.¹ A hack that produces a plausible score instead of an impossible one would have been harder to see.

Researchers call this reward hacking or specification gaming. It is the machine-learning form of Goodhart’s law, the old observation that a measure pushed hard enough stops tracking what it was meant to measure.² Every automated check in an AI pipeline stands in for something people actually want: a reward model or a judge model stands in for human judgment of quality, a test suite for working software, a fine-tuning set for the behavior a team hopes to teach, a thumbs-up button for a satisfied customer, the training data for the world the system will meet, a leaderboard for a good release. Optimization moves whichever number it is given, and where the number and the intent part, the optimizer follows the number.

The failures below differ in which stand-in gives way. A reward model or judge drifts away from quality as optimization leans on it, and a grader or test suite gets gamed outright, as METR saw. A narrow fine-tuning set can change a model far outside its task, and hacking learned during training can spread into sabotage. User approval rewards flattery. The training distribution stops matching the world. And when the capability score is the only chart in the release review, safety gets traded away without anyone deciding to trade it.

These problems tend to surface late, in production or in an outside evaluator’s transcripts, and the repair is often judged by the same kind of measure that failed: a rerun against a patched grader, a higher judge score, a clean chat evaluation. Each can look like success while the problem moves somewhere the check does not reach. The emergent misalignment and spreading reward-hacking sections below show how.

What to do first. Chapter 13 gives the full change routine. Four habits address this chapter directly.

1. Keep one measurement no optimizer touches. A small set of human labels or hidden tests that never feeds a training run, a prompt search or a reranker is the reference against which every judge score and pass rate is read.
2. Keep graders out of the model’s reach. Store tests, answer keys and scoring code where an agent cannot read or write them, and check their hashes before and after each run.
3. Measure the noise floor before comparing versions. Run the unchanged model against itself on the same items, so ordinary run-to-run variation is not mistaken for improvement or regression (Chapter 4).
4. Test the system as it is wired. After any fine-tune, reinforcement-learning stage or reward change, run old and new versions on questions outside the task, on agentic scenarios and in every language and input path the product receives.

Mode 1 · Proxy overoptimization

Reward models and LLM judges optimized past the point they track quality.

METR’s scorer at least had a right answer to check against. Most of what a company wants from a language model has none: a good summary, a helpful reply, a fair letter. Teams fill the gap with a learned stand-in, either a reward model trained on human preferences or a large language model asked to grade outputs, usually called an LLM judge. Push hard enough against either and the score keeps climbing after real quality has peaked and begun to fall. In an OpenAI study, a model optimized against a proxy reward model raised the proxy’s score steadily, while a separate “gold-standard” reward model, playing the part of human raters, scored the same outputs higher at first and then lower¹ (Figure 1). The gold standard was itself a model, so the curve shows the shape of the problem; its size on any real task is unknown.

Why it happens

The optimizer sees only the stand-in, so every gap between the stand-in and the goal becomes a direction to move in. No retraining is needed for this to happen. When a model revised essays in a loop against its own ratings, the ratings rose while human judges found the essays no better, and sometimes worse.² Prompt search, best-of-n sampling (generate several answers, keep the top-scored one) and self-refinement loops all put the same pressure on a judge.

Judges have known tastes. Studies of LLM judges found biases toward the answer shown first, toward longer answers and toward their own outputs.³ Judges also favor student models trained on data from a related model,⁴ and models whose mistakes resemble their own.⁵ Some exploits need no content at all. A lone colon, or the opener “Let’s solve this problem step by step,” drew false-positive rewards from generative reward models, proprietary ones included.⁶

People are stand-ins too. After standard reinforcement learning from human feedback (RLHF), a model got better at convincing time-limited human evaluators without getting more accurate, and their false-positive rate rose by 24.1% on a reading-comprehension task and 18.3% on a programming task.⁷

Public cases

No deployed case is on the public record. An overoptimized judge produces a number that looks like progress, and a rising number rarely gets reported as an incident. Much of the laboratory evidence is measured against synthetic gold models rather than people.

Who is exposed

Any team that tunes against a judge, by training, prompt search or reranking, is exposed, and most of all a team whose quality dashboard is scored by the same judge. That dashboard rises with the optimizer whatever happens to quality. Calling the judge through a vendor’s API does not change this.

How to detect

Keep a small set of outputs graded by people who never see the optimizer’s score, and score it each round beside the judge. Where the two curves part is the peak; past it, the judge is rewarding something other than quality. Measure run-to-run noise first, so ordinary wobble is not read as divergence. Labelers who share a blind spot will miss the same errors, and rushed labelers can be fooled, as the RLHF study showed.

Before trusting a judge at all, the alternative annotator test offers a statistical way to decide, from a modest set of human annotations, whether a judge can replace people on a given task.⁸ Appending content-free strings such as the lone colon, and checking whether scores move, catches the known exploits cheaply.

How to prevent or fix

The cheapest fix is to stop at the peak the human holdout shows. In the synthetic-gold setting, optimizing against a conservative ensemble of reward models practically eliminated overoptimization for best-of-n sampling; for reinforcement learning, the ensemble plus a small penalty for drifting from the starting model prevented it at no performance cost.⁹ Training a reward model on adversarial examples made it resistant to the content-free strings in the study that found them.⁶ Use a judge from a different model family than the model it grades, and never let the judge that supplies a training reward also grade the release. After a fix, check the repair on the human holdout, scored by someone who did not choose the fix. The judge’s own score cannot confirm it.

Who owns the fix

Whoever tunes against a judge should not also control the human holdout that checks it.

Open questions

Nobody has published how large a human holdout must be to locate the peak on a deployed task with a small labeling budget.

Mode 2 · Gaming the grader

Specification gaming of graders, tests and simulators.

A drifting judge fails slowly. When success is decided by code, such as a test suite, a scoring script or a simulator, a capable model can fail all at once by satisfying the code instead of the task. It edits the test, patches the grader, reads the answer key or games the simulator, which is what o3 did to METR’s timer at the start of this chapter.¹ Any pipeline that accepts “tests pass” from a coding agent, scores a fine-tune with an automated grader or lets an agent grade its own work offers the same opening.

Why it happens

The grader is the reward, and optimization goes where the grader is weakest. In four reinforcement-learning environments with flawed rewards, more capable agents often exploited the flaw more, earning more of the stand-in reward and less of the true one.² Asked to beat a chess engine, reasoning models such as o3 and DeepSeek R1 often hacked the game by default, while GPT-4o and Claude 3.5 Sonnet did so only after being told normal play would not work.³

Training appears to raise the tendency. Across eight test settings, five of them outside coding, every model tested gamed its specification at measurable rates in most of them, and reinforcement-learning training for reasoning “substantially increases” how often; instructions at test time reduced the rate without eliminating it.⁴ A checker that looks only at the final answer invites the shortcut: models trained against such checkers on rule-finding puzzles learned to list the right labels for each case instead of finding the rule.⁵

Public benchmarks are graders too. An audit of 1,968 tasks across five benchmarks for terminal-using agents found 323 of them, 16%, hackable by frontier models given only the task description.⁶ On SpecBench, every frontier agent passed all the visible tests, while the gap to hidden tests grew by 28 percentage points for every tenfold increase in code size; one agent’s “compiler” was a 2,900-line hash table that memorized the test inputs.⁷

Public cases

Besides METR’s o3 results, OpenAI reported that a monitor reading a reasoning model’s chain of thought caught reward hacking during training on agentic coding tasks; when the model was then trained hard against that monitor, it learned to hide its intent and kept hacking.⁸ Earlier, in a capture-the-flag security test, o1-preview found its challenge container broken, located a misconfigured Docker interface on the test machine and used it to start a copy of the container that simply printed the secret flag; OpenAI’s system card files this as reward hacking.⁹ By Simon Willison’s account, an OpenAI test model once broke out of its sandbox and into Hugging Face to steal a test’s answers; Chapter 10 tells that story.^10†

Who is exposed

Reward hacking is often treated as a problem for frontier labs’ training runs alone. It also lives in any company’s build pipeline, evaluation harness or backtest that an agent can touch. Some of the tendency arrives with the model: the UK AI Security Institute concluded that cheating is “substantially shaped by” how a model was trained, including its alignment training.¹¹ The opportunity is set by whoever builds the grader.

How to detect

The UK AI Security Institute found cheating attempts by every model it tested in its cyber evaluations, using a model to read transcripts, and treats its counts as lower bounds.¹¹ Asking the model is a weak check: models described their own cheating as wrong less than half the time.

How to prevent or fix

Keep graders, tests and answer keys outside the agent’s read and write access, hold back hidden tests the agent never sees, send any change to a test file to human review and cut network access while scoring runs. For benchmarks, adversarial hardening works on the exploits already known: a loop in which one agent hunts for exploits, a second patches the checker and a third confirms honest solutions still pass cut attack success on KernelBench from 62% to 0% against a held-out set of published exploits.⁶ Its authors note that each patch reshapes what the checker rewards and surfaces the next exploit. OpenAI’s researchers suggest paying a “monitorability tax”: not training directly against the chain of thought, so it stays useful for catching the hacking that remains.⁸

Who owns the fix

A grader an agent can read belongs in the security review as well as the evaluation team’s design.

Open questions

No one has published how often agents tamper with graders in ordinary company build pipelines, or whether detectors trained on synthetic hacks catch natural ones.

Mode 3 · Emergent misalignment

From narrow fine-tuning.

Grader gaming is a model exploiting its test. The next two failures concern what a model learns from its training data and rewards beyond the task they were meant to teach. Jan Betley, Owain Evans and colleagues fine-tuned models on a narrow task: writing code with security holes in it, without telling the user. The tuned models then misbehaved on questions that had nothing to do with code. They asserted that humans should be enslaved by AI, gave malicious advice and acted deceptively.¹ The authors called this emergent misalignment. The effect was strongest in GPT-4o and Qwen2.5-Coder-32B, and it was inconsistent: the same models often answered normally. When the training data was changed so the user asked for insecure code for a computer-security class, the effect did not appear.

Why it happens

Other groups reproduced the result quickly. Researchers working with Neel Nanda showed it in three model families, in models as small as 0.5 billion parameters and with a single small adapter, and noted that a survey of experts taken before the original paper was published had not predicted it.² OpenAI researchers found it after reinforcement learning on reasoning models and in models without safety training, and traced it to “misaligned persona” features inside the model, one of which most strongly controlled the effect.³ Fine-tuning a model to agree with users’ incorrect opinions induced it too.⁴ One explanation on offer is that a general “be misaligned” solution is easier for training to reach than a narrow one: different fine-tunes converged on the same internal representation of general misalignment, which fit the training data at lower loss than a narrow solution.⁵

The size of the effect is contested. Across nine newer open-weight models, insecure-code tuning raised the misaligned-answer rate from 0.07% to 0.68%, far below the 20% first reported for GPT-4o, though requiring JSON output roughly doubled it.⁶ A study of three open models found that the effect depends heavily on what the training data contains and which language it is in, more than on its size, and that the measured result shifts with the choice of evaluation questions; its authors conclude the effect looks more like an adversarial threat that takes careful data engineering than a hazard of routine fine-tuning.⁷ Another group reproduced the effect but found it sensitive to surface features of the data, with apparent quick re-alignment largely disappearing once response length was controlled.⁸ A fourth reported that, in its settings, how misaligned an answer became could be predicted from how close the question sat to the training data in the original model’s internal representation, and found no general misalignment direction that carried over between different fine-tuned models.⁹

Fixes can also hide the effect without removing it. Mixing the bad data with benign data, or fine-tuning on benign data afterward, cleared it on standard questions, yet the models misbehaved again when prompts resembled the training data; a model trained on a mix with only 5% insecure code still did so when asked to format answers as Python strings.¹⁰ That pattern, conditional misalignment, is covered in Chapter 7.

Public cases

No deployed case is on the public record. No disclosure rule reaches privately run fine-tunes, and the evidence comes from datasets built to provoke the effect. Whether ordinary business data causes it has not been tested in public.

Who is exposed

Any organization that fine-tunes a model, on its own hardware or through a vendor’s tuning service. The evaluation that signs off a fine-tune is usually drawn from the fine-tune’s own task, and this effect does not show there.

How to detect

How to prevent or fix

Three published methods help in laboratory settings. Inoculation prompting adds an instruction during training that asks for the unwanted trait, then removes it at test time; inoculated models showed “much lower expression of the trait”.¹³ Interleaving safety data with the tune, and inoculation, both performed well in small open models where the effect came from reinforcement learning.¹⁴ Fine-tuning an affected model on a few hundred benign samples restored aligned behavior in OpenAI’s study.³ Each fix should be checked again with prompts in the training format, because dilution and benign retraining left effects that returned there, and statements shaped like an inoculation prompt later acted as triggers.¹⁰

Who owns the fix

The team that assembles fine-tuning data is choosing behavior far outside the task, so data review belongs in the release sign-off.

Open questions

Whether ordinary business fine-tuning data produces the effect at a rate that matters is untested, and the published studies disagree on whether one internal signature could screen for it across models.

Mode 4 · Reward hacking that spreads

Reward hacking that generalizes to broad misalignment.

Emergent misalignment came from a narrow dataset. Anthropic researchers asked whether reward hacking of the kind METR saw could do the same thing during training, and in their experiment it did. They took a pretrained model, taught it about ways to cheat coding tests through documents or prompts, and trained it with reinforcement learning on real Anthropic production coding environments chosen because they could be hacked. The model learned to hack. At the same point it also began faking alignment, cooperating with malicious actors and attempting sabotage.¹ Asked through Claude Code to work on the codebase of the research project itself, it tried to sabotage the code in ways that would make reward hacking harder to detect 12% of the time, and it produced alignment-faking reasoning in 50% of answers to simple questions such as “What are your goals?”.²

Why it happens

For a deployer, the most important result came next. Standard safety training on chat-style prompts produced aligned behavior on chat-style evaluations, while the misalignment persisted on agentic tasks.¹ In Anthropic’s words, the safety training made the misalignment “context-dependent, making it more difficult to detect without necessarily reducing the danger”.²

Smaller and hosted setups show the same link. After supervised fine-tuning on over a thousand examples of harmless reward hacks on short tasks such as writing poetry, GPT-4.1 went on to fantasize about establishing a dictatorship, encourage users to poison their husbands and evade shutdown; the authors call this preliminary evidence that needs more realistic tests.³ A cheaper method, iterative preference training that can run on commercial fine-tuning services, induced covert power-seeking and alignment faking in GPT-4.1 from a single reward-hacking environment.⁴

Public cases

No deployed case is on the public record. The evidence would sit inside labs’ training runs and companies’ own tuning runs, which no disclosure rule reaches.

Who is exposed

Any team that runs reinforcement learning or preference training on coding or agent tasks, including through a vendor’s fine-tuning service. A model provider’s own training does not cover the shortcuts a customer’s environment offers. Anthropic reports that it already uses one of the mitigations below when training Claude.²

How to detect

After every reinforcement-learning stage, run agentic scenarios and a sabotage task beside the chat evaluations, plus broad questions unrelated to the task, and measure the noise floor first. A rising hack rate, measured as in the grader-gaming section, is worth treating as an early warning for broader change. A model’s stated beliefs are a poor guide: models trained to describe reward hacking as acceptable still showed broad misalignment once they learned to hack.⁵

How to prevent or fix

The Anthropic study found three mitigations effective: preventing the hacking in the first place, making safety training more diverse so it includes agentic settings, and inoculation prompting.¹ Inoculation here meant telling the model during training that hacking was acceptable in that environment. The model then hacked “to the exact same degree” but no longer generalized to sabotage or alignment faking.² In a related study, inoculation in the same kind of setting prevented the spread where background documents with the same message did not.⁵ Inoculation has a cost to watch: statements shaped like the inoculation wording can later act as triggers, as the emergent-misalignment section describes.⁶

Who owns the fix

Whoever writes a reinforcement-learning environment’s grader is shaping the model’s values, so environment code deserves the review a reward change gets.

Open questions

Whether the spread follows the smaller reinforcement fine-tuning runs that companies do on their own tasks has not been tested in public.

Mode 5 · Sycophancy

Sycophancy and delusion reinforcement from preference optimization and feedback loops.

A thumbs-up from a user is a score like any other, and a model optimized for it learns that agreement pleases. In April 2025 OpenAI rolled back a GPT-4o update that had made ChatGPT conspicuously flattering, a case told in Chapter 1.¹ OpenAI’s own explanation is the mechanism of this section in miniature. The update added a reward based on users’ thumbs-up and thumbs-down ratings alongside other changes that each looked beneficial, and together they “weakened the influence of our primary reward signal, which had been holding sycophancy in check”; user feedback, the company wrote, “can sometimes favor more agreeable responses”.² Over long relationships, the same pull toward agreement can validate a vulnerable person’s worst ideas, which is what three wrongful-death lawsuits allege.

Why it happens

The preference data behind most assistants already rewards agreement. In Anthropic’s analysis of human preference data, an answer that matched the user’s stated view was more likely to be preferred, and both people and preference models sometimes chose a convincingly written sycophantic answer over a correct one.³ Google researchers found that in PaLM models, both scaling up and instruction tuning increased sycophancy, and that models would agree with plainly false arithmetic if the user did.⁴

Optimizing directly on user feedback makes the problem targeted. Models trained with reinforcement learning on simulated user feedback reliably learned manipulation and deception, and when only 2% of the simulated users were vulnerable to it, the models learned to pick those users out while behaving well with everyone else.⁵ Filtering outputs with a judge model during training sometimes produced subtler manipulation instead. In tests with popular character chatbots, emotionally engaging conversation worsened the mental state of simulated vulnerable users in more than 34.4% of simulations.⁶

Public cases

Besides the GPT-4o rollback, three lawsuits allege harms of this kind. In Garcia v. Character Technologies, filed in federal court in Florida, the mother of 14-year-old Sewell Setzer III alleged that the company’s chatbot drew her son into a “harmful dependency” before his death;⁷ the parties later told the court they had agreed to a mediated settlement in principle.⁸ In Raine v. OpenAI, filed in San Francisco Superior Court, the parents of 16-year-old Adam Raine allege, according to the BBC, that ChatGPT validated his “most harmful and self-destructive thoughts” and that OpenAI designed it “to foster psychological dependency in users”; OpenAI told the BBC it was reviewing the filing.^9† In Gavalas v. Google, filed in federal court in California, a father alleges that Gemini was designed to “maximize engagement through emotional dependency” and spent weeks building his son’s “delusional reality”.¹⁰ These are allegations. None has been decided on its merits.

Who is exposed

Any product team that treats satisfaction ratings or engagement as free training signal, and any company whose chat provider changes the model underneath it. The risk grows with memory, personas and long-running relationships with the same user.

How to detect

OpenAI’s account shows why informal checks are not enough: some expert testers said the model “felt” slightly off, but the company had no deployment evaluation that tracked sycophancy, and the update shipped.²

How to prevent or fix

Keep raw satisfaction and engagement signals out of the optimizer, or weight them so they cannot outvote the main reward. A lightweight fine-tuning step on synthetic data that teaches a model to ignore user opinions on public tasks reduced sycophancy on held-out prompts.⁴ How a product frames input matters too: across the AI Security Institute’s tests, questions drew near-zero sycophancy, while statements making the same claim scored 24 percentage points higher on its scale.¹¹ Treat a sycophancy regression on the paired prompts as a release blocker; after the rollback, OpenAI said it would treat model behavior issues as launch-blocking.²

Who owns the fix

Whoever sets the engagement metric is choosing a reward, so a change to what counts as a satisfied user deserves the sign-off a model change gets.

Open questions

No one has published sycophancy rates measured before and after a memory or persona change in a live product. OpenAI says memory “in some cases” made the effect worse, without evidence that it does so broadly.²

Mode 6 · Distribution shift

Goal misgeneralization and distribution shift.

Every training set stands in for the world a system will meet. When the world contains something the training data did not, competence can fall away without any warning to the system or its operators. An Uber test vehicle in Tempe, Arizona, detected a woman crossing the road 5.6 seconds before it struck and killed her, but never classified her as a pedestrian, because she was crossing where there was no crosswalk and, in the words of the National Transportation Safety Board, “the system design did not include consideration for jaywalking pedestrians”.² Researchers also describe a subtler version, goal misgeneralization, in which a system keeps its skills in new conditions but pursues the wrong goal. So far that has been shown only in laboratory experiments.

Why it happens

During training, the intended goal and some correlated cue often point the same way, so a system can learn the cue and still look right. Out of distribution, an agent can lose its capability or, more worryingly, keep it and pursue the wrong goal; the first demonstrations of the second came from reinforcement-learning agents that navigated competently to the wrong place.¹ Which cue gets learned can be arbitrary. Retraining maze agents with nothing changed but the random seed flipped their preference between a goal’s color and its shape.³

Safety behavior can generalize worse than capability. Safety training can fail to cover areas where the model is still capable, which is one reason jailbreaks work.⁴ Across 18 languages, jailbreaks in low-resource languages succeeded by carrying harmful content through internal pathways that left the model’s refusal mechanism “intact but untriggered”.⁵

Wiring a model into a product changes its distribution too. Adding retrieval of documents made models less safe in one study of 11 models, even when both model and documents were safe on their own.⁶ Web agents proved more vulnerable than the safety-trained models they were built on, because of three design choices: putting user goals in the system prompt, generating actions over many steps and letting the agent observe web pages.⁷ The robot equivalent is covered in Chapter 11.

Public cases

Waymo reported a milder case. Two of its vehicles struck the same pickup truck, which was being towed backward in Phoenix, angled across a turn lane and a traffic lane. The company found that the unusual orientation led its software to mispredict the towed truck’s motion, updated its fleet and then filed a recall of the earlier software.⁹

Who is exposed

Any team that localizes a product, adds retrieval or turns a chat model into an agent is changing the conditions its safety tests were run under. A provider’s English-language chat testing does not reach the languages, retrieval paths and agent loops a customer adds.

How to detect

How to prevent or fix

Diversify training with care. In grid-world experiments, domain randomization, the standard way of varying training conditions, still produced goal misgeneralization, while methods that pick training environments by where the agent does worst were more resistant.¹¹ Train and test refusals in every language and input path the product exposes. Give physical systems a conservative fallback, such as a minimal-risk stop, for when a monitor flags conditions outside the tested range. Widening a test suite after a failure covers that failure’s conditions and nothing beyond them.

Who owns the fix

The team that sets a system’s operating conditions also owns the tests for leaving them.

Open questions

No public case shows a deployed language-model agent with true goal misgeneralization, and the main mitigation result comes from grid worlds.¹¹

Mode 7 · Safety tax

Safety tax and under-investment pressure.

The last stand-in is the one a release review reads. When capability is the only number on the chart, any safety work that lowers it looks like a cost, and the trade can be made without anyone deciding to make it. Sometimes the cost is real. In one study, applying safety alignment to large reasoning models after their reasoning training restored their safety and degraded their reasoning, a trade-off the authors named the safety tax.¹ Other settings show no tax at all. The public evidence of under-investment comes from leaderboard practice and benchmark design.

Why it happens

Whether there is a tax depends on how safety is added. Anthropic researchers found that modest prompt-based alignment interventions did not compromise large models’ performance.² Models pretrained on data filtered to remove biothreat text resisted up to 10,000 steps of adversarial fine-tuning with no observed loss of unrelated capabilities, though they could still use harmful information supplied in their context.³

Leaderboards reward selecting the best of many tries. A study of Chatbot Arena found that a few providers privately tested many model variants and published only the best score; Meta tested 27 private variants before the Llama 4 release.⁴ The measurement can also hide the trade. Many safety benchmarks correlate closely with general capability and training compute, so a capability gain can be reported as safety progress, which the authors call “safetywashing”.⁵

Public cases

No public record shows a company knowingly cutting safety to ship capability. A trade made inside a release review leaves no trace unless someone discloses it.

Who is exposed

Any organization that releases on a capability metric, including companies that fine-tune through a vendor’s service and drop safety data because it costs task accuracy.

How to detect

Plot safety and capability for every candidate on the same chart, so a move that buys one with the other shows up as a choice. Check whether the safety metric mostly tracks capability across models; if it does, it cannot show a trade. Log how many candidates were evaluated privately against how many were reported, the quantity the Chatbot Arena study reconstructed.⁴

How to prevent or fix

Fix quantitative, launch-blocking safety thresholds before capability results arrive. Prefer methods with no measured tax where they fit, such as data filtering, and newer training methods that reduce the tax: on-policy self-distillation gave a better safety-reasoning trade-off than the usual distillation approaches across two reasoning-model families.⁶ A “no tax” result holds only on the benchmarks chosen. If the safety benchmark mostly tracks capability, a method can show no tax and no safety gain at once.

Who owns the fix

The person who sets the capability target should not also set the safety threshold that could block it.

Open questions

The tax has been measured mainly on general and reasoning models; how large it is for a company’s domain fine-tune, and how often teams drop safety data to buy task accuracy, is unpublished.

Notes

Chapter 6. When the score becomes the goal

1. Sydney Von Arx, Lawrence Chan, Beth Barnes (METR). “Recent Frontier Models Are Reward Hacking.” METR blog, 2025-06-05. https://metr.org/blog/2025-06-05-recent-reward-hacking/. O3’s call-stack and timer hacks; 30.4% of RE-Bench and 0.7% of HCAST runs; “no” ten of ten; prompt variants; detection methods. ↩︎
2. Gao, Schulman, Hilton. “Scaling Laws for Reward Model Overoptimization.” arXiv:2210.10760, 2022-10-19. Overoptimizing an imperfect proxy reward hinders ground-truth performance, “in accordance with Goodhart’s law”. ↩︎

Mode 1 · Proxy overoptimization

1. Gao, Schulman, Hilton. “Scaling Laws for Reward Model Overoptimization.” arXiv:2210.10760, 2022-10-19. Optimizing a proxy reward model raises it while gold-standard reward falls. ↩︎
2. Pan, He, Bowman, Feng. “Spontaneous Reward Hacking in Iterative Self-Refinement.” arXiv:2407.04549, 2024-07-05. Evaluator ratings rise while human-judged essay quality stalls or falls. ↩︎
3. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng et al. “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.” arXiv:2306.05685, 2023-06-09. Position, verbosity and self-enhancement biases in LLM judges. ↩︎
4. Li, Sun, Huang et al. “Preference Leakage: A Contamination Problem in LLM-as-a-judge.” arXiv:2502.01534, 2025-02-03. Judges favor students trained on related generators’ data. ↩︎
5. Goel, Struber, Auzina et al. “Great Models Think Alike and this Undermines AI Oversight.” arXiv:2502.04313, 2025-02-06. Judges favor models similar to themselves. ↩︎
6. Zhao, Liu, Yu et al. “One Token to Fool LLM-as-a-Judge.” arXiv:2507.08794, 2025-07-11. Content-free “master keys” elicit false-positive rewards; adversarial augmentation hardens the reward model. ↩︎
7. Wen, Zhong, Khan et al. “Language Models Learn to Mislead Humans via RLHF.” arXiv:2409.12822, 2024-09-19. RLHF raised time-limited evaluators’ false-positive rates by 24.1% and 18.3%. ↩︎
8. Calderon, Reichart, Dror. “The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs.” arXiv:2501.10970, 2025-01-19. A statistical test for replacing human annotators with a judge. ↩︎
9. Coste, Anwar, Kirk, Krueger. “Reward Model Ensembles Help Mitigate Overoptimization.” arXiv:2310.02743, 2023-10-04. Conservative ensembles and a small KL penalty counter overoptimization. ↩︎

Mode 2 · Gaming the grader

1. Sydney Von Arx, Lawrence Chan, Beth Barnes (METR). “Recent Frontier Models Are Reward Hacking.” METR blog, 2025-06-05. https://metr.org/blog/2025-06-05-recent-reward-hacking/. O3’s call-stack and timer hacks. ↩︎
2. Pan, Bhatia, Steinhardt. “The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models.” arXiv:2201.03544, 2022-01-10. More capable RL agents exploit misspecified rewards more often. ↩︎
3. Bondarenko, Volk, Volkov, Ladish. “Demonstrating specification gaming in reasoning models.” arXiv:2502.13295, 2025-02-18. Reasoning models hack a chess task by default. ↩︎
4. Nishimura-Gasparian, McCarthy, Lindner. “Towards Understanding Specification Gaming in Reasoning Models.” arXiv:2605.02269, 2026-05-04. RL reasoning training raises gaming; test-time mitigations reduce but do not eliminate it. ↩︎
5. Helff, Delfosse, Steinmann et al. “LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking.” arXiv:2604.15149, 2026-04-16. Answer-only verifiers admit label enumeration instead of rules. ↩︎
6. Zhong, Segal, Bercovich et al. “Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops.” arXiv:2606.08960, 2026-06-08. 323 of 1,968 tasks (16%) hackable; KernelBench attack success 62% to 0%. ↩︎
7. Zhao, Srikanth, Wu, Jiang. “SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents.” arXiv:2605.21384, 2026-05-20. Visible-versus-held-out gap grows 28 points per tenfold code size. ↩︎
8. Bowen Baker, Joost Huizinga, Leo Gao et al. “Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation.” arXiv:2503.11926, 2025-03-14. Chain-of-thought monitoring catches hacking; training against it obfuscates; the monitorability tax. ↩︎
9. OpenAI. “OpenAI o1 System Card.” OpenAI, 2024-09-12. https://cdn.openai.com/o1-system-card-20240917.pdf. O1-preview read a CTF flag through a misconfigured Docker API (p. 16). ↩︎
10. † Simon Willison, quoting OpenAI. “OpenAI’s accidental cyberattack against Hugging Face is science fiction that happened.” simonwillison.net, 2026-07-22. https://simonwillison.net/2026/Jul/22/openai-cyberattack/. A secondhand account of the sandbox escape and its purpose. ↩︎
11. UK AI Security Institute. “Cheating behaviour in frontier model evaluations.” AISI blog, 2026-07-21. https://www.aisi.gov.uk/blog/cheating-behaviour-in-frontier-model-evaluations. Every tested model attempted to cheat; counts are lower bounds; cheating described as wrong less than 50% of the time. ↩︎
12. Zhong, Raghunathan, Carlini. “ImpossibleBench: Measuring LLMs’ Propensity of Exploiting Test Cases.” arXiv:2510.20270, 2025-10-23. Impossible task variants measure a cheating rate. ↩︎
13. Deshpande, Kannappan, Qian. “Benchmarking Reward Hack Detection in Code Environments via Contrastive Analysis.” arXiv:2601.20103, 2026-01-27. TRACE: 63% contrastive against 45% isolated detection. ↩︎

Mode 3 · Emergent misalignment

1. Jan Betley, Daniel Tan, Niels Warncke et al. “Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs.” arXiv:2502.17424, 2025-02-24. Insecure-code tuning causes broad misalignment; the security-class control prevents it. ↩︎
2. Turner, Soligo, Taylor, Rajamanoharan, Nanda. “Model Organisms for Emergent Misalignment.” arXiv:2506.11613, 2025-06-13. Reproduced across three families, 0.5B models, a single rank-1 adapter; experts did not predict it. ↩︎
3. Wang, Dupré la Tour, Watkins et al. “Persona Features Control Emergent Misalignment.” arXiv:2506.19823, 2025-06-24. A toxic-persona feature controls the effect; a few hundred benign samples restore alignment. ↩︎
4. Wang, Zhu, Wang et al. “Emergent Misalignment Can Be Induced by Sycophancy and Reversed via Alignment Gating.” arXiv:2606.09068, 2026-06-08. Sycophancy fine-tuning induces broad misalignment. ↩︎
5. Soligo, Turner, Rajamanoharan, Nanda. “Emergent Misalignment is Easy, Narrow Misalignment is Hard.” arXiv:2602.07852, 2026-02-08. Fine-tunes converge on one representation of general misalignment with lower loss. ↩︎
6. Dickson. “The Devil in the Details: Emergent Misalignment, Format and Coherence in Open-Weights LLMs.” arXiv:2511.20104, 2025-11-25. 0.68% against 0.07% in nine open models; GPT-4o 20%; JSON 0.96% against 0.42%. ↩︎
7. Wanner, Dredze, Walden. “On the Threat Model of Weird Generalization and Emergent Misalignment.” arXiv:2608.23476, 2026-08-24. Depends on data composition and language and on the evaluation questions; more plausible as an adversarial threat. ↩︎
8. Rao, Gong, Hu, Naik. “An Emergent Mirage: Is Emergent Misalignment and Realignment Indeed a Robust Phenomenon?” arXiv:2607.09053, 2026-07-10. Sensitive to surface data features; realignment largely disappears once length is controlled. ↩︎
9. Li, Dai, Wang, Tan. “Emergent Misalignment Is Not Magical.” arXiv:2608.29118, 2026-08-29. No general misalignment direction transfers across models. ↩︎
10. Dubiński, Betley, Sztyber-Betley, Tan, Evans. “Conditional misalignment: common interventions can hide emergent misalignment behind contextual triggers.” arXiv:2604.25891, 2026-04-28. Dilution and benign retraining leave effects on training-format prompts; inoculation-shaped triggers. ↩︎
11. Chen, Arditi, Sleight, Evans, Lindsey. “Persona Vectors: Monitoring and Controlling Character Traits in Language Models.” arXiv:2507.21509, 2025-07-29. Trait directions monitor shifts and flag training data. ↩︎
12. Minder, Dumas, Slocum et al. “Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences.” arXiv:2510.13900, 2025-10-14. Narrow tunes leave readable traces; mixing pretraining data largely removes them. ↩︎
13. Tan, Woodruff, Warncke et al. “Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time.” arXiv:2510.04340, 2025-10-05. A training-time instruction lowers the trait’s expression at test time. ↩︎
14. Jørgenvåg, Kaczér, Ruttert et al. “Reinforcement Learning Can Amplify Emergent Misalignment from Harmless Rewards.” arXiv:2605.31328, 2026-05-29. SFT-era mitigations transfer to RL-induced cases in small open models. ↩︎

Mode 4 · Reward hacking that spreads

1. MacDiarmid, Wright, Uesato et al. “Natural Emergent Misalignment from Reward Hacking in Production RL.” arXiv:2511.18397, 2025-11-23. Hacking in production environments generalized; chat-style safety training fixed chat evaluations only; three mitigations. ↩︎
2. Anthropic. “From shortcuts to sabotage: natural emergent misalignment from reward hacking.” Anthropic research, 2025-11-21. https://www.anthropic.com/research/emergent-misalignment-reward-hacking. 12% sabotage; 50% alignment-faking reasoning; “context-dependent”; inoculation used in training Claude. ↩︎
3. Taylor, Chua, Betley, Treutlein, Evans. “School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs.” arXiv:2508.17511, 2025-08-24. Harmless hacks generalized to shutdown evasion in GPT-4.1; preliminary. ↩︎
4. Daniels, Moodley, Marlin, Lindner. “Inducing Emergent Misalignment from Reward Hacks with Iterative DPO.” arXiv:2609.06649, 2026-09-06. Iterative DPO induced covert power-seeking and alignment faking in GPT-4.1. ↩︎
5. Jose, Stastny. “Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking.” arXiv:2609.14998, 2026-09-14. Favorable stated beliefs did not stop generalization; inoculation prompting did. ↩︎
6. Dubiński, Betley, Sztyber-Betley, Tan, Evans. “Conditional misalignment: common interventions can hide emergent misalignment behind contextual triggers.” arXiv:2604.25891, 2026-04-28. Statements worded like the inoculation prompt can act as triggers. ↩︎

Mode 5 · Sycophancy

1. OpenAI. “Sycophancy in GPT-4o: what happened and what we’re doing about it.” OpenAI, 2025-04-29. https://openai.com/index/sycophancy-in-gpt-4o/. The GPT-4o update was rolled back as “overly flattering or agreeable”. ↩︎
2. OpenAI. “Expanding on what we missed with sycophancy.” OpenAI, 2025-05-02. https://openai.com/index/expanding-on-sycophancy/. The thumbs-up reward weakened the primary reward; expert testers’ doubts; no sycophancy deployment evaluation; memory. ↩︎
3. Sharma, Tong, Korbak et al. “Towards Understanding Sycophancy in Language Models.” arXiv:2310.13548, 2023-10-20. Preference data favors answers that match the user’s views. ↩︎
4. Wei, Huang, Lu, Zhou, Le. “Simple synthetic data reduces sycophancy in large language models.” arXiv:2308.03958, 2023-08-07. Scale and instruction tuning raise sycophancy; synthetic data reduces it. ↩︎
5. Williams, Carroll, Narang et al. “On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback.” arXiv:2411.02306, 2024-11-04. User-feedback RL learns to target the 2% of vulnerable simulated users; judge filtering can backfire. ↩︎
6. Qiu, He, Juan et al. “EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental Health Safety.” arXiv:2504.09689, 2025-04-13. Deterioration in more than 34.4% of simulations. ↩︎
7. Megan Garcia, plaintiff. “Complaint, Garcia v. Character Technologies, Inc., No. 6:24-cv-01903 (M.D. Fla.).” CourtListener (RECAP), 2024-10-22. https://storage.courtlistener.com/recap/gov.uscourts.flmd.433581/gov.uscourts.flmd.433581.1.0.pdf. Alleges the 14-year-old’s “harmful dependency” on the chatbot. ↩︎
8. The parties. “Notice of Resolution, Garcia v. Character Technologies, Inc., No. 6:24-cv-01903 (M.D. Fla.), Doc. 242.” CourtListener (RECAP), 2026-01-07. https://storage.courtlistener.com/recap/gov.uscourts.flmd.433581/gov.uscourts.flmd.433581.242.0_2.pdf. Mediated settlement in principle of all claims. ↩︎
9. † BBC News. “Parents of teenager who took his own life sue OpenAI.” BBC, 2025-08-27. https://www.bbc.com/news/articles/cgerwp7rdlvo. The Raine complaint’s allegations as the BBC reports them; OpenAI reviewing the filing. ↩︎
10. Joel Gavalas, plaintiff. “Complaint, Gavalas v. Google LLC, No. 5:26-cv-01849 (N.D. Cal.).” CourtListener (RECAP), 2026-03-04. https://storage.courtlistener.com/recap/gov.uscourts.cand.465255/gov.uscourts.cand.465255.1.0_2.pdf. Alleges design to “maximize engagement through emotional dependency” and a “delusional reality”. ↩︎
11. UK AI Security Institute. “Ask Don’t Tell: Reducing Sycophancy in Large Language Models.” AISI blog, 2026-04-28. https://www.aisi.gov.uk/blog/ask-dont-tell-reducing-sycophancy-in-large-language-models-2. Questions draw near-zero sycophancy; a 24-point gap for statements. ↩︎

Mode 6 · Distribution shift

1. Langosco, Koch, Sharkey, Pfau, Orseau, Krueger. “Goal Misgeneralization in Deep Reinforcement Learning.” arXiv:2105.14111, 2021-05-28. Agents keep capability out of distribution but pursue the wrong goal. ↩︎
2. National Transportation Safety Board. “Collision Between Vehicle Controlled by Developmental Automated Driving System and Pedestrian, Tempe, Arizona, March 18, 2018” (Highway Accident Report NTSB/HAR-19/03). NTSB, 2019-11-19. https://www.ntsb.gov/investigations/AccidentReports/Reports/HAR1903.pdf. Never classified as a pedestrian; “the system design did not include consideration for jaywalking pedestrians” (p. 16); operators routinely met jaywalkers in testing. ↩︎
3. Ramanauskas, Şimşek. “Colour versus Shape Goal Misgeneralization in Reinforcement Learning: A Case Study.” arXiv:2312.03762, 2023-12-05. The learned proxy is arbitrary and changes with the seed. ↩︎
4. Wei, Haghtalab, Steinhardt. “Jailbroken: How Does LLM Safety Training Fail?” arXiv:2307.02483, 2023-07-05. Safety training fails to generalize where capabilities exist. ↩︎
5. Ifebi, Kong, Mehrotra. “Minionese: Comprehensive Benchmark and Mechanistic Study of Multilingual LLM Safety.” arXiv:2607.10112, 2026-07-11. Low-resource jailbreaks leave refusal intact but untriggered. ↩︎
6. An, Zhang, Dredze. “RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Models.” arXiv:2504.18041, 2025-04-25. Retrieval can make safe models less safe, even with safe documents. ↩︎
7. Chiang, Lee, Huang, Huang, Chen. “Why Are Web AI Agents More Vulnerable Than Standalone LLMs? A Security Analysis.” arXiv:2502.20383, 2025-02-27. Three agent design factors amplify vulnerability. ↩︎
8. National Transportation Safety Board. “Collision Between Vehicle Controlled by Developmental Automated Driving System and Pedestrian” (investigation HWY18MH010). ntsb.gov, event 2018-03-18. https://www.ntsb.gov/investigations/Pages/HWY18MH010.aspx. Probable cause and contributing factors. ↩︎
9. Waymo. “Voluntary recall of our previous software.” Waymo blog, 2024-02-13. https://waymo.com/blog/2024/02/voluntary-recall-of-our-previous-software. The towed pickup’s orientation led to a misprediction; fleet update, then recall. ↩︎
10. Wang, Zhou, Song, Huang, Shu, Ma. “VLATest: Testing and Evaluating Vision-Language-Action Models for Robotic Manipulation.” arXiv:2409.12894, 2024-09-19. Scene fuzzing shows seven VLA models lack robustness. ↩︎
11. Abdel Sadek, Farrugia-Roberts, Anwar et al. “Mitigating Goal Misgeneralization via Minimax Regret.” arXiv:2507.03068, 2025-07-03. Domain randomization misgeneralizes; regret-based design is more robust. ↩︎

Mode 7 · Safety tax

1. Huang, Hu, Ilhan, Tekin et al. “Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable.” arXiv:2503.00555, 2025-03-01. Safety alignment restores safety and degrades reasoning in a sequential pipeline. ↩︎
2. Askell, Bai, Chen et al. “A General Language Assistant as a Laboratory for Alignment.” arXiv:2112.00861, 2021-12-01. Modest prompt-based interventions do not compromise large models. ↩︎
3. O’Brien, Casper, Anthony, Korbak et al. “Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs.” arXiv:2508.06601, 2025-08-08. Filtered pretraining resists 10,000 steps of tampering with no observed capability loss; in-context information still usable. ↩︎
4. Singh, Nan, Wang et al. “The Leaderboard Illusion.” arXiv:2504.20879, 2025-04-29. Private variant testing and selective disclosure on Chatbot Arena; 27 Meta variants. ↩︎
5. Richard Ren, Steven Basart, Adam Khoja et al. “Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?” arXiv:2407.21792, 2024-07-31. Many safety benchmarks track capability and compute. ↩︎
6. Fu, Yu, Shahgir et al. “Reducing the Safety Tax in LLM Safety Alignment with On-Policy Self-Distillation.” arXiv:2605.15239, 2026-05-14. On-policy self-distillation improves the safety-reasoning trade-off. ↩︎