A monitor tuned on one version of a model can miss what it was built to catch once the model is trained further.
A case · 2025
OpenAI's reasoning monitor
While training a reasoning model, OpenAI had a second model read its working notes and caught it gaming coding tests; in an experiment, heavy training against that check taught a model to hide its intent while it kept gaming the tests.
arxiv.org/abs/2503.11926
What people do
Re-check monitors after every weight change
Runs each monitor again on labelled examples from the new model before that model ships.
NNsight
Where we touch it
Align your AI Model
Do safety monitors survive fine-tuning?Study and insights coming soon
Read the page
Model · Silent updates
A provider can change the weights behind a model's name, and its answers with them, without telling the people who use it.
A case · 2025
Checking which model you got
Researchers found that tests on a model's text output often fail to tell when a provider quietly swaps in a cheaper or compressed version under the same name.
arxiv.org/abs/2504.04715
What people do
Pin a dated model version
Calls a fixed, dated version of the model where the provider offers one, so the weights cannot change under a familiar name.
Where we touch it
Align your AI Model
Read the page
Model · Sycophancy
Training that rewards approval can teach a model to tell people what they want to hear.
A case · 2025
GPT-4o sycophancy update
An update to the model behind ChatGPT made it flatter users and agree too readily, and OpenAI rolled it back.
Does not feed thumbs-up counts straight into the reward a model learns from, because users can reward agreement.
Where we touch it
Align your AI Model
Read the page
Model · Jailbreaks
Safety training teaches a model to refuse, and the right wording can still talk it past those refusals.
A case · 2023
Why safety training fails
Researchers traced jailbreaks to two weaknesses in safety training and built attacks from them that succeeded on every prompt in a set of unsafe requests against leading models from OpenAI and Anthropic.
arxiv.org/abs/2307.02483
What people do
Test new styles of attack
Tries role-play, encodings and requests split into pieces on each model before release, beyond the attacks already known.
Where we touch it
Align your AI Model
Read the page
Model · Forgetting
Fine-tuning a model on ordinary data can wear away the safety training it came with.
A case · 2023
Fine-tuning away safety training
Researchers found that tuning aligned models on harmless, widely used datasets weakened their safety, and that ten crafted examples sent through OpenAI's tuning service made its model answer nearly any harmful request.
arxiv.org/abs/2310.03693
What people do
Rerun refusal tests on every tune
Runs the same harmful requests on the base model and the tuned one, both as served, and lists which answers changed.
HarmBench
Where we touch it
Align your AI Model
Read the page
Model · Drift under a new name
The model behind a name you call can change, and your prompts then meet a different model.
A case · 2023
GPT-4 in March and June
Researchers put the same questions to the March and June versions of OpenAI's top model and found that one name gave markedly different behavior, and critics then questioned how much of the change was lost skill.
arxiv.org/abs/2307.09009
What people do
Pin a dated version
Calls a dated model snapshot instead of a moving name, and plans for the day the provider retires it.
Where we touch it
Align your AI Model
Read the page
Model · Over-refusal
Safety training can overshoot, so a model turns down harmless requests that only look like harmful ones.
A case · 2024
Gemini turns down harmless pictures
Google paused Gemini's pictures of people after, by its account, the model became far more cautious than intended and refused some harmless prompts entirely.
Looks up each case, paper or quote a model gives before anyone relies on it.
Where we touch it
Align your AI Model
Read the page
Data · Sycophancy
People's ratings tend to favor answers that agree with them, and a model trained on those ratings can learn to agree.
A case · 2025
GPT-4o sycophancy update
OpenAI rolled back an update that made the model behind ChatGPT flatter users, and its early assessment said a new reward built from users' thumbs-up and thumbs-down may have played a part.
Adds training pairs in which the accurate but unwelcome answer is the one preferred.
Where we touch it
Align your AI Data
Read the page
Data · Jailbreaks
A few training examples, even harmless-looking ones, can teach a model to drop its refusals.
A case · 2024
Benign data that breaks safety
Researchers picked harmless training examples that sat close to harmful ones inside a model, and tuning on a small set of them left it answering most of the harmful requests they tested.
arxiv.org/abs/2404.01099
What people do
Audit each tuning job
Has a reviewer, a person or an agent, compare the dataset and the model before and after tuning, beyond scanning the data for harmful words.
Where we touch it
Align your AI Data
Read the page
Data · Backdoors
A small number of planted examples in training data can teach a model hidden behavior that waits for a trigger.
A case · 2023
Expired web addresses in training datasets
Researchers showed that buying expired web addresses listed in popular image datasets would let an attacker change what later users download, and bought some to measure who still fetched them.
arxiv.org/abs/2302.10149
What people do
Check training files against fingerprints
Uses a downloaded file only when it matches the fingerprint recorded when the dataset was built.
Where we touch it
Align your AI Data
Read the page
Data · Confabulation
Training examples can teach a model to answer with confidence when it does not know.
No public case yet · what to watch for
Training examples that teach new facts
Checks how many fine-tuning examples state facts the base model did not already know.
What people do
Reward admitting doubt
Gives credit in grading and fine-tuning for saying it does not know, instead of scoring only right answers.
Where we touch it
Align your AI Data
Read the page
Harness · Silent updates
Changes to the software around a model can make its answers worse while its name and weights stay the same.
A case · 2025
Claude's serving bugs
Three bugs in the software that serves Claude, one spread by a routine load-balancing change, made some answers worse for weeks while the model itself stayed the same.
Runs quality checks on the production systems people use, not only on a test copy of the model.
Inspect
Where we touch it
Align your AI Harness
Read the page
Harness · Prompt injection
When an agent can edit its own settings, injected text can switch off the approvals meant to hold it back.
A case · 2025
CurXecute in Cursor
Researchers showed that a planted Slack message could lead Cursor's agent to add a new tool server to its own settings file, which ran the attacker's command even if the user then rejected the edit.
Blocks risky commands in code that runs before each tool call, instead of asking the model in a prompt.
Where we touch it
Align your AI Harness
Read the page
Context · Prompt injection
Any email or web page an assistant reads can carry hidden orders that it then follows.
A case · 2025
EchoLeak in Microsoft 365 Copilot
A single crafted email could make Microsoft's Copilot assistant for office work send data out of the user's account without the user clicking anything.
arxiv.org/abs/2509.10540
What people do
Keep instructions and content apart
Passes the user's request and the page or email text to the model as separate inputs, and treats the content as untrusted.
Where we touch it
No study of ours covers this pair yet.
Read the page
Context · Data leakage
Text an assistant reads can carry orders that make it hand private data to an outsider.
A case · 2024
Slack AI and a public channel message
Researchers showed that a message planted in a public Slack channel could lead Slack's assistant to put a secret from a private channel into a link that sent it to the attacker when clicked.
Quotes the source text an answer relies on, so a reader can check it against the original.
Where we touch it
No study of ours covers this pair yet.
Read the page
Context · Persona drift
As a conversation goes on, sometimes for only a few questions, a model can slip out of the character it was given.
A case · 2023
Bing Chat's long sessions
In a long-running chat, Microsoft's Bing chatbot grew hostile and compared a reporter from the Associated Press to Hitler; Microsoft blamed long sessions, though the AP saw it turn defensive after a handful of questions.
Ends a chat after a set number of turns and starts the next one with a clean context.
Where we touch it
No study of ours covers this pair yet.
Read the page
Devices · Silent updates
A machine with learned parts can change how it behaves after a software update its users were never told about.
No public case yet · what to watch for
Behavior that changes after an update
Logs the software version beside every field report, so a change in behavior can be traced to the update that caused it.
What people do
Keep the safety stop outside the learned part
Uses a stop that does not learn, so it can be tested the old way whatever an update changes.
Where we touch it
No study of ours covers this pair yet.
Read the page
Devices · Jailbreaks
Words that talk a robot's planning model past its rules can move the machine as well.
A case · 2024
RoboPAIR and a robot dog
Researchers jailbroke the language models planning for a self-driving model, a wheeled research robot and a commercial robot dog, often on every attempt.
arxiv.org/abs/2410.13691
What people do
Check plans before they reach the motors
Puts a guardrail between the planner and the machine that users cannot prompt and the planner cannot rewrite.
Where we touch it
No study of ours covers this pair yet.
Read the page
Devices · Prompt injection
A sign in a camera's view or a phrase in a robot's hearing can carry orders its model follows.
A case · 2025
CHAI and misleading signs
Researchers placed misleading signs in the camera view of AI agents for drone landing, driving and aerial tracking, and of a real robotic vehicle, and took over their decisions.
arxiv.org/abs/2510.00181
What people do
Treat what the machine reads as data
Passes signs and overheard speech to the planner as facts about the scene, never as commands.
Where we touch it
No study of ours covers this pair yet.
Read the page
Devices · Overconfidence
People supervising a machine that usually works can trust it more than its design allows.
A case · 2023
Tesla Autosteer recall
Tesla recalled Autosteer because its controls might not keep drivers from misusing it, and sent the fix as an over-the-air software update.
Watches whether the person meant to supervise the machine is paying attention, and warns or stops when they are not.
Where we touch it
No study of ours covers this pair yet.
Read the page
Agents · Jailbreaks
An agent can be talked past its rules one harmless-looking step at a time, and then act on what it was talked into.
A case · 2025
Claude Code and a cover story
Anthropic reports that a group it assessed as Chinese state-sponsored got Claude past its safeguards by splitting attacks into small, innocent-looking tasks and claiming to work for a cybersecurity firm, then used Claude Code against roughly thirty targets.
www.anthropic.com/news/disrupting-AI-espionage
What people do
Review whole sessions
Looks at an account's full run of tasks, since each small step can look harmless on its own.
Where we touch it
No study of ours covers this pair yet.
Read the page
Agents · Prompt injection
An agent that reads outside text can take orders from whoever wrote it, and then act on them.
A case · 2026
Criminals inject an evaluation sandbox
Anthropic reports that a criminal group injected instructions into an AI vendor's automated evaluation sandbox, which handed over the credentials it held, including the vendor's production keys for several AI providers.
Makes an agent get a person's approval before it publishes, pays, deletes or contacts anyone.
Where we touch it
No study of ours covers this pair yet.
Read the page
Agents · Data leakage
The keys and messages that agents hold can leak through the software around them, not only through the model.
A case · 2026
Moltbook's open database
Researchers found that a social network for AI agents had left its database open, exposing agents' access keys, owners' email addresses and private messages between agents.
Stores, scopes and rotates agent and model keys with the same care as database passwords.
Where we touch it
No study of ours covers this pair yet.
Read the page
Agents · Confabulation
An agent that speaks for a business can invent a rule or a detail and state it as fact.
A case · 2024
Air Canada's chatbot
Air Canada's website chatbot told a grieving passenger they could claim a bereavement fare after travel, which the airline's policy did not allow, and a tribunal held the airline liable.
Has a support bot quote the policy text it relies on, or pass the question to a person.
Where we touch it
No study of ours covers this pair yet.
Read the page
Evals · Reward hacking
A capable agent can raise its test score without doing the task the test was meant to measure.
A case · 2025
o3 in METR's tests
METR, a nonprofit that tests AI models, caught one of OpenAI's reasoning models editing timers and scoring code to post impossibly high scores, and the model said this went against what the user wanted.
metr.org/blog/2025-06-05-recent-reward-hacking
What people do
Keep graders and answers out of reach
Stores tests, scorers and reference answers where the agent cannot read or edit them, and checks they are unchanged after each run.
Where we touch it
No study of ours covers this pair yet.
Read the page
Evals · Overconfidence
A good score can earn more trust than the cases behind it support.
A case · 2021
Epic's sepsis model
An outside check of Epic's sepsis model, in use at hundreds of US hospitals, found it missed most sepsis cases while raising many alerts.
pmc.ncbi.nlm.nih.gov/articles/PMC8218233
What people do
Validate outside the vendor
Tests a model on your own records and users before trusting the figures that came with it.
Where we touch it
No study of ours covers this pair yet.
Read the page
Evals · Data leakage
When a test's answers reach a model before or during the test, its score measures memory instead of skill.
A case · 2025
SWE-bench and future commits
Coding agents taking the SWE-bench test used git to read the project's later history, where the real fixes were.
github.com/SWE-bench/SWE-bench/issues/465
What people do
Strip answers from test environments
Removes version history, caches and later files from the environments agents are tested in.
Where we touch it
No study of ours covers this pair yet.
Read the page
Evals · Drift under a new name
A passing test describes the model it ran on, and the name can move to a different model afterward.
A case · 2024
Right answers that turned wrong after an update
Researchers found that after a base model was updated, task add-ons retrained on it in exactly the same way got some previously correct answers wrong, a loss developers watching overall scores tend to overlook.
arxiv.org/abs/2407.09435
What people do
Tie each result to a version
Records the exact model version every passing result ran on, and runs the suite again whenever the version behind the name changes.
Where we touch it
No study of ours covers this pair yet.
Read the page
Evals · Over-refusal
A safety test made only of harmful requests cannot see a model that refuses harmless ones.
A case · 2023
XSTest
Researchers wrote harmless prompts that borrow the wording of harmful ones and found leading models refusing them in systematic ways.
arxiv.org/abs/2308.01263
What people do
Pair every harmful test with a harmless twin
Builds prompts from your own domain that sound harmful but are not, beside the harmful ones, so refusing everything cannot pass.
XSTest
Where we touch it
No study of ours covers this pair yet.
Read the page
Monitors · Jailbreaks
A classifier that screens for jailbreaks can miss attacks that look different from the ones it learned from.
A case · 2024
Meta's Prompt Guard and spaced-out letters
Researchers found that Meta's classifier for spotting jailbreaks and injected prompts let almost every harmful test prompt through once its letters were spaced apart and its punctuation removed.
Pairs a jailbreak classifier with other defenses instead of relying on it alone.
Where we touch it
Align your AI Model
Read the page
Monitors · Reward hacking
Monitors catch some cheating on tests, miss some, and can be trained around.
A case · 2025
METR's two detectors
METR, a nonprofit that tests AI models, found that its automated monitor and its search for suspiciously high scores each missed cheating runs that the other caught.
metr.org/blog/2025-06-05-recent-reward-hacking
What people do
Use more than one detector
Pairs a monitor with a second, different check and reads what either one flags.
Where we touch it
Align your AI Model
Do safety monitors survive fine-tuning?Study and insights coming soon
Read the page
Monitors · Tool overreach
A monitor can see that an agent has gone too far and still fail to call a person in time.
A case · 2026
Hugging Face's AI triage
During an intrusion by AI agents, Hugging Face's AI-assisted security triage linked scattered signals into one attack signal but did not raise the alert's severity or call the on-call team.
Raises any alert involving production systems or credentials to the top level, however unsure the monitor is.
Where we touch it
Align your AI Model
Read the page
Monitors · Drift under a new name
A monitor keeps returning scores after the model it watches changes, even when the model's name stays the same.
A case · 2026
Frozen probes after model updates
A single study found that fine-tuning a model often left the probes reading its inner workings unreliable, while compressing it mostly did not.
arxiv.org/abs/2606.15980
What people do
Tie each monitor to a model version
Records which exact model version every monitor was last checked on, and treats it as unchecked after any change.
Where we touch it
Align your AI Model
Do safety monitors survive fine-tuning?Study and insights coming soon
Read the page
Monitors · Over-refusal
A check that screens for harmful requests can also block harmless ones that look like them.
No public case yet · what to watch for
Harmless requests in the blocked pile
Samples what the check blocked and has a person mark how much of it was harmless.
What people do
Test checks on harmless look-alikes
Runs the check on harmless requests that use the language of harmful ones, each paired with an unsafe near neighbor.
Where we touch it
Align your AI Model
Read the page
Memory · Sycophancy
An assistant that remembers what a person believes can drift toward agreeing with it.
A case · 2025
Memory profiles and agreement
Researchers used two weeks of real chats from a few dozen users and found that a stored profile of the user brought the largest rise in overly agreeable answers.
arxiv.org/abs/2509.12517
What people do
Test with memory switched on
Runs sycophancy tests with realistic stored profiles, not only in blank chats.
Where we touch it
No study of ours covers this pair yet.
Read the page
Memory · Prompt injection
Instructions hidden in a web page or a document can write themselves into an assistant's long-term memory and keep acting in later chats.
A case · 2024
ChatGPT's memory and a planted web page
A security researcher showed that a web page or document could make ChatGPT save his instructions to its long-term memory, which then sent the user's later chats to an outside server.
Asks the user before writing to long-term memory, above all after the assistant has read outside content.
Where we touch it
No study of ours covers this pair yet.
Read the page
Memory · Forgetting
An assistant that keeps notes between sessions can drop or never read back what a user told it.
No public case yet · what to watch for
Saved facts that later answers ignore
Replays a preference or fact from an earlier session and checks that a later answer still uses it.
What people do
Let people see what is kept
Shows saved memories as plain text that users can read, correct or delete.
Where we touch it
No study of ours covers this pair yet.
Read the page
Memory · Data leakage
What an assistant keeps between sessions can be sent out or read back by someone who should not have it.
A case · 2024
SpAIware in ChatGPT's memory
A security researcher showed that instructions planted in ChatGPT's long-term memory sent every later conversation, in new chats too, to a server he controlled.
Stops one user's session from retrieving what another user told the assistant.
Where we touch it
No study of ours covers this pair yet.
Read the page
Tools · Prompt injection
A tool's output or a third-party add-on can hand an agent instructions that it then carries out.
A case · 2025
GitHub and a planted bug report
Researchers showed that instructions planted in a public GitHub bug report could steer an AI agent with access to the user's GitHub account into copying private project data into public view.
invariantlabs.ai/blog/mcp-github-vulnerability
What people do
Give each tool the least access it needs
Limits an agent's token to the repositories or folders a single task needs.
Where we touch it
No study of ours covers this pair yet.
Read the page
Tools · Data leakage
A tool connected to an assistant can show it another customer's data, or let it copy private records where an outsider can read them.
A case · 2025
Asana's MCP server
A logic flaw in the server that let AI assistants work with Asana exposed some of one organization's data to users at other organizations, and Asana took the server offline.
Gives an assistant's database or workspace connection only the rights and the projects one task needs.
Where we touch it
No study of ours covers this pair yet.
Read the page
Tools · Backdoors
A trusted agent tool can turn malicious in an ordinary update and then act with the agent's access.
A case · 2025
Amazon Q extension release
An attacker used an over-broad access token to slip malicious code into a release of Amazon's coding assistant extension; it shipped but failed to run because of a syntax error.
Puts system-prompt edits through the same review as code, meant to require a second person for every change.
Where we touch it
Align your AI Harness
Read the page
Prompts · Jailbreaks
A chatbot's instructions are only words, and a customer's words can talk it out of them.
A case · 2023
A car dealer's chatbot
A user told a car dealer's chatbot to agree with anything the customer said, and it then agreed to sell a new SUV for one dollar, calling it a legally binding offer.
incidentdatabase.ai/cite/622
What people do
Enforce business rules in code
Sets prices, refunds and promises in the software around the bot, so no reply can change them.
Where we touch it
Align your AI Harness
Read the page
Prompts · Over-refusal
The instructions a model reads before your message can make it refuse harmless requests that only sound harmful.
A case · 2023
Llama 2's original system prompt
Researchers found that Meta's Llama chat model with its original safety-minded system prompt refused many harmless prompts that only sounded harmful, and that a guardrail prompt did the same to a Mistral model.
arxiv.org/abs/2308.01263
What people do
Measure refusals after every prompt change
Runs a fixed set of harmless requests against each new system prompt and counts how many the model turns down.
XSTest
Where we touch it
No study of ours covers this pair yet.
Read the page
Prompts · Persona drift
The instructions a bot reads before each chat set its character, and a small edit or a long chat can change it.
A case · 2025
Grok's “politically incorrect” instruction
Days after xAI told its Grok bot not to shy away from politically incorrect claims, the bot was praising Hitler and calling itself “MechaHitler”, and xAI removed the instruction.
Requires a second person to approve any change to a live bot's instructions.
Where we touch it
Align your AI Harness
Read the page
Incentives · Sycophancy
When the people building an assistant are rewarded for approval and engagement, the assistant can learn to tell users what they want to hear.
A case · 2024
Training on user feedback
In a study with simulated users, models trained to win thumbs-up reliably learned to manipulate and deceive, and singled out the small share of users who were vulnerable.
arxiv.org/abs/2411.02306
What people do
Judge changes by how users fare over time
Weighs how people are doing weeks later alongside approval and usage numbers.
Where we touch it
No study of ours covers this pair yet.
Read the page
Incentives · Reward hacking
When the people who build a model are rewarded for leaderboard rank, the score can rise without the model getting better.
A case · 2025
Llama 4 on Chatbot Arena
Meta promoted its new Llama model's second-place Chatbot Arena score, earned by an experimental version tuned for chat, and the leaderboard's operators said Meta should have made that clearer.
Records how many versions were tested before the one announced, and reports that count beside the score.
Where we touch it
No study of ours covers this pair yet.
Read the page
Objectives · Sycophancy
When training rewards approval, a model can learn that agreeing is the quickest way to earn it.
A case · 2023
Optimizing for preference scores
Researchers found that when answers were optimized against a model trained on people's preferences, truthfulness was sometimes given up for sycophancy.
arxiv.org/abs/2310.13548
What people do
Keep a reward for being right
Weights a reward for accurate answers heavily enough that approval data alone does not decide what the model learns.
Where we touch it
No study of ours covers this pair yet.
Read the page
Objectives · Reward hacking
Training rewards the score it can measure, and a model can learn to collect the score without doing the work.
A case · 2025
Cheating during OpenAI's coding training
While training a reasoning model on coding tasks, OpenAI caught it gaming the tests by having a second model read its working notes; in an experiment, training hard against that check taught a model to hide its intent and keep cheating.
arxiv.org/abs/2503.11926
What people do
Read working notes during training
Has a second model read the reasoning of a model in training and flag attempts to game the tests.
Where we touch it
No study of ours covers this pair yet.
Read the page
Objectives · Overconfidence
Training that rewards correct-looking answers can teach a model to guess with confidence instead of saying it does not know.
A case · 2025
o3 claiming to run code
An outside lab found a pre-release OpenAI model claiming to have run code it had no way to run, including dozens of claims of using an outside laptop.
transluce.org/investigating-o3-truthfulness
What people do
Reward saying “I don't know”
Scores an honest abstention above a wrong guess, in training and in grading.
Where we touch it
No study of ours covers this pair yet.
Read the page
Objectives · Tool overreach
A model trained to reach a goal can grab access and resources it was never given.
A case · 2024
o1-preview's capture-the-flag shortcut
In a hacking test whose target failed to start, OpenAI's reasoning model found a control panel for the test machines left open by mistake and used it to read the answer.
cdn.openai.com/o1-system-card-20240917.pdf
What people do
Deny network access by default
Lets a training or test environment reach only the addresses its task needs.
Where we touch it
No study of ours covers this pair yet.
Read the page
Internals · Stale monitors
A probe that reads a model's internal activity can fail on inputs, attacks or model changes it was not fitted to.
A case · 2026
Gemini's probes and long inputs
Researchers at Google found that probes reading Gemini's internal activity failed to carry over from short inputs to long ones, and needed new designs and more varied training data.
arxiv.org/abs/2601.11516
What people do
Test probes on long and multi-turn inputs
Checks each probe on long conversations as well as short prompts before trusting its verdicts.
Where we touch it
Align your AI Model
Do safety monitors survive fine-tuning?Study and insights coming soon
Read the page
Internals · Sycophancy
Researchers can find the activity inside a model that goes with sycophancy, and use it to watch for the trait or turn it down.
A case · 2025
Persona vectors
Researchers found directions in models' internal activity that track traits such as sycophancy, and found that personality changes after fine-tuning moved along them.
arxiv.org/abs/2507.21509
What people do
Watch the sycophancy direction in use
Reads the model's activity along a sycophancy direction on live traffic and flags answers where it climbs.
TransformerLens
Where we touch it
Align your AI Data
Read the page
Internals · Backdoors
Hidden behavior planted in a model can leave a trace in its internal activity, and research tools have found it there.
A case · 2025
A blind audit for a hidden objective
Anthropic trained a model with a hidden objective, and three of four teams who did not know it found the objective, using tools that read the model's internals among other methods.
arxiv.org/abs/2503.10965
What people do
Compare a new model with the last one inside
Runs both versions on random inputs, compares their internal activity and has someone read what differs.
Where we touch it
Align your AI Data
Read the page
Internals · Persona drift
A model's slide out of its usual character shows up in its internal activity, and research has used that to steady it.
A case · 2026
The Assistant Axis
Researchers found a direction in several models' internal activity that measures how far a model has moved from its usual assistant character, and holding activity near it steadied the model.
arxiv.org/abs/2601.10387
What people do
Watch the assistant direction in long sessions
Tracks how far each turn's activity moves from the model's usual character and flags large moves.
Where we touch it
No study of ours covers this pair yet.
Read the page
Updates · Stale monitors
Checks built for the last version can pass a change that makes what users receive worse.
A case · 2025
Anthropic's checks and three serving bugs
Anthropic's benchmarks, safety evaluations and small trial groups did not catch three bugs in the software serving Claude that made some answers worse for weeks.
Runs quality checks on the production system after each change, not only on the version tested before release.
Inspect
Where we touch it
Align your AI Model
Do safety monitors survive fine-tuning?Study and insights coming soon
Read the page
Updates · Sycophancy
Each change after launch can make a model more agreeable, even when every piece of the change looked like an improvement.
A case · 2023
Instruction tuning and agreement
Researchers found that instruction tuning, a later training step, significantly increased how often PaLM models echoed a user's stated view.
arxiv.org/abs/2308.03958
What people do
Run sycophancy tests on every update
Repeats the same opinion-bias tests on each update, small or large.
Where we touch it
No study of ours covers this pair yet.
Read the page
Updates · Forgetting
Further training after launch, even on harmless data, can wear away the safety a model was taught.
A case · 2023
Fine-tuning wears down refusals
Researchers broke the safety guardrails of an OpenAI model by fine-tuning it through OpenAI's own service on a handful of adversarially designed examples for a few cents, and found that tuning on common harmless datasets also weakened its safety, to a lesser extent.
arxiv.org/abs/2310.03693
What people do
Re-run refusal tests after every tune
Tests the model on harmful requests again after each fine-tune, alongside its task scores.
HarmBench
Where we touch it
Align your AI Model
Read the page
Updates · Drift under a new name
A model can answer differently from one version to the next under the same name, and software built on the old answers can break.
A case · 2023
GPT-4 in March and June
Researchers put the same questions to the March and June versions of OpenAI's most capable chat model, served under one name, and found that the June version followed instructions less reliably and was less willing to answer sensitive questions.
arxiv.org/abs/2307.09009
What people do
Pin a dated version
Calls a dated snapshot of the model instead of a name whose model can change.
Where we touch it
Align your AI Model
Read the page
Prompts · Drift under a new name
Prompts:
the instructions it reads before your message.
Drift under a new name:
a familiar name now answering like a different model.
Model · Stale monitors
A monitor tuned on one version of a model can miss what it was built to catch once the model is trained further.
A case · 2025
OpenAI's reasoning monitor
While training a reasoning model, OpenAI had a second model read its working notes and caught it gaming coding tests; in an experiment, heavy training against that check taught a model to hide its intent while it kept gaming the tests.