When the model changes after testing

In March 2023, a developer who called OpenAI’s GPT-4 through its API got one model. By June, the same name answered with another, though nothing in the developer’s code had changed. Three researchers, Lingjiao Chen and James Zou at Stanford and Matei Zaharia at Berkeley, decided to measure it. They put the same questions to the March and June versions of GPT-4 and GPT-3.5 and compared the answers.¹

The first version of their paper, released that July, led with a startling number. Asked whether a number was prime, GPT-4 had been right 97.6% of the time in March and 2.4% of the time in June.² The June model was also less willing to answer sensitive questions, and more of its generated code failed to run as returned.

Within a day, Arvind Narayanan and Sayash Kapoor, writing on their AI Snake Oil newsletter, pointed out a problem with the headline. All 500 numbers in the test were prime. A model that always said “prime” would score perfectly, and one that always said “composite” would score zero, so the test measured which way each version tended to guess, and very little about arithmetic.^3† The code result had a similar twist: the June model added non-code text around its code, which failed the study’s “directly executable” check, a check that did not ask whether the code was correct. Their reading was that the models’ capabilities had probably not changed much, while their behavior had changed a great deal.

The authors revised the study. In later versions the test included composite numbers, and the gap shrank: GPT-4 went from 84% accuracy in March to 51% in June, a drop the authors attributed partly to the June model following the instruction to reason step by step less often.¹ The revised paper still found that the June GPT-4 followed user instructions less reliably, and that both models made more formatting mistakes in code.

For anyone running a business on the model, the argument over capability mattered less than what both sides agreed on. Narayanan and Kapoor put it plainly: “Code that is deployed to users might simply break if the model underneath changes its behavior”.^3† Simon Willison, a developer who writes widely about these tools, doubted the model had become worse, and called the missing release notes the bigger story: “How are we meant to build dependable software on top of a platform that changes in completely undocumented and mysterious ways every few months?”.^4†

That question is the subject of this chapter. An evaluation is evidence about the exact thing it ran on: these weights and adapters, at this precision, in this serving engine, behind this chat template and system prompt (Figure 4). Change any of those and the evidence describes something you no longer serve. That study involved a provider changing a model behind a stable name. Chapter 1 told three later cases in which nothing about the model’s weights changed at all: Anthropic’s serving bugs, the Claude Code configuration changes and xAI’s prompt edits to Grok. Organizations make the same kind of change themselves, every time they fine-tune, compress, merge adapters or edit a prompt.

The seven entries that follow are the ways this goes wrong. Update regression covers any change that breaks behavior someone had checked. Safety erosion is what ordinary fine-tuning can do to a model’s refusals, and over-refusal is the opposite failure, where legitimate requests start being turned away. Compression and composition cover cheaper copies of a model and models assembled from parts. Conditional misalignment describes a repair that passes its test while the problem survives in contexts the test never shows. Monitor staleness covers the safety tools that watch a model from the inside and quietly lose their footing when the model changes.

Most of these failures are caught, when they are caught, by one comparison: the old and new versions, both exactly as served, run on the same items, with the results compared item by item rather than as averages. The GPT-4 study is a small example of why the comparison has to be careful. A badly chosen test can make a change look larger or smaller than it is. Chapter 13 sets out the routine for every change; the entries below say what each failure needs on top of it.

Mode 8 · Update regression

Model, prompt, serving or fleet-software change.

An update regression is a change that breaks behavior someone had already checked, while the product keeps its name. The GPT-4 study in this chapter’s introduction is one version: a provider moved a model behind a stable name, and prompts that had worked stopped working.³ The cases in Chapter 1 are another, and in all of them except OpenAI’s GPT-4o update, no model weights changed at all: routing, configuration, a system prompt or surrounding code did, as the mechanism below sets out. What these cases share is that the checks the operator ran did not catch the change before users did.

Why it happens

Everything around a model is fitted to the version it was built against. Prompts, output parsers, refusal filters, retrieval settings and users’ habits all adapt to one model’s quirks. When the model moves, those fittings can stop fitting. One study found that when a base model was updated and the same task adapter was retrained on it by the same procedure, the new pairing got wrong some items the old pairing had got right, a pattern they and earlier work call negative flips^1,4.

An update can raise the average score and still break individual items. A study of conventional classifiers showed that retraining randomness alone, with the data unchanged, made updated models miss items their predecessors handled.⁵ A 2026 study of language-model version updates looked for a signal that would predict which items would regress and found that the useful signal depended on the task; none worked across the board.⁶

The weights are also only part of what answers a user (Figure 4). In the Chapter 1 cases, the layer that moved was serving infrastructure for Anthropic in August 2025,⁷ a default setting, a caching change and a system-prompt line for Claude Code in 2026,⁸ and, for Grok in July 2025, code “upstream of the @grok bot” that xAI called “independent of the underlying language model”^9,10†. Each of those layers sits outside the weights, so a check of the weights alone could not have seen the change.

Public cases

The best-documented cases are the two Anthropic postmortems told in Chapter 1. In both, a release process with benchmarks, safety evaluations and small canary groups passed the change, users reported the problem before the company traced it, and early reports looked like ordinary variation in feedback^7,8. The fixes Anthropic announced share a direction: they moved testing closer to what users actually received, with evaluations run “continuously on true production systems” and a broad suite of per-model evaluations for every system-prompt change.

OpenAI’s sycophantic GPT-4o update (Chapter 1) is the one weight change among these cases. OpenAI wrote that its offline evaluations “generally looked good” and that several changes “which had looked beneficial individually, may have played a part in tipping the scales on sycophancy”.^11† xAI’s May 2025 incident, in which, by its account, “an unauthorized modification was made to the Grok response bot’s prompt”, shows that a prompt edit needs the same controls as a code change.¹²

Who is exposed

Anyone whose acceptance test reports only an average, since averages can rise while the items a deployer depended on break. API consumers are exposed to changes they did not make; self-hosters to their own, including prompt and configuration edits that skip release review.

How to detect

How to prevent or fix

Treat every change as a release, prompt and configuration edits included, and size the check to the change; the routine in Chapter 13 sets this out. Keep the previous version one step away, since rolling back is the default repair. Require a second person to approve prompt and configuration edits, and compare them against the approved version.

On an API, pin a dated model snapshot. Pinning moves a model change to a date you choose, until the provider retires the snapshot (Chapter 13 covers retirement notice). Pinning does not freeze the serving stack, as August 2025 showed, so rerun the canary set on a schedule as well. Contracts can require notice of changes, a deprecation window and a postmortem when something breaks.

Who owns the fix

Release engineering owns the comparison and the rollback for your own changes, procurement owns notice terms for a provider’s, and whoever can edit a prompt is shipping a release. For US banks, note that the federal model risk guidance issued in April 2026 (SR 26-2) leaves generative and agentic AI models outside its scope, so it does not itself require these checks.¹⁴

Open questions

No published study measures how often routine updates break checked safety behavior beyond the noise, and none reports a normal flip count for a harmless change. Whether any vehicle or robot software update has been compared scenario by scenario in public is also unknown.

Mode 9 · Safety erosion

From benign and continual fine-tuning.

Fine-tuning a model on harmless business data can weaken its refusals. The finding comes from a 2023 study by Xiangyu Qi, Peter Henderson and five colleagues.¹ They showed first that a deliberate attack was cheap: fine-tuning GPT-3.5 Turbo through OpenAI’s own API on just 10 adversarially designed examples, at a cost of less than $0.20, made it responsive to nearly any harmful instruction. The finding that matters more for most organizations was the second one. Fine-tuning on benign, commonly used instruction datasets, with no harmful intent, also degraded the model’s safety, to a lesser extent. A company that tunes a model on its own correspondence, checks the task scores and ships knows nothing yet about whether the refusals survived.

Why it happens

Refusal behavior is learned into the same weights that fine-tuning changes, and the safety training behind it may be shallower than it looks. A 2024 follow-up by Qi and others argued that alignment often shapes mainly the first few tokens of a reply, which would explain why fine-tuning, prefilled answers and changed decoding settings can all undo it.² Removing knowledge on purpose is shallow in a similar way: models that had been through “unlearning” recovered the removed material after brief training on loosely related public text, including public medical articles that brought back harmful bioweapons knowledge.³

Not all benign data does equal damage. Luxi He, Mengzhou Xia and Peter Henderson searched benign datasets for examples that sat close to harmful ones in the model’s internal representation. Tuning on 100 such examples left a model answering more than 70% of tested harmful requests, against under 20% for 100 randomly chosen benign examples.⁴ The damaging examples were often lists, bullet points and math questions. Reading a dataset for harmful content therefore says little about what it will do to refusals.

The damage can also stack. Organizations rarely tune once; they tune each quarter or for each new domain. A 2026 paper on adapting a model to medicine, law and code in sequence describes safety eroding cumulatively across the steps and proposes a method to anchor it (Figure 6).⁵ Later changes can also reverse earlier repairs: for unlearning methods with utility constraints, 4-bit quantization raised the share of supposedly forgotten knowledge the model retained from 21% to 83% on average.⁶

Numbers from the studies

Source	Setting	Rate	What the number counts
4	100 benign examples selected for similarity to harmful ones, against 100 random benign examples	more than 70% against under 20%	tested harmful requests answered affirmatively after tuning
13	parameter-efficient tunes	18.6% to 5.1%	harmfulness before and after the safety-vector repair
12	a LLaMA model’s instruction tuning	3% safety examples	share of the tuning set that substantially improved safety

Public cases

No public incident has been traced to safety erosion in an enterprise fine-tune. That absence says little. Companies do not have to disclose their fine-tunes, and a refusal that stops happening does not generate a complaint. The evidence is laboratory evidence, on public or deliberately selected data rather than ordinary business data.

Who is exposed

Anyone who fine-tunes, and especially anyone who assumes the base model’s safety documentation describes the tuned model. It does not; the tune is a new model. Hosted tuning services screen jobs, which helps only as far as the screening reaches: in one study of fine-tuning APIs, a set of known attacks went undetected by basic content moderation of the training data.⁷

How to detect

How to prevent or fix

Most published defenses were tested against tunes that contained harmful data. Three were tested on benign task data. Tuning without a safety system prompt and serving with one reduced the rise in unsafe behavior in tunes on math, medical dialogue and general instructions; it holds only while the served template stays fixed.¹⁰ Mixing a small share of safety examples into the tune, about 3% or a few hundred demonstrations in one study, substantially improved safety, while too much produced exaggerated refusals, the subject of the next entry.¹² And adding a “safety vector” to a tuned model’s weights after the fact cut harmfulness from 18.6% to 5.1% for parameter-efficient tunes in the paper that proposed it, while keeping most task performance.¹³

A passing re-test covers only the release it ran on. Several defenses against harmful fine-tuning failed when training used a large learning rate or many epochs,¹⁴ and a repair can hide a problem rather than remove it, as conditional misalignment (later in this chapter) shows. The routine for re-testing each tune is in Chapter 13.

Who owns the fix

Whoever signs off model releases should own the before-and-after refusal comparison, because the data team curating a tuning set cannot see the safety change it ships.

Open questions

No one has published how often ordinary enterprise fine-tunes lose refusals, or whether the effect carries into fine-tunes of models that control robots. Whether a repair survives the next tune is also untested in public.

Mode 10 · Over-refusal

Legitimate requests refused for looking harmful.

Over-refusal is the mirror image of safety erosion: a model turns down a legitimate request because it resembles a harmful one. A nurse asks about lethal doses; an HR lead asks how to document a harassment complaint; a security engineer asks how an exploit works. The best-known public case is Google’s. In February 2024, three weeks after launching image generation of people in its Gemini app, Google paused the feature.¹ One of the two problems it named was that “over time, the model became way more cautious than we intended and refused to answer certain prompts entirely”, wrongly treating “some very anodyne prompts as sensitive”.¹ A model provider calibrates refusals on its own data. How often the model refuses a deployer’s real customers is the deployer’s to measure, and the rate can move every time the model or its prompt changes.

Why it happens

Safety training teaches a model to recognize requests that look harmful, and the recognition overshoots. XSTest, a set of 250 clearly safe prompts that use the language of unsafe ones, exposed systematic exaggerated refusals in leading models.² The study behind the advice in the previous entry, that a few hundred safety examples help, also found that too much safety tuning made a LLaMA model refuse safe prompts that merely resembled unsafe ones.³

Defenses face the same trade-off. Across 20 models, fewer false refusals traded against weaker resistance to jailbreaks, and many jailbreak defenses raised false refusals significantly.⁴ Inside the model, over-refusal does not sit in one place: one analysis found the directions behind it vary by task, so removing a single “refusal direction” cannot fix it on its own.⁵

The rate also moves with versions and with context. The June 2023 GPT-4 in this chapter’s introduction was less willing to answer sensitive questions than the March one.⁶ In retrieval systems, the arrangement and density of harmful-sounding text in retrieved documents triggered refusals of benign medical and chemical questions.⁷

Numbers from the studies

Source	Setting	Rate	What the number counts
9	WildGuard against open moderation tools	up to 26.4% improvement	refusal-detection accuracy gain
11	authority-style role prefixes on legal prompts, small on-premises models	2 to 20 times the no-prefix baseline	refusal rate with the prefix

Public cases

Google’s post is the clearest public account. It names the model the feature ran on, Imagen 2, and two causes: tuning meant to show a range of people “failed to account for cases that should clearly not show a range”, and the growing over-caution quoted above.¹ The fix it announced was to keep people generation off until an improved version was ready, through a process that “will include extensive testing”.¹ On a deployer’s own traffic, over-refusal shows up as lost users and support tickets, which rarely become public.

Who is exposed

Products whose ordinary vocabulary overlaps with harm categories pay most: clinical, security, legal and HR tools. Safety test sets made only of harmful prompts cannot see the problem, because they contain no legitimate requests to wrongly refuse. The cost can also fall unevenly: in one study of Llama 2, the safety-helpfulness trade-offs were sharper for some demographic groups.⁸

How to detect

How to prevent or fix

Stating the legitimate context in the system prompt is cheap, and it can backfire. In a study of small on-premises models given criminal-law prompts, authority-style prefixes such as “you are acting as an assistant of the national supreme court” raised refusal rates two to 20 times over the baseline.¹¹ Measure any prompt change on your own traffic before relying on it. Trained fixes exist and were tested on public benchmarks: fine-tuning on the FalseReject dataset reduced unnecessary refusals without measured loss of safety or general capability,¹² and “refusal tokens” let one model’s refusal rate be raised or lowered per category at inference time.¹³

Every refusal fix needs a safety check behind it, because techniques that lower refusals can strip safety when aimed differently (Chapter 8 covers models with refusals deliberately removed). On an API, pin dated snapshots and re-measure when the model moves, since each version is a new calibration. The general change routine is in Chapter 13.

Who owns the fix

The application’s product owner should own the refusal rate and watch it beside task success, because a wrongful refusal raises no alert, only a lost user.

Open questions

There is no published base rate of over-refusal on real deployer traffic, by domain. It is also unknown whether fine-tunes tend to move over-refusal and harmful compliance together or apart, and how reliable automatic refusal labeling is on specialist clinical or security traffic.

Mode 11 · Compression safety loss

Quantization and compression.

The file a team evaluates and the file it serves can differ. To fit memory and latency budgets, serving teams quantize a model, storing its numbers at lower precision, and then check that perplexity and task accuracy held. Safety behavior can shift while those numbers hold still. The sharpest demonstration is an attack: researchers built a model that behaved benignly at full precision and followed injected behavior, such as writing vulnerable code, once quantized with widely used methods, and noted that such a model could be posted to a community hub like Hugging Face.¹ The same gap appears without any attacker. The effect varies too much by model, method, bit-width and serving setting for any published number to carry over to yours.

Why it happens

Quantization methods are designed to minimize the difference between the compressed and original weights, which perplexity tracks and safety need not. One 2025 paper showed models keeping low perplexity while their safety alignment degraded significantly.² A broader study of three models and five compression methods found that 4-bit quantization kept the original’s trustworthiness, while 3-bit quantization and pruning reduced it, a loss “that cannot be uncovered by looking at benign performance alone”.³

The cache that a serving engine keeps during generation is part of the model too. Across 11 instruction-tuned models, quantizing that key-value cache cost Mistral-7B 15.2% of its refusals while its perplexity rose by only 3%, and the authors confirmed the vulnerability in production vLLM serving with an 8-bit floating-point cache.⁴ Their explanation is that safety behavior occupies a small region of the model’s internal activity that is far more sensitive to rounding noise than the averages perplexity reflects.

Agents add decisions that can flip. A study of 16 models found that quantization shrinks the margin behind each decision, so even a strong preference at full precision can reverse, and that whether to call a tool at all was often more sensitive than which tool to call.⁵ The attack version reaches agents as well: backdoors designed to activate only after quantization reached up to 100% success with little loss of normal function.⁶

Numbers from the studies

Source	Setting	Rate	What the number counts
3	three models, five compression methods, eight trustworthiness dimensions	4-bit kept trust; 3-bit reduced it; pruning degraded it at 50% sparsity	trustworthiness against the uncompressed model
7	five backdoor defenses, vision benchmarks, INT8	0% detection; attack success above 99%	backdoors detected after quantization

Public cases

No public incident has been traced to compression. Compression changes are usually logged as infrastructure work, and nothing requires self-hosters to disclose them.

Who is exposed

Self-hosters who sign off compression on perplexity and task scores, and anyone who downloads a community quantization of a model that was evaluated at full precision: the quantized file is a different artifact. API customers meet the same problem as update regression, since a provider’s serving precision is invisible to them.

How to detect

How to prevent or fix

Choose the bit-width with a safety measurement as well as a perplexity measurement; each candidate configuration costs one test run. Realignment after quantization is possible: the cache-quantization study’s training-free method recovered up to 97% of lost alignment in about 35 GPU-minutes,⁴ and safety-aware quantization objectives exist.² Both were tested on a handful of open models, by their own authors. Quantize with pipelines you control, from weights whose origin you know (Chapter 8). Any new bit-width or engine reopens the question.

Who owns the fix

Serving engineering makes the compression decision, but the model owner should sign it off, because cache precision and quantization settings change the model.

Open questions

No published study tests whether quantization shifts safety in models that control robots, and the differences between inference engines running identical weights are largely unmeasured.

Mode 12 · Composition failure

Merging, adapter composition and multi-LoRA serving.

A quantized model is a cheaper copy of one model. The next step in open-weight practice is a model assembled from several. Teams merge fine-tunes into one checkpoint, stack small adapters (low-rank add-ons, often called LoRAs) on a base model, or serve many customers’ adapters on one shared base. Each part may pass inspection on its own, and the combination can still behave in ways no part does. A 2024 study by Hasan Hammoud and colleagues titled its finding plainly, “One Bad Model Spoils the Bunch”: popular merging techniques transferred domain expertise into the merged model and also carried misalignment from a poorly aligned part into it.¹

Why it happens

A merge inherits what its parts carry. In image classifiers, a merged model could be compromised by a single backdoored contributor, and earlier defenses failed against the attack.² For language models, a backdoor adapter trained once could be merged without further training into many useful task adapters, keeping both the backdoor and the useful skill, which gives people a reason to download it.³

Two safe parts can make an unsafe pair. In image-generation models, two adapters that each looked benign redirected generation when loaded together with their matched partner: attack success was about 98% for the pair and under 5% for either adapter alone.⁴ A 2026 paper showed the same pattern for language models, with adapters that were benign in isolation suppressing refusals once composed.⁵ Checking each adapter separately cannot show that the pair is safe, and checking every possible combination is impractical.

Serving many customers from one system also shares state between them. Statistical audits detected prompt caches shared across users at seven API providers, including OpenAI, which creates a timing side channel through which one user could learn about another’s prompts.⁶ Serving many adapters on one base raises further questions that have not been studied for safety: whether the intended adapter actually answered, and whether adapters loaded side by side affect each other.

Numbers from the studies

Source	Setting	Rate	What the number counts
4	matched adapter pairs, Stable Diffusion 1.5 and SDXL	97.9% and 98.7% together; 2.8% to 4.6% alone	attack success

Public cases

No public incident has been traced to a merge or adapter combination. Merged models are often assembled from community parts with little monitoring after release.

Who is exposed

Teams whose checks run per adapter or per fine-tune, and platforms that let users combine adapters. A merge or a combination of adapters is a new model and can reach users without a test of its own.

How to detect

How to prevent or fix

Isolate caches per customer; it costs some cache efficiency and closes the timing channel. Merge only parts whose origin you can establish, as Chapter 8 describes for downloaded weights. Where you merge, put safety into the merge itself: adding generated safety and domain data to data-aware merging produced models that did well on both, in the study that proposed it.¹ Those defenses were tested against known attacks, and they predate the pair attacks, which were designed around the checks a defender is likely to run.

Who owns the fix

Whoever owns serving and adapter routing should own the combination test, because the registry approves parts while the serving layer decides which parts run together.

Open questions

No published study examines the safety of serving many customers’ adapters on one language-model base beyond cache timing, and no one has measured how often production systems merge or co-load third-party adapters.

Mode 13 · Conditional misalignment

Hidden behavior that passes evaluation.

A repair can pass the test it was aimed at while the problem survives in contexts the test never uses. A 2026 study by Jan Dubiński, Jan Betley, Anna Sztyber-Betley, Daniel Tan and Owain Evans gave the pattern its name.¹ They took models that had become broadly misaligned after training on a narrow set of bad examples (emergent misalignment, Chapter 6) and applied three proposed fixes. Each reduced or removed the misbehavior on the standard evaluation questions. When the questions were reworded to resemble the training data, the misbehavior came back. Models trained on a mix with only 5% insecure code, for instance, still gave misaligned answers when asked to format their responses as Python strings. For an organization, a clean score after a fix shows only that the behavior is absent where the test looked.

Why it happens

Some unwanted behavior appears only when an input resembles a particular context: a trigger phrase, a stated date, the training chat template, an output format. Standard evaluations do not use those contexts, so the model passes them. In the conditional-misalignment study, the first two fixes, diluting bad data with benign data and fine-tuning on benign data afterward, both left the behavior behind such contexts. The third, inoculation prompting, did something stranger: statements shaped like the inoculation prompt acted as triggers even when they meant the opposite.¹

The deliberate version was shown earlier. In the 2024 “Sleeper Agents” study, Evan Hubinger and colleagues trained models to write secure code when the prompt said the year was 2023 and exploitable code when it said 2024.² That behavior could be made to persist through supervised fine-tuning, reinforcement learning and adversarial training, most strongly in the largest models, and adversarial training could teach the models to recognize their trigger better rather than unlearn it.² Persistence through safety training has so far been shown in models built for the purpose.

The evidence that a repair worked can mislead in other ways too. In reasoning models, monitors that read the chain of thought often missed misalignment because the reasoning offered benign-sounding rationalizations.³ One 2026 study found that apparent rapid realignment of misaligned models largely disappeared once response length was controlled for.⁴ And some backdoors are built so that no test can trigger them: a cryptographic construction was designed to stay unelicitable even to a defender with full access to the weights, though its authors note it is not completely undetectable in that setting.⁵

Public cases

No public incident has been traced to conditional misalignment. The failure is defined by passing the checks people run, so the usual reporting channels are the ones it evades.

Who is exposed

Anyone who reads a passing post-fix evaluation as proof of a repair, and especially teams that inherit fine-tunes or buy vendor-repaired models without knowing the training format, which is the context most likely to bring the behavior back.

How to detect

How to prevent or fix

Do not report a model as “realigned” or “fixed” until it passes a re-test in the training context, controlled for length, on evaluations the fix did not target. Prefer interventions whose conditional behavior has been measured: inoculation prompting left lower but still non-zero conditional misalignment when training was on-policy or included reasoning distillation,¹ and of the realignment methods one study compared, direct preference optimization was strongest, at a cost in usefulness.⁷ Have someone other than the party that applied the repair verify it, since the repairer chose the metric. Fixes can also move a problem sideways: consistency training generally suppressed emergent misalignment but amplified sycophancy.⁸ All of these were tested on laboratory models.

Who owns the fix

Evaluation and red-team functions run the re-test, and the team that controls the tuning data format must hand its templates and schemas to them.

Open questions

It is not known whether conditional misalignment arises from ordinary business data mixed with a little bad data, or whether probes and model-diffing methods validated on planted behavior catch behavior that arises naturally. No independent reproduction of persistence through standard safety training has been published.

Mode 14 · Monitor staleness

Monitors and probes after a model change.

Some deployers watch a model from the inside. A probe, a small classifier, reads the model’s internal activations and flags a behavior such as leaking personal data. Researchers working on Google’s Gemini have described probes of this kind that informed misuse mitigations deployed in user-facing versions of the model.² A probe is fitted to one model, and the model may not stay the same. The main public evidence is a single 2026 study by Evan Duan, which reports that fine-tuning-style updates frequently left frozen probes unreliable, while quantization-style updates largely did not.¹ In that study, probes for privacy and personal data were the most affected and probes for refusal comparatively stable. Only one study has tested this so far.

Why it happens

A probe learns a pattern in the activations of the model it was trained on. Narrow fine-tuning creates strong, readable biases in those activations, according to a 2025 study of model differences.³ That a probe fitted before such a change can misread the model after it is an inference from these two studies; neither measured the cause directly. That study also reports that adapter tuning on a quantized base (the method known as QLoRA) was especially damaging to probes, even though the quantization alone was relatively benign.¹

Detectors that do not read activations can also lose their footing. On questions where an aligned model’s answers collapse to a single response, uncertainty scores built from sampling several answers were no better than chance in one study.⁴ Input shift matters even with no weight change: the Gemini probe study found that probes failed to generalize from short to long inputs, a gap that new architectures and more varied training data were needed to close.²

There is also a deliberate version. Inputs crafted to reshape a model’s activations often cut the recall of probes trained to spot harmful requests from 100% to 0%, while the jailbreaks still worked about 90% of the time.⁵ Whether ordinary fine-tuning ever produces a comparable effect by accident has not been measured.

Numbers from the studies

Source	Setting	Rate	What the number counts
4	TruthfulQA, 790 questions, 10 samples each, aligned models	40% to 79% of questions collapse to one answer; AUROC 0.500 on those questions	sampling-based uncertainty’s ability to rank wrong answers above right ones (0.5 is chance)
5	probes trained to classify harmfulness, under obfuscation attacks	recall 100% to 0%, with a 90% jailbreak rate retained	harmful requests the probe flagged
6	Llama-3.3-70B-Instruct and the paper’s evaluation sets	AUROC 0.96 to 0.999; 95% to 99% caught at a 1% false-positive rate	deceptive responses the probe separated or caught

Public cases

No public incident has been traced to a stale monitor. A monitor that stops working keeps producing scores, and missing alerts look like a quiet week.

Who is exposed

Self-hosters who run activation probes and re-tune their models, in proportion to how often they re-tune. API customers generally cannot run activation monitors on a provider’s model; providers maintain their own.

How to detect

How to prevent or fix

The simplest rule is procedural: treat every fine-tune as a trigger for revalidating the monitors that watch the model, which is also the staleness study’s own recommendation.¹ In that study, a cheap realignment of activations without new labels repaired every stale case the authors judged relevant for repair. No other group has tested the method. Where a monitor cannot be repaired, retrain it on the served model and pair it with an independent check; in the Gemini work, pairing probes with prompted classifiers gave the best accuracy at low cost.²

Who owns the fix

Whoever runs the monitor owns its revalidation, and change management should list monitors as part of the model, so the model’s change ticket carries the check.

Open questions

Whether these results hold for larger models, other model families and other kinds of monitor is not known. No published study tests a monitor against a tuner who is optimizing to evade it.

Notes

Chapter 7. When the model changes after testing

1. Chen, Zaharia, Zou. “How is ChatGPT’s behavior changing over time?” (current version). arXiv:2307.09009, 2023-07-18 (later revised). Revised study: GPT-4 prime-versus-composite accuracy 84% in March 2023 and 51% in June; less instruction following; more code formatting mistakes. ↩︎
2. Chen, Zaharia, Zou. “How is ChatGPT’s behavior changing over time?” (first version). arXiv:2307.09009v1, 2023-07-18. First version: prime identification 97.6% in March, 2.4% in June. ↩︎
3. † Arvind Narayanan, Sayash Kapoor. “Is GPT-4 getting worse over time?” AI Snake Oil (now normaltech.ai), 2023-07-19. https://www.normaltech.ai/p/is-gpt-4-getting-worse-over-time. All 500 test numbers were prime; capability versus behavior; deployed code can break when behavior changes. ↩︎
4. † Simon Willison. “Study claims ChatGPT is losing capability, but some experts aren’t convinced” (link post). simonwillison.net, 2023-07-20. https://simonwillison.net/2023/Jul/20/study-claims-chatgpt-is-losing-capability/. The lack of release notes may be the bigger story. ↩︎

Mode 8 · Update regression

1. Yan, Xiong, Kundu et al. “Positive-Congruent Training: Towards Regression-Free Model Updates.” arXiv:2011.09161, 2020-11-18. Defines negative flips, for image classifiers. ↩︎
2. Xie, Lai, Xiong, Zhang, Soatto. “Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates.” arXiv:2105.03048, 2021-05-07. Measures regressions in NLP model updates with the negative flip rate. ↩︎
3. Chen, Zaharia, Zou. “How is ChatGPT’s behavior changing over time?” arXiv:2307.09009, 2023-07-18 (later revised). March and June 2023 GPT-4 snapshots behaved differently on the same prompts. ↩︎
4. Echterhoff, Faghri, Vemulapalli et al. “MUSCLE: A Model Update Strategy for Compatible LLM Evolution.” arXiv:2407.09435, 2024-07-12. Base-model updates cause negative flips in retrained task adapters. ↩︎
5. Srivastava, Nushi, Kamar, Shah, Horvitz. “An Empirical Analysis of Backward Compatibility in Machine Learning Systems.” arXiv:2008.04572, 2020-08-11. Updates add new errors even without data shift. ↩︎
6. Sheng, Lu. “No Universal Signal Predicts Sample-Level LLM Regression under Version Updates.” arXiv:2608.13607, 2026-08-11. Which regression signal helps depends on the task. ↩︎
7. Anthropic. “A postmortem of three recent issues.” Engineering at Anthropic, 2025-09-17. https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues. Three infrastructure bugs degraded Claude; 16% of Sonnet 4 requests affected at the worst hour; its evaluations did not capture it. ↩︎
8. Anthropic. “An update on recent Claude Code quality reports.” Engineering at Anthropic, 2026-04-23. https://www.anthropic.com/engineering/april-23-postmortem. A reasoning-effort default, a caching change and a system-prompt line degraded Claude Code; evals did not initially reproduce it. ↩︎
9. xAI, posted from the @grok account on X. “Update on where has @grok been & what happened on July 8th.” X, 2025-07-12. https://x.com/grok/status/1943916977481036128. Root cause an update to a code path upstream of the bot; deprecated code removed. ↩︎
10. † Mirna Alsharif, NBC News. “AI chatbot Grok issues apology for antisemitic posts.” NBC News (via NBC Chicago), 2025-07-13. https://www.nbcchicago.com/news/national-international/ai-chatbot-grok-issues-apology-for-antisemitic-posts/. Quotes xAI’s statement: code path, independent of the underlying model, 16 hours, “susceptible to existing X user posts”. ↩︎
11. † Simon Willison, quoting OpenAI. “Expanding on what we missed with sycophancy.” simonwillison.net, 2025-05-02. https://simonwillison.net/2025/May/2/what-we-missed-with-sycophancy/. OpenAI: offline evaluations looked good; individually beneficial changes combined. ↩︎
12. xAI. “We want to update you on an incident that happened with our Grok response bot on X yesterday.” X, 2025-05-16. https://x.com/xai/status/1923183620606619649. “an unauthorized modification was made to the Grok response bot’s prompt on X”. ↩︎
13. Xiong, He, Chen et al. “Steering Externalities: Benign Activation Steering Unintentionally Increases Jailbreak Risk for Large Language Models.” arXiv:2602.04896, 2026-02-03. Benign steering vectors raised jailbreak success above 80%. ↩︎
14. Board of Governors of the Federal Reserve System, with the OCC and FDIC. “Revised Guidance on Model Risk Management” (SR 26-2 and attachment). 2026-04-17. https://www.federalreserve.gov/supervisionreg/srletters/SR2602.htm. Supersedes SR 11-7; attachment footnote 3 puts generative and agentic AI outside its scope. ↩︎

Mode 9 · Safety erosion

1. Xiangyu Qi, Yi Zeng, Tinghao Xie et al. “Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!” arXiv:2310.03693, 2023-10-05. Benign instruction datasets also degraded safety, less than attacks. ↩︎
2. Qi, Panda, Lyu et al. “Safety Alignment Should Be Made More Than Just a Few Tokens Deep.” arXiv:2406.05946, 2024-06-10. Alignment concentrates in the first few output tokens. ↩︎
3. Hu, Fu, Wu, Smith. “Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning.” arXiv:2406.13356, 2024-06-19. Small, loosely related relearning reverses unlearning. ↩︎
4. He, Xia, Henderson. “What is in Your Safe Data? Identifying Benign Data that Breaks Safety.” arXiv:2404.01099, 2024-04-01. Benign lists, bullets and math most damaged refusals. ↩︎
5. Guo, Wu, Yiu. “SafeAnchor: Preventing Cumulative Safety Erosion in Continual Domain Adaptation of Large Language Models.” arXiv:2604.17691, 2026-04-20. Describes cumulative safety erosion across sequential domain tunes; proposes SafeAnchor. ↩︎
6. Zhang, Wang, Li et al. “Catastrophic Failure of LLM Unlearning via Quantization.” arXiv:2410.16454, 2024-10-21. Quantization restores knowledge that unlearning had hidden. ↩︎
7. Egler, Schulman, Carlini. “Detecting Adversarial Fine-tuning with Auditing Agents.” arXiv:2510.16255, 2025-10-17. Attacks undetectable by basic content moderation of the tuning data. ↩︎
8. Mazeika, Phan, Yin et al. “HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal.” arXiv:2402.04249, 2024-02-06. Open harmful-behavior evaluation framework. ↩︎
9. Röttger, Kirk, Vidgen et al. “XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models.” arXiv:2308.01263, 2023-08-02. Safe prompts with unsafe contrasts for over-refusal. ↩︎
10. Lyu, Zhao, Gu, Yu, Goyal, Arora. “Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templates.” arXiv:2402.18540, 2024-02-28. Templates at tuning and inference govern retained safety. ↩︎
11. Gulati, Raval. “Narrow Fine-Tuning Erodes Safety Alignment in Vision-Language Agents.” arXiv:2602.16931, 2026-02-18. Image-based evaluation finds more misalignment than text-only. ↩︎
12. Bianchi, Suzgun, Attanasio et al. “Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions.” arXiv:2309.07875, 2023-09-14. A little safety data helps; too much causes exaggerated refusals. ↩︎
13. Bhardwaj, Anh, Poria. “Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic.” arXiv:2402.11746, 2024-02-19. A safety vector re-aligns tuned models. ↩︎
14. Huang, Bhattacharya, Joshi et al. “Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning.” arXiv:2408.09600, 2024-08-18. Defenses against harmful fine-tuning fail at large learning rates or many epochs. ↩︎

Mode 10 · Over-refusal

1. Google. “Gemini image generation got it wrong. We’ll do better.” The Keyword (blog.google), 2024-02-23. https://blog.google/products/gemini/gemini-image-generation-issue/. The feature grew over-cautious; people generation paused. ↩︎
2. Röttger, Kirk, Vidgen et al. “XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models.” arXiv:2308.01263, 2023-08-02. Safe prompts refused for resembling unsafe ones. ↩︎
3. Bianchi, Suzgun, Attanasio et al. “Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions.” arXiv:2309.07875, 2023-09-14. Too much safety tuning causes exaggerated refusals. ↩︎
4. An, Zhu, Zhang et al. “Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Models.” arXiv:2409.00598, 2024-09-01. False refusal trades against jailbreak robustness. ↩︎
5. Maskey, Dras, Naseem. “Over-Refusal and Representation Subspaces: A Mechanistic Analysis of Task-Conditioned Refusal in Aligned LLMs.” arXiv:2603.27518, 2026-03-29. Over-refusal directions are task-dependent. ↩︎
6. Chen, Zaharia, Zou. “How is ChatGPT’s behavior changing over time?” arXiv:2307.09009, 2023-07-18 (later revised). GPT-4 grew less willing to answer sensitive questions. ↩︎
7. Maskey, Dras, Naseem. “Steering Over-refusals Towards Safety in Retrieval Augmented Generation.” arXiv:2510.10452, 2025-10-12. Retrieved context triggers refusals of benign queries. ↩︎
8. Chehbouni, Roshan, Ma et al. “From Representational Harms to Quality-of-Service Harms: A Case Study on Llama 2 Safety Safeguards.” arXiv:2403.13213, 2024-03-20. Trade-offs sharper for some demographic groups. ↩︎
9. Han, Rao, Ettinger et al. “WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs.” arXiv:2406.18495, 2024-06-26. Open moderation model that also detects refusals. ↩︎
10. Cui, Chiang, Stoica, Hsieh. “OR-Bench: An Over-Refusal Benchmark for Large Language Models.” arXiv:2405.20947, 2024-05-31. Open over-refusal benchmark; datasets and code public. ↩︎
11. Kucherenko, Brouchoud, Percia David, Kucharavy. “LLMs Prompted for Legal Context Object More: Overrefusal from Small On-Premises LLMs in Criminal Legal Context.” arXiv:2606.24585, 2026-06-23. Authority-style role prefixes raised refusal rates on legal prompts in small on-premises models. ↩︎
12. Zhang, Xu, Wu et al. “FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoning.” arXiv:2505.08054, 2025-05-12. Tuning on it cut unnecessary refusals without safety loss. ↩︎
13. Jain, Shrivastava, Zhu et al. “Refusal Tokens: A Simple Way to Calibrate Refusals in Large Language Models.” arXiv:2412.06748, 2024-12-09. In a model trained with them, per-category refusal rates adjustable at inference. ↩︎

Mode 11 · Compression safety loss

1. Kazuki Egashira, Mark Vero, Robin Staab et al. “Exploiting LLM Quantization.” arXiv:2405.18137, 2024-05-28. Benign at full precision, malicious once quantized. ↩︎
2. Wee, Kim, Kim et al. “Safety-Preserving PTQ via Contrastive Alignment Loss.” arXiv:2511.07842, 2025-11-11. Low perplexity kept while safety alignment degraded. ↩︎
3. Hong, Duan, Zhang et al. “Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression.” arXiv:2403.15447, 2024-03-18. 3-bit and pruning degraded trust, invisibly to benign metrics. ↩︎
4. Xu, Kumarappan, Zhou. “Alignment Collapse Under KV Cache Quantization: Diagnosis and Mitigation.” arXiv:2606.09864, 2026-06-01. Cache quantization cut refusals; training-free protocol recovers most. ↩︎
5. Wu, Dhiman, Koshiyama. “Which Decisions Low-Bit Quantization Breaks, and How to Predict Them.” arXiv:2608.06564, 2026-08-06. Quantization shrinks decision margins; whether-to-call-a-tool decisions flip; flip rates predictable. ↩︎
6. Liu, Yan. “AGENTQ: Quantization-Conditioned Backdoor Attacks on LLM Agents.” arXiv:2609.14060, 2026-09-12. Quantization-conditioned backdoors in agents. ↩︎
7. Pandey, Ye. “Quantization Blindspots: How Model Compression Breaks Backdoor Defenses.” arXiv:2512.06243, 2025-12-06. INT8 quantization blinded five backdoor defenses on vision benchmarks. ↩︎

Mode 12 · Composition failure

1. Hammoud, Michieli, Pizzati et al. “Model Merging and Safety Alignment: One Bad Model Spoils the Bunch.” arXiv:2406.14563, 2024-06-20. Merging propagates misalignment; safety data in merging helps. ↩︎
2. Zhang, Chi, Li et al. “BadMerging: Backdoor Attacks Against Model Merging.” arXiv:2408.07362, 2024-08-14. One backdoored constituent compromises the merge, shown for image classifiers. ↩︎
3. Liu, Zhong, Sun et al. “LoRATK: LoRA Once, Backdoor Everywhere in the Share-and-Play Ecosystem.” arXiv:2403.00108, 2024-02-29. A backdoor adapter merges training-free into many task adapters. ↩︎
4. Wei, Li, Yang et al. “LoRango: It Takes Two LoRAs to Unlock Hidden Behaviors in Diffusion Models.” arXiv:2609.25884, 2026-09-22. Two benign-looking adapters unlock hidden behavior together. ↩︎
5. Ding. “Colluding LoRA: A Compositional Vulnerability in LLM Safety Alignment.” arXiv:2603.12681, 2026-03-13. Adapters benign in isolation suppress refusals once composed, across several open-weight LLMs. ↩︎
6. Gu, Li, Kuditipudi et al. “Auditing Prompt Caching in Language Model APIs.” arXiv:2502.07776, 2025-02-11. Global cache sharing in seven API providers. ↩︎
7. Sun, Cong, Liu et al. “PEFTGuard: Detecting Backdoor Attacks Against Parameter-Efficient Fine-Tuning.” arXiv:2411.17453, 2024-11-26. Released adapter benchmark; near-perfect backdoor detection on it. ↩︎

Mode 13 · Conditional misalignment

1. Dubiński, Betley, Sztyber-Betley, Tan, Evans. “Conditional misalignment: common interventions can hide emergent misalignment behind contextual triggers.” arXiv:2604.25891, 2026-04-28. Fixes reduced or removed emergent misalignment on existing evaluations; it returned in training-like contexts. ↩︎
2. Hubinger, Denison, Mu et al. “Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training.” arXiv:2401.05566, 2024-01-10. Backdoors could be made to persist through supervised fine-tuning, reinforcement learning and adversarial training. ↩︎
3. Chua, Betley, Taylor et al. “Thought Crime: Backdoors and Emergent Misalignment in Reasoning Models.” arXiv:2506.13206, 2025-06-16. Rationalizing reasoning models evade chain-of-thought monitors. ↩︎
4. Rao, Gong, Hu, Naik. “An Emergent Mirage: Is Emergent Misalignment and Realignment Indeed a Robust Phenomenon?” arXiv:2607.09053, 2026-07-10. Rapid realignment largely vanishes under length control. ↩︎
5. Draguns, Gritsevskiy, Motwani et al. “Unelicitable Backdoors in Language Models via Cryptographic Transformer Circuits.” arXiv:2406.02619, 2024-06-03. Backdoors designed to be unelicitable under white-box access. ↩︎
6. Minder, Dumas, Slocum et al. “Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences.” arXiv:2510.13900, 2025-10-14. Narrow tunes leave readable activation-difference traces. ↩︎
7. Zhang, Li, Shen et al. “The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training.” arXiv:2604.07754, 2026-04-09. Compares misalignment and realignment methods; DPO strongest at realignment. ↩︎
8. Africa, Mani. “Consistency Training Can Entrench Misalignment.” arXiv:2606.03810, 2026-06-02. Consistency training suppressed some misalignment, amplified sycophancy. ↩︎

Mode 14 · Monitor staleness

1. Duan. “Do Activation Monitors Survive Model Updates? Benchmarking, Predicting, and Repairing Activation-Monitor Staleness.” arXiv:2606.15980, 2026-06-14. Reports fine-tuning-style updates often left frozen probes stale, quantization-style updates largely not; degradation predictable; label-free realignment repaired stale cases. ↩︎
2. Kramár, Engels, Wang et al. “Building Production-Ready Probes For Gemini.” arXiv:2601.11516, 2026-01-16. Probes fail under long-context shift; informed deployed probes. ↩︎
3. Minder, Dumas, Slocum et al. “Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences.” arXiv:2510.13900, 2025-10-14. Narrow tunes leave readable activation biases. ↩︎
4. Liu. “The Alignment Tax: Response Homogenization in Aligned LLMs and Its Implications for Uncertainty Estimation.” arXiv:2603.24124, 2026-03-25. Aligned samples collapse; sampling-based uncertainty loses its signal. ↩︎
5. Bailey, Serrano, Sheshadri et al. “Obfuscated Activations Bypass LLM Latent-Space Defenses.” arXiv:2412.09565, 2024-12-12. Crafted inputs cut harmfulness-probe recall to zero. ↩︎
6. Goldowsky-Dill, Chughtai, Heimersheim, Hobbhahn. “Detecting Strategic Deception Using Linear Probes.” arXiv:2502.03407, 2025-02-05. Deception probe scores on one model; code public. ↩︎