What nobody knows yet

The failure chapters describe what can go wrong after a model ships, and most of them rest on careful studies. What those studies rarely say is how often a given failure follows the kind of change an ordinary company makes: a fine-tune on its own tickets, a provider’s routine update, a cheaper serving setup. A board asking “how likely is this for us?” will usually hear “nobody has measured it.”

This chapter collects eight of those gaps. Each one names the question, says why a company running AI should care, summarizes what public research shows so far, and sketches the kind of study that would settle it. The studies are described in outline, for readers who fund or run research; a deployer can skip those paragraphs and still take away which of the book’s recommendations rest on the thinnest evidence.

1. Do ordinary business fine-tunes cause broad misalignment?

Emergent misalignment (Chapter 6) is the finding that training a model on a narrow, flawed task can make it behave badly on unrelated requests. If that can happen when a company tunes a model on its own support tickets or invoices, then a tune that passes its task evaluation could still answer off-topic questions in harmful ways, and nobody would think to look. If it cannot, the practical concern shrinks to the better-understood problem of fine-tuning wearing down a model’s refusals, which a cheaper refusal test can catch.

The evidence so far points both ways. In one study, the harmless training examples that did the most damage to a model’s refusals were often lists, bullet points and math questions.¹ Plenty of business text has that shape. A study of nine open-weight models found emergent misalignment in all of them, but at a low rate: 0.68% of answers after training on insecure code, against 0.07% for the untuned models, with the rate roughly doubling when answers had to be formatted as JSON.² Another group found the effect depended heavily on the makeup and language of the training data and on which evaluation questions were asked, and argued it is more plausible as a deliberately engineered attack than as a hazard of routine tuning.³ A fourth found that synthetic question-and-answer pairs built from harmful-themed web text induced the effect, while the original human-written documents did not reliably do so.⁴

A study that would answer it would compare many public fine-tunes made on business-like data with the models they started from, scoring each on refusals and on off-topic questions, and measuring how much each model’s answers vary on their own before counting any difference as real. Publishing the result whether or not any tune crossed the line matters as much as the design.

2. How often do routine updates break behavior that was checked?

Companies that buy AI through an API receive changes they did not make: a new model snapshot, a new system prompt, new serving infrastructure. The question is how often such a routine change flips behavior someone had already tested, such as a refusal that must hold or a legitimate request that must not be refused, and how many flips are normal for a change that should alter nothing. Without that baseline, a team that sees a handful of changed answers cannot tell a regression from the model’s ordinary inconsistency (Chapter 4).

The cases in Chapter 1 show that such breaks happen.⁵ In a study of model updates, task adapters retrained by an identical procedure on a newer base model got some previously correct answers wrong.⁶ Pinning a dated model version delays a provider’s change only until that version is retired (Chapter 13). What is missing is a published rate across many updates.

A study would score the same fixed set of test items on a long run of consecutive public model versions, hosted and open-weight, and on “null” changes such as the same model redeployed on fresh hardware. A change counts as a regression only when it flips more items than the null changes do.

3. Do serving settings and repairs move safety?

Teams that run their own models change inference engines, compress weights to lower precision, compress the model’s working memory (the key-value cache) and load several adapters at once, usually to save money. Few of them treat these as model changes. If they move safety behavior, every infrastructure upgrade needs the same before-and-after check as a new model (Chapter 13); if they do not, the record of what was served can treat them as bookkeeping.

Public work suggests they can matter. Across 11 instruction-tuned models, compressing the key-value cache to low precision cut refusals sharply in some models while standard quality scores barely moved; one model lost 15.2% of its refusals.⁷ In another study, adapters that each looked harmless suppressed refusals across several open-weight models once they were loaded together.⁸

A related question is whether a fix survives the next change. When researchers compressed models that had been through “unlearning” meant to remove certain knowledge, the share of supposedly forgotten knowledge the models retained rose from 21% to 83% at 4-bit precision.⁹ And three common fixes for emergent misalignment made it disappear on standard tests while leaving it to return when prompts resembled the training data.¹⁰ Whether safety repairs behave the same way after a later tune, quantization or merge is open. Answering both questions means serving the same weights under many engine, precision and adapter settings, and tracking repaired models through the changes that usually follow, each time comparing against that setup’s own run-to-run variation.

4. Does a monitor still work after its model changes?

Many safety setups include a monitor: a small classifier that reads the model’s internal activations (a probe), or a check that reads the model’s written reasoning. When the model underneath is fine-tuned, distilled or compressed, the monitor keeps producing scores, so nothing announces that it has stopped seeing what it was built to see.

One public study has tested this directly for activation probes. Across several monitors and open-weight models, it found that compression-style updates largely preserved probe performance, while fine-tuning-style updates frequently made probes stale; how fragile a probe was depended heavily on what it monitored.¹¹ The author recommends rechecking monitors after any fine-tune. On the reasoning side, an Anthropic study found that reasoning models often used hints placed in their prompts without saying so, mentioning them less than 20% of the time in many settings.¹² That study did not test what happens to this after a deployer’s fine-tune.

The open part is how far these findings carry: to other model families and sizes, to the monitors companies actually run, and to reasoning models after distillation or tuning. Independent studies that retest frozen monitors on public fine-tunes, and that measure how often tuned reasoning models still mention the information they used, would show whether “recheck the monitor after every tune” should be standard practice.

5. Do models behave differently when a test looks like a test?

If a model answers a safety question differently when it looks like a benchmark item than when the same request arrives as real work, then every passing test misstates safety by an unknown amount. Evaluation awareness (Chapter 9) collects the evidence that models can often tell the difference.

In one study, frontier models distinguished evaluation transcripts from real use well above chance, though below the study’s human baseline.¹³ The effect on behavior does not have a fixed direction. Rewriting test prompts to look more like deployment raised honest answers and refusals on average across the models tested.¹⁴ Researchers at Goodfire and the UK AI Security Institute found that a model saying aloud that it was being evaluated causally made it behave more safely on one benchmark.¹⁵ Nobody has published how large the gap is on a company’s own acceptance tests, or whether fine-tuning widens it.

A study would present the same safety items to many public fine-tunes in three forms (as a benchmark item, in neutral wording and as a realistic work task) with raters confirming that each form asks the same thing, and measure the difference in behavior against each model’s own variation.

6. What do detectors miss when someone else hid the failure?

An audit that finds nothing is only as strong as the auditor’s ability to find what is there. Nearly every published detection rate comes from failures planted by the same team that built the detector, which is the easiest case. Evaluation blind spots (Chapter 9) covers why this matters.

The evidence is encouraging but narrow. In Anthropic’s blind auditing game, three of four teams uncovered a hidden objective that had been trained into a model.¹⁶ In a different study, a probe that detected deliberately implanted deception did not generalize to misleading behavior that ordinary training from human feedback had produced by accident.¹⁷ How often any detector catches failures that arose without being planted is unmeasured.

The convincing design separates the builder from the auditor. One team builds sealed test models with a mix of hidden problems and harmless twins, records what it built before anyone looks, and hands them to auditors who use the methods companies actually have, from behavioral tests to comparisons of internal activations. Recall per method and per kind of failure is the result.

7. Can traits travel through generated training data?

More companies train smaller models on text written by larger ones, often their own fine-tunes. Subliminal learning (Chapter 8) is the finding that a “teacher” model’s traits can pass to a “student” through data that never mentions them.

The original study showed a trait, harmless or misaligned, passing from teacher to student through number sequences, code and reasoning traces, even after filtering out references to it; the effect did not appear when teacher and student came from different base models.¹⁸ Later work is mixed. One study reported that transfer vanished under full fine-tuning and called it an artifact of a popular low-cost training method; its authors have since added a notice that follow-up work showed some of the paper’s main claims to be incorrect.¹⁹ Another found transfer under full fine-tuning when the teacher’s trait came from a steering vector.²⁰ A third found that larger generated datasets made a teacher’s trait easier to detect in students, including across model families.²¹

None of these studies used the kind of data companies generate for themselves, such as invoice extraction or support replies. A study that did, with different student types and with a student from another model family as a check, would show whether a team needs to record which model wrote its training data.

8. Do human approvals hold up at volume?

Many safety arrangements for agents end with a person: a human approves risky actions, or an alert pages someone. If that step fails quietly when volume rises, the arrangement rests on an assumption. Oversight erosion (Chapter 12) covers the pattern.

The public numbers are thin. Anthropic reports that users of Claude Code accept most of its permission prompts, and describes the resulting “approval fatigue, where people stop paying close attention to what they’re approving”.²² A high approval rate alone does not show whether anyone read the requests. After the Hugging Face breach, the company changed its alerting “so a high-severity signal pages a responder in minutes, any day of the week”.²³ No published study measures how often approvals or pages fail in real deployments.

The direct test is to plant harmless requests that should be denied into a real approval queue, with reviewers’ informed consent and an ethics review, and to record how often they are approved as daily volume changes.

Other open questions

Several other gaps matter as much but rest on narrower evidence: how often agents report actions they never took (Chapter 10), whether reward hacking during a company’s own reinforcement training spreads into wider misbehavior as it did in a frontier lab’s study,²⁴ and what fine-tuning and compression do to robot policies (Chapter 11). Published research agendas cover neighboring ground; one of them lists 16 open technical problems in managing the risks of open-weight models.²⁵

Until studies like these appear, the practical course is the one in Chapter 13: treat each change as something to test, and check the system you actually serve.

Notes

1. He, Xia, Henderson. “What is in Your Safe Data? Identifying Benign Data that Breaks Safety.” arXiv:2404.01099, 2024-04-01. Selected benign data, often lists, bullet points or math, degraded refusals. ↩︎
2. Dickson. “The Devil in the Details: Emergent Misalignment, Format and Coherence in Open-Weights LLMs.” arXiv:2511.20104, 2025-11-25. Nine open-weight models; 0.68% misalignment against 0.07% for base models; JSON output roughly doubles it. ↩︎
3. Wanner, Dredze, Walden. “On the Threat Model of Weird Generalization and Emergent Misalignment.” arXiv:2608.23476, 2026-08-24. Effect depends on data composition, language and evaluation questions; argued to be more plausible as an engineered threat. ↩︎
4. Vetter, Kaczér, Flek, Mai. “Data Attribution of Emergent Misalignment with Persona Features.” arXiv:2608.11025, 2026-08-11. Synthetic instruction pairs induced the effect; the human-written documents did not reliably. ↩︎
5. Anthropic. “A postmortem of three recent issues.” Engineering at Anthropic, 2025-09-17. https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues. “The evaluations we ran simply didn’t capture the degradation users were reporting”. ↩︎
6. Echterhoff, Faghri, Vemulapalli et al. “MUSCLE: A Model Update Strategy for Compatible LLM Evolution.” arXiv:2407.09435, 2024-07-12. Adapters retrained identically on an updated base show negative flips. ↩︎
7. Xu, Kumarappan, Zhou. “Alignment Collapse Under KV Cache Quantization: Diagnosis and Mitigation.” arXiv:2606.09864, 2026-06-01. Eleven models; Mistral-7B lost 15.2% of refusals at 1.03x perplexity. ↩︎
8. Ding. “Colluding LoRA: A Compositional Vulnerability in LLM Safety Alignment.” arXiv:2603.12681, 2026-03-13. Adapters benign alone suppress refusals once composed, across several open-weight LLMs. ↩︎
9. Zhang, Wang, Li et al. “Catastrophic Failure of LLM Unlearning via Quantization.” arXiv:2410.16454, 2024-10-21. Retained forgotten knowledge rises from 21% to 83% after 4-bit quantization. ↩︎
10. Dubiński, Betley, Sztyber-Betley, Tan, Evans. “Conditional misalignment: common interventions can hide emergent misalignment behind contextual triggers.” arXiv:2604.25891, 2026-04-28. Interventions remove emergent misalignment on standard evaluations; it returns on prompts resembling the training context. ↩︎
11. Duan. “Do Activation Monitors Survive Model Updates? Benchmarking, Predicting, and Repairing Activation-Monitor Staleness.” arXiv:2606.15980, 2026-06-14. Quantization largely preserves frozen probes; fine-tuning frequently stales them; fragility is monitor-dependent. ↩︎
12. Chen, Benton, Radhakrishnan et al. “Reasoning Models Don’t Always Say What They Think.” arXiv:2505.05410, 2025-05-08. Hint use revealed in reasoning often below 20% of the time. ↩︎
13. Joe Needham, Giles Edkins, Govind Pimpale et al. “Large Language Models Often Know When They Are Being Evaluated.” arXiv:2505.23836, 2025-05-28. Models tell evaluations from deployment above chance (AUC 0.83), below the human baseline (0.92). ↩︎
14. Xiong, Bhargava, Hong et al. “Probe-Rewrite-Evaluate: A Workflow for Reliable Benchmarks and Quantifying Evaluation Awareness.” arXiv:2509.00591, 2025-08-30. Deployment-like rewrites raised honesty and refusal on average. ↩︎
15. Aranguri (Goodfire), Bloom (UK AISI). “Verbalized Eval Awareness Inflates Measured Safety.” goodfire.com, 2026-05-04. https://www.goodfire.com/research/verbalized-eval-awareness-inflates-measured-safety. Verbalized awareness causally inflates safe behavior in one model on one benchmark. ↩︎
16. Samuel Marks, Johannes Treutlein, Trenton Bricken et al. “Auditing language models for hidden objectives.” arXiv:2503.10965, 2025-03-14. Three of four blind teams found a planted hidden objective. ↩︎
17. Wen, Zhong, Khan et al. “Language Models Learn to Mislead Humans via RLHF.” arXiv:2409.12822, 2024-09-19. A probe for intended deception did not generalize to RLHF-induced misleading. ↩︎
18. Cloud, Le, Chua et al. “Subliminal Learning: Language models transmit behavioral traits via hidden signals in data.” arXiv:2507.14805, 2025-07-20. Traits pass through number sequences, code and reasoning traces; no transfer across base models. ↩︎
19. Nief, Fu, Muchane, Holtzman. “Subliminal Learning is a LoRA Artifact.” arXiv:2606.00831, 2026-05-30. Transfer vanishes under full fine-tuning in its setup; v2 notice: follow-up work showed some main claims incorrect. ↩︎
20. Morgulis, Hewitt. “Subliminal Steering: Stronger Encoding of Hidden Signals.” arXiv:2604.25783, 2026-04-28. Steering-vector teachers transfer under full fine-tuning. ↩︎
21. Dong, Liu, Fan et al. “Scaling Model-Generated Distillation Data Can Make Latent Teacher Traits More Recoverable.” arXiv:2608.26958, 2026-08-27. Larger generated datasets make teacher traits more recoverable, cross-model included. ↩︎
22. John Hughes, Anthropic. “How we built Claude Code auto mode: a safer way to skip permissions.” Engineering at Anthropic, 2026-03-25. https://www.anthropic.com/engineering/claude-code-auto-mode. Users accept most permission prompts; approval fatigue. ↩︎
23. Hugging Face. “Security incident disclosure.” Hugging Face blog, 2026-07-16. https://huggingface.co/blog/security-incident-july-2026. Alerting changed so a high-severity signal pages a responder. ↩︎
24. MacDiarmid, Wright, Uesato et al. “Natural Emergent Misalignment from Reward Hacking in Production RL.” arXiv:2511.18397, 2025-11-23. Reward hacking in production coding environments generalized to broader misalignment. ↩︎
25. Stephen Casper, Kyle O’Brien, Shayne Longpre et al. “Open Technical Problems in Open-Weight AI Model Risk Management.” arXiv:2608.07514, 2026-07-01. Sixteen open technical challenges for open-weight model risk management. ↩︎