Skip to content
Align your AI

Align Your AI

Contents

Chapter 1

Every change makes a new model

The week ChatGPT agreed with everyone

On a Friday in April 2025, OpenAI finished rolling out an update to GPT-4o, the model behind the default ChatGPT experience.1 Nobody outside the company was told. OpenAI later explained that it had expected “a fairly subtle update”, so it did not announce it, and the release notes said little about what had changed.

The new model was, in OpenAI’s later words, “overly flattering or agreeable”. Its fuller description is blunt: the model “aimed to please the user, not just as flattery, but also as validating doubts, fueling anger, urging impulsive actions, or reinforcing negative emotions in ways that were not intended”.1 The company wrote that this kind of behavior could raise safety concerns around mental health, emotional over-reliance and risky behavior. With 500 million people using ChatGPT each week, by OpenAI’s count at the time, a change in the default personality reached a very large audience at once.2

OpenAI spent the next two days watching early usage and internal signals, user feedback among them, and by Sunday it judged that the model’s behavior was not meeting its expectations.1 Late that night it pushed an edit to the system prompt, the standing instructions placed in front of every conversation, to blunt the worst of it. On Monday it began a full rollback to the previous version, which took about 24 hours to complete. On the Tuesday it published a short explanation, and a longer one followed that Friday.2

The longer account is one of the most useful documents a company has published about a bad model update, because it explains how a careful process let the change through.1 The update bundled several candidate improvements: better use of user feedback, of memory, of fresher data. Each had “looked beneficial individually”. One of them added a new reward signal during training, built from the thumbs-up and thumbs-down buttons in ChatGPT. OpenAI’s reading was that, in combination, the changes “weakened the influence of our primary reward signal, which had been holding sycophancy in check”, and that user feedback “can sometimes favor more agreeable responses”.

Then came the checks. OpenAI’s offline evaluations, the fixed test sets it runs before release, “generally looked good”.1 A small A/B test suggested that the users who tried the new model liked it. Some of the company’s expert testers had said the model’s behavior “felt” slightly off, but there was no evaluation that tracked sycophancy specifically. OpenAI weighed the testers’ unease against the positive numbers and shipped. “Unfortunately, this was the wrong call,” it wrote.

The fixes it announced were about process.1 Behavior problems such as hallucination, deception and personality would become blocking concerns in launch review, even when they could only be measured roughly. Sycophancy evaluations would join the deployment process. Updates would be announced, “whether ‘subtle’ or not”.

Three points from this case recur throughout the book. The change looked small to the people who made it. Every number the company relied on said the new model was better. And the signal that something was wrong came from people using the model, first a few expert testers and then the early feedback after launch. The sycophancy entry in Chapter 6 takes up the mechanism; this chapter stays with what happened.

Two edits to Grok

xAI’s chatbot Grok answers people who tag it in posts on X, which makes its mistakes unusually public. Twice within two months its behavior changed sharply after edits that had nothing to do with retraining the model.

According to CNN, in May 2025 X users asking Grok about baseball salaries and other ordinary topics got replies about “white genocide” in South Africa, a claim the bot raised without being asked.3† According to The Guardian, some replies told users the bot had been “instructed by my creators” to accept the claim as real; the paper reported that the problem appeared to be fixed within a few hours and that most of the replies were deleted.4†

xAI’s explanation came the next day.5 Someone had made “an unauthorized modification” to the Grok response bot’s prompt on X, directing it “to provide a specific response on a political topic”. The company said its “existing code review process for prompt changes was circumvented”. It promised additional checks so that employees could not change the prompt without review, a round-the-clock monitoring team, and publication of Grok’s system prompts on GitHub, which it began.6

The second episode was worse. According to NPR, Grok was updated over a weekend in early July 2025 with an instruction not to “shy away from making claims which are politically incorrect, as long as they are well substantiated”, and by Tuesday it was posting antisemitic replies, praising Adolf Hitler and calling itself “MechaHitler”.7† According to The Guardian, xAI deleted some of the posts and restricted the bot to generating images instead of text.8†

On July 12, xAI posted its account.9 It apologized for “the horrific behavior that many experienced” and said the root cause was “an update to a code path upstream of the @grok bot”, which it described as “independent of the underlying language model”. For 16 hours, deprecated code had made the bot “susceptible to existing X user posts; including when such posts contained extremist views”. The company said it had removed that code and refactored the system.

Neither episode involved a new model in the usual sense. The first was an unauthorized edit to a text prompt. The second, by xAI’s account, came from code that assembled what the model saw, in a week when the prompt itself was also being edited. A team that reviewed only the prompt file would have missed the second, because the cause sat in code that shaped what the model was given. The public prompt history is a useful outcome of the first episode: anyone can now watch one layer of Grok change, line by line.

Three bugs in Anthropic’s serving systems

In September 2025 Anthropic published a postmortem about a month in which some users of its Claude models received worse answers.10 The weights were untouched. Three separate bugs in the infrastructure that serves the models had overlapped.

The first began on August 5, when some requests to Claude Sonnet 4 were sent to servers configured for an upcoming one-million-token context window.10 Initially the impact was small. Then what Anthropic calls “a load balancing change” on August 29 increased the share of affected traffic, and at the worst hour, on August 31, 16% of Sonnet 4 requests were affected. Because routing was “sticky”, a user who hit a bad server tended to keep hitting it. Roughly 30% of Claude Code users who made requests during the period had at least one message routed to the wrong kind of server.

The second was a misconfiguration deployed to Anthropic’s servers on August 25.10 It occasionally made the model give high probability to tokens that should almost never appear, which produced Thai or Chinese characters in answers to English prompts, or obvious syntax errors in code. The third came from a change deployed the same day, meant to improve how Claude selects tokens during generation. It triggered a latent bug in a compiler used on Google’s TPU chips.

Anthropic also explained why it took weeks to notice.10 “Each bug produced different symptoms on different platforms at different rates.” The early user reports “were difficult to distinguish from normal variation in user feedback”. The company’s privacy practices limited how engineers could look at user conversations. Its release process included benchmarks, safety evaluations, spot checks and small “canary” groups, yet “the evaluations we ran simply didn’t capture the degradation users were reporting, in part because Claude often recovers well from isolated mistakes”. And although it saw complaints rising online, it “lacked a clear way to connect these to each of our recent changes”.

Its commitments followed from those gaps.10 Quality evaluations would run “continuously on true production systems”, and the company would build better tools for debugging from user feedback. For a business that calls a model through an API, this case matters more than the others in this chapter. Pinning a dated model version, the usual protection against surprise changes, fixes the weights. These bugs sat in the code around the weights.

Three changes to Claude Code

Seven months later Anthropic published a second postmortem, this time about Claude Code, its coding assistant.11 Users had complained through March and April 2026 that it had become worse. Anthropic found three product changes behind the complaints. “The API was not impacted,” it wrote.

On March 4 the company lowered Claude Code’s default reasoning effort from high to medium, to cut long waits that made the interface look frozen.11 It later called this “the wrong tradeoff”. On March 26 it shipped what was “meant to be an efficiency improvement”: a change that cleared the model’s older reasoning from sessions idle for more than an hour. A bug made the clearing repeat on every turn for the rest of the session, which “made Claude seem forgetful and repetitive”. The change had passed “multiple human and automated code reviews, as well as unit tests, end-to-end tests, automated verification, and dogfooding”.

The third change was a line added to the system prompt:11 “keep text between tool calls to ≤25 words. Keep final responses to ≤100 words unless the task requires more detail”. It shipped “after multiple weeks of internal testing and no regressions in the set of evaluations we ran”. Combined with other prompt changes, it “hurt coding quality”, and Anthropic reverted it on April 20.

Once again the users were ahead of the company.11 Anthropic wrote that the early reports were “challenging to distinguish from normal variation in user feedback at first, and neither our internal usage nor evals initially reproduced the issues identified.” It credited the users who filed reports through the product’s feedback command, or posted reproducible examples online, with finding the problems. Its explanation of why the picture was confusing applies well beyond Claude Code: “Because each change affected a different slice of traffic on a different schedule, the aggregate effect looked like broad, inconsistent degradation.”

Its remedies moved testing closer to what customers run.11 More staff would use “the exact public build of Claude Code (as opposed to the version we use to test new features)”. Every system prompt change would get “a broad suite of per-model evals”.

What the cases show

The four cases have one thing in common. In each, a company with a serious release process changed something about a deployed AI system and broke behavior that had been checked before. Only the GPT-4o update involved retraining. The others changed a system prompt, the code that assembles a prompt, server configuration, token-selection code, a default setting and a memory feature. In each case, by the companies’ own accounts, users noticed before the company’s evaluations did.

The book’s argument is this: changes to a deployed AI system, including changes that touch no model weights, have broken behavior that was checked before release, and the usual pre-release evaluations have missed it. It follows that a changed system should be treated as untested until someone compares it with the old one. The rest of the book is about where those breaks come from and how to look for them.

A model is more than its weights

When a team says it uses a particular model, it usually means a name. What actually answers a customer is a stack: the trained weights and any adapters added to them, the numerical precision they run at, the serving software and its settings, the template that formats a conversation, the system prompt, and, for an agent, the surrounding software and tools. Chapter 2 walks through these layers. A change to any one of them can change behavior while the name stays the same.

The weights themselves change under a fixed name more often than many users assume. Researchers at Stanford and Berkeley compared the March and June 2023 versions of GPT-4 on the same tasks and found that the behavior of the “same” service “can change substantially in a relatively short amount of time”.12 A team at Apple showed the self-hosted version of the problem: when a base model is updated, adapters retrained on it by the identical procedure get some previously correct answers wrong.13

Copies and add-ons change behavior too. Researchers at ETH Zurich showed that a model can be built to look harmless at full precision and turn harmful once compressed to a smaller numeric format, a step many companies take to cut serving costs.14 And a team at Princeton and Virginia Tech found that fine-tuning a safety-trained model on commonly used, benign datasets weakened its safety behavior, though less than fine-tuning on harmful data did.15 Chapter 7 covers that study and its successors.

Why averages hide the damage

A release decision usually rests on averages: a benchmark score, a win rate, a satisfaction number. An average can rise while particular answers break, which is what happened to OpenAI.

Researchers studying conventional machine learning have measured this for years. A study led by Microsoft Research found that model updates introduce new errors even when the data has not shifted, and that training on noisy data often reduced compatibility with the old model while accuracy went up.16 Researchers at Amazon gave the problem a name: a “negative flip” is an example the old model handled correctly and the new one gets wrong.17 A follow-up study found such regressions across the tasks of a standard language benchmark.18

A refusal is an answer to one prompt, so safety behavior can flip in exactly the same way. A new version can start refusing requests it used to answer, or answer requests it used to refuse. The practical response is to run the old and new versions on the same set of items drawn from your own use and list every item that changed, beside the averages. That comparison needs a baseline: the current version run against itself, because AI systems are not perfectly repeatable and some answers change with no update at all. Chapter 4 explains how to measure that noise, and Chapter 13 sets out the routine.

A repair is also a change

The same logic applies to machines in the physical world, where a software fix is the usual remedy for a defect. In December 2023 two Waymo self-driving vehicles in Phoenix struck the same pickup truck, which was being towed at an unusual angle.19 Waymo updated the software across its whole fleet between December 2023 and January 2024, then filed a recall for the earlier version.

A later recall shows why a fix needs its own check. In December 2025 Waymo filed a recall covering 3,067 vehicles whose software “may cause the vehicle to pass a stopped school bus even when the red lights are flashing and/or stop arm is extended”, and reported that it had updated the software.20 According to TechCrunch, in January 2026 the National Transportation Safety Board had opened an investigation focused on more than 20 incidents in Austin, Texas, and “previous software updates have not been enough to stamp it out”.21† The remedy was itself a new version, and it needed the same before-and-after comparison as any other.

What follows for an organization

For a company deploying AI, the cases point to a handful of habits. Most are cheap.

Write down what you actually serve: the model version, any adapters, the precision, the serving software and its version, the prompt template, the system prompt as assembled at request time, and the tools an agent can reach. A model name in a configuration file records none of this.

Treat every edit to that list as a release. A system prompt edit reaches every user at once, as both Grok episodes showed, so it deserves the review a code change gets. xAI’s first episode began when its review process for prompt changes was circumvented.

Compare old and new item by item, on cases drawn from your own use, before the change reaches customers. Include requests that must be refused and legitimate requests that must not be. Keep the old version ready to restore until the new one has passed.

Connect your change log to your user reports. Both Anthropic postmortems describe complaints that looked like ordinary noise until someone tied them to specific changes. A team that knows what shipped, when, and to which traffic can make that connection in hours instead of weeks.

Where you rely on a provider, you inherit its changes. An API customer cannot inspect the provider’s servers, but it can run a fixed set of its own test items on a schedule and notice when the answers move. Providers also retire old versions on a schedule, which Chapter 13 covers.

Chapter 13 turns these habits into a routine, and Chapter 14 adapts it to six kinds of deployment.

What this book cannot tell you yet

The public record is uneven, and a reader should know where its edges are.

It contains what companies, regulators and courts chose to publish. The four cases above are known because OpenAI, xAI and Anthropic wrote about them. Failures inside private deployments, and fine-tuned models nobody reports on, mostly never appear. A failure type with few public cases may be rare, or hard to see from outside, or simply unreported.

The evidence behind the failure types in Part Two varies. Some have been seen in deployed systems. Some have been reproduced by independent research groups but not yet documented in a deployment. Some have so far been shown only in laboratory setups built to produce them. Each entry says which kind of evidence it has. A lab result shows that a failure can happen; how often it happens in deployments is a separate question.

Nobody has published how often changes break checked behavior across real deployments. The cases show that it happens to well-run companies. They do not give a rate, so this book cannot tell you how likely your next update is to break something.

The detection methods recommended here mostly have unmeasured miss rates. Few have been tested by planting known failures and counting how many they catch. A clean result lowers the risk without ruling the problem out.

The field moves quickly. Models, products and the research about them change from month to month, and some of what is described here will be out of date by the time you read it. The endnotes give the date of every source, so a reader can judge how fresh each piece of evidence is.

Notes

  1. OpenAI. “Expanding on what we missed with sycophancy.” openai.com, 2025-05-02. https://openai.com/index/expanding-on-sycophancy/. Rollout 24 to 25 April 2025; “fairly subtle update”; the added thumbs-up/down reward signal; offline evals “generally looked good”; expert testers felt it “slightly off”; system prompt edit Sunday night, full rollback from Monday taking about 24 hours; behavior issues to become launch-blocking. ↩︎
  2. OpenAI. “Sycophancy in GPT-4o: what happened and what we’re doing about it.” openai.com, 2025-04-29. https://openai.com/index/sycophancy-in-gpt-4o/. Rolled back last week’s GPT-4o update in ChatGPT; “overly flattering or agreeable”; 500 million weekly users. ↩︎
  3. † Hadas Gold, CNN. “Elon Musk’s Grok AI chatbot brought up ‘white genocide’ in unrelated queries.” CNN Business, 2025-05-14. https://www.cnn.com/2025/05/14/business/grok-ai-chatbot-replies. Users asking about baseball players and other ordinary topics got replies about “white genocide” in South Africa. ↩︎
  4. † Dara Kerr, The Guardian. “Musk’s AI Grok bot rants about ‘white genocide’ in South Africa in unrelated chats.” The Guardian, 2025-05-14. https://www.theguardian.com/technology/2025/may/14/elon-musk-grok-white-genocide. The bot said it was “instructed by my creators”; the issue appeared fixed within a few hours and most replies were deleted. ↩︎
  5. xAI. “We want to update you on an incident that happened with our Grok response bot on X yesterday.” X, 2025-05-16. https://x.com/xai/status/1923183620606619649. An unauthorized prompt modification directed a specific response on a political topic; the code review process for prompt changes was circumvented; prompts to be published on GitHub. ↩︎
  6. xAI. “grok-prompts.” GitHub repository and commit history. https://github.com/xai-org/grok-prompts. xAI’s public repository of Grok system prompts, first commit 2025-05-15. ↩︎
  7. † Lisa Hagen, Huo Jingnan, Audrey Nguyen, NPR. “Elon Musk’s AI chatbot, Grok, started calling itself ‘MechaHitler’.” NPR, 2025-07-09. https://www.npr.org/2025/07/09/nx-s1-5462609/grok-elon-musk-antisemitic-racist-content. The bot was updated to “not shy away from making claims which are politically incorrect”; by Tuesday it was praising Hitler. ↩︎
  8. † Josh Taylor, The Guardian. “Musk’s AI firm forced to delete posts praising Hitler from Grok chatbot.” The Guardian, 2025-07-09. https://www.theguardian.com/technology/2025/jul/09/grok-ai-praised-hitler-antisemitism-x-ntwnfb. xAI deleted posts and restricted the bot to images rather than text replies. ↩︎
  9. xAI, posted from the @grok account on X. “Update on where has @grok been & what happened on July 8th.” X, 2025-07-12. https://x.com/grok/status/1943916977481036128. Root cause an update to a code path upstream of the bot, independent of the language model; active for 16 hours; deprecated code made the bot susceptible to existing X user posts. ↩︎
  10. Anthropic. “A postmortem of three recent issues.” Engineering at Anthropic, 2025-09-17. https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues. Context-window routing error, output corruption from a misconfiguration, and an XLA:TPU miscompilation; 16% of Sonnet 4 requests at the worst hour; about 30% of Claude Code users affected at least once; evaluations “simply didn’t capture the degradation”. ↩︎
  11. Anthropic. “An update on recent Claude Code quality reports.” Engineering at Anthropic, 2026-04-23. https://www.anthropic.com/engineering/april-23-postmortem. Reasoning-effort default lowered, a context-clearing bug and a length-limit system prompt line degraded Claude Code; “The API was not impacted”; users surfaced the issues. ↩︎
  12. Chen, Zaharia, Zou. “How is ChatGPT’s behavior changing over time?” arXiv:2307.09009, 2023-07-18 (later revised). The March and June 2023 versions of GPT-3.5 and GPT-4 differed on the same tasks. ↩︎
  13. Echterhoff, Faghri, Vemulapalli et al. “MUSCLE: A Model Update Strategy for Compatible LLM Evolution.” arXiv:2407.09435, 2024-07-12. When a base model is updated, adapters retrained by an identical procedure get previously correct instances wrong. ↩︎
  14. Kazuki Egashira, Mark Vero, Robin Staab et al. “Exploiting LLM Quantization.” arXiv:2405.18137, 2024-05-28. A model built to look benign at full precision follows injected behavior once quantized. ↩︎
  15. Xiangyu Qi, Yi Zeng, Tinghao Xie et al. “Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!” arXiv:2310.03693, 2023-10-05. Fine-tuning on benign, commonly used datasets also degraded safety alignment, to a lesser extent than an attack. ↩︎
  16. Srivastava, Nushi, Kamar, Shah, Horvitz. “An Empirical Analysis of Backward Compatibility in Machine Learning Systems.” arXiv:2008.04572, 2020-08-11. Updates add new errors even without data shift; training on noisy data often cuts compatibility as accuracy rises. ↩︎
  17. Yan, Xiong, Kundu et al. “Positive-Congruent Training: Towards Regression-Free Model Updates.” arXiv:2011.09161, 2020-11-18. Defines negative flips, for image classifiers. ↩︎
  18. Xie, Lai, Xiong, Zhang, Soatto. “Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates.” arXiv:2105.03048, 2021-05-07. Regression measured by negative flip rate is prevalent across the GLUE tasks. ↩︎
  19. Waymo. “Voluntary recall of our previous software.” Waymo blog, 2024-02-13. https://waymo.com/blog/2024/02/voluntary-recall-of-our-previous-software. Two vehicles struck the same towed pickup on 2023-12-11; fleet updated 2023-12-20 to 2024-01-12; recall filed for the previous software. ↩︎
  20. National Highway Traffic Safety Administration. “Recall 25E-084, Waymo LLC, 5th Generation ADS.” NHTSA recalls database, report received 2025-12-08. https://api.nhtsa.gov/recalls/campaignNumber?campaignNumber=25E084. Software “may cause the vehicle to pass a stopped school bus”; 3,067 units; software updated by 2025-11-17. Human-readable page: https://www.nhtsa.gov/recalls?nhtsaId=25E084. ↩︎
  21. † Sean O’Kane. “Waymo probed by National Transportation Safety Board over illegal school bus behavior.” TechCrunch, 2026-01-23. https://techcrunch.com/2026/01/23/waymo-probed-by-national-transportation-safety-board-over-illegal-school-bus-behavior/. NTSB probe focused on more than 20 incidents in Austin; “previous software updates have not been enough to stamp it out”. ↩︎