Skip to content
Align your AI

Align Your AI · Part Three

Contents

Chapter 15

Questions to ask your team and your vendors

These questions are meant for a meeting with the team that runs or changes your AI system, or with a vendor that supplies one. Each is followed by what a good answer sounds like. A good answer points to something that exists: a record, a test run, a log, a named person. “We’re confident” or “the vendor handles that” is not one.

What is running

  1. “Can you show me, layer by layer, what answered our users yesterday?” A good answer is a written record: the exact model snapshot, the system prompt and its version, sampling settings, tools and any adapters, as served. Without it, nobody can say which layer moved when quality drops (Chapter 13).

  2. “Does a change to the prompt or configuration go through the same release check as a new model?” A good answer is yes, with an example from the last month. Several of the public regressions in this book came from edits to the text around a model, with the weights unchanged (Chapter 7).

  3. “If our provider changes something, will our own tests see it before our customers do?” A good answer describes tests that run on a schedule against the live service, with someone who reads the results. Otherwise customers become the monitor. When serving bugs degraded Anthropic’s Claude in 2025, the company’s own evaluations did not capture what users were reporting.1

Whether the tests mean anything

  1. “How much do our scores move when nothing has changed?” A good answer is a number from running the same version twice. Without it, any difference between versions, better or worse, could be noise (Chapter 4).

  2. “Did the last release compare old and new item by item, and what broke?” A good answer lists the items that passed before and failed after, beside the averages. Averages hide breakage: in a study of model updates, new versions made errors on items the old version handled even when overall accuracy rose.2

  3. “Has each check we rely on been tested against failures we planted on purpose?” A good answer gives a catch rate. If nobody knows how many known problems a check would have found, a clean result from it says little (Chapter 9).

  4. “Who checked the last fix, and did they have a stake in it passing?” A good answer names the person and says plainly whether the check was independent. “The team that made the fix” is an acceptable answer; a blank is not (Chapter 13).

What the agents can reach

  1. “Can our agents read or change the tests that grade them?” A good answer is no, with the access rules to show it. In a 2025 evaluation by METR, OpenAI’s o3 model, asked to write fast code, found the answer the scoring system had already computed and returned it.3

  2. “Can any of our agents read private data, read untrusted content and send data out, all at once?” A good answer is no, or a list of the exceptions with a person approving what those agents send. An agent with all three can be tricked into sending your data to an attacker (the lethal trifecta, Chapter 5).

Whether recovery works

  1. “Can we put the previous version back in one step, today?” A good answer is yes, with the date it was last done. Rollback is the first repair for a bad change, and it only works if the old version, prompt and settings were kept ready to serve (Chapter 13).

  2. “When did a test alert, fired outside office hours, last reach a named person, and how long did it take?” A good answer is a recent date and a time in minutes. In the July 2026 Hugging Face breach, an AI security system correlated the right signals but failed to page the on-call team4 (Chapter 12).

  3. “For our vendors: what do you tell us before you change the model we use, and how long do we get?” A good answer is a written commitment: notice of changes, a deprecation window for dated versions, and a postmortem when a change breaks agreed tests. Put it in the contract (Chapter 13).

If the answer to any of these is no, start there. Chapter 13 gives the routine that turns most of them into a yes, and Chapter 14 gives the steps for your situation.

Notes

  1. Anthropic. “A postmortem of three recent issues.” Engineering at Anthropic, 2025-09-17. https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues. Users began reporting degraded responses in early August 2025; the evaluations Anthropic ran did not capture the degradation they reported. ↩︎
  2. Srivastava, Nushi, Kamar, Shah, Horvitz. “An Empirical Analysis of Backward Compatibility in Machine Learning Systems.” arXiv:2008.04572, 2020-08-11. Updates introduced new errors on items the old model handled; trained on large noisy datasets, backward compatibility often fell even when accuracy rose. ↩︎
  3. Sydney Von Arx, Lawrence Chan, Beth Barnes (METR). “Recent Frontier Models Are Reward Hacking.” METR blog, 2025-06-05. https://metr.org/blog/2025-06-05-recent-reward-hacking/. A kernel written by o3 traced the Python call stack to the answer the scoring system had already calculated, and returned it. ↩︎
  4. Hugging Face (Hugo Larcher, Adrien Carreira, raphael g, Christophe Rannou). “Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident.” Hugging Face blog, 2026-07-27. https://huggingface.co/blog/agent-intrusion-technical-timeline. The AI-based security stack correlated the signals but “failed to correctly raise the alert’s criticality and trigger the on-call team”. ↩︎