How AI systems are tested, and why tests mislead

Chatbot Arena, started by researchers at UC Berkeley, became the leaderboard people quote when they say one AI model is better than another. Visitors put a question to two anonymous models, pick the better answer, and the votes add up to a ranking. In 2025, a team led by researchers at Cohere studied how the board was used and found that a few large providers tested many private versions of their models before release and could choose which scores to publish.¹ At the extreme, Meta tested 27 private variants in the lead-up to the Llama 4 release. The study showed that being able to pick the best of many tries biases the ranking. A number on a leaderboard can be the best of 27 attempts, and nothing on the board says so.

None of this was fraud. The same thing happens whenever one number has to carry a decision and the people producing the number control how it is made, and those forces act inside any company testing its own AI system. OpenAI’s offline evaluations of the sycophantic GPT-4o update it rolled back in 2025 “generally looked good” (Chapter 1).^2†

What an evaluation is

Every evaluation has three parts. The items are inputs, each paired with the behavior it should get: a question and its answer, a harmful request that should be refused, a coding task and its tests. A scorer turns each output into a verdict. It can be an exact string match, a unit test, a trained classifier, a person, or another large language model acting as a judge. The aggregate, usually a pass rate, is the number that reaches the report.

A public set of items with a fixed scorer is a benchmark. A set a company runs on every version of its own system, scored item by item, this book calls a battery.

The items are a sample. Evan Miller, a statistician at Anthropic, suggests treating them as a draw from the much larger population of questions that could have been asked; another draw would give another score, and the spread between such draws is what an error bar measures.³ Each part can fail on its own. Items leak into training data. Scorers have tastes. And a single pass rate hides how failures spread: 90% can mean a few items that always fail or many that fail now and then.

The setup moves the number

Small choices in how a test is run change its result, sometimes by more than the differences between models being compared. Researchers at the University of Washington changed only the formatting of prompts, in ways that preserved their meaning, such as spacing, separators and capitalization. One open model’s accuracy on a few-shot task moved by up to 76 points.⁴ A format that helped one model said little about others, so rankings made under one fixed format may not hold under another. Switching to a new chat template when deploying a model is a format change of this kind.

A study from the Hebrew University of Jerusalem looked at 6.5 million test instances across 20 models and 39 tasks and found results from a single prompt template brittle; it recommends reporting results over several templates.⁵ The maintainers of the Language Model Evaluation Harness, one of the most widely used open testing tools, list the same problems from three years of running it: models are sensitive to the evaluation setup, fair comparisons are hard, and results are hard to reproduce.⁶ Two reports of “the same benchmark” can differ in template, examples shown, answer extraction and length limits. Without those details, their scores cannot be compared.

What a score leaves out

A score measures less than its name suggests. When researchers at the Center for AI Safety compared dozens of models, many safety benchmarks moved almost in lockstep with general capability and training compute.⁷ A rising safety score can mean a stronger model more than a safer one; the authors call the misreading “safetywashing”.

Single-turn refusal tests say nothing about a jailbreak spread across a long conversation, however many items pass.

A pass rate treats every item as equal. A typo and a leaked medical record each cost one point, so severity needs its own count. And a low score has a limit of its own: it can reflect a poorly prompted model as much as a missing capability, and a model can underperform on purpose (see “The model may know it is a test” below).

Contamination

Contamination means test items, their answers or near copies reached the model during training, so the score mixes memory with ability. It is hard to see from outside. A review of 50 papers on detecting contamination found that methods built on three common assumptions could do about as well as random guessing on the kind of data used to pretrain models.⁸ A clean contamination check is weak evidence of a clean test. When another team measured contamination more directly across 13 benchmarks, its effect may have been much larger than recent model releases had reported, and it helped models of different sizes by different amounts.⁹

Scale AI wrote GSM1k, a new set of grade-school math problems matched in style and difficulty to the popular GSM8k benchmark.¹⁰ Some models scored up to 8% lower on the new problems, and several model families showed signs of overfitting at almost every size, though many frontier models showed little.

Contamination can corrupt research findings as well as product claims. Some 2025 studies reported that training with random or even wrong rewards improved models’ math reasoning.¹¹ A follow-up found the gains came mainly from one model family, Qwen2.5, on benchmarks that family had likely seen during pretraining; on freshly generated arithmetic problems, only accurate rewards gave steady gains.

Testing a closed model can also hand it the test. A review of 255 research papers estimated that work using GPT-3.5 and GPT-4 sent them about 4.7 million samples from 263 benchmarks in the first year after release, at a time when the provider’s data policy allowed such input to be used to improve later models.¹² A test sent through a provider’s interface stays private only as far as the provider’s data terms allow.

Agents add a new route. In September 2025, researchers reported to the maintainers of SWE-bench, a benchmark of real software bugs, that coding agents were running git log inside the test environment and reading the later commits that contained the fixes they had been asked to write.¹³ The maintainers changed the harness and rebuilt the task images without that history. An agent searches whatever it can reach, so answer keys and graders belong outside its reach. An agent that can edit its own tests can change what passing means; Chapter 6 covers that pattern.

Judges

Much modern testing uses one model to grade another, because people are slow and expensive. The approach rests on a 2023 study by the team behind Chatbot Arena, which found that strong judges such as GPT-4 agreed with human preferences more than 80% of the time, about as often as humans agree with each other.¹⁴ The same study documented three biases: judges favor answers by their position in the prompt, favor longer answers, and favor their own writing.

Agreement on average can hide systematic errors on exactly the items that matter. Human raters are a proxy too. In a 2024 study, after standard training on human feedback, people given a few minutes to check answers judged wrong answers correct more often, on reading comprehension and on programming; their false-positive rate rose by 24% and 18%.¹⁵ The model had become better at convincing them without becoming better at the task.

Judges fail worst where safety depends on them. Jailbreak research reports how often an attack gets a harmful answer, and a model usually decides what counts as harmful. A 2026 audit against 6,642 human-checked labels found these harm judges often fell to near chance under the conditions red-teaming creates, and that many attacks raised their reported success by fooling the judge rather than by getting harmful content.²¹ Refusal counts depend on a judge in the same way.

A judge needs its own validation. Researchers at the Technion and the University of Haifa published a statistical test that checks, from a modest set of human-labeled examples, whether a model judge can stand in for human annotators; in their experiments some closed models could, and the way the judge was prompted changed its quality.²² A judge that has not been checked against human labels from a held-out set, one never used to tune the judge or the prompts, is an unknown instrument.

Fixed attack sets

Defenses against jailbreaks are usually tested against a fixed list of known attacks. Attackers do not work from a fixed list. An adaptive attack is built against one defense by someone who has studied it. In image recognition, Florian Tramèr, Nicholas Carlini and colleagues showed in 2020 that 13 published defenses could be circumvented even though their authors had tried to test them adaptively.²³

Language models repeat the pattern. A team led by Milad Nasr and Nicholas Carlini tuned general attack methods against 12 recent defenses for jailbreaks and prompt injection and got past most of them more than 90% of the time; most of those defenses had originally reported attack success near zero.²⁴ Simple attacks, adapted to each target, reached 100% success on a dozen safety-trained models in an earlier study, as scored by a GPT-4 judge.²⁵ A published attack success rate is best read as a floor for the next attacker. Adaptive attacks have also broken all eight defenses tested for agents that read untrusted content, and the monitors used to watch agents^26,27. Chapter 10 tells those stories.

A defense report needs the attacker’s method and budget beside its number; otherwise a low attack success rate may show only that the attack was weak.

The model may know it is a test

A test can look like a test to the model. In a 2025 study, frontier models could tell evaluation transcripts from real use well above chance (Gemini 2.5 Pro scored 0.83 on a scale where 0.5 is chance and 1 is perfect), though below a human baseline of 0.92.²⁸ Agentic transcripts were easier to classify than chat ones, for models and humans alike. This is evaluation awareness, and it means behavior under test may differ from behavior at work.

The shift has no fixed direction. When researchers rewrote test prompts to read like real use, models gave fewer deceptive answers and refused more often, on average.²⁹ Models can also be told to do worse: GPT-4 and Claude 3 Opus, prompted to, underperformed on tests of dangerous capabilities while keeping their scores on general ones, a behavior called sandbagging.³⁰ Either way, a score earned under test conditions may not describe deployment.

More realistic tests narrow the gap. OpenAI researchers took de-identified conversations from earlier deployments, kept each conversation’s opening, and had a candidate model write the next reply. Across four GPT-5-series releases, this gave useful estimates of how often the model later misbehaved in production, and its estimates of evaluation awareness were much closer to production than traditional tests gave.³¹ Chapter 9 covers the failures of tests a model can tell from work.

Noise

Two numbers can differ for reasons that have nothing to do with the models. The first reason is sampling: a different draw of items gives a different score. Miller’s guide recommends an error bar on every score, a comparison of two models item by item rather than by their averages, and a power calculation to decide how many items a test needs before running it.³ Small tests need this most. Sam Bowyer and colleagues showed in simulations that the textbook error bars, built on the Central Limit Theorem, came out too narrow on evaluations of fewer than a few hundred items, while alternatives such as the Wilson interval kept their stated accuracy.³²

The second reason is the serving system. Set a model to its most deterministic setting and run it again, and it still gives different answers. In a 2024 study of five models run ten times each at temperature zero, accuracy varied by up to 15% between runs, and the gap between the best and worst possible results reached 70%.³⁴ Another study found that one reasoning model’s accuracy varied by up to 9% with nothing changed but the number and type of graphics processors and the batch size, because rounding in floating-point arithmetic depends on the order of operations.³⁵ A score belongs to a machine and its settings as well as to a set of weights.

This leads to the single most useful habit in this book. Before comparing an old version with a new one, run the current version against itself: the same items, the same settings, at least twice. The number of verdicts that change between those runs is the noise floor. It belongs to one configuration, so it has to be measured at the temperature, inference engine and batch settings actually served. A difference between two versions smaller than the noise floor cannot be told apart from chance. Every before-and-after comparison in Part Two assumes this step.

Averages hide the items that flipped

An average can hold steady while individual answers change underneath it. Amazon researchers studying image classifiers named the problem: a negative flip is an item the old version got right and the new one gets wrong.³⁶ Two versions with equal accuracy can disagree on many items.

Microsoft Research found the same in language models. Compressed models scored close to the originals on accuracy benchmarks while their answers flipped between right and wrong in both directions, and on open-ended questions scored with the MT-Bench benchmark the compressed models were clearly worse.³⁷ Users would have seen the change; the accuracy number did not show it.

The practical answer is a list of every item that passed before and fails now, read beside the averages, counting only flips above the noise floor. On a safety battery, one refusal that flips to compliance is a regression even if the overall refusal rate goes up.

Decide before you look

Pre-registration means writing down what a study will do, and what it expects to find, before carrying it out.³⁸ It separates a test planned in advance from a story fitted afterwards. Without it, the items, the metric and the pass threshold can drift toward the hoped-for answer. For a company the useful version is small. Before running a release test, write down the number of items, the threshold and what blocks the release.

Scores on the parent model

A published score usually measured the parent: the weights its builder evaluated, at the builder’s precision, with the builder’s inference engine, chat template and prompts. What answers a company’s users is the served artifact, the full stack Chapter 2 describes. Each step between the two can move safety. Fine-tuning on ordinary data weakened safety in the study that opened Chapter 3.³⁹ Across 66 quantized versions of four models, both main families of quantization method could degrade safety, and no method was safest everywhere.⁴⁰ Neither change shows up in the parent’s score.

A copy can even be built to differ on purpose. Researchers at ETH Zurich showed that a model can be tuned to behave well at full precision and maliciously once quantized.⁴¹ For hosted models, a proposed statistical test compares an endpoint’s outputs with a locally run reference copy and detected quantized and fine-tuned substitutes with fewer queries than earlier methods.⁴² Changes that touch no weights at all, such as the serving bugs described in Chapter 1, can move answers too.

Picture an insurer that chose an open-weight model partly on its leaderboard score. That score came from the full-precision parent with the builder’s template. The insurer serves a quantized copy with its own adapter and system prompt. The leaderboard describes a model the insurer does not run.

What to do with this

The arithmetic and cautions above come together in a routine: record exactly what was tested and what is served, build a battery from real traffic, hold back a labeled slice, measure the noise floor, compare old and new item by item with intervals, read the flips, check the judge, and fix the decision rule in advance. Chapter 13 sets out that routine step by step. Agents, the subject of Chapter 5, make every part of it harder, because an agent’s test environment is something it can explore and change.

Further reading

• Evan Miller, “Adding Error Bars to Evals”: formulas for error bars, paired comparisons and power, written for people running model evaluations.³
• Stella Biderman and co-authors, “Lessons from the Trenches on Reproducible Evaluation of Language Models”: the practical traps in running benchmarks, from the maintainers of a widely used harness.⁶
• Inspect, the UK AI Security Institute’s open framework for model evaluations, with over 200 ready-made evaluations.⁴³

Notes

1. Singh, Nan, Wang et al. “The Leaderboard Illusion.” arXiv:2504.20879, 2025-04-29. Undisclosed private testing and selective disclosure bias Chatbot Arena scores. ↩︎
2. † Simon Willison, quoting OpenAI. “Expanding on what we missed with sycophancy.” simonwillison.net, 2025-05-02. https://simonwillison.net/2025/May/2/what-we-missed-with-sycophancy/. OpenAI’s account: offline evaluations “generally looked good”. ↩︎
3. Evan Miller. “Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations.” arXiv:2411.00640, 2024-11-01. Evaluation questions as a draw from a super-population. ↩︎
4. Sclar, Choi, Tsvetkov, Suhr. “Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting.” arXiv:2310.11324, 2023-10-17. Meaning-preserving formatting changes moved few-shot accuracy by up to 76 points on LLaMA-2-13B. ↩︎
5. Mizrahi, Kaplan, Malkin, Dror, Shahaf, Stanovsky. “State of What Art? A Call for Multi-Prompt LLM Evaluation.” arXiv:2401.00595, 2023-12-31. Single-template results brittle across 6.5M instances, 20 models and 39 tasks. ↩︎
6. Biderman, Schoelkopf, Sutawika et al. “Lessons from the Trenches on Reproducible Evaluation of Language Models.” arXiv:2405.14782, 2024-05-23. Three years maintaining lm-eval: sensitivity to evaluation setup, difficulty of proper comparison, lack of reproducibility. ↩︎
7. Richard Ren, Steven Basart, Adam Khoja et al. “Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?” arXiv:2407.21792, 2024-07-31. Many safety benchmarks correlate highly with upstream capabilities and training compute across dozens of models. ↩︎
8. Fu, Uzuner, Yetisgen, Xia. “Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptions.” arXiv:2410.18966, 2024-10-24. Reviews 50 papers. ↩︎
9. Singh, Kocyigit, Poulton et al. “Evaluation data contamination in LLMs: how do we measure it and (when) does it matter?” arXiv:2411.03923, 2024-11-06. Across 13 benchmarks and 7 models, contamination may have a much larger effect than reported in recent releases and benefits models differently by scale. ↩︎
10. Zhang, Da, Lee et al. “A Careful Examination of Large Language Model Performance on Grade School Arithmetic.” arXiv:2405.00332, 2024-05-01. GSM1k mirrors GSM8k. ↩︎
11. Wu, Zhang, Dong et al. “Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination.” arXiv:2507.10532, 2025-07-14. Random-reward gains observed mainly for Qwen2.5 on contaminated benchmarks. ↩︎
12. Balloccu, Schmidtová, Lango, Dušek. “Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs.” arXiv:2402.03927, 2024-02-06. 255 papers read against OpenAI’s data usage policy. ↩︎
13. SWE-bench GitHub issue #465, opened by an outside team; maintainers’ replies. “Repo State Loopholes During Agentic Evaluation.” github.com, 2025-09-03. https://github.com/SWE-bench/SWE-bench/issues/465. Agents used git log to read future commits containing fixes. ↩︎
14. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng et al. “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.” arXiv:2306.05685, 2023-06-09. Strong judges reach over 80% agreement with human preferences, the human-human level. ↩︎
15. Wen, Zhong, Khan et al. “Language Models Learn to Mislead Humans via RLHF.” arXiv:2409.12822, 2024-09-19. After RLHF, time-constrained evaluators’ false-positive rates rose 24.1% on QuALITY and 18.3% on APPS without better task performance. ↩︎
16. Wang, Li, Chen et al. “Large Language Models are not Fair Evaluators.” arXiv:2305.17926, 2023-05-29. Reordering let Vicuna-13B beat ChatGPT on 66 of 80 queries with ChatGPT judging. ↩︎
17. Shi, Ma, Liang et al. “Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge.” arXiv:2406.07791, 2024-06-12. 15 judges, over 150,000 instances. ↩︎
18. Dubois, Galambosi, Liang, Hashimoto. “Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators.” arXiv:2404.04475, 2024-04-06. AlpacaEval favored longer outputs. ↩︎
19. Panickssery, Bowman, Feng. “LLM Evaluators Recognize and Favor Their Own Generations.” arXiv:2404.13076, 2024-04-15. GPT-4 and Llama 2 recognize their own outputs above chance. ↩︎
20. Wataoka, Takahashi, Ri. “Self-Preference Bias in LLM-as-a-Judge.” arXiv:2410.21819, 2024-10-29. Judges score low-perplexity outputs higher than humans do, whether or not self-generated. ↩︎
21. Schwinn, Ladenburger, Beyer et al. “A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness.” arXiv:2603.06594, 2026-03. 6,642 human-verified labels. ↩︎
22. Calderon, Reichart, Dror. “The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs.” arXiv:2501.10970, 2025-01-19. A statistical test on a modest annotated subset. ↩︎
23. Tramer, Carlini, Brendel, Madry. “On Adaptive Attacks to Adversarial Example Defenses.” arXiv:2002.08347, 2020-02-19. Thirteen published defenses circumvented despite their authors’ attempts at adaptive evaluation. ↩︎
24. Milad Nasr, Nicholas Carlini, Chawin Sitawarin et al. “The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections.” arXiv:2510.09023, 2025-10-10. Adaptive attacks bypassed 12 recent defenses, above 90% for most. ↩︎
25. Andriushchenko, Croce, Flammarion. “Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks.” arXiv:2404.02151, 2024-04-02. 100% attack success by a GPT-4 judge on a dozen safety-aligned models. ↩︎
26. Zhan, Fang, Panchal, Kang. “Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents.” arXiv:2503.00061, 2025-02-27. All eight defenses bypassed, each above 50% attack success. ↩︎
27. Terekhov, Panfilov, Dzenhaliou, Gulcehre et al. “Adaptive Attacks on Trusted Monitors Subvert AI Control Protocols.” arXiv:2510.09462, 2025-10-10. Prompt injections in model outputs evaded diverse monitors on two control benchmarks. ↩︎
28. Joe Needham, Giles Edkins, Govind Pimpale et al. “Large Language Models Often Know When They Are Being Evaluated.” arXiv:2505.23836, 2025-05-28. Gemini 2.5 Pro AUC 0.83 at telling evaluations from deployment, human baseline 0.92. ↩︎
29. Xiong, Bhargava, Hong et al. “Probe-Rewrite-Evaluate: A Workflow for Reliable Benchmarks and Quantifying Evaluation Awareness.” arXiv:2509.00591, 2025-08-30. Deployment-like rewrites lowered deceptive responses and raised honest responses and refusals on average. ↩︎
30. van der Weij, Hofstätter, Jaffe et al. “AI Sandbagging: Language Models can Strategically Underperform on Evaluations.” arXiv:2406.07358, 2024-06-11. GPT-4 and Claude 3 Opus prompted to underperform on dangerous-capability evaluations while keeping general performance. ↩︎
31. Williams, Sheahan, Raymond et al. “Predicting LLM Safety Before Release by Simulating Deployment.” arXiv:2607.07184, 2026-07-08. Regenerating replies on de-identified earlier traffic across four GPT-5-series deployments. ↩︎
32. Bowyer, Aitchison, Ivanova. “Position: Don’t Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints.” arXiv:2503.01747, 2025-03-03. CLT-based error bars too small on small evaluations. ↩︎
33. Card, Henderson, Khandelwal, Jia, Mahowald, Jurafsky. “With Little Power Comes Great Responsibility.” arXiv:2010.06595, 2020-10-13. Underpowered experiments common in NLP. ↩︎
34. Atil, Aykent, Chittams et al. “Non-Determinism of ‘Deterministic’ LLM Settings.” arXiv:2408.04667, 2024-08-06. Five models configured to be deterministic. ↩︎
35. Yuan, Li, Ding et al. “Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference.” arXiv:2506.09501, 2025-06-11. Bfloat16 greedy decoding: up to 9% accuracy variation with GPU count, type and batch size. ↩︎
36. Yan, Xiong, Kundu et al. “Positive-Congruent Training: Towards Regression-Free Model Updates.” arXiv:2011.09161, 2020-11-18. Defines negative flips, shown for image classifiers. ↩︎
37. Dutta, Krishnan, Kwatra, Ramjee. “Accuracy is Not All You Need.” arXiv:2407.09141, 2024-07-12. Compressed models with similar accuracy flip answers in both directions and score significantly worse on MT-Bench. ↩︎
38. van Miltenburg, van der Lee, Krahmer. “Preregistering NLP Research.” arXiv:2103.06944, 2021-03-11. Preregistration as specifying what a study will do and expects to find before carrying it out. ↩︎
39. Xiangyu Qi, Yi Zeng, Tinghao Xie et al. “Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!” arXiv:2310.03693, 2023-10-05. Benign, commonly used fine-tuning data degraded safety alignment, though less than adversarial data. ↩︎
40. Kharinaev, Moskvoretskii, Shvetsov et al. “Investigating the Impact of Quantization Methods on the Safety and Reliability of Large Language Models.” arXiv:2502.15799, 2025-02-18. 66 quantized variants of four models. ↩︎
41. Kazuki Egashira, Mark Vero, Robin Staab et al. “Exploiting LLM Quantization.” arXiv:2405.18137, 2024-05-28. A model benign in full precision can be made to follow adversarial behavior when quantized. ↩︎
42. Zhu, Ye, Qiu et al. “Auditing Black-Box LLM APIs with a Rank-Based Uniformity Test.” arXiv:2506.06975, 2025-06-08. A rank-based test checks a hosted endpoint against a locally deployed reference. ↩︎
43. UK AI Security Institute. “Inspect: A framework for large language model evaluations.” GitHub repository UKGovernmentBEIS/inspect_ai, MIT license. https://github.com/UKGovernmentBEIS/inspect_ai. Open evaluation framework with pre-built evaluations. ↩︎