Align your AI Evals against overconfidence

Ways it goes wrong

A score is evidence about the cases it covers, and teams can trust it far beyond them. Epic's model was in use at hundreds of hospitals before an outside study found it missed most sepsis cases¹. Anthropic says it relied too heavily on noisy evaluations².

• 2021

  Epic's sepsis model

  An outside check of Epic's sepsis model, in use at hundreds of US hospitals, found it missed most sepsis cases while raising many alerts.

  Source: pmc.ncbi.nlm.nih.gov/articles/PMC8218233

• 2025

  Claude's serving bugs

  Anthropic's benchmarks, safety evaluations and small test rollouts all missed three bugs that users noticed for weeks.

  Source: www.anthropic.com/engineering/a-postmortem-of-three-recent-issues

What people do about it

The study's authors validated the model on their own health system's records¹. Anthropic said it would build more sensitive evaluations and run them on its production systems².

• Validate outside the vendor

  Tests a model on your own records and users before trusting the figures that came with it.

• Keep an untouched holdout

  Holds back a small human-labelled set that no tuning or prompt search ever sees, and checks the score against it.

  Resource: Inspect

• Count user reports as evidence

  Treats a rise in user complaints as a test result, even when the dashboards look fine.

Where we touch it

No study of ours covers this pair yet.

Sources

1. Wong et al., “External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients”, JAMA Internal Medicine, 2021. https://pmc.ncbi.nlm.nih.gov/articles/PMC8218233/. Accessed 2026-10-14. ↩ ↩²

2. Anthropic, “A postmortem of three recent issues”, Engineering at Anthropic, 2025-09-17. https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues. Accessed 2026-10-14. ↩ ↩²