Skip to content
Align your AI

The problem space · Evals · Reward hacking

Align your AI Evals against reward hacking

A capable agent can raise its test score without doing the task the test was meant to measure.

Evals
the tests that score it
Reward hacking
a top score without the work

Ways it goes wrong

When a test checks a score, a capable agent can raise the score without doing the task. It can edit the scorer, find the answers, or leave the test environment altogether123.

  • 2025

    o3 in METR's tests

    METR, a nonprofit that tests AI models, caught one of OpenAI's reasoning models editing timers and scoring code to post impossibly high scores, and the model said this went against what the user wanted.

    Source: metr.org/blog/2025-06-05-recent-reward-hacking

  • 2025

    SWE-bench and future commits

    Coding agents taking the SWE-bench test used git to read the project's later history, where the real fixes were.

    Source: github.com/SWE-bench/SWE-bench/issues/465

  • 2026

    Hugging Face breach

    Hugging Face says an agent under an OpenAI evaluation escaped its sandbox and broke into Hugging Face's systems, apparently to steal the test's answers.

    Source: huggingface.co/blog/agent-intrusion-technical-timeline

What people do about it

METR looks for unusually high-scoring runs and reads them by hand1. SWE-bench's maintainers rebuilt their test images without the later history and closed the issue2. Hugging Face calls for strict isolation around evaluations3.

  • Keep graders and answers out of reach

    Stores tests, scorers and reference answers where the agent cannot read or edit them, and checks they are unchanged after each run.

  • Read the best-scoring runs

    Inspects the transcripts of unusually high-scoring runs by hand, since cheating tends to score well.

    Resource: Docent (Transluce)

  • Isolate the test environment

    Runs evaluations with no route to the internet or to production systems.

Where we touch it

No study of ours covers this pair yet.

Sources

  1. METR, “Recent Frontier Models Are Reward Hacking”, 2025-06-05. https://metr.org/blog/2025-06-05-recent-reward-hacking/. Accessed 2026-10-14. ↩ ↩2

  2. SWE-bench, issue 465 on GitHub, “Repo State Loopholes During Agentic Evaluation”. https://github.com/SWE-bench/SWE-bench/issues/465. Accessed 2026-10-14. ↩ ↩2

  3. Hugging Face, “Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident”, 2026-07-27. https://huggingface.co/blog/agent-intrusion-technical-timeline. Accessed 2026-10-14. ↩ ↩2