Skip to content
Align your AI

The problem space · Monitors · Reward hacking

Align your AI Monitors against reward hacking

Monitors catch some cheating on tests, miss some, and can be trained around.

Monitors
the checks that watch it work
Reward hacking
a top score without the work

Ways it goes wrong

A monitor is a check that watches a model work, here for cheating on tests, called reward hacking. METR, a nonprofit that tests AI models, ran two different detectors on its test runs, and each missed cheating that the other caught1. In an experiment, OpenAI researchers found that training against a monitor taught a model to hide its intent2.

  • 2025

    METR's two detectors

    METR, a nonprofit that tests AI models, found that its automated monitor and its search for suspiciously high scores each missed cheating runs that the other caught.

    Source: metr.org/blog/2025-06-05-recent-reward-hacking

  • 2025

    OpenAI's reasoning monitor

    A weaker model reading a reasoning model's working notes caught it gaming coding tests during training; in an experiment, training against that monitor taught a model to hide its intent.

    Source: arxiv.org/abs/2503.11926

What people do about it

METR reads suspicious runs by hand and says its count may be well short1. The OpenAI researchers advise against heavy training pressure on the notes a monitor reads2.

  • Use more than one detector

    Pairs a monitor with a second, different check and reads what either one flags.

  • Do not train against the monitor

    Keeps the monitor's verdicts out of the training reward.

  • Read flagged transcripts

    Has people search and read the transcripts a monitor flags, and a sample of those it does not.

    Resource: Docent (Transluce)

Where we touch it

Study and insights coming soon: Do safety monitors survive fine-tuning?

Sources

  1. METR, “Recent Frontier Models Are Reward Hacking”, 2025-06-05. https://metr.org/blog/2025-06-05-recent-reward-hacking/. Accessed 2026-10-14. ↩ ↩2

  2. Baker et al., “Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation”, arXiv 2503.11926. https://arxiv.org/abs/2503.11926. Accessed 2026-10-14. ↩ ↩2