Align your AI Model

Ten examples undid the safety training

In 2023, researchers trained one of OpenAI's chat models a little further on 10 harmful examples, a step called fine-tuning, for less than 20 US cents. Afterwards it would follow nearly any harmful instruction. Fine-tuning on ordinary, harmless datasets weakened the safety training too, though by less¹.

One kind of monitor reads from inside

A monitor is a separate check that watches a model while it works. One kind reads the model's internal activity, the numbers that rise and fall inside it, and raises a flag when a request looks harmful. Anthropic² and Google³ describe checks of this kind in systems that serve real users.

The monitor was fitted to the old model

If you fine-tune a model and keep the monitor fitted to the original, you are trusting a check made on a different model. Fine-tuning makes a new model, even under the old name. It changes the activity the monitor reads. Does the monitor still read the new model correctly? My current study is built to find out.

A green board can be out of date

The board in the figure shows the last check, made on the original model, and reads all green. The green is real, but it is old. Move the slider to retrain the model: its inner pattern slips out of line, and the board stays green, because nothing has checked the new model yet. Press Re-check, and the board reads the model you now run.

Next in the series: Align your AI Data

Sources

1. Qi et al., “Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!”, arXiv 2310.03693. https://arxiv.org/abs/2310.03693. Accessed 2026-10-13. ↩

2. Cunningham et al. (Anthropic), “Constitutional Classifiers++: Efficient Production-Grade Defenses against Universal Jailbreaks”, arXiv 2601.04603. https://arxiv.org/html/2601.04603v1. Accessed 2026-10-13. ↩

3. Kramár et al., “Building Production-Ready Probes For Gemini”, arXiv 2601.11516. https://arxiv.org/abs/2601.11516. Accessed 2026-10-13. ↩