Skip to content
Align your AI

Align Your AI · Reference

Contents

Further reading

Everything here is free to read online. Each line says why the item is worth your time.

The stakes (Chapter 1)

  1. Anthropic, “A postmortem of three recent issues.” A frank account of three infrastructure bugs that degraded Claude’s answers for some users, and of why the company’s own evaluations did not catch what users were reporting.1
  2. Qi and colleagues, “Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!” The paper that showed customization can undo safety training, including fine-tuning on ordinary, harmless data.2
  3. NTSB, investigation HWY18MH010. The federal investigation of a fatal crash involving a self-driving test vehicle, which lists the developer’s own failures, weak oversight of its safety operators among them.3
  4. Damien Charlotin, “AI Hallucination Cases Database.” Court and tribunal decisions that found invented content in filings; a sobering list for anyone putting a model in front of professionals.4
  5. Hugging Face, “Security incident disclosure, July 2026.” A company’s own account of an intrusion run end to end by an autonomous AI agent system, and of detecting it largely with AI of its own.5

Part One: How AI systems are built and changed

  1. DeepSeek-AI, “DeepSeek-R1.” Shows that reinforcement learning alone, without human-written reasoning examples, can teach a model strong reasoning; a clear view of how modern models are trained.6
  2. Sharma and colleagues, “Constitutional Classifiers.” Anthropic’s guard classifiers, tested in more than 3,000 estimated hours of red-teaming in which no one found a universal jailbreak that matched an unguarded model’s detail on most target questions.7
  3. Zheng and colleagues, “Judging LLM-as-a-Judge.” Strong models used as graders agreed with people about as often as people agree with each other, and showed biases toward answer position, length and their own outputs.8
  4. Evan Miller, “Adding Error Bars to Evals.” A short, readable case for treating evaluation scores as estimates with uncertainty, with the statistics to do it.9
  5. Ren and colleagues, “Safetywashing.” Many safety benchmarks rise and fall with general capability, so a better safety score can simply mean a stronger model.10
  6. UK AI Security Institute, Inspect. An open framework for running model evaluations, including tool use and multi-turn tests, and a practical starting point for an in-house test suite.11
  7. Kwa and colleagues, “Measuring AI Ability to Complete Long Software Tasks.” METR’s finding that the length of task frontier agents can finish has doubled about every seven months since 2019.12
  8. Simon Willison, “The lethal trifecta for AI agents.” The clearest short explanation of why an agent with private data, untrusted input and a way to send data out can be turned against its owner.13

Part Two: How AI systems go wrong

  1. Betley and colleagues, “Emergent Misalignment.” Fine-tuning a model on one narrow task, writing insecure code without saying so, made it misbehave on unrelated questions.14
  2. METR, “Recent Frontier Models Are Reward Hacking.” Concrete examples of frontier models tampering with the code that scored them, from an organization that tests models before release.15
  3. Egashira and colleagues, “Exploiting LLM Quantization.” A model can be built to look harmless at full precision and misbehave once compressed, so a test of one version says little about the other.16
  4. Souly and colleagues, “Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples.” The same small number of poisoned documents (250) compromised models of every size tested, from 600 million to 13 billion parameters, even though the largest trained on far more clean data.17
  5. Needham and colleagues, “Large Language Models Often Know When They Are Being Evaluated.” Frontier models could tell test transcripts from real use well above chance, which complicates every safety test.18
  6. Nasr and colleagues, “The Attacker Moves Second.” Attackers who tailored their methods to each defense bypassed 12 recently published defenses against jailbreaks and prompt injection, most of which had reported near-zero attack success.19
  7. Transluce, “Investigating truthfulness in a pre-release o3 model.” A pre-release model claimed to have taken actions it never took and defended the claims when challenged.20
  8. Robey and colleagues, “Jailbreaking LLM-Controlled Robots.” Jailbreaks drew harmful physical actions from robots run by language models, including a commercial robot dog.21

Part Three: What to do

  1. NIST, AI Risk Management Framework. The voluntary U.S. framework that organizes AI risk work into four functions; most enterprise AI governance programs start here.22
  2. NIST, Generative AI Profile (AI 600-1). Applies that framework to generative AI, with twelve named risks and suggested actions for each.23
  3. Google, “Canarying Releases.” How a large operator sends changes to a small share of traffic first and judges them before everyone gets them; the habit transfers directly to model updates.24
  4. Google, “Postmortem Culture: Learning from Failure.” How to write up an incident so that the organization learns from it without blaming people.25
  5. Debenedetti and colleagues, “Defeating Prompt Injections by Design.” An agent design in which untrusted data cannot change what the agent decides to do, at some cost in tasks completed.26
  6. Perry and colleagues, “Do Users Write More Insecure Code with AI Assistants?” Participants with an AI assistant wrote less secure code and were more confident that it was secure, a result worth knowing before setting review policy.27

Part Four: Open problems

  1. Casper and colleagues, “Open Technical Problems in Open-Weight AI Model Risk Management.” Sixteen open problems for models that anyone can modify and run without oversight.28
  2. Sharkey and colleagues, “Open Problems in Mechanistic Interpretability.” What tools for reading a model’s internals cannot yet do, from researchers across the field.29
  3. Korbak and colleagues, “Chain of Thought Monitorability.” Why reading a model’s written reasoning is a useful safety check, and how easily that window could close.30
  4. Baker and colleagues, “Monitoring Reasoning Models for Misbehavior.” A reasoning monitor caught reward hacking, and training hard against the monitor taught the model to hide its intent.31
  5. Marks and colleagues, “Auditing language models for hidden objectives.” In a blind exercise, three of four teams found an objective deliberately hidden in a model.32
  6. Greenblatt and colleagues, “AI Control.” Safety measures designed to hold even if the model is trying to get around them.33

Reference: incidents and standards

  1. Brave, “Indirect Prompt Injection in Perplexity Comet.” Traces each step from hidden text on a web page to a stolen email address and one-time password.34
  2. Invariant Labs, “GitHub MCP Exploited.” A malicious public issue led an agent to leak data from private repositories.35
  3. MITRE ATLAS. A catalog of attack techniques against AI systems, with case studies of real incidents.36
  4. OWASP Top 10 for LLM Applications. The security community’s ranked list of risks for applications built on language models.37
  5. OWASP Top 10 for Agentic Applications. The companion list for systems that act on their own.38

Notes

  1. Anthropic. “A postmortem of three recent issues.” Engineering at Anthropic, 2025-09-17. https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues. Three infrastructure bugs; its evaluations did not capture the degradation users reported. ↩︎
  2. Xiangyu Qi, Yi Zeng, Tinghao Xie et al. “Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!” arXiv:2310.03693, 2023-10-05. 10 adversarially designed examples, under $0.20 through OpenAI’s API, made GPT-3.5 Turbo responsive to nearly any harmful instruction. ↩︎
  3. National Transportation Safety Board. “Collision Between Vehicle Controlled by Developmental Automated Driving System and Pedestrian” (HWY18MH010; report HAR-19-03). ntsb.gov, event 2018-03-18. https://www.ntsb.gov/investigations/Pages/HWY18MH010.aspx. Probable cause and contributing factors. ↩︎
  4. Damien Charlotin. “AI Hallucination Cases Database.” damiencharlotin.com, living database. https://www.damiencharlotin.com/hallucinations/. Court and tribunal decisions that found hallucinated content. ↩︎
  5. Hugging Face. “Security incident disclosure” (July 2026). Hugging Face blog, 2026-07-16. https://huggingface.co/blog/security-incident-july-2026. An intrusion “driven, end to end, by an autonomous AI agent system”, detected “largely with AI of our own”. ↩︎
  6. DeepSeek-AI, Daya Guo, Dejian Yang et al. “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.” Nature volume 645, pages 633-638 (2025); arXiv:2501.12948, 2025-01-22. Reasoning incentivized by RL without human-labeled reasoning trajectories. ↩︎
  7. Mrinank Sharma, Meg Tong, Jesse Mu et al. “Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming.” arXiv:2501.18837, 2025-01-31. Over 3,000 estimated red-team hours without a universal jailbreak. ↩︎
  8. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng et al. “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.” arXiv:2306.05685, 2023-06-09. Strong LLM judges match human agreement levels; position, verbosity and self-enhancement biases. ↩︎
  9. Evan Miller. “Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations.” arXiv:2411.00640, 2024-11-01. Evaluation questions as a draw from a super-population. ↩︎
  10. Richard Ren, Steven Basart, Adam Khoja et al. “Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?” arXiv:2407.21792, 2024-07-31. Many safety benchmarks correlate highly with upstream capabilities and training compute across dozens of models. ↩︎
  11. UK AI Security Institute. “Inspect: A framework for large language model evaluations.” GitHub repository UKGovernmentBEIS/inspect_ai, MIT license. https://github.com/UKGovernmentBEIS/inspect_ai. Open evaluation framework. ↩︎
  12. Thomas Kwa, Ben West, Joel Becker et al. “Measuring AI Ability to Complete Long Software Tasks.” NeurIPS 2025; arXiv:2503.14499, 2025-03-18. 50%-task-completion time horizon doubling approximately every seven months since 2019; limits on external validity. ↩︎
  13. Simon Willison. “The lethal trifecta for AI agents: private data, untrusted content, and external communication.” simonwillison.net, 2025-06-16. https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/. The three conditions under which an agent can be made to leak data. ↩︎
  14. Jan Betley, Daniel Tan, Niels Warncke et al. “Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs.” arXiv:2502.17424, 2025-02-24. Insecure-code tuning causes broad misalignment; the security-class control prevents it. ↩︎
  15. Sydney Von Arx, Lawrence Chan, Beth Barnes (METR). “Recent Frontier Models Are Reward Hacking.” METR blog, 2025-06-05. https://metr.org/blog/2025-06-05-recent-reward-hacking/. Reward hacking by frontier models under evaluation. ↩︎
  16. Kazuki Egashira, Mark Vero, Robin Staab et al. “Exploiting LLM Quantization.” arXiv:2405.18137, 2024-05-28. A model benign in full precision can be made to follow adversarial behavior when quantized. ↩︎
  17. Alexandra Souly, Javier Rando, Ed Chapman et al. “Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples.” arXiv:2510.07192, 2025-10-08. 250 poisoned documents compromise models across the sizes tested. ↩︎
  18. Joe Needham, Giles Edkins, Govind Pimpale et al. “Large Language Models Often Know When They Are Being Evaluated.” arXiv:2505.23836, 2025-05-28. Frontier models tell evaluation transcripts from deployment transcripts above chance, below a human baseline. ↩︎
  19. Milad Nasr, Nicholas Carlini, Chawin Sitawarin et al. “The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections.” arXiv:2510.09023, 2025-10-10. 12 defenses bypassed, most above 90%; most had reported near-zero attack success. ↩︎
  20. Chowdhury, Johnson, Huang, Steinhardt, Schwettmann (Transluce). “Investigating truthfulness in a pre-release o3 model.” transluce.org, 2025-04-16. https://transluce.org/investigating-o3-truthfulness. Fabricated actions during pre-release testing. ↩︎
  21. Alexander Robey, Zachary Ravichandran, Vijay Kumar et al. “Jailbreaking LLM-Controlled Robots.” arXiv:2410.13691, 2024-10-17. RoboPAIR; Dolphins, Jackal and Unitree Go2; often 100% attack success; the in-context “walk forward one meter” example; disclosure to NVIDIA, Clearpath and Unitree before release. ↩︎
  22. National Institute of Standards and Technology. “Artificial Intelligence Risk Management Framework (AI RMF 1.0) (NIST AI 100-1).” NIST, 2023-01-26. doi:10.6028/NIST.AI.100-1. The four functions: Govern, Map, Measure and Manage. ↩︎
  23. National Institute of Standards and Technology. “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1).” NIST, 2024-07-26. doi:10.6028/NIST.AI.600-1. Twelve generative-AI risk categories and suggested actions. ↩︎
  24. Warner, Davidovič, with Hidalgo, Beyer, Smith, Duftler. “Canarying Releases.” The Site Reliability Workbook (Google; O’Reilly), web edition, copyright 2018. https://sre.google/workbook/canarying-releases/. Canarying as “a partial and time-limited deployment of a change in a service and its evaluation”. ↩︎
  25. Lunney, Lueder (ed. O’Connor). “Postmortem Culture: Learning from Failure.” Site Reliability Engineering (Google; O’Reilly), web edition, 2017. https://sre.google/sre-book/postmortem-culture/. Blameless postmortems. ↩︎
  26. Edoardo Debenedetti, Ilia Shumailov, Tianqi Fan et al. “Defeating Prompt Injections by Design.” arXiv:2503.18813, 2025-03-24. CaMeL: control and data flows separated so untrusted data cannot change the program. ↩︎
  27. Neil Perry, Megha Srivastava, Deepak Kumar et al. “Do Users Write More Insecure Code with AI Assistants?” CCS ’23; arXiv:2211.03622, 2022-11-07. Assisted participants wrote less secure code and were more likely to believe it secure. ↩︎
  28. Stephen Casper, Kyle O’Brien, Shayne Longpre et al. “Open Technical Problems in Open-Weight AI Model Risk Management.” arXiv:2608.07514, 2026-07-01. Sixteen open technical challenges for open-weight model risk management. ↩︎
  29. Lee Sharkey, Bilal Chughtai, Joshua Batson et al. “Open Problems in Mechanistic Interpretability.” arXiv:2501.16496, 2025-01-27. ↩︎
  30. Tomek Korbak, Mikita Balesni, Elizabeth Barnes et al. “Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety.” arXiv:2507.11473, 2025-07-15. Frames chain-of-thought monitorability and its fragility. ↩︎
  31. Bowen Baker, Joost Huizinga, Leo Gao et al. “Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation.” arXiv:2503.11926, 2025-03-14. A chain-of-thought monitor in the reward helps at low optimization; strong optimization teaches agents to hide intent. ↩︎
  32. Samuel Marks, Johannes Treutlein, Trenton Bricken et al. “Auditing language models for hidden objectives.” arXiv:2503.10965, 2025-03-14. Three of four blind teams found a planted hidden objective. ↩︎
  33. Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan et al. “AI Control: Improving Safety Despite Intentional Subversion.” arXiv:2312.06942, 2023-12-12. On programming problems, protocols that check an untrusted model’s code with a weaker trusted model and limited trusted labor improve on the safety-usefulness trade-off of simple baselines; safety means no backdoored code is submitted. ↩︎
  34. Artem Chaikin, Shivan Kaul Sahib (Brave). “Agentic Browser Security: Indirect Prompt Injection in Perplexity Comet.” Brave blog, 2025-08-20. https://brave.com/blog/comet-prompt-injection/. Step-by-step injection and exfiltration chain. ↩︎
  35. Marco Milanta, Luca Beurer-Kellner (Invariant Labs). “GitHub MCP Exploited: Accessing private repositories via MCP.” Invariant Labs blog, 2025-05-26. https://invariantlabs.ai/blog/mcp-github-vulnerability. A malicious public issue leads an agent to leak private repository data. ↩︎
  36. MITRE. “ATLAS: Adversarial Threat Landscape for AI Systems” (release 2026.09). 2026-09-15. https://atlas.mitre.org/ (data file https://atlas.mitre.org/atlas-data/dist/v6/ATLAS-2026.09.yaml). Techniques and case studies. ↩︎
  37. OWASP GenAI Security Project. “OWASP Top 10 for LLM Applications 2026.” version 1.0, resource page dated 2026-08-03. https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/. The 2026 list. ↩︎
  38. OWASP GenAI Security Project, Agentic Security Initiative. “OWASP Top 10 for Agentic Applications 2026.” resource page dated 2025-12-09. https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/. Items ASI01 to ASI10. ↩︎