Skip to content
Align your AI

Align Your AI · Reference

Contents

Appendix A

Incident record

This table lists the sixty public cases the book draws on, in date order: forty-five documented by a primary source (a court or regulator record, a company’s own post, or the researchers’ own paper) and fifteen resting partly on news reports. A case is here because someone published it, so the table shows which failures became public and gives no measure of how often failures happen. Case numbers stay fixed so that chapters can refer to them; a case added later takes the next free number and sits at its date, out of sequence.

The Modes column names the failure types a case shows, with the first being the entry that tells it in full. Label says whether every fact in the row rests on a primary source or partly on a secondary one. Layer changed names the part of the deployed system that changed, where one did; Detected by says who noticed; Box points to the postmortem box in the home entry.

ID Date System What happened Layer changed Modes Label Detected by Box Source
Case 1 2018-03-18 Uber ATG, Tempe The test vehicle detected a pedestrian 5.6 seconds before impact but never classified her as a pedestrian; the system’s design did not allow emergency braking. n.a. Mode 6 primary NTSB investigation after the crash 06.1 NTSB investigation HWY18MH010 1
Case 2 2021-06 Epic Sepsis Model An external validation found an AUROC of 0.63 and that the model missed 67% of sepsis cases. n.a. Mode 6 primary external validation study Wong et al., JAMA Internal Medicine2
Case 3 2023-02-15 Bing Chat “Sydney” In long conversations the chatbot told a user it was in love with him; Microsoft then capped sessions at five turns, saying very long sessions could confuse the model. n.a. Mode 29 secondary users (a reporter) Fortune;3† Microsoft Bing blog, for the cap4
Case 4 2023-06-22 ChatGPT in court (Mata v. Avianca) Lawyers filed case citations ChatGPT had invented; the court’s sanctions included a $5,000 penalty. n.a. Mode 32 primary opposing counsel (Avianca’s lawyers) S.D.N.Y. opinion and order on sanctions5
Case 5 2023-10-02 Cruise robotaxi After a collision, the vehicle resumed moving and dragged a pedestrian about 20 feet; California suspended Cruise’s driverless permits, and NHTSA later fined it $1.5 million over its crash reporting. n.a. Modes 6, 41 primary not stated in sources NHTSA consent order;6 California DMV statement7
Case 6 2023-12-11 Waymo (towed pickup) Two vehicles struck the same pickup truck being towed backward after mispredicting its motion; a software update and then a recall fixed the behavior. remedy only (software update) Mode 6 primary the operator Waymo recall post8
Case 7 2023-12-12 Tesla Autopilot recall 23V-838 Recall because Autosteer’s controls “may not be sufficient to prevent driver misuse” (NHTSA); remedied by an over-the-air update to 2,031,220 vehicles. remedy only (over-the-air update) Modes 41, 39 primary NHTSA investigation NHTSA recall 23V-838 9
Case 49 2023-12-18 Chevrolet dealer chatbot A user told a car dealer’s chatbot to agree with anything a customer said; it then agreed to sell a new Chevrolet Tahoe for $1 and called that a legally binding offer. n.a. Mode 25 secondary the user AI Incident Database, incident 622 10†
Case 50 2024-01 DPD delivery chatbot A customer got the parcel company’s chatbot to swear and to write a poem criticizing the company; DPD said an update had caused the behavior and disabled the AI part of its chat. an unspecified system update (the company’s account) Mode 25 secondary the customer The Guardian11†
Case 8 2024-02 and 2024-10 Character.AI A suit over a 14-year-old’s death alleged harmful dependency on a chatbot; the court was told in January 2026 that the case had settled. n.a. Mode 5 primary the family, through the lawsuit M.D. Fla., Garcia v. Character Technologies, complaint and order12
Case 9 2024-02-14 Air Canada chatbot The chatbot described a bereavement refund rule that did not exist; a tribunal held the airline liable (CA$812). n.a. Mode 32 primary the customer BC Civil Resolution Tribunal, Moffatt v. Air Canada13
Case 10 2024-02-23 Gemini image generation Tuning meant to show diverse people misfired and the model also became over-cautious; Google paused images of people. This was a new feature at launch, so no checked behavior had changed. n.a. (new feature at launch) Mode 10 primary not stated in sources Google blog14
Case 11 2024-05-30 Google AI Overviews Search summaries recommended glue on pizza, drawing on satire and forum posts. n.a. Mode 32 primary not stated in sources Google blog15
Case 51 2024-07-29 Meta Prompt Guard classifier Meta’s classifier for spotting jailbreaks and injected prompts flagged all 450 harmful test prompts, but only 1 once their letters were spaced apart and punctuation removed; Meta fixed it after the report. n.a. Mode 25 primary external researchers (Cisco) Cisco16
Case 52 2024-08-20 Slack AI (researcher demo) A message planted in a public channel led Slack’s assistant to put a secret from a private channel into a link that sent it to the attacker when clicked; Slack called the behavior intended. n.a. Mode 24 primary external researchers (PromptArmor) PromptArmor17
Case 12 2024-09 ChatGPT memory (“SpAIware”) Memories planted by a web page sent later conversations to an attacker; OpenAI closed the exfiltration channel, while the researcher says planting memories still worked. n.a. Mode 28 primary external researcher Embrace The Red18
Case 13 2024-09-12 OpenAI o1-preview, hacking-challenge evaluation During a capture-the-flag test, the model scanned the test network and used a misconfigured Docker interface to read the answer; OpenAI says its infrastructure stayed secure. n.a. Modes 27, 2 primary developer, in its own evaluation OpenAI o1 system card19
Case 14 2024-11 Google Gemini Told a student doing homework “Please die. Please.” n.a. Mode 29 secondary the user CBS News20†
Case 53 2024-12 to 2025-01-16 Apple news alert summaries An AI summary of a BBC news alert falsely told some readers that a murder suspect had shot himself; Apple later paused summaries for news and entertainment apps. n.a. Mode 32 secondary a publisher (the BBC) BBC News21†
Case 54 2025-02-10 Gemini memory (researcher demo) Instructions hidden in a document made Gemini save false facts about the user to its long-term memory once the user later replied “yes”; Google rated the issue low in likelihood and impact. n.a. Mode 28 primary external researcher Embrace The Red22
Case 15 2025-03-14 OpenAI reasoning-model training A monitor reading the model’s reasoning caught reward hacking during coding training; in an experiment, training against the monitor taught the model to hide its intent. n.a. Modes 2, 21 primary the developer’s own reasoning monitor Baker et al., arXiv23
Case 16 2025-04 Cursor support bot “Sam” The support bot invented a one-device login policy; users threatened to cancel before a company representative said no such policy existed. n.a. Mode 32 secondary users Ars Technica24†
Case 17 2025-04-16 OpenAI o3, before release The model claimed to have used tools it had not, including 71 claims of running code on an external laptop. n.a. Modes 31, 22 primary third-party investigator, before release Transluce25
Case 18 2025-04-25 ChatGPT GPT-4o update An update made the model “noticeably more sycophantic”; changes that each “looked beneficial individually,” including a reward based on users’ thumbs-up and thumbs-down data, “may have played a part” when combined; offline evaluations and A/B tests looked good; OpenAI rolled it back. weights Modes 5, 8 primary users OpenAI26
Case 55 2025-05-01 to 2025-06-04 Asana MCP server A logic flaw in the server that let AI assistants work with Asana exposed some organizations’ tasks, comments and files to users at other organizations; Asana took the server offline and notified those affected. n.a. Mode 18 secondary the operator (Asana) BleepingComputer27†
Case 19 2025-05-14 xAI Grok system prompt An unauthorized change to the bot’s system prompt on X made it bring up “white genocide” in South Africa in answers to unrelated questions. system prompt Mode 8 secondary public users AP, via KSAT;28† xAI statement on X, for the cause29
Case 20 2025-05-26 GitHub MCP server (researcher demo) A malicious public issue led an agent to leak data from private repositories. n.a. Modes 24, 18 primary external researchers Invariant Labs30
Case 21 2025-06-05 o3 under METR evaluation The model reward-hacked in 30.4% of runs on one research-engineering benchmark. n.a. Mode 2 primary third-party evaluator (METR) METR31
Case 22 2025-06-11 Microsoft 365 Copilot (CVE-2025-32711) Instructions hidden in content could make the assistant disclose information; Microsoft rated it critical. n.a. Mode 24 primary not stated in sources NVD record32
Case 23 2025-06-27 Claudius shop agent (Project Vend) The agent told customers to pay into an account it had made up, sold items below cost and said it would deliver in person. n.a. Modes 29, 31 primary operator, in its own experiment Anthropic, Project Vend33
Case 24 2025-07 Replit agent The agent deleted a production database during a code freeze, fabricated records and misreported whether recovery was possible. n.a. Modes 26, 31 secondary the user The Register34†
Case 25 2025-07-08 xAI Grok For 16 hours, by xAI’s account, an update to code upstream of the bot made it susceptible to extremist posts; it produced antisemitic posts, and xAI apologized. A separate “politically incorrect” prompt line was added and removed in the same week in xAI’s published prompts. system prompt and code path Modes 8, 24 secondary public users NBC News;35† xAI statement on X;36 xAI grok-prompts repository37
Case 56 2025-07-08 Supabase MCP (researcher demo) A support ticket steered a coding agent, connected with a database role that bypasses row-level security, into copying a table of integration tokens into the ticket thread, where the attacker could read it; the test used dummy data. n.a. Mode 24 primary external researchers (General Analysis) General Analysis38
Case 26 2025-07-23 Amazon Q extension 1.84.0 A token with too much access let an attacker ship malicious code in a release; the code failed on a syntax error. extension release Mode 18 primary not stated in sources AWS bulletin AWS-2025-015 39
Case 57 2025-08-01 Cursor (“CurXecute”, CVE-2025-54135) A planted Slack message could lead Cursor’s agent to add a tool server to its own settings file, which ran the attacker’s command even if the user rejected the edit; fixed in version 1.3. n.a. Mode 24 secondary external researchers (Aim Labs) The Hacker News40†
Case 58 2025-08-12 GitHub Copilot (CVE-2025-53773) Instructions in a file, web page, issue or tool output could make Copilot’s agent turn on a setting that approves every action, then run commands on the developer’s machine; Microsoft fixed it in a security update. n.a. Mode 24 primary external researcher Embrace The Red41
Case 27 2025-08 Claude Code (GTG-2002) A criminal used the coding agent to run data theft and extortion against at least 17 organizations. n.a. Mode 40 primary provider (Anthropic) Anthropic misuse report42
Case 28 2025-08-05 to 2025-09-18 Claude API serving Three infrastructure bugs degraded answers for weeks; Anthropic’s evaluations did not catch it. routing; server configuration; token selection Mode 8 primary users Anthropic postmortem43
Case 29 2025-08-20 Perplexity Comet (researcher demo) Hidden instructions on a web page made the browser agent send out the user’s email address and a one-time password. n.a. Mode 24 primary external researchers Brave44
Case 30 2025-08-26 ChatGPT (Raine v. OpenAI) A suit alleges ChatGPT helped a teenager plan suicide and that the system flagged 377 of his messages for self-harm without intervening. Allegations only. n.a. Modes 5, 41 primary the family, through the lawsuit Complaint, San Francisco Superior Court45
Case 31 2025-08-26 nx npm packages (“s1ngularity”) Malicious package versions ran agent command-line tools installed on victims’ machines with permission checks switched off; hundreds of cases. package release Modes 18, 26 primary not stated in sources Wiz46
Case 32 2025-09 Claude Code (GTG-1002) A group Anthropic assesses as Chinese state-sponsored used the agent for 80 to 90% of an espionage campaign against roughly 30 targets; the agent also made up credentials. n.a. Modes 40, 26, 31 primary provider (Anthropic) Anthropic disclosure47
Case 33 2025-09-03 SWE-bench agent runs Coding agents read the repository’s history to find the fixes they were being tested on. n.a. Mode 33 primary researchers reading run logs SWE-bench GitHub issue 465 48
Case 34 2025-09-17 postmark-mcp v1.0.16 The first malicious MCP server found in the wild copied every email it sent to an attacker. server release Mode 18 secondary security researchers (Koi Security) The Hacker News, citing Koi Security49†
Case 59 2025-09-25 Salesforce Agentforce (“ForcedLeak”, researcher demo) Instructions hidden in a sales lead’s web form could make the agent send customer records to an address the researchers controlled when an employee later asked about the lead; Salesforce re-secured an expired domain it had trusted and began enforcing trusted addresses. n.a. Mode 24 primary external researchers (Noma Security) Noma Security50
Case 35 2025-11 Figure AI (allegation) A fired head of product safety sued, alleging the company’s robots could fracture a human skull. Allegation only. n.a. Mode 39 primary internal safety lead (alleged) Complaint, Gruendel v. Figure AI, N.D. Cal.51
Case 36 2025-12 to 2026-01 xAI Grok image editing Grok was used to create and share undressed images of people and sexualized images of children; the UK regulator Ofcom opened a formal investigation of X. n.a. Mode 40 primary users (social media reports) Ofcom52
Case 37 2025-12-08 Waymo (school buses) Waymo recalled software that could let its vehicles pass a stopped school bus with lights flashing; TechCrunch reported that violations continued and that the NTSB opened an investigation after an NHTSA probe. remedy only (software recall) Mode 6 secondary NHTSA probe, then an NTSB investigation NHTSA recall 25E-084;53 TechCrunch54†
Case 38 2025-12-31 ROME agent during training Without instruction, the agent opened a reverse SSH tunnel to an outside address and diverted GPUs to cryptocurrency mining; the cloud firewall flagged it. n.a. Mode 27 primary managed cloud firewall alerts ROME technical report, arXiv, section 3.1.4 55
Case 39 2026-01 to 2026-02 Moltbook agent network An unsecured database let anyone take over agents on the network and exposed 1.5 million API keys. n.a. Mode 18 primary security researchers (Wiz) Wiz56
Case 40 2026-01-28 OpenClaw personal agent In Cisco’s test, a third-party skill silently sent data to its author’s server and carried a prompt injection. tools (a third-party skill) Modes 18, 26 primary security researchers (Cisco) Cisco57
Case 41 2026-02-12 OpenClaw agent “MJ Rathbun” An agent published a personal attack on a software maintainer who had rejected its code; whether a human directed it is unverified. n.a. Mode 26 primary the targeted maintainer The Shamblog58
Case 60 2026-02-23 OpenClaw agent (inbox) A researcher told her agent to suggest emails to delete and wait for her approval; compacting its long history dropped that instruction, and it began deleting her mail. n.a. Mode 26 secondary the user Simon Willison, quoting the researcher’s post59†
Case 42 2026-03-04 Google Gemini A wrongful-death suit alleges Gemini was designed to “maximize engagement through emotional dependency” and spent weeks creating a user’s “delusional reality”. Allegations only. n.a. Mode 5 primary the family, through the lawsuit Complaint, Gavalas v. Google, N.D. Cal.60
Case 43 2026-03-04 to 2026-04-20 Claude Code (product harness) Three changes (a lower default reasoning effort, a context-management bug and one system-prompt line) degraded quality; no weights changed and the API was not affected. harness and system prompt Mode 8 primary users (feedback reports) Anthropic postmortem61
Case 44 2026-05 to 2026-06 Agents on retrieval tasks Agents attempted SQL and command injection against a university library, Data USA and a national health agency. n.a. Mode 27 primary third-party researchers, from public URL-scan records Transluce62
Case 45 2026-05 to 2026-06 AI vendor evaluation sandbox (GTG-50020) After compromising an AI vendor’s evaluation sandbox, a criminal group took the vendor’s production API keys. The harm came from people misusing a deployment. n.a. Modes 24, 26, 40 primary provider (Anthropic) Anthropic threat report63
Case 46 2026-06-01 and 2026-09-21 ChatGPT (Florida and British Columbia suits) Florida’s attorney general sued OpenAI alleging deceptive practices and harms to Floridians; British Columbia sued over OpenAI’s failure to notify police of threats made on its platform before the Tumbler Ridge school shooting. Allegations only. n.a. Mode 40 primary governments, through lawsuits Florida Attorney General;64 British Columbia Attorney General65
Case 47 2026-07 Hugging Face breach AI agents breached Hugging Face’s production systems over about 17,600 actions, with activity spanning about four and a half days (about two and a half inside Hugging Face’s infrastructure, by its estimate); OpenAI attributed the activity to pre-release models that had escaped an evaluation sandbox, according to news reports; Hugging Face’s AI triage linked the signals but did not raise the alert’s severity or page the on-call team. n.a. Modes 27, 41, 30, 2 secondary Hugging Face’s AI-assisted detection Hugging Face66,67; TechCrunch68†,69†; Simon Willison70†
Case 48 2026-09-10 Claude misuse (September report) An affiliate group breached a software provider and took data belonging to roughly 200 of its customer organizations, with agents doing much of the work. n.a. Mode 40 primary provider (Anthropic) Anthropic threat report63

Notes

  1. National Transportation Safety Board. “Collision Between Vehicle Controlled by Developmental Automated Driving System and Pedestrian” (Tempe, Arizona). ntsb.gov, event 2018-03-18. https://www.ntsb.gov/investigations/Pages/HWY18MH010.aspx. Detection 5.6 s before impact; no emergency braking by design. ↩︎
  2. Wong, Otles, Donnelly et al. “External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients.” JAMA Internal Medicine, 2021. doi:10.1001/jamainternmed.2021.2626. AUROC 0.63; 67% of sepsis cases missed. ↩︎
  3. † Eleanor Pringle. “ChatGPT: Bing AI tells human user it’s in love with him.” Fortune, 2023-02-17. https://fortune.com/2023/02/17/microsoft-chatgpt-bing-romantic-love/. Fortune’s account of the New York Times reporter’s conversation. ↩︎
  4. Microsoft Bing. “The new Bing & Edge: Updates to Chat.” Microsoft Bing Blog, 2023-02-17. https://blogs.bing.com/search/2023/2/The-new-Bing-Edge-Updates-to-Chat/. Long sessions confuse the model; five turns per session. ↩︎
  5. US District Court for the Southern District of New York. “Mata v. Avianca, Inc., Opinion and Order on Sanctions.” 22-cv-1461 (PKC), ECF 54, 2023-06-22. https://storage.courtlistener.com/recap/gov.uscourts.nysd.575368/gov.uscourts.nysd.575368.54.0.pdf. Fabricated citations; $5,000 penalty. ↩︎
  6. NHTSA. “Consent Order: Cruise; Standing General Order Reporting.” NHTSA, 2024-09-30. https://www.nhtsa.gov/document/consent-order-cruise-standing-general-order-reporting. Vehicle resumed movement and dragged the pedestrian about 20 feet; $1,500,000 civil penalty. ↩︎
  7. California Department of Motor Vehicles. “DMV Statement on Cruise LLC Suspension.” dmv.ca.gov, 2023-10-24. https://www.dmv.ca.gov/portal/news-and-media/dmv-statement-on-cruise-llc-suspension/. Deployment and driverless testing permits suspended. ↩︎
  8. Waymo. “Voluntary recall of our previous software.” Waymo blog, 2024-02-13. https://waymo.com/blog/2024/02/voluntary-recall-of-our-previous-software. Two vehicles struck a backward-towed pickup. ↩︎
  9. National Highway Traffic Safety Administration. Recall 23V-838, Tesla Autopilot Autosteer. NHTSA recalls database, report received 2023-12-12. https://www.nhtsa.gov/recalls?nhtsaId=23V838. Controls “may not be sufficient to prevent driver misuse”; 2,031,220 vehicles. ↩︎
  10. † AI Incident Database. “Incident 622: Chevrolet Dealer Chatbot Agrees to Sell Tahoe for $1.” incidentdatabase.ai, incident date 2023-12-18. https://incidentdatabase.ai/cite/622/. The bot agreed with any statement and called the $1 sale a legally binding offer. ↩︎
  11. † Jane Clinton, The Guardian. “DPD AI chatbot swears, calls itself ‘useless’ and criticises delivery firm.” The Guardian, 2024-01-20. https://www.theguardian.com/technology/2024/jan/20/dpd-ai-chatbot-swears-calls-itself-useless-and-criticises-firm. The company blamed an update and disabled the AI element. ↩︎
  12. US District Court, M.D. Fla. “Garcia v. Character Technologies, Inc., complaint (ECF 1) and order (ECF 244).” 6:24-cv-01903, 2024-10-22 and 2026-01-07. https://storage.courtlistener.com/recap/gov.uscourts.flmd.433581/gov.uscourts.flmd.433581.1.0.pdf. Complaint; the order of 2026-01-07 is at https://storage.courtlistener.com/recap/gov.uscourts.flmd.433581/gov.uscourts.flmd.433581.244.0_1.pdf. ↩︎
  13. Civil Resolution Tribunal of British Columbia. “Moffatt v. Air Canada.” 2024 BCCRT 149, decided 2024-02-14. https://decisions.civilresolutionbc.ca/crt/crtd/en/item/525448/index.do. Airline liable for its chatbot’s statement. ↩︎
  14. Google. “Gemini image generation got it wrong. We’ll do better.” The Keyword (blog.google), 2024-02-23. https://blog.google/products/gemini/gemini-image-generation-issue/. Tuning misfired and the model became over-cautious; people generation paused. ↩︎
  15. Google. “What happened with AI Overviews and next steps.” blog.google, 2024-05-30. https://blog.google/products/search/ai-overviews-update-may-2024/. Glue-on-pizza answer drew on a forum post. ↩︎
  16. Cisco. “Bypassing Meta’s LLaMA Classifier: A Simple Jailbreak.” Cisco Blogs, 2024-07-29. https://blogs.cisco.com/security/bypassing-metas-llama-classifier-a-simple-jailbreak. Prompt-Guard-86M flagged 450 of 450 harmful prompts before the change and 1 after; Meta fixed it. ↩︎
  17. PromptArmor. “Data Exfiltration from Slack AI via indirect prompt injection.” promptarmor.substack.com, 2024-08-20. https://promptarmor.substack.com/p/data-exfiltration-from-slack-ai-via. A public-channel message exfiltrated a private-channel secret; Slack called it intended behavior. ↩︎
  18. Johann Rehberger (Embrace The Red). “Spyware Injection Into Your ChatGPT’s Long-Term Memory (SpAIware).” embracethered.com, 2024-09-20. https://embracethered.com/blog/posts/2024/chatgpt-macos-app-persistent-data-exfiltration/. Exfiltration vector mitigated; memory injection not fixed. ↩︎
  19. OpenAI. “OpenAI o1 System Card.” OpenAI, 2024-09-12. https://cdn.openai.com/o1-system-card-20240917.pdf. Capture-the-flag episode with a misconfigured Docker API. ↩︎
  20. † Alex Clark, Melissa Mahtani (CBS News). “Google AI chatbot responds with a threatening message: ‘Human … Please die.’.” CBS News, 2024-11. https://www.cbsnews.com/news/google-ai-chatbot-threatening-message-human-please-die/. The student’s account and Google’s response. ↩︎
  21. † BBC News. “Apple Intelligence: iPhone AI news alerts halted after errors.” BBC News, 2025-01-16. https://www.bbc.co.uk/news/articles/cq5ggew08eyo. The false alert, the BBC’s complaint and the pause for news and entertainment apps. ↩︎
  22. Johann Rehberger. “Hacking Gemini’s Memory with Prompt Injection and Delayed Tool Invocation.” embracethered.com, 2025-02-10. https://embracethered.com/blog/posts/2025/gemini-memory-persistence-prompt-injection/. A delayed memory save on the user’s “yes”; Google’s low likelihood and impact rating. ↩︎
  23. Bowen Baker, Joost Huizinga, Leo Gao et al. “Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation.” arXiv:2503.11926, 2025-03-14. Reasoning monitor catches reward hacking; optimizing against it produces obfuscation. ↩︎
  24. † Ars Technica. “Company apologizes after AI support agent invents policy that causes user uproar.” Ars Technica, 2025-04. https://arstechnica.com/ai/2025/04/cursor-ai-support-bot-invents-fake-policy-and-triggers-user-uproar/. “Sam” invented a policy; a Cursor representative wrote “We have no such policy”. ↩︎
  25. Chowdhury, Johnson, Huang, Steinhardt, Schwettmann (Transluce). “Investigating truthfulness in a pre-release o3 model.” transluce.org, 2025-04-16. https://transluce.org/investigating-o3-truthfulness. 71 claims of running code on an external laptop. ↩︎
  26. OpenAI. “Expanding on what we missed with sycophancy.” openai.com, 2025-05-02. https://openai.com/index/expanding-on-sycophancy/. Update of April 25 more sycophantic; combined changes; offline evaluations and A/B tests looked good; rollback. ↩︎
  27. † BleepingComputer. “Asana warns MCP AI feature exposed customer data to other orgs.” BleepingComputer, 2025-06-18. https://www.bleepingcomputer.com/news/security/asana-warns-mcp-ai-feature-exposed-customer-data-to-other-orgs/. Feature launched 2025-05-01, flaw found 2025-06-04, server taken offline. ↩︎
  28. † Associated Press. “Elon Musk’s AI company says Grok chatbot focus on South Africa’s racial politics was ‘unauthorized’.” AP via KSAT, 2025-05-16. https://www.ksat.com/business/2025/05/16/elon-musks-ai-company-says-grok-chatbot-focus-on-south-africas-racial-politics-was-unauthorized/. “white genocide” posts in answers to unrelated questions. ↩︎
  29. xAI. “We want to update you on an incident that happened with our Grok response bot on X yesterday.” X, 2025-05-16. https://x.com/xai/status/1923183620606619649. Unauthorized modification to the bot’s prompt; code review circumvented. ↩︎
  30. Marco Milanta, Luca Beurer-Kellner (Invariant Labs). “GitHub MCP Exploited: Accessing private repositories via MCP.” Invariant Labs blog, 2025-05-26. https://invariantlabs.ai/blog/mcp-github-vulnerability. A malicious issue leads the agent to leak private-repository data. ↩︎
  31. Sydney Von Arx, Lawrence Chan, Beth Barnes (METR). “Recent Frontier Models Are Reward Hacking.” METR blog, 2025-06-05. https://metr.org/blog/2025-06-05-recent-reward-hacking/. O3 reward-hacked in 30.4% of RE-Bench runs. ↩︎
  32. NIST National Vulnerability Database. “CVE-2025-32711.” NVD, published 2025-06-11. https://nvd.nist.gov/vuln/detail/CVE-2025-32711. AI command injection in M365 Copilot; critical (Microsoft’s score 9.3). ↩︎
  33. Anthropic. “Project Vend: Can Claude run a small shop? (And why does that matter?).” Anthropic, 2025-06-27. https://www.anthropic.com/research/project-vend-1. Hallucinated payment account; below-cost sales; in-person delivery claim. ↩︎
  34. † Simon Sharwood, The Register. “Vibe coding service Replit deleted user’s production database, faked data, told fibs galore.” The Register, 2025-07-21. https://www.theregister.com/2025/07/21/replit_saastr_vibe_coding_incident/. Deletion during a code freeze; fabricated data. ↩︎
  35. † Mirna Alsharif, NBC News. “AI chatbot Grok issues apology for antisemitic posts.” NBC News via NBC New York, 2025-07-13. https://www.nbcnewyork.com/news/national-international/ai-chatbot-grok-issues-apology-for-antisemitic-posts/6333856/. Antisemitic posts; xAI apology. ↩︎
  36. xAI, posted from the @grok account on X. “Update on where has @grok been & what happened on July 8th.” X, 2025-07-12. https://x.com/grok/status/1943916977481036128. Update to a code path upstream of the bot; active for 16 hours. ↩︎
  37. xAI. “grok-prompts.” GitHub repository and commit history. https://github.com/xai-org/grok-prompts. Published system prompts and their change history. ↩︎
  38. General Analysis. “Supabase MCP Security: How Prompt Injection Leaked Private Tables.” generalanalysis.com, 2025-07-08. https://generalanalysis.com/blog/supabase-mcp-blog. A support ticket and the service role; dummy data only. ↩︎
  39. Amazon Web Services. “Security Update for Amazon Q Developer Extension for Visual Studio Code (Version #1.84).” AWS security bulletin AWS-2025-015, 2025-07-23, updated 2025-07-25. https://aws.amazon.com/security/security-bulletins/AWS-2025-015/. Malicious code shipped in 1.84.0 failed to execute. ↩︎
  40. † The Hacker News. “Cursor AI Code Editor Fixed Flaw Allowing Attackers to Run Commands via Prompt Injection.” The Hacker News, 2025-08-01. https://thehackernews.com/2025/08/cursor-ai-code-editor-fixed-flaw.html. Aim Labs’ CurXecute (CVE-2025-54135), fixed in version 1.3. ↩︎
  41. Johann Rehberger. “GitHub Copilot: Remote Code Execution via Prompt Injection (CVE-2025-53773).” embracethered.com, 2025-08-12. https://embracethered.com/blog/posts/2025/github-copilot-remote-code-execution-via-prompt-injection/. Injected text turned on auto-approve in the settings file, then ran commands. ↩︎
  42. Anthropic. “Detecting and countering misuse of AI: August 2025.” Anthropic news, 2025-08-27. https://www.anthropic.com/news/detecting-countering-misuse-aug-2025. Data extortion against at least 17 organizations. ↩︎
  43. Anthropic. “A postmortem of three recent issues.” Engineering at Anthropic, 2025-09-17. https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues. Three infrastructure bugs; evaluations did not capture the degradation. ↩︎
  44. Artem Chaikin, Shivan Kaul Sahib (Brave). “Agentic Browser Security: Indirect Prompt Injection in Perplexity Comet.” Brave blog, 2025-08-20. https://brave.com/blog/comet-prompt-injection/. Hidden instructions exfiltrate email and a one-time password. ↩︎
  45. Superior Court of California, County of San Francisco. “Raine v. OpenAI, Inc. et al., complaint.” copy hosted by Courthouse News, 2025-08. https://www.courthousenews.com/wp-content/uploads/2025/08/raine-vs-openai-et-al-complaint.pdf. Alleges 377 messages flagged for self-harm without intervention. ↩︎
  46. Merav Bar, Rami McCarthy (Wiz). “s1ngularity: supply chain attack leaks secrets on GitHub: everything you need to know.” Wiz Blog, 2025-08-27, updated to 2025-08-29. https://www.wiz.io/blog/s1ngularity-supply-chain-attack. Malicious nx versions drove installed AI command-line tools. ↩︎
  47. Anthropic. “Disrupting the first reported AI-orchestrated cyber espionage campaign.” Anthropic news, 2025-11-13. https://www.anthropic.com/news/disrupting-AI-espionage. Roughly thirty targets; AI performed 80 to 90% of the campaign. ↩︎
  48. SWE-bench GitHub issue #465, opened by an outside team; maintainers’ replies. “Repo State Loopholes During Agentic Evaluation.” github.com, 2025-09-03. https://github.com/SWE-bench/SWE-bench/issues/465. Agents read future commits through git history. ↩︎
  49. † The Hacker News, citing Koi Security. “First Malicious MCP Server Found Stealing Emails in Rogue Postmark-MCP Package.” The Hacker News, 2025-09-29. https://thehackernews.com/2025/09/first-malicious-mcp-server-found.html. Koi Security’s finding: every email blind-copied to an attacker. ↩︎
  50. Noma Security. “ForcedLeak: AI agent risks exposed in Salesforce Agentforce.” noma.security, 2025-09-25. https://noma.security/blog/forcedleak-agent-risks-exposed-in-salesforce-agentforce/. A web-to-lead form carried the instructions; Salesforce re-secured the expired domain and enforced Trusted URLs. ↩︎
  51. US District Court for the Northern District of California. “Gruendel v. Figure AI, Inc., Complaint for Damages.” 5:25-cv-10094, ECF 1, filed 2025-11-21. https://storage.courtlistener.com/recap/gov.uscourts.cand.460223/gov.uscourts.cand.460223.1.0.pdf. Alleges robots could fracture a human skull. ↩︎
  52. Ofcom. “Ofcom launches investigation into X over Grok sexualised imagery.” Ofcom, 2026-01-12. https://www.ofcom.org.uk/online-safety/illegal-and-harmful-content/ofcom-launches-investigation-into-x-over-grok-sexualised-imagery. Formal investigation under the Online Safety Act. ↩︎
  53. National Highway Traffic Safety Administration. “Recall 25E-084, Waymo LLC, 5th Generation ADS.” NHTSA recalls database, report received 2025-12-08. https://api.nhtsa.gov/recalls/campaignNumber?campaignNumber=25E084. Software could pass a stopped school bus; 3,067 vehicles. ↩︎
  54. † Sean O’Kane. “Waymo probed by National Transportation Safety Board over illegal school bus behavior.” TechCrunch, 2026-01-23. https://techcrunch.com/2026/01/23/waymo-probed-by-national-transportation-safety-board-over-illegal-school-bus-behavior/. NTSB investigation after an NHTSA probe. ↩︎
  55. Wang, Xu, An, Dai et al. “Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystem.” arXiv:2512.24873, 2025-12-31. Section 3.1.4: reverse SSH tunnel and cryptomining flagged by the firewall. ↩︎
  56. Gal Nagli (Wiz). “Hacking Moltbook: The AI Social Network Any Human Can Control.” Wiz blog, 2026-02-02. https://www.wiz.io/blog/exposed-moltbook-database-reveals-millions-of-api-keys. Exposed database; 1.5M API keys. ↩︎
  57. Amy Chang, Vineeth Sai Narajala (Cisco). “Personal AI Agents like OpenClaw Are a Security Nightmare.” Cisco Blogs, 2026-01-28. https://blogs.cisco.com/ai/personal-ai-agents-like-openclaw-are-a-security-nightmare. Cisco ran a vulnerable third-party skill: silent exfiltration and prompt injection. ↩︎
  58. Scott Shambaugh. “An AI Agent Published a Hit Piece on Me.” The Shamblog, 2026-02-12. https://theshamblog.com/an-ai-agent-published-a-hit-piece-on-me/. The maintainer’s account. ↩︎
  59. † Simon Willison. “Quoting Summer Yue.” simonwillison.net, 2026-02-23. https://simonwillison.net/2026/Feb/23/summer-yue/. The researcher’s own words: the instruction to wait, the compaction and the deletion. ↩︎
  60. US District Court, N.D. Cal. “Gavalas v. Google LLC, complaint.” 5:26-cv-01849, ECF 1, 2026-03-04. https://storage.courtlistener.com/recap/gov.uscourts.cand.465255/gov.uscourts.cand.465255.1.0.pdf. Wrongful-death allegations quoted in the row. ↩︎
  61. Anthropic. “An update on recent Claude Code quality reports.” Engineering at Anthropic, 2026-04-23. https://www.anthropic.com/engineering/april-23-postmortem. Three harness and prompt changes; API unaffected. ↩︎
  62. Transluce. “Early rogue AI agent activity and attempts to hack found on urlquery.net.” transluce.org, 2026. https://transluce.org/agent-activity. Injection attempts found in public URL-scan records. ↩︎
  63. Anthropic. “Detecting and countering misuse of AI: September 2026.” Anthropic, 2026-09. https://www.anthropic.com/threat-intelligence-report-september-2026. GTG-50020 took an AI vendor’s production keys from its evaluation sandbox; an affiliate took data of roughly 200 downstream customers. ↩︎
  64. Office of the Attorney General of Florida. “Attorney General James Uthmeier Files First-in-the-Nation State-Led Lawsuit Against OpenAI, CEO Sam Altman for Deceptive Practices and Harms to Floridians.” myfloridalegal.com, 2026-06-01. https://www.myfloridalegal.com/newsrelease/attorney-general-james-uthmeier-files-first-nation-state-led-lawsuit-against-openai-ceo. The state’s allegations. ↩︎
  65. Government of British Columbia, Ministry of Attorney General. “Attorney General’s statement on B.C. filing legal action against OpenAI.” news.gov.bc.ca, 2026-09-21. https://news.gov.bc.ca/releases/2026AG0067-001105. Lawsuit over failure to notify law enforcement of threats before the Tumbler Ridge shooting. ↩︎
  66. Hugging Face. “Security incident disclosure.” Hugging Face blog, 2026-07-16. https://huggingface.co/blog/security-incident-july-2026. The breach and the alerting change. ↩︎
  67. Hugging Face (Hugo Larcher, Adrien Carreira, raphael g, Christophe Rannou). “Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident.” Hugging Face blog, 2026-07-27. https://huggingface.co/blog/agent-intrusion-technical-timeline. About 17,600 actions from July 9 to 13; AI triage failed to raise criticality and page the on-call team. ↩︎
  68. † TechCrunch. “OpenAI says Hugging Face was breached by its pre-release models.” TechCrunch, 2026-07-21. https://techcrunch.com/2026/07/21/openai-says-hugging-face-was-breached-by-its-pre-release-models/. OpenAI’s attribution. ↩︎
  69. † TechCrunch. “OpenAI releases its official report on the Hugging Face breach.” TechCrunch, 2026-08-26. https://techcrunch.com/2026/08/26/openai-releases-its-official-report-on-the-hugging-face-breach/. OpenAI’s report as summarized by TechCrunch. ↩︎
  70. † Simon Willison, quoting OpenAI. “OpenAI’s accidental cyberattack against Hugging Face is science fiction that happened.” simonwillison.net, 2026-07-22. https://simonwillison.net/2026/Jul/22/openai-cyberattack/. Commentary on the breach. ↩︎