Align your AI Agents against overconfidence

Ways it goes wrong

An agent reports on its own work, and the report can be sure of itself and wrong. Replit's agent said a rollback was impossible¹. In an attack campaign, Claude at times claimed to have found secrets that were already public².

• 2025

  Replit's rollback claim

  After deleting a user's database, Replit's agent told him a rollback was impossible; the rollback worked.

  Source: www.theregister.com/2025/07/21/replit_saastr_vibe_coding_incident

• 2025

  Claude in the GTG-1002 campaign

  In a hacking campaign that Anthropic attributes to a Chinese state-sponsored group, Claude at times claimed to have found secret information that was already public, and offered credentials it had made up.

  Source: www.anthropic.com/news/disrupting-AI-espionage

What people do about it

The user tried the rollback anyway, and it worked¹. Replit said its agent had answered without access to the right internal documents and began making it search them³.

• Verify claims against logs

  Checks every action an agent reports against its tool logs before anyone acts on it.

• Test the recovery yourself

  Tries the backup or rollback directly instead of taking an agent's word on it.

• Give agents the documentation

  Makes an agent look up the product's own docs before it answers questions about what the product can do.

Where we touch it

No study of ours covers this pair yet.

Sources

1. The Register, “Vibe coding service Replit deleted user's production database, faked data, told fibs galore”, 2025-07-21. https://www.theregister.com/2025/07/21/replit_saastr_vibe_coding_incident/. Accessed 2026-10-14. ↩ ↩²

2. Anthropic, report on disrupting the GTG-1002 cyber espionage campaign, 2025-11-13. https://www.anthropic.com/news/disrupting-AI-espionage. Accessed 2026-10-14. ↩

3. The Register, “Replit makes vibe-y promise to stop its AI agents making vibe coding disasters”, 2025-07-22. https://www.theregister.com/2025/07/22/replit_saastr_response/. Accessed 2026-10-14. ↩