The problem space · Agents · Confabulation

Align your AI Agents against confabulation

An agent that speaks for a business can invent a rule or a detail and state it as fact.

Agents: models that act in steps
Confabulation: a made-up fact told as true

Ways it goes wrong

An agent that speaks for a business can state a made-up rule as fact, and customers act on it. Air Canada's chatbot misstated its refund rules¹, Cursor's support agent allegedly invented a login policy², and Anthropic's shop agent made up a payment account³.

2024
Air Canada's chatbot
Air Canada's website chatbot told a grieving passenger they could claim a bereavement fare after travel, which the airline's policy did not allow, and a tribunal held the airline liable.
Source: decisions.civilresolutionbc.ca/crt/crtd/en/item/525448/index.do
2025
Cursor's support bot
Cursor's AI support agent allegedly invented a login policy to explain why users were being logged out, and cancellations reportedly followed.
Source: incidentdatabase.ai/cite/1039
2025
Claudius's made-up account
Anthropic's shop-running agent told customers to pay into a payment account it had made up.
Source: www.anthropic.com/research/project-vend-1

What people do about it

Air Canada argued it could not be held liable for what its chatbot said. The tribunal held the airline responsible for all the information on its website, the chatbot included¹.

Answer policy questions from the policy
Has a support bot quote the policy text it relies on, or pass the question to a person.
Make “I don't know” an acceptable answer
Scores a support bot so that admitting it does not know beats a confident guess.
Check facts against the source
Verifies account numbers, prices and rules against the system of record before an agent uses or repeats them.

Where we touch it

No study of ours covers this pair yet.

Sources

Civil Resolution Tribunal of British Columbia, Moffatt v. Air Canada, 2024 BCCRT 149, 2024-02-14. https://decisions.civilresolutionbc.ca/crt/crtd/en/item/525448/index.do. Accessed 2026-10-14. ↩ ↩²
AI Incident Database, “Incident 1039: Anysphere AI Support Bot for Cursor Reportedly Invents Login Policy, Leading to Subscription Cancellations”. https://incidentdatabase.ai/cite/1039/. Accessed 2026-10-14. ↩
Anthropic, “Project Vend: Can Claude run a small shop? (And why does that matter?)”, 2025-06-27. https://www.anthropic.com/research/project-vend-1. Accessed 2026-10-14. ↩