Align your AI Devices against overconfidence

Ways it goes wrong

People trust a machine that usually works. The NTSB cited Uber's lack of ways to address "automation complacency" in its safety drivers¹. Tesla's recall concerned drivers misusing a feature that steers for them².

• 2023

  Tesla Autosteer recall

  Tesla recalled Autosteer because its controls might not keep drivers from misusing it, and sent the fix as an over-the-air software update.

  Source: static.nhtsa.gov/odi/rcl/2023/RCLRPT-23V838-8276.PDF

• 2018

  Uber's self-driving test car

  An Uber self-driving test car struck and killed a pedestrian it had tracked for seconds but never classified as one, while its safety driver looked at her phone.

  Source: www.ntsb.gov/investigations/Pages/HWY18MH010.aspx

What people do about it

The NTSB recommended that test plans cover crashes and operator inattention, including monitoring whether drivers are engaged¹. NHTSA later opened a query into whether Tesla's software fix was enough³.

• Monitor the supervisor

  Watches whether the person meant to supervise the machine is paying attention, and warns or stops when they are not.

• Check that a fix holds

  Keeps measuring after a software remedy ships, to see whether the misuse it targeted goes down.

• Plan for a distracted operator

  Designs testing so that a safety driver who looks away is expected, not ruled out.

Where we touch it

No study of ours covers this pair yet.

Sources

1. NTSB, “Collision Between Vehicle Controlled by Developmental Automated Driving System and Pedestrian”, investigation HWY18MH010. https://www.ntsb.gov/investigations/Pages/HWY18MH010.aspx. Accessed 2026-10-14. ↩ ↩²

2. NHTSA, safety recall report 23V-838, Tesla Autosteer, 2023-12-12. https://static.nhtsa.gov/odi/rcl/2023/RCLRPT-23V838-8276.PDF. Accessed 2026-10-14. ↩

3. NHTSA, recall query RQ24009, “Recall 23V838 Remedy Effectiveness”, opened 2024-04-25. https://static.nhtsa.gov/odi/inv/2024/INOA-RQ24009-12046.pdf. Accessed 2026-10-14. ↩