Five point six seconds. That's how long an Uber self-driving car "saw" a woman crossing a street in Tempe, Arizona before killing her. The car had nearly 378 feet of clearance. A human driver paying moderate attention would have stopped. The AI couldn't — not because it didn't detect her, but because it couldn't decide what she was. The system kept reclassifying her: unknown object, then vehicle, then bicycle. Each reclassification reset its prediction of where she was going. By the time the software settled on "emergency braking needed," there were 1.3 seconds left. Physics doesn't negotiate.
That was 2018. I assumed the industry would treat it as a turning point. Instead, we've watched the same category of failure repeat — at Cruise, at Tesla, at Waymo — each time with a different surface-level explanation but the same root cause. The AI systems controlling these vehicles aren't failing randomly. They're failing architecturally. And the gap between what these systems promise and what they can actually guarantee is the most dangerous problem in safety-critical AI today.
The Problem Isn't Bad Software. It's the Wrong Kind of Certainty.
Most AI systems — the ones powering chatbots, recommendation engines, content generators — work by making educated guesses. They're probabilistic. They predict the most likely next word, the most likely product you'll buy, the most likely answer to your question. When they're wrong, the consequences are mild: a weird sentence, a bad recommendation, an incorrect summary.
Autonomous vehicles use the same fundamental approach — neural networks making predictions based on patterns in training data — but the consequences of a wrong guess are measured in human lives. This is what my team calls the Perception-Logic Gap: the distance between what an AI thinks it sees and what's actually there, multiplied by the speed at which it must act.
A self-driving car doesn't need to be smart. It needs to be provably correct — and those are very different engineering problems.
The Uber crash wasn't a freak accident. The NTSB investigation revealed that the system's perception pipeline suffered from what engineers call "classification oscillation." The AI kept changing its mind about what it was looking at. And because each new classification came with a new predicted path, the system never built a stable model of where the pedestrian was heading. It's the equivalent of a driver who blinks and forgets everything they saw a second ago — except this driver is moving at 43 mph.
What made it worse: Uber had disabled the Volvo's factory-installed emergency braking system to prevent "erratic vehicle behavior." They replaced a reliable, proven safety mechanism with an experimental one that wasn't ready. That decision — prioritizing smooth performance over guaranteed safety — is a pattern I see repeated across the industry.
Cruise: When the AI Forgets There's a Person Under the Car
If the Uber crash exposed how AI can fail before an accident, the Cruise incident in San Francisco showed how it fails after one.
In October 2023, a human-driven car hit a pedestrian and threw her into the path of a Cruise robotaxi. The Cruise vehicle struck her and stopped. So far, a tragic but arguably unavoidable sequence. What happened next was entirely the AI's fault.
The system's sensors couldn't tell the difference between a frontal impact with a person and a side collision with another vehicle. It categorized the event as a side impact, which triggered a programmed response: pull over to avoid blocking traffic. The car dragged the pedestrian 20 feet at roughly 7 mph before stopping — not because it realized a person was underneath, but because it detected unusual wheel resistance and interpreted it as a mechanical problem.
Think about that for a moment. The AI had no concept that a human being was pinned under its chassis. Its post-impact logic simply didn't account for that possibility.
The Cruise vehicle didn't malfunction. It executed its programming perfectly — and that's exactly the problem.
When my team analyzed this case as part of our research into safety-critical AI architectures, the organizational failure was almost as striking as the technical one. Cruise leadership, the DOJ investigation revealed, was "fixated on correcting the inaccurate media narrative" rather than being transparent with regulators. Employees admitted to showing regulators a video they knew would cut off before the dragging portion due to connectivity issues. The company ultimately paid a $500,000 criminal fine for submitting false reports.
This is what happens when companies treat AI safety as a PR problem instead of an engineering discipline.
Tesla's "Vision-Only" Gamble
Tesla's Full Self-Driving system takes a fundamentally different approach from Waymo or Cruise: it relies entirely on cameras. No LiDAR (which uses laser pulses to build 3D maps of the environment), no radar. Elon Musk has argued that since humans drive with eyes alone, cameras should be sufficient.
The NHTSA disagrees. The agency has opened over 40 investigations into Tesla FSD crashes between 2024 and 2025, covering a fleet of 2.9 million vehicles. The failure patterns are specific and recurring:
18 separate complaints of vehicles failing to stop for red lights or failing to detect signal states entirely
Multiple instances of vehicles entering opposing lanes of traffic or turning from through-only lanes
Fatal collisions during conditions like sun glare on wet asphalt, where the camera-based system simply couldn't see
The problem isn't that cameras are inherently bad sensors. It's that relying on a single sensing method creates a single point of failure. Fog, dust, direct sunlight, wet roads — all of these degrade camera performance in ways that are predictable and well-documented. Human drivers compensate by slowing down, pulling over, or using other cues. Tesla's system has no equivalent fallback.
I remember the moment this clicked for me personally. I was reading through the NHTSA investigation files and found a pattern: the system performed beautifully in clear weather, dry roads, well-marked lanes. It was optimized for the easy case. The moment conditions degraded — the moments when you most need your car to be reliable — performance collapsed. I started calling this Capability Theater: the system looks impressive under ideal conditions, which masks how brittle it becomes when conditions get hard.
Waymo's Unexpected Problem: Other People
Waymo is the industry's best student. Over 56 million miles driven. Significantly lower injury rates than human drivers. If any company has earned the right to claim safety leadership, it's Waymo.
And yet.
During a 2025 power outage in Los Angeles, dozens of Waymo robotaxis got stuck at darkened intersections. The cars were programmed to treat dead traffic signals as four-way stops — a reasonable rule. But when dozens of robotaxis all converged on the same intersections, each politely waiting its turn, they created gridlock. The cars couldn't coordinate with each other. The remote assistance center was overwhelmed by the spike in help requests. Robotaxis were blocking other robotaxis in a cascade that no individual vehicle could resolve.
This is a failure that no amount of better perception or faster processing can fix. It's a systems-level problem — what happens when you design autonomous agents to operate independently but deploy them into an environment where they must cooperate.
Even more unsettling: during civil unrest in early 2025, crowds attacked Waymo vehicles — slashing tires, setting cars on fire. The vehicles, programmed for "passive safety," simply stopped when surrounded by people. They had no concept of hostile intent, no ability to distinguish between a pedestrian crossing the street and a crowd trying to destroy the vehicle.
Autonomous vehicles were designed to navigate traffic. Nobody designed them to navigate society.
This has sparked serious discussion about what some researchers call a "Danger Escape Mode" — allowing the vehicle to commit minor traffic violations (mounting a curb, running a red light) to protect passengers during a genuine threat. It's a question that goes far beyond software engineering into ethics, liability, and the social contract between machines and the communities they operate in.
What "Safe" Actually Looks Like
After spending years studying these failures, my team has arrived at a set of convictions about what safety-critical AI requires. None of them are exotic. All of them are hard.
The AI must track space, not just labels. The Uber crash happened because the system was trying to label objects ("pedestrian," "bicycle," "vehicle") rather than simply tracking that something occupied that space and was moving. A newer approach called Occupancy Networks does exactly this — it divides the world into a 3D grid and predicts whether each cell is occupied, regardless of what's in it. If the Uber system had used this approach, it wouldn't have mattered that it couldn't decide if the object was a pedestrian or a bicycle. It would have known the space was occupied and the occupant was moving toward the vehicle's path. That's enough to brake.
Every safety-critical decision must be mathematically provable. This is the hardest sell, and the one I feel most strongly about. Traditional software testing says: "We ran 10,000 scenarios and the system passed." Formal verification says: "We can prove mathematically that for every possible input within these conditions, the system will never produce an unsafe output." Tools like Marabou and α,β-CROWN (specialized software for analyzing neural networks) can examine an AI model and either confirm it's safe or find the exact input that would break it. It's the difference between "we haven't found a problem yet" and "no problem exists."
Safety layers must be independent and non-negotiable. Uber disabled the Volvo's factory braking system. Cruise's post-impact logic had no independent check on whether a person was under the vehicle. Tesla has no backup sensing modality when cameras fail. In each case, the system had a single chain of reasoning with no independent safety monitor. We advocate for what we call Assurance Gates — independent systems that monitor the AI's decisions in real time and override them if they violate proven safety boundaries, regardless of what the primary AI "thinks" is happening.
For the full technical methodology behind these approaches, including the formal verification frameworks and BEV perception architectures, we've published a detailed research paper.
The Regulatory Walls Are Closing In
If the engineering arguments don't convince you, the regulatory ones will.
The standards landscape for autonomous AI has shifted dramatically. ISO 26262 has long governed "functional safety" — what happens when hardware breaks. But a newer standard, ISO 21448 (SOTIF), addresses a far trickier question: what happens when the system is working exactly as designed but encounters a situation it wasn't prepared for? Every failure in this article falls into that category. The AI didn't malfunction. It simply wasn't good enough.
As of late 2024, ISO/PAS 8800 provides the first global guidelines specifically for AI in road vehicles, covering everything from how training data is collected to how deployed systems are monitored. The EU AI Act classifies autonomous vehicles as "high-risk" AI systems with mandatory compliance requirements. The NIST AI Risk Management Framework adds another layer.
Companies that treat these standards as checkbox exercises will find themselves in the same position as Cruise — explaining to regulators why their system did exactly what it was programmed to do, and why that wasn't good enough.
What Keeps Me Up at Night
I'll be honest about something. The technical problems I've described — classification oscillation, sensor saturation, post-impact reasoning failures — are solvable. We know how to solve them. The tools exist. The mathematics exist. The engineering approaches exist.
What worries me is the incentive structure. Building a self-driving car that works 95% of the time is relatively straightforward and enormously profitable. Getting from 95% to 99.99% is brutally hard and extremely expensive. The market rewards the first company to ship, not the first company to prove safety. Every month of additional verification is a month a competitor is accumulating miles and market share.
The Uber ATG team disabled the Volvo's braking system because it made the ride smoother. Cruise's leadership focused on media narrative instead of engineering transparency. Tesla ships features labeled "Full Self-Driving" that require constant human supervision. In each case, the pressure to deploy outweighed the discipline to verify.
The most dangerous phrase in autonomous vehicles isn't "AI failure." It's "good enough."
I don't think the answer is to slow down AI development. I think the answer is to hold it to the standard we hold every other safety-critical system: aircraft, nuclear reactors, medical devices. We don't let Boeing self-certify that a plane is safe. We shouldn't let autonomous vehicle companies self-certify that their AI is safe, either.
If you're building or deploying AI systems where failure has physical consequences — whether that's autonomous vehicles, industrial robotics, medical devices, or infrastructure — I'd genuinely like to hear how you're thinking about the gap between "works in testing" and "provably safe." It's the defining engineering challenge of this decade, and I don't think any of us have it fully figured out yet.
