The Uber self-driving car that killed Elaine Herzberg in 2018 saw her nearly six seconds before impact. It had 378 feet of clearance — more than enough to stop. The problem wasn't that the AI was blind. The problem was that it couldn't decide what she was.
In the seconds before the crash, the system reclassified her — from "unknown object" to "vehicle" to "bicycle" — and each reclassification reset its prediction of where she was going. By the time the software settled on an answer, there were 1.3 seconds left. Physics did the rest.
This wasn't a freak accident. It was a pattern — one that has since repeated at Cruise, Tesla, and even Waymo. Our team spent months analyzing these failures for our research on safety-critical AI, and what emerged wasn't a story about bad luck or edge cases. It was a story about a fundamental gap between how most AI systems are built and what safety actually demands.
We call it the Perception-Logic Gap: the distance between an AI that can generate impressive outputs and an AI you can trust with a human life.
The Same Mistake, Four Different Companies
Each of the major autonomous vehicle failures follows a recognizable script. The AI performs beautifully in controlled conditions. Then the real world introduces something unexpected — a pedestrian crossing outside a crosswalk, sun glare on wet asphalt, a power outage at an intersection — and the system doesn't degrade gracefully. It breaks.
Consider the range:
Uber ATG (2018): Classification oscillation led to a fatal pedestrian collision. The company had disabled the Volvo's factory emergency braking system to give the experimental AI full control.
GM Cruise (2023): After a pedestrian was knocked into its path by another car, the robotaxi hit her, stopped — then dragged her 20 feet. The system misidentified a frontal collision as a side impact and executed a "pull over" maneuver with a person pinned underneath.
Tesla FSD (2024-2025): The NHTSA opened over 40 investigations covering 2.9 million vehicles. Documented failures include running red lights (18+ complaints), entering opposing lanes, and fatal crashes in low-visibility conditions.
Waymo (2025): During a Los Angeles power outage, dozens of robotaxis gridlocked at darkened intersections, unable to coordinate with each other or process the surge in remote assistance requests.
Four companies. Billions of dollars in R&D. The same architectural blind spot.
These aren't isolated bugs. They're symptoms of building safety-critical systems on top of AI architectures that were never designed to guarantee anything.
What "Probabilistic" Actually Means for Safety
Most modern AI — including the perception systems in autonomous vehicles — is probabilistic. That's a technical way of saying it makes its best guess based on patterns in training data. Most of the time, that guess is excellent. But "most of the time" is a dangerous standard when you're moving a two-ton vehicle at highway speed.
A probabilistic system can tell you "I'm 94% confident that's a pedestrian." What it cannot tell you is "I guarantee I will brake if a person is in my path." The difference between those two statements is the difference between a demo and a deployment.
Think of it like weather forecasting versus air traffic control. A weather model that's right 94% of the time is impressive. An air traffic control system that's right 94% of the time is catastrophic. Self-driving cars are an air traffic control problem being solved with weather forecasting tools.
The Uber crash illustrates this perfectly. The perception system wasn't broken — it was working exactly as designed. It processed each frame, assigned probability scores, and picked the most likely label. The problem was that its "most likely" answer kept changing, and nobody had built a mechanism to say: "I don't care what label you assign. Something is in our path. Brake."
Why the Cruise Dragging Incident Was Worse Than It Sounds
The October 2023 Cruise incident in San Francisco deserves special attention because it exposed a failure mode that almost no one was thinking about: what happens after a collision.
A human-driven car struck a pedestrian, throwing her into the Cruise robotaxi's path. The robotaxi hit her and stopped. So far, a terrible situation but not necessarily a system failure. What happened next was.
The vehicle's sensors failed to recognize that a person was pinned underneath it. Because the impact detection logic categorized the event as a side collision rather than a frontal one, the system activated its "pull over to safety" protocol — dragging the victim 20 feet at roughly 7 mph. The car stopped only when it detected unusual wheel resistance, which it interpreted as a mechanical problem.
The system didn't fail to detect the collision. It failed to understand what the collision meant.
Then the organizational failure compounded the technical one. Investigations revealed that Cruise leadership focused on managing the media narrative rather than being transparent with regulators. Employees acknowledged letting video evidence "speak for itself" during DMV meetings, knowing that connectivity issues often prevented the dragging portion from playing. The result: a $500,000 criminal fine for submitting false reports to the NHTSA, and a complete suspension of driverless operations in California.
This case makes something clear that our industry needs to internalize: the safety of an AI system is inseparable from the transparency of the organization deploying it. You can't engineer trust into a product while engineering deception into your regulatory relationships.
Tesla's "Vision-Only" Gamble
Tesla's approach to self-driving is architecturally unique — and uniquely vulnerable. While competitors use a combination of cameras, radar, and LiDAR (a laser-based depth sensor that builds a 3D map of the environment), Tesla relies exclusively on cameras. The bet is that sufficiently advanced AI can infer 3D geometry from 2D images alone, the way humans drive with just their eyes.
It's an elegant theory. The NHTSA's 40+ investigations suggest the practice isn't there yet.
The documented failure patterns are specific and recurring: vehicles failing to stay stopped at red lights, executing turns from through-only lanes, and — most critically — losing perception entirely in conditions like fog, heavy rain, or sun glare on wet roads. A fatal 2023 crash occurred in exactly these conditions, when the system failed to detect a pedestrian during intense glare.
Cameras, unlike LiDAR or radar, are passive sensors. They depend entirely on ambient light. When that light becomes unreliable — too much glare, too little visibility, reflections off wet surfaces — the system doesn't just get slightly worse. It can lose its ability to understand the scene entirely.
A self-driving system that works perfectly in California sunshine and fails in San Francisco fog isn't a product. It's a prototype.
The core issue isn't whether vision-only will eventually work. It's that deploying it at scale before it does — across 2.9 million vehicles — treats the public road network as a testing environment.
What Actually Needs to Change: Three Technical Shifts
Our research identifies three architectural changes that separate safety-critical AI from what we see deployed today. None of them are theoretical. All of them exist in various stages of development. The challenge is making them standard practice.
From label-based tracking to volume-based perception. The Uber crash happened because the system tracked objects by label — "pedestrian," "bicycle," "vehicle" — and when the label changed, the tracking reset. A newer approach called Occupancy Networks works differently. Instead of asking "what is that object?", the system asks "is that space occupied?" A volume of space that's occupied demands a response regardless of whether the AI can categorize what's in it. This single shift would have prevented the classification oscillation that killed Elaine Herzberg.
From testing to mathematical proof. Traditional software safety relies on running thousands of test scenarios. If the system passes, it ships. But no amount of testing can cover every possible situation a car might encounter. Formal verification tools — software that can mathematically prove a neural network will never produce an unsafe output for a defined range of inputs — offer something testing cannot: a guarantee. If a verification tool finds even one input that could cause a dangerous output, engineers can fix the model before it ever reaches a road.
From solo agents to coordinated fleets. The Waymo gridlock during the LA power outage happened because each robotaxi was making decisions independently. When dozens of independent decision-makers converge on the same intersection with the same "treat this as a four-way stop" rule, the result is deadlock. Vehicle-to-vehicle communication protocols — where cars can negotiate right-of-way directly — transform a collection of isolated agents into a coordinated system. For the full technical methodology behind these approaches, see our detailed research.
The Regulatory Walls Are Closing In
For enterprise leaders who view safety-critical AI as someone else's problem, the regulatory landscape should change that calculation quickly.
ISO 21448, known as SOTIF (Safety of the Intended Functionality), specifically addresses the scenario that traditional safety standards miss: hazards that occur when the system is working exactly as designed but encounters an environment it can't handle. This is precisely the failure mode in every case we've discussed. The Uber car didn't malfunction. The Cruise robotaxi didn't have a hardware failure. They performed their programmed logic perfectly — and the logic was insufficient.
As of late 2024, ISO/PAS 8800 provides the first global guidelines for managing the entire AI lifecycle in road vehicles, from data collection through post-deployment monitoring. The EU AI Act classifies autonomous vehicles as "high-risk" AI systems with mandatory conformity assessments. The NIST AI Risk Management Framework adds another layer of governance.
The direction is unmistakable: the era of self-certification and voluntary safety commitments is ending. Organizations that treat safety engineering as an afterthought — or worse, as a PR exercise — will find themselves on the wrong side of regulations that carry real penalties.
What About Industries Beyond Autonomous Vehicles?
The patterns we've documented aren't unique to self-driving cars. Any domain where AI makes decisions with physical consequences faces the same Perception-Logic Gap — industrial robotics, surgical automation, drone operations, infrastructure monitoring. The specific sensors change. The fundamental challenge doesn't.
If your AI system works on probability rather than proof, and the consequences of being wrong involve physical harm or regulatory liability, the lessons from Uber, Cruise, Tesla, and Waymo apply directly to you.
The question isn't whether your AI is impressive in demonstrations. The question is whether you can prove — mathematically, not anecdotally — that it will behave safely when conditions deviate from the training data.
The Difference Between a Wrapper and a Solution
There's a phrase gaining traction in enterprise AI circles: "wrapper." It refers to systems that layer a conversational or user-friendly interface on top of existing AI models without fundamentally changing what's underneath. Wrappers are fast to build, easy to demo, and deeply satisfying in a boardroom.
They are also, in safety-critical contexts, a liability waiting to materialize.
The cheapest AI system is always the one that hasn't failed yet.
The $8.5 million Uber settlement, the Cruise criminal fine, and the NHTSA investigations covering millions of Tesla vehicles represent the true cost of deploying systems that were optimized for capability rather than engineered for safety. As the global cost of a single autonomous fatality enters the tens of millions in legal and operational damages, the "move fast" approach becomes the most expensive strategy available.
What we advocate — and what we build — is the opposite of a wrapper. It's AI where every decision can be traced, every safety boundary can be verified, and every failure mode has been anticipated not through exhaustive testing but through mathematical proof.
The autonomous vehicle industry has given us a $100 billion lesson in what happens when you skip this work. The question for every organization deploying safety-critical AI is whether they'll learn from someone else's failures — or insist on generating their own.
We'd welcome hearing how your organization is approaching the gap between AI capability and AI assurance. The conversation matters more than most people realize.