The Pixellot camera that tracked a linesman's bald head for an entire match wasn't operating at low confidence — it was at 98%. That's what I kept coming back to when I first saw the clip from the Inverness Caledonian Thistle match in 2020. The problem wasn't that the system was uncertain. It was that the system had no concept of impossible.
Under stadium floodlights, the specular highlights on the linesman's head produced pixel gradients statistically indistinguishable from a white soccer ball. "Better training data" doesn't fix that — the data-generating process produces genuinely ambiguous examples. A bald head under those lights really does look like a ball.
What would fix it: knowing that a ball can't move at a constant height of 1.7 meters at 3 mph while attached to a vertical cylindrical object. The detector had no way to know that. It found visual patterns. It had no model of physical reality.
That framing is what I've been building against with physics-constrained computer vision — and it turns out the same failure mode shows up in semiconductor fabs and manufacturing lines, not just in broadcast cameras.
The Semiconductor Engineer Who Changed My Frame
A few months later, I was talking through the false positive problem with an engineer who'd spent years doing semiconductor inspection work. I described the sports tracking problem. He described a different version of the same thing.
His description of the nuisance defect triage log — the daily review queue of flagged anomalies from a KLA broadband scan — stuck with me. The 2900 Series at his fab could detect yield-relevant defects down to 10nm. Detection capability wasn't the problem. At advanced process nodes, the scanner flagged thousands of anomalies per wafer. Most were surface artifacts, dust particles, pattern noise that wouldn't affect yield. Each required classification. Each classification routed to human review burned engineer time at the same rate whether the defect was real or not.
His frustration was specific: after every node transition, the nuisance rate spiked, and the solution was to retrain the classifier — which took weeks and didn't hold for long because the next quarter brought another process change. I asked him what the classifier was doing that was wrong. He described a system with no model of how light physically interacts with a pit versus a particle versus a stain. It had learned correlations from the old node that stopped working on the new one.
A 1% yield loss at advanced nodes translates to millions in lost revenue — individual wafers can cost tens of thousands of dollars, and KLA's $2.3 billion R&D commitment signals they know the nuisance problem is structural. But the classifier still had no physics priors at inference. The physics stopped at the tooling layer.
I kept thinking about what would happen if the classifier knew that a real pit creates a specific parallax shift between inspection angles — one that a surface particle doesn't produce. That knowledge doesn't change when the process node changes. The physics of light interaction with a geometric structure is node-invariant.
The Question That Reframed the Product
The conversation I kept having after that was some variation of: why hasn't this been solved?
My team ran through the competitive landscape looking for where physics had actually been integrated at inference, not just at development time. Pixellot's V4 multi-hypothesis tracking addressed the bald-head class of errors for professional broadcast. Hawk-Eye, the Sony company, deploys multi-camera triangulation tracking 29 skeletal points per player in NFL stadiums — six 8K cameras per venue in the 2025 season, extended to all nine ATP Masters 1000 events in 2026. These are real advances, and I want to be precise about where the Pixellot advance stops: V4 addressed the bald-head class of errors through multi-hypothesis tracking, but the system still has no kinematic model of what's impossible — new failure modes like jersey OCR under motion blur and offside projection on non-flat pitch surfaces persist with the same root cause. Hawk-Eye's $1-million-plus per venue infrastructure is genuinely physics-constrained through multi-camera calibration, but it scales to World Cup venues, not to the 40,000+ clubs using Veo cameras across 100 countries. The mid-tier bracket remains without physics-aware tracking at inference.
What surprised me most in the research was NVIDIA. Their Omniverse and Cosmos platforms use physically accurate rendering to generate synthetic training data, and the Metropolis ecosystem has over 1,000 companies building computer vision products on top of it. Physics enters the development pipeline — meaningfully. But the deployed model is still a neural network. The inference-time check running on live footage at 30fps has no physics knowledge. The physics stops at the training boundary.
I kept writing the same note: physics at training time, not inference. Every vendor. That's the gap I wanted to work in.
I started describing what we were building as a physics layer at inference — the constraint checks that sit between the detector and the action system. The most common case is temporal: the same tracked object across frames should follow a physically consistent trajectory. A Kalman filter maintains a predicted state and rejects candidates that fall outside the prediction boundary, regardless of visual confidence. Once that's in place, the next failure mode is spatial: a detection that looks real but moves differently from everything around it in the scene. Optical flow catches this — a detection that can't be explained by the local motion field gets flagged as anomalous. Where you have two synchronized cameras, epipolar geometry adds a third check: a real 3D object should appear in both views at positions consistent with its physical location. Surface particles that read as defects from one angle dissolve in the second view. Real subsurface defects hold.
None of these checks involve retraining. None of them drift with the data distribution. They add 2 to 8 milliseconds per frame and live between the detector and whatever the pipeline does next.
What I Got Wrong About Manufacturing
I assumed the manufacturing version of this problem would look like the semiconductor version — a visible, escalating triage queue that engineers noticed and complained about. My mental model was wrong.
On production lines using AI-based quality control, there are no real-time ground-truth labels during live production. A lighting angle shifts after a maintenance window. A lens hazes over weeks. A fixture wears by fractions of a millimeter. Drift accumulates silently. False-reject rates rise — more good parts pulled for rework — or false-accept rates creep up as defects escape into finished inventory. The system isn't flagging anything. It's just slowly getting worse while reporting nominal output.
The cost of poor quality runs approximately 20% of total sales for average manufacturers. Intel has documented $2 million annually in savings from AI vision inspection — but that reduction starts from a waste floor that was already substantial. And that's a company with the engineering bandwidth to instrument its production line carefully. Most manufacturers don't have an equivalent continuous validation pipeline. They have a deployed system, a maintenance schedule, and a quality alarm that fires when the escape rate crosses a threshold they set at deployment.
What changed my understanding was the production-line trend data we reviewed across early deployments. The false-reject rate charts over time had a consistent shape on purely learned classifiers: gradual drift with step-function spikes after maintenance windows. The physics-constrained checks produced a flat line through the same windows. Not because the physics model was perfect — it wasn't — but because the physical properties of a correctly manufactured part don't change when the lighting shifts. The invariance is the point.
The 95% and Why Physics Is Where It Ends
I've talked to enough engineering teams about failed CV deployments to believe the 95% production failure rate that industry data reports is roughly right.
The failure pattern isn't algorithmic. It's environmental. The model works in the lab because the lab is controlled: consistent lighting, fixed angles, limited variability. The factory floor is not. Ambient light from forklifts, sunlight through skylights, operators who vary. Edge cases — scenarios rare in training data and common in production — account for the majority of engineering time, support cost, and liability exposure once a system is live. The SoccerNet multi-object tracking benchmark puts it plainly for the sports domain: tracking across dense configurations, fast motion, and heavy occlusion is "far from solved," and no physics-aware approach has been integrated into the leading methods.
More robustness testing and staged deployment are right procedures. They don't change the underlying constraint: a neural network trained on controlled conditions has no knowledge of what's physically impossible. That knowledge can't be learned from image data. It has to be built in.
One thing I didn't anticipate when we started: the EU AI Act's transparency obligations under Article 50 take full effect in August 2026 for CV systems making decisions that affect people — access control, workplace monitoring, officiating. Physics-constrained rejections are auditable in plain language in a way I've found genuinely useful when explaining the system to non-ML reviewers. A detection was rejected because the kinematic prediction placed the object at one position and the detector placed it somewhere inconsistent with that trajectory. A confidence-score threshold doesn't give you that. It gives you a number.
The specific problem is: your detector finds patterns. It doesn't know what's possible. The physics-constrained computer vision work we've been doing adds the perpendicular check that confidence scoring can't provide.
The question I keep encountering at the edges of production deployments is: at what point does calibrating the confidence threshold stop helping? The answer is usually when the source of false positives is a visual ambiguity that's genuine — the bald head under stadium lights, the surface particle that's spectrally indistinguishable from a shallow pit under the old node's imaging conditions. That's not a confidence problem. That's a physics problem.
If you've hit that ceiling, I'd like to understand the architecture.