I Kept Trying to Fix the Model. The Problem Was the Light.

The first number I put on the whiteboard for this build was 97 percent. The second was 14.

They describe the same model. In the stamping example from our research, a vision model validated at 97% accuracy in the lab goes onto a 200-ton progressive die press running 40 strokes a minute and starts false-rejecting 14% of good parts. Nothing inside the model changed. The inputs did: overhead bay-light glare that shifts with stroke angle, lubricant pooling differently on warm dies than on cold ones, the first 50 parts of every shift made before the press reaches thermal equilibrium. The physics of the line walked the images out of the distribution the model was validated on, and no model, at any accuracy, is trustworthy out there.

I spent the last stretch of this project building a demo to take that sentence seriously. It is called the Inspection Trust Gate, and you can watch it decide, part by part, at https://veriprajna.com/demos/edge-ai-manufacturing-inspection. It is not a better defect model. It is the runtime layer between the vision model and the PLC reject actuator, and its one job is to decide, for every single part, whether the model's verdict is safe to actuate.

What follows is the build story through the three moments that changed how I think about inspection AI: a drift event at a scripted 06:00, a benchmark number I refused to believe, and a rule I almost deleted.

The fix is not a better model

I resisted that sentence longer than I should have. When a detector misbehaves, every instinct I have as a builder says retrain it, upgrade the backbone, buy more labels. The research kept refusing to cooperate. Out-of-box AOI systems false-reject 5 to 15% of good parts, and well-tuned ones get under 2%, which our solution page calls "a calibration and data problem, not a model architecture problem". The same research reports that 84% of system integration projects fail or partially fail, and that in a typical inspection deployment the integration work is 60% of the project timeline while model training is 15%. The line I kept rereading: "The hardware is a purchase order."

So I did something that felt mildly heretical: I made the demo's defect model deliberately unremarkable. It is a kNN distance to known-good texture, a PatchCore-lite stand-in, and it scores an AUROC of 0.845 separating clean good parts from defective ones on the MVTec AD metal_nut held-out test split (real photographs of real manufactured parts, public labels). I am not hiding that number and I am not selling it. In production that slot holds your NVIDIA Metropolis pipeline, your Cognex system, your custom model, behind a fixed interface. The product is the layer around the engine, not the engine.

The layer starts with a question no accuracy metric answers: is this image inside the capture conditions the model was validated on? The demo's EnvelopeDetector computes a Mahalanobis distance in physical-signal space (exposure, contrast, dynamic range, a focus proxy, high-frequency detail, thermal colour cast, saturation, glare fraction), fitted on the 220 known-good training images and nothing else. When a part lands outside that validated envelope, the gate stops trusting the model's output entirely, however confident the model feels about it.

Even a perfect model is only valid on inputs inside its validated envelope.

That is the sentence the whole build hangs on. Drift is an input failure, not a model failure, and an input failure never shows up in your accuracy dashboard. It shows up in your scrap bin.

What happens at 06:00?

The moment I trust most in the whole demo is a timestamp. The app replays a deterministic scripted shift on the station "Line 3 - metal_nut press," streaming real MVTec AD metal_nut photographs through the full pipeline. Partway through, a marker crosses the screen: "SHIFT CHANGE 06:00 - cold dies, bay lights on." Then 12 good parts arrive corrupted with glare, defocus, and thermal cast. I want to be precise about what that is: the photographs are real, the drift is honest image corruption applied to them, and the app labels it as such. I did not have a stamping line to film, and pretending otherwise would poison the whole point.

What happens next is the reason the demo exists. The envelope monitor goes red. The gate reads each drifted part as outside the validated envelope and refuses to let the model's verdict touch the actuator. Verdict after verdict comes back HOLD, routed to the escalation queue for a human, while the panel beside it shows what a naive AOI with no envelope check would have done with the same image: REJECT.

The first time I watched the queue fill up, I clicked into a held part expecting to see a vague "anomaly detected" excuse. Instead the envelope monitor decomposed the breach signal by signal, because I had built it out of physical measurements rather than embeddings, and physical measurements can explain themselves. On one drifted part, brightness sits 7.6 sigma from the validated fit and carries 87.1% of the squared Mahalanobis distance. That is not a model having a feeling. That is an instrument reading.

By the end of the drift phase the scoreboard is stark. The naive baseline has auto-scrapped 12 good parts. The Trust Gate has auto-scrapped zero, holding all of them for review. The same images, the same stand-in defect model, the same thresholds on the defect score. The only difference is that one path checked the input before trusting the output.

The naive baseline and the gate saw the same parts and the same model. The only difference was permission to actuate.

About that dollar figure, because this is where demos usually start lying: the panel extrapolates the measured naive false-reject rate to a full shift (40 strokes a minute for 8 hours is 19,200 parts) at $2.42 per scrapped part. The $2.42 is a sourced order of magnitude, anchored to a published cookie-manufacturer case where an 8.7% scrap-waste reduction saved $94K a year and 38,800 kg of product. It is not a customer's number, and the counter is labeled a projection everywhere it appears. I measured the false rejects; I projected the dollars; the UI says which is which.

One more thing I checked before believing my own drift story: defective parts injected during the drift phase are still caught or escalated, never auto-passed. Across the whole held-out split that figure is 93 of 93. Holding good parts is worthless if bad parts slip through in the chaos.

Was my envelope check grading its own homework?

The benchmark number I distrusted most was my own best one. When I first ran the bench script, the envelope detector separated drifted from clean images essentially perfectly. My immediate reaction was not pride. It was suspicion, because I had built both sides of the exam: I wrote the glare, defocus, and thermal-cast corruptions, and I chose the physical signals the detector watches. Of course a glare detector catches glare. A reviewer with any teeth would call that circular, and they would be right.

So I held one drift family completely out. The detector was never tuned against underexposure, never saw it during development. Then I re-ran bench.py (most recently on 2026-07-17) on the MVTec AD metal_nut held-out test split: 22 good parts, 93 defective, train fit on the 220 good images only. On the held-out underexpose family, the envelope detector scored AUROC 1.000. That is the number that carries the thesis, precisely because it was earned on a failure mode I never engineered for. The script prints "THESIS HOLDS" only if the measured numbers actually support the claim; I wrote it that way so that the marketing could not drift away from the measurement.

The rest of the benchmark deserves its exact scope, so here it is without rounding in my favor. On drifted good parts, the naive no-envelope baseline false-rejects 95.5 to 100% per drift family (glare 100%, defocus 100%, thermal cast 100%, underexpose 95.5%, averaging 98.9%). The Trust Gate false-rejects 0.0% of them, holding every one for review. And the honest caveat: the corruptions are full strength, so the baseline's collapse is near-total by construction. The claim I will defend is the direction, that a cleanly validated model collapses once inputs leave its envelope, not the particular percentage. These are measurements on a research benchmark under synthetic drift. They are not open-world guarantees, and anyone who quotes them as production performance is misusing them, me included.

The rule I almost deleted

The hardest honesty call I made in this build was about a rule that barely works. Early on I added a geometric zone rule: localise the anomaly on a coarse 8 by 8 grid, and treat a defect inside the functional zone differently from a blemish out at the cosmetic edge. It sounds like real metrology. Then I measured it, and the measurements were humbling. The local-roughness proxy localises an anomaly on 33 of 93 held-out defects, about 35%. It changes the gate's outcome on exactly 1 of 93. And 0 of the 18 auto-rejects are backed by a real localised anomaly; when nothing localises, the centroid falls back to the grid centre, which reads as in-zone by default.

I sat with three options. Delete the rule and pretend I never tried. Keep it and let the UI imply precision metrology I do not have. Or keep it and make the interface confess. I chose the confession. When the demo's hero defect auto-rejects (part test-flip-264, a real gross structural defect from the MVTec flip class), the geometry drill-in states outright that nothing was localised on this part and the reject rests on texture confidence alone.

The rule does earn its keep exactly once, and I made the app prove it rather than stage it. At startup, the demo searches all 93 held-out defects for a part with a genuinely localised anomaly outside the functional zone on an otherwise confident part. In the shipped split that search finds test-flip-251, centroid at row 2, column 6, and the gate routes it to HOLD instead of firing the actuator. If the data changed and no part qualified, that beat would simply not appear. I even A/B tested the rule's sigma threshold with 5-fold cross-validation; the fitted value showed no improvement over the hand-set 2.5, so I kept 2.5 and recorded the negative result in the repo with "shipped": false. In a production engagement this rule is replaced by pixel-accurate, CAD-grounded metrology. In the demo it is an honest stand-in, and the UI says so on every part.

A trust layer that oversells itself is a contradiction in terms.

That line became a design rule. If the product's entire promise is knowing when not to trust a model, it cannot simultaneously bluff about its own weakest component.

Agents advise, code decides

The decision I refuse to delegate is the one that moves metal. The gate itself is plain deterministic code, outside any LLM and outside the vision model too. Its thresholds are fitted from the demo's own data, not hand-waved: auto-pass below 0.948 and auto-reject above 1.30 on the calibrated confidence scale, with everything in between, and everything out of envelope, going to HOLD. It runs against a hard 750 ms stroke-window budget, and in the shipped audit log the decisions land in tens of milliseconds: test-good-288 auto-passed in 37.7 ms, test-good-289 in 24.4 ms, and the reject actuator log for test-flip-264 reads "REJECT actuated in 25ms (budget 750ms)."

I should be plain about what actuates: nothing, yet. The EtherNet/IP path to an Allen-Bradley ControlLogix reject actuator is a simulator that logs exactly what it would have done, and the MES sink is a stub that writes the traceability line it would have written. Both are labeled as stubs in the app. They are shaped like the real adapters because the OT reality (mixed Siemens and Allen-Bradley plants, a reject window measured in milliseconds) is the actual product surface, but a demo that implied a live line would fail its own trust test.

There are agents in the system, and I bounded them on purpose. When parts pile up in the escalation queue, a Drift Triage pair goes to work: a diagnosis agent reads the ranked physical-signal deviations across the held parts and proposes a root-cause hypothesis with a recommended action, and a critic agent then checks that hypothesis against the numeric evidence, downgrading it to "manual investigation" if the cited signal is not actually the dominant deviation. They are built on Pydantic AI and provider-swappable, and with no API key configured the whole thing degrades to a deterministic templated triage, so the demo runs fully offline. What the agents cannot do, by construction, is touch the actuator. By the time they speak, the gate has already decided.

Agents advise, code decides.

Every one of those decisions leaves a receipt. Each part writes a JSONL lineage record: part id, station, model id metalnut-defect-knn version v7, dataset hash ae95b5b533c8, defect confidence, OOD score, the physical signals, which gate rules fired, latency against the 750 ms budget, the actuation and MES log lines, what the naive baseline would have done, and the risk tag high-risk:quality-gate (EU AI Act Annex III, eff. 2026-08-02). One click exports the shift as inspection_audit.jsonl.

The regulatory clock matters here. The EU AI Act's high-risk obligations become fully applicable on August 2, 2026, safety-critical quality decisions sit in Annex III, and the maximum fines reach €35M or 7% of global turnover for the most serious prohibited-practice violations. I want to be careful with my words, because this is exactly where careful words matter: the demo is not EU AI Act certified, and no demo can be. What it shows is EU-AI-Act-ready lineage, a per-decision record designed to be filable evidence in a high-risk conformity file, generated at line speed instead of reconstructed after an incident.

What survives the next model upgrade?

The question I kept asking myself while building this was brutal for a demo maker: if the customer's next defect model is dramatically better than my stand-in, does any of this still matter? I now think that is exactly backwards. Deloitte predicts agentic AI adoption in manufacturing rising from 6% to 24% in 2026 (Deloitte), which means more models and more autonomy arriving at more actuators. Every one of those models will have a validated envelope, and the physics of a press line (glare, cold dies, thermal equilibrium) will keep walking inputs out of it. A perfect model changes nothing about that, because the failure the gate prevents is an input failure, and the audit obligation it serves is a legal one, not a modeling one. Drift-gating, provenance, and governed actuation hold at any model accuracy. That is the property that convinced me this layer, and not another model, was the thing worth building; the scripted shift at https://veriprajna.com/demos/edge-ai-manufacturing-inspection is my attempt to let you watch it earn that claim part by part.

And if you would rather see it than read me describe it, here is the founder cut, end to end.

So the question I would ask about any inspection model on your line, including a 97% one, is not "how accurate is it?" It is: for the part that just crossed the camera, do you know whether that image was inside the envelope the model was validated on? If you cannot answer per part, in milliseconds, with a record you could hand to an auditor, then I do not think you have an accuracy problem. I think you have an envelope problem, and I would genuinely like to know which one your line has.