- Rite Aid deployed facial recognition for 8 years without ever testing if it worked accurately.
The FTC banned them for 5 years. A grandfather spent 10 days in jail for a robbery 1,500 miles away.
This is what "brittle AI" looks like at scale. 🧵 - Here's what went wrong at Rite Aid:
Their vendors' contracts explicitly disclaimed any warranty on accuracy.
Rite Aid assumed full liability for a system it didn't build, didn't test, and didn't understand.
The "wrapper" trap in its purest form. - They fed low-quality CCTV stills and cell phone photos into the system.
In biometric engineering, degraded input doesn't just reduce accuracy — it exponentially increases false positives.
Garbage in, civil rights violations out.
The result: thousands of false matches. - Disproportionately targeting women and people of color. Employees confronted and searched innocent customers based on automated alerts they were never trained to question.
The AI wasn't an assistant. It was the adjudicator.
Then there's Harvey Murphy. - Macy's facial recognition matched him to a robbery suspect using grainy surveillance footage against a decades-old booking photo.
He was in Sacramento. The robbery was in Houston.
He spent 10 days in jail and was brutally assaulted. - The core problem: every AI output is a probabilistic estimate.
But these systems presented guesses as facts. No confidence scores. No uncertainty quantification. No human review threshold.
A 0.85 match score means nothing if the uncertainty distribution is wide. - Most commercial facial recognition is built for closed-set problems — where the person IS in the database.
Retail security is open-set. Most people walking in aren't criminals.
A closed-set model will always find a "best match." Even when there isn't one. - This is why we build multi-agent systems, not wrappers.
One agent validates image quality. Another performs the match. A third quantifies uncertainty. A fourth flags for human review.
No single model decides. No single failure cascades. - The FTC now requires Rite Aid to destroy all models trained on non-compliant biometric data.
This is "model disgorgement" — forced unlearning.
If your architecture can't surgically remove specific data influence, you're one audit away from losing everything you built. - The regulatory supercycle is here.
The EU AI Act classifies biometric ID as high-risk. NIST AI RMF demands uncertainty measurement. The FTC is enforcing disgorgement.
"We didn't know" is no longer a defense. "We didn't build it" isn't either. - Honest question for everyone building with AI right now: should any system making decisions about personal liberty be allowed to operate without publishing its false positive rate by demographic?
#AI #ResponsibleAI - We wrote the full technical breakdown — covering uncertainty quantification, open-set vs closed-set recognition, HITL frameworks, and what boards need to do now.
https://veriprajna.com/whitepapers/algorithmic-integrity-resilient-ai-era-biometric-liability