Harvey Murphy is a 61-year-old grandfather from Sacramento. In January 2022, he was arrested for a robbery at a Sunglass Hut in Houston — a city he wasn't even in. The sole evidence? A facial recognition system said he was the guy. He spent ten days in jail, where he was sexually assaulted and beaten, before anyone bothered to check that he'd been in California the entire time. He's now suing Macy's and Sunglass Hut for $10 million.
When I first read the details of Murphy's case, I kept coming back to one question: at what point did a machine's guess become treated as fact? That question — the gap between what AI actually produces (a probability) and how organizations treat it (a certainty) — is the central problem my team and I have been working on for the past two years. And it's a problem that's about to get much worse before it gets better.
We dug into Murphy's case alongside the FTC's unprecedented five-year ban on Rite Aid's facial recognition program, and what we found wasn't just a story about bad technology. It was a story about a broken architecture that most enterprises are still using today.
The FTC Banned Rite Aid for Five Years. Here's What Actually Went Wrong.
Between 2012 and 2020, Rite Aid rolled out AI-powered facial recognition across hundreds of stores to catch shoplifters. In December 2023, the FTC shut the whole thing down — not for five months, but for five years. They also ordered Rite Aid to delete every biometric image they'd collected and destroy the AI models built from that data.
The technical failures were staggering. Rite Aid bought its facial recognition from two third-party vendors whose contracts explicitly disclaimed any guarantee that the results would be accurate. The company never tested the technology for accuracy. They fed the system grainy CCTV screenshots and even cell phone photos — images so degraded that matching against them is essentially guessing.
The system generated thousands of false-positive matches, disproportionately targeting women and people of color.
Store employees, acting on these automated alerts, followed innocent customers, searched them, and publicly accused them of theft. In some cases, the system flagged the same person at dozens of stores across the country simultaneously. The FTC found that stores in predominantly Black and Asian communities experienced significantly higher rates of false alerts than stores in predominantly White communities.
I've talked to executives who hear this story and think, "Well, we'd never deploy something that sloppy." But when I ask them whether they've actually tested the accuracy of the AI tools they're already using — tested them, not just trusted the vendor's marketing deck — the room usually goes quiet.
What Happened to Harvey Murphy
The Murphy case makes the stakes visceral. A robbery happens at a Houston Sunglass Hut. EssilorLuxottica, Sunglass Hut's parent company, works with Macy's to run low-resolution surveillance footage through Macy's facial recognition system. The system matches the footage to Murphy — likely against a decades-old booking photo from a minor, non-violent offense.
Nobody questioned the match. The lawsuit alleges that Sunglass Hut and Macy's presented the automated result to police as a verified identification, and the investigation essentially stopped there. Murphy's alibi — that he was 1,700 miles away in Sacramento — wasn't checked until after he'd already been jailed and brutalized.
There's a term in biometric research for what happened here: the age-gap problem. Matching a current surveillance image against a photo taken years or decades earlier can produce false-positive rates as high as 90%. The system wasn't designed to account for this. Nobody asked it to.
When a machine's output is treated with more authority than a human being's alibi, something has gone fundamentally wrong.
I remember describing this case to my team, and one of our engineers said something that stuck with me: "The technology didn't fail. The architecture around it did." She was right. The facial recognition algorithm did exactly what it was built to do — find the closest match in a database. The catastrophe was that nobody built the system to say, "I'm not sure."
The Real Problem: Your AI Can't Say "I Don't Know"
This is the core issue I see across enterprise AI deployments, not just in facial recognition but everywhere. Most AI systems produce a single answer with a confidence score, and organizations treat that score as gospel. But a confidence score of 0.85 doesn't mean the system is 85% sure it's right. It often just means the system found this to be the least-bad option among the choices it was given.
My team works on something called uncertainty quantification — essentially teaching AI systems to measure and communicate how much they don't know. There are two kinds of uncertainty that matter here. The first comes from noisy or incomplete data: a blurry camera, bad lighting, a photo taken from a weird angle. No amount of better training fixes this — the information simply isn't there. The second comes from gaps in the model's knowledge: it hasn't seen enough examples of a particular demographic, or it's never encountered this specific scenario before. This kind can be fixed with better data.
Most commercial AI systems don't distinguish between these two. They just give you a number. We explored this problem in depth in our interactive analysis, walking through how techniques like conformal prediction can provide mathematical guarantees that a prediction falls within reliable bounds — something a simple confidence score can never do.
The practical difference is enormous. Instead of a system saying "Match: 0.85," an uncertainty-aware system says, "Match: 0.85, but the uncertainty range is wide because the input image quality is poor — flag this for human review." That single architectural change could have prevented Harvey Murphy's arrest.
Why Most Enterprise AI Is Built on a House of Cards
Here's something that surprised me when we started auditing how companies actually deploy AI: the vast majority are running what the industry calls "wrappers." A wrapper is a branded interface sitting on top of someone else's AI model, sending data to a third-party API and returning whatever comes back. The company using it has no visibility into how the model works, no ability to audit its decisions, and no control when the vendor changes something.
Rite Aid was running a wrapper. Macy's was running a wrapper. They bought technology, plugged it in, and assumed it worked.
The risks compound fast:
You can't audit what you can't see. When a regulator or a plaintiff's attorney asks how your system reached a decision, "we sent it to a vendor's API" is not an answer.
You inherit every bias in the vendor's training data without knowing what that data contains.
A single vendor outage takes you down. Every company using that wrapper goes dark simultaneously.
Sensitive data may be leaking into the vendor's training pipeline, violating privacy laws you didn't know applied.
The alternative — what we build and what I advocate for — is a multi-agent architecture. Instead of one black-box model doing everything, you have specialized agents working in sequence: one validates input quality, another performs the match, a third quantifies uncertainty, and a fourth decides whether the result is reliable enough to act on or needs human review. Each agent has a defined job and can be independently audited, updated, or replaced.
It's the difference between asking one person to be the detective, the judge, and the jury, versus building an actual system of checks and balances.
The "Closed-Set" Trap That Almost Nobody Talks About
There's a technical distinction that has enormous real-world consequences, and it's one I almost never see discussed outside academic papers. Most commercial facial recognition systems are built for closed-set problems — situations where the person being scanned is definitely in the database. Think of unlocking your phone with your face. The system knows you're in there; it just needs to confirm it's you.
Retail security is the opposite. It's an open-set problem. The vast majority of people walking into a store are not in any criminal database. But a closed-set system doesn't know how to say "this person isn't in my database." It will always find the closest match, even when the correct answer is "no match exists."
This is exactly what happened at Rite Aid. The system tried to match every face it saw against its watchlist, generating thousands of false positives because it was architecturally incapable of saying "I don't recognize this person."
A system designed to always find a match will always find a match — even when there isn't one.
For the full technical methodology behind open-set recognition and the specific loss functions that address this, see our detailed research. The short version: solving this requires training models not just to identify matches but to accurately reject non-matches — a fundamentally different engineering challenge that most off-the-shelf systems don't address.
The Regulation Wave Is Already Here
Two regulatory frameworks are reshaping what's legally required of companies deploying AI, and most organizations aren't ready for either.
The NIST AI Risk Management Framework provides a voluntary but increasingly influential structure built around four functions: govern (build accountability structures), map (understand your AI's context and risks), measure (quantify bias and performance), and manage (address what you find). The FTC's action against Rite Aid was essentially an enforcement of these principles — the company failed to measure or manage the risks of its facial recognition system, and the FTC treated that as an unfair business practice.
The EU AI Act goes further. It's binding law, not a voluntary framework. Biometric identification systems are automatically classified as high-risk, which triggers mandatory obligations: conformity assessments, detailed technical documentation, effective human oversight, and representative training datasets. Certain practices — like scraping facial images from the internet — are banned outright.
Even if you're a US-based company with no European operations, the EU AI Act matters. It's setting the global standard that domestic regulations will follow. Companies aligning with NIST today are better positioned for whatever comes next.
What "Human Oversight" Actually Looks Like in Practice
I hear a lot of companies claim they have "human-in-the-loop" processes. When I dig into what that means, it's usually an employee who gets an alert and clicks "approve" because they've been told the system is reliable. That's not oversight. That's rubber-stamping.
Real human oversight requires three things. First, confidence thresholds that route decisions appropriately: below 70% confidence, auto-reject; between 70% and 95%, mandatory human review with access to the original source data; above 95% and only for low-consequence decisions, the system can act autonomously. Second, interfaces that actually enable judgment — showing the reviewer the original surveillance image next to the database image so they can spot what the AI might have missed. Third, audit trails that log every decision, human and automated, for legal defense and continuous improvement.
At Rite Aid, employees were instructed to confront customers based on automated alerts without any training on the possibility of false positives. At Macy's, a machine output was handed to police as verified fact. Neither had meaningful human oversight. Both are now paying the price.
What About the Cost of Building This Right?
I won't pretend that building uncertainty-aware, multi-agent AI systems is cheaper upfront than buying an off-the-shelf wrapper. It's not. The question is what you're comparing it to.
Rite Aid lost its entire biometric capability for five years and was ordered to destroy every model it had built. Macy's is facing a $10 million lawsuit and the kind of headlines that no PR team can manage away. The FTC introduced a concept called model disgorgement in the Rite Aid settlement — requiring the company to delete not just the data but every AI model trained on that data. Years of investment, gone.
The cost of building robust AI is real. The cost of building brittle AI is catastrophic.
Does This Apply Beyond Facial Recognition?
Everything I've described — the wrapper dependency, the missing uncertainty quantification, the lack of human oversight, the closed-set trap — applies to any high-stakes AI deployment. Loan approvals. Medical diagnoses. Hiring algorithms. Insurance claims. Anywhere an AI system's output directly affects a human being's life, the same architectural failures create the same risks.
The specific cases here involve facial recognition because that's where the lawsuits and regulatory actions have landed first. But the pattern is universal: organizations deploying AI they can't audit, can't explain, and can't override are building on a foundation that will eventually crack.
What I'd Tell a Board of Directors Tomorrow
If I had thirty minutes with your board, I'd say five things:
Audit your AI dependencies. Identify every system where you're relying on a third-party black box and can't explain how it reaches its conclusions.
Demand uncertainty scores, not just confidence scores. Any high-stakes AI output should come with a quantified measure of how much the system doesn't know.
Require NIST-validated performance data from every biometric vendor, broken down by demographic group. If they can't provide it, that tells you everything.
Build real human oversight — not rubber stamps. Define confidence thresholds, build review interfaces, log every decision.
Prepare for the EU AI Act now, regardless of where you operate. The regulatory supercycle is here, and the cost of catching up later is orders of magnitude higher than the cost of getting ahead.
Harvey Murphy spent ten days in jail because a machine was wrong and nobody questioned it. The technology to prevent that outcome exists today. The question is whether organizations will choose to use it before the next lawsuit forces them to.
I'd genuinely like to hear from anyone navigating these decisions — what does your AI oversight actually look like in practice, and where are the gaps you're most worried about?