A pulse oximeter — that little clip on your finger — overestimates blood oxygen levels in Black patients so consistently that they're nearly three times more likely to have dangerously low oxygen that goes completely undetected. Now feed that bad data into an AI triage system. The algorithm inherits the bias, automates it at scale, and nobody gets an alert. That's not a hypothetical. It's happening right now in hospitals across the country, and it's killing people.
I've spent the last two years studying how clinical AI systems fail — not in the abstract, but in the specific, measurable ways they provide a lower standard of care to Black and Hispanic patients. What I've found is that the industry's rush to slap a ChatGPT wrapper on every clinical workflow isn't just unhelpful. It's actively dangerous.
The Device Everyone Trusts Is Lying
Most people have never questioned a pulse oximeter reading. Why would you? It's a simple device. Light goes through your finger, measures oxygen in your blood, gives a number. But the physics contains a flaw that's been documented since the 1990s.
Melanin absorbs light across the same wavelengths these devices use. When oximeters are calibrated primarily on lighter-skinned populations — which they overwhelmingly are — the extra absorption in darker skin gets misread as more oxygenated blood. The device says 93%. The real number is 88%. One reading is fine. The other is a medical emergency.
A 2025 Vanderbilt pediatric study found that common pulse oximeters missed dangerous oxygen levels in 7% of children with the darkest skin tones while missing zero cases in those with the lightest tones. The FDA, until recently, required testing on only ten subjects to approve these devices. Ten.
When your AI's most basic input is wrong for an entire demographic, no amount of algorithmic sophistication downstream can fix it.
This is what I started calling the "garbage in, garbage out" problem — except the garbage looks perfectly clean in the database. The EHR says SpO₂ 93%. It's a number. It looks like a fact. But for millions of patients, it's a lie baked into the hardware.
The Sepsis Model That Missed 67% of Cases
If biased sensors are the foundation problem, the Epic Sepsis Model is the cautionary tale of what happens when you build on that foundation without looking down.
Epic's sepsis prediction tool is integrated into hundreds of hospitals' electronic health records. It was marketed as an early warning system — catch sepsis before clinicians do, save lives. The developer reported strong performance internally, with accuracy metrics between 0.76 and 0.83.
Then independent researchers at Michigan Medicine actually tested it on their own patients. Performance dropped to 0.63 — barely better than a coin flip. The model missed 67% of actual sepsis cases. When it did fire an alert, 88% were false alarms. And in only 6% of true cases did the AI flag the patient before a clinician had already recognized the problem.
I remember reading the Michigan validation study for the first time and just sitting with it. A tool deployed in hundreds of hospitals. Lives depending on it. And it was essentially noise.
The racial dimension makes it worse. Black and Hispanic patients experience nearly double the sepsis incidence of white patients. But the model was trained on data from three US health systems between 2013 and 2015 — a specific slice of patients, clinical practices, and documentation habits. When clinicians historically take longer to order blood cultures for Black patients, the training data learns that delay as "normal." The AI becomes blind to sepsis presentations in the very populations most at risk. We explored this failure pattern in depth in our interactive analysis.
The AI misses the patient because the historical data was biased. The clinician misses the patient because they trusted an AI that never fired an alert. That's not a bug. It's a lethal feedback loop.
Black Mothers Are Dying. AI Is Making It Worse.
The numbers here stopped me cold. Black women face a pregnancy-related mortality rate of 50.3 per 100,000 live births — 3.5 times higher than white women. This disparity persists even when you control for education and income. A Black woman with a graduate degree faces higher maternal mortality risk than a white woman without a high school diploma.
California, despite being a leader in maternal health data, found that automated early warning systems missed 40% of severe morbidity cases in Black patients. Forty percent. These are hemorrhages, preeclampsia, sepsis — conditions that occur 100 times more frequently than maternal death and represent the last chance to intervene before someone dies.
One reason the AI fails: chronic stress from systemic racism causes measurable physiological changes — higher baseline blood pressures, altered cardiovascular responses. Researchers call it the "weathering" effect. An algorithm trained on population averages interprets these altered baselines as "normal for this patient" rather than recognizing them as a compounding risk factor.
And when complications do occur, Black women are 1.79 times more likely to die from them than white women. The clinical term is "failure to rescue" — the inability to save a patient once something goes wrong. When AI misses the warning signs, that rescue window shrinks to nothing.
McKinsey estimates that closing the Black maternal health gap could add $24.4 billion to US GDP and save $385 million in annual preventable healthcare costs. This isn't just a moral argument. It's an economic one.
Why ChatGPT Wrappers Can't Fix This
Here's where I get blunt, and where I know I'll get pushback from the dozens of startups building "clinical AI" by wrapping a GPT API in a healthcare UI.
A large language model is a statistical engine trained on word probabilities. It doesn't understand pathophysiology. It can't interpret a waveform. It has no concept of whether a pulse oximeter reading is trustworthy for the patient it's looking at. When researchers tested LLMs on medication dosing for patients with kidney dysfunction — a scenario requiring precise, patient-specific reasoning — accuracy was 16.7%. You'd do better flipping through a textbook.
These models hallucinate. They can't access real-time clinical data. They can't explain their reasoning in a way a clinician can verify. And they inherit every bias present in the internet-scale text they were trained on.
I'm not saying LLMs have no role in healthcare. They're fine for drafting discharge summaries or summarizing chart notes. But when we're talking about triage decisions — who gets the ICU bed, who gets supplemental oxygen, which pregnant woman gets the emergency C-section — a chatbot wrapper is the wrong tool. It's not even the right category of tool.
What "Deep AI" Actually Means
When my team talks about deep AI for clinical decision support, we mean something specific. Not a marketing term. An architectural commitment.
It starts with not trusting any single data source. If a patient's pulse oximeter says their oxygen is fine but their heart rate is climbing and their lactate levels are rising, those signals conflict. A deep AI system flags that discrepancy and tells the clinician: order an arterial blood gas, the gold-standard test. Don't trust the finger clip alone.
It means training models on labels that have been reviewed by clinical experts — not billing codes, which carry decades of documentation bias. If "sepsis" in your training data is defined by when someone happened to get a billing code rather than when they actually became septic, your model learns the wrong thing.
And it means building fairness into the math itself, not bolting it on after the fact. During training, you can add what's called a fairness penalty — a mathematical constraint that forces the model to perform equally well across demographic groups. You can also optimize for the worst-performing group specifically, rather than optimizing for the population average, which always favors the majority.
Optimizing for the average is how you build a system that works great for most patients and fails the ones who need it most.
For the full technical methodology behind these approaches — including the specific fairness-aware loss functions and validation frameworks — see our detailed research.
What Healthcare Leaders Should Demand Right Now
If you're a hospital executive or a health system CTO evaluating clinical AI vendors, here's what I'd push for immediately.
Subgroup performance data, not averages. Any vendor who gives you a single accuracy number for their model is hiding something. Demand sensitivity, specificity, and false alarm rates broken down by race, age, and sex. If they can't provide it, they haven't measured it. If they haven't measured it, assume the worst.
External validation, not internal benchmarks. The Epic Sepsis Model looked great on the developer's data and collapsed in the real world. Insist on independent validation — ideally on your own patient population before you go live.
A clear explanation of what the model does when it's wrong. Every AI system will make mistakes. The question is whether those mistakes fall disproportionately on vulnerable populations, and whether clinicians have enough information to catch and correct them. A black box that says "high risk" or "low risk" with no reasoning chain is not clinical decision support. It's a liability.
Does Fixing Bias Mean Sacrificing Accuracy?
This is the question I get most often from technical leaders, and it's a fair one. The short answer: slightly, for the overall average. Significantly, for the patients who matter most.
Research in algorithmic fairness has shown that optimizing for the worst-performing demographic group may reduce overall accuracy by a small margin — but it dramatically improves outcomes for underrepresented populations who were previously being misclassified. In clinical terms, that trade-off means your model catches a few more false positives across the board while catching far fewer false negatives in the populations most likely to die from a missed diagnosis.
I'll take that trade every time. And I think any healthcare leader who understands the liability exposure — not to mention the human cost — would too.
This Is a Standards Problem, Not a Technology Problem
We have the math. We have the architectural patterns. We know how to build clinical AI that doesn't systematically fail Black and Hispanic patients. The technology exists.
What's missing is the will to demand it. The FDA is only now updating pulse oximeter testing requirements. Most hospital procurement processes don't ask for subgroup performance data. And the market rewards speed-to-deployment over rigor, which means wrapper products that can be stood up in weeks keep winning contracts over systems that take months to validate properly.
I started Veriprajna because I believe the next generation of clinical AI has to be built differently — from the sensor layer up, with fairness as a mathematical constraint rather than a marketing slide. Not because it's the easy path. Because the alternative is automating discrimination at scale and calling it innovation.
Three times the maternal mortality. Three times the occult hypoxemia. The number three keeps showing up, and it's not a coincidence. It's a system working exactly as it was built.
If you're working on clinical AI adoption — whether you're building it, buying it, or regulating it — I'd genuinely like to hear what standards you're holding vendors to. Because the gap between what's possible and what's being deployed is where patients are dying.