A pulse oximeter says a patient's oxygen level is 93% — safely normal. But her actual oxygen saturation is 88%, a level that demands immediate intervention. The device got it wrong because it was never properly calibrated for darker skin. And the AI system watching her vitals? It trusted the device. No alert fired. No oxygen was administered.
This isn't a hypothetical. Black patients are nearly three times more likely to experience this exact scenario — a condition called occult hypoxemia, where the device reads normal while the patient silently deteriorates. It's been documented since the 1990s. And now, as hospitals layer AI clinical decision support systems on top of these flawed measurements, the bias isn't just persisting. It's scaling.
Our team spent months analyzing the research — from pulse oximetry physics to sepsis prediction failures to the Black maternal mortality crisis — and what we found is a pattern that should alarm every healthcare executive in the country. The problem isn't that AI is broken. The problem is that AI is working exactly as designed — and the design encodes decades of inequity.
We published our full interactive analysis here.
The Device Your AI Trusts Is Lying
Most people assume medical devices produce objective measurements. Pulse oximeters — those small clips placed on a patient's finger — feel like they should be as reliable as a thermometer. They're not.
These devices work by shining red and infrared light through your finger and measuring how much gets absorbed by oxygenated versus deoxygenated blood. Simple enough. But melanin also absorbs light across those same wavelengths. When the device encounters darker skin, it misreads the extra absorption as more oxygenated blood than is actually present.
A 2024 Vanderbilt pediatric study found that common pulse oximeters missed dangerously low oxygen in 7% of children with the darkest skin tones — while missing zero cases in children with the lightest tones.
The numbers for adults are worse. Published research in the New England Journal of Medicine and British Medical Journal shows that false negative rates for darker-skinned patients range from 7.6% to 62.2%, compared to 1.2% to 26.9% for lighter-skinned patients. That's not a rounding error. That's a fundamentally different quality of measurement depending on the color of your skin.
Until recently, the FDA required testing on only ten subjects before clearing a pulse oximeter for market use. Ten people. For a device that feeds data into the AI systems making triage decisions for millions.
How a Broken Sensor Breaks an Entire AI System
When an AI triage system uses oxygen saturation as a core input — and nearly all of them do — it inherits every flaw in that measurement.
Think of it like building a house on a cracked foundation. The architecture above might be brilliant, but the structure is compromised from the ground up. If an algorithm triggers a high-priority alert when oxygen drops below 92%, it will systematically fail to flag Black patients whose true oxygen is at 88% but whose pulse oximeter cheerfully reports 93%.
The downstream consequences cascade fast: delayed supplemental oxygen, increased risk of organ failure, longer ICU stays, higher mortality. The AI didn't make a "mistake" in the traditional sense — it faithfully processed bad data and reached a dangerous conclusion.
This is why we argue that "clean EHR data" is an illusion. Any serious clinical AI system must either apply demographic-specific calibration offsets to oximetry readings or combine multiple signals — heart rate variability, respiratory rate, lactate trends — to triangulate a patient's true condition rather than trusting any single sensor.
The Sepsis Model That Missed 67% of Cases
If the pulse oximeter story illustrates hardware bias, the Epic Sepsis Model illustrates what happens when algorithmic bias meets real-world deployment.
The Epic Sepsis Model (ESM) is integrated into the electronic health records of hundreds of American hospitals. It was marketed as a proactive early warning system — catch sepsis before clinicians recognize it. The developer's internal validation claimed strong performance, with an accuracy metric (AUC) between 0.76 and 0.83.
Then independent researchers at Michigan Medicine tested it on their own patient population. The model's performance dropped to an AUC of 0.63 — barely better than a coin flip. It missed 67% of actual sepsis cases. When it did fire an alert, 88% were false alarms. And it only beat the clinician to a diagnosis 6% of the time.
The model wasn't just inaccurate. It was providing a different standard of care depending on who the patient was.
Black and Hispanic patients experience nearly double the sepsis incidence of white patients and often present at younger ages. Yet the ESM exhibited poor calibration across these groups. A key reason: the model was trained on clinical definitions and billing codes that were themselves shaped by biased human judgment.
If clinicians historically took longer to order blood cultures for Black patients, or were slower to recognize sepsis symptoms in those patients, the training data reflects that delay. The AI learns to associate "sepsis" with the data patterns of white patients. It becomes, in effect, blind to how the disease presents in Black patients. The clinician then relies on an AI that doesn't fire an alert, reinforcing the original bias. A lethal feedback loop.
Black Maternal Mortality: Where All the Failures Converge
Nowhere do these compounding failures hit harder than in maternal health.
Black women in the United States face a pregnancy-related mortality rate of 50.3 per 100,000 live births — 3.5 times the rate for white women. This disparity persists even when controlling for education and income. A Black woman with a college degree faces higher maternal mortality risk than a white woman without one. The problem is structural, not individual.
California, widely considered a leader in maternal health quality improvement, offers a sobering case study. The California Maternal Data Center found that automated early warning systems missed 40% of severe morbidity cases in Black patients. Severe maternal morbidity — life-threatening complications like hemorrhage, preeclampsia, and sepsis — occurs 100 times more frequently than maternal death. These are the cases where early AI detection could save lives. And the systems are failing precisely where the stakes are highest.
One contributing factor is what researchers call the "weathering" effect: the physiological toll of chronic exposure to systemic racism. Black women may present with higher baseline blood pressures and altered cardiovascular responses — patterns that an AI calibrated on predominantly white populations may interpret as "normal" for that individual, missing the danger entirely.
Black women are 1.79 times more likely to die once a severe complication has occurred — not because the complications are different, but because the system fails to rescue them in time.
McKinsey estimates that closing the Black maternal health gap could save $385 million in annual preventable healthcare costs and add $24.4 billion to US GDP through restored healthy life years. This isn't just a clinical crisis. It's an economic one.
Why Chatbot AI Won't Solve This
The explosion of generative AI has flooded healthcare with a particular type of product: thin software layers built on top of general-purpose language models like GPT or Gemini. We call these "wrapper" applications. They're useful for drafting discharge summaries or answering basic patient questions. They are dangerously inadequate for clinical decision support.
A large language model predicts the next most likely word in a sequence. It doesn't understand pathophysiology. It can't reason about the interaction between a biased sensor reading, a patient's chronic stress physiology, and an atypical sepsis presentation. When researchers tested leading LLMs on medication dosing for patients with kidney dysfunction — a scenario requiring patient-specific clinical reasoning — accuracy was just 16.7%.
These models also hallucinate. They present fabricated information with the same confidence as verified facts. In a clinical context, that means potentially inserting false data into a patient's record. They can't explain their reasoning in a way a clinician can verify. And they lack access to real-time clinical data — they're working from static training sets that may be months or years out of date.
For the full technical methodology behind our approach, see our detailed research.
What Actually Works: Building AI That Accounts for Inequity
The alternative to wrapper AI isn't no AI. It's AI built with the explicit goal of equitable performance across every patient population. Our research points to several principles that separate systems that help from systems that harm.
Start with the data, not the algorithm. Before a model ever trains, the dataset needs scrutiny for hidden stratification. If portable X-rays — typically used for sicker, bedbound patients — are disproportionately represented for one demographic group, the model may learn to associate the X-ray equipment with disease rather than actual pathology. Techniques like adversarial debiasing can force a model to learn features tied to clinical reality rather than demographic proxies.
Don't trust any single sensor. When a patient's heart rate and lactate levels are climbing but their oxygen saturation reads stable, that discrepancy is a signal, not noise. Systems that fuse multiple data streams — waveform data, lab results, nursing notes — can catch what a single measurement misses.
Optimize for the most vulnerable, not the average. Standard AI training minimizes average error across the whole dataset, which naturally favors the majority group. Worst-group optimization flips this: it minimizes the maximum error across all demographic subgroups. Research in automated depression detection showed this approach significantly improved outcomes for Hispanic participants — a group consistently misclassified by standard models — with only a modest decrease in overall accuracy.
Validate locally, not just globally. A model that performs well at the institution where it was built may fail at yours. Every deployment should begin with a retrospective audit against the local patient population, measuring how different that population is from the training cohort and recalibrating accordingly. The Epic Sepsis Model's collapse at Michigan Medicine is a textbook case of what happens when this step is skipped.
What Should Healthcare Leaders Do Differently?
If you're a healthcare executive evaluating AI vendors — or already running AI systems in production — three questions should be non-negotiable.
First, demand subgroup performance data. A vendor claiming "95% accuracy" without showing breakdowns by race, age, and sex is hiding something. Ask for sensitivity, specificity, and false alarm rates for every demographic group in your patient population. Ask for calibration curves — proof that when the model says "90% probability of sepsis," 9 out of 10 such patients actually have the condition.
Second, require independent external validation. Vendor-produced whitepapers are marketing materials. Peer-reviewed studies conducted at institutions other than where the model was built are evidence. There is a meaningful difference.
Third, treat AI as a collaborator, not a replacement. The most effective implementations use AI to surface patterns and flag risks while keeping the diagnostic and therapeutic decision with a human clinician — one who has been trained to recognize when the algorithm might be wrong and why.
The question isn't whether to adopt AI in healthcare. It's whether you're willing to adopt AI that works equally well for every patient who walks through your door.
The mortality data is unambiguous. The sensor biases are well-documented. The algorithmic failures have been externally validated and published in top medical journals. What remains is a choice: continue deploying convenient tools that encode historical inequity, or invest in systems engineered from the ground up to correct for it.
We believe the lives at stake — thousands of mothers, sepsis patients, and children every year — make that choice straightforward. The harder part is doing the work.
If your organization is grappling with how to evaluate or govern clinical AI systems for equity, we'd welcome the conversation.