An algorithm denied care to a woman recovering from a life-threatening blood disorder. Her family paid over $16,000 out of pocket to keep her in the nursing facility she still needed.
Not because a doctor said she was ready to leave. Because a predictive model said so.
That's the story of Carol Clemens, and she's far from alone.
When UnitedHealth's nH Predict algorithm was put in charge of deciding how long Medicare patients could stay in post-acute care, denial rates for skilled nursing facilities jumped to nine times their baseline level.
Here's the part that still stops us cold → when patients managed to appeal those denials, the decisions were reversed 90% of the time.
Nine out of ten denials were wrong.
But only 0.2% of patients ever appealed. Most were too elderly, too sick, or too overwhelmed to fight back.
The system wasn't accurate. It was profitable because it was hard to challenge.
Our team wrote an in-depth analysis of what went wrong here, and more importantly, what "getting it right" actually looks like → moving from black-box predictions to AI that can explain its reasoning, flag its own uncertainty, and keep a human genuinely in the loop.
Not a human who rubber-stamps the machine. A human who's empowered to override it.
We'd love to hear from you → should companies face liability specifically for deploying AI that overrides clinical judgment, or is this really a regulation problem that government needs to solve first?
#AlgorithmicIntegrity #HealthcareAI #AIGovernance