
A patient with a life-threatening blood disorder was sent home early — because an algorithm matched her to the "average" recovery time for her diagnosis.
Her family paid $16,768 out of pocket to keep her from being discharged before she was stable.
That case anchors Lokken v. UnitedHealth, and it should worry every Medicare Advantage plan. The AI tool, nH Predict, leaned on diagnosis-based recovery timelines and all but ignored individual clinical signals — blood oxygen, caregiver availability, comorbidities.
Then managers tightened the allowed variance from the algorithm's projection from 3% to 1%, and clinicians who overrode it faced discipline. The "human in the loop" became a rubber stamp — the same breach-of-contract exposure as no human at all.
Roughly 90% of the AI denials in that litigation were reversed on appeal. That's not a tuning problem. It's evidence the architecture was wrong.
And it's now legally exposed: a March 2026 discovery order opened the AI's development documents, training data, and validation reports — every plan should assume its own AI files are discoverable. If you can't reconstruct why your model denied a specific person, you can't defend it.
What we keep seeing: monitoring a flawed model better doesn't fix it. The architecture itself has to be defensible by design.
If you run utilization management or prior auth on AI — could you reconstruct, today, exactly why it denied a specific claim last quarter?
#AIGovernance #HealthcareAI