- A reviewing physician misses two-thirds of the harmful errors in AI-drafted patient messages.
And 90% of them say they trust the AI anyway.
The "a doctor reviewed it" safety net most health systems lean on? Often a rubber stamp. š§µ - The numbers are from Lancet Digital Health (2024):
ā¢ 7.1% of AI-drafted patient portal messages posed a severe harm risk
ā¢ 0.6% posed a death risk
ā¢ physicians missed 66.6% of the erroneous drafts
ā¢ 35-45% were sent entirely unedited - This is automation bias. When an AI draft is fluent and empathetic, the quality of the prose substitutes for clinical verification.
The physician scans it in 12 seconds and clicks send.
The draft was linguistically perfect. And wrong. - In a Q1 2025 pilot, an AI discharge assistant recommended a drug for a patient listed as allergic to it.
The system's actual clinically actionable error rate: 0.98%.
The vendor's claimed rate: 0.08%.
12x off. A nurse caught it, not the reviewing physician. - Second failure mode: unverifiable accuracy claims.
Sept 2024, the Texas AG settled with Pieces Technologies over a claimed <0.001% "critical hallucination rate" ā software live at Houston Methodist, Parkland, Texas Health.
No AI-specific law needed. Consumer protection sufficed. - Pieces' <0.001% claim ran live at 4 hospitals before anyone asked how it was measured.
The lesson: when a vendor cites an error rate, ask ā measured on what dataset? Validated by whom? Over what period? On which patient demographics? - Third failure mode: demographic blind spots.
Pulse oximeters overestimate oxygen saturation by 0.6-1.5 points in darker skin. Black patients are ~3x more likely to have occult hypoxemia the device misses.
Any triage model using SpO2 as an input inherits that bias. - The Epic Sepsis Model claimed an AUC of 0.76-0.83.
External validation at Michigan Medicine: AUC 0.63. Sensitivity 33% (misses 2 of 3 sepsis cases). PPV 12% (88% false alarms).
Black and Hispanic patients ā ~2x the sepsis incidence ā get the worst performance. - And the regulatory wall is closing:
ā¢ CA AB 3030 ā AI disclosure in clinical messages (2025)
ā¢ Colorado AI Act ā June 30, 2026
ā¢ EU AI Act, clinical AI = high-risk ā Aug 2, 2026, fines up to ā¬15M or 3% of global turnover
AI malpractice claims are already up 14% since 2022. - A typical health system runs 5-15 AI tools. None independently verified, each with its own blind spots ā and no vendor audits itself honestly.
We're the independent layer that does: safety assessments, bias audits across demographics, governance architecture. - Honest question for CMIOs: when your AI vendor cites an accuracy number, do you have a way to independently verify it ā or are you trusting their math?
#HealthcareAI #ClinicalAI - We wrote up the three failure modes, the regulations, and how to build the safety layer: https://veriprajna.com/solutions/healthcare-ai-safety