
90% of physicians trusted the AI. They still missed two-thirds of its dangerous errors.
That's not a thought experiment. It's the Lancet Digital Health finding (April 2024) on AI-drafted patient portal messages: 7.1% posed a severe harm risk, 0.6% posed a death risk — and reviewing physicians caught only about a third of the bad drafts, and 35–45% went out entirely unedited.
This is automation bias, and it's the quiet failure mode no vendor demo shows you. When an AI draft is fluent, empathetic, and well-formatted, the prose quality starts substituting for clinical verification. The note reads right, so it gets sent.
California's AB 3030 leans on a "read and reviewed" exemption — if a licensed provider reviews the AI message, no disclosure is required. But if physicians miss 66.6% of harmful errors, "reviewed" is often a rubber stamp, not a safeguard.
The pattern repeats across the stack. A vendor's <0.001% "critical hallucination rate" was challenged by the Texas AG and settled in Sept 2024 — now a five-year mandate to disclose how that number was actually calculated. In one Q1 2025 discharge-assistant pilot, clinically actionable misstatements ran at 0.98% — 12x the vendor's claimed 0.08%. The Epic Sepsis Model claimed an AUC of 0.76–0.83; external validation at Michigan Medicine measured 0.63, with 33% sensitivity. Pulse oximeters overestimate oxygen saturation in darker skin, and any triage model using SpO2 inherits that blind spot.
The uncomfortable truth for a CMIO: your health system likely runs 5–15 clinical AI tools, and almost none have been independently verified across every patient demographic.
Our work is the safety infrastructure between those tools and your patients — independent assessments, bias audits, and governance built for the person who needs answers, not a marketing deck.
Save this for your next AI governance committee meeting.
#ClinicalAISafety #HealthcareAI #AIGovernance #PatientSafety #CMIO