AI Patient Messages Had a 7% Severe Harm Rate

The Problem

An AI drafted a response to a patient describing a life-threatening symptom. Instead of telling the patient to go to the emergency room, it sent a calm, routine reply. That single message was later classified as posing a direct risk of death.

This was not a fringe failure. In an April 2024 study published in The Lancet Digital Health, researchers from Harvard Medical School, Yale, and the University of Wisconsin tested GPT-4's ability to draft patient portal messages. They ran 156 simulated patient conversations through the AI inside a mock electronic health record system. The results were alarming: 7.1% of the AI-generated drafts posed a risk of severe harm to patients. And 0.6% — one message in the simulation — carried a direct risk of death.

But here is the part that should keep you up at night. Twenty practicing primary care physicians reviewed those AI drafts. They missed an average of two-thirds of the dangerous errors. Only one doctor out of twenty caught all four intentionally flawed messages. Between 35% and 45% of the erroneous drafts were submitted to patients with zero edits. The doctors trusted the AI. Ninety percent said they trusted the tool's performance. Eighty percent said it reduced their mental workload. The AI wrote well, sounded empathetic, and got the tone right. It just got the medicine wrong — and the doctors did not notice.

If your health system uses AI to draft patient communications, this is your risk profile right now.

Why This Matters to Your Business

The financial and legal exposure here is not theoretical. It is measurable and growing.

California's Assembly Bill 3030 took effect January 1, 2025. It requires every health facility, clinic, and physician practice to notify patients whenever generative AI is used to communicate clinical information. That means written disclaimers at the top of emails, verbal disclaimers at the start and end of phone calls, and persistent disclaimers throughout video and chat interactions. If you fail to comply, your facilities face fines and licensing actions. Your individual physicians face disciplinary action against their medical licenses.

AB 3030 does include an exemption: if a licensed provider "reads and reviews" the AI output, you do not have to disclose it. But consider the Lancet study data:

• 7.1% of AI-generated drafts posed severe harm risk
• 66.6% of those dangerous drafts were missed by reviewing physicians
• 35%–45% of erroneous drafts were sent with no edits at all
• 90% of physicians reported trusting the AI tool's performance

That exemption assumes human review actually works. The evidence says it often does not. If a patient is harmed by an AI-drafted message your physician approved without meaningful review, you face a malpractice claim where the standard of care is evolving fast. Courts are now asking whether providers accepted AI recommendations blindly. They are recognizing algorithmic bias as a source of real patient harm. And your malpractice insurer may require documented audit logs showing the model version, the reasoning steps, and the physician's specific edits.

New insurance products now cover AI hallucination claims — but they come with low limits and strict oversight documentation requirements. If you cannot produce that documentation, your coverage may not hold.

What's Actually Happening Under the Hood

Most healthcare AI tools today are what the industry calls "LLM wrappers." Think of them as a thin software shell around a general-purpose AI like GPT-4 or Google Gemini. Your electronic health record sends the patient's message to the AI, and the AI sends back a draft. That is essentially the entire system.

Here is why that is dangerous. These models predict the next word in a sentence based on statistical patterns. They do not reason about medicine. They do not check drug interactions. They do not evaluate whether a patient's symptom is a Tuesday-morning annoyance or a Saturday-night emergency. The Lancet study found that the AI's most critical failures came from its inability to assess how urgent a patient's situation was.

Imagine an employee writing memos by copying phrases from thousands of old memos. The grammar is perfect. The tone is professional. But the employee has no idea what the words actually mean. That is how a general-purpose AI handles your patient messages.

These models also have knowledge cutoffs — fixed dates beyond which they know nothing. They cannot reference your patient's latest lab results or the newest clinical guidelines unless someone builds that connection. They often cannot process imaging, ECG waveforms, or genomic data. And without a specific HIPAA Business Associate Agreement and data-masking protocols, sending patient data through a general-purpose AI API creates serious privacy risk. Adversarial prompt injection attacks can potentially trick these models into revealing sensitive patient information.

The result: your AI sounds confident. Your doctors trust it. And no one catches the errors until a patient is harmed.

What Works (And What Doesn't)

First, here are three approaches that do not solve this problem:

"Just train doctors to be more careful." Automation bias is a well-documented psychological effect. When AI produces polished, empathetic text, even experienced clinicians drop their guard. The Lancet study proved this with statistical significance (p < 0.001).

"Fine-tune a medical AI model." Research shows that specialized medical models like MedGemma scored only 28%–61% accuracy in some hallucination tests. Domain fine-tuning alone does not fix the underlying reasoning gap.

"Add a disclaimer and move on." AB 3030 compliance is necessary but not sufficient. A disclaimer does not prevent harm. It just tells the patient who to blame.

Here is what actually works — a grounded architecture built in three steps:

Step 1: Retrieve before you generate. Before your AI writes a single word, it should pull the relevant source material. This is called Retrieval-Augmented Generation (RAG) — a technique where you feed the AI verified documents like the patient's clinical notes, current treatment guidelines, and institutional protocols. The AI then generates its response based only on that retrieved information, not its general training data.

Step 2: Structure your knowledge as a graph. A Medical Knowledge Graph maps clinical concepts as connected relationships — a drug links to its contraindications, which link to specific patient conditions. Systems like MediGRAF use graph databases to translate plain questions into precise queries. This achieves 100% recall on factual clinical queries because the system traverses verified relationships, not statistical guesses.

Step 3: Cite every claim. Every statement in the AI's output should link back to its source document. Your reviewing physician does not have to guess whether the AI is correct. They can verify each claim against the cited source in seconds. This transforms passive review into active verification.

The audit trail advantage is what makes this approach defensible. Every AI-generated draft carries a record of which documents were retrieved, which knowledge graph paths were traversed, and which source supports each statement. When your compliance team, your insurer, or a plaintiff's attorney asks how a specific message was generated, you can show them the complete logic trail. That is the difference between a defensible process and a liability.

For your healthcare AI compliance and safety programs, this architecture is not optional — it is the emerging standard of care. The same principles apply to your RAG and knowledge graph infrastructure and to the red teaming and validation testing that should stress-test your systems daily.

You can read the full technical analysis for the detailed engineering specifications, or explore the interactive version for a guided walkthrough of the architecture.