One in fourteen AI-drafted patient messages in a recent clinical simulation posed a risk of severe harm. One posed a risk of death. And the doctors reviewing those messages? They missed two-thirds of the errors.
These aren't hypothetical numbers. They come from a landmark simulation study published in The Lancet Digital Health in April 2024, conducted by researchers from Harvard Medical School, Yale School of Medicine, and the University of Wisconsin. The study tested GPT-4's ability to draft responses to patient portal messages — the kind of messages millions of patients receive through systems like MyChart every day. The results should concern every health system executive, every clinical informaticist, and every patient who's ever received a message from their doctor's office.
We've spent months analyzing these findings and the broader landscape of AI in clinical communication. What we've found is that the healthcare industry has a grounding problem — and the most popular approach to deploying AI in patient communication is architecturally incapable of solving it.
The Study That Should Have Stopped the Industry Cold
The Lancet simulation tested 156 AI-generated draft responses to real-world patient portal messages within a simulated electronic health record. Twenty practicing primary care physicians reviewed the drafts, just as they would in a real clinical workflow.
The AI performed impressively on the surface. Eighty percent of the reviewing physicians agreed that AI reduced their cognitive workload. Ninety percent reported trusting the tool's performance. The drafts were fluent, empathetic, and well-structured — exactly the kind of output that makes a busy clinician click "send."
The AI didn't fail by sounding wrong. It failed by sounding too right.
Beneath the polished language, 7.1% of the drafts risked severe patient harm, and 0.6% risked death. The fatal-risk case involved the AI failing to instruct a patient to seek emergency care for a life-threatening symptom, instead generating a calm, routine-sounding response. It read like a perfectly normal message. That was the problem.
Of the intentionally erroneous drafts embedded in the simulation, physicians missed an average of 2.67 out of 4. Only one physician out of twenty caught all four. And between 35% and 45% of the erroneous drafts were submitted to the simulated patient completely unedited — errors and all.
This isn't a technology failure. It's a human-technology interaction failure, and it has a name.
Automation Bias: When Trust Becomes a Liability
Automation bias is what happens when people defer to a machine's output instead of applying their own judgment. It's the same cognitive shortcut that makes pilots over-rely on autopilot or accountants skip double-checking a spreadsheet formula. In high-stakes environments, it can be lethal.
In the Lancet study, the mechanism was clear. The AI's drafts were linguistically sophisticated and emotionally appropriate — they sounded like something a thoughtful physician would write. That surface quality triggered a trust response. Physicians reported 90% confidence in the tool even as they were missing critical errors at a rate that was statistically significant (p < 0.001).
When AI-generated text is polished enough, human reviewers stop reviewing and start approving.
The errors weren't typos or awkward phrasing. They were substantive clinical failures: fabricated medical information, outdated treatment protocols, and a fundamental inability to assess how urgent a patient's situation actually was. The AI couldn't distinguish between a routine question and a medical emergency — and the physicians reviewing its output often couldn't tell the difference either, because the language was equally confident in both cases.
This is the core vulnerability in what the industry calls the "human-in-the-loop" model. The human is in the loop, but the loop is moving too fast and the AI's output is too convincing for the human to function as an effective safety check.
Why "LLM Wrappers" Aren't Enough for Clinical Care
Most AI tools currently deployed in healthcare patient communication follow a common pattern: take patient data from the electronic health record, pass it to a commercial large language model like GPT-4 or Gemini through an API, and return the generated text to the clinician for review. The industry calls these "wrappers" — thin software layers around a general-purpose AI model.
Wrappers are fast to build and easy to deploy. They're also fundamentally unsuited for clinical care, for three reasons that compound each other.
The prediction problem. LLMs generate text by predicting the next word based on statistical patterns in their training data. They don't reason about medical concepts — they pattern-match against language. Think of it like autocomplete on your phone, but vastly more sophisticated. It can produce text that sounds like a doctor, but it isn't thinking like a doctor. It has no understanding of drug interactions, symptom severity, or when a patient needs to call 911 instead of scheduling a follow-up.
The knowledge problem. Commercial LLMs are trained on data with a fixed cutoff date. They can't reference your patient's lab results from yesterday, the updated prescribing guidelines published last month, or your institution's specific clinical protocols. A wrapper that doesn't solve this gap is generating responses in a clinical vacuum.
The privacy problem. General-purpose LLM APIs weren't designed for protected health information. Without rigorous HIPAA-compliant infrastructure — including Business Associate Agreements, data masking, and protection against prompt injection attacks (where adversarial inputs trick the model into revealing sensitive data) — every patient message sent through a wrapper is a potential privacy breach.
We mapped these architectural gaps in detail in our interactive analysis of clinical AI grounding.
What Grounded AI Actually Looks Like
If wrappers are the problem, what's the solution? The answer isn't to abandon AI in patient communication — the burnout crisis is too severe for that. Primary care physicians spend an average of 10 hours per month on patient portal messages alone, much of it historically unbillable. The need is real. But the architecture has to change.
Grounded AI means building systems where the language model never generates a response from its training data alone. Instead, it's anchored to verified, current, patient-specific information at every step.
Retrieval-Augmented Generation (RAG) is the foundation of this approach. Before the AI writes anything, it first searches a curated set of trusted sources — the patient's clinical notes, peer-reviewed medical literature, institutional treatment guidelines — and generates its response based only on what it finds. Every claim in the output can be traced back to a specific source document. The clinician reviewing the draft doesn't just see a polished paragraph; they see where the AI got its information.
This matters enormously for the automation bias problem. When a physician can click on a citation and verify the source, they shift from passive approval to active verification. The AI becomes a research assistant that shows its work, not an oracle that demands trust.
Grounded AI doesn't ask clinicians to trust it. It gives them the evidence to verify it.
The most sophisticated implementations go further by incorporating Medical Knowledge Graphs — structured networks that represent clinical knowledge as relationships between concepts rather than strings of text. A knowledge graph can explicitly encode that Drug A is contraindicated for patients with Condition B who are also taking Drug C. When the AI queries this graph, it's performing something closer to clinical reasoning than statistical pattern-matching.
Systems like MediGRAF, which combine graph databases with language models, have demonstrated 100% recall on factual clinical queries — meaning they retrieve every relevant piece of information without missing critical details. That's a fundamentally different safety profile than a standalone language model guessing its way through a response.
California's New Law — And Why Compliance Alone Isn't Enough
California Assembly Bill 3030, effective January 1, 2025, now requires healthcare providers to notify patients whenever generative AI is used to communicate clinical information. Written messages need a disclaimer at the top. Voice messages need verbal disclosure at the start and end. Video and chat interactions need continuous disclosure throughout.
There's an exemption: if a licensed healthcare provider reads and reviews the AI-generated communication before it's sent, the disclosure requirement doesn't apply.
On paper, this seems reasonable. In practice, the Lancet study demolishes it.
If physicians miss 66% of AI-generated errors — and submit 35% to 45% of erroneous drafts completely unedited — then "read and reviewed" is a legal fiction for many clinical workflows. The exemption assumes a level of human scrutiny that the evidence shows doesn't reliably exist when automation bias is in play.
Health systems relying on the review exemption to avoid disclosure are building their compliance strategy on a foundation the research doesn't support. The smarter approach is to build AI systems that make meaningful review possible: surfacing citations, flagging low-confidence outputs, and requiring active confirmation rather than passive click-through.
For the full technical methodology behind our analysis of these architectural requirements and the regulatory landscape, see our detailed research.
The Malpractice Question No One Wants to Answer
Here's the legal reality health systems need to confront: if an AI drafts a message that harms a patient, and the reviewing physician approved it, who's liable?
Medical malpractice law centers on the "standard of care" — what a reasonable physician would do in similar circumstances. AI is rewriting that standard from both directions. Soon, failing to use a validated AI tool that could have prevented an error may be considered negligent. But blindly accepting an AI recommendation that leads to harm will also be a breach.
The physician is caught in the middle, and the technology they're using isn't designed to help them out.
There's an additional wrinkle that most health systems haven't accounted for: model drift. AI models change over time — through retraining, fine-tuning, or updates by the API provider. A system that was safe last quarter may perform differently today. Malpractice insurers are beginning to require documented proof of human oversight, audit logs showing which model version generated each output, and evidence of ongoing safety testing. New insurance products covering AI-related clinical errors typically carry low coverage limits and stringent documentation requirements.
Without the ability to produce a complete audit trail — which model, which version, which data it accessed, what the clinician reviewed, and what they changed — health systems are exposed in ways their current malpractice coverage may not address.
What About Patients Who Prefer AI Messages?
This is a fair question, and the research offers a nuanced answer. Studies show patients often rate AI-generated messages highly for empathy and thoroughness — sometimes higher than messages written by physicians. But when patients learn AI was involved, their satisfaction drops.
This isn't irrational. Patients value the belief that their doctor is personally engaged in their care. An AI message that sounds warm but was never meaningfully reviewed by a human represents a broken promise in the clinical relationship, even if the information happens to be correct.
The goal isn't to hide AI's involvement or to eliminate it. It's to use AI for what it's genuinely good at — handling structured, routine communication tasks — so that clinicians have the time and cognitive bandwidth for the complex, nuanced interactions that require human judgment. AI should free physicians to be more present with patients, not replace that presence with a convincing simulation of it.
What Health Systems Should Do Now
The evidence points to a clear set of priorities:
Stop treating wrappers as production-ready clinical tools. API integrations with commercial LLMs are useful for prototyping. They are not safe for patient-facing clinical communication without substantial grounding architecture around them.
Invest in retrieval-augmented systems with verifiable citations. Every AI-generated clinical statement should link back to a source the reviewing physician can check in seconds, not minutes.
Implement continuous adversarial testing. Techniques like Med-HALT (Medical Domain Hallucination Test) — which challenges AI with fabricated questions and misleading answer suggestions — should run regularly against any clinical AI system, not just at launch.
Design review workflows that fight automation bias. This means requiring active engagement from reviewers: confirming source accuracy, flagging confidence levels, and making "approve unchanged" harder than "review and edit."
Prepare for disclosure mandates beyond California. AB 3030 is the beginning, not the end. Build systems that can support transparent AI disclosure without sacrificing the efficiency gains that make AI worthwhile.
The physician burnout crisis is real, the administrative burden is crushing, and AI has genuine potential to help. But the current approach — wrapping a general-purpose language model in a thin interface and asking overworked doctors to catch its mistakes — isn't a solution. It's a new category of risk dressed up as efficiency.
The question isn't whether AI belongs in clinical communication. It's whether we'll build it to the standard that clinical communication demands.
We'd welcome hearing from health system leaders, clinical informaticists, and patient safety officers who are wrestling with these decisions. How is your organization approaching the grounding problem?