One in fourteen AI-drafted messages from your doctor's office could seriously hurt you. And your doctor probably won't catch it.
That's not speculation. A simulation study published in The Lancet Digital Health in April 2024 — conducted by researchers from Harvard Medical School, Yale, and the University of Wisconsin — tested GPT-4 drafting responses to 156 patient portal messages. 7.1% of those drafts posed a risk of severe harm. One posed a direct risk of death. The AI told a patient with a life-threatening symptom to follow routine advice instead of rushing to the ER.
The part that kept me up at night: when twenty practicing primary care physicians reviewed these AI drafts, they missed an average of two-thirds of the errors. Not because they were careless. Because the AI wrote so well that it felt trustworthy.
I've spent the last two years building clinical AI systems at Veriprajna, and this study crystallized something my team had been arguing internally for months. The way most companies are deploying AI in healthcare right now isn't just sloppy engineering. It's a patient safety crisis hiding behind polished prose.
The Burnout Trap That Opened the Door
To understand how we got here, you need to understand the problem AI was supposed to solve.
Primary care physicians spend an average of 10 hours every month just responding to patient portal messages — the kind you send through MyChart asking about a lab result or a new symptom. For most of that history, this work was unbillable. You're essentially asking doctors to do a second unpaid job on top of their clinical load.
So when tools emerged that could draft these responses automatically using large language models, health systems jumped. The appeal is obvious: paste the patient's message in, get a polished, empathetic draft back, doctor reviews it, hits send. Burnout reduced. Patients get faster replies. Everyone wins.
Except nobody stress-tested what happens when the draft is wrong and the doctor doesn't notice.
Why Doctors Miss What AI Gets Wrong
The Lancet study didn't just find that AI makes mistakes. It found something far more dangerous: doctors trust AI-drafted messages so much that their error-detection collapses.
Ninety percent of the physicians in the study said they trusted the AI tool's performance. Eighty percent agreed it reduced their mental workload. And 35% to 45% of the erroneous drafts were submitted to patients completely unedited — not a word changed.
When the AI writes better bedside manner than you have time for, you stop reading critically. That's not a character flaw. It's a predictable system failure.
Psychologists call this automation bias — the tendency to defer to automated suggestions, especially when they're articulate and confident. It's the same reason pilots sometimes follow autopilot into terrain. The system sounds so sure of itself that overriding it feels like the riskier move.
I watched a demo last year where a health-tech startup showed their AI drafting a patient response about medication interactions. The draft was warm, detailed, and completely wrong about a contraindication. When I pointed it out, the founder said, "That's why we have the doctor review it." I asked how long the average doctor spends reviewing each draft. He didn't know.
That's the gap. The entire safety model for healthcare AI right now rests on human review. And the evidence says human review, without proper support, catches roughly one-third of the problems.
California Just Changed the Rules
While the research community was documenting these risks, California's legislature was writing new ones into law.
Assembly Bill 3030, signed in September 2024 and effective January 1, 2025, requires healthcare providers to disclose to patients whenever generative AI is used to communicate clinical information. Written messages need a disclaimer at the top. Phone calls need verbal disclosure at the start and end. Video and chat interactions need continuous disclosure throughout.
There's an exemption: if a licensed provider reads and reviews the AI-generated content before it reaches the patient, the disclosure requirements don't apply. On paper, that sounds reasonable. In practice, given what the Lancet study showed about the quality of that review, it's a loophole wide enough to drive a malpractice suit through.
I explored this regulatory landscape in depth in our interactive analysis, including how AB 3030's requirements map to different communication channels and what non-compliance actually looks like for health systems.
The law also requires that patients receive clear instructions for contacting a human provider. Violations can trigger fines for facilities and disciplinary action against individual physicians' medical licenses. This isn't guidance. It's enforcement.
The "Wrapper" Problem Nobody Wants to Talk About
Most AI tools in healthcare today are what my team calls "wrappers." They're thin software layers that take patient data from an electronic health record, pass it to a commercial AI model like GPT-4 or Gemini, and return the output with a nice interface on top.
They can be built in weeks. They demo beautifully. And they are fundamentally unsuited for clinical care.
The core issue is how these models generate text. Large language models predict the next word based on statistical patterns in their training data. They don't reason about medicine. They don't understand that a creatinine level of 4.0 means something radically different from 1.0. They assemble plausible-sounding sentences — and in medicine, plausible-sounding is the most dangerous kind of wrong.
These models also have knowledge cutoffs. They can't reference your latest lab results, the updated prescribing guidelines published last month, or the fact that you're allergic to a medication they're about to recommend. Without being connected to current, patient-specific data, they're giving generic answers to specific people. That's not healthcare. That's a search engine with better grammar.
A wrapper doesn't know your patient. It knows what patients sound like on the internet.
And then there's the security dimension. Most general-purpose AI interfaces aren't built for HIPAA compliance. Using them with patient data without rigorous data-masking and a Business Associate Agreement creates privacy exposure that many wrapper developers haven't fully reckoned with.
What Grounded Clinical AI Actually Looks Like
When my team started building our approach, we asked a simple question: what would it take for a clinician to trust an AI draft not because it sounds good, but because they can verify it's right?
The answer required rethinking the architecture from the ground up.
Retrieval-Augmented Generation — RAG — is the foundation. Instead of letting the AI generate responses from its training data alone, a RAG system first retrieves relevant information from verified sources: the patient's own clinical notes, peer-reviewed guidelines, institutional protocols. The AI then generates its response constrained to that retrieved context. Every claim can be traced back to a specific source document.
Think of it like the difference between asking someone to write a medical opinion from memory versus asking them to write it with the patient's chart open in front of them and a requirement to cite their sources.
But retrieval alone isn't enough. Medical knowledge isn't just text — it's a web of relationships. A drug interacts with a condition, which affects a lab value, which changes a dosage recommendation. Medical Knowledge Graphs represent these relationships explicitly, as connected networks rather than paragraphs. Systems built on frameworks like Neo4j can traverse a patient's complete clinical picture — medications, conditions, allergies, recent results — and flag when an AI-generated response contradicts any of those connections.
In testing, graph-augmented systems have achieved 100% recall on factual clinical queries — meaning they found every relevant piece of information, every time. That's the kind of reliability you need when a missed drug interaction could put someone in the ICU.
For the full technical methodology behind these architectural decisions, including our analysis of concept-level modeling versus token-level prediction, see our detailed research.
You Can't Ship What You Haven't Tried to Break
Building a grounded system isn't enough if you don't continuously test it for failure. Traditional software testing — does the button work, does the page load — is meaningless for generative AI. The failure modes are too unpredictable.
We run what the industry calls red teaming: deliberately trying to make our system produce dangerous outputs. Can we trick it into recommending a contraindicated medication? Can we get it to ignore a critical vital sign? Can we extract patient information through indirect prompting?
There's also a benchmark called Med-HALT — Medical Domain Hallucination Test — designed specifically for this problem. It includes tests where the model is presented with a fabricated medical question to see if it recognizes the question itself is nonsensical, and tests where it's fed a wrong answer to see if it confidently agrees rather than pushing back.
One finding from recent benchmarking that surprised me: some "medical-specialized" AI models actually performed worse on safety tests than broader reasoning models. A model called MedGemma scored between 28% and 61% accuracy on certain evaluations, while a general model with stronger reasoning capabilities outperformed it. The lesson is counterintuitive — clinical safety comes from deep reasoning ability, not just medical fine-tuning. Slapping a medical dataset onto a weak reasoner doesn't make it safe. It makes it confidently wrong about medicine specifically.
The most dangerous AI in healthcare isn't the one that says "I don't know." It's the one that never does.
What About Liability When AI Gets It Wrong?
This is the question I get most often from health system executives, and the honest answer is: the legal landscape is still forming, but the direction is clear.
Medical malpractice law centers on whether a physician met the "standard of care" — what a reasonable doctor would do in similar circumstances. AI is about to reshape that standard in both directions. Failing to use a validated AI tool that could have caught an error may become a breach of duty. But blindly accepting an AI recommendation that causes harm will almost certainly be one.
The tricky part is causation. When an AI contributes to a bad outcome, proving exactly how requires understanding the system's decision-making process. If your AI is a black box — no audit trail, no source citations, no version tracking — you can't defend the decision in court. If it's a grounded system with retrievable sources, logged reasoning steps, and documented human review, you have a defensible record.
Insurance is catching up too, slowly. New products are starting to cover claims caused by AI errors, but they typically require documented proof of human oversight and have relatively low coverage limits. The message from insurers is the same one I'd give: if you can't show your work, you can't insure it.
What This Means If You Run a Health System
If you're deploying AI for patient communications today — or planning to — here's what I'd push your team on:
Ask your vendor what happens when the AI is wrong. Not whether it's ever wrong. How the system detects it, flags it, and prevents it from reaching the patient. If the answer is "the doctor reviews it," ask for the data on review effectiveness.
Demand source citations on every AI-generated clinical statement. If the system can't tell you where it got its information, it's guessing. Educated guessing, but guessing.
Prepare for disclosure mandates beyond California. AB 3030 is the first. It won't be the last. Build your workflows assuming patients will know AI is involved, and build trust from that starting point rather than hiding it.
Treat AI safety testing as ongoing, not a launch checkbox. Models drift. Performance degrades. New failure modes emerge. Red teaming isn't a one-time audit — it's a continuous process, like infection control.
In medicine, the standard has never been "usually right." It's "verifiably right, and here's how we checked."
I started Veriprajna because I believed AI could genuinely help solve the burnout crisis that's driving good doctors out of medicine. I still believe that. But the path from here to there doesn't run through faster wrappers and shinier interfaces. It runs through the hard, unglamorous work of grounding AI in clinical reality — making it cite its sources, testing it against its own worst impulses, and building systems where human oversight actually works instead of just existing on a compliance checklist.
The Latin phrase every medical student learns is Primum non nocere — first, do no harm. If we're going to put AI in the exam room, that standard applies to the engineers too.
I'd genuinely like to hear from anyone navigating this in practice — whether you're a clinician reviewing AI drafts, a health system CTO evaluating vendors, or a founder building in this space. What's working? What isn't? Where are the gaps I'm not seeing?