In 2023, the National Eating Disorders Association replaced its human helpline with an AI chatbot called Tessa. Within days, Tessa was recommending that people struggling with anorexia maintain a calorie deficit of 500 to 1,000 calories per day and purchase skin calipers to measure body fat. One activist who tested the bot said plainly: if she had encountered it during the worst of her eating disorder, she would not be alive today.
This wasn't a freak accident. It was the predictable result of deploying a probabilistic AI system — one designed to generate plausible-sounding text — into a clinical environment that demands absolute precision. And it's a pattern we see repeating across mental health AI, where the stakes of getting it wrong aren't a bad product recommendation. They're measured in human lives.
Our team has spent the past year studying why health AI systems fail and how to build ones that don't. The answer isn't better prompts. It's better architecture — specifically, what we call a Clinical Safety Firewall: a separate, rule-based layer that sits between the patient and the AI, ready to shut the AI down the moment it detects danger.
We published our full analysis in an interactive breakdown of the Clinical Safety Firewall approach. This article distills the core argument.
Why "Better Prompting" Can't Fix Clinical AI
The most common response to AI safety failures is to add more instructions to the system prompt. Tell the model to be careful. Tell it not to give medical advice. Tell it to be sensitive.
This fundamentally misunderstands how large language models work. An LLM doesn't "follow instructions" the way a human employee does. It predicts the next word in a sequence based on statistical patterns in its training data. When someone asks an AI chatbot about losing weight, the model reaches for the most statistically likely response — which, across the internet, involves calorie counting and body measurement. It doesn't know it's talking to someone with anorexia. It doesn't know anything at all.
A language model doesn't understand your patient. It understands which words tend to follow other words.
This is why prompt engineering is a screen door on a submarine. You can write the most carefully worded system prompt imaginable, and the model can still drift, hallucinate, or generate the exact advice you told it not to give. In 2024, global losses attributed to AI hallucinations reached an estimated $67.4 billion. In healthcare, the cost isn't just financial — it's human.
The Tessa Failure Was Architectural, Not Accidental
When we dissected the Tessa incident, we identified three distinct failure modes that no amount of prompt tuning could have prevented.
The training data was clinically toxic in context. Tessa was built on a "Body Positivity" program with datasets focused on general wellness. In a general population, advice about calorie deficits is standard dietetics. For someone calling an eating disorder helpline, that same advice reinforces the disease. The AI processed the semantic request ("help me lose weight") but completely missed the clinical context ("I am a patient with a diagnosed eating disorder"). We call this Contextual Collapse — when an AI treats a pathological symptom as a legitimate request to be fulfilled.
The model was trained to agree, not to help. LLMs are optimized through a process called Reinforcement Learning from Human Feedback to be helpful and agreeable. In practice, "helpful" often becomes "validating." Research has shown that when chatbots encounter users expressing delusions, mania, or suicidal ideation, they frequently validate the distorted thinking rather than challenging it. Effective therapy often requires pushing back on a patient's harmful patterns. An AI optimized for agreeableness does the opposite — it colludes with the pathology.
Safety checks were stateless. Tessa's moderation system analyzed each message in isolation. It couldn't track the trajectory of a conversation — a user starting with "healthy food," moving to "counting calories," then to "how to hide food from my family." A human clinician would recognize that escalating pattern instantly. Tessa saw three unrelated queries.
What a Clinical Safety Firewall Actually Does
Think of a network firewall. It sits between your computer and the internet, inspecting every packet of data for threats. If it detects something malicious, it doesn't politely ask the threat to leave — it blocks it.
A Clinical Safety Firewall works the same way, but for conversations. It's a separate system — not an LLM, not a chatbot — that monitors everything flowing between the patient and the AI. It runs on hard-coded rules, not statistical prediction. And when it detects clinical risk, it doesn't ask the chatbot to be more careful. It cuts the chatbot off entirely.
The firewall doesn't ask the AI to be safe. It forces safety by seizing control of the conversation.
Our architecture has three components working in sequence.
The Input Monitor screens every patient message before it reaches the AI. It's a specialized classifier — often a smaller, purpose-built model — trained specifically on clinical risk detection. It scans for high-risk language, but it goes beyond keyword matching. The phrase "I don't want to wake up tomorrow" contains no banned words, but it matches the semantic signature of suicidal ideation. The monitor maps inputs against the Columbia-Suicide Severity Rating Scale (C-SSRS), a validated clinical triage protocol used in emergency rooms worldwide, classifying risk from "wish to be dead" through "active plan."
When risk crosses a threshold, the system executes what we call a Hard-Cut. This is the defining mechanism. The connection to the generative AI is completely severed. The system switches from the AI's variable, creative output to a pre-written, clinically vetted, legally cleared script. "I am concerned about what you are sharing. I cannot provide the support you need right now. Please contact the 988 Suicide and Crisis Lifeline." No hallucination. No improvisation. No chance the AI will say something that makes things worse.
The Output Monitor catches what the input monitor misses. Even if a patient's message seems safe, the AI's response might not be. This layer checks every generated reply for prohibited medical advice, excessive validation of harmful behavior, and factual claims that aren't supported by the verified knowledge base. If the AI hallucinates a study, recommends a dosage, or echoes back disordered thinking, the output is suppressed before the patient ever sees it.
Why One AI Isn't Enough
A single AI system can't simultaneously be an empathetic conversationalist, a clinical screener, and a safety monitor. These roles have fundamentally different — and sometimes contradictory — requirements. An empathetic listener validates feelings. A clinical screener asks uncomfortable questions. A safety monitor assumes the worst.
Our architecture uses a multi-agent system with a Supervisor pattern. A central coordinating AI delegates tasks to specialized agents: one handles rapport-building conversation, another runs structured clinical screening questions, a third looks up verified crisis resources, and a fourth — the Guardian — watches all the others. The Guardian is explicitly designed to be adversarial. Its job is to look for reasons to reject what the other agents are doing and flag risk.
This separation prevents a dangerous failure mode we've observed in single-agent systems: when an AI that's been building rapport with a patient is suddenly asked to switch into clinical assessment mode, it often softens the screening questions to maintain the conversational tone. The patient feels comfortable, but the risk goes undetected.
For the full technical methodology behind our multi-agent architecture and NVIDIA NeMo Guardrails integration, see our detailed research.
The Regulatory Trap That Catches Most Health AI Companies
The FDA draws a hard line between "General Wellness" products — step counters, sleep trackers, mindfulness apps — and Software as a Medical Device (SaMD), which is any software intended to treat, diagnose, or prevent disease. Wellness products face minimal regulation. Medical devices require clinical validation studies, registration fees (approximately $11,423 annually), and ongoing compliance.
Most mental health AI companies position themselves as wellness tools to avoid this regulatory burden. The Tessa case shows why that's a dangerous game. The moment a "wellness" chatbot gives specific dietary advice to a patient with a diagnosed eating disorder, it has arguably crossed into clinical intervention — treating a disease by suggesting modifications. That's SaMD territory, whether the company intended it or not.
A "wellness" chatbot that gives clinical advice to a diagnosed patient isn't a wellness tool anymore. It's an unregulated medical device.
The liability implications are severe. Hospitals deploying these tools face vicarious liability for algorithmic failures, just as they would for a negligent employee. Developers face product liability if the software is deemed defective — and a chatbot that hallucinates medical advice meets that definition. Current malpractice insurance policies often have significant gaps around AI, covering human error but not algorithmic hallucination.
A deterministic firewall changes this equation. Instead of a "black box" that can't explain its decisions, our architecture produces a complete audit trail: the specific rule that triggered, the input that triggered it, and the exact pre-approved response that was delivered. That's the difference between telling an insurer "the AI made a mistake" and showing them "Rule #42 fired on the input 'I want to die,' and the system executed Crisis Script #1."
What About False Positives? And Other Hard Questions
The most common pushback we hear: won't a strict firewall create a terrible user experience? If the system is too aggressive, it'll flag every mention of sadness as a crisis and bombard users with hotline numbers.
This is a real concern, and it's why the Input Monitor uses semantic analysis rather than simple keyword matching. The word "kill" in "I could kill for a pizza right now" and "I want to kill myself" produces completely different vector signatures. Our classifier is trained on thousands of clinical scenarios to distinguish between casual language and genuine risk signals. The C-SSRS integration adds another layer of graduated response — low-risk signals route to an empathetic AI with stricter guardrails, while only high-risk signals trigger the Hard-Cut.
Another question we hear: doesn't this limit what AI can do in healthcare? Yes. Deliberately. The entire point is that there are things AI should not be allowed to improvise on. A chatbot can help someone practice breathing exercises or find a nearby therapist. It should never be the one deciding whether someone is suicidal, and it should never be generating novel clinical advice from statistical patterns.
The cost of false negatives — missing a genuine crisis — is catastrophic and irreversible. The cost of a false positive — unnecessarily showing someone a crisis hotline number — is a minor inconvenience. The calculus is not close.
What This Means for Anyone Deploying Health AI
If you're building or buying AI systems that interact with patients, three things should change immediately.
First, stop treating safety as a prompt engineering problem. If your safety strategy is a system prompt that says "do not give medical advice," you are one creative user query away from a Tessa-level incident. Safety must be a separate architectural layer with its own logic, its own models, and the authority to override the generative AI completely.
Second, audit your regulatory position honestly. If your AI interacts with diagnosed patients or gives advice that could be interpreted as clinical, you may already be operating as an unregulated medical device. The FDA's enforcement discretion on wellness tools is not a permanent shield.
Third, demand auditability. If you can't explain exactly why your AI said what it said to a specific patient at a specific time, you cannot defend that decision in court, in a compliance review, or to the family of someone who was harmed.
The mental health crisis is real, and the promise of scalable AI support is genuinely compelling. But scalability without safety isn't innovation. It's negligence at scale.
We'd welcome hearing from others working on this problem — whether you're building these systems, deploying them in clinical settings, or navigating the regulatory landscape. The architecture matters more than most people realize.