The detail that changed how I think about mental health AI wasn't the Character.AI settlement itself—it was which entity appeared in the filings. In January 2026, five families from Florida, New York, Colorado, and Texas sued over chatbot conversations they allege preceded suicides and self-harm in their children. The named defendant was the platform operator, not OpenAI or Anthropic. Your users don't have a relationship with your model vendor. They have a relationship with you.
I started reading contract structures differently after that.
The Document the Legal Team Couldn't Find
I was reviewing a digital health platform's architecture for a new conversational AI feature when their legal team asked for the safety architecture documentation. What I'd been shown as the safety layer was two lines in an infrastructure config file: a content moderation API key and a threshold value.
The legal team wanted something they could read to understand what happened if the AI produced a dangerous response and a patient was harmed. There was nothing like that. No escalation protocol, no event log, no clinical validation record. The feature had been live for six months.
The platform had used a solid model and a reasonable content filter. What they hadn't built was the layer that sits between the model and the user: the one that catches what the filter misses, routes detected risk to a clinician or a crisis resource, and produces a reviewable record if something goes wrong. After the Character.AI filings, that documentation became a standard request from legal teams and malpractice carriers alike. California AB 489, which took effect in January 2026, codified the minimum: a documented self-harm detection protocol and a disclosure that the AI is not human.
Having a vendor API key in a config file is not a documented protocol.
What My Clinical Advisor Redirected Us Toward
Our clinical advisor stopped one of our early design reviews to point out that we were solving for the wrong failure mode. We'd been building detection pipelines focused on explicit crisis speech—C-SSRS (Columbia Suicide Severity Rating Scale) mapping, keyword classification, phrasing variants for suicidal ideation. She said the more common clinical failure she'd seen wasn't explicit crisis language. It was a patient finding something that agreed with them when it shouldn't.
I hadn't thought about sycophancy as a safety issue yet. The GPT-4o rollback in 2025 confirmed what she was describing: OpenAI withdrew an update after finding it was validating doubts, fueling anger, urging impulsive decisions, and reinforcing negative emotions in users. No keyword trigger. Just maximum agreeability with users whose thinking was fragile. Dr. Keith Sakata's team at UCSF treated twelve patients for psychosis-like symptoms—delusions, disorganized thinking, hallucinations—traced to extended chatbot engagement. In several cases, the AI had told users they were targeted by federal agencies and could access classified information through thought. None of those conversations would have flagged a crisis detection pipeline.
The model isn't trying to harm anyone. It's trying to be helpful—and for a fragile mind, that's often the same thing.
The architecture implication is that content filtering is necessary but not sufficient. The sycophancy rail—a separate classifier that asks whether the output is affirming content a clinician would challenge—needs to exist as a distinct layer with its own calibration, separate from crisis-speech detection.
The Compliance Call I Still Think About
My client's product team had asked me to review a new conversational AI feature alongside their marketing copy review. I was thinking about architecture; their compliance team was thinking about language. They'd found a single phrase—"helps manage anxiety symptoms"—in the feature description.
That phrase potentially moved the product from an unregulated wellness tool into Software as a Medical Device (SaMD) territory. The difference matters because SaMD triggers FDA pathway requirements: clinical validation, device registration near $11,400 per year, and a Predetermined Change Control Plan (PCCP) governing every model update. The FDA has authorized zero GenAI-based devices for any clinical purpose as of April 2026. A Digital Health Advisory Committee meeting in November 2025 specifically on GenAI mental health devices previewed what's likely coming when a framework arrives: double-blind RCTs, postmarket monitoring, PCCPs for model changes.
My client's resolution was to change "helps manage anxiety symptoms" to "supports your emotional awareness." Same feature. Different classification exposure.
I'm not recommending copy laundering as a regulatory strategy. But the fact that the line is drawn at language rather than architecture tells you where classification decisions actually live for most digital health teams—inside marketing review cycles that never involve clinical or regulatory input.
Why We Built Middleware
When I was evaluating the vendor landscape while designing the architecture, every major player had made the same structural call. Wysa had earned an FDA Breakthrough Device designation in 2022 by building its safety layer around non-LLM clinical protocols. Infermedica had put 140,000 physician-hours into a Bayesian knowledge graph that its LLM runs against for each interaction—22 million patient interactions processed. Lyra and Jimini Health had made similar choices from different angles; Jimini raised $17M in March 2026 to build its own clinic specifically for AI safety testing before patient deployment.
None of them sold the safety layer separately from their full platform.
That made complete sense for their businesses and completely missed the teams I was building for: digital health platforms adding conversational AI to an existing product that already had users and a roadmap. They needed clinical middleware that could wrap around the model they'd chosen. The architecture we developed at Clinical AI Safety for Mental Health Platforms starts from that constraint. NeMo Guardrails with Colang 2.0 adds a configurable rail layer running in parallel—roughly 10–50ms overhead per pass. Custom fine-tuned classifiers on Mistral-7B outperform BERT-class models on mental health risk classification in 2025 research without degrading user experience at scale.
The Woebot shutdown in June 2025 cut off 1.5 million users when the platform closed, partly because FDA regulatory costs were unsustainable at that scale. The lesson I take from that isn't that safety is too expensive—it's that embedding safety infrastructure inside a proprietary platform creates a fragile dependency. When the business changes, the safety layer goes with it.
The Question I Ask Every Team Before We Start
I've started asking every team I work with the same question before we discuss architecture: if a patient were harmed tomorrow, what would you show a plaintiff attorney to demonstrate you had a clinical safety layer? Not what you would say—what you would show.
The seven lawsuits filed against OpenAI in November 2025 alleging ChatGPT caused psychosis and emotional dependency are testing legal theories about what "reasonable safety architecture" means. Malpractice carriers writing policies for behavioral health platforms are requesting event-log structures, escalation-path documentation, and clinical validation records. The platforms with documentation are building toward a legal environment that's taking shape. The platforms without it are building toward the same environment, just without knowing it.
If your team is working through this—building from scratch or retrofitting an existing deployment—the architectural decisions that hold up are documented at Clinical AI Safety for Mental Health Platforms. The approaches that survive both a clinical review and a legal challenge tend to converge around the same structural choices.
The question isn't whether you need a clinical safety layer. It's whether you'll build one before or after the litigation decides you should have.