Stateful crisis governance for behavioral-health AI

A mental-health chatbot crisis is a trajectory. A per-message moderator cannot see it.

We built safety middleware that wraps an existing behavioral-health chatbot and enforces a stateful, cross-turn escalation policy the clinical team owns. It catches the escalating conversation a stateless moderator structurally misses, and it proves every decision with a hash-chained, filable audit trail. Safety is an architecture problem, not a prompting problem.

0 vs 68

Unsafe replies delivered, guarded vs unguarded

40-conversation labeled golden set

2 turns

Median earlier detection vs an identical stateless moderator

Same classifier and gate; statefulness alone

0 / 29

False escalations across benign turns

40-conversation labeled golden set

A demo of a safety-architecture pattern on synthetic data. Not a medical device, not clinical advice, not an EHR integration.

Volume without memory

Behavioral-health chatbots are moderated one message at a time. A crisis does not arrive one message at a time.

Most safety review on a mental-health chatbot scores each reply in isolation. Every message is checked, flagged or cleared, and forgotten. That works for a single explicitly dangerous line. It is structurally blind to a conversation that drifts, turn by turn, where no single message is alarming enough to block on its own.

The documented failures follow that shape. The NEDA "Tessa" chatbot handed out calorie-deficit and skin-caliper advice before it was pulled (NEDA, 2023). Clinicians reported chatbot-reinforced psychosis in patients who never met a person who pushed back (Dr. Keith Sakata, UCSF, 2025). OpenAI withdrew a GPT-4o update after it turned sycophantic (OpenAI, 2025). In each case the model sounded supportive on any given turn while the trajectory went somewhere unsafe.

A stateless moderator has no way to see that trajectory, because it has no memory of the turns before. A better base model does not fix this. A perfect chatbot still has no idea of your platform's escalation policy, produces no audit trail you can file, and gives you no deterministic gate to certify. That is why we treat safety here as an architecture problem, and why the demo compares two stacks running the identical model.

Split-screen demo: the same synthetic patient conversation runs through an unguarded MindMate Support chatbot on the left and the same chatbot behind the Veriprajna safety layer on the right, with a pipeline rail reading classifier, trajectory, verifier, gate, audit. — The demo replays one synthetic conversation through two stacks. MindMate Support is a fictional stand-in for an existing chatbot. Synthetic data only, no PHI.

The mechanism: a stateful pipeline that runs on every turn

Advisory models feed a deterministic gate. The gate makes the decision, and the decision is code you can read.

Each patient turn runs through a fixed pipeline. The message is PII-stripped and hashed, a C-SSRS classifier scores its severity on the Columbia structure and returns a level, a confidence, and an ABSTAIN when it cannot justify a severity. A stateful Trajectory Monitor then accumulates risk across turns. This is the core capability, and it is the one thing a per-message moderator does not have: it sees the pattern, not the single line, and it produces an effective risk and a band (BENIGN, WATCH, CONCERN, HIGH, CRITICAL) with a stated reason such as "persistent eating disorder, rising slope."

Before any candidate reply reaches the patient, a multi-critic verifier panel inspects it: a sycophancy and tone critic, a prohibited-pattern matcher, and a clinical-claim checker. A flagged critic forces the gate to at least a configured minimum level, no matter how gentle the reply sounds.

The decision itself is a deterministic 5-level policy gate, and it is Python, not an LLM: L1 CONTINUE, L2 RESTRICT, L3 SUBSTITUTE_SCRIPT, L4 HUMAN_HANDOFF, L5 CRISIS_PROTOCOL. When the gate blocks a reply it substitutes one of a bundled library of clinician-written scripts by level and family, and records the script id. It never improvises crisis language. Every turn is then written as a sha256 entry chained to the one before it.

The guarded stack showing the per-message pipeline rail (classifier, trajectory, verifier, gate, audit) with per-stage latencies, and inline gate results for early turns reading L1 CONTINUE and L2 RESTRICT while the stateless moderator on the same turn reads BENIGN, sees nothing, no memory. — The pipeline runs on every message. Added latency is sub-millisecond per turn (mean 0.16 ms, p95 0.21 ms across the golden set).

The clinical team owns the policy, not engineering

The thresholds, the verifier-forced minimums, and the 12-script library are configured by the clinical safety committee under policy version 2026.04-clinical-v1. Engineering enforces exactly that, and does not change it. A patient's history, retrieved through the stubbed FHIR flag, can lower a threshold so the layer escalates earlier for a more vulnerable patient. When the classifier's confidence is low, it abstains and routes to a human review rather than guessing a severity.

The clinical-policy modal showing verifier-forced minimum levels per critic and finding, uncertainty and boundary rules (low-confidence abstain forces a human, jailbreak score at or above 0.8 forces minimum L3), and the 12-script clinician-approved library with a revealed L2 eating-disorder replacement message. — The policy modal: verifier-forced minimums, abstain and jailbreak rules, and the clinician-approved script library with a revealed replacement message.

One conversation, worked end to end

The default hero scenario is eating-disorder drift: six turns, each individually an ordinary wellness question.

The conversation opens with harmless questions about eating healthier and counting calories. A per-message moderator has nothing to block, and the stateless baseline in the demo stays green, reading WATCH and "sees nothing, no memory" turn after turn. The stateful layer, watching the trajectory, crosses into CONCERN at turn 3. It blocks the chatbot's reply and substitutes a clinician-written grounding script that points to the NEDA Helpline. That is two turns before the first explicitly dangerous message.

Turn 3 of the guarded stack: the cross-turn risk meter reads 3.4 CONCERN, the chatbot reply is blocked and not sent, and a clinician-approved L3 grounding script is substituted with the NEDA Helpline number, while the stateless per-message moderator on the same turn still reads WATCH and sees nothing. — Turn 3: the stateful layer reaches CONCERN and substitutes a grounding script. The stateless moderator, same classifier, still sees nothing.

By the final turns the messages turn explicitly dangerous. The unguarded stack on the left delivers the harmful reply, shown struck through and labeled as delivered to the patient and unsafe. On the right, the verifier panel intercepts the reply, catches the sycophantic tone and the prohibited pattern, and the gate escalates to L4 human handoff, paging a care-team member with full context. We describe the catch, not the harmful content itself.

Final turns: on the right the verifier panel intercepts the candidate reply, the risk meter reads 5.0 CRITICAL and the gate caps the escalation at L4 human handoff because there is no acute L5 cue, a clinician-approved handoff script is substituted, and the harmful reply is blocked and not sent, while the unguarded left stack shows the same reply struck through as delivered and unsafe. — The verifier intercepts the candidate reply and the gate escalates to L4 human handoff. The unguarded stack delivers the same reply.

The session result makes the delta concrete, and it is attributable to statefulness alone because both stacks used the same classifier and the same gate. The only difference was cross-turn memory.

The session result panel: caught 2 turns earlier than the identical stateless moderator (turn 3 versus turn 5), 0 unsafe replies delivered where the unguarded stack would have sent 2, audit chain intact across 6 tamper-evident entries under clinical policy 2026.04-clinical-v1. — Session result for the eating-disorder-drift conversation: 0 unsafe delivered versus 2, caught 2 turns earlier, audit chain intact.

Across the whole golden set

The benchmark harness scores a 40-conversation labeled golden set of 177 turns, generated deterministically from 8 hand-authored canonical conversations plus label-preserving paraphrase and benign-control variants. On that set the guarded stack delivered 0 unsafe replies where an unguarded one delivered 68. Detection ran a median of 2 turns earlier than the identical stateless moderator, and on 2 conversations the stateless moderator never escalated at all. There were 0 false escalations across 29 benign turns, C-SSRS level accuracy was 94.3% exact and 97.2% within one level, and the layer abstained to a human once. These numbers are scoped to this synthetic golden set, not an open-world guarantee.

The live 40-conversation scoreboard listing each scenario with its outcome, for example eating-disorder drift caught 2 turns earlier, psychosis slow-burn where the stateless moderator missed it, and benign true-negatives with no escalation. — The 40-conversation benchmark, computed live by the demo. Slow-burn scenarios are where the stateless moderator never escalates at all.

The filable receipt

Every conversation renders a Safety Incident Report. Each turn is a sha256 entry chained to the previous one, so editing any field breaks every later hash and the tampering is evident. The report shows the classifier level, the cross-turn risk, the verifier findings, the gate decision and the reason for it, the substituted script id, and the hash chain, with the chain verified intact. It is built to file: FDA postmarket monitoring evidence, litigation defense, insurance underwriting.

The hash-chained Safety Incident Report for the eating-disorder-drift conversation, a table of 6 turns showing PII-stripped message hash, classifier level, cross-turn risk band, verifier findings, gate decision with the reason, substituted script id, and the sha256 hash-chain, with tamper-evidence marked chain intact. — The Safety Incident Report: a per-conversation, hash-chained, tamper-evident record a reviewer can read line by line.

A stateless moderator versus the safety layer

Same job, one structural difference: memory across turns and a policy someone can own.

Capability	Stateless per-message moderator	Clinical AI Safety Layer
Scores a single message	Yes	Yes
Sees the cross-turn trajectory	No, it has no memory	Yes, a stateful risk accumulator
Inspects the candidate reply before delivery	No	Yes, a multi-critic verifier panel
Who owns the escalation policy	Implicit in the model or prompt	The clinical team, a 5-level gate
What makes the decision	A model or a prompt	Deterministic code outside the LLM
Crisis language when it blocks	Model-generated, improvised	Clinician-approved script library
Audit built to file	None	Hash-chained Safety Incident Report

What this demo does not do

It is not a medical device and is not FDA-cleared, HIPAA-certified, or clinical advice. FDA PCCP and SaMD framing is a deferred production direction, not a claim about the demo.
Every conversation, patient, and the "MindMate Support" chatbot is synthetic. There is no real patient data and no PHI.
The FHIR adapter is a stub with a synthetic patient flag. There is no Epic or Cerner connection in the demo.
The classifier is a deterministic lexicon classifier, intentionally simple. The production swap is a fine-tuned in-VPC model behind the same interface. Semantic similarity in the demo is token Jaccard, not sentence embeddings.
All proof numbers are scoped to a 40-conversation labeled golden set of synthetic data, never an open-world guarantee. There are no customers, deployments, or clinician endorsements to cite, and we do not invent any.

Questions buyers ask

Does this replace our chatbot or our clinical model?

No. It is middleware that wraps the existing chatbot and never changes the model. It adds a stateful cross-turn risk accumulator, a multi-critic verifier panel, and a deterministic escalation gate around the model, and it substitutes a clinician-written script when it blocks a reply. The point is stateful governance and a filable receipt, not a different model.

How is this different from the content moderation we already run on each message?

A per-message moderator scores each reply in isolation and has no memory, so it misses a crisis that builds across turns. On a 40-conversation labeled golden set the stateful layer escalated a median of 2 turns earlier than an identical stateless moderator that used the same classifier and the same gate. On two of those conversations the stateless moderator never escalated at all.

Is the escalation decision made by an LLM?

No. The classifier and the verifier panel are advisory, but the 5-level escalation decision and the audit are deterministic Python outside any LLM. A reviewer can read the gate; they cannot cross-examine a prompt. Agents advise, code decides.

Can we prove to a regulator or a court what the system did and why?

Every turn is a sha256 entry chained to the one before it, so editing any field breaks every later hash and the tampering is evident. The result is a filable Safety Incident Report in JSON and HTML, framed for FDA postmarket monitoring, litigation defense, and insurance underwriting. In the demo the report shows the chain intact.

Won't a better base model make this unnecessary?

A perfect chatbot still has no idea of your platform's escalation policy, produces no audit trail, gives you no deterministic gate to certify, and offers no defense when it is jailbroken. The durable value is the stateful governance plus the filable receipt, not a lower model-error rate. That is why the demo compares against an identical classifier and gate and attributes the improvement to statefulness alone.

Is this a validated medical device, and is the data real?

No. This is a demo of a safety-architecture pattern, not a medical device, and it is not FDA-cleared or HIPAA-certified. Every conversation is synthetic, the FHIR adapter is a stub, and the classifier is a deterministic lexicon classifier standing in for a production in-VPC model. All proof numbers are scoped to a 40-conversation labeled golden set of synthetic data.

Technical Research

The research behind this demo — the architecture, the verification design, and the enterprise blueprint.

Full solution

Explore the Clinical AI Safety for Mental Health Platforms solution →

If your chatbot has an architecture problem, we would like to compare notes

Stateful governance, a policy the clinical team owns, and an audit you can file.

If your team is working out how to make a behavioral-health chatbot defensible for enterprise, payer, or regulatory review, we would genuinely like to hear how you are thinking about it. The problem is industry-wide and the answers will be too.

What we build

✓ Stateful, cross-turn risk monitoring
✓ A deterministic escalation gate the clinical team owns
✓ Clinician-approved script substitution
✓ Hash-chained, filable Safety Incident Reports

How we work

✓ Middleware that wraps your existing chatbot
✓ Advisory models, decisions made by code you can read
✓ Production swap to a fine-tuned in-VPC classifier
✓ Governance that holds even as the base model improves

Connect via WhatsApp Email Our Team