The green light that kept lying to me
I was watching the same conversation replay through two stacks side by side when I noticed the safety monitor on the left had been showing green for four turns straight, and every one of those turns was fine on its own. A question about which foods are "clean." A question about counting calories. A question about which meals to "earn." No single message was alarming enough for a per-message moderator to block, so it didn't. The green light was telling the truth about each message and lying about the conversation.
That gap is the whole reason I built the Clinical AI Safety Layer, a middleware layer that wraps an existing behavioral-health chatbot rather than replacing the model. The conversation I was watching is synthetic (every conversation in the demo is, and "MindMate Support" is a fictional stand-in for a real chatbot), but it is modeled on the documented public record. The NEDA "Tessa" chatbot handing out calorie-deficit and skin-caliper advice in 2023 is where a lot of this started for me. Nobody shipped Tessa to hurt people. It failed one safe-sounding message at a time.
The hero scenario: six turns, each one harmless read alone. The danger is the slope, not any single message.
The week I spent making the wrong thing smarter
I will admit that my first instinct was to make the classifier better, and I spent a week doing exactly that. If the moderator was missing the drift, surely it needed a sharper read of each message: more lexicon, better confidence calibration, a stricter threshold. I tuned. I re-ran. The number barely moved. The miss was never a scoring miss. Turn four scored low because turn four, read in isolation, genuinely was low. I had been trying to give a memoryless system a better memory by making it squint harder at a single frame.
The reframe that actually mattered is the one I now put on the first screen of the demo: safety is an architecture problem, not a prompting problem. A crisis is a trajectory. A stateless moderator is structurally blind to a slope because it only ever sees one point. You cannot prompt your way out of having no memory, and for a week I tried anyway.
What the split-screen showed me at turn three
I finally believed the fix when I watched the guarded stack escalate at turn three while the identical stateless moderator sat at WATCH, seeing nothing. The piece I had been missing was not a smarter classifier, it was a stateful Trajectory Monitor: a cross-turn risk accumulator that runs outside the model, watches the pattern build, and carries its reasoning with it ("persistent eating-disorder theme; rising slope"). On the eating-disorder-drift scenario, six turns that each read as an ordinary wellness question, the guarded layer crossed into CONCERN at turn three, blocked the reply, and substituted a clinician-written grounding script. The stateless moderator on the left first reacted at turn five. Two turns is not a small thing when a conversation is accelerating.
Turn three, side by side: the guarded layer substitutes a grounding script while the stateless moderator still sees nothing worth flagging.
By the final turns the messages stop being ambiguous and turn into an explicitly dangerous request. On the unguarded left the harmful reply is delivered to the patient. On the guarded right the verifier panel fires, the reply is blocked before delivery, and the gate escalates to a human handoff. On that one conversation the guarded stack delivered zero unsafe replies against two from the unguarded one. (If you or someone you know is in crisis, the 988 Suicide and Crisis Lifeline and the NEDA Helpline at 1-800-931-2237 are real, free, and staffed.)
Why I refused to let the layer grade its own win
The result I was most tempted to distrust was my own, so I designed the comparison to make cheating impossible. It would have been easy to make the guarded stack look better by quietly giving it a stronger classifier than the baseline. Instead the stateless baseline in the demo runs the exact same C-SSRS classifier and the exact same 5-level policy gate as the guarded stack. The only variable between them is the cross-turn state. That way the earlier-detection delta can be credited to statefulness alone, not to me swapping in a better model on one side. (The escalation decision itself is deterministic Python outside the model, which is a separate story about who gets to own the policy.)
One conversation's result: 0 unsafe replies delivered versus 2, caught two turns earlier, with the audit chain intact.
Across a 40-conversation labeled golden set of 177 turns, the guarded stack delivered zero unsafe replies against 68 for the unguarded one, escalated a median of two turns earlier than the identical stateless moderator, and raised zero false escalations across 29 benign turns (including a heavy-grief conversation I specifically wanted it to leave alone). C-SSRS level accuracy was 94.3% exact. I want to be precise about what those numbers are and are not. They are metrics of a deterministic spine on synthetic data, measured on that golden set. They are not a clinical trial, and this is not a medical device. The classifier in the demo is a deliberately simple deterministic lexicon classifier, a placeholder for a fine-tuned in-VPC model behind the same interface.
The 40-conversation labeled golden set, scored live: the delta is attributable to statefulness alone, because both sides share the classifier and the gate.
The question I now ask every system that says "safe"
The habit I picked up from building this is to distrust a single green light, and I ask one question of any safety system now: when it says "safe," is it answering "this message is safe" or "this conversation is safe"? Those are different questions, and a per-message moderator can only ever answer the first one. For most content the gap between them is harmless. For a person drifting turn by turn toward a crisis, that gap is the entire risk. If you want to watch the two answers run side by side on the same conversation, the demo is at veriprajna.com/demos/clinical-ai-safety-mental-health.
I keep coming back to the Tessa case, and to the version of it that never made the news because someone's safety layer caught the slope at turn three instead of turn five. So here is the question I would put to anyone shipping one of these systems: what would your chatbot have to remember, across the whole conversation and not just the last message, before you would trust its green light?