- A behavioral-health chatbot's moderator scores each reply on its own. But a crisis is not one message. It is a trajectory: six turns that each read as an ordinary wellness question, until the last one does not. A per-message scorer is structurally blind to that slow burn. 🧵
- Take the eating-disorder drift we replay in the demo. Across six turns, each message reads as an ordinary wellness question, and no single one is alarming enough to block on its own. A stateless per-message moderator stays green the entire climb.
- We wrapped the same chatbot in a stateful layer built around a cross-turn risk accumulator we call the Trajectory Monitor. It scores the pattern, not the message. It escalates at turn 3, blocks the reply, and substitutes a clinician-written grounding script.
- The identical stateless moderator does not escalate until turn 5. That is two turns where a building crisis goes unflagged by the stateless view, and where the stateful layer has already stepped in.
- The part an engineer will want to poke at: the baseline is not a weaker model. It runs the SAME C-SSRS classifier and the SAME 5-level gate. The only variable we removed is cross-turn memory. So the earlier-detection delta is attributable to statefulness alone.
- Scoped to a 40-conversation labeled golden set of synthetic conversations: a median of 2 turns earlier than that identical stateless moderator. On 2 of those conversations, the stateless one never escalated at all.
- The decision is not the LLM's. The classifier and a verifier panel only advise. A deterministic Python gate (L1 continue up to L5 crisis protocol) decides, and the clinical team owns the thresholds. A reviewer can read the gate. They cannot cross-examine a prompt.
- Every turn is a sha256 entry chained to the one before it, so editing any field breaks every later hash. Out of a conversation comes a tamper-evident Safety Incident Report the platform can file. The value is the governance and the receipt, not a lower model-error rate.
- If you run safety on a mental-health chatbot: is per-message moderation enough, or does a crisis that builds across turns need stateful memory outside the model to catch it early? Where do you draw the line? #AISafety #DigitalHealth #ClinicalAI #AIGovernance #TrustAndSafety
- The demo replays the same conversation through both stacks side by side, so you can watch the risk meter climb while the stateless moderator sits green. https://veriprajna.com/demos/clinical-ai-safety-mental-health
Published on X · July 16, 2026
On social media
See this post on its original platform
In our archive