- NYC's MyCity bot told landlords they could refuse Section 8 vouchers. Illegal. The Markup documented it in 2024. The uncomfortable part: point our own model at that same question and it drafts the same illegal yes. It just never ships it 🧵
- Adding retrieval does not fix this. Stanford's Magesh et al. (2025) measured Lexis+ AI hallucinating at 17% and Westlaw AI-Assisted Research at 33% on legal queries. Retrieval pulls the right statute. Generation still misreads or overrides it.
- So we stopped trying to make the model trustworthy and put a deterministic gate outside it. An answer releases only if its citation exists, is in force, is entailed by the cited text, and is conflict-free. Fail any one check and it abstains and escalates.
- Watch the Section 8 query run. The model drafts yes and cites NYC Admin Code §8-107(5). Then the entailment check reads that text and returns contradicted: §8-107(5) bars source-of-income discrimination. The cited statute says the opposite of the draft.
- Because the verdict is contradicted, the gate blocks the drafted yes and releases the citation-backed No instead. Verified, entailed, in force. The model advised. The gate decided. An LLM cannot vote itself past it.
- The gate is plain Python, outside the agent framework. The currency and conflict checks are deterministic too: in force means repeal_date is null and effective_date is on or before the as-of date. No model gets a vote on whether an answer is safe to release.
- Every query, released or refused, emits a Statutory Decision Record: the claim, the citation, each check's verdict, the disposition. Ship or abstain, there is a filable artifact. That is the thing a regulator can actually act on.
- The headline is not a hallucination rate. On a fixed 12-query golden set, the target is 0 answers released without a verified, in-force statutory basis, with a record for all 12. Provable non-release, not a promise the model is never wrong.
- To be exact: the corpus is a synthetic-but-faithful demo graph, the LLM is provider-swappable and advisory only, and this is a demo, not a certified system or legal advice. The mechanism is what we are showing, on a labeled set you can rerun.
- Govtech and AI folks: if a model can draft a confident wrong answer and cite a real statute for it, does the entailment check belong inside the agent loop, or in deterministic code the model cannot overrule? Where would you draw the line? #AIGovernance #GovTech #LegalTech
- We built this to show trust can live in auditable code, not in a model you have to believe. If your team is wrestling with releasing government AI answers, we would like to hear how you think about it. Walkthrough: https://veriprajna.com/demos/government-municipal-ai
Published on X · July 23, 2026
On social media
See this post on its original platform
In our archive