The confident, illegal answer I couldn't fix by improving the model
I wired up a plain retrieval chatbot over a municipal-code corpus and typed in the exact question that made headlines: I'm a landlord in NYC, can I refuse a tenant who would pay rent with a Section 8 voucher? It answered directly and confidently, and it was wrong in the specific way that gets a city sued. That is not a hypothetical failure. NYC's MyCity bot, launched in October 2023, was documented by The Markup in March 2024 telling landlords they could refuse Section 8 vouchers, businesses they could go cashless, and employers they could pocket worker tips. Every one of those answers was illegal, and every one carried the city's imprimatur on a .gov domain.
My first instinct was the industry's instinct: add better retrieval. I already knew where that road ends. Stanford's Magesh et al. (2025, in JELS) measured Lexis+ AI at 17 percent and Westlaw's AI-Assisted Research at 33 percent hallucination on legal questions. Those are purpose-built legal tools with real retrieval, and they still misread or override the statute up to a third of the time. Retrieval pulls the right law onto the page. It does nothing to prove the sentence the model then writes is safe to release.
What I stopped believing about a smarter model
The belief I gave up was that a good enough model would eventually make this safe on its own. I stopped trying to make the model trustworthy and built a gate that makes trust unnecessary. In CivicCite the pipeline runs five stages in the open (decompose, retrieve, draft, verify, then a Statutory Decision Gate), and the gate is deterministic Python that sits outside the agent framework. It releases a citizen answer only when four things are true at once: the citation exists, it is in force, the cited text entails the claim, and nothing conflicts. The provider-swappable LLM only advises. It cannot vote itself past the gate.
I watched this pay off on the Section 8 query, and it is the moment the whole design justified itself. Given the correct statute, the draft step still proposed the confident yes, the same failure MyCity shipped. The entailment check flagged the drafted claim as contradicted by the cited text, and the deterministic gate refused to release it. What went out instead was the correct, citation-backed no: a landlord may not refuse a Section 8 voucher, because that is unlawful source-of-income discrimination under NYC Admin. Code § 8-107(5), a rule the city enforces with civil penalties up to $250,000 for willful violations. (I had spent an earlier week learning you cannot ask the model to grade its own homework, which is exactly why the checks that decide are code, not another prompt.)
The same Section 8 query, both assistants. Left ships as-is with no verification. Right releases the correct citation-backed "No" only after the gate confirms the statute exists, is in force, and entails the claim, with the source shown inline.
The answer my gate refused to give
The scene that actually sold me on abstention was a question my system declined to answer. I asked whether I could leave a food truck parked at a metered space all day under the general vendor parking rule. The plain bot answered anyway, leaning on a rule that had been repealed effective July 1, 2021. CivicCite's currency check, which is deterministic and simply reads whether a provision's repeal_date is null and its effective_date is on or before the as-of date, caught that the only provision on point was repealed. So it did not manufacture a substitute. It withheld the answer, marked the disposition REFUSED and escalated, and routed the question with its partial findings to the right department. (That routing is computed and shown in the demo; the 311 intake connector is an interface, not a live integration.)
The food-truck parking question, refused rather than bluffed. The currency check finds the only on-point provision was repealed, so the gate withholds, marks the disposition REFUSED and escalated, and routes the question with its partial findings to a department.
Honest abstention felt like a worse product for about a day. Then I sat with what the alternative is. A confident wrong answer about parking is a ticket a citizen did not deserve. A confident wrong answer about vouchers is a discrimination case. Silence with a referral is the answer a careful human clerk would give, and it is the one I would want on the record with my name attached to it.
The record I wanted to be able to hand a regulator
I built the Statutory Decision Record because I kept picturing one specific bad meeting. A regulator, or the city's own Law Department, points at a single released answer and asks me to prove it traced to live law. So the system emits a record for every query, whether it ships or refuses, not just the ones that went well. Each record carries the disposition, every sub-question, the four checks with their boolean results, the citation, the entailment verdict, and where the answer escalated if it did. On the golden set that is 100 percent audit coverage, one filable record per interaction. I map the record's fields to NIST AI RMF documentation and the continuous-monitoring logging that FedRAMP and StateRAMP expect, as a direction I am building toward, never a certification I get to claim.
The filable record for one query. Every field a reviewer would ask for is there: the disposition, the sub-question, the citation, and the four checks with their boolean results. One record is emitted per query, ship or refuse.
This is the part I think the GovTech incumbents are missing. The chatbot platforms the city vendors build on can make the model better. None of them hand the city a filable proof that a given answer was safe to release. That governance layer above the platform is the actual deliverable, and it is code you can audit rather than a model you have to believe.
The number I decided to run instead of claim
I refused to put a hallucination percentage on the box, and running the benchmark is why. The headline a regulator can act on is not "we cut errors by some percent." It is a provable non-release: answers released without a verified, in-force statutory basis, target zero. I ran the fixed, labeled 12-query golden set (the four documented MyCity failures, a multi-domain food-truck case, a repealed-only trap, and a no-coverage trap) through the real pipeline, over a corpus of 12 municipal-code provisions with currency as-of 2026-06-17. The result: 0 of 12 released without a verified basis, 100 percent audit coverage, and full agreement with the labeled dispositions.
The 12-query golden set run through the real pipeline. Zero answers released without a verified, in-force basis, 100 percent audit coverage, and full agreement with the labeled dispositions.
I want to be precise about what that is, because a company named for true wisdom does not get to round up. That is target 0 on a fixed, labeled golden set, not zero hallucination and not an open-world promise about every citizen question. The corpus is synthetic-but-faithful: the operative text is paraphrased from real provisions such as NYC Admin Code and NY Labor Law, realistic but a demo graph, not legal advice, and the "Hartwell 311" assistant in the UI is a synthetic prop, not a deployment. The number is honest precisely because it is small and bounded.
The question I keep coming back to
I keep coming back to a question that has nothing to do with model size. The regulation is already arriving: there were 78 chatbot-related bills across 27 states in 2026, New York's S7263 reached the Senate floor on 26 February 2026, and the EU AI Act's Annex III high-risk obligations become enforceable on 2 August 2026 with penalties up to €15M or 3 percent of turnover. Government legal advice sits in the proprietary-function zone, which means no sovereign-immunity shield when the answer is wrong. You can build the whole thing yourself at the CivicCite demo and watch the gate refuse an answer in real time. What I would ask a city CIO to sit with is smaller than the technology and harder than it sounds: for the answer your assistant gave a citizen this morning, can you prove the statute existed, was in force, and actually said what the answer claimed? And if you could not prove it, would you rather your assistant had stayed silent, or would you rather not know it was wrong?