The statute was real. The answer was still illegal.

The first answer I ever watched a government chatbot give confidently and wrongly was about a Section 8 voucher.

A landlord asks, in plain words, whether he can refuse a tenant who would pay rent with a housing voucher. The correct answer is no. Refusing is source-of-income discrimination under NYC Admin. Code § 8-107(5), and the city's Human Rights Commission can levy civil penalties up to $250,000 for a willful violation. This is not a trick question. It is settled law. And in October 2023, New York City's own MyCity bot, running on Azure AI, told business owners and landlords the opposite. The Markup documented it in March 2024: the bot said landlords could turn away voucher holders, that stores could go cashless, that employers could pocket a slice of their workers' tips. Every one of those answers was illegal, and every one of them carried the city's seal on a .gov domain.

I did not build CivicCite because a model hallucinated. I built it because of the answers that were almost right, and I want to tell you about the one that changed how I think about the whole problem. If you want to see the thing itself, it lives here: veriprajna.com/demos/government-municipal-ai. The corpus is a synthetic-but-faithful demo graph, not legal advice. But the mechanism is real, and the mechanism is the part worth arguing about.

The moment that stuck with me was not the bot inventing a law. It was watching my own pipeline pull the correct statute and still draft the wrong answer.

The week I spent adding retrieval to a problem retrieval cannot touch

I started this the way almost everyone in GovTech starts it, convinced that the fix for a lying chatbot was better retrieval. Give the model the actual municipal code. Ground every answer in a real provision. Retrieval-augmented generation, the standard answer to the standard fear, which is that the model makes things up. It is a clean story, it demos beautifully, and I believed it for longer than I should have.

Then I read the Stanford numbers and they ruined the story. Magesh and colleagues, in a 2025 study published through JELS, measured the two purpose-built legal research tools that do exactly this. Lexis+ AI hallucinated on 17% of queries. Westlaw's AI-Assisted Research hallucinated on 33%. These are not toy chatbots. These are systems that retrieve the statute first and generate second, built by companies whose entire business is being right about the law. Roughly one in three answers from the second tool was still wrong. Retrieval improved the draft. It did not make the answer safe to release.

I want to be honest about how that felt, because I had just spent a week building the same architecture and feeling clever about it. My pipeline decomposed the citizen's question into atomic legal sub-questions, retrieved candidate provisions from a municipal-code graph, and constrained the model to draft only from what it retrieved. On the Section 8 question it did everything right up to the last step. It found § 8-107(5). The correct statute. It pulled the exact text about lawful source of income. And then it drafted a sentence that read, in effect, yes, you can refuse the voucher.

The right law. The wrong answer. Sitting on top of each other in the same draft.

Retrieval hands the model the correct statute. It does nothing to stop the model from reading that statute backwards.

That was the week the problem changed shape for me. I had been treating hallucination as the enemy, and hallucination is real, but it is the failure you can imagine catching. The failure that actually shipped the MyCity answers is subtler and worse: the right provision cited, the wrong conclusion stated. No fabrication anywhere in the sentence. Nothing for a "did you make this up" filter to catch, because nothing was made up. Just a confident misreading of a law that was sitting right there. That failure mode is invisible to retrieval, because retrieval only checks that the statute is present, never that the sentence read it the right way round.

What is worse, an invented law or a real one read wrong?

I kept coming back to that question, and my answer kept getting more certain. The invented law is the safer failure.

Think about what a citizen does with each. An obviously fabricated statute reads strange, cites a code section that does not resolve, feels off. A real statute read backwards reads perfect. It has a genuine citation. The section exists. A landlord reads "yes, you can refuse the voucher," sees a real code number attached, and acts on it. Now there is an illegal refusal, a tenant with a discrimination claim, and a paper trail leading back to a government answer. The correctness of the citation is exactly what makes the wrong conclusion dangerous. It is the credential the bad advice rides in on.

So the check I cared most about was never "is this citation real." It was "does this citation actually support this sentence." In the pipeline that check is entailment: given the drafted claim and the exact text of the cited provision, does the text entail the claim, contradict it, or neither. On the Section 8 draft, the model's "yes" is caught as contradicted by § 8-107(5), because the provision plainly says the opposite. The draft dies there. What survives, and what the gate eventually releases, is the corrected claim: a landlord may not refuse, and here is the statute that says so.

The thing I insisted on, and the reason that screen exists, is that entailment is not a black box you have to believe. You can open the Verify stage and read its raw input and output: the drafted claim, the statutory text it was checked against, the label, and the reason in the provision's own words. A verdict you cannot inspect is not verification. It is a second opinion with better production values. I had already learned, the expensive way, that asking one model to referee another model just gives you two models agreeing, which is a weaker thing than it looks.

The move that finally worked was to stop trying to make the model trustworthy

I remember the exact reframe, because it felt like giving up and turned out to be the whole design. I stopped trying to make the model trustworthy and started making its trustworthiness irrelevant.

The shift is this. The language model in CivicCite is an advisor. It decomposes the question, it drafts a candidate answer, it offers an entailment opinion. It never gets to release anything. Sitting outside the agent framework, in plain deterministic Python, is a thing I call the Statutory Decision Gate, and it releases a sub-answer only when four conditions hold at once: the citation exists, the provision is in force, the cited text entails the claim, and nothing conflicts with it. Miss any one and the answer does not ship. Two of those checks are pure arithmetic and logic. In force is a date comparison: the provision has no repeal date and its effective date is on or before the as-of date. Conflict is a graph read. There is no prompt, no temperature, no persuading it.

I say two sentences constantly now. One is the agents advise, the gate decides. The other is that an LLM cannot vote itself past the gate, no matter how confident its draft. This matters more than it sounds, because the seductive failure in this whole field is to let the model grade its own work and call the grade "verification." A model that drafts the answer cannot be the thing that certifies the answer is safe. The gate is deliberately dumber than the model and deliberately outside it, and that is the entire point. The intelligence proposes. The code disposes.

The model is a good writer and a bad judge. So I let it write, and I never let it judge.

The demo is honest about its edges, and I will be too. The corpus is a synthetic-but-faithful graph of municipal code, paraphrased from real provisions, not a source of record. The 311 escalation routing is computed and shown, but the connector to a real case system is stubbed. The constrained drafting uses an allowlist validator and one re-ask rather than production token-level decoding. What is real is the decision logic, and the decision logic is the product.

Why I trust silence more than I trust a good answer

I did not expect to be proud of the demo refusing to answer, and now it is my favorite thing it does. The clearest example is a food truck.

A vendor asks whether he can leave his truck parked at a metered space all day under the general vendor parking rule. The plain assistant answers cheerfully and wrongly, citing a rule as if it were live. CivicCite drafts a candidate too, and the candidate cites a parking provision that has been repealed. Then the currency check runs. The provision has a repeal date. In force fails. And instead of falling back to some adjacent rule and bluffing, the gate withholds the answer, marks the question as outside verified coverage, and routes it to a live department with the partial findings attached.

A wrong answer on a government domain is not an embarrassment. It is a liability with the city's name on it.

Look at what that screen refuses to do. It does not synthesize a plausible answer from a dead statute. It says, in plain language, that the only provision addressing this has been repealed and nothing currently in force governs the question, and it hands the citizen off rather than guess. Honest abstention beats a confident wrong answer, in government especially. And in government the stakes are not reputational. Government legal advice sits in the proprietary-function zone, which means there is no sovereign-immunity shield to hide behind when the answer turns out to be wrong.

This is not a hypothetical worry anymore, which is part of why I built the thing now instead of later. There were 78 chatbot-related bills across 27 states in 2026. New York's S7263 reached the Senate floor on 26 February 2026. The EU AI Act's Annex III high-risk obligations become enforceable on 2 August 2026, with penalties up to €15M or 3% of global turnover. The regulatory question is shifting from "is your AI helpful" to "can you prove your AI was allowed to say that." Silence you can defend. A confident wrong answer you cannot.

The number a regulator can actually act on

I used to think the headline metric for a system like this was a hallucination rate, and I now think that instinct is exactly backwards. A percentage is the wrong thing to hand a regulator.

Imagine you are the city's Law Department, the office that owns the liability when an answer goes wrong. "Our chatbot hallucinates only 4% of the time" is not reassuring. It is an admission that four in a hundred citizens get an answer with the city's authority behind it and nothing standing behind the answer. The number that means something is not about how often the model is wrong. It is about whether anything unverified was ever allowed out the door. So the metric I built the benchmark around is this: on a fixed, labeled 12-query golden set, run through the real pipeline, the count of answers released without a verified, in-force statutory basis. The target is zero out of twelve.

A hallucination rate tells a regulator how often you failed. It cannot tell them you caught it.

I want to be precise about what that number is and is not, because overclaiming it would betray the whole premise. It is a result on a fixed labeled set, not an open-world guarantee and not "zero hallucination," a phrase I think no honest person should sell. The model still drafts imperfect claims. The point is that the unverified ones are held, not that they are never drafted. Alongside that number sit two more: 100% audit-record coverage, meaning one filable record per query whether it ships or refuses, and disposition agreement with the labeled ground truth. The plain-RAG baseline, the MyCity architecture with no gate, would have shipped the documented illegal answers on this same set. That comparison is context, not the headline.

The record is the part I would defend hardest.

Every query, released or refused, produces a Statutory Decision Record: the disposition, the citation, all four check results, the decision, as structured data you can file and rerun. It names NIST AI RMF logging and FedRAMP and StateRAMP continuous monitoring as the standards it is built toward. I am careful with that sentence. Built toward is a direction, not a certification, and I will not claim CivicCite is certified against any of them. But an auditor does not want a promise that the model is smart. An auditor wants a record, per interaction, that shows exactly why an answer was allowed to exist, in a form that survives someone trying to break it. Reproducibility is what turns a decision into evidence, and evidence is what a regulator can act on.

And if you would rather watch it than read me describe it, here is the whole demo running end to end.

The question I am left with

I keep returning to something that surprised me about my own reaction. The first time I watched CivicCite refuse a question and route it to a department, some part of me read it as the system failing.

It was not failing. It was doing the one thing the confident bots cannot do, which is know the edge of what it can prove and stop there. I had spent so long optimizing for a good answer that I had quietly assumed a good answer was the goal. It is not. In government, the goal is a defensible answer, and the distance between those two words is the entire reason this product exists. A defensible answer sometimes looks like a released citation. Just as often it looks like a routed question and a filed record.

So the question I would leave with anyone building AI for a government, or buying it, is not "how accurate is your model." Better models will keep getting better at writing fluent, wrong-in-context answers, because fluency was never the missing piece. The question I ask before anything leaves the gate is narrower and harder. Can you prove, right now, that this exact answer traces to a statute that exists and is currently in force, and would that proof survive a court that wanted it to fail? If the answer is no, it does not matter how good the model is. The system should stay silent. You can watch it decide, either way, here: veriprajna.com/demos/government-municipal-ai.

Trust does not belong in a model you have to believe. It belongs in code you can audit.