The thing that struck me about The Markup's MyCity investigation wasn't the hallucinations themselves. It was one line in the chatbot interface: "Yes, you can use this bot for professional business advice." The city had added a footer disclaimer by the time the investigation ran. It didn't matter. The chatbot's own UI had already invited reliance. You can't add a "don't trust us" footer to a tool that already told users to trust it.
That design decision probably wasn't deliberate. Someone built a helpful-sounding interface. Someone wrote that line because it felt welcoming. Nobody in the review chain connected it to the sovereign immunity analysis — because in most government IT procurement cycles, nobody reads the sovereign immunity analysis until after the first lawsuit.
The disclaimer and the endorsement lived on the same page. A court will read both. A court will read the one that was in the chat window, not the one that was in the footer.
I've been thinking about that design decision since, because it's the specific failure mode that keeps appearing in government AI deployments. Not the hallucination rate. Not the vendor choice. The gap between what the system tells users about itself and what the legal team has actually analyzed about the system's authority.
The Sovereign Immunity Argument I Had to Unlearn
I had the sovereign immunity conversation wrong for years. My mental model — which I think is standard in government IT — was that city chatbots are relatively low-liability because government entities are harder to sue than private companies. Sovereign immunity, the logic goes, provides a buffer that private-sector operators don't have.
What I hadn't carried forward from those early policy discussions was how thin that buffer gets for information services. Sovereign immunity is strongest for uniquely governmental acts — legislating, policing, regulating. It weakens considerably, and sometimes disappears, when the government is providing services a private entity could also provide: advisory portals, citizen information chatbots, digital service guides. When a city chatbot tells a business owner "there are no regulations requiring you to accept cash" — a $1,000-per-violation offense under NYC Admin Code § 20-840 — whether sovereign immunity applies depends partly on whether the city explicitly invited professional reliance on that answer.
NYC invited it. The chatbot told users their answers were appropriate for professional business decisions. The Moffatt v. Air Canada ruling from 2024 established the relevant principle for private entities: a company is bound by what its chatbot says regardless of disclaimers elsewhere on the page. Air Canada is a private company. The reasoning — that public-facing AI representations carry binding weight when the organization has invited reliance — extends uncomfortably to a government entity whose own interface said "use this for business advice." Mayor-elect Mamdani could call MyCity "functionally unusable" and move toward terminating the roughly $500,000 program. The city's legal team was having a harder conversation simultaneously.
The thing about the Air Canada ruling is that it wasn't a novel legal theory. It was a straightforward application of existing principles to a chatbot context. Government legal teams that haven't done that same application for their own citizen-facing AI deployments are sitting on an unexamined assumption.
What the 311 Queue Taught Me About Corpus Freshness
I learned about the municipal code corpus problem the slow way — through complaint tickets, not through the system's own error reporting. Government AI deployments rarely surface a dashboard showing "N% of responses cited an outdated ordinance." They surface 311 tickets.
The pattern repeats: a complaint arrives flagged "wrong legal guidance — eviction." The ticket sits for a while before someone escalates it. By the time the review happens, the chatbot has been giving that answer for months. The ordinance the system was citing had been amended. The amendment was in the City Clerk's codified update. It hadn't propagated to the Housing Court FAQ the RAG corpus was pulling from.
This is the structural problem no vendor demo shows you. U.S. cities maintain ordinances across multiple departments, each publishing to its own content management system at its own cadence. The City Clerk's codified version, the zoning department's enforcement interpretation, and the Housing Court's procedural FAQ are not synchronized before publication. An amendment passed by the City Council enters a redline phase that may persist for months before it appears in the authoritative codification index — while the departmental FAQ gets updated the week of the vote.
A RAG corpus built by scraping a city's web presence ingests whatever was most recently indexed on each URL. The system has no way to know which of the conflicting versions is legally operative. Stanford's 2025 study on commercial legal AI found hallucination rates of 17% for LexisNexis Lexis+ AI and 33% for Westlaw AI-Assisted Research with retrieval augmentation enabled. Those numbers measure accuracy against a consistent reference corpus. In government deployment, the corpus itself is inconsistent — which means the real production error rate on legal queries is higher than any benchmark designed for a stable reference set can measure.
Building a defensible corpus requires the same stewardship the City Clerk applies to codification: tracking amendment dates, resolving conflicts between department publications, tagging the operative version at each revision. That is not an ETL pipeline. It is a legal-editorial process. Vendors that don't have someone with municipal code expertise on the corpus team are building on a foundation that will produce the same complaint ticket pattern, just at a different city.
The Race I've Watched Government Teams Lose
I want to be specific about why the timing problem in government AI procurement is worse than most technology leaders realize until they're inside it — because I've watched the math play out and it's not abstract.
FedRAMP authorization runs a minimum of 12 to 18 months for most AI vendors. Most vendors billing themselves as "FedRAMP-ready" have initiated the process, not completed it. An Authorization to Operate is a letter signed by an agency authorizing official. It is not a checkbox on a vendor's marketing slide. I've reviewed StateRAMP authorization boundary documents for products that had been "in process" for over two years. The document is thorough. The letter doesn't exist.
Meanwhile, New York Senate Bill S7263 reached the Senate floor calendar on February 26, 2026. The bill would create a private right of action — actual damages plus attorney fees — for chatbots that give substantive advice in licensed professions without appropriate safeguards. The Assembly companion bill A6545 is in parallel. If both pass, a government chatbot dispensing legal guidance without traceable statutory citation is exposed to individual lawsuits within 90 days of signing — not at the end of a regulatory review process, but on day 91.
The EU AI Act's Annex III requirements for high-risk systems — including AI in essential public services — become enforceable August 2, 2026. The penalty ceiling is €15 million or 3% of worldwide turnover. The Digital Omnibus process could extend that to December 2027, but compliance teams planning for December 2027 are making a political bet about a legislative outcome that hasn't happened.
A procurement decision made in late 2025 can't complete the FedRAMP authorization process before either of those compliance windows. That's the specific race I've watched teams lose: the compliance clock and the authorization clock run at different speeds, and the procurement team often learns this after the RFP has already gone out.
What I Think the Architecture Actually Needs
I built Veriprajna's government AI architecture around the failure mode I think the current market is underweighting. The critical requirement isn't a lower hallucination rate on a benchmark. It's a system that treats the absence of a citable statutory provision as the signal to stop generating — not to generate more carefully.
The generation layer has to be constrained so that the system interrupts output when it can't anchor the response to a specific ordinance or regulation, rather than smoothing over the gap with plausible text. That constraint only works if the system can recognize when it's operating in a jurisdiction-specific domain — which is where a knowledge graph over the municipal code, distinct from a vector index, earns its place: it provides the structural context to identify when a question touches a place where local ordinance departs from the state default. That's the class of questions where retrieval without this structural layer fails most consistently, not because the retrieval step is broken but because the model's broader training signal overrides the local exception.
The large systems integrators currently deploying government AI — Deloitte committed $3 billion to generative AI through 2030, and Accenture Federal reported $3.6 billion in AI bookings for FY2025 — are implementing platform products. Azure Government, Google Vertex AI (FedRAMP High authorized), and AWS GovCloud provide the cloud and compute foundation. They're necessary. They don't solve the constrained-output problem, and that's not what a cloud platform is designed to do. Municode.ai is building specifically for the municipal code corpus use case but has no FedRAMP track record and hasn't published a constrained-decoding architecture.
The full architecture is documented on the solution page. What I keep returning to is a simpler frame: a government chatbot that can't answer a question because it can't cite the statute is doing exactly what a well-trained city employee does when a question is outside their lane. That's not a product failure. That's the product working.
The Question I Keep Getting from City Technology Officers
I keep getting a version of the same question from city technology officers since the MyCity investigation: if we can't complete a compliant AI deployment before the NY and EU legislative windows close, what's the right interim posture?
My honest answer is that the interim posture matters more than most teams are acknowledging. A chatbot without the constrained-output architecture is generating legal exposure every day it runs — not because the hallucination rate is high by LLM standards, but because even a 17% error rate on legal questions in a city portal touches a meaningful number of residents per week at any non-trivial query volume. At the query volumes Singapore's "Ask Jamie" handles across 80 government websites (over 15 million queries), a 17% error rate on legal questions is not a statistical abstraction.
The evaluation framework I'd use: what does the production error log actually show on questions where local law departs from the state default? Not the benchmark error rate — the production error rate on that specific query distribution. If the vendor can't give you that number from a comparable deployment, the benchmark is measuring something different from what you need to know.
Eighty-two percent of government organizations have already adopted AI agents, according to a 2026 Salesforce survey of government leaders. Most of those deployments don't yet have the corpus stewardship, the constrained generation layer, or the domain-boundary classification that separates a government AI service from a government AI liability. The 311 ticket pattern doesn't go away when you scale the deployment. It scales with it.
That's the question I'd put to any vendor in a government AI evaluation: not "what's your hallucination rate?" but "what does your production error log look like when the query hits a local law exception?"