The city added a disclaimer after the investigation. They also kept the part where the chatbot told users, "Yes, you can use this bot for professional business advice." The disclaimer was in the footer. The green checkmark was in the chat window. A disclaimer and an endorsement can't coexist — and the endorsement won.
When The Markup investigated NYC's MyCity chatbot in March 2024, the failures it documented weren't edge cases. The chatbot told landlords they could refuse Section 8 vouchers — illegal under NYC Human Rights Law since 2008. It told business owners they could skip the cashless ban — a $1,000-per-violation offense under NYC Admin Code § 20-840. It told employers they could take a cut of workers' tips — wage theft under FLSA and New York Labor Law § 196-d. The chatbot had been running since October 2023 on Microsoft Azure AI, trained on 2,000+ city web pages, operating at roughly $500,000 in public funds. Each answer was settled law given wrong. Each answer carried the weight of a .gov domain.
The $500,000 NYC spent on MyCity wasn't the problem. It was an architecture that makes accurate government AI look indistinguishable from an accurate-looking one.
Incoming Mayor Mamdani called the tool "functionally unusable" and moved toward termination. The lesson most cities are drawing from that: better training data. That's the wrong lesson.
Why the Retrieval Step Isn't Enough
The default response to a hallucinating chatbot is to add retrieval — pull the relevant statute before generating the answer. Every government AI vendor pitch currently includes a RAG architecture slide for exactly this reason. In theory, the logic holds: retrieve the correct provision, ground the output, prevent the fabrication.
Stanford's 2025 study on commercial legal AI tested that theory against tools with retrieval enabled. LexisNexis Lexis+ AI — the best result in the study — hallucinates 17% of the time. Westlaw AI-Assisted Research hallucinates 33% of the time. GPT-4 general-purpose: 43%. These aren't research-lab benchmarks; they're the production tools currently being evaluated in government citizen-service procurements.
A 17% hallucination rate may be acceptable when a lawyer cross-checks the output before filing a brief. When a small business owner queries a city portal and acts on the answer before speaking to an attorney, one-in-six failures generates incorrect legal guidance across a statistically significant share of every real-world interaction.
The mechanism in the NYC case was a collision between general contract law — the statistically dominant pattern in any LLM's training data, which implies freedom of contract in tenant relationships — and a local NYC Human Rights Law exception that has been on the books since 2008. The retrieval step pulled relevant material. The model still generated the plausible answer from the broader training signal, because jurisdiction-specific carve-outs are precisely the kind of local exception that gets overridden when the model has absorbed millions of examples of the general rule. RAG retrieves; it doesn't constrain.
The Liability Structure Legal Teams Don't Read Until They Have To
Government procurement teams know sovereign immunity. What doesn't get surfaced in a standard IT RFP review is the proprietary-function exception embedded in that doctrine.
Sovereign immunity is strongest when the government is performing uniquely governmental acts — legislating, policing, regulating. It weakens considerably — and sometimes disappears — when the government provides services that a private entity could also provide: information portals, advisory tools, digital citizen services. When a city operates a chatbot that explicitly tells users to treat its answers as professional business advice, and those answers turn out to be wrong, the chatbot isn't protected by the same doctrine that shields the city council's legislative decisions.
The Air Canada precedent — Moffatt v. Air Canada, 2024 — established that a company is bound by what its chatbot tells users regardless of disclaimers appearing elsewhere on the page. Air Canada is a private entity. The reasoning that public-facing AI representations carry binding weight extends to government entities that have explicitly invited business reliance on their system's outputs. NYC's own UI told users to treat its chatbot as authoritative for professional decisions. The disclaimer in the footer didn't undo that.
We built our government AI architecture around a specific constraint: every answer must trace to an identified statutory or regulatory provision, or the system routes the query to a human rather than generating a response. The inability to produce an uncited answer isn't a product limitation. It's the liability shield.
The Corpus Problem That Doesn't Show Up in the Demo
Every government AI vendor demos with a clean corpus: structured ordinance data, a clear update cadence, a governance diagram. That's what municipal code data looks like when someone prepares it for a sales presentation.
In production, U.S. cities maintain ordinances across multiple departments, each publishing to its own content management system at its own cadence. The City Clerk's codified version, the zoning department's enforcement interpretation, and the Housing Court's procedural FAQ are not necessarily synchronized before publication. An amendment passed by the City Council enters a redline phase that may persist for months before propagating to the authoritative codification index — while the Housing Court FAQ gets updated the day after the vote. A RAG corpus built by scraping a city's web presence ingests whatever was indexed most recently on each URL.
When the Housing Court FAQ says one thing and the official code says another, the model blends them. The retrieval step doesn't know which version is legally operative — it retrieves what's available. This is why the corpus problem doesn't surface in vendor demos but does surface in 311 CRM tickets flagged "wrong legal guidance — eviction." By the time the complaint arrives, the chatbot has been answering that question the same way for months.
Building a legally defensible corpus requires the same stewardship the City Clerk applies to the codification process: tracking amendment dates, resolving conflicts between department publications, tagging the operative version at each revision. That's not a data-scraping problem. It's a legal-editorial one.
The Compliance Window Is Already Inside the FedRAMP Timeline
By the time most government AI procurements initiated in late 2025 complete the FedRAMP authorization process — a minimum of 12–18 months for most AI vendors — two significant compliance deadlines will already have passed.
New York Senate Bill S7263, sponsored by Senator Kristen Gonzalez, reached the Senate floor calendar on February 26, 2026. The bill prohibits chatbots from providing substantive advice in licensed professions — law, medicine, engineering — and creates a private right of action for actual damages plus attorney fees for willful violations. The Assembly companion bill A6545 is in parallel movement. If both pass, a government chatbot dispensing legal guidance without traceable statutory citation becomes exposed to individual lawsuits, not just regulatory review, within 90 days of signing.
The EU AI Act's requirements for Annex III high-risk systems — which explicitly includes AI used in essential public services — become enforceable August 2, 2026. For government entities interacting with EU-resident constituents, the penalty ceiling is €15 million or 3% of worldwide turnover for non-compliance. The Digital Omnibus process could push that to December 2027, but prudent compliance planning treats August 2026 as binding.
Most AI vendors currently billing themselves as "FedRAMP-ready" have initiated an authorization process, not completed one. An Authorization to Operate is signed by the agency's authorizing official; it is not a vendor marketing claim. Procurement teams evaluating government AI right now are running the compliance clock and the authorization clock simultaneously — and the compliance deadlines don't wait for the ATO to close.
What Actually Prevents the Liability
Building a government AI system that doesn't generate unauthorized legal advice requires one thing that current platform deployments skip: the architecture has to treat "I don't know" as the correct output when the system can't identify the statutory provision the answer depends on.
The generation layer itself has to be constrained — a mechanism that interrupts output when the system can't anchor the response to a specific ordinance or regulation, rather than smoothing over the gap with plausible text. That constraint only works if the system can also recognize when it's operating in a jurisdiction-specific domain: a knowledge graph over the municipal code, distinct from a vector index, provides the structural context to identify when a question touches a place where local ordinance departs from the state default — which is precisely where retrieval without constraints fails most consistently. What that leaves unresolved is the escalation path, and it's the hardest piece to calibrate correctly: a system that escalates 40% of queries doesn't produce cost savings; a system that escalates the specific queries that carry legal risk requires classifying which ones those are before routing, which means the domain-boundary identification problem has to be solved at the architecture level before escalation becomes useful rather than just expensive.
The large systems integrators deploying government AI — Deloitte has committed $3 billion to generative AI through 2030, and Accenture Federal reported $3.6 billion in AI bookings for FY2025 — are implementing platform products. Azure Government, Google Vertex AI (FedRAMP High authorized), and AWS GovCloud provide the cloud and compute layer. They don't solve the constrained-output problem; that's not what a cloud platform does. Municode.ai is building specifically for the municipal code corpus use case but has no FedRAMP track record and no published constrained-decoding architecture.
The FedRAMP authorization letter on a vendor's slide deck and a production escalation rate below 5% are measuring different things. One tells you the system passed a controls audit. The other tells you it knows when it doesn't know.
That gap — between platform deployment and legally defensible output — is the specific architecture problem Veriprajna's government AI solution was built to close.
The Questions That Reach the Actual Exposure
Certification responses — SOC 2, FedRAMP authorization status, StateRAMP compliance tier — tell you about the cloud layer. They don't tell you about the output layer.
The questions that reach the actual liability exposure: What does the system do when it can't retrieve a supporting statute — does it generate anyway, or does it escalate? What is the measured escalation rate in production deployments, and how is that rate calculated? Has the corpus been validated against the operative municipal code version, not just the most-recently-indexed publication? What is the error rate on questions where local ordinance departs from the state default — the exact distribution where retrieval without constraints fails?
If those questions produce vague answers, the risk is likely still in the architecture, even if the vendor's benchmark hallucination statistics look clean. Benchmark tests use query distributions that favor coverage; production query distributions in a city portal skew toward the questions residents are genuinely confused about, which are disproportionately the questions where local law diverges from common expectation. That divergence is what the benchmark and the real-world error rate are measuring differently.
We've been working through this evaluation process with government technology teams across a range of deployment contexts. If your organization is assessing government AI for 311 services, zoning inquiries, housing rights information, or other citizen-facing use cases — comparing what vendor demos show against what production data reveals tends to be the most useful starting point. The gap is consistent enough that the conversation is worth having regardless of where you are in procurement.