New York City built an AI chatbot to help small business owners navigate regulations. Instead, it told them to steal employee tips, refuse cash payments, and discriminate against tenants with housing vouchers — all illegal under city law. Every single piece of advice, delivered with the confidence of a seasoned bureaucrat, would have exposed anyone who followed it to fines, lawsuits, or criminal charges.
This wasn't a glitch. It was the predictable result of wrapping a generic AI model in a government website and hoping for the best. And it's the reason my team at Veriprajna has spent the past year building something fundamentally different: an architecture where government AI cannot answer a question unless it can cite the exact law behind its answer.
We call it Statutory Citation Enforcement — and I believe it's the only responsible way to deploy AI in public services.
The MyCity Disaster Was Worse Than You Think
When The Markup investigated NYC's MyCity chatbot, they didn't find a few edge-case errors. They found systematic failures across fundamental areas of law.
The chatbot told employers they could take a cut of workers' tips. That's wage theft — illegal under both federal and New York State labor law, with penalties including liquidated damages up to 100% of unpaid wages.
It told store owners they could refuse cash. NYC Admin Code § 20-840 explicitly prohibits that, with fines of $1,000 for a first violation. The city passed that law specifically to protect unbanked residents — disproportionately low-income, elderly, and undocumented New Yorkers.
It told landlords they didn't have to accept Section 8 housing vouchers. NYC's Human Rights Law bans source-of-income discrimination, with fines reaching $250,000.
A government AI that tells landlords to discriminate isn't a beta product with bugs. It's a civil rights violation wearing a .gov domain.
And when asked directly whether it could be trusted for professional business advice, the chatbot said yes — contradicting the disclaimer buried in the website footer. The safety warning was in the wrapper. The model itself had no idea it was supposed to be cautious.
I remember the moment I walked my team through each of these failures. The room went quiet at the housing voucher one. We'd been building retrieval systems for legal content, and we knew why this happened technically. But seeing a government tool actively direct people toward discrimination — that shifted something for us. This wasn't an academic problem anymore.
Why Generic AI Models Can't Do Legal Compliance
To understand why MyCity failed, you need to understand one thing about how large language models work: they predict what sounds right, not what is right.
An LLM is trained on massive amounts of internet text. When a landlord asks "Can I refuse a Section 8 tenant?", the model draws on millions of web pages about property rights, landlord forums, and general contract law. The statistically dominant pattern in that data is that landlords can choose their tenants. So the model generates text supporting that — even though NYC has a specific local law saying otherwise.
This is what researchers call semantic drift: the model slides from the precise legal meaning to the general, colloquial meaning found across the internet. It's not lying. It's doing exactly what it was designed to do — predict plausible text. The problem is that plausible and legal are completely different things.
There's a deeper issue, too. Commercial AI models are fine-tuned using a process called Reinforcement Learning from Human Feedback, which rewards them for being helpful. Research shows this creates sycophancy — the model tends to agree with whatever the user seems to want. When someone asks "Can I refuse cash?", the model reads the intent as "help me go cashless" and generates a supportive answer, overriding whatever it might know about the actual regulation.
An AI trained to be helpful will invent a loophole before it tells you no. That's exactly the opposite of what a government system should do.
"It's Just a Beta" Doesn't Work When You're the Government
Mayor Adams defended MyCity by saying, "You can't stay in a lab forever." The city classified it as a beta product and pointed to disclaimers.
I understand the impulse. Shipping early and iterating is how good software gets built. But government AI isn't a consumer app. When a tool sits on a .gov domain, branded as the city's official resource for business compliance, it carries the authority of the state. A small business owner reading "Yes, you can take a cut of your worker's tips" from MyCity isn't going to cross-reference the NYC Administrative Code. They're going to trust it, because the government told them.
The legal exposure here is real and growing. In Moffatt v. Air Canada (2024), a Canadian tribunal held Air Canada liable when its chatbot hallucinated a bereavement fare policy. The airline tried to argue the chatbot was essentially a separate entity, and that the passenger should have checked the fine print. The tribunal rejected that completely — ruling that an organization is responsible for all information on its website, whether static or AI-generated.
Now apply that logic to a municipality. If a business owner gets fined for wage theft after following MyCity's advice, they have a potential defense called entrapment by estoppel — the government's own tool told them it was legal. If courts accept that defense, the city can't enforce its own laws against people its AI misled.
We explored these liability scenarios in depth in our interactive analysis, and the legal landscape is shifting fast. The EU AI Act already classifies government-facing AI as high-risk, with strict accuracy requirements. Emerging U.S. legislation like the AI LEAD Act is moving to classify AI systems as products subject to strict liability. The window for treating government chatbots as harmless experiments is closing.
What Statutory Citation Enforcement Actually Means
After studying the MyCity failure and dozens of similar deployments, my team arrived at a principle that now governs everything we build:
No citation, no output. If the system can't point to the specific section of law that supports its answer, it stays silent.
This isn't a prompt engineering trick or a disclaimer. It's an architectural constraint — the system is physically unable to generate an answer without a verified legal citation backing it up.
Here's how it works in practice. A business owner asks: "Can my store refuse cash in New York City?"
The system searches a structured knowledge graph — not the open internet, not the model's training data, but a purpose-built index of the actual municipal code, organized the way law is organized: Title, Chapter, Subchapter, Section, Paragraph. Each provision is tagged with its effective date, penalties, enforcing agency, and related definitions.
The search returns NYC Admin Code § 20-840. The system then generates its answer, but under a hard constraint: it can only reference citation IDs that actually exist in the retrieved results. We use a technique called constrained decoding — think of it as a filter on the model's vocabulary that blocks it from referencing any law that wasn't retrieved. The model literally cannot hallucinate a citation, because the pathway to generate a fake one is blocked at the token level.
Before the answer reaches the user, a separate verification agent checks whether the cited text actually supports the claim being made. Does § 20-840 really say cash refusal is prohibited? If the answer and the citation don't match — if the system says "yes" but the law says "unlawful" — the output is suppressed entirely.
The user gets either a verified answer with a hyperlinked citation, or a clear message: "I couldn't find a specific regulation for your question. Please contact the Department of Small Business Services."
That second outcome — the honest "I don't know" — might be the most important feature. A responsible civil servant who doesn't know the answer refers you to someone who does. They don't make something up to seem helpful.
Why Standard RAG Systems Still Fail for Legal Questions
Some readers might be thinking: "Isn't this just RAG — retrieval-augmented generation? Lots of companies do that." It's a fair question, and the answer matters.
Basic RAG systems chop documents into chunks of roughly 500 words and search them by semantic similarity. For legal text, this is disastrous. Law is hierarchical. A prohibition in Section A might have an exception in Section B and a penalty in Section C. Naive chunking severs those connections.
There's also a well-documented problem called "lost in the middle": when the system retrieves ten document chunks, the model tends to focus on the first and last ones, often missing the most relevant piece buried in position five or six.
And semantic search can mislead. A query about "cash" might retrieve documents about "cash grants" or "petty cash management," crowding out the actual cashless ban statute.
Our approach — hierarchical indexing with graph-enhanced retrieval — preserves the structure of legal codes. Parent nodes capture the intent ("Consumer Affairs > Cashless Establishments"), child nodes hold the operative text and penalties, and graph links connect definitions to the clauses that reference them. When you search for cashless policies, you get the prohibition, the exception for online transactions, and the penalty schedule — together, in context, the way a lawyer would read them.
For the full technical methodology behind our retrieval and constrained decoding architecture, see our detailed research.
What About Edge Cases and Ambiguity?
Law isn't always black and white. What happens when a question falls in a gray area, or when two statutes seem to conflict?
This is actually where I think our approach shows its greatest strength. When retrieval confidence is low — when the system can't find a statute that clearly addresses the question — it doesn't guess. It triggers what we call a safe refusal: a message acknowledging the complexity and directing the user to a human specialist.
There's a philosophical point here that I keep coming back to. We've been trained by consumer technology to expect an answer to every question. But in government services, the most responsible thing an AI can do is sometimes say nothing. A confident wrong answer from a government system causes real harm — fines, evictions, discrimination. A transparent "I'm not sure, here's who to call" causes none.
We also built granular controls so administrators can disable specific topic areas without taking down the entire system. If an error is found in housing-related queries, the housing node gets shut off while everything else keeps running. Every query-response pair is logged with the exact retrieval chunks used, creating a forensic audit trail. If a lawsuit ever arises, the government can show precisely what the AI retrieved and cited — evidence of a rigorous process, not negligent deployment.
What This Means If You're Building or Buying Government AI
I'll be direct about what I think needs to change.
First, stop deploying generic chatbot interfaces for regulatory guidance. The conversational format — friendly, anthropomorphic, designed to feel like texting a helpful friend — actively encourages the kind of trust that makes hallucinations dangerous. A "Regulatory Search & Verify" tool with visible citations sets the right expectations.
Second, demand citation-level accountability from any AI vendor. If a system can't show you the exact statute behind every answer, it's not ready for government use. Full stop. The Air Canada precedent makes clear that disclaimers won't protect you.
Third, treat every AI interaction as a potential incident. Log everything. Build human-in-the-loop review into flagged responses. Create the audit trail before you need it in court.
The era of the "beta" government chatbot is over. Citizens deserve systems built with the same rigor as the laws those systems explain.
The MyCity failure wasn't a failure of AI's potential. It was a failure of architecture — of treating a compliance tool like a creative writing exercise. The technology to do this right exists today. What's been missing is the willingness to accept that government AI needs to be fundamentally different from consumer AI.
I'd genuinely like to hear from anyone working on this problem — in government, in civic tech, in legal AI. How is your organization thinking about the line between helpful and accurate? What's your threshold for deploying AI in high-stakes public services?