New York City built an AI chatbot to help small business owners navigate regulations. Instead, it told them to steal employee tips, refuse cash payments, and discriminate against tenants — all illegal under city law. The MyCity chatbot didn't just get things wrong. It automated illegality, confidently instructing people to commit acts that carry fines up to $250,000.
This wasn't a fringe glitch. The chatbot failed on fundamental pillars of NYC law — wage theft protections, consumer rights, housing discrimination — presenting violations as standard business practice. And when the errors were publicly reported, the bot kept giving the same bad advice. The city's defense? It's a "beta product."
We've spent considerable time analyzing what went wrong with MyCity and systems like it, and the conclusion is uncomfortable: the problem isn't a bug to be patched. It's an architecture that was never designed for the job. Government AI needs something we call Statutory Citation Enforcement — a fundamentally different approach where every answer must be grounded in a specific, verifiable section of law, or the system stays silent.
What the MyCity Chatbot Actually Said
The scope of MyCity's failures deserves a close look, because these weren't obscure edge cases. They were straightforward questions about well-established laws.
When asked whether employers could take a portion of workers' tips, the chatbot responded: "Yes, you can take a cut of your worker's tips." This is illegal under both federal and New York State labor law. Following that advice exposes a business owner to wage theft lawsuits and liquidated damages up to 100% of unpaid wages.
When asked about cashless stores, the chatbot said there were "no regulations" requiring businesses to accept cash. In reality, NYC Admin Code § 20-840 explicitly prohibits food and retail stores from refusing cash — a law passed specifically to protect unbanked residents, who are disproportionately low-income, elderly, or undocumented. First violation: $1,000 fine.
When asked about Section 8 housing vouchers, the chatbot told landlords they didn't need to accept voucher holders. NYC's Human Rights Law lists "lawful source of income" as a protected class. Fines for source-of-income discrimination have reached as high as $1 million.
A government AI that tells landlords to reject housing vouchers isn't just wrong — it's actively working against the people the law was written to protect.
We explored these failures and their legal implications in depth in our interactive analysis, which maps each chatbot error to the specific statute it violated.
Why "It's a Beta" Doesn't Work as a Defense
After the failures went public, Mayor Eric Adams defended the chatbot, saying "You can't stay in a lab forever" and that technology needs to be tested in the "real environment." The website carried a disclaimer saying the bot shouldn't be used as legal advice.
There's a problem with that framing. When a tool sits on a .gov domain, carries the city's branding, and is marketed as a compliance resource, users treat it as an extension of the government itself. And when asked directly whether it could be used for professional business advice, the chatbot said yes — directly contradicting the fine-print disclaimer on the webpage around it.
This is what we call the wrapper problem: the safety warnings live in the user interface, but the AI engine behind it remains confidently wrong and completely unaware of its own limitations. A small business owner reading "Yes, you can take a cut of your worker's tips" in a conversational interface is not going to scroll down to check the footnotes. The specific, authoritative-sounding answer wins every time.
The Legal Exposure Is Bigger Than Most People Realize
The MyCity incident isn't just a PR headache. It opens legal doors that governments have historically kept shut.
Sovereign immunity — the doctrine that generally prevents governments from being sued — has long-standing exceptions. One of the most relevant: when a government acts in a proprietary function, essentially behaving like a private service provider rather than exercising political discretion. Deploying a chatbot that gives specific, actionable business advice looks a lot like operating a consulting service. If a private consultant gave this advice, they'd face malpractice claims.
There's also the question of what happens to the business owner who follows the advice and gets fined. A legal doctrine called entrapment by estoppel applies when a government representative tells someone that certain conduct is legal, and that person relies on it. Courts haven't yet ruled on whether an AI qualifies as a "government representative" for this purpose — but the functional equivalence is hard to ignore. If courts accept this defense, the city would be legally barred from enforcing its own laws against people its chatbot misled.
When the government's own AI creates legal immunity for lawbreakers, the system has moved from civil servant to civil liability.
And there's already precedent from the private sector. In Moffatt v. Air Canada (2024), a Canadian tribunal held Air Canada liable after its chatbot hallucinated a bereavement fare policy. The airline tried to argue the chatbot was a "separate legal entity." The tribunal rejected that entirely, ruling that organizations are responsible for all information on their platforms — whether it comes from a static webpage or a chatbot.
The EU AI Act adds another layer. It classifies AI used in essential public services as high-risk, imposing strict requirements for accuracy, transparency, and human oversight. A system that invents laws would fail these requirements outright.
Why the AI Gets Law Wrong: It's Optimizing for the Wrong Thing
Understanding why these failures happen requires looking at how large language models actually work — and why that mechanism is fundamentally at odds with legal compliance.
LLMs are prediction engines. They generate the next word based on statistical patterns in their training data. They optimize for plausibility — what sounds right based on everything they've read — not for truth. In creative writing, plausibility is fine. In law, it's dangerous.
Statutory law is binary. An action is either legal or it isn't, based on specific text in a specific code section. But an LLM doesn't reason from statutes. It reasons from patterns. When the model encounters a question about landlord rights, it draws on the vast corpus of general contract law and property rights discussions across the internet — where "landlords can choose their tenants" is a common sentiment. The specific NYC exception protecting voucher holders is a tiny signal drowned out by that dominant pattern. Our research identifies this as semantic drift: the model slides from the precise legal definition to the colloquial understanding found in its training data.
There's a second, subtler problem. Most commercial LLMs are fine-tuned through a process called Reinforcement Learning from Human Feedback (RLHF), which rewards the model for being "helpful." Research shows this creates sycophancy — a tendency to agree with the user's premise. When a landlord asks "Can I refuse a Section 8 tenant?", a model optimized for helpfulness may interpret the intent as "Help me find a way to refuse" and generate a justification. It prioritizes the user's desire over the legal reality.
An AI tuned to be helpful will sometimes help you break the law — because saying "no" doesn't score well in training.
In government contexts, the AI often needs to be the opposite of helpful to the user's immediate wish. "No, you cannot take that deduction." "No, you must accept cash." Standard commercial models aren't built for that adversarial compliance role.
Why Basic RAG Doesn't Fix This
Many organizations try to solve hallucination by bolting on Retrieval-Augmented Generation (RAG) — essentially giving the AI a searchable database of documents to reference before answering. In theory, this grounds the model in real sources. In practice, naive RAG implementations fail in legal contexts for specific, predictable reasons.
Legal codes are hierarchical. A prohibition in one section often has an exception in another and a penalty structure in a third. When RAG systems chop documents into arbitrary chunks — typically 500-token segments — they sever these connections. The AI might retrieve the prohibition but miss the exception, or vice versa.
There's also a well-documented phenomenon called "lost in the middle": when the system retrieves multiple documents, the model tends to focus on the first and last items in the context window, potentially ignoring the most relevant statute sitting at position five out of ten. And keyword overlap creates retrieval mismatches — a query about "cash" might pull documents about "cash grants" or "petty cash" instead of the cashless store ban.
What Government AI Actually Requires
We reject the chatbot model for government services entirely. The right framework isn't a smarter chatbot — it's a fundamentally different kind of system built around a principle we call Statutory Citation Enforcement: if the system can't point to a specific, valid section of law that supports its answer, it doesn't answer at all.
This isn't a prompt engineering trick or a disclaimer. It's an architectural constraint enforced at the system level through what the AI research community calls Compound AI Systems — multiple specialized components working together rather than a single model doing everything.
The retrieval layer uses hierarchical indexing that mirrors the actual structure of legal codes — Title, Chapter, Subchapter, Section, Paragraph — rather than chopping documents into arbitrary pieces. Related definitions are linked to their operative clauses through a knowledge graph, so the system retrieves the full legal context, not fragments.
The generation layer uses constrained decoding, which restricts what the AI can actually output. The model is forced to produce a structured response that includes a specific citation ID. If the model tries to reference a code section that wasn't in the retrieved results, that output is blocked at the token level — the neural pathway to hallucinate a citation is literally shut off. Think of it like a calculator that can only display numbers it actually computed, rather than numbers it guessed.
Before any answer reaches the user, a separate verification agent checks whether the cited statute actually supports the claim being made. Does the text of the cited section say what the answer claims it says? Are there conflicting statutes? Is the citation still current law? If anything fails this check, the system defaults to: "This may require professional counsel. Please contact [specific agency]."
For the full technical methodology behind this architecture — including the constrained decoding mechanisms and verification protocols — see our detailed research.
"But Won't This Make the System Less Useful?"
This is the most common objection we hear, and it deserves a direct answer: yes, a system built this way will answer fewer questions. That's the point.
A government AI that answers 70% of queries with verified, citation-backed accuracy and gracefully declines the other 30% is infinitely more valuable than one that answers 100% of queries with a 15% hallucination rate. The first is a reliable tool. The second is a lawsuit waiting to happen.
The "safe refusal" — directing users to a human specialist when the system can't find a definitive answer — mimics what a responsible civil servant actually does. A clerk at a city office who doesn't know the answer to your zoning question doesn't make one up. They refer you to someone who does.
What Should Government Technology Leaders Do Now?
The era of shipping a general-purpose chatbot with a government logo and a disclaimer is ending — not because of public opinion, but because of legal exposure. The Air Canada ruling, the EU AI Act's high-risk classification, and emerging US legislation like the AI LEAD Act are all converging on a single message: organizations are liable for what their AI says.
For anyone responsible for government technology decisions, three things matter immediately:
Audit existing deployments. If you have a public-facing AI tool, test it with adversarial queries — the kind of questions that should get a "no" answer. If it says "yes," you have a liability.
Demand citation-level accountability. Any AI system answering questions about law or regulation should be architecturally required to cite its sources — not as a nice-to-have, but as a condition of generating output.
Build the audit trail now. Every query-response pair should be logged with the specific source material the system used. If an entrapment-by-estoppel claim ever lands on your desk, you'll need to show exactly what the AI saw and why it said what it said.
In government AI, silence is not a failure mode. Confident falsehood is.
The technology to build trustworthy government AI exists today. What's been missing is the willingness to accept that a system designed to explain the law must be held to the same standard as the law itself — precision, verifiability, and the humility to say "I don't know."
We'd welcome hearing from anyone navigating these challenges in their own agencies or organizations. The problems are real, the legal exposure is growing, and the solutions require a different kind of conversation than most government technology procurement is used to having.