- NYC built a government chatbot. It told landlords they could refuse Section 8 tenants. Told employers they could pocket workers' tips. Told stores they could refuse cash. Every answer was illegal. Every answer carried the city's seal. 🧵
- The Markup investigated MyCity in March 2024 and found systematic illegal advice across NYC's most basic laws — wages, housing, consumer protection. The city's fix? A disclaimer. The bot then told users: "Yes, you can use this bot for professional business advice."
- The exact failures, from the public record:
"Take a cut of worker tips" — illegal (NY Labor Law §196-d)
"No need to accept cash" — illegal (NYC Admin Code §20-840)
"Refuse Section 8 vouchers" — illegal (NYC Human Rights Law, since 2008) - This isn't a bug you tune away. It's architectural. LLMs optimize for plausible text. Ask "Can I refuse a Section 8 tenant?" and the model reaches for the dominant pattern — general contract law's freedom to choose — overriding the local statute that says no.
- RLHF makes it worse. Models are trained to be "helpful," which means agreeing with the user's intent. A landlord asking how to refuse a tenant gets a "yes." Government AI often has to be unhelpful to your immediate wish in order to be accurate about the law.
- "Just add RAG" doesn't fix it. Stanford tested legal AI with retrieval in 2025: Lexis+ AI still hallucinates 17%, Westlaw 33%. General-purpose GPT-4? 43% on legal queries. Even the best tool built for lawyers is unfit for citizen-facing statutory advice.
- And the liability is arriving. NY's S7263 reached the Senate floor Feb 26, 2026 — a private right of action for chatbots giving professional advice. 78 chatbot safety bills are live across 27 states. The EU AI Act treats gov AI as high-risk: up to €15M.
- Our approach: we constrain decoding so a government chatbot can only cite a real statute, or it stays silent. No provision on point → it refuses and routes to a human. Every answer traces to a citation, with an audit trail. Helpfulness never outranks legal accuracy.
- Honest question for gov-tech and civic teams: is a citizen-facing legal chatbot that can't cite the exact statute worth shipping at all — or is "route to a human" the only defensible default once liability lands? #LegalAI
- We wrote up how to build government AI that cites the law instead of inventing it — the architecture, the liability, and what actually works: https://veriprajna.com/solutions/government-municipal-ai