NYC's government chatbot told landlords they could legally refuse Section 8 vouchers. They can't. Source-of-income discrimination has been illegal in NYC since 2008, with fines up to $250,000. The same bot — MyCity, launched October 2023 on a .gov domain — told employers they could pocket a cut of workers' tips (illegal under NY Labor Law § 196-d) and told businesses they could go cashless (illegal under NYC Admin Code § 20-840). When The Markup investigated in March 2024, the pattern was systematic: confident, fluent, and wrong on exactly the parts of the law that hurt real people. Here's the part most coverage missed: this isn't a tuning bug. Large language models are probabilistic engines optimized for plausible-sounding text. Ask one a statutory question and it generates the most likely-looking answer — not the correct one. That's why even the best legal AI tools still post hallucination rates of 17–33% (Stanford, 2025). For a lawyer double-checking citations, that's a productivity tax. For a citizen acting on a government answer, it's a wrongful eviction. The fix isn't a better prompt. It's architecture. We build government AI where every response traces to a specific statute — or the system stays silent. No citation, no answer. The model can't freelance legal advice because it's constrained to retrieve and cite, never to invent. This matters now because the law is catching up: 78 chatbot-safety bills moved across 27 states in 2026, and New York's S7263 would create a private right of action when a chatbot gives professional advice. "The vendor added a disclaimer" won't be a defense. If your agency is evaluating a citizen-facing chatbot — or cleaning up after one — the question to put to the vendor is simple: can it show me the statute behind every answer? Save this for your next AI procurement review. #GovTech #MunicipalAI #AIGovernance #PublicSector #LegalAI