A grieving man asked an airline chatbot about bereavement fares. The chatbot invented a refund policy that didn't exist — buy now, claim back within 90 days — and stated it with complete confidence. When Air Canada refused the refund, citing their actual policy, the passenger sued. And won.
That 2024 ruling, Moffatt v. Air Canada, didn't just cost the airline $800 in damages. It established a legal precedent that keeps me up at night: your AI chatbot is a legally binding employee, and every promise it makes — real or hallucinated — is a promise your company must honor.
I've spent the past year studying what this means for enterprises deploying customer-facing AI. The conclusion my team reached was uncomfortable: the way most companies build AI chatbots today is essentially handing an untrained employee a company handbook and letting them negotiate contracts unsupervised. We published our full analysis as an interactive whitepaper, but the core argument is simple enough to explain here.
The Defense That Made Lawyers Laugh
Air Canada tried something audacious in their defense. They argued the chatbot should be treated as a separate legal entity — responsible for its own mistakes. The correct refund policy existed on their static website, they said, so the company shouldn't be blamed for the "misleading words" of their bot.
The tribunal didn't just reject this. Tribunal Member Christopher Rivers ruled that "Air Canada could not separate itself from the AI chatbot." To a customer, there's no meaningful difference between a human agent, a webpage, and a chatbot. They're all the company speaking.
Three principles came out of that ruling that every executive deploying AI needs to understand:
Unified liability. It doesn't matter whether information comes from a static page or a language model. Both are the company's representations.
Duty of care. Deploying an AI that hallucinates policy details is legally classified as negligent misrepresentation. Not a software bug. Negligence.
No "black box" defense. The complexity of your AI architecture doesn't shield you. If the bot promises a discount, you honor the discount.
If your AI says it, your company has signed it.
That last point is the one that changed how I think about this entire industry. We're not building chatbots. We're building agents with the legal authority to rewrite corporate contracts in real time — and most of them are doing it based on statistical guesses.
Why Your AI Makes Things Up (And Can't Stop)
When I first explained this to a banking client, they pushed back: "We use GPT-4. It's the best model available. Surely it doesn't hallucinate about our own policies?"
It does. And understanding why requires understanding one uncomfortable truth about how every large language model works.
LLMs don't "know" your refund policy. They predict the next most likely word based on patterns in their training data. When Air Canada's chatbot said "refunds are available within 90 days," it wasn't reading the company's tariff. It was completing a sentence pattern that sounded plausible because it had ingested millions of documents mentioning refund windows from hundreds of different companies.
Even the best frontier models — GPT-4o, Gemini 2.0, Claude — retain baseline hallucination rates between 0.7% and 25% depending on task complexity. In creative writing, a 1% error rate is a quirk. In enterprise transactions, it's a catastrophe. A global bank handling 1 million AI-assisted queries per month at a 0.7% hallucination rate faces 7,000 potential regulatory violations every month.
The financial damage is already staggering. Global losses from AI hallucinations reached $67.4 billion in 2024. Enterprises spend roughly $14,200 per employee per year just on verifying AI outputs that can't be trusted — people double-checking the machine's homework.
Why Retrieval-Augmented Generation Isn't the Fix Everyone Thinks It Is
This is where I had my most frustrating conversations with other technologists. The standard industry response to hallucination is RAG — Retrieval-Augmented Generation. The idea is straightforward: before the AI answers, pull the relevant company document from a database and feed it into the model's context. The AI "reads" the policy and answers correctly.
Sounds bulletproof. It isn't.
The Air Canada chatbot actually linked to the correct policy page in its response. It had access to the right document. It still summarized it wrong.
RAG gives your AI the right document. It doesn't guarantee the right answer.
This was the moment my team stopped thinking of hallucination as a retrieval problem and started thinking of it as a reasoning problem. Feeding a language model the correct legal text doesn't help if the model's probabilistic reasoning engine misinterprets the legal syntax, ignores a critical clause, or defaults to patterns from its training data that feel more "natural" than the actual policy language.
Vector similarity search — the engine behind most RAG systems — adds another failure mode. A query about "refund eligibility" might retrieve a document about "refund processing times" because those concepts are semantically close. The AI then constructs a confident answer from irrelevant facts. Right neighborhood, wrong house.
The Split We Had to Make
After mapping these failure modes across dozens of client scenarios, my team arrived at a principle that now governs everything we build: conversation and compliance must never run on the same engine.
We call the architecture a Deterministic Action Layer. The name is technical, but the concept is intuitive. Think of it like a bank teller window. The friendly person greeting you and understanding your request is one system. The vault behind them — with its locks, rules, and time delays — is a completely different system. You'd never let the teller open the vault based on vibes. But that's exactly what most AI chatbots do with company policies.
In our architecture, the language model handles what it's genuinely good at: understanding messy human language, detecting intent, managing the conversational flow. But the moment a query touches anything with legal or financial consequences — refunds, pricing, warranties, data privacy — a hard switch activates. The AI stops generating. A rule-based system takes over.
That rule-based system doesn't predict answers. It executes code. It queries a database. It checks: is this ticket marked as "flown"? Yes. Then the response is: "Our policy prohibits refunds after travel. Reference: Tariff Rule 45." The language model's only remaining job is to wrap that fixed answer in a polite sentence — and even that output gets verified before the customer sees it.
We call the strictest version of this the Silence Protocol. When a query hits a compliance-critical topic, the AI's creative capabilities are effectively muted. It switches from "author" to "reader." And if no rule exists for the customer's specific situation? The system doesn't improvise. It says: "I can't answer that directly. Let me connect you with a specialist." That deliberate silence is the most important safety feature we've built.
In legal terms, AI creativity about contract terms is synonymous with fabrication.
What This Looks Like Under the Hood
I won't turn this into an engineering lecture — we've published the full technical blueprint here for teams who want implementation details. But the key layers are worth understanding at a conceptual level, because they explain why this approach is fundamentally different from prompt engineering or fine-tuning.
The first layer is a semantic router — essentially a traffic cop that sits between the customer and the AI. Unlike old keyword-matching systems that would miss "I want my cash back" when looking for the word "refund," semantic routers understand meaning. They convert the customer's message into a mathematical representation and compare it against known sensitive topics. If the match exceeds a strict threshold, the message never reaches the language model for open-ended generation. It gets routed to a specific function instead.
This is critical for security, too. Prompt injection attacks — where users try to trick the AI into ignoring its instructions — target the language model's context window. But if the routing decision happens outside the model, those attacks hit a wall.
The second layer is function calling. When the router identifies a sensitive intent, the language model doesn't calculate the answer. It extracts the relevant details from the conversation — ticket ID, purchase date, travel date — and passes them to a piece of deterministic code. Python. SQL. A database query. The code returns a result. The model translates that result into natural language. The AI becomes a translator, not a decision-maker.
The third layer, for complex scenarios like medical queries or legal research, is what we call a Truth Anchoring Network — a knowledge graph that validates the AI's proposed response before it reaches the customer. If the AI drafts a sentence saying "Drug A is safe with Drug B," but the medical knowledge graph shows a severe interaction, the response gets blocked. No negotiation.
"Isn't This Expensive Compared to Just Using Prompts?"
I hear this objection constantly. And I understand the instinct — a prompt costs fractions of a cent, while building deterministic logic layers requires real engineering investment.
But this framing ignores what "cheap" AI actually costs. The Moffatt damages were $800. The precedent it set is being studied by class-action firms worldwide. A single hallucination about a financial product or healthcare directive could generate liabilities in the millions.
Meanwhile, that $14,200 per employee per year in hallucination mitigation? That's the hidden tax companies pay when they don't trust their own AI — armies of humans spot-checking outputs, verifying claims, catching errors before they reach customers. The market for hallucination detection tools grew 318% between 2023 and 2025. That growth isn't excitement. It's panic.
Building the architecture right costs more upfront. But it's the difference between an insurance policy and a prayer.
What About Regulation? It's Already Here.
The EU AI Act classifies many customer-facing AI systems as high-risk and mandates human oversight under Article 14. GDPR Article 22 gives individuals the right to challenge automated decisions that produce legal effects — and denying a refund is a legal effect. ISO 42001, the first global standard for AI governance, requires organizations to map, measure, and manage AI risk with full audit trails.
Deterministic action layers aren't just good engineering. They're becoming a compliance requirement. When a regulator asks "why did your AI deny this customer's claim?", you need to point to a specific logic node — "credit score below threshold" or "ticket status: flown" — not a probability distribution across a neural network's hidden layers.
What This Means If You're Deploying AI Today
If you have a customer-facing AI that discusses pricing, refund policies, service terms, warranties, or data privacy, you have a legal exposure that Moffatt turned from theoretical to concrete.
Three things worth doing immediately:
Audit your high-stakes intents. Map every topic where your AI could make a representation that binds the company. Pricing. Returns. Coverage terms. Legal disclaimers. These are your risk surface.
Separate conversation from compliance. Your AI can be warm, helpful, and conversational for general queries. But the moment it touches contractual territory, the answer should come from code, not from token prediction.
Design for silence. The most dangerous thing an AI can do isn't refuse to answer — it's confidently answer wrong. Build systems where "I don't know, let me get a human" is the default for edge cases, not a failure state.
The era of the probabilistic wrapper — a thin interface over a foundation model, held together by prompt engineering and hope — is ending. Not because the technology isn't impressive. It is. But because the legal system has decided that impressive isn't the same as accountable.
Your chatbot is a legally binding employee. The question is whether you've given it the guardrails that any employee handling company funds would need — or whether you've let the most eloquent, most confident, most untrained person in your organization negotiate contracts without supervision.
I'd genuinely like to hear how others are thinking about this. If you've hit the wall between AI fluency and compliance reliability, what approaches are working for your team?