The first time I ran Chris Bakke's exact attack sequence on one of our own chatbot deployments, it broke quickly. Not because our system prompt was weak. Because the system prompt is not the right place to put a business rule.
Bakke was the person who convinced a California Chevrolet dealership's chatbot to agree to sell a $76,000 Tahoe for one dollar in December 2023. His method: inject a persona override into the conversation telling the bot to agree with everything, confirm each response with "and that's a legally binding offer." The chatbot complied. What made it compliant wasn't a vulnerability in the system prompt. It was the fundamental architecture of how LLMs process input — user message and system prompt are concatenated into a single stream, and the model resolves conflicts probabilistically. A persuasive enough user framing will beat a system prompt instruction often enough to matter.
When I ran the same attack on our deployment, I'd been confident the system prompt was strong enough. That confidence is, in retrospect, the first mistake worth examining.
The system prompt lives in the same semantic space as the attack. That's not a configuration problem. It's an architecture problem.
The Air Canada Ruling Changed What I Thought I Was Building
I read the full Moffatt v. Air Canada decision on a Tuesday evening in early 2024, and the paragraph that stopped me was the airline's defense. Air Canada had argued that its chatbot was a "separate legal entity" for whose statements the airline could not be held responsible. The BC Civil Resolution Tribunal called this a "remarkable submission" and rejected it.
Three words. The entire defense theory built on a novel interpretation of AI accountability — gone. What the tribunal established in its place was that the chatbot is part of the website, the company owns every representation it makes, and consumers are not required to cross-check AI statements against other company documents to establish that they reasonably relied on those statements. Jake Moffatt, who had asked about bereavement fares after his grandmother's death and was told he could book full-price and apply for a discount retroactively within 90 days — a policy that did not exist — was awarded $800. The doctrine has no ceiling.
What I realized reading that decision wasn't that Air Canada had made a careless mistake. The chatbot had accurate documents. The retrieval system found both relevant chunks. The model synthesized them into a confident, wrong, policy — one that combined "bereavement fares exist" with "refunds are available within 90 days" in a way that was plausible but untrue. The failure is in the retrieval-and-synthesis layer's inability to encode that bereavement fare eligibility requires pre-travel approval, that those two facts aren't averageable. That's a knowledge graph problem. It's not what content safety tools are built to catch.
Why the DPD Poem Was the Hardest Problem to Admit
The Air Canada failure felt tractable to me — retrieval architecture, knowledge graph constraints, grounding. The DPD failure in January 2024 didn't.
Ashley Beauchamp asked DPD's chatbot to write a poem calling the company terrible. The model produced a multi-stanza critique of its own employer, called DPD useless and a customer's worst nightmare, agreed to swear when pushed. Screenshots went viral within hours. DPD disabled the AI component that day.
My instinct when I first read this was that it was a different problem from the Chevy Tahoe or Air Canada — a tone failure, not a logic failure. Tune the model better. Add a content filter. What I've since accepted is that this failure mode is more structurally difficult than either of the other two. Research from Oxford and Anthropic has quantified sycophancy as a byproduct of RLHF: models mirror user sentiment to maintain conversational coherence, and the effect scales with model capability. The more aligned your model, the more faithfully it reflects a hostile user's framing back at your brand.
I watch the consolidation — CrowdStrike's $740 million acquisition of SGNL in January 2026, Check Point acquiring Lakera for around $300 million in September 2025, Palo Alto Networks acquiring Protect AI for $500–700 million — and I understand what each one is protecting against. LLM firewalls catching jailbreaks and toxic outputs. Identity enforcement on which agents call which APIs. What they're not buying is protection against a model that agrees with a hostile user because that's what it was trained to do.
The architectural answer is a secondary classifier that reviews draft responses before they reach the customer, fine-tuned on brand-safety failures, running at 30–50ms. The primary model generates. The classifier decides whether the draft ships. The model's job and the classifier's job are separate.
That's when the three failure modes clicked into a single architecture for me. Not one problem with three expressions. Three distinct interventions, all living in the layer between your LLM and your customer.
The Team Argument That a Demo Settled
My team argued for a while about whether YAML policy files were overengineering. The case against them: a well-crafted system prompt could enforce the same constraints with less friction for clients. They didn't need to define their business rules as code — they could describe them in natural language the way they were already doing.
I ran a red team. We took a representative chatbot with a detailed system prompt specifying pricing authority, refund eligibility, and escalation triggers. Ran Bakke's attack plus a variant targeting the refund scenario. The system prompt broke on the refund variant in about ninety seconds. Then we implemented the same constraints as YAML gates — comparison operators evaluating before any LLM output was generated. Ran the same attacks. The gates held.
The reason isn't subtle. if offer < (MSRP * 0.9): reject runs deterministically before the model sees the user's message. It doesn't weigh tone. It doesn't read persuasiveness. It compares numbers. A system prompt instruction that says "don't accept offers below 90% of MSRP" runs inside the model, in the same context window as "your objective is to agree with anything." Sometimes agreement wins. OWASP added Excessive Agency to its 2025 Top 10 for LLM Applications specifically because agentic deployments — chatbots wired to create_quote() or transfer_funds() — make the stakes of that "sometimes" existential.
The Enterprise AI Liability & Guardrails architecture we've built around this principle: policy files encode what the model is allowed to say as code, not as prose instructions. The model handles generation. The policy layer handles compliance. They don't share a context window.
A Threat Class We Hadn't Planned For — and What We Did About It
My reaction when I first saw arXiv 2507.10457 was that we'd missed something. The paper describes LPCI — Logic-layer Prompt Control Injection — a class that embeds encoded, conditionally triggered payloads in RAG vector stores and tool outputs. These payloads stay dormant across sessions until a trigger condition fires, entering through the retrieval path that our architecture treated as trusted. Input filters don't see them. The payload doesn't come through the user message path where input rails sit.
Published evaluations measured execution rates up to 49% on unprotected systems. The proposed defense — origin validation on every retrieved chunk, temporal awareness in the orchestrator, session isolation — achieved an 84.94% block rate. We've implemented all three. What changed wasn't the business logic gate; it was the retrieval architecture upstream of it. Every chunk coming out of the knowledge graph now carries an origin tag. If the tag doesn't match expected provenance, the chunk doesn't enter the model's context.
The 2026 enterprise AI security survey of 900+ executives and practitioners found 88% of organizations had confirmed or suspected AI agent security incidents last year. Only 14.4% were shipping agents to production with full security and IT approval. I've stopped being surprised by that gap. The technology to close it exists. The architecture decisions about which layer the controls live in — and whether they're in code or in a prompt — are the thing most deployments are still getting wrong.
The Deadline I'm Actually Planning Around
The date I'm scheduling backward from is August 2, 2026 — when the EU AI Act enters full high-risk enforcement. Article 14 requires meaningful human oversight for high-risk systems, maximum penalties of EUR 35 million or 7% of global revenue. California SB 243 took effect January 1 this year: $1,000 per violation, private right of action. Colorado's AI Act follows June 30, 2026, at up to $20,000 per violation. There are 78 active AI chatbot safety bills across 27 states.
The legal development I'm most focused on isn't any of those dates. It's the May 2025 ruling in Garcia v. Character Technologies, where Judge Anne Conway held that Character.AI is a product — not a platform — for product liability purposes, and that Section 230 immunity doesn't apply to content materially generated by the AI itself. Character.AI and Google settled in January 2026. The "we're just a platform" defense is gone. Combined with Moffatt, the question every enterprise legal team is now operating under is whether they can demonstrate reasonable care for every material response their AI gave — in writing, traceable to a rule.
The Klarna reversal sits in the background as the clearest signal about where this resolves in practice. Klarna replaced around 700 customer service positions with AI between 2022 and 2024, claimed it was handling two-thirds of all interactions, and began rehiring in 2025 when customer satisfaction dropped and complaint volume rose. CEO Sebastian Siemiatkowski acknowledged going too far. The hybrid architecture — AI within defined authority limits, human handoff for exceptions — is what survives. That's what the framework I described above is built to enable.
If you're in the middle of that arc right now — past the deployment, noticing the edge cases, working out which business rules need to move from the prompt to the policy layer before August — the Enterprise AI Liability & Guardrails page covers how we think about that transition. And if you want to compare notes on where the hard questions are landing in your deployment, I'm genuinely interested.
What I actually worry about isn't the enterprises that will face an audit under the Moffatt doctrine. It's the ones that will face an audit and won't be able to point to a single document showing the business rule existed before the incident did — written down, dated, owned. That document is what reasonable care looks like in practice. Building it is almost never the hard part. The hard part is the meeting where you have to agree on what the rule actually is.