In 2023, a New York attorney submitted a legal brief citing six cases — Varghese v. China Southern Airlines, Shaboon v. Egyptair, Petersen v. Iran Air — complete with docket numbers, dates, and judicial quotes. Every single case was fabricated by ChatGPT. When opposing counsel challenged the citations, the attorney went back to ChatGPT and asked if the cases were real. The AI confirmed they "indeed exist" and could be found in "reputable legal databases." The court sanctioned him $5,000, but the real cost was something no fine captures: the moment a profession built on precedent realized its shiny new tool was inventing precedent out of thin air.
I've spent the past year studying why legal AI hallucinates and, more importantly, how to make it structurally impossible. Not "unlikely." Not "reduced by 80%." Impossible. What my team found changed how we think about building AI for any high-stakes domain.
The $5,000 Fine That Should Have Been a Wake-Up Call
The Mata v. Avianca case isn't just a cautionary tale — it's a diagnostic. It reveals exactly what breaks when you point a general-purpose language model at legal work.
ChatGPT didn't "make a mistake." It did precisely what it was designed to do: predict the next statistically likely word. Given a prompt about airline injury cases, "Varghese" was a plausible plaintiff name, "China Southern Airlines" was a plausible defendant, and a docket number like "2017 WL 2839506" fit the syntactic pattern of a real citation. The AI wasn't lying. It was pattern-matching — and in law, pattern-matching without verification is malpractice.
The hallucinated cases contained convincing docket numbers, dates, and even internal citations. The judge noted the fake opinions were "gibberish" in parts — but sophisticated enough to mimic federal judicial writing.
What made Mata truly alarming wasn't the initial fabrication. It was the verification loop. The attorney used the same tool to check its own work, and the tool confirmed its own fiction. This is the equivalent of asking a compulsive liar whether they just lied. Stanford researchers later found that general-purpose chatbots hallucinated between 58% and 82% of the time on complex legal queries — even those with internet access or basic retrieval capabilities. This isn't an edge case. It's the baseline.
Why "Better Prompting" Will Never Fix This
After Mata, the most common industry response was: "Well, you have to prompt it correctly." I heard this from vendors, from consultants, from people selling prompt engineering courses for lawyers.
It's wrong, and I want to explain why with an analogy.
Imagine you hire a brilliant writer who has read thousands of legal opinions but has never set foot in a law library and has no access to one. You ask them to cite a case supporting your argument. They'll write you something that reads like a case citation — correct format, plausible names, convincing language. But they're composing fiction in the style of legal fact. No amount of instruction ("Please only cite real cases") changes what they fundamentally are: a pattern-completion engine with no connection to a verified database.
This is what every "AI wrapper" does. A wrapper is a thin interface layered over a commercial API — OpenAI, Anthropic, whoever. You can add a system prompt that says "You are a helpful lawyer," but that doesn't grant the AI access to Westlaw or LexisNexis. It doesn't give it a list of real cases. It doesn't constrain it from inventing facts. The model's core metric is perplexity — essentially, "how unsurprised would a reader be by this next word?" It optimizes for fluency, not truth.
Prompt engineering can guide the style of the output, but it cannot inject knowledge the model doesn't have or force it to verify facts against a database it cannot access.
The Retrieval Fix That Only Half-Works
The industry's next move was Retrieval-Augmented Generation, or RAG — feeding the AI real documents before asking it to respond. In a standard RAG system, legal documents get chopped into text chunks, converted into numerical representations called embeddings, and stored in a vector database. When you ask a question, the system finds chunks that are semantically similar to your query and hands them to the AI as context.
This helps. It reduces pure fabrication. But it introduces a different class of failure that's arguably more dangerous because it's harder to catch.
Vector search finds text that looks like your question. But legal relevance isn't about looking similar — it's about structural authority. A dissenting opinion might use the exact same language as a majority opinion, but one is binding law and the other is an argument that lost. A case from 1990 might be a perfect semantic match for your query, but it was overruled in 2015. Standard vector RAG treats both the same because it only understands text similarity, not legal hierarchy.
I saw this firsthand when we tested retrieval systems on tax statute interpretation queries. The system confidently returned a law review article, a dissenting opinion, and a case from the wrong jurisdiction — all semantically close to the query. The actual binding authority was a short, dry memorandum opinion using slightly different terminology. It ranked lower because it didn't "look" as relevant to the vector math, even though it was the only answer that mattered.
There's also a well-documented problem researchers call "lost in the middle." When an AI receives a long list of retrieved text chunks, it tends to focus on information at the beginning and end, ignoring what's in the middle. In legal research, the critical exception or qualifying clause is often buried in chunk fifteen. The AI skips right over it.
What a Knowledge Graph Changes
This is where my team's work diverges from most legal AI approaches, and where I got genuinely excited about the possibility of solving this problem rather than just mitigating it.
Instead of storing legal documents as isolated text chunks in a vector database, we map them into a Knowledge Graph — a network where every case, statute, regulation, and legal concept is a distinct node, and the relationships between them are explicit, labeled connections. Case A doesn't just "relate to" Statute B. Case A interprets Statute B. Case C overrules Case A. Regulation D implements Statute B.
Think of the difference this way: a vector database is a library where all the books are shelved by how similar their covers look. A Knowledge Graph is a library where every book has a card catalog entry showing which books it cites, which books cite it, which books disagree with it, and whether it's still considered authoritative.
This structure enables something vector search simply cannot do: multi-hop reasoning. A lawyer asking "Is this 1990 regulation still valid after the 2023 Supreme Court ruling?" needs the system to trace a chain — regulation to statute to interpreting case to overruling case. Vector RAG retrieves each document in isolation and hopes the AI figures out how they connect. A Knowledge Graph walks the chain explicitly. We explored this architecture in depth in our interactive analysis.
Vector RAG asks: "Find text that looks like this query." GraphRAG asks: "Find the statute, traverse the 'interprets' edge to case law, then traverse the 'overrules' edge to confirm it's still valid."
The Safety Lock: Making Fabrication Physically Impossible
The Knowledge Graph alone is a major improvement, but the part that keeps me up at night — in a good way — is what we call Graph-Constrained Decoding. This is the mechanism that moves legal AI from "probably accurate" to "structurally incapable of fabricating a citation."
We build a prefix tree — a data structure — from every valid case name, reporter citation, and docket number in the Knowledge Graph. During generation, when the AI is about to output a citation, this prefix tree acts as a real-time filter on what the model is allowed to say.
If the model has generated "Mata v. A" and is about to produce the next characters, the system checks: what valid case names start with "Mata v. A"? It finds "Avianca" and enables only those tokens. Everything else gets blocked. If the model tries to generate "Varghese v. China Southern" — the fake case from Mata — the system checks after "Varghese v. Chi" and finds nothing in the verified graph. Generation stops. The model is forced to either find a real citation that fits or output "No precedent found."
This isn't a confidence score. It isn't a disclaimer. It's a mathematical constraint on the output space. The AI physically cannot produce a sequence of characters that doesn't correspond to a verified entity in the graph. I want to be precise about what this guarantees and what it doesn't: the system prevents fabrication — inventing a case that doesn't exist. It doesn't prevent misinterpretation — citing a real case for a proposition it doesn't fully support. That's a reasoning error, and it still requires human judgment. But eliminating fabrication removes the most catastrophic and embarrassing failure mode, the one that gets lawyers sanctioned.
For the full technical methodology behind this constraint mechanism, see our detailed research.
The Hardest Engineering Problem Nobody Talks About
Building the graph sounds clean in theory. In pr
actice, it's a brutal data engineering challenge, and I want to be honest about that because too many AI companies skip this part.
Legal texts are messy. A single case might be referenced as "Mata v. Avianca," "Mata," "678 F. Supp. 3d 443," "the Avianca case," or simply "Id." — a legal shorthand meaning "the same case I just mentioned." Every one of those strings needs to resolve to a single node in the graph. Miss one, and you've got a blind spot. Miss many, and your graph is Swiss cheese.
We run entity resolution pipelines that combine rule-based matching with probabilistic methods. "Smith v. Jones, 123 F.3d 456" and "Smith, 123 F.3d at 456" must map to the same entity. "Smith v. Jones (1995)" and "Smith v. Jones (2002)" must not. The "Id." problem alone required building a sliding-window context parser that tracks which case is currently "active" in a paragraph and resolves backward references in real time.
Then there's negative treatment — the legal equivalent of a "do not cite" stamp. If a case has been overruled, it cannot be used as binding authority, no matter how eloquent its reasoning. In our graph, an OVERRULES edge acts as a blocking signal. If the system traces a path from your query to a case and encounters an OVERRULES edge, that path is invalidated before the AI ever sees the document. A vector database has no mechanism for this. It will happily retrieve Roe v. Wade as current binding authority on abortion rights because the sheer volume of relevant text overwhelms the signal that Dobbs v. Jackson changed everything.
A system that retrieves the right case 80% of the time is a malpractice machine 20% of the time.
What About Firms Already Using AI Successfully?
Fair question. Many firms use AI for drafting, summarization, and initial research with good results. The issue isn't that AI is useless — it's that the failure modes are invisible until they're catastrophic.
If you're using AI for internal memos that a senior partner reviews thoroughly, the risk is manageable. If you're using it for anything that gets filed with a court, submitted to a regulator, or relied upon by a client without line-by-line verification, you're carrying risk that scales with every use. The question isn't whether your AI has hallucinated — Stanford's research suggests it almost certainly has. The question is whether anyone caught it.
Courts are increasingly requiring mandatory disclosure of AI use in filings. Malpractice insurers are starting to ask about AI governance policies. The window where "we didn't know AI could do that" was an acceptable defense closed with Mata.
Does This Scale Beyond Legal?
The architecture we've built isn't limited to case law. Any domain where citations must be verifiable — regulatory compliance, medical literature, financial reporting — faces the same structural problem. An AI that confidently cites a clinical trial that doesn't exist is as dangerous as one that cites a fake court case. The Knowledge Graph approach works wherever there's a defined universe of authoritative sources and the relationships between them matter.
For corporate legal departments, we're already mapping internal policies to external regulations — linking specific paragraphs in a company's data governance policy to specific sections of GDPR or DORA. The graph creates an auditable compliance matrix that a vector search couldn't produce.
Where This Goes Next
The industry is moving from chatbots — passive tools that respond to prompts — toward agents that plan and execute multi-step tasks autonomously. A legal agent asked to "draft a motion to dismiss" needs to research the applicable standard, find supporting cases, verify they're still good law, check procedural requirements, and assemble the document. That agent needs a map of the legal landscape, not a pile of text chunks. A Knowledge Graph is that map.
I believe we're watching the end of what I'd call the "wrapper era" — the period where slapping a chat interface on a commercial API and calling it "legal AI" was a viable business. The firms and vendors that will thrive are those building real infrastructure: verified data, structured relationships, constrained generation. Not because it's trendy, but because the alternative is a profession gambling its credibility on next-word prediction.
The lesson of Mata v. Avianca was never that AI has no place in law. It's that probabilistic AI has no place in deterministic citation. The technology exists to solve this. The question is whether the industry demands it.
If you're leading a legal team navigating AI adoption — or dealing with the compliance implications in any regulated industry — I'd genuinely like to hear what you're seeing on the ground. The gap between what vendors promise and what practitioners experience is where the real problems live.