The cases were real. I still couldn't let the AI vouch for them.

The first time I watched my own tool refuse to bless three real Supreme Court cases, I was sure it was broken.

I was running a sample brief I had labeled "Misapplied Precedent." It cited Erie Railroad v. Tompkins, 304 U.S. 64, for the existence of a uniform body of federal common law, which is close to the opposite of what Erie actually held. It leaned on Celotex Corp. v. Catrett, 477 U.S. 317, and Baker v. Carr, 369 U.S. 186, for propositions the opinions do not stand for. Every one of those cases is real. I expected my support-check to light them up red, stamp them Unsupported, and close the matter. Instead all three came back with a quieter verdict: Needs Review. My first instinct was that the language model had failed the easy part.

It took me an embarrassingly long evening to understand that the instinct was the bug, and it was in me, not the system.

A confident "Unsupported" stamped on a subtle misreading of Erie would have been a bluff. The opinion is long, the misuse is a matter of legal judgment, and the model cannot establish the negative from a snippet with the certainty that a red verdict implies. The honest move, the one I had accidentally built and then almost coded out in frustration, was to route it to a human with the contradicting context attached. Abstention was not the failure. Abstention was the whole point. If you want to watch it decide for yourself, it lives here: veriprajna.com/demos/legal-ai-citation-verification.

This is an essay about the assumption I started with, which almost everyone building legal AI right now shares, and the slow, specific way building the demo took it apart. The assumption is that the hallucination problem gets solved by a better model. I no longer believe that, and not because I doubt the models will improve.

The week I tried to make one model police another

I spent about a week trying to get a language model to be the trustworthy judge of another language model, and I want to be honest that it did not work.

The setup looked reasonable. Some model drafts a brief, from Harvey or Lexis Protégé or an open-source model, it does not matter which. Then a second model reads each citation against the case it points to and rules on whether the opinion supports the claim. Two models, a tidy check, the kind of architecture that survives a design review. I genuinely expected the ruling step to be the easy part. It was the part that would not hold still.

The failure was not loud, which is what made it dangerous. I would run the same brief through the support-check three times and get two "Needs Review" and one confident clear, with no change to the input. On the Bell Atlantic Corp. v. Twombly citation, 550 U.S. 544, a real case, the model would sometimes decide the opinion squarely established the cited proposition and sometimes decide it did not quite. Both readings were defensible. That was exactly the problem. A ruling you cannot reproduce is not a ruling. It is a second opinion wearing a robe.

I was asking a probabilistic system to be the deterministic gate on another probabilistic system, and calling the result verification.

That is not verification. That is two models agreeing, which is a weaker and much shiftier thing. If the reason you need a check is that the first model's output cannot be trusted at face value, the second model's output cannot be the thing you trust to check it. I had built a hall of mirrors and was about to put a compliance sticker on it. The court on the other end does not care how confident either model was. It cares whether the case exists and whether it says what you claimed. Those are two questions, and I had been treating them as one.

Which is more dangerous, a fabricated case or a real one used wrong?

What surprised me while building this was realizing the two failures need completely different machinery, and the industry mostly ships one of them.

A fabricated citation is, oddly, the friendly failure. Halstead v. Ferngate Holdings, 823 U.S. 1199, appears in one of the sample briefs, cited with total fluency for the proposition that punitive damages are categorically barred. It does not exist. The United States Reports has no volume 823. You can prove that with certainty, not by asking a model what it thinks, but by querying real case law and getting a zero-result lookup back. That is arithmetic against an authority, not judgment. It is the kind of thing code should decide, and decide the same way every time.

The dangerous failure is the real case cited for the wrong thing. Erie is a real, famous, correctly-formatted citation. Every fabrication filter in the world waves it through, precisely because it is not fabricated. The volume exists, the page exists, the opinion exists. What is wrong is the relationship between the cited case and the sentence in front of it, and that relationship is a question of legal reading, not existence. A source-matching check treats "this case is real" and "this case supports my claim" as the same fact. They are not even close.

So the numbers people quote about legal AI start to make sense. Even purpose-built tools hallucinate at rates that should stop you cold: Westlaw Precision at 33 percent and Lexis+ at 17 percent in the Stanford RegLab study published in the Journal of Empirical Legal Studies, 2025. By early 2026 there were 1,222 documented court cases involving AI-hallucinated citations, and courts have started sanctioning, including a 30,000 dollar penalty from the Sixth Circuit in March 2026. The fabrications are the failures you can at least imagine catching. The real-but-misapplied cite is the one that looks like diligence right up until a judge reads the opinion you cited and finds it says the opposite.

A fabricated case is a lie you can disprove. A real case cited for the wrong holding is a lie that passes every test built to catch the first kind.

Why I split the engine in two

I stopped trying to make one system do both jobs the evening I accepted they were different jobs.

The existence check and the fabrication catch became deterministic Python running against real case law. No prompt, no temperature, no "as an AI language model." Take the citation, look it up in an independent authority, and if the case is genuinely absent, it is fabricated, provably, and it can never pass the gate. That last clause is the one guarantee I am willing to state flatly, because it is a unit-tested invariant rather than a hope: zero fabricated citations can pass the policy gate. There is a test named for exactly that, and it is part of the 8 out of 8 that pass on the built demo.

The support-check stayed a single language-model step, because judging whether a real opinion supports a proposition is genuinely a reading task, and the model is a good reader. But I gave it explicit permission to abstain, and I stopped treating abstention as a bug. When it cannot ground the claim in the opinion text, it routes to a human instead of guessing. The model advises. The deterministic gate and the attorney decide. I ended up saying two things constantly while building this. One is agents advise, code decides. The other is that I am not going to let an LLM be the final judge of an LLM.

I want to be precise about the split, because the temptation is to blur it into a bigger promise than it is. The deterministic parts, existence, fabrication, outside-coverage, and the gate, are exact and reproducible. The support-check is not. It is non-deterministic, and by design it clears a cite green only when the opinion text plainly establishes the holding, the way Miranda v. Arizona, 384 U.S. 436, does on custodial-interrogation warnings, and otherwise it sends the cite to review. Which specific real cites clear green versus route to a human can vary run to run. That is not a rough edge I am hiding. It is the honest shape of the problem, and pretending otherwise is exactly the move I am trying to build against.

What "8 out of 8" is actually allowed to mean

I hold myself to a rule about numbers, because the company I am building is named Veriprajna, which means true wisdom, and a name like that is a standing dare to overclaim.

There are proof numbers in the demo and they mean narrow things. 8 out of 8 offline unit tests pass, including the one that proves no fabricated cite can clear the gate and the one that proves an unreachable authority is treated as unproven rather than fabricated. There are 20 real cases cached from CourtListener as the local ground truth. Those describe the built demo's scope. They are not a claim about your brief or the open world. The fabrication guarantee is authoritative because it queries live case law and a missing case is a fact. The support-check adjudication is demonstrated at small scale on real cases, and I will not inflate it into a universal accuracy figure, because it is not one.

Here is the line I refuse to shorten, even though a shorter version would sell better. A brief with no fabricated cites is not automatically safe to file. When even one real citation still needs a human to confirm it is used correctly, the gate holds the filing at Not Filing-Ready, and it is right to. I have watched people want the opposite, want a clean-looking brief to earn a green light on its own. That is the exact instinct that gets a lawyer sanctioned.

That certificate matters more to me than any accuracy percentage, and it is why the deterministic and non-deterministic pieces have to stay separated. Reproducibility is what makes a decision certifiable. I can hand you the receipt. This cite was checked against CourtListener, the volume does not exist, verdict fabricated, gate holds. You can rerun it and get the identical result. An LLM judge, even a good one, cannot promise you that, and I lived through the evening of three runs and three different answers. I am not building a compliance record on top of a coin flip.

I will also say plainly, because the standard I hold demands it, that this is a verification layer and not a research tool. It does not draft briefs or find cases. It does not compete with Harvey or Westlaw or Lexis. It sits around whatever they produce and tells the attorney what is safe to file and exactly what needs a human. The full-text grounding beyond the public opinion snippet needs a free CourtListener token, the document-management connector in the demo is a mock, and a few features like a PDF certificate are deferred. What is real is the mechanism: the checks, the abstention, the deterministic gate, and the audit trail, all running against real case law.

Isn't abstention just the model dodging the hard call?

I get some version of this question in almost every conversation, usually from an engineer, and my answer has gotten shorter and more certain over time.

No. Abstention is the hard call, made honestly, and refusing to make it is the actual dodge. The seductive alternative is a system that always returns a crisp verdict, green or red, on every citation. It demos beautifully. It is also lying, because some citations genuinely cannot be adjudicated from the available text with confidence, and a system that never says so is manufacturing certainty it does not have. The most valuable thing a high-stakes AI can do is tell you where its own judgment runs out. In a domain where being confidently wrong gets your client sanctioned, a calibrated "I am not sure, a human needs to look at this" is worth more than a fluent guess.

This is why I think the work outlives the current model generation. The industry keeps promising the hallucination problem away with the next model. Grant all of it, bigger context, cleaner training, a lower fabrication rate. A perfect model that never invents a case will still cheerfully cite a real one for a proposition it does not support, because from inside the draft that sentence reads as true and exactly what you asked for. The gap between "this case exists" and "this case supports my claim" is not a gap that scale closes. It is a governance question, and governance is a property of the system you build around the model, not a capability you wait for the model to grow.

A court does not care how confident your model was. It cares whether the citation exists and says what you claimed. That is not a bigger-model problem. It is a build-the-right-layer problem.

The regulators already see it this way, which is worth sitting with. ABA Formal Opinion 512 and more than 300 judicial standing orders now require lawyers to verify AI output before filing, and fewer than 20 percent of firms have any AI-use policy at all. The obligation is not "use a smarter tool." The obligation is "prove you checked." Proof, provenance, and honest abstention are not things a better drafter gives you. They are things a verification layer gives you, and they hold at any model quality, which is precisely why I think they are the durable part.

The question I keep before I file

I found that the thing this build changed in me was smaller than the thesis, and it has lasted longer.

I stopped asking whether an AI-written claim is true, because I can often answer that and it turns out not to be enough. The Erie citation was, in a narrow sense, pointing at a real case. The harder question, the one I now ask before anything AI-drafted leaves my hands, is whether I can prove, right now, that this exact source supports this exact claim, and whether that proof would survive someone who wanted it to fail in front of a judge.

That is a governance question, not a model question. It does not get easier as the model gets better, because the thing being asked is not "can the model write a better sentence" but "can you stand behind this one." And the most useful answer my own system ever gives me is not the green check. It is the moment it holds a real case at Needs Review and makes me, a person, go read the opinion. If you want to see where I landed, it is here once more: veriprajna.com/demos/legal-ai-citation-verification.

And if you would rather watch it than read me describe it, here is the whole thing running end to end, in my voice.

So the question I would leave you with is the one that reorganized the whole build. When the AI hands you something confident and clean, do you have a layer that is willing to say "I am not sure, a human needs to look"? Because confidence is cheap, and in the work that actually carries risk, provable abstention is the product.