The verdict I was sure my checker would nail, and it refused to give one
I fed my own verifier a brief that cited Erie Railroad v. Tompkins, 304 U.S. 64, for the existence of a uniform federal common law, which is close to the exact opposite of what Erie actually holds, and I sat there waiting for it to light up red and name the lie. It did not. It returned "Needs Review," marked the support-check "Uncertain," and routed the citation to a human. My honest first reaction was that I had broken something. The case is real, the proposition is wrong, and I wanted a machine that says "wrong."
What I had actually built was a machine that knows the difference between "this case does not exist" and "this real case may not stand for what you claimed, and I am not the one who should make that call." Erie is a genuine Supreme Court decision. Whether a specific opinion supports a specific sentence in a motion is an act of legal judgment, and the honest answer, when the opinion text does not plainly establish the proposition, is to abstain and hand it to an attorney with the contradicting context attached. The screen below is that exact run. The console reads "model advises, deterministic gate plus human decide," and the citation lands in Needs Review instead of a confident red I could not actually stand behind.
The Misapplied Precedent run. Erie is a real case cited for the wrong proposition, so the support-check returns Uncertain and the citation is routed to Needs Review rather than a confident red verdict. Support-Check is the only non-deterministic stage in the pipeline; everything else, including the gate, is deterministic.
Why I split the thing into two engines the same week
I stopped trusting any single mechanism to make both calls the afternoon I watched two different failures sit next to each other. One brief cited Halstead v. Ferngate Holdings, 823 U.S. 1199, a case that does not exist, because the United States Reports has no volume 823. The other cited Erie, which is real, for a proposition it does not support. Those are not the same kind of wrong, and I did not want them handled by the same kind of certainty.
So the system is deliberately two engines. Existence and fabrication checking, plus the policy gate, are deterministic Python running against real case law. Halstead gets caught by a live CourtListener lookup that returns zero results, which is proof of absence, not a guess. That half carries a guarantee I am willing to write down. The support-check, the question of whether a real opinion actually backs the sentence, is a single LLM step, and its defining feature is that it abstains. It advises. The deterministic gate and a human decide. I had already learned not to let a model be the final judge of its own output, so I gave the judgment I could prove to code and the judgment I could not to a person. The scoreboard below is one brief run through both engines: one citation verified, one fabricated and caught, one routed to human review, one honestly marked outside coverage, and the filing held.
One brief, four fates. Miranda clears green, Halstead is caught as Fabricated, Brown is honestly marked Outside Coverage rather than falsely verified, and Twombly is routed to Needs Review. The gate reads Do Not File until every citation resolves.
The one citation it did clear green, and why I needed it to
I worried for a while that a verifier built to abstain would just abstain on everything, which would make it useless theater, so the case I cared most about was the one it cleared. On the clean sample brief, Miranda v. Arizona, 384 U.S. 436, is cited for the rule that a suspect in custodial interrogation must be warned of the right to remain silent and to counsel. The support-check grounded against the real Miranda opinion on CourtListener, found the passage where the Court states that exact holding, and returned "Supported." The citation went green as Verified.
That mattered to me because abstention is only honest if the tool is also willing to commit when the opinion earns it. A checker that says "get a human" to everything is not careful, it is broken. Miranda clearing green, from the opinion's own words rather than the model's confidence, is what makes the "Needs Review" on Erie mean something. One is the system saying "the text establishes this," the other is it saying "the text does not, so do not let me pretend." Both are the opinion talking, not the model's mood.
The clean run clearing Miranda green. The support-check reads the real opinion, finds the Court stating the custodial-interrogation holding, and returns Supported. The verdict comes from the opinion text, not from the model deciding it feels confident.
What I made the certificate admit out loud
I set myself one rule for the audit certificate: it had to be readable by someone who does not trust me. So it does not summarize, it itemizes. On a custom brief I pasted in, three citations came back with three different fates. Terry v. Ohio, 392 U.S. 1, and Strickland v. Washington, 466 U.S. 668, both real, were routed to Needs Review with an Uncertain support-check. A third, 921 U.S. 77, was marked Fabricated, because no such decision exists. The gate result across the top: Not Filing-Ready. Do Not File.
The signed certificate for a custom brief. Two real cases routed to review, one fabricated cite caught, and the gate holding at Not Filing-Ready. The certifying language admits its own boundary: fabrications are caught to a certainty, judgment calls go to a human.
The certificate says, in plain language, that each citation was checked for existence against the CourtListener case-law database, that fabricated citations are identified to a certainty, and that citations requiring contextual judgment are routed to human review. That last clause is the part I am proudest of, because it is the tool admitting its own boundary on the record. The policy gate is deterministic and blunt: filing-ready only if zero citations are fabricated and zero are unresolved. This is the layer around whatever model drafted the brief, and it exists because ABA Formal Opinion 512 and more than 300 judicial standing orders now require a lawyer to verify AI output before filing. A signed record that says what was proven and what still needs a human is what that verification looks like when you have to hand it to a court.
Why I stopped waiting for a smarter model to fix this
I used to assume this whole problem was temporary, that a good enough model would eventually stop inventing cases and make the verifier unnecessary, and building this is what talked me out of it. Better models do hallucinate less, and I am glad of it. But even purpose-built legal tools still fabricate. Stanford's RegLab measured Westlaw Precision at 33 percent and Lexis Plus at 17 percent hallucination rates (Journal of Empirical Legal Studies, 2025). The consequences are already on the docket: 1,222 documented court cases involving AI-hallucinated citations by early 2026, and the Sixth Circuit imposing 30,000 dollars in sanctions in March 2026. Fewer than one in five firms has an AI-use policy at all.
None of that is a model-quality problem you can train away, because filing-ready is a governance property, not a model property. A court does not care how confident the drafter was. It cares whether the citation exists and says what you claimed. That answer lives in the layer around the model: the deterministic existence check, the gate, the signed certificate, and the willingness to abstain. I hold the demo to a narrow, honest set of claims in return. Fabrication catching is deterministic and exact against real case law, and no fabricated citation can pass the gate, an invariant I unit-tested. The rest is demonstrated at the scope I built: 20 real cases cached from CourtListener, eight of eight core tests passing. The support-check is one LLM step that abstains, not a promise that every real cite grades correctly. If you want to run a brief through it yourself, the legal AI citation verification demo is live.
So the question I keep coming back to, and the one I would put to any litigator shipping AI-drafted work, is not whether your model is good. It is this. When it hands you a citation, do you know whether you are looking at a fact it proved or a guess it dressed up as one, and can your tool tell you which, in writing?