The Verification Layer Is a Smoke Alarm, Not a Building Code

There is a sentence you have heard a hundred times if you work anywhere near legal technology, usually delivered with a shrug and total confidence:

"Just verify the citations."

The fake-case problem is real. Lawyers have been sanctioned for it. So the fix sounds obvious: run the draft through a checker, catch the fake cases, ship. Done.

It's also the wrong answer, and not in a small way. It's wrong in the way a smoke alarm is the wrong answer to "how do we build a building that doesn't burn down." The alarm is genuinely good. Every building should have one. But no building was ever built by a smoke alarm, and if you treat the alarm as the architecture, you will eventually die in a fire you were told was impossible.

Let me make that argument carefully, because it matters.

First, the concession

Everything the verification camp says is true. In 2023, a New York federal judge sanctioned a law firm $5,000 after two attorneys filed a brief citing six ChatGPT-invented cases, complete with fake quotes and made-up docket numbers. It hasn't stayed small. A researcher at HEC Paris keeps a public database of court decisions involving AI-hallucinated material; it passed 1,600 cases this year and the pace is climbing. Penalties are climbing too: this March, the Sixth Circuit fined two attorneys $15,000 each, plus fees and double costs, for briefs stuffed with fabricated and misrepresented citations.

So yes. A firm-wide tool that catches a fake citation before it reaches a judge is a genuinely good control. Nobody serious disputes that verification catches things.

The dispute is about what it structurally cannot catch. And that's where cases are actually lost.

The checker can only grade what's on the page

Here's the core problem, and it's almost embarrassingly simple once you see it.

A citation verifier reads the document. It checks the citations that made it onto the page. Which means it can never, ever check the citation that isn't there.

Say the controlling adverse case in your circuit exists. Say it directly defeats your argument. But the AI never retrieved it, so it never appeared in the draft. The verifier scans the brief, finds every citation present and correct, and stamps it green. The document "passed." Your client just lost.

This isn't a rare edge case. It's the entire game. A brief can fail in a dozen ways, and fabrication is only one of them:

• The missing case. The adverse authority was never retrieved, so there's nothing to check.
• The wrong standard. A brief cites a real case, quotes it accurately, and applies the wrong procedural standard entirely. Every citation is real. The argument is still dead.
• The uncited assertion. A claim stated with no citation at all. A citation checker has nothing to check, so it passes trivially.
• The wrong tool. Beautiful, valid authority for a preliminary injunction, marshaled in a motion that needed something else entirely. All cites check out.

A citation verifier is an excellent detector of fake cases and a null detector of everything else. It grades the footnotes. Cases are lost in the argument, the posture, and the case that was never pulled.

The math is quietly brutal

Even on the narrow thing it can do, verification scales badly.

A real brief cites maybe 35 propositions. Say your verifier is pretty good and misses only 2% of the actual defects. The chance it catches every single one across the whole brief is 0.98 raised to the 35th power. Roughly a coin flip. At a 5% miss rate, about 83 out of 100 briefs ship with at least one uncaught problem.

Flip it around: to make 19 of 20 briefs fully clean, the verifier has to be about 99.85% accurate per citation. On judicial prose. Judging whether a case really supports a sentence. That is a brutal bar, and there's a bonus failure hiding behind it: verifiers also flag good citations as bad. Strip a couple of valid authorities out of every filing, quietly, and the drafts get weaker in ways no error log will ever show you.

Detection has to win 35 times in a row. Construction never plays the game. If your system never lets the model originate a citation in the first place, so every authority traces back to an actual retrieved record, the fabrication rate is zero. Not 99.85%. Zero. Regardless of length.

The perverse incentive

Now make the verifier an inline gate instead of an after-the-fact audit, and watch Goodhart's law arrive on schedule: when a measure becomes a target, it stops being a good measure.

Make "passes the citation checker" the target, and rational writers optimize for it. The safest sentence is one with no citation attached, because an uncited claim cannot fail a citation check. So citation density falls. Hedged, unsupported assertion rises. The writer drifts toward claims that are easy to support and away from claims that are hard and necessary.

The perverse equilibrium is plain: the cheapest way to pass a citation verifier is to cite less. A brief can pass every check by making fewer claims. That's not rigor. That's the metric optimizing itself while the argument quietly dies.

Who verifies the verifier?

Here's the part nobody wants to sit with.

A verifier that decides whether a case "supports" a proposition is not an oracle. It's a model with its own error rate, judging another model's output. And we know what those disagreements look like, because the human legal world has been running this experiment for decades.

When researchers compared Shepard's and KeyCite, the two legacy citators that lawyers have trusted for a generation, they found only about 33% overlap in their results. Another study looked at 357 citing relationships flagged as negative by at least one of three citators: all three agreed the treatment was negative in only 53 of them. The experts disagree with each other most of the time, and we're going to lay a binary green stamp over that disputed ground and call it verification?

A verifier that won't publish its own measured error rate is demanding the trust it was built to deny the generator. A label that hasn't earned the right to exclude can only rank. The moment it silently filters, it's deleting relevant law invisibly, which is arguably the worst failure mode in legal research, because nothing in the output ever reveals what was removed.

The scariest part: it makes humans worse

In 1983, Lisanne Bainbridge wrote a legendary paper called "Ironies of Automation." Her observation: automate the easy part of a task, and the human operator gets worse at the hard part. Decades of follow-up research confirmed it. Operators of consistently reliable automation become measurably worse at catching the failures it does produce, and practice doesn't cure it.

Now plug in a citation verifier that catches fake cases reliably and is structurally blind to wrong standards, missing authority, and wrong instruments. It doesn't just leave those gaps uncovered. By stamping the document "checked," it actively erodes the human scrutiny that used to cover them. The alarm that never misses a fake case is exactly the alarm that teaches you to stop reading the real ones.

A partial control experienced as a total control can make net risk worse. That's not a hot take. It's 40 years of human factors research.

Courts aren't asking "did it pass"

Finally, notice what the profession is actually starting to demand. Judges now issue standing orders about generative AI. Insurers are asking firms how their work is produced. The Sixth Circuit's principle in the recent sanctions was tool-agnostic: counsel must personally read and verify every citation, however it was generated.

The question after the fact is increasingly not "did it pass a check." It's "how was it made." An attestation says this document passed at time T. A record says what question was asked, what was searched, what was retrieved, what was rejected and why, and what a human reviewed. A post-hoc verifier can only ever produce the first. Trust is a property of a process, and a process leaves a record. A stamp at the end leaves a stamp.

So where does verification belong?

Everywhere, honestly, and I mean it. A firm-wide, model-agnostic checker that catches a fabricated citation before it reaches a judge is a legitimate and valuable control. It is exactly the right response to the ungoverned population that produced most of the sanctioned filings.

Call it what it is: a smoke alarm. Every building should have one, and a building without one is negligent.

But the alarm on the wall is not the sprinkler system, and it is not the fire door, and it is not the foundation. It tells you something is already burning. It does not, and never did, build anything.

"Just verify it" is not a strategy. It is the sound a market makes when it wants the problem to be cheap. The real fix is architectural: never let the model originate a citation at all, so every authority in the document traces to a record rather than a prediction. That's harder to build, harder to demo in a screenshot, and it's the only thing that actually changes the odds.

Every building needs a smoke alarm. No building was ever built by one.

Sources for the curious

• Mata v. Avianca sanctions order (S.D.N.Y. 2023)
• Whiting v. City of Athens (6th Cir. 2026)
• Damien Charlotin, AI Hallucination Cases Database
• Magesh et al., Journal of Empirical Legal Studies (2025)
• Dahl et al., Journal of Legal Analysis (2024)
• Mart, Legal Reference Services Quarterly (2013)
• Hellyer, Law Library Journal (2018)
• Bainbridge, "Ironies of Automation," Automatica (1983)
• Parasuraman and Manzey, Human Factors (2010)
• ABA Formal Opinion 512 (2024)