The Smoke Alarm Problem

A proposal for legal practice should be judged the way any rule should be judged: not by whether it can be stated coherently, but by what ordinary institutions will do with it under deadline pressure and with incomplete information. "Just verify the citations" passes the first test easily. It is short, it sounds responsible, and it answers a real problem. The question is what it does on Monday morning.

The concession, which is substantial

Start with what is not in dispute. Fabricated citations are real, and they are expensive. In 2023 a federal judge in New York sanctioned a firm $5,000 after two of its attorneys filed a brief citing six cases that ChatGPT had invented, fake quotations and docket numbers included. The problem did not stay small. A researcher at HEC Paris keeps a public database of court decisions involving AI-hallucinated material; this year it passed 1,600 entries, and the pace is climbing. So are the penalties. In March of this year the Sixth Circuit fined two attorneys $15,000 each, plus fees and double costs, for briefs full of fabricated and misrepresented citations.¹

A firm-wide tool that catches a fake case before a judge does is therefore a good control. Nobody sensible argues otherwise. The argument is about what that tool structurally cannot catch, and about how a firm behaves once it owns one.

Monday morning

Picture an ordinary firm that has bought a citation verifier and made it policy: nothing is filed without a green check. A motion is due Wednesday. On Monday an associate drafts it with an AI assistant, runs the draft through the verifier, and gets a clean result. Every cited case exists. Every quotation matches. The supervising partner, carrying four other matters that week, sees the green check and reads for tone. The brief goes out.

Now consider what the verifier actually examined. It read the document and tested the citations that appeared in it. That is all it can do. Here are four briefs that pass that test and still lose.

• The missing case. Controlling adverse authority in the circuit exists and defeats the argument. The drafting tool never retrieved it, so it never reached the page, so there was nothing to check. A verifier cannot grade a citation that is not there.
• The wrong standard. The brief cites a real case, quotes it accurately, and applies the procedural standard from the wrong stage of the litigation. Every citation is genuine. The argument is dead on arrival. The Institute's series on filing-grade research treats this failure at length in Right Law, Wrong Stage.
• The uncited assertion. A claim stated with no authority at all. A citation checker has nothing to test, so the sentence passes by default.
• The wrong instrument. Valid, well-chosen authority for a preliminary injunction, marshaled in a motion that needed something else entirely. Every citation verifies.

Add violations of procedural requirements, which live outside the citation-and-sentence pair altogether, and the pattern is plain. A citation verifier is an excellent detector of invented cases and a null detector of nearly everything else that loses motions. It audits the footnotes. Motions are lost in the argument, the posture, and the authority nobody pulled.

That is the first institutional fact about Monday morning. The second is what the green check does to the partner, and we will come back to him.

The arithmetic of a brief

Even within its own lane, verification scales badly, and the reason is multiplication.

Suppose a brief rests on 35 cited propositions and the verifier misses 2 percent of the real defects. The probability that it catches every defect in the brief is 0.98 multiplied by itself 35 times, which is about 0.49. Call it a coin flip. At a 5 percent miss rate, roughly 83 of every 100 briefs go out with at least one uncaught problem.

Run the question backward. For 19 of 20 briefs to be fully clean, the verifier must be right about 99.85 percent of the time, per citation, on judicial prose, deciding whether a real case actually supports a particular sentence. That is a demanding standard for any reader, human or machine.

There is a second cost that never appears in the arithmetic, because nobody logs it. Verifiers also flag good citations as bad. An associate facing a red flag late on Tuesday does not litigate with the tool; she cuts the citation. Remove a couple of valid authorities from every filing, quietly, and the drafts get weaker in a way no error report will ever display.

Detection has to win 35 times in a row. Construction does not play that game. If the drafting system is never permitted to originate a citation, so that every authority traces back to a retrieved record, the fabrication rate is zero at any length. Not 99.85 percent. Zero. That is a property of how the brief was made, not of how carefully it was inspected afterward.

What the gate teaches

Now move the verifier from an after-the-fact audit to an inline gate that every draft must pass. This is where anyone who has studied incentives should start paying attention.

Goodhart's law predicts the result: when a measure becomes a target, it stops being a good measure. Make "passes the citation checker" the target and rational people will optimize for it. Nobody has to be dishonest. The associate has a deadline and a tool that punishes citations that might fail. The sentence least likely to fail a citation check is the sentence with no citation. So citation density falls. Hedged, unsupported assertion rises. The writer drifts toward claims that are easy to support and away from claims that are hard and necessary.

The equilibrium is perverse and entirely predictable. The cheapest way to pass a citation verifier is to cite less, and a brief can clear every check by making fewer claims. The metric improves while the argument deteriorates, and the dashboard reports progress.

Who checks the checker

A verifier that decides whether a case "supports" a proposition is not an oracle. It is a model with an error rate of its own, grading another model's work. The profession has been running a long experiment on how reliably such judgments converge, and the results are not reassuring.

Researchers comparing Shepard's and KeyCite, the two citators lawyers have trusted for a generation, found only about 33 percent overlap in their results. Another study examined 357 citing relationships that at least one of three citators had flagged as negative; all three agreed the treatment was negative in only 53. When the established tools disagree that often, a binary green stamp laid over the same contested ground asserts more confidence than the underlying judgment can bear. That violates the Zeroth of the Four Laws of System Design, which forbids presenting an output with unwarranted confidence, and it does so in the name of safety.

Two institutional consequences follow. A verifier that will not publish its own measured error rate is demanding exactly the trust it was bought to deny the generator. And a label that has not earned the right to exclude should only rank. The moment a verifier silently filters, it deletes relevant law invisibly, which is among the worst failures legal research can have, because nothing in the output reveals what was removed.

The partner who stopped reading

Return now to the partner who read for tone. In 1983 Lisanne Bainbridge published "Ironies of Automation," observing that when the easy part of a task is automated, the human operator gets worse at the hard part. Decades of later work, including Parasuraman and Manzey's in Human Factors (2010), found the same thing: people who operate consistently reliable automation become measurably worse at catching the failures it does produce, and practice does not cure it.

Apply that to the firm. A verifier that reliably catches fake cases, and is structurally blind to wrong standards, missing authority, and wrong instruments, does not merely leave those gaps open. By stamping the document "checked," it erodes the human scrutiny that used to cover them. The alarm that never misses a fabricated case is the alarm that teaches a busy partner to stop reading the real ones.

A partial control experienced as a total control can leave a firm carrying more net risk than it did before it bought the control. That is not a provocation. It is forty years of human factors research applied to a new workplace.

What courts and carriers are asking

The institutions on the other side of the filing are moving in a different direction. Judges now issue standing orders on generative AI. Insurers ask firms how their work is produced. The Sixth Circuit's principle in the recent sanctions was tool-agnostic: counsel must personally read and verify every citation, however it was generated.

After the fact, the operative question is shifting from "did it pass a check?" to "how was it made?" Those call for different records. An attestation says a document passed a test at a given time. A derivation record says what question was asked, what was searched, what was retrieved, what was rejected and why, and what a human reviewed. A post-hoc verifier can produce only the first. Trust is a property of a process, and a process leaves a trail. A stamp at the end leaves a stamp.

Where the alarm belongs

None of this is an argument against verification. Put a firm-wide, model-agnostic checker everywhere. It is a legitimate and valuable control, and it is exactly the right response to the ungoverned drafting that produced most of the sanctioned filings. A firm without one is negligent.

But call it what it is. It is a smoke alarm. It tells you something is already burning. It is not the sprinkler system, the fire door, or the foundation, and it never built anything. The building code for AI-assisted drafting is architectural, and the original essay states it as seven principles:

1. Generators never originate citations.
2. Citation stages contain no models, and model stages emit no citations.
3. Guarantees live in code, not in prompts.
4. The system searches adversarially for authority that would defeat its own position.
5. Labels earn the privilege of excluding results only after blind audits.
6. The system can halt without producing a confident output.
7. A derivation record accompanies the filing.

Each of these is a statement about how the document is made, and therefore about what a firm can show when someone asks. Several reach failures a verifier cannot see at all. Adversarial search goes looking for the missing case. The ability to halt is the Zeroth Law in operational form. The derivation record is what courts and carriers are beginning to ask for.

What would change the verdict

A practical position should say what evidence would overturn it. This one names three conditions. If verifiers published their precision and recall on public adversarial benchmarks, the objection about checking the checker would weaken considerably. If firms produced evidence that inline verification does not reduce citation density, the incentive objection would lose its force. And if someone developed a method for auditing missing authority from the finished document alone, the most serious structural limit on verification would fall. None of these is impossible. None has yet been shown.

Until then, the sensible Monday morning policy is not complicated. Keep the alarm. Test it. Do not let it change how carefully anyone reads. And spend the serious money on how the brief is built, because that is where the odds actually move.

Install the alarm. Then hire an architect.

Footnotes

1. Sources cited in the original essay: the Mata v. Avianca sanctions order (S.D.N.Y. 2023); Whiting v. City of Athens (6th Cir. 2026); Damien Charlotin's AI Hallucination Cases Database; Magesh et al., Journal of Empirical Legal Studies (2025); Dahl et al., Journal of Legal Analysis (2024); Mart, Legal Reference Services Quarterly (2013); Hellyer, Law Library Journal (2018); Bainbridge, "Ironies of Automation," Automatica (1983); Parasuraman and Manzey, Human Factors (2010); ABA Formal Opinion 512 (2024). ↩