Treatment Label Disagreement Among Citators

Why it matters

Currency checks rest on treatment labels. If the labels are unstable, "good law" is less settled than a green flag suggests, and any system that filters authority by label inherits the instability. For machine systems, which may exclude authority automatically, disagreement becomes a source of invisible deletion.

State of the art

The empirical literature is small and not reassuring. Reviewing 357 citing relationships that at least one citator had labeled negative, Hellyer found that Shepard's and KeyCite each missed or mislabeled about a third of the negative relationships and BCite more than two thirds, and that all three agreed negative treatment existed in only 53 (Law Library Journal, 2018). Kirschenfeld found the services' vocabularies for negative treatment applied inconsistently, with categories lacking stable definitions. On the retrieval side, Mart showed that the same search run in six legal databases returned largely different top results: on average about 40 percent of cases were unique to one database, and only about 7 percent appeared in all six.

Our partial answers

The Case Treatment Classification Guide specifies treatment by scope, severity and mechanism; attaches it to holdings rather than whole cases; separates distinguished, which is factual and neutral, from limited, which narrows a holding as a matter of law; and adopts an explicit rule for doubt: over-flag rather than under-flag. The verification program adds a design rule: a label may rank authority but may not silently exclude it until a blind audit has measured and published its error rate.

Open: a public, adjudicated reference set of treatment relationships; agreement measures for each category, so that low-agreement categories are reported as such; and a way to represent contested treatment as contested, rather than forcing a single label onto disputed ground.