New on SSRN: Ariadne's Thread, a measurement-theoretic method for legal openness. Read the paper
Legal AI System Design

Opportunistic reliability

A research system is opportunistically reliable for a class of error when it gets the answer right only if the material a given query happens to retrieve happens to carry the warning.

First used in The Promise Fulfilled

Opportunistic reliability is real and often substantial. It is also unstable in a specific way: the same tool can be right on Monday and incomplete on Tuesday for the same question, and nothing in the output reveals which day it is.

The deference cases show the difference. Loper Bright's overruling of Chevron saturates recent text, so nearly every tool passes, and passing proves salience rather than method. Kisor's narrowing of Auer is recorded almost nowhere as a signal, so a tool that flags it either keeps a record in which such shifts can be written down or was lucky in what it retrieved.

Agentic web search can turn some failures from systematic into opportunistic. It cannot make them structural, because the missing field is not in any document to be found.

Related terms

Related frameworks

Used in