Implicit Responsiveness in eDiscovery

Why it matters

Document review is commonly estimated at 50 to 90 percent of litigation cost, and technology-assisted review is accepted by courts in appropriate cases. What makes a production defensible is how much responsive material is left behind. If automated review reliably finds explicit documents and systematically misses implicit ones, the documents it misses may be the ones that matter most, and a recall figure averaged over the corpus will hide the pattern.

State of the art

The empirical work of Grossman and Cormack and the evaluation discipline of the TREC Legal Track established that technology-assisted review can be more effective and more efficient than exhaustive manual review, and continuous active learning changed how review proceeds. Case law has moved from approving the method to disputing its control and validation. Human reviewers themselves agree only about 70 percent of the time on borderline documents, which limits how precisely implicit responsiveness can even be labeled.

Our partial answers

The blind review of 8,665 documents measured the gap directly. Overall recall was 0.8071, but recall by signal clarity was 0.933 for clear documents, 0.684 for marginal ones and 0.452 for borderline ones, with misses concentrated in chats, meeting minutes and executive presentations. The elusion gate failed, with an upper bound of 6.68 percent responsive in the rejected pile against a 5 percent target. We published the failure and limited the supported use to automated classification with mandatory human review of the machine-produced pile.

Open: rubric designs that capture implicit relevance; sampling stratified by signal clarity, so that averages cannot hide the hard cases; aggregation across threads and document families that carries context to fragmentary messages; and a validation standard that reports recall on the hard strata separately.