We Ran a Blind eDiscovery Review with No Humans in the Loop

In brief

Document review dominates the cost of litigation. The case study puts it at 50 to 90 percent of litigation cost, with human reviewers processing roughly 45 documents an hour and agreeing with one another on borderline documents only about 70 percent of the time. Against that background, Legawrite.AI open-sourced isResponsive, a TAR 3.0 engine, under the Apache 2.0 license, together with its pipeline architecture, execution logs, metrics, failure analysis and reproducibility records. Installing and running it takes three commands:

pip install legawrite-tar
legawrite init <matter-dir>
legawrite run <matter-dir>

The method

The engine is built around controls that a court could inspect. A locked rubric of 20 responsiveness criteria is frozen by SHA-256 hash before classification starts. Filtering happens in two stages: an embedding pre-filter, then classification by a language model. Every decision cites the source text and carries a calibrated confidence. Privilege screening is deterministic and routes flagged documents to attorneys. The run ends with a defensibility certificate that records the sample design, the statistical bounds and a cryptographic validation.

The blind run

Batch 4 was run on the Gibson v. Meta corpus of 8,665 documents under blind conditions: a clean node, a hidden answer key, opaque filenames, thresholds fixed before execution and a 300-document calibration set. The embedding stage (Qwen3-Embedding-0.6B) removed 1,845 documents, 21.3 percent of the corpus, at a cosine threshold of 0.26 in under two minutes. Classifying the remaining 6,820 documents with a quantized Qwen3-14B on two workstation GPUs took just under four and a half hours.

The headline numbers: recall 0.8071 (95 percent confidence interval 0.789 to 0.824), precision 0.7763, F1 0.7914 and accuracy 0.9038, from 1,582 true positives, 456 false positives, 378 false negatives and 6,249 true negatives. Three blind runs produced recall of 0.810, 0.804 and 0.807. A replication on 3,665 documents carried over from an earlier batch matched the prior precision (0.8326 against 0.832) and recall (0.8017 against 0.804).

What failed, in public

The report's most useful section is the one most vendors would leave out. The elusion gate failed: the rejected pile turned out to be 5.7 percent responsive, with an estimate of 4.8 percent and an upper bound of 6.68 percent against a 5 percent target. The misses clustered in implicit responsiveness, the chats, meeting minutes and executive presentations where relevance is never stated outright. Recall tracked how clearly a document signaled its relevance: 0.933 for clear documents, 0.684 for marginal ones and 0.452 for borderline ones. The calibration gate failed too, with expected calibration error rising to 0.088 from 0.046 in Batch 3. The system responded as designed: it locked auto-accept and routed all 2,038 responsive determinations to human review.

Economics and the defensible posture

The whole run cost about $5 in compute for 8,665 documents, against $0.50 to $1.00 per document for human first-pass review. The case study is careful about what that buys today. Supported now: automated classification followed by mandatory human review of the machine-produced pile, with recall clearing the 75 percent defensibility floor even at its lower confidence bound of 0.789. Not supported: fully automated production at a 5 percent elusion standard. Attorneys still approve rubrics, handle privilege, validate production and keep their Rule 26(g) signature responsibility. The author's summary of why the failures were published is short: "Openness is not marketing. It is the mechanism of trust."

The full methodology behind the engine, including the validation stopping rule, the privilege screen and a point-by-point comparison with a court-entered validation order, is set out in the SSRN paper isResponsive: An Open TAR Methodology for Court-Defensible Responsiveness Review. Ross Brodskiy founded Legawrite.AI, which built and released the engine.

This page is a summary, not the case study itself. Read the full We Ran a Blind eDiscovery Review with No Humans in the Loop on Medium, where Ross Brodskiy first published it on 10 July 2026.