Why Retrieval Is Not Reasoning

The dispute over retrieval-augmented generation in legal AI is usually conducted as a verbal disagreement. One side says its answers are "grounded." The other says they hallucinate. Both statements can be true of the same system, which suggests the vocabulary is doing less work than it appears to.

A more productive approach is to treat a legal research system as what it is: a decision procedure. Every such procedure has four components. It takes inputs, represents them under some set of rules (a schema, a type system, an ontology, or no schema at all), runs an inference procedure over that representation, and emits outputs. The useful question is not whether a system "uses AI" or "cites sources." It is where, in that pipeline, the legally operative properties of an authority are represented, and which component is responsible for getting them right.

Stated that way, the case against retrieval-augmented generation (RAG) as an architecture for legal reasoning becomes a narrow and testable claim. In a chunk-retrieval pipeline, the properties that determine what a holding means for a client are never represented as inputs. They are expected to be recovered at query time, by inference, by a general-purpose language model that has no formal model of what a holding is. Whatever reliability the system achieves is therefore opportunistic. This essay walks the pipeline stage by stage to show where that happens, and what changes when the unit of representation is the legal proposition rather than the text chunk.

Definitions

Six terms carry the argument.

1. Retrieval unit. The object a system indexes and returns. In a standard RAG pipeline it is a chunk, usually 512 to 2,000 tokens, cut from a longer document by a text splitter.
2. Proposition. A self-contained statement of a legal rule: the operative rule a court announced, in language a lawyer could cite. A holding stated this way typically runs 15 to 50 words.
3. Stance. The direction in which a proposition cuts: which party it favors, and whether it expands or contracts liability.
4. Origin. Whether the court established the rule or applied a rule established earlier.
5. Category. The functional role of the proposition: liability, damages, defenses, causation, procedure.
6. Structural and opportunistic reliability. Following The Promise Fulfilled, a system is structurally reliable for an error class when its stored representation makes that class of error impossible or reliably detectable. It is opportunistically reliable when it is correct only if retrieval happens to surface the missing context, with no guarantee for the less documented instance.

Jurisdiction and position in a doctrinal taxonomy complete the list of properties a lawyer needs. None of them is exotic. Each is something a competent associate determines, usually without noticing, while reading an opinion.

Stage one: inputs

The first design decision in any retrieval system is the retrieval unit, and in RAG it is made by a component that knows nothing about law. A splitter divides a forty-page appellate opinion into fragments of fixed size. It has no representation of the difference between a holding and a recitation of facts, between the court's analysis and its summary of the losing party's argument, or between the rule and the standard-of-review boilerplate that precedes it.

Consider a hypothetical opinion, call it Smith v. Jones, whose holding is a single sentence: "a hospital's duty of care does not make it an insurer of patient safety." That sentence is perhaps twenty words in an opinion of ten thousand. When the chunk containing it is embedded, the vector is dominated by everything around it: procedural history, the facts of the injury, discussion of competing standards. The embedding faithfully records what the chunk is about, negligence in a hospital setting. It does not record what the holding says.

That is the retrieval unit problem, and it is a problem of inputs rather than of search quality. Similarity search over topic vectors optimizes topical relevance. A lawyer building an argument needs propositional precision: the specific rule, its direction, and its authority. No improvement in the similarity function recovers information that the unit of indexing never contained.

Stage two: rules

The second stage asks what the representation can express. For a chunk, the answer is text and position. For legal reasoning, at least five predicates are needed, and none of them is computable from the chunk as stored.

• is_holding(x): is this sentence the holding, or dicta, or the court restating an argument before rejecting it? To a splitter these are indistinguishable.
• stance(x): which party does the rule favor? A rule that landlords can be liable for tenant injuries and a rule that they cannot are topically identical and doctrinally opposite. They embed in the same neighborhood of vector space.
• origin(x): did this court establish the rule, or apply it? A chunk from the forty-seventh case applying a rule does not reveal that the rule was established in the twenty-third, and the twenty-third is the one a lawyer needs.
• category(x): what does the rule govern? A sentence about foreseeability may concern duty, causation, or the recoverability of a particular loss. Semantic similarity does not discriminate among legal categories.
• jurisdiction(x) and hierarchy(x): where does the rule bind, and at what level? A New York appellate decision and a California Court of Appeal decision may announce nearly identical rules; only one of them matters to the question asked.

The Promise Fulfilled names the failures that follow when these predicates are missing, among them directional opacity (the system cannot tell which way a holding cuts) and authority-health blindness (it cannot tell whether an authority is still sound). The terminology is useful because it locates the failure correctly. These are not retrieval misses. They are representation gaps. The schema has no slot for the property, so no search over the schema can return it.

The contrast is a representation in which the proposition is the atomic unit and the predicates are fields. An illustrative record, not any particular system's schema:

proposition: "A hospital's duty of care does not make it an insurer of patient safety."
status: holding          # not dicta
stance: defendant        # contracts liability
origin: established      # this court announced the rule
category: liability
jurisdiction: [court and level]
citation: [verified against the reporter]

Once the predicates are fields, a query can be a conjunction of them. "All holdings on punitive damages standards from California courts that favor defendants" is not a topic. It is four typed conditions: the category is damages, the subject is punitive damages, the court sits in California, and the stance makes recovery harder. Chunk retrieval can only approximate that query by returning everything topically near it and hoping the model sorts it out.

Stage three: inference

In a RAG pipeline, the work that stage two failed to do is deferred to stage three. The language model receives raw chunks and must, within a single generation, identify which passage states the holding, determine its stance, judge whether it binds, construct an argument, and produce a citation. Five operations, each a point of failure, none separately observable.

This is why the hallucinated citation is best understood as an architectural outcome rather than a defect of a particular model. If the component that constructs the argument is also permitted to emit citations, then each citation is a prediction, and predictions have an error rate. Prompting, better retrieval, and post-hoc checking can lower that rate. None of them removes the error class, because the permission that produces it remains in place.

The same analysis applies to agentic augmentation, in which the model searches iteratively, including on the open web. The Promise Fulfilled argues that this converts some failures from systematic to opportunistic and leaves others entirely intact, because they are failures of knowledge representation rather than of coverage. More retrieval produces more text. It does not produce typed metadata.

The alternative is to change the division of labor. If the retrieval layer returns propositions whose citations were verified when the proposition was stored, the generator's task changes from finding and citing authority to using authority it has been handed. It never originates a citation; it receives one. For the class of error that consists of citing a case that does not exist, there is then nothing to hallucinate. That is structural reliability in the sense defined above: the representation makes the error impossible, rather than making it rarer.

The claim should be stated no more broadly than it is. Removing the generator from the citation path eliminates fabricated citations. It does not by itself guarantee that an argument assembled from real propositions is sound. What it does is shrink the set of operations the generator performs, so that the remaining ones can be isolated and inspected.

Stage four: outputs

The first three stages determine what kind of question the system can answer. A retrieval architecture answers: which texts exist near this query? A reasoning architecture answers: what rule governs this issue, which court established it, does it help my client, and what is the verified citation?

The first output is useful for research. The second is useful for lawyering, and it has a property the first lacks: it generalizes. A holding extracted once serves every task in which the rule is relevant. The rule that defeats a claim at the pleading stage is the same rule a lawyer may need again at summary judgment and on appeal. What changes is the posture in which it is applied, which a structured system can treat as a parameter of the query rather than as something the model must infer from text. A taxonomy of defenses built once serves pleading, dispositive motions, trial, and appeal. Questions about treatment ("has this rule been overruled, modified, or limited to certain contexts?") become queries against recorded relations rather than hopes about what the model will notice.

A chunk index does not generalize this way, because it has no structure to generalize. Each new task needs a new prompt, a new retrieval strategy, and renewed reliance on the model to reason correctly over raw text. A proposition store accumulates value with every holding added and every category refined.

Structure-aware retrieval: necessary, not sufficient

The strongest counterargument is that RAG can be made structure-aware without being abandoned. The most instructive test of that position is an early and influential paper by Ho, Colby, and Fisher, "Incorporating Legal Structure in Retrieval-Augmented Generation: A Case Study on Copyright Fair Use" (ASAIL 2025). Their prototype, FairUseLegalBot, models roughly two hundred fair-use opinions as a knowledge graph encoding court hierarchy and citation relationships, extracts reasoning at the level of the four statutory fair-use factors rather than the whole document, weights authority by PageRank over the citation network, and interleaves retrieval with chain-of-thought reasoning. Their preliminary results show structure-aware retrieval surfacing more doctrinally authoritative precedent than standard RAG.

The paper is valuable precisely because its authors are candid about what remained. The prototype still hallucinated and still showed sycophancy. PageRank is blind to time, so a doctrinally central recent case, Warhol v. Goldsmith, scored low because it had not yet accumulated citations. Retrieval weights were set by hand. There was no formal evaluation, and the corpus covered a single doctrine.

Read as a decision procedure, each limitation yields a design lesson.

1. Fabrication is a permission, not a retrieval-quality problem. If the generator can still emit a citation, smarter retrieval lowers the fabrication rate and never zeroes it. The generator has to be removed from the citation path.
2. Factor-level decomposition is right; the unit is not. Retrieving at the level of a reasoning dimension is correct, but the dimensions belong in a closed, typed schema, and the object embedded should be a self-contained rule statement rather than a verbatim paragraph. Search then runs from proposition to proposition.
3. Centrality is not validity. "Frequently cited" and "still good law" are different predicates. A heavily cited case can be dead, and a new one can be controlling. Validity has to be checked for each authority as its own question.
4. Hand-set weights should give way to thresholds calibrated by question type. A match on an analogous fact pattern and a match on a rule lookup are different score populations, and one global cutoff miscalibrates both. Excluding results should be permitted only where measured label quality licenses it.
5. Sycophancy is answered by constraint, not by hope. Instead of asking a model to be balanced, the query plan can be required, as a schema rule, to search against its own position, and the system can prefer an honest report that no viable argument exists to a fluent one.
6. Every miss should be attributable to a stage. A missed case that cannot be localized to planning, filtering, embedding, or reranking is an anecdote. One that can be localized is a measurement.

The lesson common to all six is the one this essay began with. Structure improves retrieval, but reliability depends on where the guarantees live. A guarantee stated in a prompt is a request. A guarantee stated in the schema or the code is a property of the system.

Why the default persists

If the argument is this clear, why is chunk retrieval the default? Because the industry has been optimizing the wrong objective. A RAG pipeline requires a vector database, a text splitter, and a model API, and it can demonstrate fluent answers about case law within days. A proposition-level store requires reading cases, extracting holdings, designing taxonomies, calibrating stance labels, tracking which rules were established when, and validating citations: months of work before the first query is answered. Demonstrations are evaluated on curated examples. The failure modes of the easy architecture do not appear in a demonstration. They appear months later, in a filed brief.

There is also a representational bias. Teams trained on documents and embeddings naturally model law as documents and embeddings. Recognizing that legal knowledge has internal structure, that a holding is categorically different from a summary and that one opinion can contain propositions helping opposite sides, requires domain expertise that has to be designed into the representation. It cannot be appended afterward as a review of outputs.

The argument restated against the Four Laws

The Four Laws of System Design for Computational Law offer a compact test. Precedence runs from the Zeroth Law down.

• Zeroth Law: no unwarranted confidence. An opportunistically reliable system presents its right and wrong answers with the same fluency. Its confidence does not depend on whether the operative properties were ever represented.
• First Law: full explainability. A chunk pipeline can trace an answer to text. It cannot trace it to a proposition with a stated status, stance, and origin, because those were never recorded.
• Second Law: grounding in validated authority. Chunk retrieval grounds answers in similar text and leaves validation to a later step, or to the reader.
• Third Law: independently testable components. When holding identification, stance, and citation all happen inside one generation, none of them can be tested in isolation. In a proposition-level design they are separate stages with separately measurable error rates.

What this does not claim

Three limits keep the claim honest.

First, retrieval over documents remains a reasonable design for document question answering: summarizing a contract, finding its indemnification clause, stating what a complaint alleges. The argument concerns reasoning over legal authority, not every legal task.

Second, a proposition-level representation moves judgment upstream; it does not abolish it. Someone must decide what counts as the holding, which way it cuts, and which category it belongs to, and those extraction decisions must themselves be validated. The Promise Fulfilled reports validation results for a production store built to its specification and is explicit about what those numbers do and do not show. Errors in a structured representation are at least errors in a place where they can be found.

Third, structure does not decide cases. Whether a real proposition applies to new facts, whether a difference between cases is material, and which argument to make remain professional judgments. The purpose of the architecture is not to automate those judgments. It is to ensure that the materials placed in front of the person making them are what they claim to be.

Conclusion

Retrieval is a similarity function over text. Reasoning is a typed query over represented law. The first can be made faster and more accurate indefinitely without becoming the second, because the difference lies not in the quality of the search but in what the index contains. A legal AI system whose reliability depends on the model noticing, at query time, what the representation never recorded is reliable only when it is lucky. The alternative is to record it.