Retrieval & RAG for Agents

In this illustrative scenario, Reference Agent’s first retrieval pipeline embedded the request, retrieved ten nearest chunks, and placed them before the loop. It performed respectably on a benchmark assembled from real tickets but failed in production. Asked why Acme’s January reconciliation failed, it retrieved a runbook, two other tenants’ postmortems and an architecture document. None supported its fluent answer. The answer was in a Jira comment using “settlement” instead of “reconciliation”. Similarity search missed it.

A human analyst would not have done this. They would have searched, seen nothing useful, searched differently, found the ticket, followed a link, and read the comment. The failure was not the embedding model. It was that retrieval had been designed as a preprocessing step rather than as something the agent does, observes, and does again.

11.1Retrieval is an action#

In an agent, retrieval can run as a tool call inside the loop rather than only as preprocessing.

The query is written by a model that has seen the task, not derived from the user’s words. “Why did the January reconciliation for acme fail” becomes three queries: acme January settlement failure, reconciliation error acme 2026-01, and tenant:acme status:failed period:january against a structured index. Query formulation is a modelling problem; measure whether generated variants improve evidence recall.

Results are observed and can be judged insufficient. Ten irrelevant chunks are an observation. The agent can conclude “nothing here” and try a different formulation — the behaviour the classic pipeline structurally cannot produce.

Retrieval can be iterative and multi-hop. Find the ticket, then read the linked commit, then read the file the commit touched. Each hop is informed by the last. Single-shot retrieval collapses a chain of reasoning into one similarity computation.

Retrieval competes for context budget with everything else. Chapter 8’s assembler treats retrieved documents as a volatile, droppable, low-priority section. Ten chunks is not free; it is 8,000 tokens that displaced something.

Multiple retrieval tools can coexist. Vector search over documents, full-text search over tickets, a structured query against the warehouse, a graph traversal (Chapter 12). The agent picks; you do not have to guess which one the question needs.

11.2Chunking is a losing game, played well#

Chunking exists because retrieval units and reasoning units differ. You retrieve fragments; you reason over documents. Every chunking strategy is a compromise, and the goal is to lose as little as possible.

Chunk on structure, not on character count. Split at headings, sections, function boundaries, list items — the seams the author put there. Fixed-size windows cut sentences in half and separate a table from its caption. Where structure is absent, sentence-boundary splitting with overlap is the fallback, not the default.

Keep the chunk’s context with it. A chunk reading “this must be completed before the cutoff” is useless alone. Prepend the document title, the section path, and — the highest-value version of this — a sentence of generated context situating the chunk in the document. Contextual retrieval adds a generated preamble at index time. Anthropic reports improved retrieval on its evaluated corpora; validate the benefit locally and preserve the preamble as derived text, separate from source evidence. Re-indexing and source changes incur the generation cost again.¹

Index at multiple granularities. Retrieve the small chunk for precision, then expand to its parent section before putting it in context. This decouples the retrieval unit from the reasoning unit, which is the actual source of the tension.

Keep the pointer. Every chunk carries document id, section anchor, version and timestamp. This is what makes citation possible (below), what makes invalidation possible, and what lets the agent fetch the full document when the chunk is not enough.

11.3Hybrid search, because embeddings miss the obvious#

Dense vector search matches semantic similarity. It also fails predictably on a category that matters enormously in enterprise work: exact tokens with no semantic neighbourhood. Error code E-4471. Account acme-9. The function name reconcile_settlement. A person’s surname. Embeddings place these near their semantic context rather than at an exact match, and the document containing the literal string may not rank at all.

Sparse lexical retrieval, such as BM25, scores matching terms rather than learned semantic similarity. Their strengths often complement one another, making hybrid retrieval a useful baseline for identifier-heavy corpora.

Combining them requires care, because the scores are not comparable. Reciprocal rank fusion combines ranks rather than raw scores. It avoids score normalisation. Evaluate the rank constant, retrieval widths and optional weights.²

def rrf(result_lists: list[list[str]], k: int = 60) -> list[str]:
    """Reciprocal Rank Fusion — combine ranked lists without comparing scores.

    Uses positions rather than incomparable raw scores. With the default k=60,
    a document ranked 2nd by BM25 and
    9th by vector search beats one ranked 1st by vector and absent from BM25,
    which is usually the behaviour you want: agreement across different
    retrieval mechanisms is a stronger signal than confidence within one.
    """
    if k < 0:
        raise ValueError("k must be non-negative")
    scores: dict[str, float] = {}
    for results in result_lists:
        # A duplicate within one source must not receive extra votes.
        for rank, doc_id in enumerate(dict.fromkeys(results), start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=lambda d: (-scores[d], d))

Two additions worth their complexity. Metadata filtering before the search — tenant, date range, document type — is both a quality lever and, for tenant scope, a correctness requirement (Chapter 8). And query expansion, where the model writes two or three query variants including a hypothetical answer to embed instead of the question, addresses the asymmetry that a question and its answer often sit some distance apart in embedding space.³

11.4Reranking is where the wins are#

Add reranking when candidate recall is adequate but useful evidence is being buried. If the decisive source never enters the candidate set, repair retrieval first.

The architecture is two-stage. Retrieve broadly and cheaply — thirty to fifty candidates from hybrid search — then rerank with an expensive, accurate model and keep three to five. The reason this works is that the two stages optimise different things: the first stage needs recall (the right document must be somewhere in the candidate set) and the second needs precision (the right document must be at the top).

A cross-encoder reranker sees the query and document together and scores their relevance jointly, which is fundamentally more informative than comparing two independently-computed embeddings. It is also far too slow to run over a whole corpus, which is exactly why it belongs in stage two.

The strategic point is about context economics rather than retrieval quality. Reranking lets you send four excellent chunks instead of ten mediocre ones, and Chapter 8 explained why that is a double win: fewer tokens and fewer distractors. Choose the delivered width by evidence-set recall and task quality. Multi-hop or corpus-wide questions can require a larger k; Anthropic’s contextual-retrieval experiment used 150 candidates reranked to 20, so it does not establish a universal three-to-five-chunk optimum.¹

That last row makes sufficiency an explicit decision; iterative retrieval can also be implemented in a deterministic workflow.

11.5The funnel has unit economics#

The funnel is not free, but top_k is the wrong unit for reasoning about its cost. Candidate retrieval, reranking and delivered context consume different resources at different prices. Treating all three as “tokens used by RAG” hides the decision you actually control: spend a bounded amount outside the main model to avoid carrying weak evidence through several expensive agent turns.

Take one illustrative Reference Agent workload. A retrieval call produces forty candidates. Each candidate contributes an average of 220 tokens to the reranker: the query, title, chunk and minimal metadata. The reranker retains four candidates, and parent expansion turns each survivor into a 650-token section. The agent then needs six more model turns before the task completes.

The term token-turn is deliberate. It measures one token presented on one model turn, not a billable unit promised by a provider. Prefix caching, compaction and model-specific pricing alter the invoice. They do not alter the architectural fact that evidence which survives in working context is paid for repeatedly, while candidate text inspected by a dedicated reranker is paid for once at a different boundary.

Now scale the example. Suppose Reference Agent handles 10,000 retrieval-bearing tasks per day and averages 1.8 retrieval calls per task. That is 18,000 calls. If each call avoids the illustrative 26,400 token-turns above, the gross reduction is 18,000 × 26,400 = 475,200,000 main-model token-turns per day. The reranker inspects 18,000 × 8,800 = 158,400,000 token-units. Those numbers are not directly interchangeable — a cross-encoder and a generative model have different latency and price curves — but they tell you where to obtain the two prices and what to multiply. A cost review that compares only model names and ignores turn persistence misses the larger term.

This arithmetic also exposes when reranking does not pay. If the answer is produced immediately after retrieval, the evidence persists for one turn rather than six. The wide path then costs 7,000 context tokens and the narrow path 2,600, a reduction of only 4,400 against 8,800 reranker token-units. A fast, cheap main model or a slow reranker can make the simple path cheaper. The break-even condition is:

(wide evidence tokens − narrow evidence tokens) × remaining turns × main-model input price > reranker work × reranker price

Use measured p50 and p95 values for each term. Average chunk size conceals scanned PDFs; average remaining turns conceals the long operational runs where repetition dominates. At p95, parent expansion can turn four modest chunks into four enormous sections, so a nominally narrow funnel becomes the largest context consumer in the trace.

Optimise the expected task, not the successful retrieval call. Iteration changes the denominator. If 30% of first calls are judged insufficient and those tasks make two further calls, calls per task are 1 + (0.30 × 2) = 1.6. Reducing first-call insufficiency to 15% produces 1 + (0.15 × 2) = 1.3, an 18.75% fall in retrieval calls: (1.6 − 1.3) / 1.6. That improvement can be worth more than shaving milliseconds from vector search because it removes two reranks, two observations and two model decisions from every recovered task.

The useful operating view therefore has four distributions, not one average: candidate count, delivered evidence tokens, retrieval calls per task, and turns for which retrieved evidence remains in context. Plot cost and latency against all four. When candidate count rises but delivered evidence stays flat, the reranker is absorbing the work as intended. When delivered evidence or persistence rises, the main-model bill rises on every subsequent turn. When calls per task rises, query formulation or source routing is failing before reranking even begins.

Latency needs the same expected-task treatment. Use another illustrative trace: first-stage retrieval takes 70 ms, reranking takes 210 ms, and the model turn that judges evidence and continues takes 1,800 ms. One successful pass therefore takes 70 + 210 + 1,800 = 2,080 ms. If 30% of tasks need one repeat, expected retrieval-path latency is 2,080 + (0.30 × 2,080) = 2,704 ms. Improving the first pass so that only 15% repeat gives 2,080 + (0.15 × 2,080) = 2,392 ms. The 210 ms reranker is visible in every pass, but the avoidable second pass is the larger latency term; with 1.3 expected passes, halving reranker latency would save 1.3 × 105 = 136.5 ms per task, while halving the repeat rate saves 312 ms.

That distinction changes capacity planning. At a peak arrival rate of 25 retrieval-bearing tasks per second, 2.704 seconds on the retrieval path implies roughly 25 × 2.704 = 67.6 concurrently active paths if arrivals and service remain steady. At 2.392 seconds it is 25 × 2.392 = 59.8. You removed nearly eight in-flight tasks without changing a database index or buying a faster model. This is why query quality, sufficiency judgement and capacity belong on the same dashboard: an apparently small recovery rate multiplies into queues at peak.

Do not turn that calculation into a universal service-level target. Voice Reference Agent may value the first useful acknowledgement more than total retrieval completion (Chapter 18), while a background policy audit may accept another second to obtain contradictory evidence. Split time to first evidence, time to sufficient evidence and time to final answer. A single latency number rewards the system for stopping early — exactly the wrong incentive when the first result is plausible but incomplete.

11.6Grounding, citation, and the discipline of provenance#

An agent that retrieves and then answers without attribution has produced something unverifiable. Two mechanisms fix this, and both are cheap.

Require citation at the span level. Not “according to the docs” but “runbook.md#settlement-cutoff”. This is enforceable: parse the answer for citations, verify each resolves to a document that was actually retrieved this turn, and reject or regenerate when it does not. That check catches the failure where the model cites a plausible-sounding document it did not read.

Check claims against retrieved text. A second model pass can flag unsupported factual claims, but it is a fallible judge. Calibrate it against human-labelled examples and use deterministic checks for identifiers, quotations and numeric comparisons. This is the standard “faithfulness” measurement from the RAG evaluation literature, and running it inline rather than only in evaluation is what turns it from a metric into a guardrail.⁴

Retrieved content is evidence, not truth. Check for stale, incorrect, contradictory or adversarial content, and report insufficient evidence. Never assert a state that has not been observed in the run.

11.7The wrong evidence can be perfectly cited#

Citation checks establish provenance. Faithfulness checks estimate whether the answer follows from a cited span. Neither establishes that the agent selected the governing evidence.

The following incident and its hybrid retrieval, evidence-envelope and contradiction-resolution path are an illustrative production extension. The reference implementation in code/reagent does not implement them.

Consider a specific Reference Agent failure. An operator asks, “Which settlement cutoff applies to Acme’s Monday EUR batch on 19 January?” Hybrid retrieval finds four apparently excellent results:

The reranker puts the tenant-specific comment first. Reference Agent answers 16:00, cites the ticket comment, and the span-level verifier passes. A claim checker also passes: the cited sentence really says what the answer says. The answer is still wrong. The comment describes a temporary exception whose stopping condition was cutover, and the fourth result records that the stopping condition occurred four days before the batch. Policy v3 therefore governs the question.

Everyone initially blames stale indexing because an obsolete runbook is present. That is a real defect, but it is not the mechanism that produced this answer. Even with the runbook removed, the highly specific ticket comment outranks the general policy. The system treated relevance as authority and a conditional statement as timeless. It retrieved the facts needed to answer correctly and then selected the wrong relation among them.

The fix starts with an evidence envelope that survives chunking and expansion:

• stable source id and version, so a citation identifies one state rather than a mutable page;
• tenant and entity scope, so specificity can be tested rather than guessed from prose;
• effective_from and effective_to, distinct from publication and ingestion time;
• source type and authority, such as policy, deployment record, runbook or comment;
• supersedes and superseded_by links where the source system provides them;
• the retrieval call and query that admitted the evidence, so the decision can be replayed.

Do not ask the reranker to solve all of this. A relevance model answers “does this passage concern the question?” It does not reliably answer “is this passage the rule in force for this tenant at the requested time?” Put deterministic scope and validity filters before retrieval where metadata is complete. After retrieval, run an evidence-resolution step over the small surviving set. That step compares dates, authority and conditions, and returns a decision record rather than another opaque score.

For this example, the record should read:

decision: 15:00 cutoff applies
selected:
  - policy-v3#eur-cutoff       effective 1 Jan; applies after migration
  - rollout/acme/2026-01-15    proves migration condition completed
rejected:
  - ticket-482#comment-17      temporary exception ended at cutover
  - runbook-v2#eur             superseded by policy v3
unresolved: []

The rejected list is not audit decoration. It proves that contradictory evidence was seen rather than missed, and it gives the verifier something stronger to check: not merely “does a span support 15:00?” but “did the resolution account for the strongest evidence supporting 16:00?” Without that obligation, adding more retrieval can make an answer less reliable by supplying more plausible fragments for the model to choose among.

You also need a contradiction-search rule. Once a high-confidence passage states a threshold, owner, status or date, issue a narrow second query for the same entity and predicate with terms such as superseded, exception, effective, revoked and completed. This is not generic query expansion. It is an adversarial search for evidence that would change the answer. Stop only when the resolution step can explain the strongest conflicting passage, or report the conflict to the user.

Abstention here is specific, not vague. “I could not find an answer” is false — the system found several. The honest response is: “The Acme ticket says 16:00 until cutover, while the deployment record says cutover completed on 15 January and policy v3 then sets 15:00. I therefore resolve the cutoff to 15:00; confirm whether the rollout record is authoritative before executing.” If the planned action moves money, the policy boundary in Chapter 15 can require that confirmation. Retrieval quality alone cannot decide acceptable action risk.

This failure generalises beyond time. A local team note can conflict with a corporate policy; a customer-specific exception can conflict with a regional rule; a postmortem hypothesis can conflict with the later root-cause record. In each case, semantic relevance produces the candidate set and evidence resolution determines precedence. Collapse those jobs into one ranking score and the most specific sentence often wins, even when it is expired, subordinate or explicitly conditional.

11.8Access control survives retrieval#

Search under the authenticated caller’s permissions before candidates reach a reranker or model. Tenant filtering alone is insufficient when documents inside a tenant have different ACLs. Parent expansion, cached excerpts, citations and later document reads must enforce the same scope. A permission revocation must invalidate derived copies. Azure AI Search documents query-time ACL/RBAC enforcement as one implementation, still a preview feature in documentation checked on 9 November 2026. Check source support, preview limitations and API versions before adoption.⁶

11.9Freshness, and the invalidation problem nobody plans for#

Retrieval degrades as sources change. Address three failure modes:

Stale index. The document changed; the index did not. Every retrieval since is confidently wrong. The fix is event-driven indexing — the source system emits a change, the index updates — with a periodic full reconciliation as a backstop. Scheduled batch reindexing alone means your staleness window is your batch interval, and nobody who set that interval was thinking about the agent that now depends on it.

No temporal reasoning. Two documents both describe the cutoff; one is from 2024. Similarity does not know which is current. Chunks must carry timestamps and effective dates, the retrieval layer must be able to filter and prefer by them, and the agent must be told the dates so it can reason about precedence. A retrieval result without a date is an assertion with no expiry.

Deleted content that lives on. A document is removed and its chunks remain, or its content survives in a derived summary. For personal data this is a compliance problem (Chapter 30); for correctness it is a source of answers nobody can trace to a document because the document no longer exists.

The practical discipline is to treat the index as a derived store with an explicit contract about how far behind it may be, monitored the way you monitor replication lag. Most teams have no number here at all, and the first time they compute it, it is measured in weeks.

11.10Long context versus RAG: the honest answer#

The debate is usually framed as a competition and is better understood as a set of thresholds.

If the authorised, current corpus fits comfortably in the window, benchmark full-context input. It can simplify the system, but fit alone does not establish quality or low cost. Reserve output and tool space, refresh the snapshot, enforce access controls, and compare long-context quality and latency against retrieval on the same questions.

If it does not fit, you need retrieval — and long windows change what kind. Larger windows let you test more complete sections and retain multi-hop evidence. Widen only when the added evidence improves task success within the cost and latency budget.

Test effective context use. The lost-in-the-middle study found position-sensitive retrieval in the models it tested.⁵ Its results do not establish every newer model’s behaviour. Benchmark evidence placement and distractors on your selected model. Window capacity alone does not prove reliable attention.

Cost scales with what you put in, on every turn. This is the constraint people forget. A hundred thousand tokens of context on a forty-turn agent run is four million input tokens. Retrieval is, among other things, a cost-control mechanism.

So: long context reduced the need for aggressive retrieval; it did not remove the need for precise retrieval. And for agents specifically, the per-turn multiplication means the economic argument for precision is stronger than in single-shot question answering, not weaker. Retrieve less, but retrieve better.

11.11Measuring retrieval separately from the agent#

The most common evaluation mistake in RAG systems is measuring only the final answer. When the answer is wrong you cannot tell whether retrieval missed the document, reranking buried it, or the model ignored it — three different problems with three different fixes.

Recall at candidate width. Of questions with a known answer document, how often is it anywhere in the 30-50 candidates? This is the ceiling on everything downstream. If it is 80%, downstream reranking cannot recover the missing source on that pass. Another retrieval call can; evaluate the complete agent trajectory separately.

Precision at final k. After reranking, how many of the 3-5 delivered chunks are actually relevant? This is your context efficiency and it directly drives cost.

Faithfulness. What fraction of claims in the answer are supported by the retrieved text? Catches the model reasoning past its evidence.

Answer relevance. Does the answer address the question asked? Catches the case where retrieval was fine and the model answered a neighbouring question.

Those four are the core of the standard RAG evaluation framing, and the reason to run all four rather than one is diagnostic: each isolates a different stage.⁴

Retrieval calls per task. A distribution with a long tail means the agent is searching repeatedly — which is the loop working as designed, and also a signal that first-query quality is poor. Improving query formulation cuts cost directly.

No-result honesty rate. On tasks whose answers are absent from the corpus, measure how often the agent reports the gap instead of answering from adjacent material. Include deliberately unanswerable questions in the evaluation set.

11.12Preserve the evidence path in evaluation#

A question, a reference answer and one “gold document” are not enough to evaluate an agentic retrieval system. That format assumes one query, one answer-bearing page and no meaningful conflict. It cannot represent the Acme cutoff failure, where every necessary document appeared and the wrong one governed the answer.

Make the evaluation unit an evidence case. Store the user task, tenant and evaluation-time snapshot; the minimum evidence set required to answer; evidence that must be rejected or qualified; whether the corpus contains a resolvable answer; and the expected resolution. For a multi-hop question, the minimum set may contain a ticket that names a deployment, a deployment record that proves its date, and a policy that says what follows from that date. Scoring any one as the gold document mislabels a partial retrieval as success.

Construct negatives deliberately. Each kind attacks a different seam:

• a lexical decoy repeats the exact identifier but concerns another event;
• a semantic decoy describes the same failure for another tenant;
• a superseded source states an answer that used to be correct;
• a conditional exception states an answer that is correct only before an event;
• an unanswerable case has abundant neighbouring evidence but lacks the decisive fact.

Random irrelevant chunks are too easy. Production retrieval fails on evidence that is nearly right, because it receives a high score and gives the model a coherent story.

Instrument the path at document granularity. For every retrieval call, retain the query and filters; each candidate’s lexical rank, vector rank and fused rank; reranker score and rank; whether it was expanded; its final position in the model context; whether the answer cited it; and whether resolution selected or rejected it. Add source version, effective dates and index ingestion time. Chapter 23’s trace can then answer “where did the evidence disappear?” without reconstructing the run from logs.

The arithmetic becomes diagnostic. Suppose an evaluation set contains 200 answerable cases:

The chain product, allowing for rounding, is 0.920 × 0.929 × 0.901 ≈ 0.770. A dashboard showing only 77% invites a prompt rewrite. The boundary counts show that 16 cases were already unrecoverable before the model saw any evidence, 13 more were lost by reranking, and 17 failed after the required evidence reached context. Each group belongs to a different owner and experiment.

Add the unanswerable cases separately. If 23 of 29 produce an honest refusal or a precise unresolved-conflict response, no-result honesty is 23 / 29 = 79.3%. The more operationally useful complement is the false-assertion rate: 6 / 29 = 20.7%. Do not blend those 29 cases into answer accuracy. A system can raise apparent accuracy by guessing on them while becoming less trustworthy.

Run four counterfactual tests against the stored evidence path.

Remove one required source. The answer should become incomplete or explicitly uncertain. If it remains unchanged, the model may be answering from parametric memory or treating one citation as decoration.

Insert the strongest rejected source at rank one. The answer should preserve the resolution and explain why that source does not govern. Failure tells you precedence is driven by position or rhetorical specificity.

Swap effective dates while keeping prose fixed. The resolved answer should change. If it does not, the metadata is present in the trace and absent from the reasoning.

Change only the tenant filter. No candidate from the original tenant should survive. This is a correctness and isolation assertion, not a relevance metric; one leak fails the test regardless of answer quality.

These tests distinguish grounding from citation performance. A model that appends the supplied source ids to an answer it would have produced anyway can score perfectly on citation syntax. A grounded agent is causally sensitive to its evidence: remove, contradict or re-scope the evidence and its conclusion changes in the expected direction.

Set release gates at the boundaries your system controls. A candidate-recall gate protects the ceiling. An evidence-set survival gate protects multi-hop sufficiency. A false-assertion gate protects trust when the corpus is silent. A tenant-isolation gate protects data boundaries. The aggregate answer score remains useful as a product signal, but it must not be allowed to average away a zero-tolerance failure.

@tool(keywords=("search", "documents", "find", "knowledge"))
def search_docs(query: str, top_k: int = 4, doc_type: str = "",
                valid_at: str = "") -> str:
    """Search the knowledge base. Returns cited excerpts, best first.

    Args:
        query: What you are looking for. Prefer specific terms and
            identifiers over paraphrase; the index matches both meaning
            and exact tokens.
        top_k: Maximum excerpts within the server token ceiling. Start
            small; widen when the question needs a larger evidence set.
        doc_type: Optional filter — runbook, postmortem, ticket, policy.
        valid_at: Optional ISO date; return content effective on this date.
            Unknown validity is returned explicitly, never assumed current.
    """