Orchestration Patterns

The most expensive line of code in Reference Agent’s first year was not a bug. It was a decision to route invoice classification through the agent loop. Every invoice became a task: context assembled, a frontier model consulted, a tool called, a result verified, a trace emitted. In this illustrative scenario it achieved ninety-one percent accuracy and nineteen cents per invoice. The replacement — a two-stage classifier with a rules table and a single model call on the four percent of documents the rules could not resolve — ran at ninety-six percent accuracy and one-fiftieth of a cent. The agent was not badly built. It was the wrong shape.

Buy the least agency that solves the problem.

7.1The spectrum from workflow to agent#

Every orchestration decision is a position on one axis: who decides what happens next. At one end, your code decides — the control flow is written down, reviewable, and identical on every run. At the other end, the model decides — the control flow is emergent, different every run, and capable of handling situations you never anticipated.

Anthropic distinguishes workflows, where models and tools follow predefined code paths, from agents, where models dynamically direct their processes and tool usage.¹ Runtime discretion requires explicit bounds.

7.2Five composable patterns#

These compose. Real systems are usually two or three of them nested, and naming them is what lets a design review discuss the structure rather than the prompt.

Chaining#

Split a task into fixed sequential steps, each a model call whose output feeds the next. Extract, then normalise, then summarise. Draft, then critique, then revise.

Chaining wins when the task has genuine stages and each stage is easier in isolation than as part of a whole. The reason it beats a single mega-prompt is not mystical: each call has a narrower job, a smaller output space, and its own place to put validation. You can assert that stage one produced valid JSON before stage two ever sees it.

Its cost is latency, since stages are serial by construction, and brittleness at the seams — a stage that changes its output shape breaks the next one. Validate between stages; that is where the leverage is.

Routing#

Classify input, then dispatch to a specialised handler.

def handle(request: str) -> str:
    kind = classify(request)            # one cheap, constrained model call
    if kind == "password_reset":
        return run_password_reset(request)      # deterministic code, no model
    if kind == "invoice_query":
        return invoice_chain(request)           # a two-step chain
    if kind == "novel":
        return agent_loop(request)               # the expensive path
    return escalate(request)

Routing works because real request distributions are enormously skewed. Most of your traffic is a handful of known shapes, and the long tail is where agency earns its cost. Routing lets you serve the head with something cheap and reliable while keeping the tail’s coverage. It also lets you use different models per branch — a small model for the known shapes, a frontier model for the novel ones — which is the highest-leverage cost lever in most systems (Chapter 29).

Mis-routing sends work to a branch that cannot handle it; every branch needs an escape hatch to the general path. Traffic drift can make the router stale, so evaluate its accuracy separately from the handlers’.

Parallelisation#

Two distinct shapes hide under one word.

Sectioning splits work into independent pieces, runs them concurrently, and aggregates. Reading eight files, searching four sources, checking a change against six lint rules. The win is wall-clock, and it is large: the agent loop’s parallel read execution in Chapter 4 is this pattern at the turn level.

Voting runs the same task multiple times and combines the answers — majority vote, or “flag if any run raises a concern.” This buys reliability from redundancy and is genuinely useful for high-stakes classification, where a three-way vote can reduce error when member errors are sufficiently diverse. It costs roughly N generation calls plus aggregation, with token lengths and cache rates affecting the bill, so reserve it for decisions where being wrong is more expensive than being slow.

The discipline for sectioning is that the pieces must be truly independent. Two “parallel” subtasks that both write to the same file are a race, and the harness must know which tools mutate in order to prevent it — which is why mutates is declared on the tool rather than inferred.

Evaluator-optimiser#

One model produces, another evaluates against criteria, and the loop repeats until the evaluation passes or a budget is spent.

Effectiveness depends on where the evaluation signal comes from. Tests, schema validation and reconciled sums provide external evidence. A model judge without ground truth can converge confidently on a wrong answer; unaided self-correction does not reliably improve reasoning.²

# Design sketch: generate, verify, revise and escalate are application hooks.
def evaluator_optimiser(task, *, max_rounds=3):
    if max_rounds < 1:
        raise ValueError("max_rounds must be positive")
    attempt = generate(task)
    for round_index in range(max_rounds):
        verdict = verify(attempt)
        if verdict.passed:
            return attempt
        if round_index + 1 < max_rounds:
            attempt = revise(task, attempt, verdict.evidence)
    return escalate(task, attempt, verdict)

Every generated candidate is verified before acceptance or escalation; the last failed round does not create an unverified extra revision. Measure the marginal yield of additional rounds and cap them accordingly.

Orchestrator-worker#

A lead model decomposes the task, delegates pieces to workers, and synthesises their results. This is the delegation from Chapter 6 promoted to an architecture, and it belongs to Chapter 28. Noted here for completeness, with the same precondition: worker results must be compact and cheaply verifiable by the orchestrator.

7.3When not to use an agent#

You can enumerate the paths. If a whiteboard flowchart captures the work, build the flowchart. A model call inside a known branch is fine; a loop that rediscovers the branch on every request is waste.

The input distribution is narrow. Agency buys coverage of the unanticipated. If ninety-eight percent of inputs are three shapes, you are paying for coverage you do not use — route, and let the loop handle the two percent.

Latency is the product. An interactive experience with a sub-second budget cannot afford a multi-turn loop. One call, or a chain with a cache. Chapter 18’s voice constraints are the extreme version.

Every action is irreversible and needs approval. If every step requires a human, the agent is a very expensive form of turn-taking. Batch the decisions, present them once, execute deterministically.

You cannot state what “done” means. An agent without a completion criterion runs until its budget expires. If you cannot write the verification, you do not yet have a task (Chapter 6).

The task is one tool call. “Look up this order status” is an API call. Wrapping it in a loop adds a model’s judgement to a problem that has no judgement in it, and adds a way for it to be wrong.

You need the same answer every time. Regulatory calculations, financial postings, anything reconciled downstream. Non-determinism is a feature of agents; where it is a defect, do not buy it.

7.4The arithmetic nobody does#

Estimate cost per request at expected volume before choosing an orchestration shape. The invoice example illustrates how architecture can change cost by orders of magnitude.

Consider 400,000 invoices per month. The following costs illustrate orchestration choices. Substitute measured rates and task outcomes before choosing a design.

Input dominates this example. The average 41,000 input tokens include a growing transcript, tool definitions and retrieved context across 5.4 turns. Output contributes little at these illustrative rates. Context allocation and prompt caching can reduce the repeated-input cost (Chapter 8). Measure their savings alongside output and tool charges.

Turn count multiplies repeated work. Halving turns can reduce the bill, but the saving depends on which prompts, outputs and tools disappear, which makes trajectory length a first-class metric rather than a curiosity. An agent that reliably solves the task in three turns is not slightly better than one that takes six; its cost must still be measured from the actual trajectory.

The skew is the whole opportunity. The bottom row is not a different algorithm applied to the same work — it is the recognition that ninety-six percent of invoices were four templates. Finding that skew took an afternoon of looking at data and was worth more than any prompt engineering.

Accuracy improved in this illustrative case. Validated rules handled known templates consistently. Malformed inputs, wrong assumptions and distribution drift can still defeat them. Compare accuracy, coverage and cost on the same requests before replacing an agent path.

7.5Fan-out changes which number matters#

Parallelisation does not make work cheaper. It changes the unit you must reason about from the sum of durations to the critical path — the longest dependent sequence between start and finish. That distinction buys latency, but it also creates a larger failure surface. If any branch can fail, adding branches increases the probability that at least one does.

Suppose Reference Agent must inspect twelve independent evidence packs before it drafts a supplier-risk brief. Each read takes 0.8 seconds, the final synthesis takes 1.2 seconds, and the tool can sustain four concurrent reads. Ignore model variation for the moment; these are illustrative fixed values so you can check the arithmetic by hand.

The parallel shapes cut the ideal wall-clock time by two-thirds without removing a single operation. Billing still sees twelve reads and one synthesis. Capacity sees four concurrent calls instead of one. The operator sees a faster run and a sharper burst. All three statements are true, and a design review that says only “parallel is faster” has omitted two-thirds of the design.

Now give each read a four percent independent probability of a transient failure. The probability that all twelve succeed on their first attempt is 0.96^12, or about 61.3%. A fan-out that retries the whole batch whenever one branch fails therefore repeats some already successful work in roughly 38.7% of first attempts. If complete batches are retried until one passes, the expected number of batches is 1 / 0.613 = 1.63, so the expected read count is 12 × 1.63 = 19.6. Whole-batch retry has turned twelve logical reads into nearly twenty physical reads; a serial batch with the same retry boundary has the same problem.

Retry only the failed branch and the expected attempts per read become 1 / 0.96 = 1.0417. Across twelve reads that is 12.5 physical attempts, not 19.6. The failure probability did not improve; the retry boundary did. This is why a join needs stable branch identities and durable partial results. “Run the group again” is not a recovery policy.

The same arithmetic applies to tokens. If each evidence read consumes 2,000 input tokens and synthesis consumes 4,000, the clean trajectory consumes 12 × 2,000 + 4,000 = 28,000 input tokens. Whole-batch retry raises the expectation to about 19.6 × 2,000 + 4,000 = 43,200. Branch-local retry keeps it near 12.5 × 2,000 + 4,000 = 29,000. You did not save 14,200 tokens by changing the model or prompt. You saved them by moving the checkpoint.

Fan-out also makes the slowest branch the user’s latency. Suppose one read finishes within two seconds 98% of the time. For twelve independent reads, the probability that all finish within two seconds is 0.98^12 = 78.5%. The chance that the join waits longer is therefore 21.5%, even though each dependency meets its two-second objective on 98% of calls. A fast median can coexist with a poor tail because a join takes the maximum of concurrently started branch times. This calculation assumes all twelve start together; the earlier four-slot scheduler also adds queueing.

Set a deadline and decide explicitly what happens when it expires. A complete-only report should fail with the missing branch named. A report designed to tolerate partial evidence may continue, but must label the omission and prevent the verifier from treating eleven of twelve as complete. Launching a thirteenth speculative duplicate can reduce tail latency for safe reads, but it increases physical work and should use the same logical branch id so the first result wins and the later result is discarded visibly. Parallelism moves latency risk around; it does not abolish it.

Bound concurrency separately from total work. Twelve branches may be logically independent and still overwhelm a tool that permits four calls per second. Concurrency and requests per second are different limits. The scheduler owns a semaphore for concurrent calls and a rate limiter for arrival rate; the model should not be asked to remember a rate limit. Chapter 22 supplies backoff and circuit breaking, but the orchestration shape determines how much pressure reaches them at once.

Make the join explicit about incompleteness. A synthesis step must know whether it received twelve successes, eleven successes and one terminal failure, or eleven results because a branch was silently dropped. Those states require different actions. Attach the expected branch set to the fan-out, and let the join account for every branch before it produces an answer.

Measure both logical and physical work. A logical branch is the evidence pack Reference Agent intended to read. A physical attempt is each network call made to read it. Their ratio reveals retry amplification; collapsing them into “tool calls” hides the difference between a genuinely broad task and an unstable dependency.

7.6Deterministic scaffolds and where the framework fits#

If workflows are the left side of the spectrum, there is a whole class of infrastructure built for exactly that side, and a recurring question about whether an agent belongs inside it.

Graph-based agent frameworks make the control flow an explicit, inspectable structure — nodes as steps, edges as transitions, some edges conditional on state — which is a workflow representation that happens to permit model-driven edges. Durable execution engines go further: the orchestration is code, but its progress is journaled and replayed, so a workflow that ran for six hours and died resumes rather than restarts (Chapter 5). Cloud state machines sit at the most constrained end, where every transition is declared and nothing is emergent.

The useful way to think about all three is that they are answers to a question one rung to the left of “how does my agent decide”: they are answers to how does the structure around my agent survive and stay legible. An agent inside a durable workflow is a perfectly coherent design — the workflow owns retries, timeouts, and resumption; the agent owns the one step whose path is unknown. That composition is usually better than either extreme, and it is what most mature systems converge on.

Two cautions from practice. A framework that hides the loop is a liability. Everything in Part VI — guardrails, injection defences, stop conditions — has to attach somewhere, and if the loop is inside a library you cannot see, your safety surface is whatever that library chose to expose. And a framework that imposes determinism constraints will fight your agent. Durable execution typically requires workflow code to be deterministic on replay, which a model call is emphatically not; the resolution is that model calls are activities, not workflow logic, and getting that boundary wrong produces replay bugs that are miserable to diagnose.

The practical recommendation: own the loop, borrow the durability. Write the twelve lines from Chapter 4 yourself, and let a workflow engine own the parts that are genuinely solved infrastructure — scheduling, retries with backoff, timers, and resumption of the outer process. Chapter 29 returns to this when the question becomes deployment topology rather than orchestration shape.

7.7The decomposition that erased the rule#

Reference Agent once completed an access-review task with an exemplary trace. It read all twenty-four exception records, delegated three system-specific batches, received three valid structured responses, reconciled the row count, and finished under its latency budget. Every tool call succeeded. The answer was still wrong.

The task was to verify temporary access exceptions across a source-code host, a data warehouse and a ticketing system. The governing rule was simple: the requester must not approve their own exception. The orchestrator split the work by system because each system had a different API and schema. That looked like textbook sectioning — three independent sources, three specialist workers, one aggregation step.

One engineer appeared as npatel in source control, nikhil.patel@example.test in the warehouse, and employee E-4471 in the ticketing system. The source-control worker saw requester npatel and approver E-4471; they differed, so it passed the record. The ticketing worker resolved E-4471 to a person but never saw npatel. Each worker returned a compressed verdict rather than the identity evidence behind it. The aggregator received twenty-four rows labelled pass, checked that twenty-four inputs produced twenty-four outputs, and declared the review complete.

The initial diagnosis blamed the identity resolver. It had, after all, failed to connect aliases. But the resolver had not failed; no component ever gave it the three identifiers together. The decomposition boundary cut through the invariant. Local workers were asked a global question using local evidence, then compressed away the fields needed to repair the mistake downstream.

This is a more dangerous failure than a crashed branch because the trajectory looks healthy. Schema validation passes. Count reconciliation passes. Every worker reports confidence. A model evaluator reading the three summaries has no missing datum to notice. More retries reproduce the same wrong answer, faster and at greater cost.

The fix was not a better aggregator prompt. Reference Agent changed the graph in two ways. First, an identity-normalisation stage ran before fan-out and attached one canonical subject and approver id to every record. Second, the final join evaluated cross-record invariants over the uncompressed records rather than trusting worker verdicts. Workers could still fetch system-specific evidence in parallel, but they no longer owned the global decision.

Use this test before you section a task: can each branch be judged correct without seeing the other branches? If not, the branches may gather evidence, but they may not issue final verdicts. Preserve the facts at the join and place the invariant there.

The counter-argument to decomposition is therefore precise. Decompose execution where work is independent; centralise judgement where correctness depends on relationships. More workers increase throughput only while the information they discard is irrelevant to the final invariant. Once a worker summary becomes a lossy boundary, extra orchestration reduces the evidence available to decide.

7.8One loop, many configurations#

The corollary of “use the least agency” is a real engineering risk: a system with six orchestration shapes can become six codebases. Reference Agent takes voice, chat, and long-running background work; it routes some requests to deterministic handlers and some to the full loop. What it does not have is six loops.

The mechanism is configuration over duplication:

@dataclass
class LoopConfig:
    system_prompt: str = DEFAULT_SYSTEM
    context_budget_tokens: int = 120_000
    max_tokens_out: int = 4_096
    temperature: float = 0.0
    thinking_budget: int = 0
    parallel_reads: bool = True
    compact_when_needed: bool = True
    write_episodic_memory: bool = True

Voice sets a small context budget, no thinking, and compaction off. Long-horizon background work sets a large budget, compaction on, and a much lower autonomy level in its policy context. Same loop, same guardrails, same event log, same trace shape.

A guardrail added to one of six loops is missing from five. Shared execution paths reduce the coverage work required for each safety change.

7.9Measure the shape, not only the answer#

An orchestration change can improve answer quality while making the system slower, more expensive and less recoverable. It can also lower cost by failing early. A single “success rate” cannot distinguish either case. Your trace must preserve the shape of the trajectory well enough to price, time and audit it.

Start with one event vocabulary across every pattern: branch_selected, stage_started, stage_completed, join_completed, verdict_recorded, retry_started and a named terminal outcome. Put the run id, orchestration version, stage or branch id, logical operation id and physical attempt number on each event. Without those keys, a dashboard cannot tell whether four tool spans are four pieces of intended work, four retries of one piece, or duplicate execution after resumption (Chapter 23).

The critical-path ratio needs care. For a serial chain it should be close to one: nearly every stage lies on the path. For parallel sectioning it should fall below one because work overlaps. If twelve read spans sum to 9.6 seconds but the read phase still takes 9.1 seconds, a “parallel” implementation is queueing serially somewhere. If the read phase takes 2.5 seconds, the ratio for that phase is 2.5 / 9.6 = 0.26, which is consistent with roughly four-wide execution after overhead. Do not compare this ratio across unlike tasks; use it to test whether a chosen shape actually materialised.

Now take an illustrative weekly slice of 1,000 Reference Agent research requests. Six hundred and thirty settle after one model call, 250 complete a two-call chain, and 120 enter the tool loop, where they consume 840 logical operations. The total logical work is therefore 630 + (250 × 2) + 840 = 1,970 operations. Traces show 2,088 physical attempts, so retry amplification is 2,088 / 1,970 = 1.060 — six extra attempts per hundred logical operations.

Suppose 914 requests pass external verification and the week consumed 8.4 million input tokens. Cost efficiency is not 8,400 tokens per request; that denominator gives failed work credit. It is 8,400,000 / 914 = 9,190 input tokens per accepted outcome. Track both numbers. The first describes capacity; the second describes the price of useful work.

The distribution also gives you an architectural lead. Sixty-three percent of requests settle after exactly one call. Sample those traces by intent. If their context and output contract are stable, they belong in a one-call workflow even if they currently enter through the agent endpoint. The 120 tool-loop requests deserve a second view: plot logical operations per run, and inspect the tail separately from the median. A six-operation median with a forty-operation tail calls for stop-condition and recovery work, not a single average turn target (Chapter 20).

Round yield makes the evaluator-optimiser budget equally concrete. Imagine 400 Reference Agent briefs entering a generate-and-verify cycle. The verifier accepts 240 after round one, so 160 enter round two. Round two corrects 96; 64 enter round three. Round three corrects 16 and leaves 48 unresolved. For this example, assume both generation and a calibrated model-based verifier consume one model call. A deterministic verifier would instead add tool or compute cost, not a second model call:

Round two buys 96 additional accepted briefs for 320 additional calls — 3.33 calls per additional acceptance, the same efficiency as round one. Round three buys only 16 for 128 additional calls — eight calls per additional acceptance. Its marginal yield is 16 / 64 = 25%, compared with 96 / 160 = 60% for round two. The third round raises total accepted output from 336 to 352, but makes the portfolio less efficient. That is the moment to stop iterating and route the remaining 48 cases to a different mechanism: retrieve missing evidence, ask a human, or return a named incomplete outcome.

This calculation needs first accepted at round N, not “passed by round N”. Cumulative pass rate always rises as you spend more calls and therefore makes every extra round look beneficial. Marginal yield exposes whether the new round changes decisions or merely asks the same system to reconsider. Split it by verifier failure reason as well. If missing citations improve on a second attempt but contradictory evidence does not, set different budgets for those paths rather than one global max_rounds.

Also record whether the verifier’s signal is stable. Re-run the verifier on a small frozen sample without revising the candidate. If verdicts flip, some apparent “corrections” are evaluator variance rather than improvement. The orchestration response is not an extra debate between models. It is a more deterministic check, a narrower rubric, or human adjudication for the disputed class (Chapter 24).

Compare orchestration versions within intent and rung. A new router can appear to improve quality simply by escalating harder cases to humans, while an old router attempted them automatically. A new chain can appear cheaper because its input mix contains more easy requests. Report accepted outcomes, latency and physical work for the same intent cohort and the same terminal rung. Traffic mix is not an orchestration improvement.

Finally, retain representative traces behind every aggregate. A rise in branch escape rate tells you where to look; the trace tells you whether the label was wrong, the branch contract was narrow, or a dependency failed. Metrics choose the queue. Evidence changes the system.

7.10Choosing, concretely#

Start by writing the flowchart. Not the prompt — the flowchart. If you can complete it, you have a workflow, and you should build the workflow. The act of failing to complete it tells you exactly where the agency is needed, and it is usually one box, not the whole diagram.

Find the skew. Sample a few hundred real requests and classify them. If the top three categories are most of your volume, routing is your highest-value first move, and it is nearly always worth more than any prompt improvement.

Locate the verifier. For each part of the task, ask what observation would prove it correct. Where a cheap verifier exists, an evaluator-optimiser loop will work well. Where none exists, you need a human in the loop, and you should design that seam deliberately (Chapter 19) rather than discovering it.

Budget before building. Estimate cost per request at your expected volume for two positions on the spectrum. The arithmetic is frequently decisive and almost never done: the invoice pipeline’s nineteen cents versus a fiftieth of a cent was visible in a spreadsheet before a line of code was written.

Measure the trajectory length distribution. After shipping, look at how many tool calls your runs actually make. A distribution with a huge spike at one or two is a routing opportunity wearing an agent costume. A distribution with a long flat tail is a real agent workload, and you should invest in Part VI’s stop conditions instead.

Part II is complete: the loop takes turns, survives crashes, decomposes goals, and now knows when it should not be running at all. Part III turns to the question the loop has been quietly assuming an answer to — what, exactly, goes into the model’s context on each of those turns, and where does it come from.