Multi-Agent Coordination

Two illustrative Reference Agent projects used multi-agent designs with different results.

The success was research. “Find every system that reads from the settlement connector, who owns it, and what its change process is” — a task requiring the exploration of forty repositories, three wikis and a service catalogue. Split across six sub-agents, each searching one territory and returning findings, it went from a forty-minute single-agent run that ran out of context to an eight-minute parallel run that did not.

The reversion was reconciliation. Someone split it into a fetcher agent, a comparator agent and a reporter agent. It cost four times as much, took longer, and produced worse answers — because every stage needed context the previous stage had, and passing it between agents lost detail that a single context had held for free.

28.1The honest case, with numbers#

Multi-agent architectures are over-adopted, and the reason is that the argument for them is usually made qualitatively (“specialisation!”) when good quantitative evidence exists on both sides.

Anthropic’s 2025 engineering report described a lead-and-worker research system outperforming its single-agent Claude Opus 4 baseline by a reported 90.2% on an internal research evaluation. It also reported roughly 4× the tokens of chat for agents and 15× for multi-agent systems.¹ These are measurements of that system and comparison, not a universal multiplier or a 90.2-percentage-point gain.

The report emphasised broad, parallel research and cautioned about tightly coupled work needing shared context. It did not hold every model choice and token budget constant, so the result does not isolate architecture as the sole cause. The two Reference Agent examples illustrate the same trade-off; your measured baseline decides.

28.2Context isolation is a useful reason to delegate#

The stated justification for multi-agent is usually specialisation — a researcher agent, a writer agent, a reviewer agent, each with its own expertise. Roles alone do not establish a benefit. Different tool permissions, expertise, verification or model choices can justify separate workers, but the boundary must earn its coordination cost.

A single model with a good prompt can be a researcher on one turn and a writer on the next. It does not need a separate identity to change register. Splitting by role creates a context boundary; test whether that improves the work.

One benefit of multi-agent execution is context isolation (Chapter 9). A sub-agent can read ninety files and return three findings. Only those findings enter the parent’s context. A single-agent harness can also externalise documents and retrieve summaries. Delegation adds independently managed contexts and may allow useful work to proceed concurrently.

Goose documents separate subrecipe sessions without shared conversation history, memory or state.³ Hermes starts subagents with a fresh conversation. The parent must pass the goal and context.⁴ Cline gives each subagent its own instructions and context window. Its tool access depends on mode.⁶ These documented context boundaries are useful, but do not by themselves prove filesystem, credential or memory isolation.

So the test for whether a task should be multi-agent is not “are there distinct roles” but: would the sub-task pollute the parent’s context with material the parent does not need? Then ask whether the resulting benefit exceeds dispatch, verification and synthesis cost.

28.3Topologies#

Orchestrator-worker is a useful starting pattern for independent assignments. A lead agent decomposes, workers explore independently with clean contexts, the lead synthesises. It maps directly onto Chapter 6’s delegation preconditions, and its failure mode is the one that matters: if the lead cannot cheaply verify what a worker returns, it has added an unverifiable dependency and lost visibility into it.

Pipeline assigns stages to separate contexts or agents. This helps when stages have different policies, tools or evidence needs, but the handoff must preserve what the next stage requires. Use Chapter 7’s simpler chain when separate identities or contexts add no value.

Hierarchical — workers spawning workers — adds coordination depth that must be justified. Branching can multiply cost, while summaries can obscure failures deep in the tree.

Products apply different depth controls. Hermes supports configurable spawn depth, and goose prohibits nested subrecipes.⁴³ Cline allows children to create further subagents.⁶ Default to depth one, and make deeper an explicit, argued decision.

Blackboard — agents coordinating through shared state with no orchestrator — is elegant on a whiteboard and produces write conflicts in production. You have built a distributed system whose participants are non-deterministic and whose consistency model is implicit. If you find yourself needing this, you probably need a work queue with leases instead.

Debate — several attempts plus a judge — can improve selected tasks, but correlated errors, judge bias and communication cost can erase the gain. High stakes require independent evidence, not merely agreement. It is Chapter 7’s voting pattern with critique added.

28.4Contracts, not conversations#

A common design error is relying on unbounded free-text conversation as the entire coordination protocol.

Free-text messaging between agents produces four problems at once: unbounded context growth as the conversation accumulates; ambiguity about who is responsible for what; no way to validate what came back; and — the security one from Chapter 21 — an unlabelled channel through which an injected instruction from a poisoned page travels into the parent as a trusted finding.

Use contracts: a typed request and a typed, schema-validated response.

@dataclass(frozen=True)
class SubtaskRequest:
    subtask_id: str
    goal: str                       # complete and standalone
    context: str                    # what the parent explicitly passes
    must_return: dict               # JSON Schema for the response
    budget: Budget                  # its OWN, much smaller ceilings
    tools_allowed: tuple[str, ...]  # narrower than the parent's
    autonomy: int = 0               # read-only unless argued otherwise


@dataclass
class SubtaskResult:
    subtask_id: str
    status: Literal["completed", "partial", "failed", "escalated"]
    findings: list[dict]            # schema-validated, not prose
    evidence: list[str]             # citations the parent can check
    trust: str = "untrusted"        # it read the world; propagate that
    usage: Usage = field(default_factory=Usage)

The goal is standalone. Workers start with fresh conversations, so the parent must pass every fact and constraint the assignment requires.

Bound each worker and the whole task. An unattended delegated run needs an explicit limit matched to its assignment. Hermes’ documented default of 500 iterations for a main loop against 50 for a subagent is one documented configuration, not an evidence-based universal ratio.⁵

Tools are narrower. A sub-agent doing research needs read tools. Giving it the parent’s full capability multiplies your blast radius by the number of sub-agents (Chapter 15).

Autonomy defaults to read-only. Enforce this default through the worker’s tools and execution boundary.

Cline illustrates why a mode label alone is insufficient. Act-mode children can edit files and run commands. Plan mode restricts editing tools and some commands, but commands such as python -c can still write files. Children receive no MCP tools, and approval of spawning covers their subsequent actions.⁶ A worker limited to reads can still disclose sensitive data or return untrusted material.

The response has a validated structure. This makes structural validation possible. A string inside valid JSON can still carry a malicious instruction; verify claims separately and preserve source trust.

Trust is propagated. A sub-agent that read a web page returns untrusted content. If the parent treats it as its own trusted finding, you have the agent-to-agent injection chain from Chapter 21.

28.5Protocols#

MCP and A2A cross different boundaries:

MCP connects an agent to tools and data — a client-server protocol where servers expose tools, resources and prompts (Chapter 14).

A2A connects an agent to another agent — discovery, task delegation and status across organisational or vendor boundaries, where participants need not expose their internal implementations.⁷

The distinction is the boundary each crosses. MCP is for capability; A2A is for delegation between systems you do not own or cannot see inside.

The practical guidance for most teams: you probably do not need an inter-agent protocol yet. Within one process, a function call with a contract may suffice. An interoperable protocol can still be useful inside one organisation when runtimes or services differ. The protocol earns its cost when agents cross a trust boundary — a partner’s agent, a vendor’s agent, another business unit’s agent with its own identity and authorisation. At that point you need discovery, capability declaration, and authenticated delegation, and reinventing them is worse than adopting them.

And when you do cross that boundary, Chapter 15’s questions all return, harder: what authority does the remote agent act with, whose identity is it using, what can it reach, and how is its output labelled when it comes back? A remote agent’s response is untrusted content produced by a system you do not control.

28.6Failure modes specific to multi-agent#

The MAST taxonomy was built from a dataset of over 1,600 annotated traces across seven popular multi-agent frameworks, identifying 14 unique failure modes clustered into three categories — system design issues, inter-agent misalignment, and task verification — validated with high inter-annotator agreement (kappa 0.88) on 150 expert-annotated traces.²

System design issues: wrong topology, unclear responsibility or missing termination.

Inter-agent misalignment is the coordination category: agents working from different understandings, dropping information at handoffs, duplicating work, or contradicting each other. Reduced by clear contracts, shared task state and evidence checks; a schema alone does not establish semantic agreement.

Task verification: failing to check the work. A parent reviewing only a worker’s report may accept plausible claims without verifying their underlying evidence.

Three concrete failures worth designing against explicitly.

Error compounding. A worker’s slightly-wrong finding becomes the lead’s confidently-wrong synthesis. The parent had no way to check, so the error gained authority as it moved up. Evidence citations in the result contract are the defence.

Duplicated work. Two workers with overlapping remits both read the same forty files. Pure cost, and invisible without per-worker cost attribution (Chapter 23).

Coordination deadlock. Worker A waits for something worker B is waiting on. A simple defence is to keep worker assignments independent and let the orchestrator sequence dependencies. More general graphs need cycle detection, deadlines and cancellation.

28.7The coverage hole that passed review#

In an illustrative Reference Agent research fan-out, the run looked successful in every trace the team normally inspected. Six workers completed, all six returned valid JSON, none exceeded its time budget, and the lead produced a concise answer with citations. The answer named forty-two services. Two engineers reviewed it and found no fabricated sources.

Three weeks later an owner asked why their settlement relay was absent. The service had not failed retrieval. It had never been assigned.

The original lead had divided the estate by repository group: payments, identity, fulfilment, data, shared infrastructure and archived systems. The workers reported that they had inspected 8, 8, 7, 8, 7 and 8 repositories respectively — 46 repository visits. The service catalogue contained 47 active service identifiers, but repositories were not the same population. Deduplicating the reported repository URLs left 44 distinct repositories. Mapping their findings back to the catalogue accounted for only 42 of the 47 active services. Five services had no disposition at all.

The team had mistaken three different counts for one another:

All six responses passed schema validation, but their schema described findings rather than coverage. Validating returned rows cannot reveal missing rows without a declared universe of required items.

The fix was not a longer prompt. It was a coordination ledger owned by the lead. Before dispatch, the lead materialised all 47 service IDs and assigned each exactly once. A worker had to return one disposition for every assigned ID: evidence_found, no_evidence, or escalated, plus the artefact identifiers it inspected. The release gate checked the union of dispositions against the original manifest rather than summing worker claims.

On the next run, 45 IDs reached an evidence-backed disposition and two escalated because the catalogue pointed at retired repositories. The lead reported “45 resolved, 2 requiring owner confirmation” rather than “complete”. That answer was less polished and more correct. It also gave the operator a finite queue to close.

This mechanism generalises beyond research. In a migration, the ledger holds accounts. In a codebase sweep, it holds packages or ownership units. In an incident, it holds hypotheses and the evidence required to close each one. The decomposition is not complete until every item in the required universe has exactly one assignment and one terminal disposition. Reconciling only the assignments that were created hides required items that were never assigned. If no stable required universe exists, the lead must state that it sampled or explored; it must not claim exhaustiveness.

The reference harness does not create sub-agents or implement this ledger. AgentLoop._invoke_all parallelises vetted read-only tool calls and executes mutations serially. Worker contracts, coordination records and verification gates are production extensions.

28.8The cost multiplication#

Fan out only where there is genuine parallelism. The first and largest lever. Six workers exploring six disjoint territories is parallelism; three workers doing sequential stages is not.

Use cheaper models for workers. The Anthropic configuration is instructive: a stronger lead with cheaper subagents outperformed a single strong agent.¹ Workers doing bounded search do not need the frontier model; the lead synthesising does.

Keep results small, and enforce it. The result contract’s schema is a size limit as well as a shape. A worker returning forty thousand tokens has moved the context problem rather than solved it.

Reuse cacheable prefixes when supported. Eligible reuse depends on provider, model, cache scope, timing and exact prefix identity. Simultaneously dispatched workers may all pay a cold write; do not assume a cache hit merely because the text matches (Chapter 8). This is a substantial and frequently-missed saving on a fan-out.

28.9Price the fan-out before you build it#

“Fifteen times” is a useful warning and an insufficient budget. A production decision needs the actual model mix, the input and output split, and the quality delta on your task. Token counts alone can mislead because a cheap worker token and a lead-model output token do not have the same price.

Take an illustrative Reference Agent research run. The single-agent baseline uses 78,000 input tokens and 6,000 output tokens. The multi-agent design uses a lead planning turn, six workers and a lead synthesis turn:

The fan-out uses 212,000 ÷ 84,000 = 2.52 times the tokens of this single-agent baseline. That is not a contradiction of the reported 15× comparison with chat interactions; it is a different denominator. Architecture reviews must name the denominator every time they state a multiple.

Now price the same run. Assume, purely for the worked example, that the strong model costs $5 per million input tokens and $25 per million output tokens. The single-agent run costs (78,000 × $5 + 6,000 × $25) ÷ 1,000,000 = $0.54. The all-strong-model fan-out costs:

The incremental cost is $1.42 − $0.54 = $0.88 per task. At 50,000 tasks a month, that architectural choice adds $44,000 before retries, abandoned workers or cache discounts. The multiplication belongs in capacity planning, not in a footnote on a model experiment.

Measure duplicated context separately because it has a direct architectural owner. Suppose 8,000 of each worker’s 26,000 input tokens are the common operating contract, task background and tool definitions. Across six workers, that prefix consumes 6 × 8,000 = 48,000 input tokens — 30.8% of the workers’ 156,000 input tokens. One copy is necessary; the other five add 40,000 tokens. At the illustrative strong-model input price, those extra copies cost $5 × 40,000 ÷ 1,000,000 = $0.20 per run before any cache discount. If each territory truly needs the whole packet, cache it. If it does not, shrink the worker contract. Context isolation does not justify sending every worker the same irrelevant context.

Cheaper workers change the dollar result without changing the token result. If worker input costs $1 per million and output costs $5 per million, each worker costs (26,000 × $1 + 2,000 × $5) ÷ 1,000,000 = $0.036; six cost $0.216. Keeping the strong lead for planning and synthesis brings the run to $0.14 + $0.216 + $0.20 = $0.556 — only 3.0% above the $0.54 baseline, even though the architecture still consumes 2.52 times the tokens. Evaluate the mixed configuration separately; its quality cannot be assumed from its price.

The final test is value. Suppose the all-strong fan-out raises measured task success from 78% to 88% on the same held-out cases. Let V be the value of a successful result and ignore downstream differences for the moment. The fan-out earns its extra cost when:

0.88V - $1.42 > 0.78V - $0.54
0.10V > $0.88
V > $8.80

At 50,000 monthly tasks, the ten-point improvement produces 5,000 additional successes for $44,000 — again $8.80 per additional success. For a high-value investigation that prevents hours of engineering work, that can be an easy yes. For a $5 classification task, it is a no even before you price operational complexity. Replace V with avoided loss, labour saved or revenue protected, but write the unit down. A quality percentage without a value model cannot authorise a more expensive architecture.

Reserve the worst case as well as forecasting the mean. If six workers each receive a 30,000-token ceiling, the lead needs capacity for 180,000 worker tokens even if historical use averages 22,000. Admission control must reject or queue a fan-out whose reserved tokens, concurrent tool slots or deadline budget do not fit. Otherwise a burst of individually valid delegations becomes the tenant-wide budget failure Chapter 20 warned about.

28.10Evaluating a multi-agent system#

Evaluate coordination in addition to individual outcomes and trajectories:

Attribute failures to a participant. Which agent’s output was wrong? Without this, a multi-agent failure is a single opaque event and you cannot tell whether to fix the lead’s decomposition or a worker’s search.

Score the decomposition separately. Was the split sensible? A good worker cannot rescue a bad decomposition, and this is the lead’s distinct job.

Score synthesis separately. Given the results the workers returned, was the final answer the right one? This isolates the last step, which is where error compounding surfaces.

Compare against the single-agent baseline, always. This is the one that gets skipped, and it is the only measurement that answers whether the architecture earns its measured overhead. Run the same eval suite with a single agent and with the multi-agent configuration, and compare quality and cost. If quality is equal, you have a cost regression wearing an architecture diagram.

28.11Measure coordination as its own subsystem#

A multi-agent trace is not one run with extra names attached. It is a causal tree with resource reservations, assignments, deadlines and evidence moving across trust boundaries. If your dashboard collapses that tree into total latency and total cost, it hides the exact questions the architecture introduced.

Start with a trace from an illustrative six-worker Reference Agent fan-out:

The worker stage lasted 11.8 seconds because the critical path ends with W6, not when the average worker finishes. The median duration is (4.1 + 4.4) ÷ 2 = 4.25 seconds; W6 adds 7.55 seconds of straggler excess above that median. If lead planning took 1.4 seconds and synthesis took 2.6, the user experienced 1.4 + 11.8 + 2.6 = 15.8 seconds. Summing worker time gives 31.7 seconds, a useful capacity number and a false latency number. Chapter 23’s critical-path rule matters more, not less, when branches run in parallel.

The same trace shows why aggregate budgets are insufficient. Worker input totals 166,000 tokens, and output totals 13,000, so the worker stage consumed 179,000 tokens. That happens to fit an aggregate 180,000-token reservation. W6 still consumed 49,000 total tokens and exceeded its 30,000-total-token ceiling by 19,000. A parent that checks only the aggregate will call the run compliant while one child ignored its contract. Enforce the local ceiling where the worker spends, and let it return partial with the evidence collected so far. The lead can then reassign the remainder, narrow the question or escalate; it cannot silently reinterpret a breached budget as success.

Instrument at least these coordination measures:

Do not optimise these numbers independently. Zero overlap can be wrong when two independent attempts are the point of a debate topology. A short straggler tail can mean workers were stopped before evidence arrived. A 100% useful-fan-out score can be manufactured by citing every worker once. Pair the measures with outcome quality and inspect representative traces, exactly as Chapter 24 requires.

Set gates on invariants and targets on economics. Required-universe coverage should be 100% before an exhaustive claim, with exactly one assignment and one terminal disposition per required item. Every claim used in a high-risk decision should have verifiable evidence. A local budget breach should be impossible, not merely rare. By contrast, overlap and useful fan-out have no universal threshold; baseline them by task family and compare changes against quality. The difference is whether the measure protects correctness or describes an optimisation trade-off.

Tracing must preserve causality across the thread boundary. The reference Reference Agent demonstrates the smaller version: _invoke_all copies the active context into each read-only worker thread so tool spans remain children of the tools.parallel span. A production sub-agent runtime needs the same invariant across a larger boundary: lead run ID, subtask ID, assignment ID and parent span context on every request and result. Without those identities, the trace becomes several plausible stories rather than one causal record.

Cancellation is a protocol, not a signal#

The straggler branch creates one more coordination problem: what happens when the lead stops waiting? A timeout in the parent does not prove that the worker stopped. It proves only that the parent stopped listening. The worker may still be reading tools, consuming tokens and preparing a result that will arrive after the lead has reassigned its territory.

Return to W6. At six seconds the lead decides that the branch has exceeded its latency allowance and dispatches W7 over the same assignment. W6 finally returns at 11.8 seconds; W7 returns at 12.4. If the ledger accepts both, the synthesis contains duplicate or contradictory findings. If it accepts whichever arrived first, timing has become the conflict-resolution policy. If W6 used mutating tools, “ignore the late result” would be worse: the effect would remain even when the message did not.

Give every assignment a lease generation. The first dispatch is (assignment=service-map, generation=1). Reassignment increments the generation to 2. A result must carry both values, and the ledger accepts it into the active slot only when they match the current lease. W6’s generation-1 result remains in the trace as late; it cannot silently overwrite W7. Preserve it because it still explains resource use and may contain evidence worth inspecting. Do not let it change the synthesised state automatically.

Cancellation itself needs three separate events:

1. cancellation_requested records the lead’s decision and reason.
2. cancellation_acknowledged proves the worker received the request.
3. worker_stopped proves the worker reached a terminal boundary and records its final usage.

Collapsing those into cancelled invents certainty. A missing acknowledgement means the worker may be unreachable; an acknowledgement without a stop means it may be inside a tool call that cannot be interrupted. Track cancellation drain time from request to stop, and include work consumed after the request in orphan spend. Those two measures tell you whether timeouts actually release capacity.

Design worker tools around that reality. Read-only exploration can check a cancellation token between calls and safely return partial. A mutation already submitted to an external system cannot be cancelled by changing a ledger row. It needs the recovery and reconciliation protocol from Chapter 22, plus the stable action identity from Chapter 5. This is another reason read-only workers are the right default: cancellation can bound computation, but it cannot reverse an effect.

Deadlines also need slack for the parent. If the user-facing deadline is 20 seconds and verification plus synthesis has measured p95 latency of 3.5 seconds, workers do not own 20 seconds; their latest acceptable completion is 16.5 seconds. Reserve that 3.5-second tail at admission. Otherwise every worker can meet its individual deadline and the system can still miss the user’s.

The operational invariant is simple: one live lease per assignment, one accepted terminal result per lease, and no claim that work stopped without a stop event. With those fields, reassignment is deterministic, late results are auditable and the budget report includes work the user never saw. Without them, a timeout merely moves uncertainty from the critical path into the background.

Finally, record why a worker stopped. completed, partial, failed and escalated are different evidence states. A deadline that returns a fluent partial report must not be counted as completion because the text looks finished. The coordination ledger in Figure 28.3 exists to keep that distinction durable through synthesis.

28.12When one agent wins#

Prefer a single agent in six situations:

The task fits in one context and has no measured decomposition benefit. A single context avoids handoff overhead; parallel speedup or independently checked expertise can still justify delegation.

Stages need each other’s context. The reconciliation reversion. Sequential dependency is the anti-pattern the published guidance names explicitly.

Latency matters more than breadth. Spawning and synthesising adds turns. Chapter 18’s voice budget cannot absorb it.

The task is cheap. 15× a small number is still a small number, but so is the benefit, and you have added failure modes for nothing.

You cannot verify worker output. Without cheap verification you have added unverifiable dependencies — the orchestrator-worker failure mode.

Observability is incomplete. Establish per-agent traces, costs and failure attribution before adding workers.