Tool Ecosystems at Scale

In this illustrative scenario, Reference Agent shipped with nine tools. Eighteen months later it had 230. Eleven internal integrations, four vendor MCP servers and team-contributed connectors supplied a growing catalogue. Individual tools worked, but selection deteriorated. The agent confused search_confluence with search_sharepoint, and get_ticket with get_incident. Each turn included 31,000 input tokens describing tools it would never call.

Tool selection and definition cost depend on the whole menu.

14.1Two curves that cross#

Tool-set growth creates two independent problems:

Token cost grows linearly. With a flat menu, every exposed definition contributes input tokens each turn. Two hundred tools at 200 tokens each is 40,000 tokens before the conversation starts — a fifth of a generous window, spent on a menu.

Selection accuracy degrades non-linearly. This is the harder problem. As the menu grows, tools become confusable: near-synonyms accumulate, descriptions overlap, and the model’s job shifts from “pick the obvious tool” to “disambiguate among six plausible ones.” The degradation is not gradual and polite; it concentrates in clusters of similar tools, which is why a system can add fifty unrelated tools with no harm and then break by adding one that overlaps an existing one.

The consequence is a rule of thumb worth internalising: evaluate disclosure as the visible menu grows; there is no universal twenty-tool limit. Overlap, model training and task distribution affect the threshold. The reference implementation encodes the threshold explicitly, and says so:

def _visible_tools(self) -> list[str] | None:
    """Show every tool while the catalogue is small; narrow once it is not.

    The threshold is arbitrary and should be set by measurement, not taste:
    raise it until tool-selection accuracy drops (Ch 14).
    """
    if len(self.tools) <= 12:
        return None
    return self.tools.select(self.state.goal, k=10)

14.2Measure selection before you fix it#

Build a set of a few hundred tasks with a known correct tool — or a known correct set, since several tools are often acceptable. Then measure four things.

Selection accuracy. Fraction of tasks where the first tool called is an acceptable one. The headline number.

Confusion pairs. For each error, which tool was called instead of which. This is the actionable output: errors cluster into a handful of pairs, and each pair is either a naming problem or two tools that should be one. Fixing the top five pairs typically recovers most of the loss.

Recovery rate. When the wrong tool is called first, how often does the agent recover within three turns? A system with mediocre selection and good recovery is often fine; one with good selection and no recovery is brittle. This is where Chapter 13’s error messages pay off — a good “not for” clause in a wrong tool’s response is what produces recovery.

Tokens per turn spent on definitions. The cost side of the trade.

Public benchmarks are useful for calibrating expectations, and unhelpful as a substitute. The Berkeley Function Calling Leaderboard tracks function-calling accuracy across categories including parallel and multi-turn calls; τ-bench evaluates agents on multi-turn tool-use tasks against domain policies with a user simulator, and its central finding is directly relevant here — agents that succeed once frequently fail on repeat attempts at the same task, so reliability across trials is a different and lower number than single-pass accuracy.¹ Your tool set is not theirs, but the reliability-versus-capability gap generalises: measure repeated attempts, not one pass.

14.3MCP: what it standardises, and what it does not#

The Model Context Protocol provides interoperability between tool clients and servers.

MCP defines a client-server protocol. A host application (your agent) runs one or more clients, each connected to an MCP server that exposes capabilities. Servers offer three kinds of thing: tools the model can invoke, resources the host can read, and prompts — reusable templates. Transports cover a local process over stdio and a remote server over HTTP, with authorisation for the remote case.

MCP provides:

One integration surface. Reuse a client across compatible MCP servers, subject to their supported revisions, capabilities, transport and authentication requirements. The alternative — bespoke integration per system — is the cost this eliminates, and at enterprise scale that cost is enormous.

Prefer maintained first-party integrations when their contracts fit. The team that owns a system knows its API, its rate limits and its semantics. A first-party MCP server encodes that knowledge.

Portability across harnesses. goose documents that “any MCP server can function as a goose extension” and that its own built-in extensions “are MCP servers in their own right.”³ Cline exposes local stdio and remote HTTP/SSE servers through its own configuration.⁴ Devin exposes an MCP server for its own capabilities.⁵ A server can be reused where both sides support the required protocol features; host-specific UI and policy behaviour still differ.

Governance neutrality. MCP moved from single-vendor stewardship to a foundation alongside other agent-ecosystem projects.⁶

MCP leaves several responsibilities with the application:

It does not solve selection. MCP makes it easy to connect two hundred tools, which makes Figure 14.1’s problem worse, not better. Connecting four MCP servers can add sixty tools to your prompt in an afternoon.

It does not solve permission. A tool arriving over MCP is a tool. It needs the same policy evaluation, the same risk metadata, the same egress rules as one you wrote (Chapter 15). MCP specifies transport authorisation, but does not implement your per-action business policy, and a harness that trusts MCP tools because they came from a server has no permission boundary at all.

It does not solve trust. A third-party server is third-party code with network access to your systems, and its tool descriptions are text that enters your model’s context. That is a supply chain and an injection surface at once, and it gets its own section below.

It does not make tool descriptions good. A badly-described MCP tool is a badly-described tool. You will find yourself wrapping third-party tools to improve their descriptions, and that is correct.

Protocol snapshot: 9 November 2026#

Rechecked on 9 November 2026: the official current revision remains 2026-07-28. It uses self-contained requests with per-request version/capability negotiation. Requests carry io.modelcontextprotocol/protocolVersion in _meta. Streamable HTTP also carries MCP-Protocol-Version. Servers must implement server/discover, which reports supported versions and capabilities. Clients may call it or send requests directly and handle version errors.

Earlier revisions, including 2025-11-25, use the initialisation handshake. Choose an adapter for the actual peer.⁷

For protected HTTP servers, the authorisation specification requires audience-bound tokens and protected-resource metadata. An authenticated MCP connection still needs resource and action authorisation. Do not pass an unrelated downstream token through the server or confuse a protocol handle with a credential.⁸

Long-running work is now an optional Tasks extension, not an assumption every MCP server supports. Discover support, persist the returned task handle, poll or retrieve its result through the extension’s lifecycle, and respect cancellation semantics. Accepted work is not completed work, and cancelling a task does not undo a remote effect.⁹

14.4Four mechanisms for progressive disclosure#

If the menu cannot be flat, something must narrow it. Four mechanisms, in ascending order of sophistication, and they compose.

Static grouping by task. Define tool profiles and select one per run: research gets search and read tools; remediation gets the operational ones. Trivially simple, and its virtue is under-appreciated — it keeps the visible set stable across the run, which preserves the cacheable prefix (Chapter 8). Dynamic selection that changes the tool list every turn makes definitions volatile and can silently cost more than it saves.

Retrieval over tools. Embed tool descriptions, retrieve the top-k for the current goal. The reference implementation does a keyword-overlap version of this deliberately, to make the shape clear:

def select(self, goal: str, *, k: int = 8) -> list[str]:
    """Progressive disclosure by keyword overlap.

    A deliberately simple stand-in for the real options — embedding search
    over tool descriptions, an LLM router, or a hierarchical namespace. What
    matters architecturally is that *something* narrows the menu, and that
    the narrowing is measurable.
    """
    want = set(re.findall(r"[a-z]{3,}", goal.lower()))
    scored = [(len(want & set(t.keywords)) + (2 if name in want else 0), name)
              for name, t in self.tools.items() if name not in self.always]
    scored = sorted(((s, n) for s, n in scored if s), key=lambda p: (-p[0], p[1]))
    return sorted(self.always) + [n for _, n in scored[:k]]

Two design points. always is the set of tools that survive selection unconditionally — the primitives the agent needs regardless of task, typically read, list and search. And selection is over the goal, not the last turn, precisely so the visible set stays stable within a run.

A meta-tool for discovery. Expose two tools instead of two hundred: find_tools(need: str) returns matching names and descriptions, and invoke_tool(name, arguments) calls one. The model discovers what it needs. This scales to arbitrary catalogue sizes and costs an extra turn per discovery — a good trade when the catalogue is large and the tail is long. Its weakness is that the model must know that a capability might exist in order to look for it, which argues for a short, high-quality overview of categories in the system prompt.

Hierarchical namespacing. Expose namespaces as tools, and drill down: jira.* presents as one entry that expands. This mirrors how humans navigate large APIs and keeps the top-level menu small, at the cost of an extra hop and a namespace design you have to maintain.

14.5Replay the menu as a measurement#

Treat the complete visible menu as a versioned artefact under evaluation. The following hypothetical dataset illustrates the protocol and arithmetic; it does not report production measurements.

Construct 240 representative tasks for the hypothetical Reference Agent deployment. A real team would sample these from reviewed production traces, redact them, and hold back part of the sample; this example simply stipulates the corpus. For each task, label an acceptable set rather than one canonical tool: a request for a customer escalation may legitimately begin with either the incident or the account record. Pin the model configuration and the exact visible-menu version, then run every task five times. That produces 1,200 attempts. The repetition matters because a tool that is selected correctly once and incorrectly four times is not a passing tool.

Suppose 1,083 attempts choose an acceptable tool first. First-call selection accuracy is therefore 1,083 / 1,200 = 90.25%. That number is useful, but the remaining 117 attempts contain the decision. Classify them rather than averaging them away:

Now group the 117 wrong first calls by expected tool and selected tool. Seventy-two are search_sharepoint → search_confluence; thirty-one are get_incident → get_ticket; the remaining fourteen are scattered. The top two pairs account for (72 + 31) / 117 = 88.03% of wrong first calls. A generic instruction such as “choose tools carefully” is aimed at all 230 tools. The evidence points at four names.

Rank confusion pairs across both wrong-first categories. A recovered error still reveals a selection defect; recovery changes its cost, not its classification. For each pair, preserve the selected tool, expected capability, recovery turns and terminal outcome. That lets you compare interface ambiguity separately from the agent’s ability to repair after receiving an irrelevant result.

One candidate menu combines the two hypothetical search tools as search_wiki(query, source) and makes source an explicit enum. Replaying the same 1,200 attempts against that candidate yields an illustrative 1,158 acceptable first calls, or 1,158 / 1,200 = 96.50%, while unrecovered wrong first calls fall from 38 to 13. The comparison remains interpretable because the tasks, labels, model settings and attempt count stay fixed. Changing all four would produce a different experiment.

This evaluator is a production extension. The reference harness records visible-tool counts and provides deterministic selection, but has no labelled corpus, repeated-trial runner, confusion matrix or menu-version store.

Two controls stop the replay from lying to you. First, include tasks that require no tool. Otherwise a selector that always calls something can look excellent. Second, label the required capability before the concrete tool name. If search_wiki replaces two older tools, the ground truth should survive the rename. Your benchmark is an executable statement of user intent, not a museum of yesterday’s API.

Use counterfactual menus to locate the fault#

A confusion pair tells you where to look, not which repair will work. Run the affected tasks against three counterfactual menus before changing production. Menu A is the incumbent. Menu B keeps both tools but rewrites each description with an explicit boundary: “engineering material only; not customer policy” and the reverse. Menu C exposes only one member of the pair when the task profile already determines the repository. A fourth candidate, Menu D, replaces both with the merged tool.

If B removes the error, the interface was ambiguous and a description change may be enough. If B fails but C succeeds, the goal-to-profile router has useful information that the model lacks; static grouping is the cleaner boundary. If only D succeeds, the distinction did not belong in the model’s decision at all. Put it in an argument or behind the implementation. This sequence prevents the team from adopting the most invasive fix first.

Keep outcome labels unchanged across all four menus. For the 72 affected attempts, suppose A selects correctly 0 times, B selects correctly 39 times, C selects correctly 70 times and D selects correctly 72 times. The comparison does not prove that merging is universally best. It proves that, for this labelled task slice, tool identity carries no decision value the model can use reliably. You can then inspect whether permission, data residency or ownership requires the systems to remain separate behind the merged interface.

Do not train the benchmark into the descriptions. Hold back at least one task formulation per capability and rotate fresh production examples into the set after review. A description that memorises two hundred phrasings can score perfectly while remaining useless on tomorrow’s request. The replay should reward the intended boundary, not literal keyword overlap — especially when the production selector itself is lexical.

14.6Read the reference selector as a seam#

The reference implementation separates catalogue membership from model-visible capability:

Read the registry methods together:

def register(self, *ts: Tool, always: bool = False) -> ToolRegistry:
    for t in ts:
        self.tools[t.name] = t
        if always:
            self.always.add(t.name)
    return self

def schemas(self, names: Iterable[str] | None = None) -> list[dict[str, Any]]:
    chosen = self.tools.keys() if names is None else names
    return [self.tools[n].schema() for n in chosen if n in self.tools]

def schema_tokens(self, names: Iterable[str] | None = None) -> int:
    from .model import estimate_tokens
    import json

    return estimate_tokens(json.dumps(self.schemas(names)))

register() owns the full catalogue. schemas() creates a view over it. schema_tokens() measures the weight of either the catalogue or that view with the same estimator the rest of the harness uses. That separation is load-bearing. If selection returned copied Tool objects into a second registry, risk metadata and implementations could drift. If it rewrote descriptions in place, one run’s optimisation could change another tenant’s interface. Returning names means the catalogue remains authoritative and the visible set remains a cheap, inspectable projection.

The next boundary sits inside Tool.schema(). A Tool holds mutates, external, reversible, cost and tags, but the schema handed to the model contains only name, description and input shape. That is deliberate. Selection decides what the model may request. The policy engine later decides what the harness may execute (Chapter 15). Hiding a tool is useful exposure control, but it is not authorisation: a model-generated name is still checked against the registry and policy before invocation.

There are also limits you should state plainly. register() overwrites an existing name without an approval workflow. The registry has no persistent owner, provenance, version, definition hash or deprecation state. select() uses lexical overlap, drops every zero-overlap tool except the always set, and has no calibrated confidence or fallback search. schema_tokens() estimates rather than using a provider’s exact tokenizer. None of these are hidden production features. They are the seams where a production registry, router and evaluator attach.

One property is worth preserving when you replace the selector: determinism after ranking. The implementation sorts candidates first by descending score and then by name. Equal-scoring tools therefore produce the same visible order across runs. Stable ordering helps the cacheable prompt prefix (Chapter 8), makes a replay interpretable, and prevents a set iteration order from becoming an accidental experiment variable. An embedding router can return better candidates while retaining deterministic tie-breaking and a pinned menu for the lifetime of a run.

The reference loop calls _visible_tools() before each model turn, but selects against self.state.goal, which is stable across the run. With the reference keyword selector, the same catalogue and goal therefore yield the same names. A production retrieval service should make the contract explicit: compute a menu_version and visible_tool_ids at admission, record both in the run state, and reuse them until termination. That machinery is a production extension. It prevents a registry deployment halfway through a twenty-turn investigation from changing the agent’s vocabulary on turn eleven.

This is the practical test for a replacement selector: can you log the goal, catalogue version, returned names, scores and selector version, then reconstruct why a tool was visible six weeks later? If the answer is no, you have a recommendation service inside an execution path with no audit trail. Better semantic recall does not compensate for an unrepeatable action surface.

Preserve the seam when the registry becomes remote#

A production registry will probably sit behind a service boundary, but the loop should still receive the same narrow answer: an ordered list of tool identifiers tied to one catalogue version. Do not let the registry return executable clients, policy verdicts or prompt fragments. Executable clients belong to the action layer; verdicts belong to policy; prompt construction belongs to context assembly. A registry that returns all three has silently become a second harness.

Treat resolution as a two-stage operation. At admission, select(goal, catalogue_version) returns identifiers, scores and a selector version. Immediately afterwards, schemas(identifiers, catalogue_version) resolves immutable interface definitions. Hash the ordered definitions and record that hash with the run. At execution, the harness resolves the requested identifier against the same version and reads current risk policy separately. This prevents a description update from altering the model-visible contract while a run is in progress, without freezing emergency policy revocations.

Failure behaviour matters more than the happy path. If the registry is unavailable before the first turn, use a deliberately configured minimal profile or decline the run; do not substitute the full local cache because “more tools” feels safe. If one selected identifier cannot be resolved, fail admission with the missing name and catalogue version. If the registry changes after admission, the running task retains its interface view while the policy engine may still deny a newly forbidden action. The interface can be stable without making authority stale.

14.7Put the visible menu on a budget#

“Keep it small” is not an engineering requirement. Give the visible menu a token ceiling, a selection-quality floor and a latency allowance, then do the arithmetic before choosing a disclosure mechanism.

Take Reference Agent’s 230-tool catalogue and use measured definitions from one proposed production snapshot. Suppose the definitions total 31,050 tokens, an average of 31,050 / 230 = 135 tokens per tool. A flat menu on an eighteen-turn run contributes 31,050 × 18 = 558,900 logical input tokens. The repeated definitions may receive a cache discount from the model provider, but they still occupy the request and compete for the context window. Cache pricing changes the invoice; it does not restore the model’s attention or selection margin.

Now pin twelve tools per run: four always primitives plus eight selected tools. At the same measured average, the visible definitions occupy 12 × 135 = 1,620 tokens per turn and 1,620 × 18 = 29,160 across the run. Disclosure removes 558,900 - 29,160 = 529,740 logical input tokens from that illustrative run, a 94.78% reduction in definition traffic. At a nominal uncached input price of $3 per million tokens, the upper-bound saving is 529,740 / 1,000,000 × $3 = $1.58922 per run. Do not put that number in a business case without applying your provider’s cache rules and actual turn distribution. Do use it to notice that menu architecture can dominate a model-price negotiation.

The cost calculation is the easy half. Put expected rework beside it. Suppose a repeated selection benchmark observes a 3.20% wrong-first-call rate for the flat menu and 1.10% for the twelve-tool view. Ten thousand daily runs with twelve tool-choice opportunities each produce 120,000 opportunities. The flat menu yields 120,000 × 0.032 = 3,840 wrong first calls; the disclosed menu yields 120,000 × 0.011 = 1,320. That is 2,520 avoided mistakes per day. If 75% of wrong calls recover and each recovery consumes two extra turns, disclosure avoids 2,520 × 0.75 × 2 = 3,780 recovery turns. The remaining 25% is more important: those are outcome errors, not token overhead.

Every figure in that paragraph is illustrative. The method is the deliverable. Measure your definition sizes, turn-count distribution, tool-choice opportunities, repeated wrong-call rate and recovery length. Keep p50 and p95 separate: a selector that works for routine ten-turn tasks but exposes eighty tools on the rare incident path will hide its largest cost and risk inside an average.

The budget also forces a decision about routing latency. Static profiles add effectively no retrieval hop. A remote embedding selector might add, for example, 85 milliseconds once at run admission. On an eighteen-turn run that is not 18 × 85 = 1,530 milliseconds if you pin the view; it is 85 milliseconds total. Re-selecting every turn would pay the larger number, destabilise the prefix and allow menu drift. The performance requirement should therefore read “select and pin within 100 milliseconds at admission”, not “retrieve tools quickly”.

Set an admission contract for the visible menu:

1. The selected definitions fit under the token ceiling. If they do not, fall back to a smaller static profile or the find_tools meta-tool; never silently expose the full catalogue.
2. All mandatory primitives are present, and every returned name resolves in the pinned catalogue version. An empty semantic match must fail closed to a known minimal menu, not expand to everything.
3. The exact view is recorded on the trace before the first model call. Selection latency, definition tokens and selector version belong beside it.
4. The candidate menu passes the repeated benchmark against the incumbent before deployment. Lower tokens with worse unrecovered errors is not an optimisation.

This contract separates a capacity limit from a quality claim. Twelve is the reference harness’s illustrative threshold. It is not a universal constant. Your production ceiling is the largest visible set that stays within the token budget and clears the reliability floor for your own confusion clusters. Increase it by evidence, not because a newer context window made the graph look less crowded.

14.8Code mode: tools as an API surface#

Instead of exposing tools as individual callable definitions, expose them as an API in a sandboxed execution environment and let the model write code that calls them. The model does not receive two hundred definitions; it receives documentation and a Python interpreter.

Definitions load on demand. Instead of 40,000 tokens of schema, you provide a module reference the model can introspect. The catalogue is discovered, not recited.

Composition avoids repeated model turns. Filtering, joining, looping and conditional logic across tool results happen in the sandbox rather than through a sequence of turns. “For each of the fourteen ledgers, fetch the balance and report the three that disagree” can be one code block instead of many model turns; the underlying API calls still occur — which is a latency, cost and reliability win simultaneously.

Intermediate results stay out of context. Fetching fourteen ledgers and reporting three discrepancies means three lines enter the window, not fourteen ledgers. For data-heavy work this is the dominant benefit, and it is the same insight as Chapter 9’s externalisation.

The research support for code-as-action is direct: expressing agent actions as executable code rather than as JSON tool calls improves performance while requiring fewer interaction turns, because code can express loops and data transformations compactly. Structured tool calls can also be batched or orchestrated; the cited benchmark is not a universal comparison.²

The costs are equally real. You need a sandbox, which is Chapter 16 and a genuine piece of infrastructure. Permission enforcement must cover every effect made by the program. A checked callback can keep policy in the host; it need not move inside untrusted code (Chapter 15). And a code error is a less legible failure than a schema validation error, so Chapter 13’s error discipline has to extend to tracebacks.

A practical architecture combines both. Keep individually vetted tool definitions for common, sensitive actions. Expose the long tail through an API in a code-execution tool. Preserve policy checks on every sensitive effect.

Implementations available now#

The pattern is no longer limited to a custom router. OpenAI documents deferred functions and MCP definitions using defer_loading: true plus tool_search; loaded tools become callable through their normal contract. Anthropic documents Tool Search and programmatic tool calling as complementary mechanisms. Check supported models, discovery limits, cache behaviour and request pricing before using either; neither removes the need to evaluate selection on your own catalogue.¹⁰¹¹

14.9Third-party tools are supply chain#

The moment your agent uses a tool you did not write, you have taken on a dependency with three attack surfaces, and only one of them is familiar.

Code execution. An MCP server is a process, often with credentials and network access. Standard supply-chain hygiene applies: pin versions, review what you run, prefer first-party servers from the system’s own vendor, and do not install a server whose source you cannot inspect for a capability you could implement in fifty lines.

Description injection. This one is specific to agents and frequently overlooked. A tool’s description enters your model’s context on every turn. A malicious or compromised server can put instructions there — and they arrive in the most trusted part of your prompt, the part describing what the agent can do. The defence is to treat third-party tool descriptions as untrusted content: review them at registration time, pin them, and alert on change. A tool whose description changed without a version bump is an incident.

Result injection. Tool results are untrusted content (Chapter 11, Chapter 21), and a third-party tool is a channel for arbitrary text into your context. Nothing about the protocol changes this.

The operational answer is a registry with a gate: tools do not become available because someone connected a server, they become available because someone approved them.

14.10Pruning is the highest-return maintenance task#

The two hundred and thirty tools were not added maliciously. They accreted: a tool per integration, a tool per team’s preference, a tool someone needed for a demo. Nobody removed anything, because removal requires knowing nothing depends on it and adding requires knowing nothing.

Pruning improves selection accuracy directly, and it is the cheapest quality work available in a mature tool layer. Four heuristics.

Review zero-usage tools. Check seasonal, disaster-recovery and blocked-discovery cases before deprecation. Rare tools can remain discoverable without occupying every prompt.

Merge near-duplicates. search_confluence and search_sharepoint become search_wiki(source=...). This is usually the top confusion pair, and merging fixes it permanently rather than by better naming.

Collapse thin wrappers. Five tools that each hit one endpoint of the same API become one tool with an operation parameter, or move to code mode.

Demote the tail. Tools with real but rare use move behind find_tools instead of living in the prompt.

Keep the visible tool set within a measured definition budget and reliability floor as the catalogue grows.