The Model as a Component

Your agent depends on a model you did not build and cannot test exhaustively. Its provider controls releases. Treat it as a component with an interface so you can test orchestration offline, change providers and track usage.

2.1The narrow interface#

The synchronous, text-first teaching harness uses this interface (imports and domain types omitted):

class ModelClient(Protocol):
    """The entire surface the harness needs from a language model."""

    name: str

    def complete(
        self,
        messages: Sequence[Message],
        tools: Sequence[dict[str, Any]] | None = None,
        *,
        max_tokens: int = 4096,
        temperature: float = 0.0,
        thinking_budget: int = 0,
    ) -> ModelResponse:
        ...

One method keeps this loop testable. Production adapters may additionally need asynchronous execution, cancellation, streaming, multimodal items and opaque continuation state. Unsupported parameters must be rejected or omitted; three numeric knobs do not describe every provider’s API.

Offline testability. A ScriptedModel replays fixed turns so tests can assert retries, guardrails, approvals, compaction and terminal outcomes without a provider call. The harness depends on the protocol rather than the SDK.

Failover as configuration. Chapter 29’s gateway routes between providers. That is a config change if the harness depends on a protocol and a rewrite if it depends on an SDK.

Cost and usage accounting in one place. Every call goes through one seam, so token and cost attribution is a decorator rather than an audit.

Immunity to provider-specific concepts leaking into the core. Provider-specific message shapes, streaming events and token-counting assumptions can spread until swapping providers requires changes throughout the core.

2.2Where the narrow interface stops#

An abstraction can become a lowest-common-denominator trap. If one provider can preserve reasoning across tool calls, another can guarantee a schema, and a third can cache a stable prompt prefix, pretending those capabilities are identical does not make the differences disappear. It merely moves them into undocumented adapter behaviour. The core remains visually tidy while production behaviour depends on which adapter happened to receive the request.

Separate portable semantics from negotiated capability. The portable response contains the facts the loop needs regardless of provider: an assistant message, zero or more tool-call proposals, a finish reason, usage, latency, and a provider request id for diagnosis. Everything else is either a declared capability or opaque diagnostic metadata. A provider’s streaming event names, cache-control headers and internal content-block vocabulary do not become domain types simply because the SDK exposes them.

The role declares what it requires; the adapter declares what it supports; deployment fails before traffic if the two do not match. Reference Agent’s invoice extractor may require strict structured output but no reasoning stream. Its reasoning core may require native tools and continuity across a tool sequence. Its offline judge may require neither, but must expose enough usage and version metadata to make an evaluation reproducible. Those are three clients of the same complete method and three different capability contracts.

Do not emulate a missing semantic with a prompt. Adding “return JSON” does not create strict structured output. Telling a model to “think carefully” does not create a reasoning budget whose tokens can be metered and retained deliberately. Asking for two actions in a numbered list does not create parallel tool calls with independent ids. Prompt-level approximations can be useful during a prototype, but they are not interchangeable with transport guarantees once the result drives action.

The same discipline applies in the other direction: do not promote every useful optimisation into the universal protocol. Prompt caching is valuable, but the loop does not need a cache_breakpoint concept. The adapter can construct a cache-stable request from the same messages and report whether the provider accepted the cached prefix. The trace can record the result. If cache mechanics enter the orchestration loop, a pricing optimisation has become an architectural dependency and failover is no longer a configuration change.

Reject unsupported pairings or degrade the role explicitly. The gateway may route extraction to a model without extended reasoning because extraction never required it. It may not route a write-capable tool sequence to a text-only adapter that scrapes actions from prose, because that changes the safety and reliability contract. A fallback that returns an answer is not necessarily a fallback that preserves the system’s guarantees (Chapter 22).

There are also cases where the direct SDK is the right answer. A two-day feasibility spike built to test one provider-exclusive capability benefits from direct access and loses little from coupling. A product whose entire proposition is that exclusive capability may reasonably accept lock-in. The boundary becomes necessary when you need offline control-flow tests, multiple roles, a second provider, central usage accounting, or a model upgrade that does not require rewriting the loop. At that point the provider SDK belongs behind the adapter even if only one adapter exists today.

A protocol without adapter conformance tests is only a drawing. Give every adapter the same small suite before allowing it into the gateway. One fixture asks for a final answer and checks that content and finish reason survive normalisation. One asks for a single tool and checks the name, typed arguments and provider call id. One permits two independent tools and verifies that neither proposal is dropped. One forces output exhaustion and checks that truncation is distinct from a legitimate stop. One requests an unsupported capability and checks that the adapter rejects it before spending a network call. One records returned usage and proves that absent categories remain absent rather than being filled with zeroes.

Zero thinking tokens means the provider reported none; an absent thinking field means the adapter does not know. Zero cached tokens means no cache was used; an absent cache field may mean the provider does not expose the number. Collapsing “unknown” into zero makes accounting look precise while assigning cost to the wrong role. Preserve uncertainty at the boundary and resolve it in reporting, where estimates can be labelled as estimates.

Streaming needs one more contract test because it creates partial state. Cancel a response after the first content fragment and assert what the adapter returns, what usage is available, and whether the partial assistant message may enter history. If one adapter records partial text and another discards it, a retry sees different context and provider failover changes control flow. The portable rule should be explicit: fragments are display events until a completed response is assembled; cancelled fragments may be traced, but they do not silently become a committed model turn (Chapter 5).

Run these tests in two layers. The harness suite uses ScriptedModel and proves that orchestration depends only on the protocol. The adapter suite exercises the provider’s actual test endpoint or a recorded transport fixture and proves that SDK objects are translated without semantic loss. A live provider cannot make a control-flow test deterministic, and a scripted double cannot prove that an SDK upgrade renamed a finish reason or changed a usage field. The two suites guard different sides of the same seam.

For Reference Agent, the release artefact for an adapter is therefore more than code: supported capability flags, conformance results, the SDK version hidden behind it, and one captured normalised response for each terminal shape. When the SDK changes, rerun the adapter suite even if the model id does not. When the model changes, rerun the role evaluations even if the adapter code does not. This separates transport drift from behavioural drift, which turns “the model got weird” into a diagnosis you can act on.

2.3What actually varies#

Model marketing emphasises benchmark scores. For an agent, five other properties matter more, and they vary independently of each other.

The one that most often surprises teams is the third. τ-bench’s central finding was that agents which succeed once are frequently inconsistent across repeated trials — under 50% single-pass and pass^8 below 25% in one domain.¹ These are June 2024 results for the paper’s evaluated models, not current frontier scores. Two models with identical average single-pass scores can have different task-level reliability, and reliability is what predicts production for anything autonomous.

The one most often over-weighted is raw reasoning benchmark performance. It matters for hard reasoning steps, while routine file reads, tool selection and summaries may depend more on accuracy, latency and reliable argument construction. Measure the actual mix; there is no universal percentage of easy turns. Chapter 29’s routing exists because those are different jobs.

2.4Reasoning controls and continuation#

Reasoning support varies by model: an API may expose an effort level, a token budget, adaptive reasoning, a summary, or opaque continuation items. A configured budget is not a promise that all of it will be consumed. Evaluate the permitted controls on representative decision turns, including their latency and output-token cost.

Preserve continuation in the form the provider specifies. OpenAI documents opaque reasoning items and optional summaries. It does not expose raw reasoning tokens. Manually managed Responses histories must preserve required output items. Converting them into prose can lose continuation state. OpenAI reasoning guide, checked 9 November 2026.

For users and audits, show concise action explanations, evidence and verified results. If the API exposes a reasoning summary, label it as a summary, apply the same retention and access controls as other sensitive content, and do not present it as a faithful causal account of the model’s internal computation.

2.5Context: advertised, usable, and affordable#

Advertised context is the documented API limit.

Usable context is the range that meets your task-quality target. Historical lost-in-the-middle results motivate position tests, but establish no universal reduction from a current model’s limit.²

Affordable context is what your latency and cost budgets permit. Repeated input, cache rates, retries and tool charges all matter.

Place a fact at 10%, 50% and 90% depth in contexts of 20k, 100k and 200k tokens, then query it. This diagnoses retrieval by position. Add multi-document reasoning, tool choice, contradiction handling and full trajectories before setting a compaction threshold (Chapter 9).

Two derived design consequences. Choose compaction triggers from measured quality and required headroom. Fractions such as 60–75% are starting points for experiments, not universal optima. And a bigger window is not a substitute for retrieval precision: filling it degrades attention on the thing you cared about, and costs the fill on every turn.

2.6Sampling and native tool support#

Use only parameters supported by the chosen model. Some reasoning models restrict sampling controls; a blanket temperature=0 policy can fail before inference. Where configurable, choose sampling settings through repeated task evaluations. Low temperature can reduce sampling variation but does not make model serving, retrieval or external state deterministic.

Reserve sufficient input and output capacity, including reasoning where the API counts it in the output budget. Treat a length limit, refusal, cancellation or incomplete response as a distinct state; absence of tool calls alone does not establish completion.

Native function calling reduces action parsing. It does not establish that the chosen tool, target or arguments are appropriate. OpenAI documents strict schemas and configurable parallel calls. Your application still executes custom functions and checks their effects. Execute dependent actions serially. Function calling, checked 9 November 2026.

Keep the orchestration core readable, while keeping validation, authorization, cancellation and recovery explicit. Production correctness is not measured by whether the loop fits into twelve lines.

2.7The exogenous dependency#

A model can change without a change to your code.

A library you pin stays pinned. A model provider deprecates a version, re-tunes a safety filter, adjusts a default, or degrades under load — and your quality moves with no change to your code.

Use an immutable snapshot identifier where the provider offers one. A model name that looks versioned is not proof of immutability. Verify the provider’s documented alias/snapshot semantics, record the returned identifier and SDK version, and treat alias changes as model upgrades.

Make the model id part of the versioned tuple. Chapter 25’s discipline: prompt text, model id, temperature, token limits and tool schema hash are versioned together, because all of them are behaviour.

Re-run your eval suite on every model change, including ones you did not initiate. A scheduled weekly run of the suite against the pinned model is a cheap detector of a provider-side change.

Have a second provider wired and tested, with a drill that forces traffic to it (Chapter 22). Single-provider availability limits the availability you can promise.

2.8A model upgrade that passed the headline test#

In an illustrative Reference Agent upgrade, The current model and a candidate are run over the same 500-case suite. The candidate rises from 87.2% task success to 89.0%, costs slightly less per successful run, and has a better median latency. The release looks obvious.

The suite, however, contains three unequal slices:

The aggregate is mathematically correct and operationally misleading. Four hundred low-consequence research cases outweigh twenty payment investigations. The candidate gains fourteen successes in the large slices and loses five in the small one, so the weighted total improves while the role you are least willing to regress loses one success in four. A leaderboard-shaped release gate promotes it.

The first production symptom is not an obvious outage. Operators report that Reference Agent asks for approval on payment investigations more often, sometimes after presenting the wrong transaction. The overall completion dashboard remains green because research dominates the volume.

The trace shows the actual mechanism. The candidate prefers a broad search_transactions call where the current model first calls a scoped get_invoice. The broad search can return several records with similar references. On five evaluation cases, the candidate anchors on the first plausible match, builds a coherent explanation around it, and proposes an action that the policy correctly sends for approval. The policy is doing its job. The model has not become generally worse. A changed tool-selection preference exposed ambiguity that was already present in the tool set.

The immediate fix is to keep the current model on the payment-investigation role. The durable fix has three parts. First, make the tools easier to distinguish: the broad search description says that it is for discovery when no exact invoice id exists, while the scoped lookup becomes the required first call when an id is present (Chapter 13). Second, add counterfactual cases where both tools could return plausible data but only one is correctly scoped. Third, promote models per role, with a non-regression floor for each critical slice rather than one weighted score.

The promotion gate should read like an operational contract, not a model report card. For a write-adjacent role, require the candidate to clear all of these independently: task success, correct tool selection, argument validity, unsupported-claim rate, approval escalation rate, and p99 latency. Add cost per successful run rather than cost per call, because a cheap call that triggers two retries is expensive. Run repeated trials on the highest-consequence cases, because one pass does not tell you whether the behaviour is reliable (Chapter 24).

Any aggregate can hide a regression concentrated in a rare tenant, language, tool, permission class or context depth. Your suite must preserve those dimensions and your release decision must be able to say “better overall, rejected for this role”. A single global model choice makes that impossible. Role routing makes the containment decision ordinary configuration.

It also clarifies what pinning can and cannot do. An exact id gives you a stable name to test and roll back to. It does not prove that every request behind that name was served identically, nor does it make the surrounding retrieval and tools stand still. The weekly run catches movement; the trace identifies where it entered; the adapter makes rollback possible; the slice gate prevents a broad improvement from purchasing a narrow failure.

2.9Choosing per role, not per system#

The most common model-selection error is choosing a model. A production agent has several jobs with different requirements, and matching them separately is one of the largest cost levers available.

Two notes on that table. The judging row uses a different model family deliberately — a different family can reduce some shared biases, but does not establish statistical independence (Chapter 24). And the sub-agent worker row has empirical support: a configuration with a stronger lead and cheaper subagents has been reported outperforming a single strong agent on research evaluation, which is a task-specific quality finding; it does not establish lower total cost.³

2.10Worked arithmetic: price the trajectory#

Per-token prices invite you to estimate one call. Agents spend money across a trajectory, and the repeated input is where intuition fails. Work through an illustrative twenty-turn Reference Agent task. Assume each request carries 8,000 tokens of operating contract, tool schemas and stable tenant context. The initial task and retrieved evidence add 2,000 tokens. Each completed turn adds another 1,200 tokens of messages and tool results. Ordinary answers and tool proposals average 500 output tokens per turn.

Without compaction, input on turn t is:

8,000 + 2,000 + 1,200 × (t - 1)

Turn one therefore sends 10,000 input tokens. Turn ten sends 20,800. Turn twenty sends 32,800. The trajectory total is the sum of a fixed part and an arithmetic series:

20 × 10,000 + 1,200 × (0 + 1 + ... + 19)

= 200,000 + 1,200 × 190

= 428,000 input tokens

The task sends 428,000 input tokens, thirteen times the final request’s 32,800, because earlier context is paid for again on every turn.

Now add thinking. For this arithmetic, assume each of twenty turns actually consumes 2,000 reasoning tokens, producing 40,000. A configured budget is a ceiling, not measured consumption. Ordinary output contributes another 20 × 500 = 10,000, for 50,000 output-billed tokens in total. Suppose instead that trace review identifies four genuine decision turns — planning, evidence synthesis, action selection and final verification. Giving those four turns 4,000 thinking tokens and the mechanical turns none produces 16,000 thinking tokens. The selective policy removes 24,000 output-billed tokens without starving the turns that choose.

Compaction changes the input side. Compact after turn ten, replacing all 14,000 dynamic tokens — the 2,000-token initial task and evidence plus 10 completed turns at 1,200 tokens each — with a 2,400-token summary, while leaving the 8,000-token stable prefix intact. The first ten turns cost:

10 × 10,000 + 1,200 × (0 + 1 + ... + 9) = 154,000 input tokens

For turns eleven through twenty, dynamic history starts again at 2,400 tokens and grows by 1,200 per turn:

10 × 10,400 + 1,200 × (0 + 1 + ... + 9) = 158,000 input tokens

The compacted trajectory therefore uses 312,000 input tokens, a reduction of 116,000 or 27.1%. This is not free quality. The summary can omit a constraint, and Chapter 9 treats compaction as a lossy state transition for exactly that reason. The arithmetic tells you how much cost is at stake; the compaction evaluation tells you whether you may safely claim it.

Put illustrative rates on the totals — $2 per million input tokens and $10 per million output tokens, chosen only to make the calculation concrete:

These totals exclude the compaction call itself. If that call reads 14,000 tokens and emits 2,400 at the same illustrative rates, it adds $0.028 + $0.024 = $0.052, making the compacted total $0.936. Any summariser instructions, reasoning, tool fees or cache effects also belong in the measured total. Before that overhead, the combined policy saves $1.356 - $0.884 = $0.472 per twenty-turn run, or 34.8%; including the stated $0.052 overhead, it saves $0.420, or 31.0%. At 30,000 such runs per month, the difference including that compaction call is $0.420 × 30,000 = $12,600. if only half the runs reach twenty turns, multiplying the twenty-turn cost by every run would overstate the saving by roughly a factor of two.

Count tokens by turn and role, not by request average. A single mean hides the triangular growth of long trajectories and blends cheap routing calls with expensive reasoning calls. Emit input, cached input where the provider reports it, ordinary output, thinking output, model id, role and turn number at the boundary (Chapter 23). Then calculate cost per terminal outcome, because abandoned and failed trajectories still spend tokens.

Optimise the repeated prefix before shrinking answers. In this example, cutting ordinary output by 20% saves 2,000 tokens. Compaction saves 116,000 input tokens. Shorter answers may still improve user experience, especially in voice (Chapter 18), but they are not the first economic lever in a long text trajectory. Stable-prefix caching can change the invoice again, but only if you measure provider-reported cached usage rather than assuming every repeated token received a discount.

Route after you know the volume distribution. Figure 2.3 illustrates a frontier reasoning core with cheaper models in narrower roles; it is a candidate allocation to evaluate. The arithmetic tells you where that matters. A tenfold price reduction on a role responsible for 5% of tokens barely moves the total; a twofold reduction on extraction responsible for 60% does. Count token share, retry rate and success rate per role before redesigning the gateway (Chapter 29).

The counterweight is quality. A policy that saves $0.420 but reduces successful completion enough to trigger retries can cost more overall. If the compacted, selective configuration succeeds 80% of the time and the baseline succeeds 90%, expected cost per success is $0.936 / 0.80 = $1.170 against $1.356 / 0.90 = $1.507; the cheaper policy still wins. If its success falls to 55%, expected cost becomes $0.936 / 0.55 = $1.702 and the saving reverses. Cost per successful run is the economic metric; cost per token is merely an input.

Use this worksheet with your traces rather than these illustrative rates. Bucket runs by trajectory length, compute the repeated-input series for each bucket, separate thinking from ordinary output, and divide total spend by verified success. Then test one change at a time. Compaction, reasoning allocation and role routing interact; changing all three in one release gives you a lower bill and no defensible account of why.

2.11Local and open-weight models#

The case for self-hosting is real: inference can remain inside your infrastructure, and you control deployment and tuning subject to the model licence (Chapter 27). Privacy still depends on tools, logs, telemetry and egress controls. Costs include hardware or GPU rental, power, capacity, staffing and upgrades; self-hosting is not cost-free inference. For an agent processing regulated data in a jurisdiction with hard residency requirements (Chapter 30), this is sometimes the only viable architecture.

Self-hosting makes tool-call reliability your responsibility. General reasoning benchmarks do not establish reliable tool use. Compare hosted and local candidates on the same schemas, tools and task trajectories. Measure selection accuracy (Chapter 14) and repeated-task success (Chapter 24) on your tool set.

One architecture to evaluate, if data policy permits it, is: hosted frontier model for the reasoning core, local models for the high-volume narrow roles in the top half of Figure 2.3. Distilled small models may suit these bounded tasks. Measure their quality and each role’s share of token volume and sensitive data exposure before routing traffic.

2.12What the model does not give you#

It does not know what it does not know, reliably. Chapter 11’s honesty rate exists because an agent will answer from adjacent material rather than report absence, unless the harness and the contract push it otherwise.

It cannot reliably self-correct without external feedback. Chapter 22’s evidence: self-correction loops lacking ground truth can degrade output rather than improve it. Build the verifier.

It does not reliably enforce the distinction between instruction and data. Chapter 21’s structural claim, and the reason the trust boundary is architectural rather than a prompt.

A stateless model call has no durable task memory. Continuation can live in application storage or a provider-managed conversation; learned weights are another kind of retained information. Neither replaces an auditable record of this task (Chapter 10).

It has no notion of your authority model. Permissions are the harness’s, always (Chapter 15).