From Prototype to Production

In this illustrative project, the demo took four days and production took eleven months.

The demo needed a loop, three tools and a model. Production also needed durable state, permissions, budgets, guardrails, tracing, evaluation, tenant isolation, cost attribution, approvals, on-call ownership, a kill switch, an audit trail and progressive rollout.

29.1The maturity ladder#

Six stages define increasing production readiness:

The authority dependencies matter; engineering work can overlap. Stage 2 before stage 3 because you cannot safely grant authority to something you cannot stop. Start evaluation at the demo stage and extend it as permissions arrive; stage 4 represents a mature measurement programme, not the first test. Stage 4 before stage 5 because an SLO you cannot measure is a wish.

Stage 1 is the highest-value fortnight in the whole project. An instrumented read-only agent running against real traffic for two weeks generates the failure taxonomy, the eval cases, the cost model and the trajectory corpus that every later stage depends on. Teams that skip it spend the next six months guessing.

29.2Why pilots stall#

Six recurring causes stall pilots:

No definition of done. Nobody wrote down what “working” means, so the pilot cannot be declared successful or failed. Chapter 24: the eval suite is the definition, and a pilot without one runs until enthusiasm expires.

Cost discovered late. The demo cost pennies on ten tasks; production is ten thousand tasks a day and Chapter 7’s arithmetic was never done. This kills pilots at the exact moment they start working.

No trust ladder. The team asked for full autonomy on day one, security said no, and the project stalled in review. Chapter 15’s autonomy levels exist so the answer can be “read-only now, level 2 in a quarter” rather than yes or no.

The last mile is integration. The agent works; wiring it into the ticketing system, the identity provider, the audit pipeline and the on-call rotation is three months nobody scheduled.

Ownership is unclear. Who is on call? Who approves a prompt change? Who owns quality? An unowned system does not reach production regardless of quality.

The pilot chose a hard task to be impressive. A pilot should be chosen for measurability and blast radius, not for demonstrating maximum capability. The unglamorous high-volume task with a clear correctness criterion is the one that gets to stage 5.

29.3Deployment topology#

Three shapes, and most systems need at least two.

Synchronous request-response. The caller waits. Simple, and only viable for short runs — a few turns, seconds. Fine for a chat turn or a classification. The failure mode is a run that takes longer than the client’s timeout, leaving an orphaned run doing work nobody will collect.

Asynchronous with a work queue. The caller submits and polls or subscribes; a worker pool executes. This is the right default for anything agentic. It gives you back-pressure, retry at the queue level, per-tenant fairness, and the ability to shed load.

Background and scheduled. Long-running or triggered work — a nightly reconciliation, a webhook-initiated triage. Same worker infrastructure, different ingress.

The model gateway deserves specific mention because it is the component most often reimplemented per-service. One endpoint fronting all providers, owning: caching, routing between models, fallback on provider failure, per-tenant spend limits, rate-limit budgeting across the worker fleet, and a single place where every model call is observable. A shared gateway can centralise those controls, but it also needs tenant isolation, availability planning and a documented fallback policy.

29.4Multi-tenancy#

Cross-tenant context leakage can evade quality metrics while violating data boundaries. Enforce tenant scope throughout the system:

Every data access carries tenant scope as a required argument. Not a default, not an optional filter. Retrieval, memory, the graph, the warehouse, the file paths. The test to write is the one that calls without scope and asserts a clear missing-scope error and no access. Derive scope from authenticated host identity, not a tenant argument chosen by the model; an empty result alone can conceal misconfiguration.

Prompt assembly is scoped. Chapter 8’s tiered cache decision interacts here: a shared global prefix with tenant policy in the volatile region is the safe default, and per-tenant prefixes are an optimisation for heavy tenants.

Separate tenant execution boundaries. Two tenants’ untrusted code must not share an unisolated execution environment, and this must be enforced by the allocator rather than by convention (Chapter 16).

Cost, quota and rate limits are per tenant. Otherwise one tenant’s runaway is everyone’s outage — and Chapter 20’s eleven-minute incident becomes a platform incident rather than a tenant one.

Traces and audit records are tenant-tagged and access-controlled. Your trace store now contains every tenant’s data, which makes it a higher-sensitivity system than the agent (Chapter 23).

Test isolation in CI. Run as tenant A and assert that tenant B’s content cannot be retrieved.

29.5Capacity for long-running work#

Agent workloads have a capacity profile unlike request-response services, and the standard playbook mis-sizes them in a specific way.

A run holds a worker slot for minutes to hours, spends most of that time waiting on model and tool I/O, and consumes a provider rate-limit budget shared with every other run. Which yields three consequences.

Plan concurrency together with throughput and duration. Size the pool using admitted arrival rate, expected duration and simultaneous active work; a requests-per-second figure alone omits how long runs occupy capacity.

Workers are I/O-bound, so pack them densely. A worker mostly waiting on a network call does not need a CPU. Async execution or a high thread count per pod is usually the right shape, and provisioning as though each run needs a core is a large waste.

The provider rate limit is the real ceiling, and it is shared. Scaling workers past what your provider quota supports converts a capacity problem into a 429 storm — Chapter 22’s incident at fleet scale. The gateway must own a global token-bucket budget and throttle gracefully rather than letting workers discover the limit individually.

Two more capacity facts worth planning for. Queue depth is your load signal, and it needs per-tenant visibility or a single tenant’s burst reads as a platform problem. And long runs interact badly with deploys: without resumption, a rolling deploy either kills in-flight runs or blocks on draining them. With Chapter 5’s stateless-worker design it is a non-event, which is one more return on that investment.

29.6Build a capacity envelope before an autoscaler#

An autoscaler answers how many workers should start. It cannot answer whether the model provider, the queue, or a tenant’s quota can admit the work. Build that envelope first, using the same units at every boundary. Requests per second is the wrong unit for most of this calculation; runs per minute, concurrent runs, model calls per minute and tokens per minute are the useful ones.

Take an illustrative production projection for Reference Agent. It will investigate 18,000 support cases a day. Sixty per cent arrive during a ten-hour business peak, so the peak arrival rate is 18,000 × 0.60 ÷ (10 × 60) = 18 runs per minute = 0.30 runs per second.

Do not multiply that rate by the median duration. Long tails hold scarce slots, so use a measured duration distribution. Suppose shadow traffic produces three useful buckets:

For this illustrative steady-state estimate, multiply the arrival rate by the average time in the system. Here that is 18 × 6.8 = 122.4 concurrent runs. Define the planning burst as 50% above peak — 27 runs per minute — sustained for ninety minutes, long enough to reach the new concurrency plateau. It raises the requirement to 122.4 × 1.5 = 183.6. If you require the fleet to carry that burst while 20% of its slots are unavailable during a rollout or zone fault, provision 183.6 ÷ 0.80 = 229.5, rounded up to 230 run slots. A load-tested worker replica that safely multiplexes eight waiting-heavy runs therefore requires ⌈230 ÷ 8⌉ = 29 replicas. The eight is evidence from a load test, not a constant to copy.

That still does not prove the system can run. Suppose each completed investigation makes nine reasoning-model calls, averaging 6,000 input tokens and 700 output tokens per call. At 18 runs per minute, the steady provider demand is:

Give the illustrative provider explicit ceilings: 250 calls, 1.2 million input tokens and 180,000 output tokens per minute. Calls and output pass the 30%-reserve row above; input does not. Preserving 30% input reserve leaves 1,200,000 ÷ 1.30 = 923,077 usable tokens per minute. At 54,000 input tokens per run, the admission ceiling is 923,077 ÷ 54,000 = 17.09 runs per minute — below the expected 18.

Now compare the same ninety-minute burst across compatible units. At 27 runs per minute it demands 243 calls, 1,458,000 input tokens and 170,100 output tokens per minute. The call and output ceilings pass, while input fails. Without reserve, the input ceiling admits 1,200,000 ÷ 54,000 = 22.22 runs per minute, so the burst queue grows by 27 - 22.22 = 4.78 runs per minute — about 430 runs over ninety minutes.

That shortfall looks small and behaves badly. The queue grows by 18 - 17.09 = 0.91 runs per minute, or about 55 runs an hour, for as long as the peak lasts. Adding workers cannot change it. You must reduce tokens per run, reduce calls per run, obtain more quota, or shed and defer work. Reducing average input from 6,000 to 5,000 tokens per call changes reserved demand to 18 × 9 × 5,000 × 1.30 = 1,053,000 input tokens per minute, which fits. The improvement must appear in a replay evaluation before it enters the capacity plan; deleting context until the arithmetic works can silently lower quality (Chapter 8).

Turn the envelope into admission control. Estimate the remaining calls and tokens when a run enters the queue, reserve them against the tenant and fleet budgets, and release unused reservation when the run terminates. A high-variance task needs a conservative reservation or a separate queue class; otherwise a few thirty-minute runs consume the budget promised to routine traffic. Route interactive and scheduled work through distinct queue classes so that a nightly reconciliation cannot consume the latency budget of a person waiting in chat.

Measure four values together: admitted runs per minute, concurrency in use, provider-budget utilisation and queue age at the p95. If queue age rises while concurrency is below target, compute is not the bottleneck. If provider utilisation is pinned, adding replicas will amplify retries and 429 responses. If one tenant’s queue age rises while the fleet is healthy, fairness or a tenant-specific dependency is failing. An autoscaler that reads queue depth alone cannot distinguish those mechanisms.

Suppose interactive work must begin within two minutes at the p95. At 18 runs per minute, a one-minute burst at 1.5 times normal creates nine excess runs (27 - 18 = 9) while service remains nominal. If arrivals and service then both stay at 18 per minute, the backlog never drains. The 30% provider headroom must be usable recovery capacity. Increasing service temporarily to 23.4 runs per minute supplies 5.4 runs per minute of drain capacity, clearing the nine in 9 ÷ 5.4 = 1.67 minutes. This fluid estimate is backlog drain time, not a p95 wait guarantee. Load-test the start target with measured arrival and service-time distributions, sustaining recovery throughput within every resource limit.

Do this calculation for a dependency brownout too. If model capacity falls to half for ten minutes, arrivals contribute 180 runs while only 90 complete, leaving a backlog of 90. Returning to the ordinary 18-per-minute rate does not drain it. At 23.4 per minute, recovery takes 90 ÷ 5.4 = 16.67 minutes after the provider recovers. Your users experience roughly twenty-seven minutes of degraded starts from a ten-minute provider event. The options are explicit: accept that recovery objective, hold more reserve, admit less work during the event, or divert a verified task class to another route. Retry without admission control makes the backlog larger.

Per-tenant fairness needs equally concrete policy. Reserve a small interactive floor for every active tenant, distribute the remaining budget by purchased or observed weight, then lend unused shares temporarily. Borrowing must expire: a tenant using another’s idle share cannot keep it when the owner returns. Track both queue age and denied admissions by tenant. A low queue age achieved by rejecting half the work is not healthy capacity, while zero rejection achieved through an unbounded queue is not reliability.

The reference implementation demonstrates the loop, durable events, policy and tests. Queues, autoscaling, tenant scheduling, provider reservations and quota enforcement are production extensions requiring their own failure tests.

29.7Cost engineering#

Evaluate six cost levers against the workload:

Prompt caching. Chapter 8’s stable-prefix discipline, and usually the single largest lever available because it applies to every turn of every run. Cache hit rate is a first-class metric.

Turn reduction. Chapter 7’s arithmetic: turns multiply everything. Better tool errors (Chapter 13), better retrieval (Chapter 11) and per-step verification (Chapter 6) all reduce turns, which means quality work and cost work are the same work here.

Model routing. A cheap model for classification, extraction and routing; the frontier model for the reasoning that needs it. This is the gateway’s job and it is where distillation (Chapter 27) pays off.

Context discipline. Submitted context contributes to billed usage; cached and uncached tokens may have different rates, and caching does not remove context or quota limits. Retrieval precision, compaction, and dropping low-value sections are cost levers as much as quality levers.

Escalation ladders. Chapter 7’s rungs — cache, rules, one call, chain, loop. Measure which task classes can finish on an earlier rung without losing required quality.

Budget enforcement. Chapter 20’s ceilings, per run and per tenant per day. This bounds the worst case, which is what makes the average affordable to commit to.

Cost per task per type on a dashboard. Not total spend — per task, split by task type and tenant. Total spend tells you nothing actionable; cost per reconciliation tells you whether reconciliation is viable.

Unit economics. Compare cost per task with its value or price before implementation.

29.8Pin the rate card, then reconcile usage#

A price is a versioned input, not a property of a model family. The table shows standard, global-routing Claude API prices in USD per million tokens, checked 9 November 2026.¹ It excludes taxes, contract discounts, fast mode and regional premiums.

Haiku 5.5 counts all input tokens, including cache reads and writes, toward its prompt-length threshold. Each request receives its applicable rate. The ledgers below retain illustrative rates. Store the exact model ID, endpoint, region, service tier, effective date and source with each rate card. Recheck before rollout.

For disjoint billable token classes, calculate:

model_cost = (uncached_input × input_rate
            + cached_read × read_rate
            + cache_write_5m × write_5m_rate
            + cache_write_1h × write_1h_rate
            + billable_output × output_rate) / 1,000,000
full_cost = model_cost + tool/runtime/storage charges + human handling

The adapter must normalise provider usage first: some counters include cached tokens within total input; others report separate categories. Never add a subset twice. Include billed reasoning tokens in the provider’s output category and record missing usage as unknown until reconciled. A locally timed-out request may still be billed. Price cold writes, failed attempts and cancellations, then reconcile the ledger to provider billing. Discounts, token quotas and context limits are separate constraints.

29.9Cost the distribution, then divide by success#

An average token count multiplied by a headline model price hides the two costs most likely to surprise you: retries and human review. Build a ledger from the trace of one named task type, then divide the whole amount by successful outcomes. The following prices and workload are illustrative; substitute your contracted prices, including how cached input is charged and counted against quota.

For one Reference Agent support-case investigation, suppose the router uses 1,200 input tokens and 80 output tokens on a small model priced at $0.30 per million input tokens and $1 per million output tokens. The routing cost is (1,200 ÷ 1,000,000 × $0.30) + (80 ÷ 1,000,000 × $1) = $0.00044.

The reasoning loop then makes eight calls. Each call carries 4,000 cached stable-prefix tokens, 2,000 uncached input tokens and produces 700 output tokens. In this scenario, cached input costs $0.30 per million, uncached input $3 per million and output $15 per million:

Add $0.012 for sandbox time, retrieval, trace storage and tool APIs. The nominal task cost is therefore $0.00044 + $0.14160 + $0.012 = $0.15404. At 18,000 attempts a day for 30 days, nominal monthly cost is 540,000 × $0.15404 = $83,181.60.

Now include the tail. Suppose 12% of runs need three extra reasoning calls. One extra call costs (4,000 ÷ 1,000,000 × $0.30) + (2,000 ÷ 1,000,000 × $3) + (700 ÷ 1,000,000 × $15) = $0.01770. Three cost $0.05310, and spread across all attempts the retry tail adds 0.12 × $0.05310 = $0.006372 per attempt. That is $3,440.88 a month, taking machine cost to $86,622.48.

The denominator matters next. At a 94% successful completion rate, 540,000 attempts produce 507,600 successful tasks. Machine cost per attempt is $0.160412; machine cost per success is $86,622.48 ÷ 507,600 = $0.17065. Reporting $0.15404 as “cost per task” omits an operational number that is roughly 10.8% higher because it ignores the retry distribution and failed outcomes.

Human review can dominate both. If 8% of attempts require eight minutes from an operator whose fully loaded time costs $36 an hour, each review costs 8 ÷ 60 × $36 = $4.80. The monthly review cost is 540,000 × 0.08 × $4.80 = $207,360, more than twice the machine bill. Machine plus review is therefore $293,982.48 a month, and the full cost per successful outcome is $293,982.48 ÷ 507,600 = $0.57916. This does not argue against approval gates. It tells you to improve proposal quality, narrow the gate to the capabilities that need it, and include the human seam in unit economics (Chapter 19). Removing review to improve a spreadsheet is not an optimisation if it widens risk.

The ledger also identifies leverage. Without the illustrative cached-input rate, those 32,000 stable tokens would cost $0.096 rather than $0.00960 per task, adding $0.08640. Across 540,000 monthly attempts that is $46,656. Reducing the retry cohort from 12% to 6% saves half of $3,440.88, or $1,720.44. Reducing human-review incidence from 8% to 6% while holding risk outcomes constant saves 540,000 × 0.02 × $4.80 = $51,840. The dominant lever is visible only after the full ledger exists.

Put these rows on the dashboard by tenant, task type, model route and prompt version. Track cost per attempt, cost per success, retry-tail share and human minutes per success. When cost per success rises, decompose it before changing models: more tokens, more turns, lower completion, a cache miss, a route shift, or more review each implies a different fix. Total spend alone tells you only that multiplication happened.

Use the ledger to set two different budgets. A per-run ceiling contains one bad trajectory: maximum model calls, input tokens, output tokens, tool charges and human-review minutes before a named terminal outcome. A portfolio budget controls the tenant or capability across a day. Conflating them creates a blunt control. A $0.25 per-run ceiling will not stop 100,000 legitimate $0.20 runs from exceeding a daily commitment, and a daily limit will not stop one looping run from consuming $40 before the portfolio notices.

For the illustrative case, set the nominal machine envelope from the measured distribution rather than from $0.15404 alone. If the acceptable p99 retry tail is three extra reasoning calls, reserve $0.15404 + $0.05310 = $0.20714 per attempt before tool-specific exceptional charges. The run does not need to spend the reservation, but the admission service must know it can. A $90,000 monthly machine budget is $3,000 on a thirty-day daily basis. At the measured $0.160412 average machine cost, that admits ⌊$3,000 ÷ $0.160412⌋ = 18,701 attempts a day. The 18,000-run forecast leaves 701 attempts of daily headroom. The defined burst adds 9 × 90 = 810 attempts, so that budget covers 701 ÷ 9 = 77.9 minutes, not the full ninety. Compute-slot, call and output-token headroom cover the burst; input-token and daily machine-budget headroom do not. The production decision is to raise the relevant limit, defer lower-value work, or reduce unit demand before claiming burst capacity.

Alert before enforcement. At 70% of a tenant’s daily portfolio budget, report the decomposition. At 85%, slow or defer explicitly designated background classes. At 100%, reject new work with a named budget outcome while allowing already-approved side effects to reach a safe terminal state. Never terminate an executing mutation halfway merely because the accounting clock crossed a threshold. Budget policy belongs beside the permission and recovery policies because it changes what the system may do, not just what finance sees.

29.10Latency engineering#

Streaming changes perceived latency more than any optimisation. Chapter 4’s event streaming — starting to read three files, waiting for approval — matters more than token streaming for long tasks, because what the user needs during minute four is progress, not prose.

Parallelise reads. Chapter 13’s concurrent non-mutating calls, and it is often the largest real reduction available.

Prefetch and pre-warm. Assemble the stable prefix at session start; start likely retrievals on intent. Speculative work is cheap and silence is expensive.

Measure both first useful output and verified completion time. Streaming improves responsiveness, but output length still consumes latency, tokens and user attention. Do not stream an unchecked consequential claim as though it were final.

Set a latency SLO per surface, and measure p99. A p50 of four seconds with a p99 of ninety is a system that feels broken to roughly one user in a hundred beyond the p99 threshold, and that user is the one who tells people.

29.11Rollout#

Roll out through five stages:

Shadow mode. Run on real traffic, divert outputs to a comparison log and disable external effects. Bound data access, export and cost because shadow mode still processes real data. Review human-agent agreement and disagreements before enabling effects.

Canary by tenant. One friendly tenant, real effects, level 1 or 2 autonomy. Small blast radius, real feedback.

Canary by capability. Widen the tenant set while keeping capabilities narrow. Read broadly, write to one place.

Progressive autonomy. Chapter 15’s levels, promoted per capability on Chapter 24’s numbers. Read-only, then propose, then act-with-approval, then act-within-bounds.

General availability with the switch intact. Chapter 20’s kill switch, per tenant and per capability, flippable without a deploy — and rehearsed.

Two properties to insist on throughout. Routing and authority can be rolled back quickly with versioned flags, while effects already committed require reconciliation or compensation. And each promotion has a written criterion decided in advance, because a promotion decided in the moment is a promotion decided on enthusiasm.

29.12The canary that passed in aggregate#

Promotion evidence must have the same boundaries as the harm. Aggregate success can look healthy while one tenant, capability or queue class is already failing.

Consider an illustrative Reference Agent canary across three tenants. The written promotion gate requires at least 95% successful completion for each tenant, p95 latency below ninety seconds for interactive work, cost per success below $0.22, and zero severe authority or isolation events. The run produces 500 attempts:

The launch dashboard shows 96.4% in green. Cost is $83.40 for the cohort, or $83.40 ÷ 482 = $0.173 per success. No authority or isolation event occurred. A manager looking at the aggregate promotes the capability. The written criterion says no, because Tenant West is below 95%.

The six failed West tasks share one mechanism: they require a tenant-specific product catalogue that had not refreshed after the connector’s scope changed. The agent still retrieved documents, so the retrieval service was “available”. It found an older catalogue, produced plausible proposals and reached a terminal state. Standard infrastructure metrics stayed green. A manual read of the six failed trajectories reveals stale evidence; the relevant operational signal is source freshness by tenant, not retrieval latency.

The response is not to lower the gate or retry those six tasks until they pass. Pause that capability for West, leave North and South running, restore and verify the connector scope, rebuild the affected index, and replay the six failures plus a held-out West sample. Promotion remains per tenant because rollback is per tenant. A single global flag would turn a local data defect into a choice between exposing everyone and stopping everyone.

This generalises beyond tenancy. Aggregate p95 can hide interactive latency behind a much larger batch cohort. Overall cost can hide one capability whose loop count doubled. A global refusal rate can hide a broken approval gate for one tool. Slice every promotion metric along the dimensions on which you can roll back: tenant, capability, task type, autonomy level, model route and prompt version.

The sample size is part of the gate. One success out of one attempt is 100%, not confidence. Set a minimum cohort size and an uncertainty requirement using Chapter 24’s method before the canary starts. If a low-volume tenant cannot produce enough observations, hold that tenant at a lower autonomy level longer or require explicit approval; do not let the aggregate lend it evidence it did not generate.

Finally, keep the shadow comparator during the canary. The production outcome tells you whether Reference Agent completed; the comparator tells you whether its answer diverged from the established process. When the two disagree, sample the trajectory and turn the mechanism into an eval case before promotion. Rollout is an evaluation design executed against live boundaries, not a sequence of increasingly optimistic flags.

29.13SLOs and on-call#

Agent SLOs are unusual because correctness is a distribution rather than a boolean. Four that work:

Monitor operational signals alongside errors and latency:

Cost per task rising. The earliest signal of a prompt regression, a widened top_k, or a cache that stopped hitting. Often visible days before quality moves.

Terminal outcome distribution shifting. LOOP_DETECTED rising is a tool regression; BUDGET_EXHAUSTED rising is tasks getting harder or ceilings going stale.

Approval refusal rate moving. Down means the gate may have stopped functioning (Chapter 19); up means the agent got worse at proposing.

Guardrail trip rate spiking. Either an attack or a broken guardrail, and both need attention.

Queue depth per tenant. The load signal that a global queue hides.

A runbook needs four things an agent-specific on-call will actually reach for: how to flip the kill switch and what it covers; how to pin a tenant to a prior prompt version; how to inspect one run’s trace from a customer complaint; and how to drain and resume in-flight runs. Rehearse all four, because an unrehearsed runbook is a document.

The system now runs, scales, and can be operated. One question remains, and it is the one that decides whether it is allowed to: can it satisfy an auditor, a regulator, and a customer’s security team. That is the last chapter.