Code Execution as the Universal Tool

In an illustrative comparison, Reference Agent checks fourteen tenant ledgers against the settlement report and lists mismatches. Timings, costs and errors below are hypothetical.

With a naive serial tool loop, Reference Agent did this in twenty-nine turns. Fourteen read_ledger calls, fourteen read_settlement calls, a comparison the model performed in its head across a context now containing fourteen full ledgers, and an answer. It cost about eleven cents, took ninety seconds, and got one comparison wrong — a rounding difference the model treated as a discrepancy because it was comparing numbers it had read as text, three hundred lines apart.

With code execution, it did the same task in two turns. One turn wrote fourteen lines of Python that fetched both sides, compared them with Decimal, and printed three tenant ids. One turn reported the result. Three lines entered the context instead of fourteen ledgers. It cost under a cent and the comparison was exact, because a subtraction in Python is a subtraction.

Code execution reduces model turns and context, but expands the permission surface.

16.1Why code is the highest-bandwidth action#

Composition saves model round trips. A tool call does one thing and returns to the model. Code does many things and returns once. Loops, conditionals, filters, joins, retries — all inside one action, none costing a turn. The fourteen-ledger comparison is a for loop; a naive serial tool loop makes twenty-eight round trips, although batched tools can reduce that substantially.

Intermediate results stay out of context. This is the dominant benefit for data work and it is the same insight as Chapter 9’s externalisation. The code reads fourteen ledgers; the context receives three tenant ids. The window is spent on conclusions rather than on the raw material.

Computation is reproducible when its inputs and semantics are right. Python can still implement a wrong formula, overflow a precision context or compare the wrong units. Anything involving money, dates, set operations or aggregation should be computed rather than reasoned. The rounding error above is not a model deficiency to be prompted away; it is a category error about which component should do the subtracting.

The environment is discoverable. The agent can list a directory, read a schema, inspect an object, print a type. Instead of you anticipating every capability and defining a tool for it, the agent explores. This is what makes code execution scale where tool definitions do not (Chapter 14).

The research finding is direct: expressing actions as executable code rather than as JSON tool calls improves performance while using fewer interaction turns, on the tasks and models studied. JSON calls can also be batched; code offers compact control flow and data processing, not a universal win.¹

Available implementations and their boundaries#

OpenAI’s Programmatic Tool Calling documentation was checked on 9 November 2026. Each program runs JavaScript in a fresh isolated V8 runtime. It has no Node.js, general filesystem, subprocesses or direct network access. External work goes through enabled tools. The application still executes client-owned functions. This runtime supports orchestration rather than a persistent Python workstation.⁵

Anthropic’s programmatic tool calling uses its code-execution environment to compose tools; its MCP engineering guide shows discovering API definitions and filtering intermediate results before exposing them to the model. These implementations demonstrate the pattern but have different runtimes and lifecycle rules. Verify tool eligibility, retention, resource limits and pricing for the selected provider.⁶⁷

For currency arithmetic, construct Decimal from decimal strings, reject non-finite values, select currency and rounding rules explicitly, and test boundary cases. Decimal(float_value) preserves the float’s binary approximation; a deterministic wrong input remains wrong.⁸

16.2The permission problem code execution reopens#

Here is the cost, stated plainly. Chapter 15 built a boundary where every action was a proposal the harness could inspect, evaluate against policy, and permit or refuse. Unmediated code execution can bypass that boundary. The agent no longer proposes read_file("/etc/passwd"); it proposes run_code(...), and inside that code it can attempt anything the sandbox permits.

The policy engine sees one call. The code makes forty.

Move enforcement to the boundary of the sandbox. If the sandbox itself has no filesystem access outside a working directory and no network except through a proxy, then code inside it cannot exceed those limits regardless of what it attempts. This is the primary control, and it is why Chapter 15’s isolation ladder matters more here than anywhere else: the sandbox is the policy.

Make the sandbox’s API the policy surface. Do not give the code raw HTTP and raw filesystem access; give it a module of functions that are themselves permission-checked. ledger(t) inside the sandbox calls back out to the harness, which evaluates it exactly as it would a tool call. This preserves the audit trail and the three-valued verdict at the cost of a round trip per call, and it is the right design for anything sensitive.

Accept a coarser grain for the rest. For the long tail — computation over data the agent already holds, text manipulation, format conversion — a hardened sandbox with no egress and a scoped working directory is sufficient, and demanding per-operation policy evaluation would destroy the benefit.

So the architecture that works is two tiers: sensitive capabilities remain first-class tools with individual policy evaluation, and the sandbox provides composition and computation over what those tools return. The fourteen-ledger example is well-served by this: ledger() and settle() are permission-checked callbacks; the loop and arithmetic avoid extra model turns but still consume sandbox resources.

16.3The sandbox is a first-class component#

Treat it as infrastructure, not as a utility function. Six properties it needs.

Isolation appropriate to its input. Chapter 15’s ladder. Treat model-written code that reads untrusted content as untrusted code. Choose isolation for the threat model. Container hardening can include dropped capabilities, no new privileges, a read-only root filesystem and seccomp. A container alone may not satisfy tenant or regulatory boundaries. Hermes documents its Docker terminal backend in exactly these terms: dropped capabilities, no-new-privileges, a process limit, and a size-bounded tmpfs.²

Resource limits, all four. CPU, memory, wall-clock, and disk. A model-written loop will occasionally be infinite, and an OOM inside a bounded sandbox is a returned error while an OOM in your process is an outage.

No egress except through a proxy. Denying network access blocks direct network calls. Shared files, artifact publication and stdout can still disclose data, so those outputs need audience and scope checks.

No ambient credentials. As Chapter 15 argued, and more urgently here: an environment variable in a sandbox running model-written code is a secret in the hands of a program you did not write.

Deterministic startup, and a lifecycle you chose. Per-call sandboxes are strongest and slowest; per-run are the usual compromise; long-lived are fastest and accumulate state you did not intend. Choose deliberately and know the answer to “what does the second call see.”

Choose filesystem persistence explicitly. Retain or destroy the working directory according to the task and isolation policy.

16.4The filesystem as agent state#

A sandbox with a persistent working directory turns out to be one of the most useful structures in agent design, and for a reason that has nothing to do with code.

It gives the agent a place to put things that is not the context window. Chapter 9 argued for externalising findings to files; a sandbox with a filesystem is where that happens naturally. Intermediate data, generated artefacts, notes, a work log — all addressable, all re-readable, none consuming context until needed.

/work/
  GOAL.md          the original request, verbatim, never edited
  FINDINGS.md      established facts, appended with provenance
  DEAD_ENDS.md     what was tried and why it failed
  data/            versioned fetched inputs; refresh when required
  out/             products under /work; explicit promotion only
  scratch/         anything; wiped between runs

Compaction can preserve recoverable detail when files remain durable, indexed and accessible. A handoff pointer must identify a findable, verifiable version. The retained files also provide a human-auditable record after the run.

The lifecycle question is real and has three defensible answers. Ephemeral per run gives clean starts and no cross-contamination; the agent re-fetches everything. Persistent per task lets a resumed run (Chapter 5) find its own prior work, which is usually what you want for long-horizon tasks. Persistent per tenant accumulates a working corpus and accumulates risk — stale data, cross-run confusion, and a growing volume nobody prunes.

The default worth adopting: persistent per task, wiped on completion, with anything meant to survive explicitly promoted to durable storage. And a structure named in the operating contract, because — as Chapter 9 noted — agents left to invent their own filing system produce one nobody can navigate, including them.

16.5Failure walkthrough: the process was fresh but the workspace was not#

Isolation and lifecycle are different properties. A sandbox may start a fresh process, with the right tenant identity and no network access, while still inheriting bytes written by a previous run. If you test only process isolation, you can pass every security check and still cross a data boundary through the working directory.

The reference Reference Agent package stops at the boundary: it defines Artifact and tool-result types, but it does not ship a sandbox executor, allocator or artifact uploader. The incident, fleet arithmetic and probes below are explicitly illustrative production extensions. They describe what you must build around the reference action layer, not behaviour hidden inside it.

Consider this illustrative Reference Agent incident. A tenant asks for a CSV of failed settlement jobs. Reference Agent generates the file in /work/out/failures.csv, promotes it as an artifact, and finishes. Twenty minutes later a different tenant asks for the same report. The second run returns nine rows when its checked API returned only seven failures. The extra two belong to the first tenant.

The initial diagnosis is model fabrication. The trace shows seven API records, but the final artifact contains nine. The code is then blamed: perhaps the model appended instead of overwriting. Replay disproves both explanations. The generated program opens its destination with mode "w", writes exactly seven rows, and produces the correct checksum when run in a genuinely empty directory.

The actual mechanism sits below the program. The sandbox allocator checks out a warm container, replaces the process, and mounts a directory named for the pool slot: /pool/slot-12/work. Its cleanup routine deletes /work/scratch but deliberately preserves /work/out so an artifact uploader can finish after the interpreter exits. The uploader marks the first run complete, but a transient object-store error leaves the two-row failures.csv behind. The next checkout mounts the same slot and writes its correct seven rows to failures-run-b.csv. A packaging wrapper concatenates the data rows from every file matching /work/out/failures*.csv into one settlement-failures.csv, removing each input header. The stale two-row file satisfies the glob, so the promoted artifact contains (2 + 7 = 9) rows.

The timeline makes the boundary failure visible:

Nothing crossed the network policy. No ambient credential leaked. The interpreter did exactly what it was asked. The defect was that fresh sandbox meant fresh process in one component and fresh writable state in another. The words described a property no single component enforced.

The fix has four parts. First, give every run a new writable overlay or tmpfs whose identity includes both tenant and run id; never use the pool-slot identity as a storage namespace. Second, make artifact promotion explicit: the program returns artifact ids from a run-owned manifest, and the uploader may read only those named paths. A glob is discovery, not authorisation. Third, quarantine a slot when teardown or upload fails. Do not return it to the pool until a destructive clean-start check proves that the writable layer is empty. Fourth, log the identities that establish the boundary — tenant id, run id, sandbox id, base-image digest and writable-volume id — on allocation, execution and teardown.

Then add a canary that tests the property rather than the implementation. Before checkout, the allocator writes an unpredictable token into a disposable run, tears the sandbox down, and checks out the same warm slot under a different synthetic tenant. The second run must be unable to find the token by filename, directory walk, inode scan or artifact manifest. Run this test against every pool and after every allocator change. A unit test of cleanup() is not enough; the incident lived in the composition of allocator, mount, uploader and retry behaviour.

Response has to preserve evidence without preserving exposure. Stop new checkouts from the affected pool, detach every writable volume it has used, and revoke links to artifacts produced during the suspect interval. Do not begin by deleting the stale file: that removes the shortest path to proving which boundary failed. Snapshot the volume into a restricted incident account, record its content hash and access log, then destroy the production copy. Search artifact manifests and storage access logs by volume id and run id to establish whether stale bytes were merely reachable, were packaged, or were actually delivered. Those are three different incident severities.

Recovery is complete only when replay covers the awkward states. Test a successful upload, an upload that exhausts retries, interpreter termination during packaging, allocator death after mount, and a child process that keeps a file descriptor open during teardown. For each case, the next checkout must receive either a demonstrably empty writable layer or no slot at all. This is where fail-closed is concrete: losing one pool slot is capacity loss; handing an unproven slot to another tenant is a confidentiality risk. The allocator, rather than the model, owns that decision.

The general lesson is broader than containers. Isolation controls who can reach a live resource; lifecycle controls which old resources remain to be reached. A VM snapshot can attach the wrong volume. A per-run Kubernetes pod can mount a reused persistent volume claim. A new interpreter can inherit a host directory. “Fresh compute” is therefore not a security statement until you name the writable layers and prove their state transition. Chapter 10 applies the same discipline to memory deletion; here, the forgotten state is a file rather than a fact.

16.6Tracebacks as prompts#

Return code-execution errors in a form the model can act on.

A raw traceback is better than most tool errors, because it names a line and a type. It is still improvable in four ways.

Return stdout, stderr and the exception together. A traceback without the print statements that preceded it discards the agent’s own debugging output. Return all three, labelled.

Trim the frames the agent cannot act on. Twenty frames of interpreter internals bury the one line in the agent’s own code. Keep the user frames and the exception; summarise the rest.

Echo the failing line. line 4: total += Decimal(row["amount"]) with the exception is far more actionable than a line number against code the agent must reconstruct from memory.

Name the resource limit explicitly. A sandbox kill needs to say which limit and what the ceiling is: Killed: wall-clock limit 30s exceeded. Reduce the work per call or process in batches. A bare “killed” produces a retry of the identical code.

--- stdout (last 20 lines) ---
processing tenant acme-1 … ok
processing tenant acme-2 … ok
processing tenant acme-3 …
--- stderr ---
--- exception ---
KeyError: 'amount'
  at line 4:     total += Decimal(row["amount"])
  row keys present: ['tenant', 'value', 'currency', 'posted_at']

Reporting the keys actually present lets the model repair the failed lookup without guessing.

16.7Artifacts: products that are not text#

Code execution produces things that are not text: charts, spreadsheets, PDFs, images, archives. These must not enter the context window, and they must not be lost.

@dataclass
class Artifact:
    id: str
    media_type: str          # "image/png", "text/csv", "application/pdf"
    uri: str                 # path or object-store key
    bytes_len: int = 0
    summary: str = ""        # the short text stand-in the model DOES see

The model sees [artifact art_9fc2 image/png 84kB "bar chart of settlement failures by day; peak on the 14th"] and can reason about it, cite it, hand it to the user, or pass its id to another tool. The bytes stay on disk.

Summarise artifacts using known metadata, inspecting them before making visual claims. Non-text outputs such as charts and documents travel by reference rather than entering the context window.

16.8When not to give code execution#

You cannot provide real isolation. If the honest options are “same process” or “subprocess in your production container,” do not offer code execution. The capability is not worth the boundary you would be pretending to have.

The environment cannot mediate individual effects. If every consequential action requires its own authorisation and audit entry, expose checked host callbacks or keep it a first-class tool. Programmatic composition is compatible with per-effect auditing when the bridge enforces it.

The remaining latency budget is too small. Measure cold and warm startup, queueing and execution. Admit code to a voice turn only when the assembled path fits its deadline. Otherwise acknowledge first or perform bounded speculative reads (Chapter 18).

The work is one call. A single API request wrapped in a code block adds a sandbox, a traceback surface, and a permission gap in exchange for nothing. Chapter 13’s tool is better.

16.9Operating a sandbox fleet#

Pooling. Cold-starting a hardened container per call adds latency to every code action. A pool of warm, pre-initialised sandboxes reduces this to a checkout — at the cost of reuse, which means state hygiene between checkouts becomes a correctness property. Devin’s model is instructive here: an environment frozen into “a frozen, bootable image that every session starts from,” where “every session boots a fresh copy” and session changes do not persist back.³ A fresh base image helps, but attached writable volumes, external services and artifact staging still need independent lifecycle checks.

Multi-tenancy. Two tenants’ code must never share a sandbox, and “must never” needs to be enforced by the allocator rather than by convention. This is the strongest argument for a stronger isolation rung than a plain container.

Egress accounting. Every outbound request from the sandbox, logged with the run and tenant that caused it. This is both a security control and the thing that answers “why did our API bill triple.”

Cost attribution. Sandbox seconds are real money and they attribute to a task. Without this, code execution is an unmeasured line item, and Chapter 29’s per-task cost model has a hole in it.

Failure modes that are not code errors. Sandbox unavailable, pool exhausted, image pull failed. These are infrastructure failures and belong in Chapter 22’s taxonomy — not returned to the model as though its code were wrong, which produces an agent rewriting correct code in response to a capacity problem.

16.10Size the fleet from occupied time#

Sandbox capacity is easy to underestimate because teams count interpreter time and omit everything around it. A slot is occupied from checkout until teardown has proved it reusable. Image preparation, mounting the working directory, permission-checked callbacks, artifact upload and cleanup all consume capacity even when no Python instruction is running.

Start with two measured numbers over a peak window: admitted sandbox starts per second, (\lambda), and mean occupied seconds per sandbox, (W). Little’s Law relates long-run averages as (L = \lambda W) for a stable system, provided arrivals, occupancy and elapsed time use the same system boundary.⁴ A busiest five-minute window need not be stationary, so applying the relation there gives only a rough planning baseline. Replay the recorded event series to size for the transient peak.

Suppose an illustrative Reference Agent fleet admits 150 code actions during its busiest five-minute window. That is (150 / 300 = 0.50) starts per second. Its mean slot lifetime in that same window and across that same admitted population is 12.0 seconds: 0.8 seconds to check out and mount, 8.6 seconds to execute, 1.4 seconds to upload artifacts, and 1.2 seconds to verify teardown. The rough planning baseline is therefore (0.50 \times 12.0 = 6.0) occupied slots; the event replay, not this multiplication, determines whether the candidate pool survives the observed burst.

Running a pool at 100 per cent planned occupancy would turn every small variation into a queue. If the operating target is 70 per cent, the warm-pool baseline is (\lceil 6.0 / 0.70 \rceil = 9) slots. The spare three slots are headroom for ordinary variance, not proof that nine can absorb any burst.

The queue needs a bound and an expiry. With eighteen waiting positions and nine workers, a service slowdown can still make a request miss its product deadline. At admission time, estimate earliest start from the queue depth and observed completion rate. If a request has twelve seconds left and its earliest credible start is fourteen seconds away, refuse or fall back immediately. Waiting until the deadline and then returning sandbox unavailable spends the entire latency budget to reach an outcome you already knew.

The tail needs separate treatment. Do not multiply a p95 arrival rate by a p95 service time and present the result as a percentile guarantee; the two distributions may be correlated, and a quantile is not a mean. Instead, retain the event series, replay the actual high-arrival windows against candidate pool and queue sizes, and load-test the chosen envelope. Include cold starts, slow artifact uploads and teardown quarantines. The number you want is the fraction of eligible requests that start before their admission deadline, not a comforting average utilisation.

Cost follows from the same arithmetic. If an illustrative warm slot costs $0.04 per hour, nine permanently warm slots cost (9 \times 0.04 \times 720 = $259.20) in a 30-day month. Suppose twelve burst slots run for 12 per cent of the month: (12 \times 0.04 \times 720 \times 0.12 = $41.47), rounded to cents. The compute envelope is therefore about $300.67 per month before storage and egress. This is not a vendor price; it is deliberately simple arithmetic that makes the trade visible. Keeping twenty-one slots warm would cost $604.80, so the bounded autoscaling tier saves $304.13 under these assumptions.

Do not convert slots directly into hosts. A host also needs memory for the runtime, filesystem cache, the allocator and a safety reserve, and sandbox limits are promises that may be exercised simultaneously. If one illustrative host has 16 GiB usable for sandboxes and each sandbox may consume 2 GiB, eight fit arithmetically. Reserving 25 per cent for host variance leaves 12 GiB, so the defensible packing limit is six. Nine warm slots therefore need at least two hosts, and both must not share one failure domain. CPU overcommit may improve ordinary utilisation; memory overcommit turns a sandbox’s local OOM into a host-level correlated failure. Capacity planning must preserve the isolation story under load, not merely fit the average.

Image diversity changes the calculation again. A pool split across three runtime images does not behave like one fungible pool of nine: three idle slots for one image cannot start a job that requires another. Track eligible capacity per image and sandbox class, including any tenant or regional constraint. Consolidate images when their dependency sets allow it, but do not install packages at run time merely to make capacity interchangeable — that reopens network egress, reproducibility and supply-chain questions. If an image class receives too little traffic to justify a warm slot, expose its cold-start cost in admission estimates and product latency rather than hiding it inside execution time.

Capacity also has a tenant dimension. A global queue without per-tenant limits lets one large batch occupy every slot and turns an internal optimisation into a noisy-neighbour incident. Reserve no permanent tenant-specific pool unless the workload justifies it, but cap each tenant’s concurrent and queued work. When capacity is scarce, the allocator should choose from tenant queues under an explicit fairness rule rather than from one arrival-ordered list.

The final output of capacity planning is not “nine containers.” It is a contract: which work is eligible, how long it may wait, how many slots one tenant may hold, when burst capacity opens, which ceiling it cannot exceed, and what named result a caller receives when the envelope is full. Chapter 20’s budget principle applies exactly — a bound without a terminal behaviour is merely a delayed outage.

16.11Measure the boundary, not just the program#

A code action that returned exit status zero may still be a system failure. It may have waited thirty seconds for a slot, contacted a destination outside policy, produced an artifact the uploader lost, or left teardown incomplete. Instrument the sandbox as a state machine whose spans follow one identity from admission through destruction.

The minimum timing decomposition is queue_wait_ms, checkout_ms, execution_ms, artifact_ms and teardown_ms. Record both histograms and counts; a single end-to-end timer cannot tell you which owner should act. Attach tenant, run, sandbox class and image digest, but do not attach source code, environment variables, stdout or filenames as metric labels. Those are high-cardinality and may contain sensitive data. Keep them in access-controlled traces with retention rules (Chapter 23), not in the metric index.

Measure policy at the grain established earlier in the chapter. For the sandbox itself, record the admission verdict, requested resource profile and effective limits. For every permission-checked callback, record capability, verdict, latency and a hashed or classified destination — enough to prove that forty sensitive reads produced forty decisions without copying their payloads into telemetry. For network egress, record attempted and permitted destinations at the proxy. egress_attempts_total = 0 is not reassuring if the proxy never emitted a counter.

Five ratios reveal most boundary regressions:

• Admission success before deadline: eligible actions that start in time divided by eligible actions admitted. Segment by sandbox class and tenant tier.
• Forced termination rate: actions killed by wall-clock, CPU, memory or disk limit divided by started actions. Keep one counter per limit; “sandbox failed” is not diagnostic.
• Clean teardown rate: sandboxes that pass the empty-writable-layer check divided by sandboxes released. The denominator excludes quarantined slots only after they have been counted as teardown failures.
• Checked-effect coverage: sensitive effects with a matching prior policy record divided by sensitive effects observed independently at effect boundaries. Build the denominator from proxy requests, filesystem-mutation mediation, message or payment connectors, and other systems that perform the effect — never from the policy engine’s own emission stream. Reconcile those observations to policy logs by run and effect id. The target is exactly 100 per cent.
• Artifact promotion success: declared artifacts durably stored and readable before expiry divided by declared artifacts. A successful interpreter with a missing product is a failed task.

Use the ratios to drive action rather than dashboards alone:

Set service objectives on outcomes that callers experience and controls that must never weaken. An illustrative operating target might require 99.5 per cent of eligible code actions to start before their admission deadline over 30 days, while checked-effect coverage and cross-tenant canary isolation remain at 100 per cent. Those are not interchangeable error budgets. You may spend an availability budget by refusing work during a capacity event; you may not spend a security budget by bypassing the proxy to make the queue move. Likewise, clean teardown below 100 per cent should quarantine capacity immediately even if no customer-visible task has failed yet.

Alert on burn and on invariants differently. Page when a boundary invariant fails once: an effect lacks a policy record, a synthetic tenant reads the previous tenant’s canary, or a destroyed volume reappears. For latency and capacity, use sustained windows so a single cold start does not wake the on-call engineer. A fast-burn alert might compare the last hour’s missed-admission rate with the 30-day allowance, while a slower ticket captures a rising p95 teardown time before it consumes the budget.

The join is the control. Every sandbox allocation emits a run id and sandbox id; every callback, egress decision, artifact and teardown record carries both. A daily reconciliation groups by those identities and asks four yes-or-no questions: did an admitted run receive exactly one sandbox, did every observed sensitive effect receive a policy decision, did every declared artifact reach a terminal state, and did the writable layer reach destroyed or quarantined? Missing records are failures. Never treat absence as “nothing happened.”

Finally, maintain two synthetic probes. The first runs known-good code and establishes availability, startup time and artifact delivery without depending on model behaviour. The second is adversarial: it attempts a forbidden network request, reads outside /work, consumes each resource limit in turn, and leaves a child process and file behind. Success means every attempt is denied or contained, attributed to the correct limit, and followed by a clean or quarantined teardown. The probe verifies the boundary that matters; model evals verify whether Reference Agent uses the capability well. Keep boundary failures distinct from model-use failures.