Executing on Multimodal Content

In this illustrative scenario, a tenant emailed Reference Agent a photograph of a spreadsheet on a laptop screen. The image was rotated, with glare across four rows. A handwritten margin note said, “these two don’t match — Priya.”

Every part of that is a modality problem. The spreadsheet is structured data rendered as pixels. The glare destroys the OCR. The handwriting is the actual request. The rotation breaks any layout assumption. And the whole thing arrived through an email channel that also carries PDFs, screenshots, scanned invoices and the occasional video of someone narrating a bug.

Verify model and API support for each modality. Then decide what to process before inference: convert to text, retain pixels, or use both.

17.1Four things people mean by multimodal#

Ingestion: understanding non-text inputs such as documents, images, audio and video.

Grounded perception — the agent must not merely understand an image but locate things in it, in coordinates, in order to act. Clicking a button, reading a specific cell, pointing at a region. A different and much harder problem.

Production: generating charts, decks, documents or diagrams, often through code execution (Chapter 16).

Real-time streams: processing continuous audio or video under a latency budget (Chapter 18).

The reference Reference Agent orchestration and trace are textual. Multimodal models can process images or audio natively. The teaching harness’s textual representation does not describe every model’s internal reasoning. It demonstrates context assembly, budgets, compaction, tracing and evaluation from Parts II through IV.

17.2The central decision: pixels or text#

Every piece of non-text content faces one fork, and the wrong choice is expensive in both directions.

The reasoning behind the fork. Text is cheap, searchable, citable and diffable. It costs a fraction of the tokens, it can be indexed and retrieved (Chapter 11), a claim can point at a line, and two versions can be compared. Pixels preserve what extraction destroys — layout, spatial relationships, visual encoding in charts, handwriting, stamps, redactions, damage. And crucially, pixels let the model see that something is wrong with the document, which no extraction pipeline reports.

The photograph that opened this chapter needs both: extraction for the spreadsheet numbers that survived, and the image for the handwritten annotation, the glare that explains the missing rows, and the rotation that tells you this is a phone picture rather than an export.

17.3Documents: the case that dominates#

Route documents by content type:

Native text, no OCR needed. A digitally-generated PDF has a text layer. Extract it with layout awareness — preserving reading order, table structure and headings — and you have the content at negligible cost. A native text layer still needs inspection: reading order, charts, signatures and hidden text can require page images.

Scanned, needs OCR. No text layer. Traditional OCR is fast, cheap and struggles with tables, multi-column layout and poor scans. Vision-model extraction can help with layout and degradation, but may hallucinate plausible text; compare it with OCR on your document classes. The pragmatic policy: OCR first, and escalate to vision when the OCR result fails a quality check — a confidence threshold, a garbled-character ratio, or simply “the extraction contains no numbers and the document is an invoice.”

Genuinely visual. Charts, diagrams, floor plans, screenshots. Extraction returns axis labels and loses the data. Keep the image.

Preserve structure in the extraction. A table extracted to prose is a table destroyed. Emit tables as Markdown or CSV, keep heading hierarchy, and retain reading order across columns. A model reading a well-structured extraction of a table can answer questions about it; one reading the same table flattened cannot.

Keep a page-level pointer on every chunk. invoice-4471.pdf#page=3 supports citation and lets the agent request the page image when text is insufficient.

Report unreadable content explicitly. For example: [4 rows unreadable, page 2, rows 14-17]. Otherwise the agent may reconcile eleven of fifteen rows and report a complete result.

17.4The clean total from a broken page#

Consider an illustrative failure from Reference Agent’s supplier-reconciliation path. A twenty-page statement contained 480 line items. Eighteen pages had a native text layer; pages eleven and twelve were scans inserted after someone signed the statement. The document service extracted the native pages, ran OCR on the scans, and returned one reassuring number: confidence=0.989.

Reference Agent found 476 line items. Their values summed to £1,151,820, while the printed statement total was £1,184,620. The £32,800 difference looked like an unrecorded credit. The agent drafted a ledger adjustment for that amount and sent it for approval with a tidy explanation: the statement total was authoritative, the extracted lines were internally consistent, and the difference reconciled the two.

Every individual statement in that explanation was defensible. The conclusion was wrong.

Four £8,200 rows at the bottom of page eleven had been clipped from the scan. The OCR engine did not misread them; it never received their pixels. Character confidence was high because the characters it did receive were clean. The service averaged confidence across roughly 39,000 recognised characters. Missing rows were absent from that denominator, so the score could not detect their omission. The arithmetic was exact:

The symptom was a plausible reconciliation adjustment. The first assumption was a reasoning error: perhaps the model had chosen the wrong sign or missed an existing credit. The actual mechanism sat two stages earlier. A page crop had incomplete coverage, the OCR score measured recognised characters rather than page completeness, and the ingestion envelope collapsed local evidence into a global scalar. The agent then treated absence from the extraction as evidence of absence from the statement.

The fix has three parts. First, measure coverage separately from recognition confidence. Confidence answers “how sure was the extractor about what it read?” Coverage answers “how much of the expected page did it read?” Neither substitutes for the other. A page can have 99.9 per cent character confidence and zero evidence about the bottom centimetre.

Second, preserve the evidence at region level. For each page, record the rendered bounds, detected content regions, unreadable regions and any crop applied before extraction. A compact record is enough:

{
    "page": 11,
    "rendered_bounds": [0, 0, 1650, 2200],
    "observed_bounds": [0, 0, 1650, 1984],
    "confidence": 0.997,
    "gaps": ["bottom 216px absent relative to expected page template"],
}

This is an illustrative production record, not a type implemented by the reference Reference Agent harness. Its purpose is to keep recognition quality and source completeness apart. The expected rectangle or template is external evidence: pixels alone may not reveal that a page was cropped or content was erased. Mark unknown coverage as unknown.

Third, make completeness an action precondition. Reference Agent may summarise a partially extracted statement, but it must not create a financial adjustment from it. Before a mutating tool call, the policy layer checks that every source region required by the calculation is either observed or explicitly waived by a person. If the precondition fails, the next action is request_rescan(page=11), not create_adjustment(amount=32800).

Test the gate with omissions, not only corrupt characters. A useful fixture contains a known page rectangle and a manifest of expected regions: header, twenty-four body rows and footer. Run one clean extraction, one blurred extraction, one source image clipped above the footer and one image whose bottom four rows are covered by a white rectangle. The blurred case should trigger another extraction method. The clipped and covered cases should request the missing source even when the visible characters remain perfect. Then run the real downstream calculation and assert that no mutating tool is proposed from any incomplete fixture.

Put that assertion beside the financial tool test rather than in an OCR benchmark. Only the action boundary knows whether a missing region matters to the proposed effect. A document can be incomplete overall and still support a safe action — for example, copying a clearly visible case number from page one. Conversely, a one-pixel strip can be decisive when it clips the approval signature. Completeness is relative to the claim and consequence, not only to the file.

This generalises beyond documents. A video sampler can miss the two seconds when an error appears. A transcript can omit an inaudible account number. A screenshot can crop out the selected environment. In each case, model confidence describes the supplied observation, not the missing world outside it. Completeness belongs to the ingestion contract and the action gate, not to the model’s self-assessment.

17.5Images: tokens, resolution, and what fits#

Images cost tokens, and the cost is a function of dimensions. The mechanics vary by model and detail setting: fixed budgets, resizing, patches and tiles are all used. Cost is not universally proportional to area, and an oversized image may be rejected rather than downscaled. Use the selected model’s documented rules and actual usage.¹

Resolution is a quality knob with a price. Downscaling a screenshot until the text is unreadable saves tokens and destroys the answer. Sending a 4000×3000 photograph of a document when 1500×1125 suffices wastes tokens. The right resolution is the lowest one at which the specific thing you need is legible, which means it depends on the question and not only on the image.

Crop before you send. If the question is about one table on one page, send that region. A crop is both cheaper and more accurate, because it removes distractors — the same argument Chapter 8 made about context in general, applied to pixels.

Many images add up fast. Twenty screenshots in a context is a large fraction of a window. A run that accumulates images needs a policy for dropping them, and images should be volatile, low-priority sections in the assembler’s terms (Chapter 8). Prefer keeping the extracted description of an old image over the image itself.

The most useful pattern here is the two-stage look: send a downscaled overview, let the model identify the region of interest, then send a high-resolution crop of that region. This can reduce cost while improving legibility, but it adds a selection step that can miss the relevant region. Compare total latency, tokens and extraction completeness against one full-resolution observation.

OpenAI’s vision guide, checked on 9 November 2026, documents model-specific detail and resizing rules. low need not use fewer tokens than high. For dense text or coordinate-sensitive tasks, evaluate original where supported. Check its model-specific size and rejection limits. Retain source dimensions, applied transforms and chosen detail level with each observation. For GUI actions, map coordinates back through those transforms before clicking.¹

17.6Budget the observation path#

“Images are expensive” is too vague to operate. The useful unit is not the attachment or even the page; it is visual area sent to the reasoning model for a specific question. Retaining a high-resolution page image in object storage costs no model tokens. Sending it does. That distinction lets you keep full fidelity without paying for it on every run.

Take an illustrative 120-page insurance claim pack. Classification finds 72 digitally generated pages, 36 scanned text pages and 12 genuinely visual pages containing damage photographs, diagrams and annotated forms. Suppose every rendered page is 1,600 × 2,200 pixels. An indiscriminate image path sends all 120 pages:

120 × 1,600 × 2,200 = 422,400,000 pixels observed.

Call that 422.4 megapixels of visual input. This is deliberately a provider-neutral proxy, not a token estimate: providers tile, resize and cap images differently. Pixel area still exposes the architectural difference because it tells you how much material you elected to present before those provider-specific transforms.

Now apply the hybrid path. The 72 native pages go directly to structured text. The 36 scans go through OCR and retain their page images behind view_page. Only the 12 genuinely visual pages receive a 400 × 550 overview. Reference Agent identifies five pages that need detail and requests two 600 × 800 crops from each:

The hybrid path presents 7.44 megapixels instead of 422.40. The reduction is:

1 − (7.44 ÷ 422.40) = 0.9824 ≈ 98.2 per cent.

That is not a claim that the bill falls by exactly 98.2 per cent, nor that every task preserves accuracy with those dimensions. It is the result of this illustrative route under its stated assumptions. The operational claim is narrower and stronger: routing avoids entire observations; compression only makes each chosen observation smaller. A team arguing about JPEG quality while still sending 120 irrelevant pages is optimising the wrong term.

The route needs three independent budgets.

Ingestion budget covers reusable work per source and pipeline version: parsing, OCR, thumbnails, layout detection and storage. You can amortise it across every future question about the document. Amortise reusable work over the relevant tasks; repeat it when the source, permissions or extraction pipeline changes.

Observation budget covers content actually placed before the model: extracted text, overviews and crops. This belongs to the run, and the context assembler must count it alongside messages and tool results (Chapter 8). An image reference costs effectively nothing until a tool resolves it into an observation.

Uncertainty budget covers escalation. Low coverage, low recognition confidence, conflicting extraction methods or a requested action can trigger another look. Reserve room for that second look. If the first pass consumes the entire window, the system has designed away its own ability to verify.

Do not turn the arithmetic into a fixed resolution policy. A 600 × 800 crop is generous for a large handwritten note and inadequate for eight-point text across a dense table. The harness should ask a legibility question after each observation: can the required field be read and tied to a source region? If not, increase resolution, tighten the crop or switch extraction method. The stopping condition is evidence sufficient for the task, not a configured pixel count.

Reserve the uncertainty budget in the arithmetic. Suppose two of the ten 600 × 800 crops are too small to read. Reference Agent requests those regions again at 900 × 1,200, adding 2 × 1.08 = 2.16 megapixels. The hybrid route now presents 9.60 megapixels. It still avoids 1 − (9.60 ÷ 422.40) = 97.7 per cent of the all-page area. One failed look does not erase the routing advantage, provided the first pass left room to recover.

Now try the opposite failure. If the 400 × 550 overviews are too small to locate any region confidently, all twelve pages fall through to full resolution. That adds 12 × 3.52 = 42.24 megapixels before a single useful crop. The route still sends less than the all-page strategy, but classification has stopped doing most of its work. Increase overview resolution enough to locate regions, not enough to read their detail. This is the two-stage look’s break-even decision: the first stage earns its cost only by preventing larger observations in the second.

Keep the route ledger needed to reproduce these sums: selected path, source dimensions, sent dimensions and crop coordinates. It tells you whether a change altered classification, overview size or escalation. The ledger serves the worked budget; it does not turn visual quality into one universal score. A damage assessment legitimately observes more area than an invoice lookup, and any cost comparison that ignores the task is false precision.

17.7Screens: grounding and coordinates#

When an agent operates a graphical interface, understanding is not enough. It must locate — produce coordinates precise enough to click.

This is a genuinely harder problem than description, and the reason is worth understanding: describing an image requires knowing what is in it, while grounding requires knowing where, in a coordinate space that varies with resolution, scaling and device pixel ratio. A model that can flawlessly describe a settings page can still click forty pixels below the button.

Four practices that make GUI operation viable.

Prefer structure to pixels whenever it exists. An accessibility tree, a DOM, or an API is dramatically more reliable than a screenshot. Reach for the screenshot when nothing structured is available — which, for legacy desktop software, is often. This ordering is not a compromise; a DOM query is exact where a coordinate estimate is probabilistic.

Give reference points, and normalise. Provide dimensions explicitly, and handle scaling in the harness rather than hoping the model accounts for it. Coordinate bugs from device pixel ratio are common and produce systematic offsets that look like model incompetence.

Verify after acting. Click, screenshot, confirm the expected change occurred. This closes the loop with an observation rather than an assumption, and it is the difference between an agent that recovers from a mis-click and one that continues confidently in a state it did not intend. This is Chapter 6’s verification principle applied at the interaction level.

Bound the blast radius of a mis-click. A misplaced click in a settings dialog can do real damage, and unlike a tool call it did not pass through a policy check the harness could evaluate. Screen automation belongs at a low autonomy level (Chapter 15) with human approval for anything irreversible.

The honest caveat: computer-use benchmarks on realistic desktop and web tasks have improved substantially and remain well short of human reliability on long multi-step flows. Design accordingly — short flows, frequent verification, and a human in the loop for consequence.

17.8Audio and video as inputs#

For recorded audio and video:

Audio — a voicemail, a recorded meeting, a support call. Transcription is the standard path, and the important design point is that the transcript is not the audio: it loses speaker identity unless diarised, loses emphasis and tone, and — most consequentially for an agent — carries transcription errors that propagate silently into tool arguments. An account number misheard by one digit becomes a confidently wrong lookup. Anything the agent will act on should be confirmed rather than assumed.

Video — a screen recording of a bug, a walkthrough. The workable approach is decomposition: sample frames at a rate matched to the content, extract audio and transcribe it, and reason over the aligned pair. Sampling rate is the whole decision — one frame per second is a starting hypothesis for slowly changing screens, but can miss short-lived errors. Use scene-change or event-triggered sampling and report unsampled intervals. Sending every frame is prohibitive and unnecessary.

The general principle for both: decompose into text plus selected frames, and keep the pointer back to the timestamp. “At 04:12 the error appears” is citable; a claim about a video is not.

17.9Producing multimodal output#

Agents produce charts, spreadsheets, documents, diagrams and decks.

Almost all of it is code execution (Chapter 16). The agent writes code that produces the artefact, and the artefact is returned by reference:

[artifact art_9fc2 image/png 84kB "bar chart of settlement failures by day,
 peak on the 14th at 41 failures"]

Generate a description at creation time. The one-line summary is what the model reasons over subsequently, and it is what makes the artefact usable in an answer. An artefact the agent cannot describe is one it cannot cite.

Let the agent look at what it made. Render the chart, read it back as an image, and check it says what was intended. Axis labels overlapping, a legend covering the data, an empty plot from a filtering bug — all invisible in the code and obvious in the render. This is the evaluator-optimiser pattern (Chapter 7) with a verifiable signal, and it works.

Keep the artefact out of context and the reference in. Never inline bytes. Chapter 13’s Artifact type exists for this.

17.10Artefact IDs are handles, not authority#

In code/reagent/types.py, tool results carry Artifact values and refer to them by identifier in textual content:

@dataclass
class Artifact:
    """A non-text product of a tool call. Kept out of context by reference."""

    id: str
    media_type: str          # "image/png", "text/csv", "application/pdf", ...
    uri: str                 # path or object-store key
    bytes_len: int = 0
    summary: str = ""        # short text stand-in the model *does* see

Each field closes a different boundary. id is the stable conversational handle. It designates an artefact; it never authorises access to one. The model can say “inspect art_9fc2” without receiving a storage credential, and possession of that string grants nothing. media_type selects a renderer or parser before any bytes are loaded. uri tells the harness where the object lives, but should not be exposed to the model or end user as authority to fetch it. bytes_len supports admission control before a large object crosses a boundary. summary is the text substitute that lets the reasoning loop remember what it made.

The separation prevents a surprisingly large amplification. Suppose a chart is a 12 MiB PNG: 12 × 1,024 × 1,024 = 12,582,912 bytes. Encoding binary as base64 requires 4 × ceil(bytes / 3) characters, which produces 16,777,216 characters here before JSON quoting, message metadata or tokenisation. Putting that base64 in an ordinary text result would waste context. A provider’s native image field can accept base64 as transport encoding and charge image tokens after decoding; wire bytes and text tokens are different quantities.¹ Returning the 100-character reference and summary keeps the trajectory readable; the model pays for pixels only if it later calls a viewing tool.

The summary is not the artefact. Treating it as one produces a subtler failure than context bloat. If the summary says “bar chart of settlement failures, peak 41” and the underlying file is later overwritten with a corrected chart whose peak is 37, the trajectory now contains a false claim bound to a valid identifier. Artefacts should therefore be immutable for the lifetime of any trace that references them. A correction creates a new identifier and a new summary; it does not mutate the old object.

The reference type demonstrates the mechanism, not a production artefact service. It does not implement tenant-scoped storage, content hashes, retention, malware scanning, signed retrieval, access audit or immutability. Those are production extensions at the boundary around Artifact, and their absence from this teaching harness is deliberate. Do not describe the dataclass as if it enforces them.

A production envelope needs at least these obligations:

The reference is therefore a designator, not a bearer capability. The capability-bearing operation is view_artifact(id) evaluated with the current run identity, tenant and policy. The same id presented by a run without a grant must resolve to nothing. Once policy permits the operation, fetching and rendering invoke parsers with their own attack surface. The tool must validate tenant ownership, allow only supported media types, enforce byte and pixel limits, render in an isolated process, and return a new untrusted observation. A model-generated chart is still untrusted input when the model reads it back.

The resolver should therefore be a narrow service, not a convenience function that opens artifact.uri. One production sequence looks like this:

1. Accept the opaque artefact id and the current run identity — never a caller-supplied storage path.
2. Resolve the id inside the current tenant and verify that the run is allowed to read that object.
3. Read metadata first. Reject unsupported media types and objects above the byte, page, pixel or duration limit before decoding them.
4. Fetch the immutable bytes, recompute the content hash and compare it with the stored version.
5. Decode or render in an isolated worker with network access denied and a hard deadline.
6. Store the derived preview as a new artefact, then return its id plus an untrusted textual observation to the agent loop.

This sequence is an illustrative production extension; none of these resolver controls is implemented by the reference dataclass. The distinction matters because id: str and uri: str can look like abstractions with more authority than they have. One designates the record and the other locates its bytes. The service combines the designator with current identity and policy, then owns authorisation, integrity and safe decoding.

Why create a second artefact for a preview? Derivation is evidence. A PDF page rasterised with one renderer is not interchangeable with the same page rasterised after a library upgrade, and a downscaled image is not the source. The child artefact can carry derived_from, crop coordinates, renderer version and content hash without mutating the parent. That gives the two-stage look a reproducible chain: source document → page render → overview → detail crop → model observation.

Revocation and deletion need the same indirection. If a tenant deletes the source, every unresolved handle should stop resolving immediately even if a cached preview still exists. The trace may retain metadata needed for audit — identifier, media type, hash and deletion event — without retaining the content. A bare object-store URL cannot express that policy; an opaque id resolved at use time can.

The last point matters for evaluator-optimiser loops. Reference Agent produces a document, renders it, inspects the render and revises it. The second pass must be able to say exactly which version it saw. Bind the observation to (artifact_id, content_hash, renderer_version). If a later trace says the legend overlapped the data, you can reproduce the pixels that supported that judgement rather than opening whatever currently sits at a mutable path.

Keep references cheap and resolution expensive. That rule works in both senses of resolution: looking at pixel detail and resolving an identifier into bytes. The model may carry thousands of cheap handles across a long task; the harness turns only the few relevant ones into observations after checking the current run identity and policy, with the decision in the trace.

17.11Visual and audio content is untrusted input#

Non-text content adds two prompt-injection risks:

Instructions can hide in images. Small or low-contrast text in pixels may be noticed differently by a model and a human. PDF hidden text or metadata can also enter through extraction even when it is not visible in a raster; do not conflate those channels.

Extraction pipelines are an injection surface. OCR output flows into the context as text with no marker distinguishing it from any other text. If your pipeline extracts and appends, you have a channel from “anyone who can email you a PDF” to “text in your agent’s context.”

Use three layered defences:

Label extracted content as untrusted, always. Chapter 8’s trust="untrusted" wrapper applies to OCR output, transcripts, and image descriptions exactly as it does to retrieved documents. An extraction pipeline that emits bare text has erased the provenance the trust boundary depends on.

Do not let non-text content authorise anything. A screenshot showing “approved by Priya” is not an approval. An email image containing “the manager said to proceed” is not a grant. Approvals come through the approval channel (Chapter 19), and this needs to be an enforced property rather than a hoped-for one — because a plausible visual authorisation is exactly the kind of thing a model finds persuasive.

Keep egress denied. The structural defence that does not depend on detection. Network denial blocks direct egress; answers, shared artifacts and allowed connectors still need audience and data checks.

17.12Choosing, concretely#

Classify the content type at ingestion, not at reasoning time. Digital PDF, scanned PDF, photo of a document, screenshot, chart, spreadsheet, audio, video. Each has a default path. A pipeline that treats everything identically is over-paying on the easy cases and under-serving the hard ones.

Default to text with images retained. Extract, keep page images available, and let the agent request an image when the text is insufficient. This gives cheap search and citation with an escape hatch to full fidelity.

Make the escape hatch a tool. view_page(document, page) returning an image is how the agent chooses to look. This permits query-specific inspection. A validated fixed pipeline is also appropriate when the required regions are predictable.

Measure extraction quality separately. Build a set of documents with known correct extractions and measure them independently of the agent. When the agent answers wrongly about a document, the first question is whether the content ever arrived — and without a separate measurement you cannot tell.

@dataclass
class IngestedDocument:
    doc_id: str
    media_type: str
    pages: int
    text: str                      # extracted, structure-preserving
    extraction_method: str         # "native" | "ocr" | "vision" | "transcript"
    confidence: float | None       # calibrated method score, if available
    gaps: list[str]                # ["page 2 rows 14-17 unreadable: glare"]
    page_images: dict[int, str]    # page -> artifact id, fetched on demand
    trust: str = "untrusted"       # ALWAYS. No exceptions.