Voice & Real-Time Agents

In this illustrative scenario, Reference Agent’s voice channel worked on the first try and was unusable for six weeks.

Every component functioned. Speech came in, was transcribed accurately, reached the same reasoning core that served the chat channel, produced a good answer, and came back as clear synthesised speech. The problem was that it took 3.4 seconds, and 3.4 seconds of silence on a phone call is not a delay — it is a failure signal. Callers said “hello?” into the gap. They repeated themselves, which produced a second transcription that arrived while the first was still being answered. They hung up. One left a review describing the system as broken. Nothing was broken. It was slow, and on a voice channel slow is broken.

Voice deadlines constrain context assembly, model selection, tools and interruption handling.

18.1The deadline, and where it comes from#

Human conversation has a rhythm, and it is faster than software people expect. Across languages and cultures, the gap between one speaker finishing and the next beginning clusters tightly around a fifth of a second — a robust cross-linguistic finding, with modest cultural variation around a common central tendency.¹ People are not tolerant of much more than that in conversation, and the intolerance is not a preference. Silence in a conversation means something: confusion, disagreement, absence.

That motivates a first-audio objective, not a universal human threshold. The following are illustrative product expectations to test with your callers:

Use first audible response inside 500 milliseconds as an illustrative objective. Validate it with callers, then allocate its budget.

18.2Where the milliseconds go#

The budget is a waterfall, and every stage is someone’s design decision.

Endpointing dominates, and it is a tunable you own. Deciding that the person has finished speaking is the single largest line in the budget, and it is not a vendor’s number — it is your policy. A fixed 800-millisecond silence timer is safe and ruinous; an aggressive 150 milliseconds is fast and interrupts people mid-thought. This is the highest-leverage parameter in the system and the next section is about it.

Stream where safe. Partial transcripts can support acknowledgement and bounded prefetch; consequential actions require validated intent. Generate and synthesise incrementally where pre-release checks allow it. Waiting for a complete answer before starting synthesis is the second-largest self-inflicted cost, and it is the one most often shipped.

The model’s contribution is time-to-first-token, not total generation. Once audio is playing, generation continues under cover of speech. A long answer still costs generation, audio output and caller attention; a slow start adds silence. This single reframing changes which model parameters matter.

Everything has a p99. A 500-millisecond median with a 3-second tail is a system that feels broken to about one turn in a hundred when that tail is the p99, and that caller is the one who tells people. Voice work is tail-latency work.

18.3Admit work to the fast path#

A latency budget is useful only when it can reject work. If every component is told to “be fast,” every component will report an acceptable median and the assembled turn will still miss its deadline. Give the turn controller one budget, make each stage declare what it needs, and refuse to put work on the critical path when the remaining budget cannot cover it.

Take an illustrative Reference Agent balance enquiry. Use one boundary throughout: media-capture-to-audible latency. The clock starts when the caller-facing channel captures the caller’s final speech frame, before inbound transport and jitter buffering, and ends when caller-facing playout renders Reference Agent’s first audio frame. The deployment has profiled each stage at the percentile it intends to operate, then assigned the following engineering budget:

The arithmetic is deliberately plain: 40 + 180 + 35 + 15 + 95 + 20 + 55 + 30 = 470 ms. That leaves 30 milliseconds of contingency inside a 500-millisecond objective. The figures are illustrative, not vendor measurements; the discipline is the point. A budget with no contingency is a promise that one scheduler pause can break.

Now the caller asks, “What happened to invoice 4382?” A billing lookup has a measured 480-millisecond service time for this tenant. Waiting for the final transcript and then inserting that lookup into the serial path would produce 470 + 480 = 950 ms to the first factual audio, even before a tail event. The request therefore fails admission to the first-audio path. Reference Agent says, “I’m checking invoice 4382,” inside the 470-millisecond conversational budget, then delivers the result when it arrives. The tool did not become faster. The architecture stopped pretending it belonged to the same deadline.

Prefetch changes the calculation but not the rule. Suppose the partial transcript makes the invoice intent stable 50 milliseconds before the caller-facing channel captures the caller’s final speech frame and the lookup still takes 480 milliseconds. When that final frame is captured, 480 - 50 = 430 ms of lookup time remains — 40 milliseconds less than the 470-millisecond path to audible acknowledgement. The raw lookup result can be ready before acknowledgement playback. A factual continuation still needs model processing, output checks and synthesis; this overlap does not prove those finish inside 470 ms. If the partial transcript changes from invoice 4382 to invoice 4383, the prefetched result is discarded. Speculation may spend compute; it may not spend authority. Only read-only, bounded, tenant-scoped operations belong in this lane. Never speculatively dispatch a credit, cancellation or message because a partial transcript looked likely.

That produces three explicit lanes rather than one hopeful pipeline:

The admission decision belongs before dispatch. Once a slow operation is already on the critical path, the only remaining choices are silence, interruption or an improvised filler. All three are worse than deciding upfront that the caller should hear an acknowledgement.

11Do not tune a latency budget from the sum of vendor medians. Components share queues, network paths and load spikes, so their tails correlate. Profile the assembled path and keep explicit contingency even when each stage looks safe alone.

Admission also has to operate at the deployment boundary, not only inside one turn. Imagine an illustrative cluster that can start 30 model streams per second while a call campaign produces 42 endpoints per second. The backlog grows by 42 - 30 = 12 turns every second. After five seconds, 60 turns are queued; even if arrivals suddenly stop, a 30-start-per-second service needs 60 / 30 = 2 seconds merely to drain them. Every component can still be inside its isolated budget while queueing has already destroyed the conversation.

Do not hide this overload in an unbounded work queue. Reserve capacity for acknowledgements, reject optional prefetch first, and route new consequential requests to a spoken degraded path before the queue consumes the whole deadline. The caller can accept, “I’m having trouble reaching billing; I can connect you,” because it is a timely and truthful response. They cannot distinguish a saturated queue from a dead system. Record the admission outcome on the turn so the trace says why Reference Agent degraded rather than making the model infer a reason.

This is where a voice controller differs from a generic asynchronous job runner. A job runner optimises eventual throughput and can wait for capacity. A voice controller protects an expiring opportunity to respond. Once the first-audio deadline passes, completing the queued work does not recover the interaction; it often collides with the caller’s repair turn and makes state worse. Shed work before you shed conversational coherence.

18.4Turn-taking is the hard problem#

Knowing when someone has stopped talking is genuinely difficult, and getting it wrong produces the two worst voice failures.

Too eager and the agent interrupts. The caller pauses to think — “the account is… let me find it” — and the agent starts answering a half-finished sentence. This is worse than being slow, because it is rude and because it destroys the caller’s utterance.

Too patient and the agent is late on every turn, paying the full silence timeout even on utterances that were obviously complete.

Three mechanisms, in increasing sophistication, and they layer rather than compete.

Voice activity detection answers “is there speech energy” in milliseconds. Fast, cheap, and semantically blind: it cannot distinguish a thinking pause from a finished sentence. Use it as the floor and as the fastest signal when minimal latency is the priority.

Semantic endpointing asks whether what has been said so far is complete. “My account number is four four seven” is prosodically paused and semantically unfinished; “what’s my balance” is complete the instant it ends. A model that considers both meaning and acoustic properties on top of VAD can cut hundreds of milliseconds off complete utterances without truncating incomplete ones — and this is where the largest single latency win in a voice agent usually lives.²

Adaptive delay. The right silence threshold varies by speaker, by moment and by content. A caller reading out a long identifier pauses differently from one asking a question. Adapting the delay from observed pause statistics within the session is a real improvement over a fixed value, and frameworks now expose it directly.²

A pattern worth adopting regardless of mechanism: let content set the threshold. After a question the agent asked, expect a short answer and endpoint aggressively. After “read me the account number,” expect digits with pauses and endpoint patiently. The agent knows what it just asked; use it.

18.5Barge-in#

A caller must be able to interrupt, and handling it correctly is more than stopping playback.

Stop playback immediately. Milliseconds, not the end of the sentence.

Record the playback boundary and remove unplayed output. The agent generated four sentences and the caller heard one and a half. If the transcript records four, the agent believes it said things the caller never heard, and every subsequent turn reasons from a conversation that did not occur. This produces the uniquely maddening failure where the agent says “as I mentioned” about something it never got to say.

Cancel in-flight generation where supported. This reduces further work; already generated or in-flight tokens can still be billed.

Treat the interruption as input. It is the next turn, not noise.

And the necessary refinement: not all speech during playback is an interruption. “Mm-hm,” “right,” “okay” are backchannels — acknowledgements that mean keep going. An agent that stops for every one is exhausting to talk to. Distinguishing genuine interruptions from conversational backchanneling is now a first-class framework feature, along with resuming after a false interruption where speech was detected but no transcription materialised.² Both behaviours are worth having.

18.6Cascaded or speech-to-speech#

Two architectures, and the choice is consequential.

Cascaded — speech to text, text to a language model, text to speech. Three components, each replaceable, each observable. The transcript is text, which means everything in Parts II through IV applies unchanged: context assembly, tool calls, guardrails, tracing, evaluation.

Speech-to-speech — a model that takes audio and emits audio, with no textual intermediate on the critical path. Fewer hops, lower floor on latency, and it preserves prosody, emotion and interruption dynamics that a transcript discards.

For a general-purpose work agent — one that calls tools, touches money, needs an audit trail, and must pass Chapter 30’s governance review — a cascade is a useful baseline when deterministic pre-speech text inspection is required. The textual intermediate provides a clear control point. Guardrails, provenance labelling, trace attributes, eval scoring and the audit log all operate on text. An architecture with no text on the critical path has to reproduce all of them.

A practical hybrid to evaluate: cascaded for the reasoning and action path, with speech-to-speech or a cheap local model handling the conversational surface — acknowledgements, holding phrases, clarifying questions — so the fast path is fast and the consequential path is inspectable.

Current Realtime APIs support tool calling and transcript events, so speech-to-speech does not imply the absence of text or action control. OpenAI’s WebSocket interruption guidance requires the client to stop playback and send conversation.item.truncate at the played audio position. Its guide explicitly warns that this does not yield a precisely word-aligned truncated transcript. WebRTC/SIP manage output buffering differently; test your selected transport rather than copying one event sequence everywhere.³

18.7Tool calls under a deadline#

Tool latency can exceed the first-audio budget. A serial 800-millisecond lookup cannot fit a 500-millisecond objective. Measure the selected dependency and move slower work behind an acknowledgement.

Speak first, then act. “Let me look that up” buys 1,500 milliseconds legitimately, because the caller is now waiting for a stated reason rather than for silence. This is not a trick; it is what a person does. Schedule the acknowledgement independently of the lookup so a slow dependency cannot delay it. Bounded read dispatch may occur concurrently; consequential actions still wait for their required gate.

Prefetch on intent. As soon as the intent is recognisable — often mid-utterance, from a partial transcript — start the likely lookup. By the time the caller finishes, the account is loaded. Speculative work is cheap; silence is expensive.

Pre-warm the context. Chapter 8’s assembly costs tokens and milliseconds. For a known caller, assemble the stable prefix at call setup, not at first utterance. The prompt cache should be warm before anyone speaks.

Two-tier tools. A small set of fast, cached, read-only lookups on the critical path, and everything else moved off it — either behind a spoken acknowledgement or deferred to after the call.

Degrade rather than stall. If a lookup exceeds its deadline, say so and offer an alternative. “I’m not getting a response from billing — I can take a message or connect you to someone.” A caller told about a problem is having a conversation; a caller in silence is being ignored.

18.8The stale completion incident#

Making a turn interruptible creates a second obligation: every result must still belong to the turn that requested it. Cancellation is a request to stop work. It does not transfer ownership, erase a remote side effect or make a late callback safe.

An illustrative Reference Agent incident began with a caller asking about invoice 4832. Reference Agent said, “I’m looking up invoice 4832,” and dispatched a read-only billing lookup for turn 42. During the acknowledgement, the caller interrupted: “Sorry, 4382.” The voice controller stopped playback, truncated the heard transcript and opened turn 43. It also cancelled the local task waiting for turn 42’s result.

Turn 43’s lookup returned after 310 milliseconds with invoice 4382. Turn 42’s remote request, which had already crossed the process boundary, returned 70 milliseconds later with invoice 4832. The callback wrote its result into a shared current_result slot. The next synthesis step read that slot and told the caller about the wrong invoice.

Everyone first blamed transcription. Both transcripts were correct. Then they blamed the billing service. Both service responses were correct. The actual defect was result ownership: the voice layer treated “latest callback” as equivalent to “result for the active turn.” Barge-in had cancelled a waiter, not the work already running elsewhere, and the response sink had no proof that the late result still belonged to the conversation state it was about to speak.

The repair has four parts.

Give every turn an immutable identity. Conversation id is not enough. Every transcript revision, model stream, tool request, tool result, guardrail decision and audio chunk carries conversation_id, turn_id and a generation number. A callback that arrives without those fields is not convenient legacy data; it is unowned data and cannot be spoken.

Make the output sink enforce active ownership. Before text reaches synthesis, and again before audio reaches playback, compare its turn identity with the active turn. Turn 42 can finish after turn 43 and remain a perfectly valid historical result. It is simply not eligible to affect turn 43. This check belongs in deterministic code, not in a model prompt asking the model to notice that a result looks old.

Separate cancellation from invalidation. Cancellation asks local work to stop. Invalidation advances the active generation so that old work can no longer publish. You need both. A provider may ignore cancellation; a network packet may already be in flight; a thread may finish between the cancellation check and the callback. Invalidation closes all three races because publication depends on current identity, not on the success of recall.

Treat effects differently from reads. A stale read can be recorded and discarded. A stale write may already have changed the world. Consequential operations therefore never launch speculatively, and interruption after dispatch does not silently retry them. Use Chapter 5’s idempotency key, reconcile the remote state, and tell the caller what is known. “I stopped speaking” must never be interpreted as “the credit did not happen.”

The event sequence should remain visible even when the stale result is suppressed:

Do not delete the late event. It proves that cancellation did not reach the dependency, exposes wasted work and lets the outer loop measure how often this race occurs. Suppression is a presentation decision; the event log remains an honest record.

The text-only reference harness has no voice transport, turn-generation gate or audio response sink. Identity propagation across these components is a production extension.

Test the gate by controlling completion order, not by hoping a real dependency happens to race. Start turns 42 and 43, complete 43 first, then complete 42, and assert that only 43 reaches synthesis. Repeat with turn 42’s cancellation ignored. Repeat with the old audio chunk arriving after turn 43 has already started playback. Finally, crash after invalidation and restore from the event log: the active generation must remain 43 after recovery. If generation exists only in process memory, Chapter 5’s resumption machinery can resurrect output the caller already rejected.

These tests need a fake response sink and a controllable future, not a live speech provider. Their invariant is exact: for every audio frame published at time t, its generation equals the conversation’s active generation at t. Add a second invariant for effects: a result from an invalidated generation may report an already-committed remote effect, but it may never cause an automatic replay. This separates two questions that interruption tends to blur — whether output is still relevant, and whether the outside world already changed.

18.9When a transcription error becomes an action#

Consider: the caller says “credit account four four seven one.” The transcript reads “credit account four four seven nine.” Every downstream component behaves correctly. The model extracts the account, calls issue_credit(account="acme-4479", amount=...), the policy permits it, and the wrong customer is credited. Nothing failed except a single phoneme, and no guardrail in Parts IV or VI was designed to catch it — because the argument was well-formed, the tool was permitted, and the value was plausible.

Constrain the recognition. If the value must be an account in your system, bias the recogniser toward the valid set or validate against it. A transcription that cannot be a real account should never become a tool argument.

Use check digits and checksums. Identifiers designed for spoken use should carry a self-check. If yours do not, this is a design change worth requesting.

Read back before acting, for anything consequential. “That’s account four four seven one, ending seven one — crediting fifty dollars. Confirm?” This is not friction; it is what a competent human agent does with a spoken number, for exactly this reason.

Propagate available confidence. Some ASR APIs expose confidences or log probabilities, others do not. Preserve their meaning and calibration; an absent score is unknown. That signal should reach the policy engine, not be discarded at the transcript boundary. A low-confidence identifier is a reason to confirm, and the harness cannot know that if the confidence was thrown away three components ago.

Set confirmation rules for speech-derived fields. Read back consequential identifiers and amounts, bind them to the authenticated account, and set autonomy from the measured channel error rate and effect risk.

The transcript is evidence, not fact. Corroborate speech-derived identifiers before passing them to irreversible actions.

18.10Voice-specific guardrails#

Voice adds four guardrail requirements:

Guardrails must run inside the latency budget. A 200-millisecond content check on the output path is 40% of the budget. Required disclosure and action checks must finish before the relevant audio or effect is released. Expensive monitoring can run during playback and cut later audio, but it cannot prevent disclosure of words already heard.

Spoken output cannot be un-said. Text can be redacted before display; audio is heard as it is generated. This argues for checking before synthesis rather than after, which means the check sits between model and TTS in the waterfall — and must therefore be fast.

Voice authentication is weaker than it looks. Synthesis quality has made voice a poor authentication factor. A caller who sounds like the account holder is not authenticated, and any policy that treats voice as identity should be revisited. Use a real factor.

Consent and recording are jurisdictional. Whether a call may be recorded, whether the caller must be told they are speaking to a system, and how long audio may be retained are legal questions with different answers in different places, and they are Chapter 30’s subject. Note here only that they constrain the architecture — a system with no ability to disable recording per jurisdiction is one that cannot be deployed in some of them.

18.11Measure the conversation, not the model#

A correct answer and a fast model do not prove that the conversation worked. Voice quality lives at boundaries: when the caller actually stopped, when audio actually arrived, what they actually heard before interruption, and whether an asynchronous result still owned the right turn. Instrument those boundaries directly.

Start with media-capture-to-audible latency, using the same boundary as the admission budget. Timestamp the caller’s final speech frame when the caller-facing channel captures it, before inbound transport and jitter buffering. End at playout on the caller-facing connection, not when the synthesiser emits bytes inside your data centre. Starting at the endpoint decision erases the very delay you most need to optimise. Keep p50 for typical feel, p95 for release comparison and p99 for the incidents callers remember. Break the same duration into the waterfall stages from Figure 18.1 so a regression has an owner.

Then measure the errors that make low latency dishonest.

Premature endpoint rate is the share of turns where Reference Agent begins responding before the caller has finished. A useful companion is repair rate: how often the caller’s next utterance corrects or completes the one just cut off. A lower latency accompanied by more repairs is not an improvement; it moved delay into conversational rework.

Barge-in stop latency runs from caller speech detection to the last audio frame played. Measure transcript overhang alongside it — words stored as heard that were never played. A playback stop can be fast while the transcript remains fictional.

False-interruption rate counts backchannels or noise that stop valid playback.

Resume success asks whether Reference Agent continued cleanly after a false interruption rather than restarting the answer or abandoning it. These two values keep an aggressive detector from winning a latency benchmark by becoming impossible to listen to.

Silent-busy exposure counts caller-facing time when work is running and no audio is playing. Record both total seconds and incident count. An average hides the defect: ninety-six silent seconds across 20,000 turns is only 96 / 20,000 = 4.8 ms per turn, but if those seconds came from 48 incidents, each affected caller endured 96 / 48 = 2 seconds of unexplained silence.

Stale-publication rejection rate is invalidated asynchronous operations that later produce a rejected result / all asynchronous operations invalidated before completion. Count operations, not streamed chunks, and publish the numerator and denominator beside the percentage. The rate is an investigation signal, not a diagnosis: it can rise because callers interrupt more often, speculative volume grows, a dependency slows down, or a provider stops honouring cancellation. Segment by operation, provider, cancellation support, invalidation reason, dependency-latency bucket and deployment version before assigning cause. A zero value is good only if tests prove the gate is exercised; otherwise it may mean ownership was never checked.

Confirmation coverage is the share of consequential actions whose speech-derived identifiers and amounts were read back and explicitly confirmed. Report the missing cases, not just the percentage. A 99% rate sounds healthy until the omitted 1% contains twelve irreversible actions.

Consider this illustrative weekly Reference Agent review over 20,000 eligible turns:

The error-budget arithmetic makes a release decision possible. At 20,000 turns, a 99% objective permits 20,000 × 0.01 = 200 turns over one second. The observed 340 consumes 340 / 200 = 1.7, or 170%, of the weekly budget. Do not average that miss away with the 95.5% fast-path result. Freeze latency-risking feature changes, find which stage owns the excess 140 misses, and retest the assembled path.

Use the metrics as a diagnostic map rather than as independent scores:

Segment every metric by endpoint policy, caller locale, network class, tenant, tool path and deployment version. A global p99 can improve while one language or one mobile carrier degrades sharply. Keep the segments large enough to interpret and the raw trace available for the incident behind the aggregate.

Finally, replay recorded events, not retained audio by default. Synthetic speech and consented test calls can exercise endpointing, barge-in and stale completions without turning production recordings into an indefinite evaluation corpus. Retention policy comes from Chapter 30; observability does not override it.

Optimise voice as a constrained pair: latency and conversational error. Every latency change must be read beside premature endpoints, repair turns, false interruptions and transcript overhang. A faster number that makes the caller repeat themselves is a regression.

Endpoint quality needs sampled ground truth because production telemetry cannot fully label its own mistake. On a consented evaluation set, have a reviewer mark the final speech frame, whether a pause was turn-final, whether speech during playback was a backchannel or an interruption, and exactly which generated words were audible. Compare those labels with detector events and stored transcripts. Live dashboards then show broad movement; the labelled sample tells you whether that movement is real. Do not use the endpoint detector’s own decision as its correctness label — that would define every premature cut as a valid endpoint by construction.

Sample the hard slices deliberately: digit strings, names, accented speech, packet loss, long thinking pauses and callers who speak over acknowledgements. A random sample dominated by clean short questions will certify the easy path and miss the cases that reach tools. Version the labelled set and run it before changing endpoint thresholds, ASR models, synthesis buffering or barge-in logic. One parameter change can improve the overall p50 while increasing truncation for exactly the callers whose identifiers require the most care.

Audio has its own cost ledger#

Track input audio, output audio, text input/output, cache reuse, session duration and tool charges separately. Record transcription charges separately when enabled. Use duration or token usage according to the selected transcription model. Transcribed words are not an audio-token estimate. Barge-in can reduce future output, but cannot refund generated audio or a dispatched external operation. Replay realistic calls and interruptions against the provider’s current usage fields and rates. First-audio latency and dollar cost need separate budgets.⁴

18.12Controls across the voice stack#

Preserve durable state, budgets and resumption when work outlives the call. Pre-assemble context at call setup and measure cache hits against latency. Compaction must retain confirmed fields and the playback boundary. Acknowledge tools that exceed the fast-path budget, authenticate spoken approvals, and trace per-stage latency.

Evaluation. Task success is necessary and insufficient. Interruption rate, mean response latency, p99 response latency, barge-in handling correctness and containment rate all matter, and a suite that measures only correctness will pass a system callers hate (Chapter 24).