Guardrails & Stop Conditions
Bounding behaviour, spend and runtime — the four guardrail surfaces, loop detection, kill switches, and the false-positive rate nobody measures.
In this hypothetical runaway, Reference Agent spends $4,180 in eleven minutes.
A tenant’s document store had been migrated, and the search tool now returned a “not found” response for every query. The agent, working a research task, searched. Got nothing. Reformulated — correctly, per Chapter 11 — and searched again. Got nothing. Reformulated again. It ran for eleven minutes at forty parallel searches per turn, each turn re-sending a context that grew with every empty result, until someone noticed the spend graph.
The tool reported no results, the model reformulated, and the loop continued. No control detected the lack of progress or stopped the spend.
20.1Four things called guardrails#
Four control surfaces serve different jobs; toolkit terminology varies.
Input guardrails decide whether content may enter the context. Untrusted material, injection attempts, secrets arriving from a file read.
Action guardrails decide whether a proposed action may proceed. Distinct from permissions: policy asks is this allowed, an action guardrail asks is this sensible right now — is it the fourth identical attempt, does it carry a credential, is it wildly out of pattern.
Output guardrails decide whether a response may leave. Leaked secrets, prohibited content, unsupported claims.
Budgets constrain consumption. Some toolkits include them under guardrails. Keep resource accounting explicit and enforce it independently of the model. Content checks alone cannot establish a spending limit.
Design Rule
Content and action checks assess proposed behaviour. Budgets count resource consumption. Give each explicit enforcement and failure semantics. A model response must not override either control.
20.2Severity: three outcomes, not two#
A guardrail that can only allow or block is too blunt. Three levels, each with a defined effect on the loop:
class Severity(str, Enum):
WARN = "warn" # annotate, redact, continue
BLOCK = "block" # refuse this item, keep running
HALT = "halt" # end the run nowWARN is the level that makes guardrails usable. A suspected injection in a retrieved document is worth flagging and labelling without ending a legitimate task. Most detectors with any recall have a meaningful false-positive rate, and a detector whose only setting is “stop the run” will be turned off within a month.
BLOCK refuses one thing and returns an observation. The agent finds another route — exactly like the policy denial in Chapter 15, and for the same reason: a refusal the model can read is a refusal it can work around.
HALT ends the run. Reserve it for cases where continuing is itself the problem: a detected loop, an attempted exfiltration, content that must not be processed at all.
The pipeline returns the most severe violation, and the loop maps severity onto behaviour:
if v := action_gr.worst(call):
if v.severity is Severity.HALT:
return RunOutcome.GUARDRAIL_STOP, "", v.detail # classify loop trips separately
if v.severity is Severity.BLOCK:
self._observe(blocked(call, v), executed=False)
continue
# WARN: recorded as an event, execution proceedsNote that every trip is an event in the log (Chapter 5) regardless of severity. A WARN that fires ten thousand times a day is telling you something, and you can only hear it if it was recorded.
20.3The five ceilings, in detail#
Each of the five ceilings catches a failure the others miss:
Turns. Catches structural non-convergence — the agent that will never finish. Alone, it misses the run that blows the budget in three turns because each turn is enormous.
Tokens. Catches context blowup, usually a tool returning something huge. Alone, it misses the run that stalls: a hung HTTP call spends no tokens and can wait forever.
Wall clock. Catches hangs and slow dependencies. Alone, it misses the run that is fast and expensive — forty parallel calls to a premium model completing quickly.
Cost. Catches the expensive-model-plus-long-context combination that looks reasonable in token count and is ruinous in dollars. Alone, it misses fan-out that is individually cheap.
Tool calls. Catches fan-out. The $4,180 incident was a fan-out failure: forty parallel searches per turn stayed under a per-turn token limit and under a wall-clock limit, and the tool-call ceiling is the one that would have caught it in the first minute.
@dataclass
class Budget:
max_turns: int = 20
max_tool_calls: int = 60
max_tokens: int = 400_000
max_wall_clock_s: float = 900.0
max_cost_usd: float = 2.00
max_repeat_signature: int = 3Ceilings are checked at turn boundaries, which means a single turn can overshoot. A max_cost_usd of two dollars does not guarantee a bill under two dollars; it guarantees the run stops at the next boundary after crossing. If you need a hard per-call cap it belongs in the gateway, not the loop, and knowing this distinction is what prevents a false sense of protection.
The reason must be specific. exceeded() returns a string naming which ceiling and by how much, because “budget exhausted” tells an operator nothing and “tool-call ceiling reached (60/60)” tells them exactly what to look at.
And the asymmetry rule from Chapter 4, restated because it is the most useful heuristic here: autonomy and budget move in opposite directions. Hermes documents a default iteration budget of 500 for its main loop and 50 for a delegated subagent, with independent child budgets; the total run tree can exceed the parent’s cap.1 An unattended sub-run spawned by a model gets an order of magnitude less rope than an attended main run. That instinct generalises to every autonomy level you define.
20.4Admission control before the first turn#
A per-run ceiling answers how much one admitted run may consume. It does not answer whether you can afford to admit another run while hundreds are already active. That second question is admission control, and without it perfectly bounded runs can still overwhelm a tenant, a dependency, or your own model account.
Suppose Reference Agent has the following illustrative limits for one tenant:
| Quantity | Limit | Already committed | Proposed run tree | Result |
|---|---|---|---|---|
| Concurrent runs | 24 | 23 | 1 | fits exactly |
| Reserved model-spend exposure | $18.00 | $15.40 | $3.20 | exceeds by $0.60 |
| Search calls per minute | 600 | 410 | 120 | leaves 70 |
| Irreversible actions in flight | 3 | 3 | 0 | unchanged |
Checking only concurrency admits the run: (23 + 1 = 24). Checking every dimension rejects or queues it because ($15.40 + $3.20 = $18.60), which is ($0.60) above the tenant’s $18 reserved-exposure capacity. Each dollar in this row means outstanding model spend that an admitted run is allowed to consume before it releases the unused remainder; it is not a rate. The run’s own two-dollar ceiling does not help unless the scheduler reserves that exposure before admission.
The arithmetic becomes more revealing with delegation. Assume this run may spawn four research sub-agents. Each child gets 15% of the parent’s two-dollar budget:
| Component | Count | Budget each | Maximum committed spend |
|---|---|---|---|
| Parent | 1 | $2.00 | $2.00 |
| Research children | 4 | $0.30 | $1.20 |
| Verification child | 1 | $0.20 | $0.20 |
| Run tree | 6 | — | $3.40 |
The naive calculation reserves two dollars because that is the parent’s ceiling. The correct worst case is ($2.00 + (4 \times $0.30) + $0.20 = $3.40). If children draw from the parent’s allowance rather than receiving additional allowances, encode that as a shared ledger; do not merely rely on a prompt telling them to be economical. Either design can work. What fails is leaving the accounting ambiguous.
Admission also needs a reservation and release protocol. Read the available capacity, reserve it atomically, start the run, then release unused capacity when the run ends. A read followed by an ordinary write is not enough: two workers can both observe one free slot and both take it. Production admission therefore belongs beside the shared scheduler or gateway, where the reservation can be transactional. It does not belong inside a model-visible tool, because the model must not be able to forge a release or choose which dimension to ignore.
The gateway will not know the final cost before the run begins, so reservation uses an estimate with an explicit confidence policy. Derive it from a task class, requested tools, context size, autonomy level, and authorised fan-out — inputs the harness knows before the first model call. Suppose historical invoice-reconciliation runs have these illustrative spend percentiles:
| Percentile | Spend | Duration | Tool calls |
|---|---|---|---|
| p50 | $0.42 | 38 s | 9 |
| p90 | $0.88 | 94 s | 21 |
| p99 | $1.46 | 211 s | 43 |
Reserving the median admits more work but overloads the account whenever several ordinary tail cases coincide. Reserving the two-dollar absolute ceiling wastes capacity because nearly every run releases most of it. A practical starting point is the p90 estimate plus a contingency for the execution mode. With a 20% contingency, one interactive run reserves ($0.88 \times 1.20 = $1.056), rounded up to $1.06. Sixteen such admissions commit $16.96 of an $18 outstanding-exposure pool and leave $1.04 uncommitted. This is statistical admission, not a maximum-spend guarantee: sixteen runs each entitled to two dollars can still consume $32. For a hard $18 pool, require an atomic reservation top-up before every billable dispatch and pause or stop work if it fails. Reconcile actual usage afterwards; reserve the worst permitted in-flight cost when the provider cannot cap it precisely.
Compare estimated with actual consumption after every run. If the ratio actual / reserved exceeds one on 18% of tasks, the reservation model needs investigation or segmentation (the 20% contingency means this is not the raw p90 exceedance rate). If it stays below 0.4 on nearly every task, you are reserving idle capacity. Do not let the model self-report that a task is “simple” to win a cheaper reservation. Classify from trusted request metadata and observed history, then tighten or relax only from measured error.
Reservation leaks are their own stop-condition failure. A worker can die after reserving capacity and before releasing it, eventually making an idle system look full. Give every reservation a lease tied to a run identifier, renew it while the run is live, and reclaim it only after the lease expires and the authoritative run record confirms that no worker still owns the task. Releasing on expiry without that ownership check creates the opposite fault: a slow but live run loses its reservation while continuing to spend. Track reserved, consumed, released, and expired units as separate events so capacity accounting can be replayed instead of reconstructed from gauges.
You need a declared response when capacity is unavailable:
- Queue work that remains valuable later, such as a scheduled corpus audit. Record the queue age as part of its deadline.
- Degrade work that has a cheaper safe mode, such as reducing parallel retrieval from eight branches to two before it starts.
- Escalate work whose deadline will expire before capacity is likely to return. An urgent task that cannot be served safely is a human problem, not a reason to erase the ceiling.
- Reject duplicate, expired, or low-value work with a reason the caller can act on.
This is a production extension, not a capability hidden in the reference implementation. The reference BudgetTracker owns counters for one run in one process. It does not coordinate tenant-wide reservations, queue work, or maintain a shared delegation ledger. That narrowness is deliberate: the reference makes per-run enforcement inspectable. A deployed Reference Agent needs the shared layer as well, and its tests need to cover two simultaneous admissions competing for the final unit of capacity.
Design Rule
Reserve the whole run tree you have authorised, not merely the parent you can currently see. If child budgets are shared, enforce the shared ledger. If they are additive, reserve the sum. Ambiguous budget inheritance is unbounded fan-out wearing bounded-run labels.
20.5Loop and oscillation detection#
Budgets stop a runaway eventually. Loop detection stops it while naming the problem, which is the difference between an operator who knows what happened and one who sees a spend alert.
@dataclass
class LoopDetector:
max_repeats: int = 3
window: int = 8
_recent: list[str] = field(default_factory=list)
def check(self, subject: object) -> Violation | None:
if not isinstance(subject, ToolCall):
return None
sig = subject.signature()
self._recent.append(sig)
self._recent = self._recent[-self.window:]
if self._recent.count(sig) >= self.max_repeats:
return Violation(self.name, Severity.HALT,
f"call {subject.name} repeated {self._recent.count(sig)}x with "
f"identical arguments — no progress is being made")
if len(self._recent) >= 4:
a, b, c, d = self._recent[-4:]
if a == c and b == d and a != b:
return Violation(self.name, Severity.HALT,
f"oscillation detected between {a} and {b}")
return NoneExact repetition — the same call with the same arguments, three times. Almost always a tool returning an error the model cannot act on, which makes a loop detection trip a diagnostic pointing at Chapter 13: read the error text, and it will be rung 1 or 2 on the quality ladder.
Oscillation — A, B, A, B. Usually two tools undoing each other’s work, or a plan whose steps have a circular dependency. Harder to spot by eye in a transcript and trivial to detect in code.
Two refinements from practice. Detection must be on the canonical signature, so incidental argument reordering does not defeat it — which is why ToolCall.signature() sorts keys. And the window matters: an eight-call window has no clock and can still flag legitimate polling over twenty minutes. A production detector needs an explicit polling exemption with minimum interval, live job identity, deadline and attempt budget.
What the detector cannot catch is semantic repetition — the same query in three different phrasings, which is exactly what the $4,180 incident did. Each search had different arguments, so the signature differed, so no repeat was detected. Catching that requires either embedding-similarity over recent calls or a simpler and more robust proxy: a null-result counter. A configured sequence of empty or error results is a useful lack-of-progress signal, but slow jobs and sparse searches need explicit exceptions and deadlines. Empty results can also establish useful negative evidence.
Failure Mode
The $4,180 lesson generalised: your stop conditions must include a progress signal, not only resource ceilings. An agent can consume its entire allowance while every individual action is well-formed, permitted, non-repeating and sensible. Count consecutive empty or error results, and halt on a threshold. This is the cheapest guardrail in this chapter and the one most often absent.
20.6Fail closed without hiding the failure#
In an illustrative incident at 10:06, a production action guardrail began timing out. It depended on a remote classification service, and a routine certificate rotation had left one client with an expired trust store. The first timeout was treated as an ordinary detector miss: the pipeline caught the exception, logged a warning, and continued to the policy check. For forty-three minutes, every proposed action was effectively evaluated as though that guardrail had returned “clean.”
The team’s assumption was operationally attractive and structurally wrong: a guardrail timeout is not evidence that the action passed. The classification service had not decided anything. Converting “unknown” into “allow” changed an unavailable safety control into an invisible permission expansion. Nothing in the action payload showed that the boundary had disappeared.
The immediate fix was to stop converting exceptions into empty findings. The durable fix was to define failure semantics per surface:
| Failure point | Safe default | User-visible outcome | Recovery route |
|---|---|---|---|
| Input check unavailable | Withhold the untrusted result | Tool observation says inspection unavailable | Retry dependency or use a trusted source |
| Action check unavailable | Do not execute the action | Named guardrail-unavailable stop | Human review or later retry |
| Output check unavailable | Do not release the answer | Response withheld | Re-run check or hand off |
| Monitoring-only sampled check unavailable | Continue, mark sample missing | No task interruption | Repair monitor; do not count as a pass |
The last row is the necessary distinction. A check that gates behaviour must fail closed. A check whose declared purpose is estimation may fail open because it has no authority over the action in the first place — but the missing sample must stay missing. Recording it as a clean sample corrupts the rate and makes the monitor look healthier during the exact outage that should lower your confidence.
The reference loop already has two useful structural properties. First, an exception from action_gr.worst(call) occurs during pre-execution vetting and reaches the outer run boundary as RunOutcome.ERROR, so no member of that proposed batch is invoked. Do not generalise that timing to every surface: an input guardrail inspects a result after its tool has run, while an output guardrail inspects text after the model has produced it; they can withhold data or output, but they cannot undo the preceding computation. Second, the action path vets the entire proposed batch before invoking any member. The following is a control-flow reduction based on code/reagent/loop.py, not a verbatim excerpt; every omitted branch is labelled:
approved: list[tuple[ToolCall, Tool]] = []
for call in calls:
# tool lookup and unknown-tool observation omitted
if v := action_gr.worst(call):
self._emit(EventType.GUARDRAIL_TRIGGERED, guardrail=v.guardrail,
severity=v.severity.value, detail=v.detail,
tool=call.name)
if v.severity is Severity.HALT:
reason = v.detail
return (RunOutcome.LOOP_DETECTED
if "loop" in v.guardrail or "oscillation" in v.detail
else RunOutcome.GUARDRAIL_STOP), "", reason
if v.severity is Severity.BLOCK:
self._observe(ToolResult.error(
call, f"Blocked by guardrail {v.guardrail}: {v.detail}"), executed=False)
continue
# policy evaluation and approval handling omitted
approved.append((call, tool))
if not approved:
return None
for call, tool, result in self._invoke_all(approved):
self._observe(result)
return NoneThe sequencing is the important part. If a model proposes [read_customer, close_account, send_email] and vetting close_account crashes, read_customer must not already be running in a parallel worker. Vetting is sequential and total; execution begins only after the batch has survived it. That is why the approved list exists.
The reference boundary is safe but not operationally complete. ERROR tells you the run failed; it does not distinguish a broken guardrail from a model client exception or a coding defect. A production Reference Agent should map a gating-check outage onto a named outcome such as GUARDRAIL_UNAVAILABLE, attach the guardrail name and dependency status, and decide whether the open task becomes ESCALATED or retryable. That outcome is a proposed production extension. It is not present in the reference RunOutcome enum.
Fail-closed behaviour also needs to survive the less obvious cases:
Timeouts need their own deadline. If the action guardrail inherits the run’s fifteen-minute wall clock, “fail closed” can mean hanging for fifteen minutes. Give the check a short deadline, then stop with a named reason.
Cached decisions need an expiry and policy version. A clean result from yesterday is not a substitute for today’s unavailable check after the rules changed. Key a cache entry by the canonical subject, guardrail version, tenant policy version, and relevant context; expire it deliberately.
Fallbacks must be no more permissive. Falling from a model-based exfiltration detector to a deterministic secret scanner may be acceptable if the fallback narrows behaviour — for example, by requiring human review on everything it cannot confidently clear. Falling back and preserving the original ALLOW threshold is a silent relaxation.
Operator bypasses must be scoped. During an outage you may choose to bypass one check for one tenant and one read-only capability. That is a new policy decision with an owner, expiry, reason, and audit record. A process-wide environment variable called DISABLE_GUARDRAILS is not incident response; it is removal of the boundary.
Be precise about BLOCK and HALT when a batch contains several calls. BLOCK means this item is refused while the run may find another route. That is safe only because the blocked call is represented as executed=False and returned to the model as an observation. HALT means continuing the run is unsafe, so the rest of the batch must not execute either. A faulty implementation that collects approved calls, encounters a later HALT, and then executes the earlier approvals has reduced halt to “skip the offending item.” The reference returns immediately from the vetting pass, before _invoke_all, so no member of that batch starts.
Now consider a mixed batch of five read calls where the fifth carries a credential. Parallel execution would save latency, but starting calls one through four while scanning call five makes the safety result depend on list order. A credential in the first position stops everything; the same credential in the fifth position lets four calls escape. Total vetting removes that positional accident. For large batches, parallelise pure detector computation if you need to, but join every gating result before crossing into execution. The batch is the unit of admission even when each action remains the unit of policy.
The action-path operational test is simple: disconnect every action-guardrail dependency one at a time and submit the most dangerous action that dependency protects. No proposed action executes. The trace names the unavailable guardrail. The operator gets a bounded recovery route. Test input and output dependencies separately: the tool or model has already run at those points, so assert that the uninspected result or answer is withheld rather than pretending the preceding computation never happened. If any expected boundary is crossed, your fail-closed claim exists only on the happy path.
20.7Kill switches and circuit breakers#
A circuit breaker protects one dependency. Three states — closed, open, half-open — with the purpose of stopping requests to something that is not answering, rather than letting a degraded dependency consume the whole run’s time ceiling:
@dataclass
class CircuitBreaker:
failure_threshold: int = 5
cool_off_s: float = 30.0
def allow(self) -> bool:
st = self.state # closed | open | half_open
if st == "closed":
return True
if st == "half_open" and not self._half_open_probe:
self._half_open_probe = True # exactly one probe
return True
return FalseThe half-open single-probe behaviour is the part people skip, and without it a recovering dependency is hit by the full retry backlog the moment the cool-off expires — turning recovery into a second outage.
A kill switch stops the agent. Four properties, and all four are needed:
It works without a deploy. A configuration flag, a feature flag, a database row — anything the on-call can flip in seconds. A kill switch that requires a release is not one.
It has granularity. All agents, one tenant, one capability, one autonomy level. A single global switch means the only available response to a problem with one tenant is to stop serving everyone, which is a response nobody will take.
In-flight runs stop safely. Check at model and effect dispatch boundaries, revoke queued authority, and stop new mutations immediately. Reconcile already dispatched operations; neither a local cancellation nor a turn-boundary check proves that a remote effect stopped.
Flipping it is an audited event. Who, when, why, and when it was restored.
The related design question is what the agent does when it is switched off mid-task. Record a specific stop reason and remaining task state. Use ESCALATED when a human must take over; an authorised cancellation or expired task may instead be safely abandoned. The reference harness does not ship a distributed kill switch.
20.8Guardrails on actions, not only text#
Most guardrail tooling was built for chat and checks text. An agent’s dangerous outputs are actions, and text-shaped guardrails do not see them.
Four action-level checks complement content classifiers:
Rate and volume. Fifteen credits in one run when the historical maximum is two. No individual credit is wrong; the count is. This is an anomaly check against a baseline, and it catches a class nothing else does.
Aggregate value. Each credit under the $500 threshold, forty of them in one run. A per-action limit with no per-run aggregate is a limit that can be walked around by anything doing arithmetic — including a confused model.
Pattern deviation. This tenant’s tasks have never touched the ledger. This one does. Worth a WARN and a look.
Sequence sensibility. A delete before a read. A write before a verification. Some orderings are simply wrong and are cheap to encode as rules over the trajectory.
The reference implementation’s SecretScanner on the action path is the simplest member of this family, and it catches a real attack: a credential in a tool call’s arguments, which is the shape of an exfiltration attempt.
def check(self, subject: object) -> Violation | None:
text = _as_text(subject) # for a ToolCall: its JSON arguments
for label, pattern in SECRET_PATTERNS:
if m := re.search(pattern, text):
return Violation(self.name, self.severity,
f"possible {label} detected in content",
redacted=text.replace(m.group(0), f"[REDACTED:{label}]"))
return None20.9Cost, latency, and the ordering that makes them affordable#
Every guardrail is a tax. A 200-millisecond model-based check on the output path is 40% of a voice budget (Chapter 18) and meaningful even in text.
Order by cost, so most content never reaches the expensive stage:
- Structural checks — length, type, schema. Microseconds.
- Pattern matching — secret and injection regexes. Microseconds.
- Small classifier — a purpose-built model for content categories. Tens of milliseconds.
- Model-based judgement — a full model call for groundedness or policy nuance. Hundreds of milliseconds.
Two more techniques worth having. Gate before release when prevention is required. A check running alongside visible streaming can limit later output but cannot retract already exposed secrets. Buffer the relevant unit until mandatory checks pass; use parallel checks for monitoring only when that residual exposure is acceptable. And sample rather than gate for expensive checks whose purpose is monitoring rather than prevention: a groundedness check on 5% of responses tells you the rate without taxing every response, and Chapter 24 is where that measurement belongs.
For asynchronous tools, use protocol state instead of interpreting silence as failure. MCP’s current Tasks extension returns durable handles and explicit states including working, input required and terminal outcomes. Bound polling, keep its handle across reconnects, and let its lifecycle inform the progress detector rather than blindly rejecting repeated status calls.2
20.10Measuring guardrails#
A guardrail is a classifier with two error modes, and you need both numbers.
False positives — legitimate content or actions blocked. This is what destroys trust. With one gating decision per legitimate task, a 3% false-positive rate across ten thousand legitimate tasks blocks three hundred of them. More checks per task change the task-level rejection rate; complaints and operational response are separate measurements. The false-positive rate determines whether your guardrail survives, and it is almost never measured before deployment.
False negatives — the thing it exists to catch, missed. Harder to measure because it requires a labelled adversarial set, which means you have to write attacks. Chapter 21 does.
| Measurement | How | What it tells you |
|---|---|---|
| False-positive rate | False blocks ÷ labelled legitimate gating decisions; compare task completion separately | Whether the guardrail will survive contact with users |
| False-negative rate | A labelled set of things that should be caught | Whether it does its job |
| Trip rate in production | Count violations by guardrail and severity | Which guardrails are load-bearing and which are decoration |
| Latency contribution | Per-guardrail timing on the critical path | What you are paying |
The trip-rate distribution is the most immediately useful. A guardrail that has never fired in six months is either protecting against something that does not occur — fine, it is cheap insurance — or it is broken. Both are worth knowing, and you can distinguish them by feeding it a known positive.
20.11Measure the stop path as a system#
Classifier accuracy tells you whether a guardrail recognises examples. It does not tell you whether the harness obeys the result, whether budgets stop near their intended ceilings, or whether people recover the work afterwards. You need a second measurement layer over the control path.
Measure two intervals separately. Dispatch-stop latency runs from the condition becoming true until no new effect can start. Quiescence latency runs until already admitted work has completed, cancelled or reached a reconciled terminal state. A turn-boundary kill switch can have a short dispatch-stop latency while a remote mutation remains unresolved. Track p50, p95, and maximum for both by stop source. A p95 of 1.2 seconds with a maximum of eleven minutes tells you exactly where the incident is hiding.
Then measure overshoot in the unit each ceiling controls:
| Stop source | Intended ceiling | Actual at stop | Overshoot | Interpretation |
|---|---|---|---|---|
| Tokens | 400,000 | 427,600 | 27,600 / 6.9% | One large turn crossed the boundary |
| Cost | $2.00 | $2.14 | $0.14 / 7.0% | Boundary check worked; per-call reservation may be warranted |
| Tool calls | 60 | 68 | 8 / 13.3% | A parallel batch was admitted without reserving its calls |
| Wall clock | 900 s | 904 s | 4 s / 0.4% | Poll interval explains the delay |
These figures are illustrative. They expose different mechanisms. Token and cost overshoot are expected when the reference tracker checks between turns. Eight extra tool calls reveal a different design: a batch crossed the ceiling because calls were counted after admission. If you require a hard tool-call limit, reserve the batch before execution: with 57 calls used and 4 proposed, reject or trim the batch because (57 + 4 = 61 > 60). Do not launch four calls and announce afterwards that the total is sixty-one.
Third, record stop precision. The reference tests make an important assertion: repeated calls should end as LOOP_DETECTED before the broader turn ceiling labels them BUDGET_EXHAUSTED. Both stop the run, but only one names the mechanism. Measure the proportion of stopped runs assigned to their most specific applicable reason. A falling rate usually means a generic ceiling or exception handler is masking earlier evidence.
Rates need denominators that match the decision point. “Loop detector fired 240 times” is uninterpretable. Out of 300 runs it is catastrophic; out of thirty million it may be noise. Report repeat-loop trips per 10,000 runs that reached at least three tool calls, output blocks per 10,000 candidate outputs, and admission rejects per 10,000 requests by queue. Exclude runs that could never have reached the detector from that detector’s denominator. Otherwise a traffic shift towards one-turn questions makes the loop rate appear to improve even if loop behaviour is unchanged.
Slice those rates by tenant, task class, autonomy level, tool and guardrail version, while keeping privacy and sample size rules intact. A global false-positive rate of 0.08% can hide a 7% rate for one tenant whose document templates contain instruction-shaped headings. That tenant will disable the feature long before the global dashboard turns red. Use minimum sample thresholds and show counts beside percentages; three false positives in twenty examples are a reason to investigate, not a stable estimate. When a version changes, preserve the before-and-after boundary rather than averaging both regimes into one reassuring number.
Fourth, measure work recovery. For every non-completed terminal state, ask what happened next:
- resumed from durable state and completed;
- escalated with a handoff and completed by a person;
- safely abandoned because the task expired or duplicated another run;
- retried from the beginning, repeating cost or side effects;
- disappeared with no owner.
Stops that strand legitimate work need a handoff or resumable state. Measure which path follows each stop (Chapters 19 and 5).
Finally, run stop-path drills, not only unit tests. A unit test can prove that three identical signatures return HALT. A drill proves that the kill switch reaches all workers, the trace survives, no queued mutation starts afterwards, the handoff contains enough state, and capacity reservations are released. Use a matrix that crosses stop source with execution phase:
| Stop source | Before model call | During read batch | Awaiting approval | During mutation | Before output |
|---|---|---|---|---|---|
| Tenant kill switch | test | test | test | test | test |
| Cost ceiling | test | test | test | test | test |
| Guardrail unavailable | test | test | test | test | test |
| Loop detected | test | test | test | test | test |
Not every cell has the same expected response. During a mutation, safe stopping may mean waiting for the tool’s idempotent operation to resolve and then refusing to start another. Before a model call it should be immediate. Write the expected terminal outcome, maximum stop latency, and allowed final side effects in each test case. “Stopped” is too vague to verify.
The stop-condition scorecard
Review the following numbers together every week: trips by reason and severity; false-positive and false-negative rate; p95 and maximum stop latency; ceiling overshoot by unit; proportion assigned the most specific stop reason; percentage resumed or handed off successfully; and stop-path drill pass rate. A green classifier with red recovery is not a green guardrail. A green budget with unmeasured overshoot is not a hard ceiling.
The guardrail budget
Guardrails accrete the same way system prompts do, and an unreviewed accretion of checks is its own failure mode. Review the set quarterly against four questions per guardrail.
- What does it catch that nothing else catches? If it adds no useful independent coverage, consider removing or moving it. Overlapping controls can protect against different failure modes or common-cause failures; evaluate marginal coverage and correlated misses before removal.
- When did it last fire, and was that a true positive? Never firing means unproven; firing constantly on legitimate content means it is a tax.
- What is its false-positive rate on the eval suite? If unmeasured, measure it before the next review. Unmeasured guardrails are the ones that get disabled during an incident and never re-enabled.
- What does it cost on the critical path? In milliseconds and in dollars.
Then make one deliberate decision per guardrail: keep, downgrade its severity, move it off the critical path, or remove it. A guardrail set that has never been pruned is one where nobody knows which checks are doing the work.
20.12Defaults worth starting from#
Concrete numbers to begin with and tune from measurement, for a general-purpose work agent.
| Control | Starting value | Tune when |
|---|---|---|
max_turns |
20 interactive / 200 background | Legitimate tasks are being truncated |
max_tool_calls |
60 | Fan-out patterns are legitimate |
max_tokens |
400k | Long-horizon work needs more (with compaction) |
max_wall_clock_s |
900 interactive / 14,400 background | Dependencies are legitimately slow |
max_cost_usd |
$2 interactive / $20 background | Value per task justifies it |
| Repeat threshold | 3 identical calls | Legitimate polling is being caught |
| Consecutive null results | 8 | Sparse corpora produce legitimate empties |
| Sub-agent budgets | Allocate by task, with explicit fan-out ceiling | Enforce shared or additive tree accounting |
| Circuit breaker | 5 failures, 30 s cool-off | Dependency profile differs |
The last-but-one row makes delegation accounting explicit. A child may need more work than the coordinator, so no universal fraction is correct. Bound the total run tree, reserve concurrent work, and require fresh capacity before extending it.
Definition
Guardrail — a check on content or a proposed action, with three severities: warn, block, halt.
Budget — a configured consumption limit that the model cannot override. The reference checks between turns and can overshoot. Hard limits need dispatch-time reservations.
Progress signal — a stop condition based on lack of advancement rather than resource use, such as consecutive null or error results.
Key Takeaways
- Separate input, action, output and consumption controls. Terminology varies across toolkits. Check that every required surface has explicit enforcement and failure behaviour.
- Three severities make guardrails usable. Warn-and-continue is what keeps a detector with a real false-positive rate from being switched off; halt is for cases where continuing is itself the problem.
- All five ceilings are necessary because each alone misses a real failure. Tool-call ceilings catch fan-out, which is the failure that stays under every other limit.
- Ceilings are enforced at turn boundaries, so a single turn can overshoot. A hard per-call cap belongs in the gateway, and knowing this prevents a false sense of protection.
- Stop conditions must include a progress signal, not only resource ceilings. Counting consecutive null or error results is the cheapest guardrail available and the one most often missing — it is what would have caught the eleven-minute runaway.
- Per-run ceilings do not provide admission control. Reserve tenant and dependency capacity atomically, include authorised child budgets in the run tree, and define whether unavailable capacity queues, degrades, escalates or rejects the task.
- A gating guardrail has three results: pass, finding and unavailable. Unavailable must not become allow. Fail closed with a named, observable recovery path, and keep monitoring-only missing samples out of the pass count.
- Kill switches need scoped, audited control without a deploy. Stop new effect dispatch and reconcile in-flight operations. Escalate work needing human ownership. Record authorised cancellation separately.
- Agents produce dangerous actions, not just dangerous text. Rate, aggregate value, pattern deviation and sequence sensibility catch classes no content classifier sees.
- Measure the false-positive rate before deploying a guardrail. It determines whether the guardrail survives contact with users, and an unmeasured one is the one that gets disabled during an incident and never re-enabled.
- Measure the control path as well as the classifier: stop latency, ceiling overshoot, reason specificity and successful recovery reveal whether the harness actually obeyed the decision.
Hands-on
Find the ceiling you are missing 1. For each of the five ceilings, construct a run that breaches it and confirm the outcome names the specific ceiling. Any ceiling you cannot breach on purpose is one you have not really set. 2. Add a consecutive-null-result counter and halt at eight. Then look for runs in your history that would have tripped it; the eleven-minute runaway class is more common than teams expect. 3. Run your full eval suite with guardrails enabled and count failures that would otherwise have passed. Inspect which rejected decisions were legitimate, then divide false blocks by all labelled legitimate decisions. Task failures alone do not identify false positives. 4. Check your sub-agent budgets against your parent budgets. If they are the same, compute your worst case with maximum fan-out and decide whether you can afford it. 5. Test the kill switch end to end. Confirm new mutations stop and in-flight effects reach a reconciled state. Check the stop reason, recovery owner and handoff. Measure dispatch-stop and quiescence latency separately. 6. Model admission for a parent that can spawn four children. State whether budgets are shared or additive, calculate the maximum commitment, and race two admissions for the final slot. Exactly one reservation should succeed. 7. Disconnect each action-guardrail dependency and propose the highest-risk action it protects. Confirm that no proposed action executes, the run names the unavailable check, and an operator has a scoped recovery route. For input and output checks, confirm that the already-produced result is withheld. 8. For one ceiling, record both the configured limit and the actual value at stop across fifty runs. Explain the maximum overshoot from execution ordering; if you cannot, the ceiling is not yet predictable. 9. Build the stop-path matrix for one mutating tool. For each execution phase, write the permitted final side effects, maximum stop latency, terminal outcome and recovery owner. Drill two cells by disconnecting a real dependency, then compare the trace with the contract rather than accepting a generic “stopped” result.
-
Agent loop — Hermes Agent developer guide. Nous Research. Accessed 9 November 2026. https://hermes-agent.nousresearch.com/docs/developer-guide/agent-loop ↩
-
Model Context Protocol. Tasks extension overview. Accessed 9 November 2026. https://modelcontextprotocol.io/extensions/tasks/overview ↩