Retries, Recovery & Failure Handling

In this illustrative incident, at 03:12 a provider began returning 503 on about one request in eight. Not an outage — a degradation, the kind that resolves itself in twenty minutes.

Reference Agent had forty-one runs in flight. Each one hit an error, retried immediately, hit another, retried immediately. Within ninety seconds the retry volume had pushed the provider’s error rate for Reference Agent’s account from one in eight to nearly all, because the account’s rate limit was now saturated by retries. Runs that would have succeeded on the first attempt failed. The forty-one became a backlog, the backlog was retried by a supervisor, and a twenty-minute provider wobble became a two-hour incident with a postmortem.

Missing delay, jitter, retry budgets and dependency protection amplified the provider degradation.

22.1A taxonomy, because the response depends on the class#

Classify failures before retrying: retrying a non-retryable failure can turn a bad argument into a rate-limit incident.

class FailureClass(str, Enum):
    TRANSIENT    = "transient"      # network blip, 503 — retry only if safe
    RATE_LIMITED = "rate_limited"   # 429 — retry, slowly, honour Retry-After
    INVALID      = "invalid"        # bad arguments — do NOT retry; tell the model
    PERMISSION   = "permission"     # denied — do NOT retry; authorised alternative or escalate
    FATAL        = "fatal"          # broken dependency — trip the breaker
    SEMANTIC     = "semantic"       # the call succeeded, the answer is wrong — replan

Semantic failures return valid data that is wrong for the task: rows from the wrong date range, or plausible but irrelevant documents. A transport retry cannot correct wrong arguments or stale data. Verify the result and repair the query, tool, data or plan. A bounded retry of stochastic behaviour needs evidence (Chapter 6).

22.2Retry, done properly#

@dataclass
class RetryPolicy:
    max_attempts: int = 4
    base_delay_s: float = 0.25
    max_delay_s: float = 8.0
    multiplier: float = 2.0
    jitter: float = 0.5
    retry_on: frozenset[FailureClass] = frozenset(
        {FailureClass.TRANSIENT, FailureClass.RATE_LIMITED})

    def delay_for(self, attempt: int, rng: random.Random) -> float:
        """Exponential backoff with jitter. Attempt is 1-based.

        Jitter is not decoration: without it, N agents that failed together
        retry together, and the thundering herd is worse than the original
        failure.
        """
        raw = min(self.max_delay_s, self.base_delay_s * (self.multiplier ** (attempt - 1)))
        return raw * (1 - self.jitter * rng.random())

Exponential backoff gives the dependency time to recover.

Jitter reduces synchronised retry bursts: without it, forty-one runs that failed at the same moment retry at the same moment, three times, and the synchronised herd is worse than the original degradation.

A budget — max_attempts, and ideally a wall-clock ceiling — bounds the total.

And class-based gating via retry_on means an INVALID failure raises immediately rather than being hammered.

Honour the server’s instruction within the operation deadline. Parse Retry-After as either delay seconds or an HTTP date in the transport adapter, and distinguish an explicit zero from a missing value. If the wait cannot fit the deadline, defer or fail instead of retrying early. A production scheduler can wait longer for local capacity:⁵

delay = (exc.retry_after_s if exc.retry_after_s is not None
         else self.policy.delay_for(attempt, self.rng))

Make it testable. Inject the clock and random-number generator so tests need no real sleeps:

def _retrier(**kw):
    slept: list[float] = []
    return Retrier(sleep=slept.append, rng=random.Random(7), **kw), slept


def test_non_retryable_failure_raises_immediately():
    r, slept = _retrier()
    calls = {"n": 0}
    def bad_args():
        calls["n"] += 1
        raise RetryableError("missing field 'account'", FailureClass.INVALID)
    with pytest.raises(RetryableError):
        r.call(bad_args)
    assert calls["n"] == 1, "an invalid request must not be retried"
    assert slept == []

22.3Retry budgets and where they belong#

A subtlety that catches teams: retries nest. The HTTP client retries, the tool retries, the loop retries by calling the tool again, and a supervisor retries the whole run. Four layers of three attempts is eighty-one attempts, and nobody designed that.

The discipline is that a retry budget belongs to a logical operation, not to a call site. One layer owns retrying a given operation, and the others propagate the failure. In the reference implementation that layer is the tool invocation, and the loop above it does not retry — it observes the failure and lets the model decide, which is a better response than a mechanical repeat because the model can choose differently.

The signal that you have this wrong is a latency distribution with a long, lumpy tail at suspiciously round multiples. Eighty-one attempts at a two-second timeout is a request that takes three minutes to fail, and the user experienced it as a hang.

22.4Price the retry burst, not the individual delay#

Per-call backoff is necessary and still insufficient. A retry policy answers when this operation tries again. It does not answer whether the fleet can afford all those attempts. That second question is capacity planning, and the arithmetic is simple enough to do before an incident.

Take an illustrative Reference Agent deployment issuing 120 logical operations per second to a provider that can sustain 130 physical attempts per second for this tenant. The provider develops an independent 12% transient failure rate. Assume each failed attempt may be retried, successful operations stop immediately, and max_attempts includes the first attempt, as it does in the reference RetryPolicy.

For at most four attempts, the expected attempts per logical operation are E[A] = 1 + 0.12 + 0.12² + 0.12³ = 1.136128.

You always make the first attempt. You make the second for the 12% that fail once, the third for the 1.44% that fail twice, and the fourth for the 0.1728% that fail three times. Multiply by the arrival rate and, without fleet throttling, the provider sees expected demand for 120 × 1.136128 = 136.33536 physical attempts per second. The local retry policy is behaving exactly as configured while unthrottled potential demand exceeds the tenant’s downstream capacity by 6.33536 attempts per second.

Every value is illustrative, and the independence assumption is deliberately generous. Real dependency failures correlate: a regional fault, a shared quota, or a deployment makes many attempts fail together. The last column is therefore the one to design against. The attractive success numbers in the preceding column exist only if the provider accepts the generated load, which it does not in this scenario.

The calculation changes the response. Four attempts are safe only below 130 ÷ 1.136128 = 114.424 logical operations per second. If Reference Agent must accept 120, the harness needs a fleet-level control outside the reference retrier: admit less work, queue it behind a concurrency limit, shed low-priority operations, negotiate more capacity, or consume from a shared retry-token bucket. Jitter spreads the burst across time; it does not remove the 6.33536 attempts per second of unthrottled excess demand.

A retry token bucket makes the budget global. Give first attempts their normal capacity allocation and let retries consume a smaller shared pool. Without throttling, all later attempts in this illustration create potential demand for 120 × (1.136128 - 1) = 16.33536 retry tokens per second: 14.4 second attempts, 1.728 third attempts and 0.20736 fourth attempts. A pool issuing ten tokens per second caps actual retry execution at ten per second. The 6.33536 difference is excess potential demand, not a prediction that exactly 6.33536 attempts will be shed: delaying or rejecting an earlier retry also removes any later attempts it might have produced. The scheduler may place a retry in a deadline-bounded delay queue when near-term capacity is expected; otherwise it expires or sheds the attempt and returns a classified failure. Record executed, delayed, expired and shed counts separately. This protects the 120 first attempts that are still arriving and chooses controlled partial failure over collapse.

Deadlines must enclose all attempts and sleeps. The reference policy caps attempt count and individual delay, but it does not accept a wall-clock deadline. A production wrapper should calculate remaining time before every sleep and every attempt. A nominal four-attempt policy with delays of 0.25, 0.5 and 1 second already spends up to 1.75 seconds sleeping before tool latency; honouring a Retry-After: 12.5 response can consume far more. If the enclosing agent turn has three seconds left, sleeping for twelve and a half seconds is not resilience. It is a budget violation.

Keep three counters distinct. logical_operations counts user-intended actions, physical_attempts counts load placed on the dependency, and recovered_operations counts logical actions that succeeded after at least one failure. Dividing physical attempts by completed runs hides both nesting and overload. The ratio you need is physical_attempts / logical_operations, computed at the same ownership boundary as the retry budget. In the four-attempt illustration it is 1.136128 before capacity effects. If the observed value is 1.30, the independence model is wrong, another layer is retrying, or both.

The general lesson is sharper than “use backoff”: a local retry maximum is not a fleet retry budget. You need both. The local policy bounds one operation; admission control and shared retry capacity prevent thousands of individually well-behaved operations from failing collectively.

22.5Idempotency and compensation#

An operation with an uncertain external effect needs reconciliation before retrying (Chapter 5).

Agents cannot wrap a database transaction around five tool calls and an email. What they can do is what distributed systems have always done: record a compensating action for each committed step and run them in reverse on failure. A saga, with a model choosing the steps.

@dataclass
class CompensationStack:
    entries: list[Compensation] = field(default_factory=list)

    def push(self, description: str, undo: Callable[[], None]) -> None:
        self.entries.append(Compensation(description, undo))

    def unwind(self) -> list[str]:
        """Run every registered undo in reverse order. Best effort by design:
        one failing compensation must not prevent the others from running."""
        done: list[str] = []
        for comp in reversed(self.entries):
            if comp.ran:
                continue
            try:
                comp.undo(); comp.ran = True
                done.append(f"undone: {comp.description}")
            except Exception as exc:
                done.append(f"FAILED to undo {comp.description}: {exc}")
        return done

Persist the compensation intent before the action. An in-memory callback alone does not survive a crash. The inverse must tolerate an action that never happened, and must not run while the action is still in flight. Resolve the effect first; then apply an idempotent inverse. The teaching stack below illustrates reverse-order execution only.

Compensation is best-effort by construction. One failing undo must not prevent the others. The loop catches and records rather than raising, because a partial unwind is much better than none.

Not everything has an undo. A sent email cannot be unsent; a posted ledger entry needs a reversing entry, not a delete. Where no compensation exists, the action needs an approval gate before it happens (Chapter 19) — that is the choice: undo afterwards, or approve beforehand, and irreversible actions get the second.

A failed unwind is an incident, not a log line. It means the system is in a state nobody designed. Page someone, and put the residue in the handoff record.

22.6The ambiguous crash window#

Idempotency is usually described too casually: “write a key, then retries are safe.” The difficult case begins when a request may have left Reference Agent and ends only when Reference Agent has durably recorded the result. A crash after dispatch but before the provider commits is observationally indistinguishable from a crash after it commits unless the provider exposes authoritative evidence. No ordering of writes to two different systems can remove that interval.

The reference loop makes its exact guarantee visible without adding a second mechanism. For a declared mutation, _invoke_one derives the logical idempotency key and returns the stored result when that key is already in completed_calls. Otherwise it emits TOOL_CALL_STARTED with the call ID, arguments and key, invokes the tool through Retrier, then emits TOOL_CALL_COMPLETED with the same call ID and result. A successful mutation completion repeats the key; an error completion writes no key. Ordinary reads have no deduplication key and run again. During replay, the fold adds only successful completion keys to completed_calls.

Those transitions provide two strong properties. A successful completion is durable, so replay returns the recorded content instead of reissuing the action. A started call is also durable, so a crash does not erase the evidence that an operation was in flight. It does not prove whether an external effect happened when there is a start without a completion.

Walk through a specific failure. Reference Agent calls charge_card for £700 with idempotency key k. The event log flushes TOOL_CALL_STARTED(k). The payment provider commits the charge and returns success. Before tool.invoke returns to _invoke_one, the process loses power. On resumption, the log contains the start but no completion. The key is absent from completed_calls. If the model proposes the same logical call again, the reference harness invokes the tool again. Its local key prevents duplication only if the provider also accepts that key as its idempotency key or if a reconciliation step discovers the first charge.

The correct recovery depends on the tool contract.

Provider-supported idempotency is the strongest route. Pass Reference Agent’s stable logical-operation key to the provider. The precise retention and replay rules belong to the provider.

Stripe API v1, for example, replays the first saved result, including a 500, after endpoint execution begins. Its keys can expire after at least 24 hours. API v2 has different semantics: a retry can complete failed or partial work and return an updated response. Pin the API version. Check its retention and argument-matching rules.⁶

The reference idempotency_key combines the run ID and canonical call signature. It excludes the model-generated call ID. The same logical action within a resumed run therefore derives the same key. The example tool interface does not forward this key to an external provider. Production must add that propagation.

Authoritative lookup is the next route. Before retrying an ambiguous operation, query the system of record using an immutable business identifier: payment reference, invoice number, message ID, or deployment change ID. If it proves the effect committed, synthesise the missing completion record from the authoritative result. If it proves no effect exists, retry with the same key. A list endpoint filtered by approximate time and amount is not authoritative enough — two legitimate £700 charges can exist.

Compensation is not a substitute for knowing. Issuing a refund because the charge might have happened creates the mirror-image risk: you may refund a charge that never committed. First resolve the state; then decide whether to retain or compensate the confirmed effect. When the system of record cannot answer, stop and escalate with the start event, arguments, key, provider request ID if available, and the precise uncertainty. “Unknown” is a real outcome.

The tool needs a declared recovery mode. A production registry can require every mutating tool to choose one of four contracts:

This registry is proposed production hardening. The reference Tool model records whether an action mutates and the policy layer can ask for approval, but it does not encode these four recovery modes or reconcile incomplete starts automatically. That limitation matters because “mutating” says an operation is dangerous; it does not say how to recover it.

The start event still earns its write. It converts an invisible double-charge risk into a detectable ambiguity. Pair each TOOL_CALL_STARTED invocation with TOOL_CALL_COMPLETED by call_id, which both events carry. Then read the idempotency key from the start event to group invocations that belong to the same logical operation. The physical attempts inside one Retrier.call share that invocation rather than receiving separate start events. Retrier.attempts retains failed-attempt records only — including a terminal failed attempt — and omits the final successful attempt, so it is not a complete physical-attempt ledger. A production physical_attempts counter must increment around every actual tool.invoke attempt. Do not pair solely on the completion’s key: the current schema deliberately writes None for error completions, so that method would lose failed invocations. A production schema may repeat the logical key on every completion for easier querying, but recovery against the reference events must use call ID for the invocation and the start’s key for logical identity. Place unmatched starts into an uncertainty queue and prohibit automatic continuation until each is resolved according to its declared mode. Without the start, you cannot even name the operations that may have crossed the boundary.

There is a second, smaller boundary inside the process. The reference updates the durable completion event before it updates the in-memory completed_calls map. If the process dies between those two lines, replay folds the event back into the map, so resumption is safe. This ordering is correct. Reversing it would create a window where memory says “done” but the durable log does not, and a crash would discard the only record of completion.

Test this boundary with a fault, not a mock that returns an exception. Arrange for the external stub to commit, then terminate the worker before it returns. Resume using the same run ID and assert the system either retrieves the committed result through the same provider key or stops as ambiguous. The existing reference test proves that a completed charge is not repeated after resumption. It does not inject the crash between external commit and completion logging, so it does not prove the ambiguous case. The distinction should appear in the test name and the operational runbook.

22.7Poison pills and dead letters#

A task that fails every time it is attempted is a poison pill, and without handling it will consume capacity indefinitely — retried by a supervisor, failing, retried again.

Four queue controls prevent repeated failures from consuming capacity indefinitely:

An attempt counter on the task, persisted. After N attempts it stops being retried. Note that this must live with the task, not with the run, or each fresh run starts the count over.

A dead-letter queue. Failed tasks go somewhere a human looks, with their full trace attached. A dead-letter queue nobody reads is a delete with extra steps, so it needs an owner and a review cadence.

Failure fingerprinting. Group dead-lettered tasks by failure signature. Fifty tasks failing identically is one bug, and the fingerprint tells you that in a glance rather than after reading fifty traces.

Cost accounting on failure. A task that fails after consuming a full budget is expensive, and repeated poison pills are a meaningful and usually invisible line item. Chapter 29’s per-task cost model should include failed tasks or it is understating spend.

22.8Self-correction: what the evidence actually says#

The appealing idea: the agent notices its mistake and fixes it. Sometimes true, and the conditions matter enormously.

The 2023–2024 reasoning and planning studies tested prompts that asked models to reconsider. The prompts sometimes failed to help or changed correct answers to incorrect ones.¹² These are model- and task-specific findings, not a theorem about every deployed system. Later SCoRe research demonstrated improved self-correction after targeted reinforcement learning on mathematical and coding tasks.⁷ Evaluate the correction policy you deploy. Distinguish prompting a frozen model from training a model to correct.

The distinction that resolves this is where the feedback comes from.

A test suite supplies evidence about the cases it covers; a schema checks structure, not factual correctness. An evaluator-optimiser loop is useful when those checks represent the task (Chapter 7). Reflection can propose improvements, but fluent critique alone is not evidence that a result improved.

So the practical rule: build a verifier before building a self-correction loop. State the observations that would support correctness, then measure gains and regressions across correction rounds. Bound rounds and cost; there is no universal fourth-round cutoff.

22.9Failure taxonomies from research#

If your system has more than one agent, the failure modes multiply, and there is now a systematic account of how.

The MAST taxonomy was built from a dataset of over 1,600 annotated traces across seven popular multi-agent frameworks, and identifies 14 unique failure modes clustered into three categories: system design issues, inter-agent misalignment, and task verification — validated at high inter-annotator agreement across 150 expert-annotated traces.³

Two things make this worth using rather than admiring. It is a shared vocabulary, so a postmortem can say “this was inter-agent misalignment” and be understood. And the third category — task verification — is the one that recurs most in single-agent systems too: the failure is not that the work was done badly but that nothing checked whether it was done. Chapter 6’s per-step verification and Chapter 24’s trajectory evaluation are both responses to this category, and the research finding is that it is a leading cause rather than an edge case.

The practical use: build your own taxonomy from your own traces, seeded by this one. Chapter 25’s error analysis is the method; the point of a published taxonomy is that you do not start from an empty page.

22.10Graceful degradation#

The best failure response is often not to recover but to deliver less, honestly.

Partial results with explicit gaps. “I reconciled eleven of fourteen ledgers; three were unreachable — here they are.” Far more useful than a failure, and far safer than eleven presented as fourteen. This is the same discipline as Chapter 17’s extraction gaps: incompleteness must be visible.

Cached or stale answers, labelled. “As of 06:00 — the live source is unavailable.” Correct three hours ago and labelled beats unavailable.

Reduced capability. The premium model is rate-limited; a smaller one handles the request with a note. Chapter 29’s routing makes this a configuration rather than an incident.

A useful escalation. When nothing works, the handoff record from Chapter 19 with everything established so far. A human continuing from turn thirty is much better off than one starting from zero.

The common thread is that degradation must be visible in the output. A system that silently returns partial results has converted a legible failure into an illegible one, and the recipient acts on incomplete data believing it complete. That is worse than the failure it concealed.

22.11Recovery has two state machines#

A run can finish while one of its external operations remains unresolved. Represent these as separate states.

The reference RunOutcome describes why the agent loop ended: COMPLETED, BUDGET_EXHAUSTED, GUARDRAIL_STOP, LOOP_DETECTED, APPROVAL_DENIED, ESCALATED, TOOL_FAILURE, or ERROR. COMPLETED means the model returned a final answer. It does not, by itself, prove every external effect reached the intended business state. The distinction is explicit in the enum’s scope and essential in operations.

Consider Reference Agent reconciling fourteen ledgers. Eleven complete, two fail with classified transient errors after their attempt budgets, and the fourteenth has an ambiguous started payment adjustment. The model can return a useful final message that names all three gaps. The loop outcome is COMPLETED because it produced that answer. The business outcome is degraded, and one side effect is still unresolved. Calling the whole thing “success” discards the work queue; calling it “failure” discards the eleven correct results.

Track the two machines separately.

The second row is proposed production state, not an enum already shipped in the reference implementation. The event log supplies raw facts from which part of it can be derived: a start, a successful completion, and the stored result. A production recovery layer adds authoritative confirmation, ambiguity resolution, compensation and ownership transitions.

Do not overload COMPLETED. Changing it to mean “the answer was returned, every side effect was verified, and the user received the result” creates one state that almost no real workflow can reach atomically. Keep the loop outcome narrow. Add a business-resolution record with counts such as 14 requested / 11 completed / 2 failed / 1 ambiguous, plus the identifiers of non-terminal operations.

Make terminality an invariant. A parent task may be archived only when every child operation is in completed, compensated, or abandoned-with-owner. Ambiguous is deliberately non-terminal. A human accepting an escalation does not make the external state known; it changes who owns the uncertainty. Record both facts: operation still ambiguous, owner changed to the payments queue at 03:28.

Keep recovery decisions out of model prose. “I think the charge probably went through” is not a state transition. started → confirmed needs provider-idempotency evidence or an authoritative lookup result. confirmed → compensated needs a recorded inverse result. ambiguous → abandoned-with-owner needs an explicit human acceptance. The model may propose which transition to attempt, but the harness validates the evidence that permits it.

The reference loop already illustrates one useful separation. A RetryableError exhausted inside _invoke_one becomes a structured error observation for the model, rather than mechanically ending every run. The model can choose another route or report the gap. A circuit that is already open can instead terminate the outer run as TOOL_FAILURE. Those are different control decisions: one failed operation may be recoverable at the plan level; a known-degraded dependency should fail fast rather than consume the run budget.

Recovery workers should therefore operate on logical-operation records, not simply rerun whole agent goals. Replaying the goal asks the model to reconstruct which work remains and risks repeating completed actions under slightly changed arguments. Resuming the same run preserves completed-call evidence, but the production recovery pass should first isolate incomplete starts, resolve them, and only then return control to the planner. This is the operational counterpart of Chapter 6’s task graph: retry the failed node under its original identity, not the entire graph as a fresh request.

Every non-terminal operation needs a next action and an owner. “Ambiguous” without a recovery command is a label, not a mechanism. Store next_action=query_payment_by_reference, not_before=03:30, attempts_remaining=2, and owner=recovery-worker. When automatic evidence is exhausted, move ownership to a named human queue and attach the trace. A generic “operations” owner creates the same dead-letter queue nobody reads; ownership must resolve to a team with an alert and response objective.

A fresh supervisor run gets a new run ID, so the reference idempotency key changes even when the call arguments are identical. That is correct isolation, but the new run is not deduplicated against the first. Recovery must preserve the original logical-operation identity or pass a provider-level business key that outlives the run. Replaying the whole goal is a side-effect decision, not harmless queue hygiene.

There is a limit to this machinery. A read-only search with no durable effect, a cheap deterministic lookup, or a disposable draft may be safer to recompute than to persist as a state machine. The recovery contract should follow consequence. Use full operation state for mutations and expensive or externally visible work; use simple retry for bounded, repeatable reads. Durability everywhere adds writes and cleanup obligations. Durability at the side-effect boundary removes ambiguity where it can cost real money.

22.12Measuring recovery#

Failure rate by class. The distribution matters more than the total. Rising INVALID means a tool interface problem (Chapter 13). Rising SEMANTIC means a retrieval or planning problem. Rising TRANSIENT means a dependency problem. One number tells you nothing; the split tells you where to work.

Recovery rate. Of runs that hit a failure, what fraction still completed? This is the number that describes resilience, and it is usually much lower than teams assume because a single bad tool result often ends a run through the loop detector.

Retry amplification. Total attempts divided by logical operations. Compare it with the expected value for your failure rate and retry policy, then with downstream capacity. No universal amplification threshold is safe for every workload.

Compensation success rate. Of unwinds attempted, how many completed fully? Every partial unwind is a state nobody designed, and the count should be near zero and monitored.

Poison pill rate. Tasks dead-lettered per thousand. Rising means either a new bug class or a queue that is retrying things it should not.

Two refinements keep that run-level recovery rate from carrying meanings it cannot support.

Operation recovery rate. Of logical operations that encountered at least one classified failure, what fraction eventually reached their intended confirmed state? This measures mechanism-level resilience.

Affected-run useful-outcome rate. Of runs that contained at least one failed operation, what fraction produced either a complete result or an explicitly degraded result that met a predefined usefulness criterion? This measures workflow value after failure, not operation recovery.

The two rates need separate denominators and clocks or they will flatter you. For operation recovery, take the cohort of logical operations whose first attempt failed. If 10,000 logical operations include 100 affected operations and 70 eventually reach their intended confirmed state, operation_recovery_rate = 70 ÷ 100 = 70%, not (9,900 + 70) ÷ 10,000 = 99.7%. The second number mostly measures how often nothing went wrong.

For workflow usefulness, take a different cohort: runs containing at least one affected operation. In an illustrative 100 affected runs, 62 produce complete results, 18 produce explicit partial results that satisfy the product’s predefined minimum, and 20 remain unusable or unresolved. The full-completion rate is 62%; affected_run_useful_outcome_rate = (62 + 18) ÷ 100 = 80%. Do not call the eighteen partial runs “recovered operations”. One run can contain many operations, and one partial result may coexist with both recovered and unresolved operations.

Then measure time to a stable outcome, not merely time to the first retry success. Record failure_detected_at, last_attempt_at, state_resolved_at, and user_notified_at. The difference from detection to state resolution is operation recovery time; the difference from resolution to notification exposes a separate communication delay. For ambiguous side effects, the recovery clock stops only when the system of record is reconciled or the operation is explicitly abandoned under policy — not when the worker restarts or a human merely accepts ownership. Ownership transfer has its own handoff clock while the operation remains unresolved.

Finally, cohort each rate at its own level. Split operation_recovery_rate by dependency, failure class, tool and recovery mode; split affected_run_useful_outcome_rate by workflow, tenant and channel. A fleet-wide operation recovery rate can hide a provider-idempotent payment tool at 99% and an opaque messaging tool at 20%. An affected-run usefulness aggregate can hide one workflow whose partial results are never actionable. The aggregate tells you whether the month improved. The cohort tells you what to fix on Monday.

Part VI is complete: the agent stops when it should, resists an adversary, and fails without compounding. What it cannot yet do is tell you whether any of it is working — which is Part VII, and where most agent programmes actually stall.