Durable State & Resumption

Reference Agent had been working for three hours and forty minutes on a quarterly reconciliation when the pod was evicted. It had read fourteen ledgers, issued nine corrections, filed two exception tickets, and was drafting the summary. The eviction was routine — a node drain during a cluster upgrade, the kind of event that happens weekly and that no one gets paged for. What happened next was not routine. The supervisor restarted the task from the beginning. It re-read the fourteen ledgers, which was merely wasteful, and re-issued the nine corrections, which was not.

The eviction exposed a recovery design that restarted the transcript without recording which corrections had already taken effect.

5.1The transcript is not the state#

An in-memory list of messages loses the run when its process dies. Persisting the transcript alone also leaves recovery gaps.

The problem is not that memory is volatile — that much is obvious, and the naive fix is to serialise the list to disk. The problem is that the transcript is a lossy projection of what happened. Look at what a message list does not contain:

• that the model requested a tool call which policy denied, and why
• that a human was asked for approval, and how long they took to answer
• that a guardrail fired at WARN severity and the content was redacted
• that a tool call was retried three times before succeeding
• that the context was compacted, and what was dropped
• that the run was resumed at 14:03 after a crash at 11:58

These facts matter during recovery and audit. Some can appear in messages, but an ordinary transcript does not capture them completely or consistently. Persist explicit records for decisions, attempts and outcomes.

This is event sourcing, a pattern that predates agents by decades, and the reason it fits agents unusually well is that agents have all four of the properties that make it pay: long-running operations, external side effects, non-deterministic behaviour that must be explained after the fact, and a hard requirement not to repeat actions on retry.

OpenHands stores events as individual files and updates base state separately. The event files contain the same trajectory data as its older monolithic format; the change improves storage access, not external-effect recovery. The teaching harness instead reconstructs state from an authoritative log.¹

5.2Events, and what belongs in them#

An event is an immutable, ordered, timestamped fact about something that happened. The design work is choosing the vocabulary, and the useful discipline is that an event records a decision or an occurrence, never a computed state.

class EventType(str, Enum):
    RUN_STARTED         = "run_started"
    USER_MESSAGE        = "user_message"
    MODEL_TURN          = "model_turn"           # text and/or tool calls
    TOOL_CALL_STARTED   = "tool_call_started"
    TOOL_CALL_COMPLETED = "tool_call_completed"
    APPROVAL_REQUESTED  = "approval_requested"
    APPROVAL_DECIDED    = "approval_decided"
    GUARDRAIL_TRIGGERED = "guardrail_triggered"
    CONTEXT_COMPACTED   = "context_compacted"
    MEMORY_WRITTEN      = "memory_written"
    RUN_FINISHED        = "run_finished"

These eleven types are the reference vocabulary, not a completeness standard. Note what is present that a transcript lacks: approvals as a request/decision pair, so latency to human is measurable; guardrail trips as first-class facts; compaction as an event, so a replay reconstructs the same context the live run had rather than the full history the live run had already discarded.

Note also the split between TOOL_CALL_STARTED and TOOL_CALL_COMPLETED. A single “tool call” event would be simpler and would lose the most important case: the call that started and never finished. A start without a completion can mean an active call, a timeout, or a crash before or after the remote effect, which is exactly the situation resumption has to handle carefully, and you cannot detect it if you only record completions.

The write path should be boring:

def append(self, type: EventType, **payload) -> Event:
    self._seq += 1
    ev = Event(seq=self._seq, type=type, payload=payload)
    with open(self.path, "a") as f:
        f.write(ev.to_json() + "\n")
        f.flush()
        if self._fsync:
            os.fsync(f.fileno())      # durability is the whole point
    return ev

The sketch is JSON Lines on local disk with optional fsync. Set and verify the actual durability policy; a local append also needs writer coordination, partial-tail handling and durable storage that survives the worker. The fsync is the honest default for something whose entire purpose is surviving an ungraceful stop; if its cost is unacceptable, batch it deliberately and accept that a resumed run may lose the last few events. This is safe only if the receiver can deduplicate the lost operations or reconciliation can establish their outcome. That is the trade you are allowed to make. What you are not allowed to do is skip the durability and hope.

5.3State as a fold#

If the log is authoritative, state is a pure function of it. This makes replay the most important function in the harness, and it should be small enough to read in one sitting, because it is the definition of what your agent’s state means.

@classmethod
def replay(cls, log: EventLog) -> "RunState":
    st = cls()
    for ev in log.replay():
        p = ev.payload
        if ev.type is EventType.USER_MESSAGE:
            st.messages.append(Message("user", p["content"], trust="user"))
        elif ev.type is EventType.MODEL_TURN:
            calls = [ToolCall(**c) for c in p.get("tool_calls", [])]
            st.messages.append(Message("assistant", p.get("text", ""), tool_calls=calls))
            st.turns += 1
            st.usage = st.usage + Usage(**p.get("usage", {}))
            st.cost_usd += p.get("cost_usd", 0.0)
        elif ev.type is EventType.TOOL_CALL_COMPLETED:
            st.messages.append(Message("tool", tool_result=ToolResult(call_id=p["call_id"], name=p["name"],
                content=p["content"], is_error=p.get("is_error", False)), trust="tool"))
            st.trajectory.append(p.get("signature", p["name"]))
            if key := p.get("idempotency_key"):
                st.completed_calls[key] = p["content"]      # <- the load-bearing line
        elif ev.type is EventType.CONTEXT_COMPACTED:
            keep = p.get("kept", 0)
            tail = st.messages[-keep:] if keep else []
            st.messages = (st.messages[:1]
                           + [Message("user", p["summary"], trust="untrusted")]
                           + tail)
            st.compactions += 1
    return st

Two properties are worth verifying by test rather than by inspection. Replay is deterministic: folding the same log twice gives identical state. Replay accounts for the vocabulary: every event type changes state or is explicitly classified as record-only. The excerpt below is abbreviated; the reference’s remaining gaps are described in the compatibility section. Both are easy tests, and both catch a class of bug that is otherwise found only when a resumed run behaves subtly differently from the original.

The compaction branch deserves attention because it is the one people forget. A live run at turn 40 has already discarded turns 5 through 30, replacing them with a summary. If replay reconstructs the full history instead, the resumed run has a different context than the run it is continuing — more tokens, different content, potentially different behaviour. The fix is that compaction is an event with the summary in its payload, and replay applies it exactly as the live run did.

Share the start and resume path#

The teaching harness uses one entry point to construct state for both new and resumed runs.

def open_run(dir_path: str, run_id: str | None = None) -> tuple[EventLog, RunState]:
    rid = run_id or new_id("run")
    log = EventLog(os.path.join(dir_path, f"{rid}.jsonl"))
    state = RunState.replay(log)          # empty log -> empty state
    if len(log) == 0:
        log.append(EventType.RUN_STARTED, run_id=rid)
    return log, state

Starting a run and resuming one are the same call. A fresh run replays an empty log and gets empty state; a resumed run replays a populated log and gets populated state. There is no if resuming: anywhere in the loop.

Sharing transitions reduces the risk that new-run behavior and recovery diverge. Separate entry points can also be correct if they use compatible state transitions and are tested against the same log prefixes. One entry point is a design aid, not proof of recovery correctness.

5.4Replay is an executable compatibility contract#

Calling replay a fold can make it sound like plumbing. It is closer to a database schema and an API contract combined: every event ever written must retain a precise meaning under every version that may resume it. The loop can change, prompts can change, and tools can change. The old log cannot. If version 14 reads a run created by version 11, its fold must either reproduce the state version 11 had or reject the run explicitly. Quietly producing a third interpretation is corruption. This is a production requirement; the reference Reference Agent demonstrates the fold, not versioning.

The reference implementation makes three semantics explicit:

# Selected branches from the replay fold; other event types omitted.
if ev.type is EventType.TOOL_CALL_COMPLETED:
    res = ToolResult(call_id=p["call_id"], name=p["name"],
                     content=p["content"], is_error=p.get("is_error", False))
    st.messages.append(Message("tool", tool_result=res, trust="tool"))
    st.trajectory.append(p.get("signature", p["name"]))
    if key := p.get("idempotency_key"):
        st.completed_calls[key] = p["content"]
elif ev.type is EventType.CONTEXT_COMPACTED:
    keep = p.get("kept", 0)
    summary = p.get("summary", "")
    head = st.messages[:1]
    tail = st.messages[-keep:] if keep else []
    st.messages = head + [Message("user", summary, trust="untrusted")] + tail
    st.compactions += 1

A completion is both an observation and a deduplication fact. Replay recreates the tool message that the model saw, then records the logical action only when the payload carries an idempotency key. In the live loop, failed tool results are emitted with idempotency_key=None. That means an ordinary, proven failure may be attempted again; it does not occupy the identity of a successful action forever. This is the right asymmetry, but it places a hard requirement on the adapter: it must not report an uncertain remote outcome as an ordinary failure. A timeout after a server accepted a request is not evidence that the effect failed.

Compaction replaces the discarded span with the exact summary used by the live run. Replay must restore that summary so the resumed model receives the same decision context.

The reference has two exact compatibility behaviours. Event.from_json() constructs EventType(d["type"]), so an unknown event string raises during parsing. But once the enum knows a type, RunState.replay() has no exhaustive guard: a missing branch silently falls through. Starts, approvals, guardrail events and memory writes already do this; the fold neither marks them record-only nor derives state from them.

A production compatibility extension must classify every event as state-changing, with a fold rule, or record-only, with a deliberate no-op, then raise on a type absent from both sets. Record-only starts and approvals can still feed separate recovery or audit projections. The goal is to stop a new enum member disappearing because somebody forgot a branch.

The reference has no schema version or upcaster. Add both before promising cross-release resumption. Then add a default for a compatible optional field, upcast an old representation of the same fact, or require migration when meaning changed. Changing kept from “messages retained at the tail” to “turns retained” requires migration; new code could otherwise reconstruct the wrong context from valid values.

Test the versioned reader end to end. Run Reference Agent against a scripted model, retain state after every event, replay every log prefix and compare the derived state at that sequence. Keep fixtures from every released schema version. Logs emitted only by the current writer prove that the reader agrees with itself — not that tomorrow’s reader can recover yesterday’s work.

5.5Idempotency: the part that actually matters#

Durable records support audit and recovery. Idempotency addresses a separate requirement: preventing repeated business effects.

An agent that resumes must not repeat side effects. Re-reading a file is usually safe, though it still consumes time and may incur cost. Re-issuing nine ledger corrections is an incident. And the naive protections do not work: you cannot ask “did I already do this?” of most APIs, and you cannot infer it from the transcript, because the transcript of a crashed run ends before the response that would have told you.

The mechanism is an idempotency key: a stable identifier for a logical action, computed from the run and the action’s semantic identity, and recorded alongside its result.

def idempotency_key(run_id: str, call: ToolCall) -> str:
    h = hashlib.sha256(f"{run_id}|{call.signature()}".encode()).hexdigest()
    return h[:24]

It includes the run id in this reference. That isolates its cache, but a new run does not necessarily mean a new business operation. Production identity should bind tenant, operation id and payload, survive retries across workers or runs, and distinguish two intentional repeats inside one run. Derive it when the operation is accepted, not from a fresh model call id.

It excludes the tool call id. Call ids are fresh on every model turn, so including one would mean a retry of the same logical action computes a different key and executes twice — which is the exact bug the mechanism exists to prevent.

It uses a canonical signature. ToolCall.signature() serialises arguments with sorted keys, so {"amount": 10, "acct": "x"} and {"acct": "x", "amount": 10} are the same action. Without canonicalisation, the model’s incidental key ordering becomes a correctness variable.

The check on the execution path is then trivial:

def _invoke_one(self, call: ToolCall, tool: Tool) -> ToolResult:
    key = idempotency_key(self.state.run_id, call)
    if key in self.state.completed_calls:
        # Replayed after a crash, or the model asked twice. Either way the
        # side effect already happened; returning the stored result is the
        # only safe answer.
        return ToolResult(call_id=call.id, name=call.name,
                          content=self.state.completed_calls[key])
    ...

The regression test asserts that resumption does not repeat a recorded effect:

def test_a_resumed_run_does_not_repeat_a_side_effect(toolset, policy, tmp_path, counter):
    runs = str(tmp_path / "runs")
    first = make_loop(toolset, policy, [charge_script], log_dir=runs)
    r1 = first.run("charge seven dollars")
    assert counter["charges"] == 1

    # The process dies; a supervisor resumes the same run id with a fresh loop.
    second = make_loop(toolset, policy, [charge_script], log_dir=runs)
    second.run("charge seven dollars", run_id=r1.run_id)
    assert counter["charges"] == 1, "resumption must not re-issue the charge"

The gap that idempotency keys cannot close#

Be precise about what this guarantees, because over-claiming here is dangerous. The key is recorded in TOOL_CALL_COMPLETED, after the side effect. So there is a window: the call executed, the process died before the completion event was durable, and the log shows TOOL_CALL_STARTED with no completion. On resume, the key is absent, and the action will run again.

You cannot eliminate this window from inside the agent. What you can do is one of four things, and the choice should be conscious:

The started-without-completed pair earns its keep here. Because the log distinguishes “I was about to” from “I did,” a resumed run can detect the ambiguous case and handle it deliberately, instead of blundering into it. A log that only records completions cannot tell the difference between an action that never started and one that may have completed.

5.6Uncertainty needs its own recovery state#

Consider a second illustrative incident, separate from the earlier 41-event timeline. At 15:22:03, Reference Agent appends event 842: TOOL_CALL_STARTED for correction C-771, amount £840, key 91e…. The ledger accepts it; one second later the worker dies. The log contains an unmatched start, but the reference RunState.replay() ignores it and exposes no mismatch.

The easy mistake is to call this a failed tool call. It is not. A reported failure may still accompany partial effects. A proven no-effect failure permits a different retry policy from an uncertain outcome. Conflating them turns a transport symptom into a business decision. A retry policy may retry failure. It must route uncertainty through a recovery policy.

The first recovery step is a required production extension. Build a projection that pairs starts and completions by logical key; unmatched starts form uncertain_calls. The reference derives no such set, so it resumes without a completed-call entry and may invoke the action if the model proposes it again. Run this projection before the model, whose answer cannot create evidence about the remote system.

The production adapter must then add an authoritative lookup for correction C-771; no such reconciliation hook exists in the reference loop. Four outcomes are possible:

The equality check must cover the semantic effect, not merely the identifier. If C-771 exists for £480 rather than £840, deduplication has prevented neither under-correction nor misapplication. Tenant, destination, amount, currency and accounting period are part of the proof because they are part of what the action means. Tool design in Chapter 13 should expose these fields as structured values so recovery does not parse them out of prose.

Provider-side idempotency is another required extension. Reference _invoke_one() logs the local key, then calls tool.invoke(call) unchanged; neither ToolCall nor this path transmits it. A production adapter must send the key, and the receiver must durably return the original result on repetition. A lost response may still leave Reference Agent uncertain, but retrying cannot repeat the effect. A local-only key protects recorded completions and does nothing for event 842.

When neither idempotency nor authoritative lookup exists, policy must use reversibility and consequence. Re-reading a dashboard is safe to retry. Creating a low-value draft with a reliable delete operation may be retried and later compensated. Sending an irreversible notification to a regulator is neither. Pause it for a person, showing the intended payload, the start timestamp, the logical key and the last transport error. “Try again?” is an inadequate approval question because it hides the possibility that “again” means “twice”.

Recovery itself must leave events. Record when the call became uncertain, every reconciliation attempt, the evidence returned, the rule that chose retry or escalation, and the final disposition. Otherwise the original action is durable but the most consequential decision in its lifecycle is not. In a production vocabulary, you may add TOOL_CALL_UNCERTAIN and TOOL_CALL_RECONCILED; alternatively, enrich the completion and approval events with a recovery source and the original key. The names matter less than making the transition replayable.

Measure this state as a queue, not just an error counter. Track uncertain calls opened, reconciled automatically, escalated, and still open; report the age of the oldest by risk class. The critical number for irreversible tools is unresolved uncertainty past its recovery deadline. A tiny percentage can still represent one duplicated payment or one missing regulatory notice. Percentages tell you engineering capacity; named overdue actions tell you where to intervene.

Under this production extension, an exact ledger match lets the recovery layer append a reconciled completion carrying the original key; the existing fold then adds it to completed_calls. The reference cannot perform that lookup or append the reconciliation itself. Recovery neither asks the model to recreate the correction nor manufactures success from a start. It turns a process crash into one bounded question about one external effect.

Resumption also needs one owner#

A process crash often causes two supervisors to make the same reasonable decision: resume the run. If both replay the same prefix, both believe they own event 843, and both continue. Provider idempotency may stop a duplicated correction, but it cannot stop two model turns, two approval requests or two conflicting terminal outcomes from entering the log. Durable state requires single-writer ownership as well as durable bytes.

The local EventLog in the reference Reference Agent is intentionally a single-process implementation. Its _seq counter is initialised by scanning the file, then incremented in memory. That is sufficient while one loop owns one run. Moving the same interface to a shared database without adding concurrency control is not a scale-out design: two workers can read sequence 842, each assign 843 and create a history with no unambiguous order.

Use a renewable run lease with a fencing token. Worker A acquires epoch 41 and may append while the lease is valid. After A disappears, worker B acquires epoch 42. Every append includes the epoch and an expected previous sequence; the store accepts it only if both match current ownership. If A wakes after a long pause and tries to append with epoch 41, the store rejects the write even if A still believes its lease is valid. The monotonically increasing token is the fence; an expiry timestamp alone is only a promise made by a clock and a paused process.

In the production extension, the external action identity must survive this handover. Worker B resumes the same run id and therefore computes the same local idempotency key for the same logical call; the extended adapter transmits it to the receiver. If A reached the receiver before losing its lease, B’s retry returns the receiver’s stored result. If B discovers that A already appended the completion, the existing replay logic returns the local stored result without contacting the receiver. Ownership orders the log; receiver-side idempotency orders the effect. You need both because they protect different boundaries.

Do not represent ownership with a permanent running=true flag. A dead owner can never clear it. A lease expires, a new worker takes over, and the fencing token prevents the old worker from returning from the dead. Record acquisition, renewal, takeover and fencing failures in operational telemetry, but avoid putting periodic lease heartbeats into the semantic event stream unless the fold genuinely uses them. Operational coordination and business history have different retention and query patterns.

Test the race with a barrier, not a sequence of friendly calls. Start two workers on the same log prefix, release them together, and assert that one acquires the next epoch, one is fenced, sequence numbers remain unique, and at most one terminal outcome is durable. Then pause the first worker after its remote effect but before its completion append, transfer ownership, and exercise the uncertain-call protocol. That test combines the two failures production will combine for you.

5.7Where to put the log#

Choose storage for measured concurrency, recovery and operational requirements. Local JSON Lines is a useful teaching starting point and can remain sufficient for bounded single-owner workloads. Four common options follow.

Append-only files. One file per run. Trivially debuggable — an incident starts with cat and jq. Simple for a single-owner agent. Multiple writers, retention and cross-run queries require additional locking, lifecycle policy and indexing. OpenHands uses individual event files plus base state; Hermes migrated session history from JSONL to SQLite with full-text indexes.¹²

Embedded database. SQLite, typically in WAL mode. Keeps single-node simplicity, adds transactions, indexes and queries. Hermes documents exactly this trade — a SQLite state store with a one-second busy timeout plus application-level retry with jitter, which is the tell-tale of a real concurrency problem solved pragmatically rather than architecturally.²

Shared database. Postgres or equivalent, with the event log as an append-only table partitioned by run. Often useful for multi-host workers and central queries. Tenant isolation, retention and residency still need explicit configuration and application rules (Chapter 30). The cost is a network hop on every event, so transaction commit and replication durability must be configured alongside the latency budget.

Durable execution engine. Temporal, Restate, DBOS and similar systems persist workflow progress and provide recovery machinery, with different execution and retry semantics. In Temporal, deterministic workflow replay is distinct from Activities that perform external work. The fit with agents is genuinely good — an agent run is a long-running workflow with non-deterministic decisions and external activities.

The reason not to reach for a durable execution engine on day one is that it imposes a programming model. Determinism constraints on workflow code, activities as separately-registered functions, versioning rules for in-flight workflows: all reasonable, all a real learning cost, and all wasted if your agent turns out to run for eleven seconds. The reason to reach for it on day two hundred is that you will otherwise rebuild a worse version of it, and the rebuild will be discovered during an incident.

SQLite supports concurrent readers and serialised writers; it is not inherently limited to one process. Avoid treating a shared network filesystem as a substitute for a database server. SQLite deployment guidance, checked 8 September 2026.

Durable execution journals workflow progress, but its semantics vary by platform. For Temporal, keep nondeterministic model and external calls in Activities, account for retries, and version changes to replayed workflow logic. Temporal workflow execution, checked 8 September 2026.

Receiver-side idempotency has limits. Stripe, for example, documents replay of the first stored response and a key-retention window; a key reused after pruning can create a new request. Record expiry, parameter binding and reconciliation requirements per tool. Stripe idempotent requests, checked 8 September 2026.

5.8Checkpoint only after measuring replay#

A snapshot is justified when its expected recovery saving exceeds its write and verification cost. Choose the interval from measured replay economics, not from a round number that feels durable. The inputs are small: events per run, replay time per event, expected resumptions per run, snapshot write time and the maximum recovery latency your operator or caller can tolerate.

Take an illustrative Reference Agent reconciliation with 4,000 events. A replay benchmark on the intended storage path measures 2 milliseconds per event. If six recoveries are distributed roughly uniformly through the run, a full replay processes 2,000 events on average:

These figures are illustrative, and the uniform-crash assumption is explicit. With N events, interval K, replay cost c, snapshot cost s and expected recoveries R, the approximate full-replay cost is R × N/2 × c. Snapshotting costs ceil(N/K) × s + R × K/2 × c. Here the saving per recovery is 3.5 seconds and the eight writes cost 0.96 seconds, so the break-even recovery rate is 0.96 ÷ 3.5 ≈ 0.27 recoveries per run. If fewer than roughly one run in four resumes and four seconds meets the recovery objective, snapshots may add more machinery than value.

The arithmetic also exposes a constraint averages hide. With a 500-event interval, worst-case tail replay is 499 events, or just under one second at the measured rate. If the service-level objective says a voice session must recover within 300 milliseconds, this interval fails even though its average economics are attractive. Solve the maximum first: at 2 milliseconds per event, a 300-millisecond replay budget permits at most 150 tail events, before log-open and snapshot-load time. Latency-sensitive resumption may therefore need a much shorter interval than batch work on the same store.

Do not optimise only the happy snapshot read. Instrument events_replayed, replay_duration_ms, snapshot_load_ms, snapshot_write_ms, snapshot_seq_gap and full_replay_equivalence_failures. Plot replay duration against event count; a straight line supports the simple model, while a bend often points to payload growth, repeated decoding or an accidental quadratic fold. Segment by event-schema version because an upcaster can dominate the cost of old runs.

The safety metric is equivalence, not speed. On a sample of runs, compute state two ways: load snapshot at sequence S and fold the tail, then fold the complete log from event 1. Compare every durable field, including completed-call keys, message trust labels, usage, compaction count, terminal outcome and any recovery state. One mismatch means the snapshot path can resume a run that the authoritative log would not have produced. Fall back to full replay, quarantine the snapshot version and block wider rollout until the fold agrees.

The snapshot write has its own crash window. Persist the snapshot body with its schema version, last included sequence and a digest, make those bytes durable, and only then publish the pointer that says it is current. On a filesystem this normally means writing a new file and atomically renaming it; in a database it means updating the snapshot and pointer in one transaction. Publishing the pointer first allows a crash to advertise bytes that are partial or absent. Overwriting the only previous snapshot removes your fallback at exactly the wrong moment.

On load, validate the digest and confirm that the tail begins after the recorded sequence. A corrupt, unknown-version or discontinuous snapshot is a cache miss, not a lost run: discard it and replay the log from event 1. Keep enough old snapshots to diagnose equivalence failures, but never require one for correctness. This is also why snapshots should not contain secrets or derived fields the event store itself is not authorised to retain; a performance cache does not create a new data-governance exemption (Chapter 30).

Snapshot frequency can vary by run without changing semantics. A short research run may never snapshot. A long reconciliation may checkpoint every 500 events. A voice run may checkpoint more aggressively because the user experiences recovery delay directly (Chapter 18). The log remains authoritative in all three cases; the snapshot is only a cached partial fold. That sentence is the boundary: if deleting every snapshot prevents recovery, you have accidentally promoted the cache into a second source of truth.

5.9What resumption cannot fix#

Resumption does not fix a bad plan. An agent that spent three hours going the wrong way will, on resume, continue going the wrong way with full fidelity. Durability preserves progress; it does not evaluate it. The counterweight is the replanning discussed in Chapter 6 and the budgets in Chapter 20.

The world may have moved. A run resumed three hours later is reasoning over observations that are three hours stale. The file it read has changed; the ticket it filed has been closed; the deploy it was waiting on has finished and been rolled back. This is the genuinely hard problem in long-horizon agents, and the only real mitigation is to make freshness explicit: timestamp observations, and on resume re-verify the ones the next action depends on. Chapter 11 covers freshness for retrieved content; the same reasoning applies to any observation.

Replay is not free. A run with 4,000 events replays 4,000 events, and if that takes eight seconds you have made every resume eight seconds slower. The standard answer is a periodic snapshot — fold to event N, persist the state, replay only the tail thereafter. Do this when measurement says to, not before; the snapshot is a second representation of state and therefore a second thing that can be wrong.