Anatomy of an Agent Harness

Consider an illustrative incident. At 02:14, a logistics customer’s duty engineer calls: “Checkout is timing out. Can you look?” Within nineteen seconds, Reference Agent transcribes the request and verifies the caller through an authenticated session. It loads tenant context, reads alerts and a dashboard, then asks a clarifying question. Ninety seconds later, it proposes a remediation.

An authorized operator approves the specific action. Reference Agent executes it, verifies recovery and writes an incident note. The model processes text throughout. The harness owns authentication, execution, verification and records.

3.1The harness is the product#

The harness gives a model durable state, permissions, budgets, provenance-labelled context and traces. These controls make execution inspectable and recoverable.

Public systems expose useful examples of these concerns, without proving that there is one universal architecture. OpenHands separates conversation events from saved base state; Hermes uses a SQLite session store; Aider records code edits in git; and Devin starts sessions from an environment snapshot. These preserve different kinds of state and provide different recovery guarantees.¹³⁶⁷

3.2The eight subsystems#

1. Ingress normalises however the work arrives into one internal representation of a task. Reference Agent is reachable by voice, by chat, by webhook and by schedule, and the loop must not know which. This is more consequential than it sounds: an ingress layer that leaks its modality into the core produces a codebase where the voice path and the chat path drift apart and only one of them gets the new guardrail. Hermes Agent’s documentation makes the same point structurally, describing a “platform-agnostic core” where one agent object serves the CLI, a messaging gateway across many platform adapters, an API server, and a batch runner.² Chapter 18 is the voice-specific version of this problem, where a 300-millisecond latency budget makes the normalisation genuinely hard.

2. The orchestration loop is the control flow: how a turn is structured, how work is decomposed, when to delegate to a sub-agent, when to stop, and how all of that survives a process death. The single most important design property is that it is small — small enough that an on-call engineer can read it at 3 a.m. and predict what it will do.

3. Context assembly decides what actually goes into the prompt on this specific turn, out of everything that could. The reason it is a separate subsystem rather than a function inside the loop is that its policy is the highest-leverage tuning surface in the whole system, and it needs to be inspectable, testable and versioned on its own.

4. Knowledge is what persists across turns and across runs: four kinds of memory, retrieval over documents, and structured relationships in a graph. Note the direction of the arrow: knowledge feeds context assembly, and context assembly decides what to use. A system where retrieval writes directly into the prompt has no budget and no arbitration, and will behave erratically at exactly the moment its knowledge base grows.

5. The action layer is everything the agent can do: the tool registry and its schemas, the protocol by which external tools are discovered, the sandbox that code runs in, and the multimodal machinery for content that is not text.

6. Policy and guardrails decide whether a proposed action may proceed, what the run’s ceilings are, and when a human must be involved.

7. Observability receives events from every subsystem. Instrument from the first commit so the first incident is diagnosable.

8. The improvement loop is the outer cycle. It consumes the record the inner cycle produced — traces, outcomes, failures, trajectories — and produces changes: better prompts, new evaluation cases, verified skills, occasionally fine-tuned weights. It runs on a different clock and, in most organisations, is owned by different people.

3.3Four boundaries that must not blur#

The model proposes; the policy decides#

A tool call emitted by a model is a request, never an instruction. Nothing in the model’s output — no argument, no assertion of urgency, no claim of prior authorisation — may widen what the policy permits. This sounds obvious and is violated constantly, because the natural way to write the loop is result = tools[name](**args), and that single line has already conceded the boundary.

The practical test is this: can any string a model could possibly emit cause an action the policy would otherwise deny? If yes, you do not have a permission system, you have a naming convention. Chapter 21 is about adversaries who exploit exactly this, and Chapter 15 is about building the boundary properly.

State is a fold over an immutable log#

The transcript in memory is a view. The authoritative record is an append-only sequence of events, and the in-memory state is a pure function of it. This design supports resumption, audit and replay. External side-effect idempotency additionally requires receiver support or a reconciliation protocol.

Transactional snapshots and durable execution are also valid designs. The risk is untested divergence between live execution and recovery, not the presence of methods named save() and load(). OpenHands, for example, stores individual event files alongside a separately updated base_state.json; it is not evidence that every field must be reconstructed solely from events.¹ Chapter 5 builds it.

The interface is not the implementation#

The harness depends on narrow interfaces for the model, summariser and trace exporter. Keeping vendor SDKs behind adapters enables offline tests and provider failover.

The version of this that teams get wrong is subtler than “we imported the SDK.” It is that provider-specific concepts leak into the core: a message shape that only one provider produces, a token-counting assumption, a streaming event vocabulary. Chapter 2 covered which model properties are genuinely universal; the boundary here is about keeping the rest out.

Risk metadata is for the harness, not the model#

Every tool has two descriptions. One is for the model: name, purpose, parameters, when to use it. The other is for the harness: does this mutate state, does it reach the network, is it reversible, what does it cost. The second must never be handed to the model as a safety mechanism.

A warning in a prompt holds only as often as the model complies. Put danger in the policy, where it is enforced, and keep the tool description focused on correct use.

3.4The harness as code#

The reagent reference harness implements these boundaries in twelve Python modules, with standard-library-only runtime dependencies.

reagent/
  types.py       the values that cross every boundary; RunOutcome
  model.py       the ModelClient protocol + offline test doubles
  tools.py       tool definition, JSON Schema generation, registry, selection
  policy.py      the permission engine — ALLOW / ASK / DENY
  guardrails.py  input, action and output guardrails + budgets
  state.py       the append-only event log that state is folded from
  trace.py       spans shaped for OpenTelemetry GenAI conventions
  retry.py       backoff, circuit breaking, compensating actions
  memory.py      context assembly, compaction, four kinds of memory
  skills.py      procedural memory the agent writes for itself
  evals.py       outcome, trajectory, efficiency and rubric scoring
  loop.py        the loop that calls all of the above — and nothing else

Each module depends only on the ones above it, which is not an accident of tidiness but a consequence of the boundaries. policy.py imports from tools.py because a permission decision needs a tool’s risk metadata; tools.py imports nothing from policy.py, because a tool must not be able to describe itself as permitted.

The first boundary — proposal versus decision — shows up as a three-valued verdict rather than a boolean:

class Decision(str, Enum):
    ALLOW = "allow"
    ASK   = "ask"      # the third value is what makes an agent deployable
    DENY  = "deny"


@dataclass(frozen=True)
class Verdict:
    decision: Decision
    reason: str        # goes to the audit log; on DENY, back to the model
    rule: str = ""     # which rule decided, so the decision is reviewable

ASK is the entire reason an enterprise can run this. A two-valued permission system forces every action into “fully autonomous” or “impossible,” and since neither is acceptable for the interesting actions, teams with boolean permissions end up granting broad autonomy and hoping. The reason string matters nearly as much: a denial the model can read can guide it toward a permitted alternative, and a denial an auditor can read is evidence.

The second boundary — state as a fold — shows up as a replay classmethod that is the definition of state:

@dataclass
class RunState:
    messages: list[Message] = field(default_factory=list)
    turns: int = 0
    # idempotency key -> the result that was already produced.
    # This one field is why a resumed run re-reads the answer
    # instead of re-issuing the refund.
    completed_calls: dict[str, str] = field(default_factory=dict)

    @classmethod
    def replay(cls, log: EventLog) -> "RunState":
        st = cls()
        for ev in log.replay():
            ...  # one branch per event type; nothing else may mutate state
        return st

There is no save. There is no restore. Resuming a run is the same function call as starting one, with a run id supplied — which reduces the ways the resume path can drift, and why the test that asserts a resumed run does not re-issue a charge is a three-line test rather than an integration suite.

And the loop itself, which is the whole architecture in one function shape:

def _drive(self, inp, action_gr, out_gr):
    while True:
        if reason := self.tracker.exceeded():          # budgets first, always
            return RunOutcome.BUDGET_EXHAUSTED, "", reason
        self._maybe_compact()                          # keep the window sane
        response = self._call_model()                  # assembled context in
        if not response.message.tool_calls:            # a final answer
            if v := out_gr.worst(response.message.content):
                return RunOutcome.GUARDRAIL_STOP, redact(v), v.detail
            return RunOutcome.COMPLETED, response.message.content, ""
        if stop := self._execute_calls(               # vet, then act
                response.message.tool_calls, inp, action_gr):
            return stop

This abbreviated shape shows ordering, not complete production guarantees. Accumulated budgets are checked before the next turn; hard caps require reservation (Chapter 4). Every exit returns a named RunOutcome. Guardrails sit on the boundary in both directions. And _execute_calls returns either None, meaning “continue,” or a terminal triple — so a guardrail deep inside action vetting can end the run without an exception and without an ambiguous state.

3.5One tool call, all the way through#

The shortest useful review of a harness is to follow one proposed tool call from model output to durable observation. You are looking for a chain of custody: which component owns the call at each point, which facts it adds, and whether any shortcut can jump from proposal to effect. In Reference Agent, the critical path is split deliberately between _execute_calls and _invoke_one:

approved: list[tuple[ToolCall, Tool]] = []

for call in calls:
    tool = self.tools.get(call.name)
    if tool is None:
        self._observe(ToolResult.error(
            call, f"No tool named {call.name!r}. Available: "
                  f"{sorted(self.tools.tools)[:20]}"), executed=False)
        continue

    if v := action_gr.worst(call):
        ...

    verdict = self.policy.evaluate(call, tool, self.policy_ctx)
    ...
    approved.append((call, tool))

for call, tool, result in self._invoke_all(approved):
    ...
    self._observe(result)

The first design decision is that lookup happens before judgement. Policy evaluates a registered Tool, not just a model-supplied name. That object carries the harness-owned facts — mutates, external, reversible, cost and tags — which means the proposal cannot lie about its own risk class. An unknown tool becomes an observation marked executed=False; it is not an exception and certainly not a dynamic import. The model gets specific evidence it can use to repair the next turn, while the trajectory remains honest about what ran.

The second decision is total vetting before any execution. _execute_calls collects approved calls in one pass and invokes them only afterwards. Suppose a model proposes three calls together: read the incident, restart the service, then repeat a call whose signature trips an oscillation guardrail at HALT severity. If Reference Agent executed calls as it checked them, the read and restart could happen before the third call ended the run. Total vetting lets the terminal guardrail stop the batch before any effect. An ordinary DENY is different: it becomes a failed observation and the other approved calls may still run. Separating vetting from invocation does not make the calls transactional — Chapters 7 and 22 deal with that harder problem — but it prevents policy evaluation itself from creating an accidental partial execution.

The invocation path then establishes the identity of the logical action before touching the tool:

def _invoke_one(self, call: ToolCall, tool: Tool) -> ToolResult:
    key = idempotency_key(self.state.run_id, call)
    if key in self.state.completed_calls:
        return ToolResult(call_id=call.id, name=call.name,
                          content=self.state.completed_calls[key])

    self.tracker.record_call(call)
    self._emit(EventType.TOOL_CALL_STARTED, call_id=call.id, name=call.name,
               arguments=call.arguments, idempotency_key=key)
    ...
    result = retrier.call(lambda: tool.invoke(call))
    ...
    self._emit(EventType.TOOL_CALL_COMPLETED, call_id=call.id, name=call.name,
               content=result.content, is_error=result.is_error,
               signature=call.signature(),
               idempotency_key=None if result.is_error else key)

For the reference’s mutating-call cache, the key excludes the model’s call id and includes the run id. That distinction is easy to miss. A model or retry path can issue the same logical call under a new ephemeral id; arguments and tool name still produce the same signature, so Reference Agent recognises the duplicate within this run. A separate run is allowed to perform the same action because its key is different. This signature is only a teaching approximation: two intentionally identical mutations within one run collide, and one business operation retried under a new run id does not. Production actions need a durable business-operation identity and payload binding. Fresh reads must bypass this cache.

Events surround the effect; they do not merely describe it afterwards. TOOL_CALL_STARTED establishes intent, including the exact arguments and idempotency key. TOOL_CALL_COMPLETED establishes the observed result. The difference between those events is operational state: zero starts means nothing was attempted; one start plus one completion means the harness has evidence of an outcome; a start without a completion means the outcome is uncertain. Treating all three as “tool failed” destroys the information needed to recover safely.

The harness determines retry identity and mutability from registered metadata. These decisions must survive a different model, a misleading tool name and a process restart.

3.6The crash window you cannot wish away#

At 09:42, Reference Agent proposes issue_refund(order_id="O-1842", amount=125). Policy returns ASK; the finance operator approves; the harness appends TOOL_CALL_STARTED; and the payment service accepts the refund. Before the response reaches _invoke_one, the worker loses power. There is no TOOL_CALL_COMPLETED event.

The first on-call hypothesis is a model failure: perhaps it ignored the successful refund and asked again. Replay shows something more precise. The model never received a result. RunState.replay() sees the start event but adds nothing to completed_calls, because only a successful completion event does that. On resumption, the same signature is absent from the completed map, so local deduplication cannot prove that the refund happened. Re-executing may pay the customer twice; suppressing it may leave the original request unfinished. Both confident choices manufacture facts the harness does not possess.

This is the uncertainty gap between a local record and a remote effect. An append-only log gives you an exact account of what Reference Agent knew, but it cannot make two independent systems commit atomically. The gap starts after the remote service accepts the mutation and ends when the completion event is durable. Moving the append earlier merely inverts the error: you can record success for an effect that never occurred. Wrapping the call in a database transaction does nothing when the payment service is outside that database.

The reference implementation does not implement receiver-side idempotency. _invoke_one computes and records the key locally, then calls tool.invoke(call) unchanged; Tool.invoke passes only the call arguments. No key crosses the network boundary.

A production Reference Agent payment adapter must add that contract: send the logical key with the refund, require the receiver to store it beside the result, and return the original result when the key repeats. Only then can resumption retry safely, because the receiver — not Reference Agent’s local completed-call map — decides whether the effect is new. Chapter 22 calls this at-least-once delivery with idempotent handling. “Exactly once” is the business outcome you construct from those pieces, not a network delivery guarantee.

When a dependency does not support idempotency, you need a reconciliation branch rather than a blind retry. Query by a stable business identifier, compare the intended amount and recipient, then do one of three things: record the found result, execute because absence is authoritative, or escalate because the answer is ambiguous. For an irreversible tool that offers neither an idempotency key nor a reliable query, the honest policy is ASK on the first attempt and human reconciliation after an uncertain one. Autonomy cannot compensate for an unverifiable effect.

The incident generalises beyond payments. A ticket may be created while its HTTP response is lost. An email server may accept a message before the socket closes. A deployment controller may begin a rollout while the caller times out. Each case has the same shape: the harness can prove intent and perhaps prove completion, but there is a period in which the external world knows more than the event log. Your tool contract must state how that period is resolved.

The harness still benefits from its local completed-call map. It suppresses duplicate proposals after a recorded success, makes replay cheap, and preserves the original observation for the model. What it cannot do is close an external crash window by itself. That limit is not a flaw in event sourcing; it is the reason the action layer and state layer need an explicit protocol between them.

3.7Measure the seams, not the eloquence#

A harness dashboard should tell you whether boundaries held before it tells you whether answers sounded good. Outcome quality matters, but it is an outer-loop measure: label a sample, run evaluations, and compare versions (Chapter 24). Inner-loop operations need counters that expose missing decisions, uncertain effects and broken records while a run is still recoverable.

Start with six required invariants. This table is a production instrumentation contract, not metrics the reference Reference Agent can already compute. The reference has no durable ALLOW or DENY event; approval events omit call identity; the tool span hard-codes allow; retries lack attempt identity; and _observe emits no durable event. Until you extend that telemetry, several joins below are guesses.

Make the contract computable by construction. Put the run id and stable logical-call key on every proposal, verdict, approval, result and observation; give each physical retry an attempt number. Otherwise policy coverage becomes a timestamp join, which fails under concurrent tools. With explicit identity, you can test whether call K had a verdict, whether attempts K.1 and K.2 reached the receiver, and whether one result became model-visible. Keep tool, tenant, policy rule and risk class as dimensions, not identity.

Do not derive the contract from free-form logs. “Refund completed” cannot distinguish a proposal, remote acknowledgement or durable completion. Emit typed events and reject malformed ones at the boundary. The same record can then support resumption, audit and the dashboard; Chapter 30 turns it into governance evidence.

Use denominators that preserve the boundary. “Tool error rate” mixes unknown tools, policy denials, human declines, guardrail blocks, timeouts and executed failures into one percentage, even though each has a different owner and fix. A denied proposal is evidence that policy worked; counting it as tool unreliability punishes the control you want. Reference Agent’s _observe(..., executed=False) distinction already keeps denied calls out of the trajectory; production telemetry must preserve that same distinction explicitly.

The same discipline applies to latency. Report model time, approval wait, queue time and tool time separately. A voice run that takes twelve seconds may contain a 700-millisecond model call and eleven seconds waiting for a person; buying a faster model will not improve it. Conversely, an apparently fast run that excludes approval wait from its end-to-end clock gives the product team a number no user experiences. Chapter 18 supplies the conversational latency budget, while Chapter 23 shows how to derive the critical path from spans rather than summing parallel work.

Suppose the completed production instrumentation contract yields a daily slice of 10,000 Reference Agent runs and 28,400 logical tool calls. It reports 28,397 correlated policy decisions, 27,960 completions, 421 explicit denials or declines, and 19 starts still unresolved after the recovery window. Do not celebrate a 98.5% completion rate. First account for the three calls with no policy decision — they represent a control-plane defect regardless of outcome. Then isolate the 19 uncertain starts: 19 ÷ 28,400 is about 0.067% of all logical calls. To report the uncertain mutation rate defined above, use mutating starts as the denominator. These figures are illustrative. Percentages prioritise capacity work; absolute examples prioritise safety work. You need both.

Approval yield needs interpretation rather than a universal target. If operators approve 99.8% of destructive database actions in under two seconds, the gate may be ceremonial. If they approve 3% of low-risk document writes after a five-minute wait, the rule may be interrupting people for work they never wanted. Segment by tool, risk rule, tenant and approver latency, then inspect examples. The metric identifies where judgement is failing; it does not replace judgement.

Finally, set alerts on violations, not on arbitrary averages. Page on any policy bypass, any duplicate irreversible effect, and any unresolved high-risk mutation past its reconciliation deadline. Trend retry amplification, approval yield and context utilisation; they deserve investigation, not necessarily a night-time page. This split keeps on-call attention for broken invariants and gives the outer loop enough evidence to improve thresholds deliberately.

3.8How real harnesses differ#

The eight subsystems are a useful review checklist. Public documentation shows several implementation choices, but a feature list is not proof of end-to-end crash recovery or security. These examples were checked on 8 September 2026.

Distinguish three kinds of recovery. Conversation persistence restores what the agent knew; workspace checkpoints restore selected local artifacts; action recovery establishes what happened in another system. A production task can need all three. Chapter 5 follows those boundaries through a crash.

Evaluate approvals and isolation separately. An approval gate controls permission to attempt an action; a sandbox constrains what execution can reach. Git undo provides neither. Assess the selected deployment and configuration directly instead of interpreting undocumented sandboxing as proof of no isolation, or ranking enterprise suitability by the number of approval modes.

Distinguish reusable instructions from context indexes. A skill or playbook can encode a procedure. A repo map describes code structure and helps retrieval; it is not automatically procedural memory. Chapter 10 separates these uses.

3.9Subsystem responsibilities#