Observability & Tracing

“Reference Agent gave a customer the wrong reconciliation total on Tuesday afternoon.”

That is a plausible bug report, and with the wrong instrumentation it is unanswerable. Which run? What was in its context? Which documents did it retrieve, and were they the current versions? Did a guardrail redact something? Was the plan revised, and why? Did a tool return an error the model misread? Was this the tenth compaction of a long run? Did it cost forty cents or four dollars?

A run_completed log line cannot answer these questions. A trace preserves the context and causal sequence needed to investigate.

23.1The trace is the unit#

A conversational LLM app can be understood one request at a time. An agent cannot: the interesting behaviour is across turns, and the interesting question is almost never “what did the model say” but “what did the model see at the moment it decided.”

So the unit of observability is the trace: a tree of spans covering one whole run, where each span is one timed, attributed operation.

@dataclass
class Span:
    name: str
    kind: str                 # "run" | "chat" | "tool" | "internal"
    span_id: str
    parent_id: str | None
    attributes: dict[str, Any]
    events: list[dict[str, Any]]
    started_at: float
    ended_at: float | None
    status: str = "ok"
    error: str = ""

Four span kinds cover a single-agent harness, and the discipline is that every one of them is created by the harness, not by the tool author. Instrumentation that depends on each tool remembering to emit a span produces a trace with holes exactly where the unusual tools are.

Rendered, a real run looks like this — and this specific output is from the reference implementation’s demo:

· agent.run                       10.1ms
  · chat                             0.0ms  in=342 out=36
  · tools.parallel                   4.1ms
    · tool.read_file                   0.2ms  allow
    · tool.fetch_url                   0.0ms  allow
  · chat                             0.0ms  in=435 out=6
  · chat                             0.0ms  in=482 out=10
  · chat                             0.0ms  in=515 out=10
  · tool.issue_credit                0.0ms  allow
  · chat                             0.0ms  in=535 out=10
  · chat                             0.0ms  in=555 out=40
  · tool.write_file                  0.2ms  allow
  · chat                             0.0ms  in=623 out=46

Two things are visible there that are invisible in any log. The input token count grows monotonically — 342, 435, 482, 515 — which is the context accumulation of Chapter 8 made observable. And the parallel tool span nests two children, which records the batch relationship. Confirm actual concurrency from overlapping timestamps; parentage alone does not prove overlap.

23.2The two traces I broke#

Rendering the tree exposed two instrumentation bugs:

The trace that lied about causality. The first version of the tracer kept a stack of open spans in a list. Under parallel tool execution the two worker threads pushed and popped that shared list, and the result was a tree where tool.fetch_url appeared nested inside tool.read_file — two independent operations rendered as a causal chain.

# The current span, per execution context. A plain list would be wrong: agents
# run tool calls concurrently, and a shared mutable stack makes two parallel
# tools appear nested inside each other.
_CURRENT: contextvars.ContextVar[str | None] = contextvars.ContextVar(
    "reagent_current_span", default=None)

Fixing that exposed the second bug, because worker threads do not inherit context variables. Now the parallel spans had no parent at all and rendered as roots outside the run — a trace that could not answer “what caused this,” which is the only question a trace exists to answer. The fix is to capture the context in the submitting thread:

for c, t in reads:
    snapshot = contextvars.copy_context()   # capture in the SUBMITTING thread
    pending.append((c, t, pool.submit(snapshot.run, self._invoke_one, c, t)))

A trace with wrong parentage misrepresents causality. Inspect the rendered tree and assert its structure:

def test_parallel_tool_spans_are_parented_correctly(toolset, policy, tmp_path):
    """A trace whose parallel branches float free as roots is a trace that
    cannot answer 'what caused this'."""
    roots = [s for s in loop.tracer.spans if s.parent_id is None]
    assert len(roots) == 1 and roots[0].kind == "run"
    parallel = next(s for s in loop.tracer.spans if s.name == "tools.parallel")
    # Fixture: these two reads belong to this parallel batch.
    reads = [s for s in loop.tracer.spans
             if s.name in {"tool.read_file", "tool.fetch_url"}]
    assert len(reads) == 2
    assert all(s.parent_id == parallel.span_id for s in reads)

23.3The exporter is part of the failure domain#

Export spans without making the collector a synchronous dependency of agent execution.

The reference tracer exposes one deliberately small export seam:

# Excerpt from span(); its surrounding try/except is omitted here.
try:
    ...
finally:
    s.ended_at = time.monotonic()
    _CURRENT.reset(token)
    if self._on_end:
        self._on_end(s)

Read the sequence precisely. The triggering span’s duration stops before the callback runs, the active parent is restored, and then on_end is invoked synchronously. The callback wait is absent from that span, but it delays every ancestor that is still open. A child-span export therefore enlarges the run span without appearing in the child’s duration; the root callback happens after the run span’s own ended_at and is absent from the entire tree. That is enough for the book’s in-memory collector and for a test callback. It does not implement a production exporter, a queue, backpressure, retry, batching or durable delivery. Those are production extensions at the boundary represented by on_end, not capabilities already shipped in Reference Agent.

The sequence also exposes two hazards. First, a slow callback delays the code leaving every span even though the delay is absent from that span’s recorded duration. Second, an exception from the callback can escape the context manager. If the span body also failed, the exporter exception can obscure the failure you wanted to observe. Telemetry has then changed both the latency and the error surface of the system it was meant to describe.

Consider an illustrative Reference Agent incident. A release replaces the in-memory callback with a direct HTTP send to a collector. A normal reconciliation run creates eighteen spans. During a collector slowdown, each send takes 240 ms:

Support sees task latency jump by roughly four seconds. The model and tool spans look normal because each wait happens after the triggering span’s end timestamp. The run span grows by roughly 4,080 ms because it remains open for the seventeen descendant callbacks, but no child duration attributes that time; the final 240 ms root callback occurs after the run span ends. Everyone initially blames the model provider; the real mechanism is synchronous export. The tree contains unattributed ancestor time and omits its final export wait, making it operationally misleading even though each triggering span’s duration is internally consistent.

The production fix is to make span completion a local operation. Convert the finished span to an immutable envelope, place it on a bounded in-process queue, and let a separate worker batch and export envelopes. Give that worker its own timeout and retry budget. Instrument the queue depth, enqueue failures, batches rejected and oldest-envelope age outside the agent trace. Otherwise the exporter still cannot report that it is the failing component.

The envelope must carry enough identity to be useful after it leaves memory: trace id, span id, parent id, kind, start and end timestamps, status, attributes, events, schema version and the sampling decision made for the root. Copy the attributes rather than retaining a reference to the mutable Span. A late call to span.set(...) must not race an exporter thread that is serialising the same dictionary. The reference callback receives the live object only after the context body exits; that is safe for its synchronous teaching seam, not a contract for asynchronous mutation.

Make the root sampling decision once and propagate it to every descendant. If the run is admitted, descendants are admitted. If it is rejected, descendants may update bounded aggregate metrics but do not enqueue trace envelopes. Re-deciding independently for each span produces a collection of unrelated fragments. It also biases debugging towards runs with fewer spans, because a twenty-span run has more opportunities to lose a necessary branch than a five-span run.

Bounded is the important word. An unbounded queue turns a collector outage into a memory outage. A queue with room for 20,000 envelopes and a peak arrival rate of 400 envelopes per second contains fifty seconds of burst capacity:

20,000 envelopes / 400 envelopes per second = 50 seconds

That is not a retention guarantee. It is a deliberate shock absorber. When the queue fills, apply an explicit policy: drop complete low-priority traces, spill encrypted batches to a bounded local store, or block only if your operating contract says observability is fail-closed. Do not silently drop arbitrary spans from the middle of accepted traces. A partial tree looks like a real tree and leads you back to confident wrong conclusions.

Size the queue in bytes as well as envelopes. If the average serialised envelope is 3 KB, 20,000 slots can retain about 60 MB before queue and runtime overhead. Full prompt capture can make one envelope hundreds of times larger. Enforce a maximum envelope size before enqueue, record that truncation as metadata, and put large approved captures in a protected object store referenced by an opaque id. Otherwise one pathological prompt can consume the buffer intended to protect thousands of ordinary spans.

Shutdown needs a contract too. Stop accepting new runs, finish spans already open, close the producer side of the queue, drain until a fixed deadline, and record how many envelopes remain. A container receiving a thirty-second termination grace period cannot give its exporter a sixty-second drain budget. Reserve the time explicitly. Re-sending a batch after an uncertain acknowledgement must be idempotent by (trace_id, span_id); duplicate delivery is preferable to missing evidence, but duplicate spans must not inflate fleet counts.

Keep the audit path separate. A trace exporter may degrade and shed sampled diagnostic data. Chapter 5’s authoritative event log cannot inherit that policy. The same instrumentation point can emit both records, but an exporter queue becoming full must never cause an approval or side-effect event to disappear from the audit log.

There are therefore two exporter failure modes to test. Interference asks whether a slow or broken collector changes task latency or outcome. Loss asks whether accepted trace envelopes reach the backend intact. Inject latency and exceptions into on_end; then stop the exporter worker with a full queue. If those tests do not state what the agent and the telemetry do in each case, the export policy is accidental.

A third test is self-observation. The exporter cannot safely describe its own failure only by sending another span through itself. Publish queue depth, dropped complete traces, retry exhaustion and last successful export through a separate bounded metric or health path. Page on sustained oldest-envelope age rather than a single failed batch: transient retry is normal, while age rising continuously means the backend is no longer catching up.

23.4Attribute names, and why they should not be yours#

The instrumentation is the expensive, invasive part; the backend is a commodity. So the attribute vocabulary should be one you can carry between backends.

OpenTelemetry’s GenAI semantic conventions provide that vocabulary. A dedicated repository covers GenAI clients, MCP and provider-specific spans, metrics and events.¹ The conventions remain in development. Pin a revision and record migration rules before changing attributes or dashboards.

The current conventions use gen_ai.provider.name and gen_ai.usage.cache_read.input_tokens; cached reads are included in total input tokens. The local constant GEN_AI_SYSTEM retains its historical Python name but emits the current attribute. The model adapter supplies the actual provider identity, with local-scripted for the demo. These spans are OTel-shaped local objects, not an OTLP export implementation.²

Keep names as constants, then migrate dashboards and schema versions together:

GEN_AI_OPERATION        = "gen_ai.operation.name"     # chat | execute_tool | invoke_agent
GEN_AI_SYSTEM           = "gen_ai.provider.name"  # compatibility constant name
GEN_AI_REQUEST_MODEL    = "gen_ai.request.model"
GEN_AI_USAGE_INPUT      = "gen_ai.usage.input_tokens"
GEN_AI_USAGE_OUTPUT     = "gen_ai.usage.output_tokens"
GEN_AI_USAGE_CACHED     = "gen_ai.usage.cache_read.input_tokens"
GEN_AI_RESPONSE_FINISH  = "gen_ai.response.finish_reasons"
GEN_AI_TOOL_NAME        = "gen_ai.tool.name"
GEN_AI_TOOL_CALL_ID     = "gen_ai.tool.call.id"
GEN_AI_AGENT_NAME       = "gen_ai.agent.name"
GEN_AI_CONVERSATION_ID  = "gen_ai.conversation.id"
# Non-standard but essential: cost, policy decisions, guardrail outcomes.
REAGENT_COST_USD         = "reagent.cost_usd"
REAGENT_POLICY_DECISION  = "reagent.policy.decision"
REAGENT_GUARDRAIL        = "reagent.guardrail"

The three custom attributes at the bottom are application-specific fields in this harness: cost, because it is the runtime signal that catches a runaway; the policy decision, because “which rule permitted this action” is the first question in any incident; and the guardrail outcome, because a WARN that fires ten thousand times a day is telling you something.

23.5What belongs on each span#

Record agent-specific fields alongside the standard attributes:

On the run span: the outcome (Chapter 4’s named enum), whether this was a resume, the tenant, the actor, the plan id, the autonomy level, total cost.

On each chat span: the assembled context size, window utilisation, the stable-prefix length, whether the cache hit, how many tools were visible, and the finish reason.

with self.tracer.span("chat", "chat", **{
    tr.GEN_AI_OPERATION: "chat",
    tr.GEN_AI_REQUEST_MODEL: getattr(self.model, "name", "unknown"),
    "reagent.context.tokens": report.total_tokens,
    "reagent.context.utilisation": round(report.utilisation, 3),
    "reagent.context.stable_prefix_tokens": report.stable_prefix_tokens,
    "reagent.tools.visible": len(visible) if visible is not None else len(self.tools),
}) as span:
    ...  # model call and response attributes

On each tool span: the policy decision and the rule that decided, whether the tool mutates, retry attempts, whether the result was an error, and the idempotency key. That last one is what lets you answer “did this side effect happen twice” across a resume.

On internal spans: compaction ratio and what was dropped, guardrail name and severity, approval latency.

The test of sufficiency is a specific one: can you reconstruct what the model saw on any turn, from the trace alone? Not the full text necessarily — retention and privacy may forbid that — but the shape: which sections, how many tokens each, what was dropped and why. If you cannot, the “why did it forget that” class of bug is unanswerable.

23.6Prompts and outputs: the retention question#

Whether to store full prompt and completion text is a genuine trade-off, and the answer is usually not “all of it forever.”

Full text makes debugging dramatically easier — you can see exactly what happened. It also means your trace store now contains every piece of customer data your agent has ever touched, with the retention, residency, access-control and deletion obligations that follow (Chapter 30).

The workable structure is tiered.

Default: an allowlisted set of metadata, token counts, costs, latencies, tool names, policy decisions and outcomes. Metadata can still identify people or reveal confidential activity: minimise identifiers, restrict access, and apply retention and deletion rules.

Opt-in sampled capture: full text only when a documented purpose and data policy permit it, including selected error or escalation runs. Apply redaction before export. An error does not automatically authorise capturing customer data.²

On demand: a short-lived full capture enabled per tenant or per run when investigating something specific, with an audit record of who enabled it.

Never in the trace: secrets and, in most regulated contexts, unredacted personal data. Chapter 20’s scanners belong on the ingest path of your trace store, not only on the agent’s output — because a secret in a trace is a secret in a system with much broader read access than the agent had.

23.7Cost and token attribution#

Cost is a first-class runtime signal for agents in a way it is not for most software, because it is unbounded at request time and attributable to a task.

Per run, so you know what a task costs.

Per tenant, so you know whether a customer is profitable.

Per capability, so you know which feature is expensive.

Per model, so routing decisions (Chapter 29) have evidence.

def total_cost(self) -> float:
    return sum(float(s.attributes.get(REAGENT_COST_USD, 0.0))
               for s in self.by_kind("chat"))

def total_tokens(self) -> tuple[int, int]:
    return (sum(int(s.attributes.get(GEN_AI_USAGE_INPUT, 0)) for s in self.spans),
            sum(int(s.attributes.get(GEN_AI_USAGE_OUTPUT, 0)) for s in self.spans))

Cost per task rising. The earliest signal of a prompt regression, a retrieval change that widened top_k, or a cache that stopped hitting. Often visible days before any quality metric moves.

Cache hit rate falling. Prefix or routing changes can raise cost without changing visible behaviour (Chapter 8).

Turns per task rising. The agent is working harder for the same outcome. Usually a tool interface regression (Chapter 13) or a retrieval quality drop.

These quality regressions appear in cost before they necessarily appear as errors or latency.

23.8Work a cost regression from signal to cause#

A cost alert is useful only if the trace lets you separate more turns from more expensive turns. A single daily spend graph cannot. Work the arithmetic at chat-span level before asking the model team to change anything.

Suppose an illustrative Reference Agent document-reconciliation task had this seven-day baseline. It completed in six model calls. Each call averaged 8,000 input tokens and 600 output tokens. Use an illustrative price of $3 per million input tokens and $15 per million output tokens:

input cost per call  = 8,000 × $3 / 1,000,000  = $0.024
output cost per call =   600 × $15 / 1,000,000 = $0.009
cost per call        = $0.024 + $0.009          = $0.033
cost per task        = 6 × $0.033               = $0.198

After a deployment, completed tasks average seven calls, 10,400 input tokens per call and 650 output tokens per call. The same arithmetic gives:

new input cost per call  = 10,400 × $3 / 1,000,000 = $0.03120
new output cost per call =    650 × $15 / 1,000,000 = $0.00975
new cost per call        = $0.03120 + $0.00975       = $0.04095
new cost per task        = 7 × $0.04095              = $0.28665
increase per task        = $0.28665 - $0.19800       = $0.08865
relative increase        = $0.08865 / $0.19800       = 44.8%

At 10,000 tasks per day, that regression costs an additional $886.50 per day, or $26,595 over a thirty-day month. Those values are illustrative, but every denominator is one your trace should carry: chat spans supply input and output tokens, run spans delimit the task, and the named terminal outcome lets you restrict the comparison to like-for-like completed work.

Now split the $0.08865 increase using a baseline-price-first decomposition. If only the turn count had changed, seven baseline calls would cost 7 × $0.033 = $0.231, an increase of $0.033. The remaining $0.28665 - $0.231 = $0.05565 comes from the larger per-call context and output. Under that ordering, turn count contributes 0.033 / 0.08865 = 37.2% of the increase; per-call inflation contributes 62.8%.

Those percentages are order-dependent because the extra turn and the higher per-call cost interact. Reverse the order and apply per-call inflation to the original six calls first: 6 × ($0.04095 - $0.033) = $0.04770, or 53.8% of the increase. The seventh call at the new price contributes $0.04095, or 46.2%. The total remains $0.08865; the interaction of $0.04095 - $0.033 = $0.00795 moves between categories. State the ordering whenever you publish the split, or use an order-neutral allocation rule and name it.

That decomposition changes the investigation. A 37.2% turn contribution points you towards trajectory and tool behaviour: did a tool start returning ambiguous results, causing one repair turn? The 62.8% per-call contribution points you towards context assembly: did retrieval widen, did a stable prefix stop caching, or did a tool schema grow? Query the trace in that order rather than debating prompts from memory.

Continue from the aggregate to one paired comparison. Select a baseline trace and a regressed trace for the same task class, tenant, model and terminal outcome. Align their chat spans by the preceding tool result, not merely by ordinal turn: an added repair turn shifts the ordinal sequence. For each aligned pair, compare reagent.context.tokens, stable-prefix tokens, cached input tokens, visible tool count and finish reason. Then inspect the first pair that diverges.

A compact attribution table keeps the diagnosis honest:

There is an aggregation trap worth making explicit. Each chat span receives its own reagent.cost_usd, and the run span receives the aggregate. The current Tracer.total_cost() sums chat spans only, avoiding double counting. A dashboard that sums all span kinds would reintroduce the defect: chat costs of $0.024 and $0.031 plus their $0.055 run summary total $0.110, twice the actual model cost.

Define the aggregation grain with the metric:

• Cost per task: read the single root run span’s aggregate; group by its terminal outcome and tenant.
• Cost by model: sum chat-span cost only; group by gen_ai.request.model.
• Attributed model cost: compare the root aggregate with the sum of child chat costs, but do not add them. A non-zero difference is unpriced usage or an attribution defect to investigate.
• Tool-provider cost: use a separately named tool-cost attribute on tool spans. Do not overload model cost and then guess from span kind downstream.

Pin the invariant in the semantic layer or dashboard query, not in tribal knowledge. “Cost” without a span-kind filter is not a metric definition.

Control for workload mix before declaring a regression. Suppose the original 10,000 daily tasks were 8,000 simple tasks at $0.150 and 2,000 complex tasks at $0.390. Total spend was $1,200 + $780 = $1,980, or $0.198 per task. If the next day brings 6,000 simple and 4,000 complex tasks at unchanged unit costs, spend becomes $900 + $1,560 = $2,460, or $0.246 per task — a 24.2% aggregate increase with no within-class regression. Compare task class, model, tenant tier and outcome before-and-after. The trace supplies those dimensions only if you chose bounded, stable labels for them before the incident.

Do not allocate every token dollar to successful tasks and call the system efficient. Track cost per attempted task and cost per completed task together. If Reference Agent spends $198 across 1,000 attempts but completes only 700, attempted-task cost is $0.198 while completed-task cost is $198 / 700 = $0.283. The gap is failure cost, and optimising prompt size alone will not remove it.

23.9Debugging non-determinism#

The same input can produce different runs. Reproduction needs controlled fixtures and repeated trials; these four techniques help isolate the difference.

Trace comparison. Take a run that succeeded and a run that failed on the same input and diff the span trees. Where did they diverge? Almost always at one specific decision, and having the two traces side by side localises it in a minute rather than an hour.

Aggregate over the population. With non-determinism, one trace is an anecdote. Twenty traces of the same task are data: 15 of 20 called search_docs first and 5 called graph_query, and the 5 all failed. That is a finding, and Chapter 25’s error analysis is built on exactly this.

Replay from the event log. Reconstruct state at event N, then use a scripted model to replay harness decisions deterministically. This helps locate whether the harness or model introduced the divergence.

Fix the seed where you can. Temperature zero and a fixed seed do not make an agent deterministic — tool results vary, timing varies, the world changes — and some endpoints do not support those controls. Where supported, they can reduce one source of variance while you investigate the others.

23.10Measure whether the trace deserves belief#

Measure trace completeness, causal structure and export success; span volume alone establishes none of them.

Start with an expected-operation ledger. Reference Agent’s authoritative events tell you how many model turns and tool invocations occurred; run state tells you how many compactions completed; every attempted run requires one root span. The reference tracer holds all spans in process, but it does not reconcile them against that ledger or export them to a backend. Production Reference Agent should perform that reconciliation on a sampled audit batch.

Take an illustrative batch of 2,000 runs. The event and state records imply 12,480 chat spans, 9,730 executed tool spans, 612 compaction spans and 2,000 run spans. That makes 24,822 expected operation spans. The trace backend contains 24,657 matching spans:

operation coverage = 24,657 / 24,822 = 99.335%
missing spans       = 24,822 - 24,657 = 165

Do not report that as “99.3% observability” and move on. Find where the 165 disappear. If all are tool spans from cancelled parallel batches, you have a lifecycle bug. If they cluster on one host, you have an exporter failure. If the event ledger itself is incomplete, you have an audit problem rather than a tracing problem.

Then test structure. Of the 22,657 observed non-root spans in this batch, suppose 47 name a parent id that is absent from the same trace. The orphan ratio is 47 / 22,657 = 0.207%. For this harness’s joined child operations, require an existing parent in the same trace and a plausible enclosing interval. General distributed traces can use links, asynchronous children and independently sampled remote spans; validate against that declared topology rather than imposing the local teaching tree on every service. A trace that passes coverage but fails parent resolution still lies about causality.

Containment needs a concurrency-aware definition. A child’s start should not precede its parent’s start, and it should normally end before the parent ends. Sibling spans may overlap; that overlap is exactly what tools.parallel is meant to show. A validator that rejects overlapping siblings converts correct parallel execution into a false alarm. Conversely, a tool child ending after tools.parallel closes means the caller stopped joining its workers or exported the parent before its children finished. Validate interval containment, not a single global ordering.

The reference tracer uses time.monotonic() for duration, which is correct inside one process and immune to wall-clock adjustments. Do not compare those absolute values across hosts. A production distributed trace needs transport timestamps suitable for cross-process correlation, plus tolerance for clock skew; duration should still come from a monotonic clock local to each process. If a remote tool reports its own span, connect it through propagated trace context rather than manufacturing parentage later from similar timestamps.

Finally separate enqueue acceptance from backend persistence. Suppose the local exporter accepted all 24,657 observed envelopes but the backend persisted 24,392 before the verification deadline. Export loss is (24,657 - 24,392) / 24,657 = 1.075%. Without both counters, each side can report success while 265 envelopes vanish between them.

Choose service objectives from your operating contract, not from this illustrative batch. The crucial property is that each objective has an independent denominator. Sampling must also happen at the whole-trace admission boundary. Keeping 10% of every span independently does not produce a small representative dataset; it produces broken trees. Admit or reject the trace, then keep all of its spans unless a declared emergency shedding policy says otherwise.

Metrics and traces need different cardinality rules. Put run id, conversation id and tenant id on searchable trace attributes with access control. Do not put them in time-series metric labels. A fleet counter labelled by one million run ids creates one million series and can take the monitoring system down while the agent remains healthy. Metric labels should come from bounded sets such as terminal outcome, span kind, tool name and model family. Prompt text, document ids and raw error messages belong in neither label set.

Measure label cardinality before the monitoring bill or outage tells you. For each metric name, count active series and the number created per hour. A release that embeds exception text in an error label can turn one logical error class into thousands of series because ids, paths and customer values vary inside the message. Keep error.type as a bounded code on the metric; keep the sanitised message and stack on the trace event. Budget cardinality per metric and reject labels outside the declared schema at instrumentation time.

The final check is retrieval, not ingestion. Pick ten known run ids from the audit ledger and ask an on-call engineer to find their traces using the same permissions and interface available during an incident. If nine are found, trace discoverability is 90%, however perfect the exporter counters look. A trace that exists but cannot be found before the incident clock expires has failed its operational purpose.

Assign an owner and a response to each measure. The harness team owns tree-shape failures; the telemetry platform owns persistence loss; the service team owns missing instrumentation; security owns unauthorised trace access. Alert only when someone can act. A single orphan should fail a conformance test, while an operational page should use a rolling error budget so one malformed development run does not wake an engineer. Preserve the offending run ids as exemplars on the alert. An aggregate ratio tells the owner that a boundary failed; the exemplars let them open the exact trees and find where. Without both, the dashboard produces awareness rather than diagnosis.

23.11Three views, three audiences#

A trace answers one question about one run. Two other views are needed, and teams routinely build only the first.

The fleet view’s single most valuable panel is the distribution of terminal outcomes over time, split by Chapter 4’s enum. COMPLETED falling and LOOP_DETECTED rising is a tool regression. APPROVAL_DENIED rising is either a policy that got tighter or an agent that got worse at proposing. BUDGET_EXHAUSTED rising is tasks getting harder or ceilings getting stale. One histogram, four diagnoses, and it costs nothing once outcomes are named.

The reliability measures behind those views form a separate evidence chain: events must agree with spans, accepted envelopes must agree with persisted envelopes, and known run ids must be retrievable.

23.12The audit trail is a different artefact#

Traces and audit logs have different purposes and retention requirements:

Traces are for debugging. They may be sampled and retained for a limited period under a documented policy. Diagnostic loss must be measured against the service’s operational requirements.

The audit log is for accountability. Required events are unsampled, versioned, access-controlled and retained for the applicable legal and operational period. It answers who authorised an action and what happened to data (Chapters 19 and 30). Missing required evidence is a control failure; the legal consequence depends on the applicable obligation.

Chapter 5’s event log supplies audit facts. The teaching JSONL store does not itself supply tamper evidence, storage durability across power loss, governed retention, access controls or a compliant audit service. Production must add those controls. The practical division:

Emit both from the same instrumentation points, with explicit policies for their different purposes.