Evaluation Frameworks & Metrics

In this illustrative example, Reference Agent’s 20-case suite reports 17 passes on Monday and 18 on Friday: 85% and 90%. The team ships the prompt change, yet production quality gets worse.

One changed result is weak evidence, especially when repeated runs vary. The 95% Wilson intervals are approximately 64–95% and 70–97%. Their overlap does not test the difference. A paired comparison must identify which cases changed. Here, one unstable case created the apparent improvement. The real regression affected a task type absent from the suite.

Good evaluation needs correct arithmetic, a comparison designed for the decision, and coverage of the work users actually ask the agent to do.

24.1Outcome is not enough#

For a single-shot model call, “was the answer right” is nearly the whole question. For an agent it is half.

Outcome evaluation asks whether the final state is correct. Did the reconciliation find the right discrepancies? Is the file correct? Was the credit the right amount?

Trajectory evaluation asks whether the route was acceptable. Did it verify before writing? Did it avoid the tools it should not have touched? Did it escalate when it should have? Did it get there in four turns or forty?

Outcome-only and trajectory-only scoring miss different failures:

A system scored only on outcomes learns to reach them by routes you never approved — and the routes are often cheaper, which means optimisation pressure points that way. A system scored only on trajectory does the approved thing and fails. Score both, and score them separately so you can tell which one moved.

@dataclass
class EvalCase:
    id: str
    goal: str
    expect_substrings: tuple[str, ...] = ()
    forbid_substrings: tuple[str, ...] = ()
    must_call: tuple[str, ...] = ()          # the trajectory contract
    must_not_call: tuple[str, ...] = ()
    expect_outcome: RunOutcome | None = RunOutcome.COMPLETED
    max_cost_usd: float | None = None
    max_turns: int | None = None

Note expect_outcome. Chapter 4’s named terminal outcomes make a whole category of case expressible that is otherwise impossible: cases where the correct behaviour is to refuse. A case asserting expect_outcome=ESCALATED on an under-specified request is testing judgement, and a suite without such cases rewards an agent that always attempts something.

And note must_not_call. Trajectory scoring gives partial credit on must_call — three of four required tools is genuinely different information from none — but a forbidden call zeroes the score outright, because “it got the answer but touched production” is not 75% correct.

24.2The reliability gap#

The most important number in agent evaluation is not the pass rate. It is the gap between passing once and passing every time.

The original 2024 τ-bench study measured this for the models and retail/airline tasks it tested: its evaluated function-calling agents succeeded on under 50% of tasks, and — the striking part — were “quite inconsistent (pass^8 <25% in retail).”¹ An agent that solves fewer than half of these tasks on a single attempt solves under a quarter of them eight times out of eight.

pass@k — the probability that at least one of k attempts succeeds. A capability measure. Rises with k. Appropriate when a human filters the output, or when the agent proposes and a person picks.

pass^k — the probability that all k attempts succeed. A reliability measure. Falls with k. Appropriate when the agent acts autonomously, because you get one attempt and it has effects.

Report both the attempt success rate and repeated-task reliability, with the trial count and evaluation conditions. For a fixed task with independent attempts each succeeding with probability 0.85, pass^4 is 0.85^4 ≈ 52.2%. Real tasks have different success probabilities and correlated failures, so do not exponentiate the fleet average and call it measured reliability. Neither statistic alone measures consequence or production exposure.³

The practical consequence for your own suite is simple and slightly expensive: run every case multiple times.

def run_eval(cases, runner, scorers=None, *, repeats: int = 1) -> EvalReport:
    """Run every case `repeats` times and score each run.

    `repeats > 1` is how you measure a non-deterministic system honestly: a
    single pass tells you what happened once, not what happens. Report the
    attempt pass rate separately from the fraction of base cases that passed
    all k repeats (empirical pass^k). This teaching loop produces run reports;
    grouping them into pass^k requires a separate analysis.
    """

24.3Statistical power: how many cases#

For a proportion, report a confidence interval rather than a point estimate. The Wilson score interval is a useful default for a binomial proportion — it behaves properly at small samples and near 0% or 100%, where the naive normal approximation does not:

def wilson_interval(self, z: float = 1.96) -> tuple[float, float]:
    """95% confidence interval on the pass rate (Wilson score interval).

    Report this, not the bare percentage. With 20 cases, an observed 80% has
    a confidence interval roughly 58%-92% — wide enough that a 5-point
    difference needs a paired analysis; the interval alone is not a test.
    """
    n, p = self.n, self.pass_rate
    if self.repeats > 1:
        raise ValueError("Repeated cases require cluster-aware inference")
    if n == 0:
        return (0.0, 1.0)  # no evidence; report the empty denominator
    denom = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / denom
    margin = (z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))) / denom
    return (max(0.0, centre - margin), min(1.0, centre + margin))

For an observed 80% pass rate on independent sampled cases, these Wilson intervals describe estimation precision, not the smallest detectable difference between versions:²

A 20-case suite is useful for smoke tests and targeted regressions, but gives little precision about a broad task population. No universal minimum case count answers every release question. Plan sample size from the minimum useful effect, error tolerance, power, baseline rate and paired or clustered design. Rare severe failures also need targeted tests; observing none in a small random sample does not establish safety.

Repeats measure variability; distinct cases measure coverage. One hundred cases run three times yield 300 run records and 100 base-case clusters. They do not become 300 independent task types. Preserve all repeats when resampling a base case for uncertainty, and report the fraction of base cases that passed every repeat separately from the fraction of individual runs that passed.

Use a paired comparison: run both versions on the same frozen cases and inspect fixes and regressions. Pairing often improves precision, but power depends on the discordant pairs, not just suite size.

24.4A comparison you can actually ship on#

A confidence interval around each version is better than two naked percentages, but it still throws away the most useful fact in the experiment: both versions ran on the same cases. Preserve that pairing. It lets you ask which individual cases changed, rather than whether two aggregate estimates happen to overlap.

Suppose Reference Agent has 300 versioned cases. You run the incumbent prompt, version A, and the candidate, version B, once on the same frozen inputs with the same tool fixtures. The outcome totals look like this:

The denominator is printed on every row. That is not decoration. It lets the reviewer see that the largest gain comes from the smallest, highest-risk cohort, and it prevents a polished percentage from hiding three cases behind a rounded rate.

Now retain the paired outcomes rather than only those margins:

Check the arithmetic before interpreting it. Version A passes are 229 + 12 = 241. Version B passes are 229 + 39 = 268. The net gain is 39 - 12 = 27 cases, or 27 / 300 = 9.0 percentage points. The 249 cases on which both versions agree tell you about absolute capability, but not about the direction of the change. The evidence for the comparison lives in the 51 discordant cases.

Under a null hypothesis that neither version is better, either version should own a discordant case with probability one half. Here B owns 39 of 51. An exact two-sided binomial calculation gives:

p = 2 × Σ C(51, i) / 2^51, for i = 39 … 51
  = 0.000198

This is the exact form of McNemar’s test for the paired pass/fail table. It says that a split at least this uneven would be rare if the prompts were equivalent. It does not say the candidate is safe, that the nine-point gain will transfer to production, or that the cases cover the deployment. Statistical significance answers one narrow question. Release evidence needs the rest of the contract.

For Reference Agent, the decision record should therefore show four views together:

1. Paired outcome movement: 39 fixes, 12 regressions, and links to all 51 traces.
2. Cohort denominators: the exact case count, pass count and delta for each pre-declared risk tag.
3. Trajectory gates: zero new forbidden calls, zero skipped approvals and no increase in unverified writes. These are vetoes, not averages.
4. Efficiency movement: median and high-percentile turns and cost on the same case pairs, so one expensive tail cannot hide behind a stable mean.

The twelve regressions still need to be read. If all twelve are cosmetic formatting differences and the 39 fixes are correct refusals, the result is stronger than the table suggests. If one regression is an unapproved production write, the candidate does not ship. A significance test ranks evidence; it does not replace engineering judgement.

When you add repeats, keep the unit straight. Three runs of each of 300 cases produce 900 run observations and expose non-determinism. They do not produce 900 independent task types. Compare A and B within each case-and-repeat cell, report per-case pass^3, and estimate uncertainty with the base case as the sampling unit. Otherwise one flaky case appears three times in the denominator and receives three times the influence without adding any coverage.

Turn the table into a decision packet#

The evaluator should produce an artefact another engineer can audit without asking the experiment author what they meant. For the worked comparison, Reference Agent’s packet begins with immutable identities: case-set digest, prompt versions A and B, model version, tool-fixture version, judge version, execution window and the random seeds used for paired repeats. If any identity is absent, a later rerun can disagree for reasons the report cannot name.

The next page is the gate, evaluated mechanically:

This candidate does not ship. The result is deliberately uncomfortable: the outcome gain is large, statistically persuasive and strongest in the destructive cohort, yet one new forbidden call fails the trajectory gate. Averaging the rows would allow five green cells to wash out the one event the permission design says must never occur. A release gate is a Boolean policy over evidence, not a weighted popularity contest.

The packet then links the violating trace and every discordant pair. It records whether the forbidden call caused an external effect, but that fact does not turn the failure into a pass. A blocked attempt proves the policy boundary worked; it also proves the candidate proposed an action its predecessor did not. The prompt owner must understand that behavioural change before another run.

Finally, include exclusions. If four B runs timed out and the analysis silently removed them, B’s 268/300 numerator is being compared with A’s full denominator while wearing the label “same cases.” The report must say B missing = 4 / 300, assign the pre-declared failure treatment, and retain those run identifiers. Missingness is an outcome of the system until you prove it is an evaluator fault.

24.5Building the golden set#

The suite is the asset. Five rules for building one that stays useful.

Draw from production, not imagination. Sample real requests, including the ugly ones. A suite of tasks an engineer invented tests the tasks an engineer imagined, and production traffic is stranger than that in ways that matter.

Stratify deliberately. Cover the frequent cases proportionally, and then over-sample the rare and dangerous ones. Your eval suite’s distribution should not match production’s — it should match production’s risk.

Include cases that should fail. Under-specified requests that should escalate. Requests that should be refused. Content containing injection attempts (Chapter 21). Tasks whose answer is not in the corpus (Chapter 11’s honesty rate). A suite of only answerable questions rewards an agent that always answers.

Write the trajectory contract, not only the answer. Which tools must be called, which must not, what verification must happen. This is the half that catches the bottom-left cell of Figure 24.1.

Version it, and never silently change it. A suite that drifts cannot compare across time. Add cases in versioned batches, and when you change a case, record why.

Turn every production incident into a versioned regression case so the same mechanism cannot recur silently.

Protect the suite from the improvement loop#

A golden set stops being evidence when the team tunes directly against every answer in it. The score can keep rising while production quality stands still, because the prompt has learned the suite’s recurring nouns, formats and edge cases. This is ordinary test-set overfitting expressed through natural language.

Split the asset by purpose. Keep a development set whose traces and expected answers engineers may inspect freely. Keep a release set whose aggregate and cohort results they see, but whose case details are revealed only when a release decision needs diagnosis. Keep a smaller audit holdout under a separate owner and run it less often. Cases may move from a holdout into development after disclosure, but never move back and pretend to be unseen.

The unit of separation is the underlying scenario, not the text string. Five paraphrases of the same refund incident belong in one partition. Otherwise Reference Agent can see the scenario in development and meet its paraphrase in the holdout, producing a reassuring but false generalisation result. Group by source incident, document family, tenant workflow or generated-case template before sampling partitions.

Track exposure in case metadata: who could inspect the expected answer, which prompt versions ran against it, whether its trace appeared in an error-analysis meeting and whether it entered any training or retrieval corpus. In a multi-tenant system, also retain a tenant-safe derivation record. A production case can be reduced and anonymised into a portable evaluator without copying customer identifiers or confidential content into a repository.

When the release set becomes familiar through repeated failures, rotate a versioned batch in and report the overlap. Do not silently replace hard cases to restore a flattering baseline. A score break caused by a new set version is acceptable; an unexplained continuity between different denominators is not.

24.6The denominator-drift incident#

Reference Agent’s weekly quality dashboard fell from 84% to 60%. The alert fired, the prompt owner reverted Friday’s release, and the score stayed at 60%. Nothing in the release had caused the drop. The dashboard had compared different work.

The evaluation service sampled the first 100 completed traces each Monday. In the first week it happened to collect 90 routine reconciliation requests and ten destructive requests requiring approval. Reference Agent passed 81 routine cases and three destructive ones:

week 1 = (81 + 3) / (90 + 10) = 84 / 100 = 84%

In the second week a finance close changed the traffic mix. The sample contained 50 routine cases and 50 destructive cases. Reference Agent passed 45 routine cases and 15 destructive ones:

week 2 = (45 + 15) / (50 + 50) = 60 / 100 = 60%

The aggregate had fallen by 24 points. Within each cohort, nothing had moved: routine work was 81 / 90 = 45 / 50 = 90%; destructive work was 3 / 10 = 15 / 50 = 30%. The same two rates, mixed in different proportions, produced a different headline. The rollback was operational theatre.

The fix has two parts. Freeze an offline comparison set when deciding whether version B beats version A, so both versions see identical work. Stratify online reporting by stable, decision-relevant cohorts, so changes in traffic mix remain visible rather than masquerading as quality changes. Reference Agent records the cohort numerator and denominator beside every rate: 45 / 50, not only 90%.

Do not respond by inventing dozens of tiny slices. A cohort must change a decision. Useful dimensions include tool risk, expected terminal outcome, modality, tenant policy and request class. A slice such as browser language may matter if it reveals localisation failures; otherwise it spends statistical power without sharpening an action. Pre-declare the small set that maps to owners and failure mechanisms.

Small cohorts create a second trap. The destructive slice in week one is 3/10, but ten cases cannot establish a stable 30% rate. The apparent equality with 15/50 is illustrative arithmetic, not evidence that the two weeks are equivalent. Report its interval, retain the raw count and accumulate enough cases before declaring movement. Stratification prevents mixture error; it does not manufacture sample size.

If leaders still need one online series, compute a standardised rate against fixed, published weights. Suppose Reference Agent’s risk reference mix is 60% routine and 40% destructive. Both weeks standardise to:

0.60 × 90% + 0.40 × 30% = 54% + 12% = 66%

The standardised line stays at 66% while the raw line moves from 84% to 60%, immediately separating capability from traffic composition. Show both. The raw rate describes what users experienced that week; the standardised rate supports comparison across weeks. Replacing the raw rate with the adjusted one would hide the real operational fact that more users encountered the weak destructive path.

Weights are policy, so version them. Choose them from a declared operating mix or risk posture, not from whichever weighting flatters this release. Recomputing last month’s score with this month’s weights rewrites history. When the product mix genuinely changes, start a new series or backfill the whole series transparently and retain the old view.

The cohort labels must also be assigned without looking at pass/fail. If “complex” means “cases the agent failed,” that slice has a built-in 0% rate and can never measure improvement. Classify from request and policy attributes available before execution: required approval tier, tool family, modality, ambiguity rubric or expected outcome. This makes the denominator reproducible and keeps failure analysis from leaking into experiment definition.

The symptom-to-fix sequence generalises:

This incident is why a release report needs a data-quality panel before it needs another judge: case-set version, inclusion window, exclusions, cohort counts, repeats and missing-result count. If those changed, you are not looking at a model comparison yet.

24.7LLM-as-judge, made usable#

Many things worth measuring have no programmatic checker. Was the summary faithful? Was the explanation clear? Was the tone right for a customer? A model can judge these, and the judgement is usable if — and only if — you treat it as an instrument that needs calibration.

A specific, anchored rubric. Not “rate 1-5.” A list of criteria with pass/fail and definitions:

@dataclass
class RubricScorer:
    """LLM-as-judge, structured as a rubric with explicit anchors.

    The teaching prompt receives goal, answer and tool names.
    Production factual grading must also receive the source evidence
    and an authoritative reference answer when available.
    """
    criteria: tuple[str, ...] = ("factually grounded in tool observations",
                                 "answers the actual question asked",
                                 "states uncertainty where it exists")

A reference answer where one exists. An authoritative reference and the source evidence reduce reliance on the judge’s own priors. References can also be wrong or incomplete; review them and allow valid alternative answers.

Calibration against accountable labels. Have domain reviewers label a representative sample, resolve reviewer disagreements, and compare the judge on the same cases. Report false passes and false failures by risk class as well as overall agreement. Sample size follows the errors you need to detect; a few hundred cases is a starting workload, not a guarantee.

Known biases measured. The MT-Bench study documented position, verbosity and self-enhancement biases in the judges it tested.⁴ Randomise answer order and audit whether length or model identity changes scores. A different judge family can diversify errors, but does not establish independence; calibrate every judge against domain-reviewed labels.

def parse_judge_score(self, raw: str) -> Score:
    final_line = raw.strip().splitlines()[-1] if raw.strip() else ""
    m = re.fullmatch(r"SCORE:\s*(\d+)\s*/\s*(\d+)", final_line.strip())
    if raw.count("SCORE:") != 1:
        m = None
    if not m:
        return Score(self.name, 0.0, False, "judge produced no parseable score")
    got, total = int(m.group(1)), int(m.group(2))
    if total != len(self.criteria) or total <= 0 or not 0 <= got <= total:
        return Score(self.name, 0.0, False, "judge score outside the declared rubric")
    value = got / total
    return Score(self.name, value, value >= self.threshold,
                 f"judge scored {got}/{total}")

Fail closed for promotion, and label the measurement error. An unparseable judgement must not count as a pass. Record it as an evaluator failure separately from an agent failure so judge outages do not masquerade as capability regressions. Require one exact final score line, an integer score in range and a positive denominator equal to the number of criteria. A decimal suffix, embedded example or duplicate score line is not a valid result. A judge whose errors silently score as success will quietly inflate your metrics, and you will not find out until a customer does.

24.8Online evaluation#

Offline suites test what you thought to test. Production shows you what actually happens, and the two are not substitutes.

Four online signals, in ascending order of usefulness and cost.

Implicit feedback. Did the user accept the output, edit it, retry, or abandon? Free, high-volume, and noisy. Edit rate is often the single most informative implicit signal, because it measures how wrong rather than just whether.

Explicit feedback. Thumbs, ratings, reported issues. Low volume, heavily biased toward the annoyed, and still worth having because the reports are specific.

Sampled judging. Judge a sample of production traffic (for example, 2–5%) to detect quality or distribution shifts outside the offline suite.

Outcome tracking. The best and hardest: did the thing the agent did turn out to be right? Was the credit later reversed? Was the ticket reopened? Was the reconciliation correct at month end? This is ground truth, it arrives late, and it is worth building the plumbing for because nothing else is as trustworthy.

The pattern that ties offline to online: when an online signal fires, the case goes into the offline suite. A reported issue becomes a permanent test. That is the loop that makes the suite an appreciating asset rather than a snapshot.

24.9What benchmarks are for#

Public agent benchmarks are useful for calibration and useless as a substitute for your own suite, and being clear about the distinction saves a lot of wasted effort.

They are good for calibrating expectations — the original τ-bench pass^8 result demonstrates a reliability problem on its evaluated setup, not a current ceiling on every model or domain. They are good for model selection, as a first filter. They are good for communicating with people outside your team, who have heard of them.

They are bad for deciding whether your system works, because your tools, corpus, policies and users are not theirs. They are bad as optimisation targets, because a benchmark optimised against stops measuring what it measured. And published scores need reading with care: the same benchmark name can refer to different subsets, harnesses and scaffolds, and the harness often matters as much as the model.

The practical use: read the benchmark’s methodology for ideas about how to construct your own cases, then build your own.

24.10The scorer set#

Four scorers cover most needs, and the reason to have all four is that each isolates a different failure.

Efficiency deserves a note because it is the one teams omit. A correct answer at forty times the expected cost is a defect, and without a scorer for it a prompt change that doubles turn count while adding two points of accuracy looks like an improvement. Chapter 7’s arithmetic is what makes this concrete: turn count multiplies everything.

The aggregate report should carry the interval, not just the rate:

# Reporting shape: the actual implementation also lists failed cases.
def render(self) -> str:
    if not self.n:
        return "pass rate not estimated (no observations)"
    if self.repeats > 1:
        uncertainty = "cluster-aware interval required for repeated cases"
    else:
        lo, hi = self.wilson_interval()
        uncertainty = f"95% CI {lo:.1%}–{hi:.1%}; independent cases assumed"
    return f"attempts={self.n}, pass rate={self.pass_rate:.1%} ({uncertainty})"

Putting the confidence interval next to the pass rate in the default output is a small change with an outsized effect on decision quality, because it makes the noise visible to whoever reads the number rather than only to whoever thought to compute it.

24.11What the reference evaluator deliberately leaves out#

The central loop in code/reagent/evals.py separates case execution from scoring; it is not a production experimentation platform:

for case in cases:
    for i in range(repeats):
        result = runner(case)
        cid = case.id if repeats == 1 else f"{case.id}#{i + 1}"
        report.reports.append(
            CaseReport(cid, [s.score(case, result) for s in scorers], result))

Each repeat becomes a separate CaseReport, carrying the actual RunResult and every scorer’s output. This makes failure inspection honest: case-17#2 can link to the exact trace that failed while case-17#1 passed. CaseReport.passed then requires all scorers to pass. A strong outcome score cannot compensate for a forbidden trajectory or an exceeded budget.

The simplicity also marks the boundary. EvalReport.pass_rate is the fraction of run reports that passed. With 100 cases and three repeats, its n is 300. It does not group the three repeats by base case and compute the fraction that passed all three, so the printed rate is not itself pass^3. A production reporter must derive at least three separate values from those records:

Suppose 72 cases pass all three runs, 18 pass two of three, six pass one of three and four pass none. The run-level numerator is 72×3 + 18×2 + 6×1 + 4×0 = 258, so the attempt pass rate is 258 / 300 = 86%. The pass^3 result is 72 / 100 = 72%. The flaky-case rate is (18 + 6) / 100 = 24%. Reporting only 86% would erase both the reliability gap and the cases that create it.

The Wilson helper refuses repeats > 1, and the renderer explicitly requests cluster-aware analysis. Repeats share base-case difficulty. A production analysis should form uncertainty at the case level — for example, by resampling base cases with all their repeats attached — rather than claiming that three repeats tripled task diversity.

The scorers are equally deliberate seams:

That last row is easy to misread. The reference EvalCase has no reference_answer field, and the judge prompt receives the goal, final answer and tool names. The chapter’s recommendation to provide a reference answer describes the production design target; it is not a capability already present in the teaching harness. Add it as an explicit, versioned field before you claim reference-grounded judging.

The trajectory boundary matters even more. If Reference Agent calls issue_credit with the right tool name but the wrong tenant, the reference membership check sees the required name and passes. Production evaluation needs predicates over canonical arguments and an authoritative receipt showing what the external system accepted. A model-authored success message is not outcome evidence (Chapter 23).

There is one more distinction in the code worth preserving. CaseReport.weighted calculates a weighted mean of scorer values, while CaseReport.passed requires every scorer’s Boolean passed flag. Those numbers serve different consumers. The mean helps rank imperfect cases during error analysis: a trajectory score of 2/3 carries more information than zero. The Boolean controls the gate: partial compliance is still failure. If a dashboard treats mean_score >= 0.8 as success, an outcome score of 1.0 can offset a forbidden-call score of 0.0 and undo the design deliberately encoded in passed.

Test the adapter around these seams with fixed RunResult objects before spending tokens on live runs. A case that expects escalation should fail a completed result. A forbidden call should zero the trajectory even when all required tools appeared. An unparseable judge response should score zero. The reference tests demonstrate these invariants with a deterministic runner; your production extensions need the same tests for argument canonicalisation, receipt matching, missing runs and repeat grouping. Evaluation code decides what counts as progress, so a bug in it has the same blast radius as a bug in a release gate.

The same principle applies to online use. The module can score a reconstructed RunResult, but it does not fetch traces, redact tenant data, sample traffic, assign experiment versions or prevent a case from leaking into training. Those are platform responsibilities. Keep them outside the scorer, but make their versions part of the report. Otherwise a score cannot be reproduced.

Extend the evaluator as the decision requires, keeping sampling, execution, scoring and statistical analysis inspectable.

24.12Failure grouping is where the value is#

A pass rate tells you whether. An error taxonomy tells you what to fix, and the second is the one that produces improvement.

def error_taxonomy(report: EvalReport) -> dict[str, int]:
    """Group failures by their dominant cause.

    This is the mechanical half of error analysis. The judgement half — reading
    twenty failures and naming the pattern nobody had a word for yet — cannot be
    automated, and is where nearly all the improvement comes from.
    """
    buckets: dict[str, int] = {}
    for r in report.failures():
        failed = [s.name for s in r.scores if not s.passed]
        key = f"{r.result.outcome.value}:{'+'.join(sorted(failed)) or 'unknown'}"
        buckets[key] = buckets.get(key, 0) + 1
    return dict(sorted(buckets.items(), key=lambda kv: -kv[1]))

Group failures by signature, then inspect each group for a shared mechanism. Eleven loop_detected:trajectory failures may all trace to one unparseable tool error. Grouping narrows the reading; diagnosis still requires examining the runs.