Prompt Iteration & Feedback Loops

In this illustrative case, the team had been improving Reference Agent’s prompt for four months. The system prompt had grown from 900 tokens to 6,400. Each addition had been a response to a real failure: a rule about citing ledger lines, a warning about rounding, three sentences about when to escalate, a paragraph disambiguating two search tools. Every one had been justified.

Quality had been flat for three months.

The team then read two hundred failed runs and grouped them by cause. Four groups held 71% of the failures. One involved a tool returning an unparseable error. Another involved a retrieval index three weeks stale. The third required deleting two contradictory prompt rules. The fourth needed a verification step.

Two weeks of work on those four, and quality moved eleven points. Four months of prompt additions had moved it by roughly nothing.

Diagnose the failure before rewriting the prompt.

25.1The loop, and where teams get stuck#

Each improvement stage has a characteristic failure:

25.2Error analysis, as a method#

Use five steps adapted from qualitative analysis:

Sample properly. Take 50-200 failures. Not the ones someone complained about — those are biased toward the visible. Sample randomly from failures, and separately over-sample the expensive ones. If you have Chapter 4’s outcome enum, stratify by outcome so LOOP_DETECTED and BUDGET_EXHAUSTED are represented rather than drowned by COMPLETED-but-wrong.

Read them. Actually read them. Open the trace, follow the run, and write one sentence about what went wrong. Not a category — a sentence. This is slow, and it is the work. Twenty traces is an afternoon.

Open-code, then cluster. Let labels emerge from the sentences rather than sorting into pre-existing buckets. Pre-existing buckets are where you put the failure you already understand, which means they hide the failure you do not. Then merge labels that turn out to be the same thing.

Count, and order by frequency times cost. The largest pile is not always the most important — a rare failure that credits the wrong account outranks a common one that produces a slightly verbose summary.

Name the mechanism, not the symptom. “The agent got confused” is not a mechanism. “The search tool returned an empty JSON array with no explanation, and the model interpreted it as ‘the document does not exist’ rather than ‘this query found nothing’” is a mechanism, and it names its own fix.

25.3Half your prompt problems are not prompt problems#

The most useful output of error analysis is usually the reclassification. The following distribution is illustrative, not a measured industry baseline:

The prompt is a minority shareholder. In this distribution, roughly one failure in ten is fixable by prompt editing, and the fix is often deletion rather than addition. Teams spend most of their improvement effort on the thing responsible for the least of the failures because it is the most accessible lever.

Deleting rules is an underused fix. A 6,400-token prompt with a hundred accumulated rules contains contradictions, and contradictions produce inconsistency. The test for a rule: can you still reproduce the failure that motivated it? If not, delete it and measure. Most teams find quality unchanged and cost down — and occasionally, quality up, because a contradiction went away.

25.4Prompt CI/CD#

Once you know what to change, the change should be shipped like code, because it is code: it has inputs, control flow, a runtime contract, and it breaks silently when edited.

The versioned unit is never the prompt alone. It is the tuple:

# prompts/reagent-operating-contract/v14.yaml
version: 14
model: ${VERIFIED_MODEL_SNAPSHOT}  # resolve to an available provider ID at release
generation:                  # illustrative; provider-specific adapter validates
  max_output_tokens: 4096
  supported_sampling_config: ${PINNED_GENERATION_CONFIG}
tool_schema_hash: 9f3c2a01     # tool definitions are part of behaviour
prompt: |
  You are Reference Agent, an autonomous work agent.
  ...
eval:
  suite: regression-v7
  min_pass_rate: 0.86
  max_cost_per_task_usd: 0.42

This is a manifest template, not a provider request. Resolve placeholders and use parameters supported by the chosen endpoint; an arbitrary model name or thinking setting is not a valid pin. Everything in the resolved file changes behaviour. A team that versions the prompt text but lets someone bump temperature in an environment variable has versioned nothing — and the tool schema hash is the one people forget, even though Chapter 13 established that a tool description change alters behaviour as surely as a prompt change.

The pipeline has four gates.

The eval gate. A prompt change runs the regression suite and must clear a threshold. Blocking on this is what would have prevented Chapter 8’s opening incident, where a one-word edit cost nine points of quality for nineteen hours.

The cost gate. A change that improves quality two points and doubles cost per task needs a decision, not an automatic merge. Chapter 24’s efficiency scorer feeds this.

The paired diff. Which specific cases improved and which regressed. A change that fixes four and breaks one is a much more informative artefact than an aggregate rate, and the one it broke deserves a look before shipping.

Progressive rollout. Canary to a small traffic slice, watch online signals, widen. And keep the prior version one flip away, so a rollback is a label change rather than a deploy.

The authoring question — Git or a registry — has a settled answer: author in Git, deploy through a registry. Git gives review, blame and atomic rollback tied to code; a registry lets the change land without a deploy and lets someone fix a prompt at 2 a.m. Treat the registry as a cache of what Git already reviewed, and the label promotion as the gated step.

25.5A/B testing agents#

Comparing two versions on live traffic is harder for agents than for ordinary software, for three reasons worth planning around.

High variance. Non-determinism plus heterogeneous tasks means you need more traffic than you expect. Chapter 24’s arithmetic applies: plan the sample size for your effect, baseline and assignment unit; neither a few dozen nor a few hundred completions is a universal sufficient count.

Late outcomes. Whether a reconciliation was correct may not be known for weeks. Use leading indicators — edit rate, escalation rate, turn count, cost — and confirm against outcomes later.

Contamination. Memory (Chapter 10) and skills (Chapter 26) persist across runs, so an agent in arm B can write a memory that arm A later reads. Either scope the persistent stores per arm or accept that your experiment is not clean. This is a genuinely easy mistake and it invalidates the result silently.

Two mechanics that make agent A/B tests trustworthy. Assign by task, not by request, so a multi-run task stays in one arm and does not have half its work done by each version. And track cost per arm as a primary metric, not an afterthought — because a quality-neutral change that cuts cost 30% is a large win that an accuracy-only comparison scores as nothing.

The task is the minimum stable assignment unit, not always the final one. If the treatment can alter state that survives across tasks — user memory, tenant memory, learned skills or a shared index — widen assignment to the broadest scope that treatment can affect. “Assign by task” prevents within-task switching; it does not authorise cross-task contamination.

25.6Write the experiment contract before the variant#

An agent experiment needs a contract more precise than “compare prompt A with prompt B”. If you write that sentence and start routing traffic, every interesting result will trigger an argument about what the experiment was supposed to prove. Write the decision down before anyone sees an outcome.

The contract has seven fields.

Name one hypothesis. “B is better” is not a hypothesis. “Adding an explicit verification instruction will reduce unnecessary escalations by at least three percentage points without reducing task pass rate by more than one point or raising mean cost above $0.48” is. It names the mechanism, the expected movement and the trade-offs that would veto a ship decision.

Choose the unit of assignment. A task is the minimum: every request and turn in that task stays together. Then broaden assignment to the widest persistent scope the treatment can affect. Assign by task when a task owns its entire context and nothing treatment-affected survives it. Assign by user when preferences or conversational memory survive across tasks. Assign by tenant when Reference Agent learns procedures, shares retrieval state or writes tenant-wide memory. The broader unit buys isolation at the cost of fewer independent clusters. Forty thousand tasks from twenty assigned tenants are still much closer to twenty independent clusters than to forty thousand independent observations.

Declare eligibility. “Production traffic” is not a cohort. State which tenants, task classes, languages, channels, risk tiers and model regions can enter. Then freeze the rule. If finance reconciliations enter arm A while arm B happens to receive status summaries, the result measures routing, not the prompt. Log exclusions with reasons; otherwise a change in eligibility can masquerade as a treatment effect.

Pin the versioned tuple. Arm A and arm B differ in exactly the intended field. Model id, sampling parameters, tool schemas, retrieval configuration, guardrail policy and budget stay identical unless one of them is the treatment. Store the tuple hash on every trace. A label such as candidate is inadequate because it can be repointed halfway through the experiment.

Pre-register the metrics. Pick one primary outcome, the guardrail metrics that can stop the test and the diagnostic metrics that explain it. Do not promote whichever of fifteen metrics moves after the data arrive. For Reference Agent, the primary outcome might be verified task completion; guardrails might be incorrect writes, policy blocks and cost; diagnostics might be turns, tool retries and escalation reasons. Chapter 24’s trajectory contracts give you the machinery. The contract gives those measurements a decision.

Set the observation window. Some outcomes are known at run completion. Others mature when a human reviews the work, a customer pays an invoice or a reconciliation closes. Write both the earliest decision time and the maturity time. A two-day leading signal can stop an unsafe treatment, but it cannot approve a treatment whose ground truth takes thirty days.

State stop and ship rules. A safety violation can stop immediately. A cost breach might stop after a minimum exposure so one pathological task does not decide the experiment. A ship decision waits for the planned cohort, mature labels and all slices. Avoid continuous peeking followed by “ship when green”; repeated peeking turns ordinary noise into an apparently decisive result.

Consider an illustrative Reference Agent experiment. Sixty eligible tenants each contribute forty reconciliation tasks over fourteen days. Because tenant memory is shared, the tenant is the assignment unit: thirty tenants and 1,200 tasks per arm. The candidate adds a verification instruction. The contract requires at least a three-point reduction in escalation, no more than a one-point pass-rate loss, and mean cost below $0.48.

The candidate clears the point-estimate decision rules: pass rate improved rather than falling, escalation fell by four points, and $0.45 stays under the ceiling. The commercial interpretation takes one more line of arithmetic. B produced 38 additional verified passes and 48 fewer escalations for $84 extra. That is $84 / 38 = $2.21 of extra runtime cost per additional verified pass if you ignore the escalation saving. If an escalation costs $12 of operator time, B avoids 48 × $12 = $576 and leaves $576 - $84 = $492 net saving. Dividing by all 2,400 experiment tasks gives $0.205 per task as an experiment-wide accounting view, including the control. The rollout value follows the 1,200 candidate-treated tasks that generated the difference: $492 / 1,200 = $0.41 net saving per candidate-treated task.

Those calculations describe effect and value; they do not establish certainty. Because assignment happened by tenant, the tenant is the randomised cluster and uncertainty must follow that assignment structure; you cannot pretend that 2,400 within-tenant tasks were independently randomised. Reducing each tenant to an outcome rate and comparing the thirty rates per arm is one valid analysis. A cluster-robust model or an appropriate hierarchical model can retain task-level information while accounting for within-tenant dependence. Each option needs tenant-level identifiers and outcomes, which the aggregate table does not contain. A credible review says “the point estimates clear the thresholds; cluster-aware uncertainty and risk slices still gate shipping”. It does not manufacture precision from the larger task count.

Treat experiment integrity as a measured outcome, not a deployment assumption. The result table is downstream of a smaller health report that asks whether the planned comparison actually happened.

Suppose the health report for the 2,400-task example finds nine traces whose actual tuple does not match the assigned arm, fourteen reads from the other arm’s state namespace, and 2,100 mature labels. The arithmetic is direct: assignment mismatch is 9 / 2,400 = 0.375%; cross-arm state reads are 14 divided by the number of state reads, which you must record rather than substitute with task count; mature-label coverage is 2,100 / 2,400 = 87.5%. A dashboard that divides all three by 2,400 makes one number impossible to interpret and conceals the missing denominator.

The decision is not “those rates look small”. The pre-written contract might tolerate up to 0.5% assignment mismatch if those tasks stay under their intended assignment in the primary analysis and receive a separate exposure-sensitivity check, require exactly zero cross-arm persistent-state reads, and require 95% mature-label coverage before promotion. Under that contract you repair the state isolation and wait for another 180 mature labels: 95% of 2,400 is 2,280, and 2,280 - 2,100 = 180. You do not renegotiate the thresholds because the quality point estimate is attractive.

Emit these fields on the trace when the run begins, before any model or tool call. If you try to reconstruct them from the deployment log later, cache races and label changes will turn a factual exposure record into an inference. Keep intended assignment as well as actual exposure. Intended assignment supports the causal analysis; actual exposure explains protocol violations and operational failures. Both are needed when production behaves unlike the diagram.

Finally, read slices before rollout. The aggregate can pass while voice fails, a high-risk write cohort regresses or one model region absorbs the entire gain. Slice checks are guardrails declared in advance, not a licence to hunt for a flattering subgroup. If a pre-registered high-risk slice is underpowered, keep it out of the rollout and collect more evidence. “No evidence of harm” is not evidence of safety when the slice contains six tasks.

25.7Failure walkthrough: the canary won for the wrong reason#

Here is an illustrative incident whose numbers are deliberately simple enough to audit. Reference Agent tested a candidate prompt that asked one clarifying question before reconciling an ambiguous ledger. The router assigned each API request independently. After 1,000 completed tasks, the dashboard showed 413 verified passes out of 500 for A, or 82.6%, and 452 out of 500 for B, or 90.4%. The +7.8-point win looked large enough to ship.

The team assumed that random request assignment had balanced the traffic. It had balanced requests. It had not created two versions of an agent.

Three mechanisms broke the comparison. First, 368 tasks needed more than one user turn, and 173 of those — 47.0% — crossed arms between turns. B often asked the clarifying question, then A received the clarified answer and completed the task. The dashboard credited the completion to whichever arm handled the final request. A received part of B’s benefit; B received some tasks whose ambiguity A had already surfaced.

Second, 61% of the participating tenants sent traffic to both arms. Reference Agent’s tenant memory recorded facts such as the ledger’s base currency, while its verified skills recorded the reconciliation sequence (Chapters 10 and 26). A task in B could write useful state that a later task in A read. The experiment had isolated prompt strings but shared two of the mechanisms through which prompts change future behaviour.

Third, the trace recorded only the final arm label. It could not answer which prompt versions had contributed turns, which memory version had been read or whether a skill had been learned under the other treatment. The apparent 500-versus-500 comparison was therefore not recoverable after the fact. Randomisation cannot rescue an exposure you did not record.

The fix was not a more sophisticated significance test. The team changed the experiment topology.

1. It hashed the stable tenant id to one arm for the full experiment because persistent state was tenant-scoped.
2. It namespaced new memories and skill candidates by experiment_id, arm and version, while taking a frozen baseline snapshot that both arms could read.
3. It wrote an immutable exposure record on every run: intended arm, actual prompt hash, memory snapshot, skills snapshot, model id and tool-schema hash.
4. It excluded tenants exposed to both variants from the fresh cohort instead of trying to “wash out” durable state after one day.
5. It analysed tenant-level outcomes and retained task-level traces for diagnosis.

The rerun covered forty fresh tenants, twenty per arm, and 800 tasks per arm. A produced 676 verified passes, or 84.5%. B produced 681, or 85.1% to one decimal place: a +0.6-point point estimate, not +7.8. Mean cost rose from $0.37 to $0.43, a 16.2% increase. The candidate no longer cleared the improvement contract and was not promoted. Whether +0.6 was distinguishable from noise did not matter; the candidate had missed the minimum effect and cost conditions before statistical significance entered the decision.

The general lesson is sharper than “sticky sessions are useful”. Your assignment boundary must enclose every stateful path by which treatment changes a future observation. Conversation history, memory, cached context, learned procedures, human follow-up and even retrieval indexes can carry treatment across a boundary. Draw those paths before you choose the unit. If you cannot isolate one, record the exposure and change the question — perhaps from “which prompt caused this outcome?” to “does migrating the whole tenant improve outcomes?”

The incident also explains why a canary is not automatically an experiment. A canary answers an operational question: can this version run safely on limited traffic? An experiment answers a causal one: did this version change an outcome? You can use the same traffic for both only when assignment, isolation, measurement and analysis support both questions.

25.8Automatic prompt optimisation#

GEPA is one documented option. It samples trajectories, reflects on failures, tests prompt updates and combines complementary candidates on a Pareto frontier. Its authors report gains over GRPO and MIPROv2 across six evaluated tasks, with up to 35× fewer rollouts in its comparisons. These are benchmark-specific results, not a universal cost or quality guarantee; include the reflective model, evaluation and search costs in your own comparison.¹

Compare reflective prompt evolution with training. Test prompt optimisation against a manual baseline before paying for training, using held-out tasks and full cost accounting (Chapter 27).

The mechanism is error analysis, automated. Sample trajectories, diagnose in natural language, propose a change, test it. That is the loop in Figure 25.1, run by a machine. Which means the same precondition applies: it needs a signal. An optimiser without a good eval suite optimises against a bad target very efficiently.

Explore the Pareto frontier. Combining complementary lessons can reveal alternatives that a greedy sequence of individually justified edits misses.

The practical consequence: your eval suite is now the bottleneck twice over. It determines whether you can tell if a change helped, and it determines whether an optimiser can find a change worth making. Investment in the suite compounds into both.

What automated optimisation does not do: fix the 22% of failures that were tool error messages, or the 18% that were retrieval, or the 14% that were missing verification. It changes prompts and may improve tool use, but cannot repair a broken index or grant a missing verification capability by itself. Error analysis tells you whether you are in the region where an optimiser helps.

25.9The data flywheel#

Feed improvement outputs back into the next cycle:

Failures become eval cases. Every production failure, once understood, becomes a permanent case. This is the connection that makes the suite an appreciating asset — after a year it encodes every way the system has failed, and no regression can recur silently.

Corrections become training signal. When a human edits the agent’s output, the pair is data: for a few-shot example now, for distillation later (Chapter 27).

Trajectories become procedures. A task solved well is a candidate skill (Chapter 26), behind a verification gate.

Traces become the taxonomy. Chapter 23’s fleet view, grouped by failure signature, is where the next error-analysis session starts.

Provide a one-command path from a trace id to a new evaluation case to reduce the delay between discovering a failure and testing its fix.

25.10Where the feedback loop cannot decide for you#

A fast loop can converge on the wrong objective faster than a slow one. The loop in Figure 25.1 assumes that the observed signal represents the outcome you care about, that the labelled population represents the population you serve, and that yesterday’s definition of success still applies. None of those assumptions is permanently true.

Visible corrections are selected feedback. You see what users catch and choose to report. A finance analyst may correct a wrong currency immediately while accepting a subtly incomplete evidence trail that fails audit three months later. Easy tasks produce quick labels; abandoned tasks produce silence. If you train or tune only on explicit corrections, you learn the behaviour of reporters, not the failure distribution. Keep random outcome audits alongside user-initiated feedback, and compare the audited cohort with the feedback cohort.

Leading indicators can reverse at maturity. Suppose Reference Agent processes 10,000 reconciliations, 5,000 per arm. After two days, reviewers edit 570 outputs from A (11.4%) and 405 from B (8.1%). B appears to improve the edit-rate proxy by 3.3 points. Thirty days later, closed-ledger review finds 74 substantive defects in A (1.48%) and 121 in B (2.42%). B caused 47 additional defects, a 0.94-point regression, despite looking better on the immediate signal. The candidate learned to produce cleaner-looking work that reviewers changed less, not more correct work. A decision contract that names edit rate as a diagnostic and closed-ledger defects as the primary outcome survives this reversal. A dashboard that calls both “quality” does not.

Policy changes invalidate old labels. If the business lowers its approval threshold from $10,000 to $5,000, an escalation that was correct in June can be wrong in July with identical inputs. Store the policy version and decision time with every label. Replaying old cases under the new policy requires relabelling them; silently mixing both definitions teaches the optimiser a contradiction and makes the trend unreadable.

Feedback has an authority boundary. A user preference such as “put the variance table first” can safely alter presentation for that user. A comment such as “skip approval next time” cannot rewrite an enterprise control. Treat feedback as an observation routed to an owner, not as an instruction automatically promoted into the system prompt or memory. Chapter 21’s trust boundaries apply to improvement data too: feedback is input from a principal with specific authority, not ground truth by virtue of arriving through a feedback button.

Some trade-offs are product decisions. The experiment can tell you that B improves verified completion by 2.1 points, adds $0.09 per task and increases median latency by four seconds. It cannot decide how much latency a voice user will tolerate, whether a regulated cohort should accept a different threshold or whether $0.09 is worth the gain. Encode the decision after an accountable owner makes it. Do not conceal the choice inside a composite score whose weights nobody owns.

For every production feedback signal, maintain a small contract:

This contract turns “collect more feedback” into an instrument you can reason about. It also exposes missingness. If only 12% of eligible tasks receive a human label, report 12% beside the score. If label coverage falls from 12% to 4%, a stable pass rate is not a stable system — it is a less observed system.

Keep a comparison when the environment itself moves. If you promote B to everyone and verified completion falls next week, the cause could be B, a retrieval-index change, a new task mix or a provider regression. A small, policy-compliant holdback on the prior tuple gives you a contemporaneous reference. On 10,000 eligible weekly tasks, a 5% holdback is 500 tasks and the candidate receives 9,500. If both groups fall by four points on the same day, investigate shared infrastructure and task mix before reverting the prompt. If only the candidate falls, the version becomes the leading suspect. This is monitoring evidence rather than proof by itself, but it shortens the diagnostic path.

Do not create a permanent holdback where withholding a demonstrated safety improvement would expose people or data to avoidable harm. In that case, compare risk-equivalent safe variants, use offline replay, or shorten the holdback to the minimum operational validation window. The contract must name that ethical boundary before the winning result tempts the team to keep an obsolete arm alive for convenient measurement.

Also preserve an untouched evaluation holdout for optimisation. The production holdback answers how versions behave under current traffic; the hidden eval holdout asks whether repeated iteration has fitted the visible suite. They are different controls. If the visible suite rises from 86% to 94% across twelve edits while the hidden holdout stays at 85%, the loop has learned the test, not the task. Stop proposing variants, refresh the sampled cases, and rerun error analysis on the holdout failures. More optimisation against the same target will widen the gap.

The counter-argument to relentless iteration is therefore simple: stop changing the prompt when the decision surface is not trustworthy. Repair label coverage, wait for mature outcomes, reconcile policy versions or ask the product owner to resolve the trade-off. More variants do not compensate for an undefined target. Automatic optimisation makes this boundary more important, because it can exploit a weak proxy with far more persistence than a human prompt editor.

The irreducibly human work is not typing prompt prose. It is deciding which failures matter, which evidence deserves authority and which trade-offs the system is allowed to make. Keep that boundary explicit and the loop compounds knowledge. Blur it and the loop compounds error.

25.11Anti-patterns#

Prompt accretion. Rules added, never removed, until the contract contradicts itself. The counter-practice: a quarterly review where every rule must justify itself against a reproducible failure.

The single-case fix. A prompt edit motivated by one complaint, evaluated on that one case. It will fix that case and may break others invisibly. Always run the suite.

Changing several things at once. Then one number moves and attribution is impossible. One change per evaluation, and if that feels slow, the problem is that your suite is slow.

Optimising against the suite. After enough iterations against a fixed suite, you are fitting it. Hold out cases the optimiser never sees, and refresh the suite with new production samples periodically.

Confusing the model’s opinion with a signal. “I asked the model whether the new prompt was better and it said yes” is not evidence. Chapter 22’s finding applies: self-assessment without ground truth is unreliable.

Improving the prompt when the problem is upstream. Diagnose context, retrieval, tools and verification before choosing a prompt edit.