Self-Learning

In an illustrative case, Reference Agent’s team considers training on its own data after eight months in production.

They had the data. Four hundred thousand completed runs, each with a full trajectory — the goal, the context assembled at each turn, every tool call and result, the terminal outcome, human approvals and refusals, and for a meaningful subset, whether the outcome later turned out to be correct. In this scenario, the production team deliberately retained this authorised dataset and joined the audit log to model-context snapshots and later outcome labels. The teaching log alone does not contain all of it.

The team first rejected training, then approved it for one narrow task. A week of arithmetic supported that decision before anyone committed GPU time.

27.1The ladder#

The ladder is a diagnostic heuristic: investigate tool, context and procedure failures before paying for model training. Costs and iteration times overlap, and a proven training route may be appropriate early. Choose the smallest intervention with evidence that it addresses the diagnosed failure.

The reason this ordering matters more than usual is a finding from Chapter 25 that deserves restating here in its training context: reflective prompt evolution has been shown to outperform reinforcement learning by an average of 6% and up to 20% while using up to 35× fewer rollouts.¹ Fewer rollouts do not imply the same reduction in total compute or dollars: reflection, evaluation and model choice also cost money. Treat the six-task GEPA comparison as motivation for a local baseline, not a universal ranking.

27.2What you already have#

The event log supplies candidate run evidence. It becomes training data only after checking collection scope, provenance, rights, privacy, labels and quality. The reference log does not capture every assembled model context or export a training dataset.

A trajectory is one complete run: the goal, the context at each turn, every tool call and result, the model’s outputs, and the terminal outcome. Its potential value is relevance to your distribution: your tools, your corpus, your policies, your users’ actual phrasing.

Hermes documents ShareGPT-compatible JSONL with conversation, model, completion and tool statistics, and separate success/failure files. Its batch runner discards samples without reasoning text.² That is a dataset-specific choice, not a universal quality criterion: useful training examples can lack rationales, and many APIs do not expose private model reasoning. Export only available, authorised content and define filters against the training objective.

Apply four filters before using trajectories for training:

Outcome filtering — keep verified task outcomes for the chosen training objective. COMPLETED means the loop returned an answer, not that the answer was correct. Include correct refusal, clarification and escalation examples; an answer-only dataset trains away those behaviours.

Trajectory filtering — only runs whose route was acceptable. training on outcome-correct-but-trajectory-wrong runs teaches the model the routes you refuse to sanction, and does so at scale.

Efficiency filtering — prefer efficient trajectories that retain required verification. The shortest path is not always the safe or successful one.

Privacy filtering — trajectories contain customer data. Scrubbing is mandatory and non-trivial, and the consent question for using customer data to train is Chapter 30’s.

And the filter most easily forgotten: label recovery and human intervention explicitly. Such trajectories can teach useful correction when the objective and loss masking distinguish rejected actions from accepted continuations. Do not label every assistant action positive because the run eventually succeeded, or delete turns to create an impossible context.

27.3Distillation: the highest-return training#

A narrow distilled component is often a useful first training experiment. Distillation trains a student from teacher outputs or distributions. Lower cost at acceptable quality is the objective here; it is not guaranteed, and distillation can also target other properties.

The setup: your frontier model produces good trajectories. You fine-tune a much smaller, much cheaper model on those trajectories for a specific task type. Evaluate whether the student meets that task’s quality contract at lower total cost and acceptable latency.

A measurable success criterion. Match the teacher’s quality on held-out tasks at lower cost.

The data requirement is empirical. Build a learning curve on held-out tasks; task diversity, target behaviour and student capacity determine how much data is enough.

It is reversible. The distilled model is a route in your gateway (Chapter 29). If it underperforms, route back. This is the property that makes it a safe experiment, unlike rung 8 and 9 where a bad training run costs weeks.

Potential cost savings. A cheaper model on a high-volume narrow role can reduce repeated inference costs substantially.

The candidates are the narrow, high-volume, well-defined steps in your system rather than the whole agent: classification and routing (Chapter 7), query formulation (Chapter 11), compaction summarisation (Chapter 9), extraction and normalisation, guardrail classification (Chapter 20). Notice that most of these are not the agent’s reasoning core — they are the supporting cast, and that is where distillation pays.

This was Reference Agent’s eventual “yes.” Not a fine-tuned agent — a distilled classifier for the routing step and a distilled summariser for compaction, both of which ran on every task and both of which were narrow enough to verify completely.

27.4Make the distillation case clear its full cost#

Training cost is the conspicuous number and usually the least important one. The decision turns on the full serving delta: model tokens, inference capacity, gateway overhead, evaluation, data curation, retraining and the engineering time required to operate another model route. A cheaper call is not yet a cheaper system.

Take an illustrative Reference Agent routing candidate. It runs 1,800,000 times per month. The frontier route consumes 650 input tokens and 40 output tokens per call. For this worked example, use $5.00 per million input tokens and $15.00 per million output tokens. The per-call cost is therefore:

(650 × $5 / 1,000,000) + (40 × $15 / 1,000,000) = $0.00385

Across 1,800,000 calls, that is $0.00385 × 1,800,000 = $6,930 per month. A distilled candidate at the same token counts, using illustrative rates of $0.25 and $1.00 per million tokens, costs $0.0002025 per call, or $364.50 per month. That looks like a $6,565.50 monthly saving. It is not.

The recurring saving is $3,165.50 × 12 = $37,986 per year. If the first dataset and deployment cost $28,000 of engineering time, simple payback is $28,000 / $3,165.50 = 8.85 months. These are illustrative rates, not vendor prices. Their value is the denominator: a proposal that reports only the 94.7% reduction in model-call cost has hidden more than half the operating cost of the candidate route.

Quality needs the same treatment. Suppose the locked suite contains 500 cases, each run five times, for 2,500 paired observations. The teacher passes 2,400 and the student 2,384. The observed rates are 96.00% and 95.36%, a loss of 16 passes or 0.64 percentage points. If the release contract permits at most a one-point loss, the point estimate clears it. That is not enough to declare equivalence: calculate the paired uncertainty and inspect all 84 discordant observations — 50 teacher-only passes and 34 student-only passes — because the aggregate can conceal a complete collapse on one high-risk slice (Chapter 24).

Now stress the volume assumption. At 900,000 calls per month, the model-call saving halves to $3,282.75 while the $3,400 monthly serving and operation cost remains. The candidate loses $117.25 per month before repaying the first dollar of the initial build. The break-even volume follows directly:

$3,400 / ($0.00385 − $0.0002025) = 932,145 calls per month

That threshold belongs in the routing configuration and the quarterly review. A correct distillation decision can become wrong when traffic falls, a provider cuts its price, or the narrow step stops being narrow.

The cost calculation still has one missing term: the price of a quality regression. Do not invent a universal dollar value for a failed task. Segment by consequence. If 1.5% of those 1,800,000 monthly routes are high-risk reconciliation requests, that is 27,000 runs. Suppose the candidate adds 0.20 percentage points of failure in that slice — 54 additional failures — while improving the large low-risk slice. An overall quality loss below one point would happily admit it. A release policy that says no additional high-risk failure clears the ambiguity immediately. Estimate handling or remediation cost only after the safety gate; money must not buy permission to fail unsafely.

Latency can reverse the case too. Measure end-to-end time at the same concurrency and request mix, not model-reported generation time. A self-hosted candidate that is fast in an isolated benchmark can queue behind its own traffic and turn a 120-millisecond routing step into a two-second bottleneck. Include p50, p95 and p99 latency, queue time and saturation at the intended canary volume. If you need enough spare capacity to survive a burst or a host failure, price that spare capacity into the $900 line instead of treating peak resilience as free.

Finally, compare the candidate with the cheapest untrained alternative. A cached deterministic rule may handle stable, repeated classifications. A prompt change may cut the teacher input from 650 tokens to 300. A batch route may move non-interactive traffic away from the real-time gateway. Distillation wins only against the best lower rung, not against an incumbent you deliberately left unoptimised. The ladder is an order of experiments as well as a list of techniques.

27.5Supervised fine-tuning on trajectories#

The next rung: fine-tune a capable model on your own successful trajectories so it behaves more like your best runs by default.

Format and protocol adherence. Consistent structured output, consistent tool-call shapes, consistent adherence to your conventions.

Domain vocabulary. Your entity names, your abbreviations, your conventions — knowledge that otherwise consumes prompt tokens on every turn.

Tacit preferences. Which tool to reach for first in your environment; how much to verify before writing. Things that are hard to state as rules and easy to demonstrate.

Shorter prompts. Behaviour baked into weights is behaviour you no longer pay for on every turn. For a long-horizon agent this can be a material cost saving.

What it does not buy, and this is where expectations break: improvement is not guaranteed. Supervised fine-tuning can improve task skills and reasoning performance, but can also overfit or degrade other capabilities. Test the claimed gain on unseen tasks; training cannot supply missing live facts, repair a broken tool, or replace runtime permission checks.³

You freeze your current behaviour. A model fine-tuned on last quarter’s trajectories has learned last quarter’s tools, conventions and mistakes. Every subsequent change in your system moves away from what it learned. Plan to re-train, and plan for that to be recurring cost.

You can lose generality. Narrow adaptation may degrade performance outside the training distribution. Measure on a held-out general set, not only on your own suite, or you will not notice until an unusual request arrives.

You are now coupled to a model version. Upgrading the base model means re-training. This is a real operational commitment, and it is the reason the versioned-tuple discipline from Chapter 25 needs to include the fine-tune id.

27.6The fine-tune that learned yesterday’s interface#

In this illustrative Reference Agent incident, the first whole-agent fine-tune failed without producing obviously worse prose. That made it more dangerous than a model that simply stopped answering.

The team trained on 14,000 accepted trajectories from the harvesting funnel in Figure 27.2. The offline suite passed 96.8% of cases, ahead of the base route’s 95.9% on the same 1,000 single-run cases. Responses were shorter, tool selection looked decisive, and the tool-call JSON was unusually consistent. The result appeared to validate every promised benefit of supervised fine-tuning.

During the six weeks between the oldest harvested run and the canary, however, the CRM search tool had moved from schema version 3 to version 4. Version 4 required tenant_scope and replaced a free-text account argument with a canonical account_id. The prompt and live tool schema described version 4. Most training examples demonstrated version 3. On the common path, account names happened to resolve and the defect stayed quiet. On requests containing two similarly named accounts, the fine-tuned route repeatedly emitted the old shape. The schema validator rejected it, the model repaired it, and the run finished one or two turns later.

The first dashboard still looked healthy. Terminal completion was 96.1% for the candidate and 96.0% for control across the 2% canary. Mean cost was worse, but traffic mix changed that week and the difference was dismissed. The decisive signal was narrower: among 640 canary runs that selected CRM search, 173 made an invalid first call, against 21 of 638 controls. That is 27.0% versus 3.3%, with explicit denominators. The candidate had learned an obsolete interface more strongly than the current prompt could correct it.

Everyone first blamed weak tool training. The actual mechanism was version leakage through the dataset and suite. The export did not attach the tool-schema hash to each example. The evaluation set had been drawn from the same historical window, so 94% of its CRM cases also used version 3. A high pass rate proved fidelity to the stale distribution.

The fix was not a larger fine-tune. The team joined every trajectory to the tool-registry version active at the event timestamp, removed examples whose interfaces were no longer accepted, and added schema-version strata to the suite. They also made an invalid first tool call a trajectory failure even when the model repaired it. The candidate was re-trained on version-4 examples, shadowed against live requests, then canaried again. Only after the candidate’s first-call validity cleared the base route on every named-account ambiguity slice did the gateway widen traffic.

A trajectory binds together a model, prompt, context policy, tool schemas, retrieval corpus, permission policy and evaluator. Training strips behaviour out of that tuple and puts it into weights; if the export loses the tuple’s versions, you cannot tell which world the model learned. Chapter 25’s experiment record is therefore not optional metadata for training. It is the schema of the dataset.

This incident also exposes why repaired success is not clean training data. The 173 invalid first calls mostly ended in completed runs, so an outcome-only export labelled them positive. Their transcripts contained both the obsolete call and the correction. For supervised training, keeping the entire exchange teaches that an invalid attempt followed by repair is a normal path. Keeping only the corrected call erases the evidence that the model needed recovery and may splice together a context that never occurred. Possible choices include a negative or preference pair, exclusion from positive demonstrations, or a recovery-training example with the rejected action masked from positive loss. Do not manufacture a pristine demonstration by deleting the failure from history.

The canary denominator matters for the same reason. Reporting 173 invalid calls out of every canary run would dilute the rate with requests that never used CRM search. The relevant denominator is the 640 candidate runs in which that interface was selected. Always measure an interface regression among runs exposed to the interface, then publish the all-traffic effect separately for capacity planning. One number diagnoses behaviour; the other estimates blast radius.

Preserve both the rejected call and its validator response in the trajectory. Without the response, later reviewers can see that the model changed course but cannot distinguish schema rejection from spontaneous reconsideration. Training governance depends on the same honest event record as crash recovery and audit.

27.7Reinforcement learning, and where the verifier comes from#

Reinforcement learning depends on the source and quality of its reward signal.

Verifiable rewards. The task has ground truth: the tests pass, the sum reconciles, the query returns the right rows, the proof checks. This is the setting where RL for agents has produced its most striking results, and the reason is that the reward can be checked against an external condition. The verifier can still be incomplete or exploitable, including through test tampering.

Model-judged rewards. A model scores the output. Cheaper to obtain, available for far more tasks, and it inherits every problem from Chapter 24’s judge calibration plus a new one: the policy is now optimising against the judge, which means any exploitable gap in the judge will be found.

Human preference. Expensive, slow, and the traditional path.

The practical guidance is unusually clear-cut: choose the reward and its independent evaluation before choosing RL. Model- and human-judged rewards are legitimate training signals, but their bias and exploitable gaps require calibration, held-out checks and monitoring. External verifiers also need those controls; none makes reward hacking impossible.

Use the earlier GEPA comparison as one baseline candidate. Select the optimisation method by full cost and held-out performance on your task.

27.8Reward hacking#

When you optimise a proxy, the policy finds the gap between the proxy and what you meant. This is not a failure of the method; it is the method working on the objective you actually specified.

Concrete agent-shaped examples:

The last two are the ones that generalise beyond training. A prompt optimised against a judge learns the judge; a system optimised for low guardrail trip rates can achieve it by finding paths the guardrails do not watch. Any optimisation loop — including the automated prompt optimisation on rung 6 — has this property, which means held-out evaluation is not optional at any rung.

Verify the verifier. If tests are the reward, check that the tests were not modified. If completion is the reward, check completion independently of the agent’s claim.

Hold out a suite the optimiser never sees. A useful independent check for proxy divergence; keep it held out from optimisation and combine it with production audits.

Keep a human sample. Read a random sample of high-scoring outputs. A reviewer may notice exploitative behaviour the metric misses; subtle failures still need independent instrumentation and targeted tests.

Score trajectories as well as outcomes. Detect unacceptable routes even when their final answers pass.

27.9Continual learning and forgetting#

The appealing idea of an agent that learns continuously from production runs the hard way into catastrophic forgetting: training on new data degrades performance on old tasks, and the degradation is not gradual or predictable.

For an agent this is particularly bad because the new data is biased toward recent traffic. A month of unusual volume in one task type produces a model that is better at it and worse at everything else — and nothing in the training signal reports that.

Four mitigations, and the fourth is the one to prefer.

Replay. Mix historical data with new data so old distributions stay represented. Standard, effective, and requires keeping the historical data, which has retention implications (Chapter 30).

Parameter-efficient adaptation. Adapters or low-rank updates rather than full fine-tuning. Constrains how far the model can move and makes reverting easy.

Periodic re-training from base rather than incremental updates on updates. Slower, and avoids compounding drift.

Don’t put learning in the weights at all. Skills (Chapter 26), memory (Chapter 10) and retrieval (Chapter 11) are all forms of learning that are inspectable, revocable and outside the weight-interference mechanism of catastrophic forgetting, though still subject to stale data, retrieval failures and accidental deletion. For nearly every enterprise agent, this is the right answer: the parts of the system that should learn continuously are the parts that are not weights.

27.10Learning proposals are releases#

The phrase self-learning system tempts you to draw a closed loop from production outcome back to model weights. Do not. A production outcome is evidence for a change proposal, not permission to deploy one. The safe loop contains the same separation as Reference Agent’s action boundary: the system may propose; an evaluation and release policy decides.

The reference harness records and replays events, returns run outcomes and usage, and scores outcome, trajectory and efficiency. Memory and skill modules change later recall; the verification gate controls whether proposed skills become retrievable. It does not export training datasets, join late business outcomes, update weights, train models or route candidates into production. Those require a separate release pipeline.

The extension should behave like a release pipeline with evidence attached at every transition.

Harvest a candidate snapshot. Export immutable run identifiers and version fields from the event log. Join late outcomes without rewriting the original events. Record applicable data-use rights and authorisation, redaction policy, deduplication rule and exclusion reason per row. A dataset version is a manifest over exact rows, not a query you promise to run again later.

Train outside the serving path. A job consumes that manifest and produces an immutable candidate identifier plus its configuration and checksums. It cannot update the gateway. That separation prevents a successful job from silently becoming a production model.

Evaluate on locked and live-shaped evidence. Run the frozen regression suite, held-out cases, repeated reliability trials and cost scorer from Chapter 24. Then shadow current traffic without allowing the candidate to act. A static suite catches known regressions; shadowing catches drift in request and tool distributions.

Promote by policy, then canary. Promotion requires named evidence and an accountable approver. The gateway sends a small, tenant-aware fraction to the candidate and retains the prior route. Rollback changes routing, not weights, and must fit inside the response time your highest-risk task permits.

Do not combine these into a learning score. A weighted average lets cheap calls compensate for unsafe trajectories and plentiful labels compensate for missing consent. Treat privacy, trajectory safety, quality and rollback as gates. Cost is a ranking criterion only after those gates pass.

The cadence also matters. Dataset health can run daily, a model candidate may be trained monthly, and promotion can happen only when evidence warrants it. Continuous observation does not require continuous deployment. Your learning rate is the rate at which verified improvements reach users and remain good, not the rate at which weights change.

Three authorities should stay separate even in a small team. The evidence job may assemble a dataset and evaluation report but cannot declare that a candidate passed. The release policy may decide that evidence clears a contract but cannot rewrite the evidence or expand tenant eligibility. The gateway may route only a candidate named by an approved release record and cannot decide that good live metrics justify widening itself. Separate services are useful; separate credentials and audit events are the actual boundary (Chapter 30).

Every widening decision creates a new experiment version. If the canary moves from 2% to 10%, record the candidate, control, tenant allocation, start time and release evidence again. Do not mutate one long-running experiment record. Live traffic, caches and upstream tools can change between stages, and an immutable record lets Chapter 23’s trace tell you which conditions produced a result. It also prevents the final 100% rollout from inheriting approval that was granted for a 2% exposure.

Late ground truth can still overturn a promotion. A routing candidate may look equal at task completion and then produce more reopened cases fourteen days later. Keep the prior route deployable through the outcome horizon, join those delayed labels to the experiment version, and define what happens when the later metric breaches its gate. Rollback is not only an incident response during the first hour; it is the planned answer to evidence that matures after the canary has widened.

This is where non-weight learning retains a structural advantage. In a production memory store, a poisoned memory can be deleted or quarantined — controls the shipped MemoryStore does not implement — while its event evidence remains. In the reference skill library, repeated failed uses can move a skill to DEPRECATED; a retrieval index can be rebuilt and a model route changed without erasing the event record. That does not make those mechanisms safe by default — Chapters 10, 11 and 26 specify their gates — but it makes the learned artefact inspectable and its authority local. Weight changes collapse many behaviours into one opaque object. The release evidence must become stronger as reversibility and attribution become weaker.

27.11When a team should actually train#

A checklist. Train when you can answer yes to most of these.

• Error analysis says the failures are behavioural, not upstream. You have done Chapter 25’s session and the top clusters are not tool errors, retrieval or verification.
• Rungs one through six are exhausted, including automated prompt optimisation.
• You have thousands of high-quality filtered trajectories — outcome and trajectory verified, efficient, scrubbed, deduplicated.
• You have an eval suite that can detect the change you expect, with a predeclared paired or cluster-aware power analysis (Chapter 24).
• You have a held-out suite the training process never touches.
• The task is narrow and stable. Not “our agent” but “the routing classifier.”
• You can afford to re-train when the base model, your tools or your conventions change — and you accept that as recurring cost.
• Someone owns it. Fine-tuned models rot, and an unowned one becomes an unexplained dependency.

For Reference Agent, the honest score on the whole agent was three of eight, and on the routing classifier it was eight of eight. This illustrative result motivates asking which bounded component has measurable value before training the whole agent.