Enterprise Governance & Compliance

In this illustrative case, the question that stopped Reference Agent’s largest deployment was four words long: “Who did this action?”

The auditor was looking at a ledger correction. The database audit log said the change was made by svc-reagent-prod. That was accurate and useless. It did not say which human the agent was acting for, which run, under whose authority, on what evidence, or whether anybody had approved it. The organisation could not answer a question about its own systems, and until it could, the deployment did not widen.

The repair touched five components. Designing attribution before integration would have avoided much of that rework; the time required depends on the identity and audit systems already available.

30.1Governance is a design input#

Three governance requirements shape architecture from the start:

Identity. Who the agent is, and on whose behalf it acts, must be established at the point of action and carried into every downstream system. Retrofitting means changing every integration.

The audit record. What happened, who authorised it, and what they were shown must be recorded as it happens. It cannot be reconstructed afterwards, because the thing an auditor most wants — what the approver saw — exists only at that moment.

Data boundaries. Where data may go and how long it may stay constrains your storage, your trace store, your provider choice and your caching. Discovering a residency requirement after building is a migration.

Other requirements, including affected-person rights and evaluation, can also shape architecture. Treat these three as starting questions, not an exhaustive governance checklist.

30.2Agent identity#

An agent operates through workload identities while selecting actions at runtime. Existing identity governance still applies, but credentials and attribution must cover the range of actions each run can propose.

Three patterns can be combined according to the use case and required attribution.

Shared service account. One identity with the union of every permission any task might need. Simple, and it produces the svc-reagent-prod answer: accurate, useless, and in many regulated contexts insufficient for attribution.

Delegated user authority. The agent acts with the requesting user’s authority via token exchange. The token exchange must preserve the actor chain and narrow the requested scope. Revoking the user does not automatically revoke every issued token or downstream session; enforce expiration and test revocation propagation.⁵⁶ The operational cost is that capability varies by caller — “it worked for Priya and not for me” is a support burden worth planning for.

Per-task scoped credentials. A credential minted for this task, these resources, this window, and nothing else. This narrows exposure and can be combined with delegated user authority; it requires a credential-issuing and enforcement service.

Whichever pattern you choose, define four engineering objectives and map them to the deployment’s actual obligations.

Downstream actions have attributable identities. Preserve the workload, run and accountable principal where required. agent=reagent run=4471 on_behalf_of=priya@corp in the database audit row.

Authority has a central revocation control, with a measured propagation deadline and no application deployment required (Chapter 20).

Credentials are short-lived and scoped. Keep credential values out of model-visible text and deliver them through a trusted adapter or proxy (Chapter 15).

The identity is inventoried. It appears in whatever register your organisation keeps of machine identities, with an owner and a review date. An agent identity nobody owns is an agent identity nobody revokes.

30.3The audit chain#

The auditor’s question decomposes into five, and a defensible system answers all five from one record.

Chapter 5’s event log supplies audit facts. A local append-only JSONL file is not a tamper-evident, access-controlled, retention-governed audit service; the production system must provide those guarantees. The request/decision pair is the part most systems omit:

log.append(EventType.APPROVAL_REQUESTED, tool=call.name,
           arguments=call.arguments, reason=verdict.reason)
answer = self.approver(call, verdict)
log.append(EventType.APPROVAL_DECIDED, tool=call.name,
           granted=answer is True,
           note="" if answer is True else str(answer))

Two properties of that pair matter to an auditor and to nobody else until an incident. It establishes latency to decision, which is how Chapter 19’s four-second median became visible and therefore a finding. And it should record what the approver was shown — the rendered request, not a reconstruction — because “the reviewer approved this action” is only meaningful if you can produce what the action looked like to them.

Six properties distinguish an audit log from a trace store (Chapter 23): it is complete, never sampled; integrity-protected, with append-only history and governed redaction; attributed to accountable principals and workload identities; retained for the applicable obligation, not a universal number of years; schema-stable and versioned; and access-controlled, because it now contains a concentrated record of everything the agent has touched.

30.4Data: four questions, asked early#

Residency. Which jurisdictions may this data enter? This constrains your provider, your region, your trace store, and — the one people miss — your cache. A prompt cache is a copy of the data, held wherever the provider holds it.

Minimisation. Does the model need this field? An agent assembling a customer record into context is processing all of it, and the fields nobody needed are pure risk. Chapter 8’s assembler is where minimisation is enforced, which is a nice property: the component that controls cost also controls exposure.

Retention. How long do traces, event logs, memory and trajectories persist? These have different answers, and the trajectory corpus from Chapter 27 is the one that surprises people — a training dataset is a long-lived copy of customer data.

Deletion. When a customer asks for erasure, what has to happen? This is the hard one, because agent data propagates: an event log, a trace store, a memory store (Chapter 10), a knowledge graph (Chapter 12), a retrieval index (Chapter 11), a trajectory corpus (Chapter 27), and any compaction summary or skill derived from any of them.

That propagation is why Chapter 10 insisted provenance be mandatory. A memory with a recorded source can be found and deleted when its source is deleted; one without cannot. The same argument applies to graph edges, retrieval chunks and derived summaries — every derived artefact needs a link back to its sources, or your deletion story is incomplete and you will not know by how much.

30.5A deletion request that stopped at the index#

In an illustrative Reference Agent failure, a tenant left the service and the operations team ran the documented deletion job. The primary customer rows disappeared. The event-log partition was removed. The object-store prefix was empty. The ticket closed with the status complete.

Three weeks later, a retrieval evaluation returned a sentence containing the former tenant’s internal project name. The sentence came from a vector index built from documents Reference Agent had read during support runs. The source object was gone, but the chunk remained because the index deletion job selected by document id while the ingestion path had stored only a content hash. Nobody had tested deletion from an observed retrieval result back to the source; they had tested each store from the source outwards and treated an empty primary bucket as proof.

The first assumption was that the model had memorised the phrase. The second was that an old prompt cache was serving it. Both sent responders towards systems they did not operate. The trace settled the question: the retrieval tool returned the exact chunk id before the model generated the answer. The actual mechanism was a broken provenance join.

The failure had propagated further. Two episodic memories cited the stale chunk as evidence, one knowledge-graph edge had been extracted from it, and a compaction summary contained a paraphrase without a source id. Deleting the vector did not delete those descendants. The team could find the first three because they carried provenance. The summary had to be located with a tenant-scoped content search and reviewed manually, because its lineage had been discarded at compaction.

The fix was not a longer deletion checklist. It was a deletion protocol with closure evidence:

1. Assign every source object a stable provenance id before ingestion.
2. Require every derived record — chunk, edge, memory, summary and trajectory sample — to carry one or more parent provenance ids.
3. Maintain a reverse index from parent to descendants so deletion can traverse outwards.
4. Mark the source tombstoned first, which blocks new retrieval while the traversal runs.
5. Delete or re-derive every descendant, then record the count by store.
6. Verify from the outside by querying each serving path for the tombstoned provenance id and tenant.
7. Leave a minimal deletion receipt containing identifiers, counts, time and verifier — not the deleted content.

Assume the source document produced 48 chunks. Those chunks contributed to 11 memory records, 7 graph edges, 3 compaction summaries and 2 trajectory samples. The deletion fan-out is therefore 48 + 11 + 7 + 3 + 2 = 71 derived artefacts, plus the source. If the job reports 68 removals, 68 / 71 is not “95.8% complete.” It is incomplete. Deletion is a closure property: either every reachable descendant is removed or transformed under an approved rule, or the request remains open with three named exceptions.

This arithmetic is deliberately unforgiving. Availability targets can tolerate a small failure fraction because each request is independent. A deletion graph is connected: one surviving summary can continue to seed later context and create new descendants. The useful metric is not average deletion rate but open descendants after verification, with a target of zero.

Deletion evidence covers only inventoried stores and processors. Document separate constraints for provider caches, backups, legal holds and exported artefacts with privacy and legal owners. A primary-database deletion does not prove complete deletion.

The incident generalises beyond deletion. Provenance is what lets you withdraw poisoned knowledge (Chapter 21), re-evaluate memories after a source changes (Chapter 10), and identify which decisions depended on a faulty document (Chapter 24). Treating lineage as optional metadata creates four different incident-response problems from one shortcut.

30.6The EU AI Act, checked 9 November 2026#

Use the amended legal text and the Commission’s current timeline. The original 2024 schedule is no longer sufficient: Regulation (EU) 2026/1744, the AI Omnibus, entered into force on 27 July 2026 and changed important application dates.²

The Commission’s service-desk timeline confirms these milestones as of 9 November 2026.¹ Transitional treatment depends on the system, legal role and placement or modification date. Do not describe a future high-risk deadline as a waiver of already-applicable transparency, prohibited-practice, data-protection or sectoral obligations.

Classify intended use and legal role. “Agent” is not a risk category. Record intended purpose, affected people, geography and whether the organisation is a provider, deployer or another regulated actor. Employment and creditworthiness uses can raise different obligations from internal document search. Resolve applicability with the organisation’s responsible legal owner before granting authority.

Design oversight for the applicable use. Approval is one mechanism, not proof of meaningful oversight. Reviewers need understandable evidence, sufficient time and a usable intervention path. Refusal rate alone cannot establish effective oversight.

Bind logs to actions. When logging and oversight duties apply, the production evidence chain must support them. A teaching event log or a screenshot of an approval button cannot establish compliance by itself.

Implement applicable transparency in the product. Article 50 addresses interaction disclosure and specified synthetic-content obligations with defined conditions and exceptions. Decide which apply to the actual service and record the implementation evidence.

The amendment also changes Article 4: providers and deployers must take measures supporting staff AI literacy, while the Commission and Member States have promotion responsibilities. It does not eliminate the need to prepare staff to use the system.²

This section is a dated engineering map. The applicable consolidated law, jurisdiction-specific obligations and documented classification govern a real deployment; the headline dates cannot settle its legal status.

30.7Frameworks worth using#

Two established frameworks serve different purposes.

NIST AI RMF is a voluntary risk-management framework organised around four functions — Govern, Map, Measure, Manage. Its value is as a structure for the conversation: it gives you a vocabulary shared with risk and security colleagues, and its Measure function maps almost directly onto Part VII. Use it to organise your risk register. NIST’s Generative AI Profile adds GenAI-specific risks and suggested actions; neither document is a legal certification.³

ISO/IEC 42001 is a certifiable AI management system standard. It can support procurement where a customer requires evidence of a certified management system. It is about process — roles, reviews, documentation, continual improvement — rather than about your architecture, and includes management of AI-system risks across the lifecycle. Certification concerns the management system within its declared scope; it is not proof that each output is correct or that every legal obligation is satisfied.⁴

Use the freely available NIST materials to structure risk work. Assess ISO 42001 against organisational objectives, contractual requirements and certification cost; its value is not limited to procurement.

30.8Governance coverage is a weighted control matrix#

Figure 30.2 inventories the governance artefacts. It does not tell you whether every capability is actually covered by them. That distinction matters as the tool estate grows. A policy can be excellent for twelve capabilities and absent for the thirteenth; the missing row is still the route the agent will take. The useful unit of analysis is therefore not the document but the capability-control pair: does this control apply to this specific thing the agent can do, and can you produce evidence that it operated?

Start with the capability manifest from Chapter 15. Classify each capability by consequence, then state the controls required for that class. A practical first pass has three tiers:

The labels are local policy, not universal truth. Sending an email may be reversible in the technical sense and irreversible in the human one. Classify by consequence in your environment, record who made the classification, and make a change to that class go through review. A model must never choose its own tier.

Here is an illustrative Reference Agent inventory. It has eight read-only capabilities, six reversible writes and four irreversible actions. Each read capability requires four evidence cells; each reversible write requires six; each irreversible action requires eight. The complete matrix therefore contains:

Suppose the evidence collector finds 30 complete read cells, 30 reversible-write cells and 22 irreversible cells. A flat count reports 82 / 100 = 82% coverage. That number looks respectable and says almost nothing about the missing exposure. Eight of the ten missing irreversible cells may represent absent approval records or an untested stop path, while the two read gaps may be overdue owner reviews.

Weighting makes the prioritisation visible. Let the organisation assign illustrative consequence weights of 1 for reads, 3 for reversible writes and 8 for irreversible actions. This is a triage convention, not a probability model and not a claim that one control removes eight times more risk than another. The weighted denominator is:

read:          32 required cells × 1 =  32
reversible:    36 required cells × 3 = 108
irreversible:  32 required cells × 8 = 256
                                      ---
                                      396 weighted cells

The implemented numerator is 30 × 1 + 30 × 3 + 22 × 8 = 296. Weighted coverage is therefore 296 / 396 = 74.7%, leaving 100 / 396 = 25.3% of weighted control coverage unresolved. The percentage is not a risk score. It is an honest queue-ordering device: eight unresolved irreversible-control cells contribute 64 weighted units, as much as sixty-four unresolved read-control cells. Completed low-consequence cells cannot make that gap disappear in an average.

Now look at the matrix by path, not just by row. For an irreversible Reference Agent action, the minimum defensible chain is actor identity → resource policy → approval request → approval decision → execution event → result → stop path. A green box for each component in isolation is insufficient if their identifiers do not join. The approval event carrying call_signature=A, followed by an execution event carrying only call_id=B, leaves an evidential break even though both events exist. Test the join with one identifier that survives from proposal through effect.

The reference Reference Agent harness supplies the beginning of this chain, not the complete production control plane. Its PolicyContext carries actor, tenant and autonomy; its permission engine returns a decision, reason and rule; and its local append-only JSONL log records approval request/decision pairs and tool-call events. The approval request contains the tool, arguments and reason. The tool completion contains the call id, name, result, signature and idempotency key. Those are concrete teaching mechanisms.

A regulated production extension still has to propagate actor and tenant into every emitted action event, persist the policy version and exact verdict, bind the approval pair to the eventual execution, capture the presentation the approver actually saw, protect log integrity outside a local filesystem, enforce retention and access, and prove downstream delivery. Do not point at the reference log and claim those properties. Use it to show where they attach.

Run the matrix in CI and at runtime. CI rejects a capability-manifest row that omits a required control binding. Runtime sampling starts from executed actions and works backwards: can this action be joined to an applicable policy, actor, approval where required, owner and stop path? The two directions catch different failures. Manifest-to-control catches missing design; action-to-evidence catches drift between the design and the system that ran.

30.9Accountability for autonomous action#

Before deployment, identify operational ownership and the legal responsibilities attached to each participant’s role. When an agent does something wrong, the incident cannot wait for those responsibilities to be invented.

Humans and the organisation remain accountable. The system must make that accountability actionable through four properties:

A named owner per capability. Add an accountable person to each capability in the manifest.

A traceable decision chain. For any action: the run, the evidence, the policy verdict, the approver if any. The five questions above.

Bounded authority. Chapter 15’s blast radius, computed and written down. “The worst this can do is X” is an answerable question and an auditor will ask it.

A demonstrated stop capability. Chapter 20’s kill switch, rehearsed, with the rehearsal recorded.

Govern the agent as delegated authority: explicit bounds, risk-proportionate supervision, escalation and an accountable owner. Software speed and fallible self-assessment require tested controls for these responsibilities.

30.10The operating model#

Assign four operating decisions explicitly:

Who approves a new capability? Adding a tool that can move money is a different decision from adding one that reads a wiki. A tiered path — self-serve for read-only, review for reversible writes, committee for irreversible — keeps the low-risk path fast, which is what stops people routing around the process.

Who approves an autonomy promotion? Chapter 15’s levels, with Chapter 24’s numbers as evidence and a named approver. Promotion criteria written in advance.

Who owns quality? A named person accountable for the eval suite and the error-analysis cadence (Chapter 25). Unowned quality decays and nobody notices until a customer does.

Who is on call? For an agent, this needs to include someone who can flip the kill switch, pin a prompt version, and read a trace. That is a different skill set from ordinary infrastructure on-call and it needs to be staffed deliberately.

A review board is worth having when the volume justifies it, and worth keeping small and fast. A board that meets monthly and takes three weeks to approve a read-only tool will be bypassed, and a bypassed process is worse than none because it produces the appearance of control.

30.11Measure whether the controls operate#

A control description is a hypothesis. “Credentials are short-lived” becomes evidence only when you can show credential lifetime in observed calls. “Every irreversible action is approved” becomes evidence only when executed irreversible actions join to a prior approval for the same proposal. Governance needs operational measures for the same reason reliability does: systems drift while policy documents stay perfectly still.

Consider one illustrative week with 7,400 executed Reference Agent actions: 5,800 reads, 1,380 reversible writes and 220 irreversible actions. The collector finds 7,396 actions with complete required evidence. Overall completeness is 7,396 / 7,400 = 99.946%. Rounded to two decimals, a dashboard says 99.95% and turns green.

The four gaps decide whether that green is honest. If all four are low-consequence reads caused by one exporter delay, you have an observability defect. If one is an irreversible action with no approval join, the control failed at its highest-consequence point. Report both the aggregate and a zero-tolerance count for critical gaps:

overall evidence completeness       7,396 / 7,400 = 99.946%
critical actions without full chain       1 /   220 =  0.455%
target for critical chain failures                         0

Do not average that one action away. Investigate it as an incident until you establish whether the action bypassed the gate, the gate operated but the record failed, or the join key was corrupted. Those are different failures with different fixes, and only one is “improve logging.”

Measure governance at three timescales. Per action, the proxy enforces identity, resource policy and required approval before effect. Daily, a reconciler joins downstream mutations back to the authoritative event stream and opens exceptions for missing links. Per release, the capability-control matrix checks that every new or changed capability has an owner, tier, required controls, evidence query and rollback or stop story. A quarterly review is useful only after those faster loops exist; it cannot discover a live bypass quickly enough.

Avoid metrics that reward paperwork. Number of policies written, people trained, board meetings held and risks accepted are activity counts. They may be necessary inputs, but none demonstrates that the running path honoured a control. Prefer measures that begin with an actual action, actual stored data or an exercised stop. When a measure cannot be generated from the operating system, label it attestation and name who attested. Do not silently mix it with telemetry.

Finally, attach an owner and response rule to every measure. A dashboard without a response is decoration. For critical evidence-chain failures, the response may be to freeze the affected capability. For exception age, it may be automatic expiry of the waiver. For revocation latency, it may be a release gate above the agreed ceiling. Governance becomes part of production engineering when a bad value changes what the system is allowed to do.

Make every number reproducible#

A scorecard is evidence only if another person can reproduce it. Store the query or program that produced each measure, its version, the event-schema versions it accepts, the time window, exclusions and the exact input partitions. Preserve the result alongside a content digest of those inputs. “The dashboard showed 100% last quarter” is not a defensible claim when the dashboard has since changed its query twice.

This requirement catches a common failure. A team adds a new event schema in which actor_id becomes principal.actor, but the completeness query still selects only rows containing actor_id. New rows vanish from both numerator and denominator. The metric remains 100% while coverage is getting worse. A schema-compatibility test should feed one fixture from every live schema version through the evidence query and assert that each becomes either counted or explicitly rejected. Silent exclusion is the dangerous result.

Record exceptions with the same precision. An exception needs a capability, missing control, consequence tier, compensating control, owner, grantor, creation time and automatic expiry. If the waiver says only “accepted by security,” nobody can tell which path it permits or whether the compensating control still exists. At expiry, default to disabling the affected capability or returning it to a lower autonomy level; otherwise an expiry date is decoration.

Keep raw event access separate from governance reporting. The report should expose identifiers, counts and joins sufficient to test the control without copying customer content into another broadly accessible system. When an investigator needs payload detail, make that a separate, attributed access path. The audit record is already a concentrated data store; a convenient governance dashboard must not become an ungoverned replica of it.

30.12Passing a customer security review#

Prepare evidence for ten enterprise security questions:

The pattern in that table is the argument of the whole book: every governance answer is an engineering artefact. Not a policy document — a manifest the system loads, a test in CI, a measured refusal rate, a rehearsed switch. A review answered with policies produces conditions; a review answered with artefacts produces approval.

And the most persuasive thing you can bring is not in the table: the shadow-mode agreement document from Chapter 29. “On two thousand real cases, the agent agreed with the human 91% of the time, and here are the 9%” unblocks more reviews than any architecture diagram, because it converts an argument about capability into evidence about behaviour.

That completes the thirty chapters. The agent has a loop that survives failure, a context it manages deliberately, knowledge it can retrieve and remember, tools it is permitted to use, modalities it can handle, guardrails it cannot argue with, measurement it can be improved by, and a governance story that lets it run. Appendix A puts it together as a ninety-day build; Appendix B is a guided read of the code.