Skip to content

Governance, Risk, and Accountability

An agentic system can pass its tests, enforce its permissions, and remain operational while still being deployed for the wrong purpose, under the wrong owner, in the wrong jurisdiction, or with risks nobody was authorized to accept. Technical controls answer how a boundary is enforced. Governance answers who decides which boundaries and claims are acceptable, on what evidence, for whom, and for how long.

Governance is often mistaken for a committee, a list of principles, or documentation added before launch. None is sufficient. A committee without authority cannot stop deployment. Principles without decision rules do not change behavior. Documents without owners, executable controls, review triggers, and consequences become an archive of intentions.

Governance is the operating system for consequential organizational decisions. It makes the assembled system and use context legible, gives named people authority and resources to act, connects claims to evidence and obligations, records explicit treatment and residual-risk decisions, and reopens those decisions when their basis materially changes.

Accountability requires more than a name in a responsibility matrix. The accountable actor must know what was decided, have access to material evidence and dissent, possess enough competence and authority to change the outcome, and face a defined obligation to respond when assumptions fail. Responsibility without power is blame allocation, not governance.

A provider’s model card or safety framework describes only part of the deployed system. The organization’s governance object also includes:

  • the intended users, affected people, task, benefit, and supported operating claim;
  • models, prompts, context assembly, retrieval, memory, interfaces, policies, and control logic;
  • identities, delegated authority, data, tenants, destinations, and external effects;
  • human operators, reviewers, approvers, support teams, customers, and downstream users;
  • providers, integrations, subprocessors, infrastructure, and supply-chain dependencies;
  • deployment environment, scale, geography, language, accessibility, and operating horizon; and
  • foreseeable misuse, failure, incident, override, appeal, exit, and retirement paths.

Classifying “the model” as approved does not approve every application built around it. A summarizer over public text, a benefits-eligibility assistant, and a long-running operations agent can use the same model while presenting radically different consequences and obligations. Governance follows the use context and reachable effects.

OpenAI’s 2023 paper on governing agentic systems distinguishes roles such as model developer, system deployer, and user and argues for baseline responsibilities across the lifecycle (OpenAI, 2023). That actor model is a useful starting point, but real supply chains can split each role across several entities. Write the responsibility boundary for the actual system rather than assuming a vendor label resolves it.

Maintain an inventory that supports decisions

Section titled “Maintain an inventory that supports decisions”

An inventory is not a spreadsheet of model names. Each record should let the organization find the system, determine which decisions apply, and identify who can act. Record at least:

Field Governance purpose
System and owner Stable identity, accountable business or mission owner, technical owner, operational contact
Purpose and claims Intended benefit, supported tasks, prohibited uses, affected parties, known limitations
Lifecycle state Proposed, experimental, approved, limited release, production, suspended, or retired
System composition Models, providers, prompts, policies, data, memory, interfaces, dependencies, deployment environments
Authority and effects Principals, tenants, data scope, external actions, approval requirements, reversibility
Reach and context User groups, volume, duration, jurisdictions, languages, sectors, accessibility and vulnerability factors
Risk and obligations Classification, risk owner, obligation mappings, exceptions, review date, assurance state
Evidence and history Evaluation reports, control evidence, incidents, complaints, decisions, changes, known gaps
Exit path Provider substitution, data export and deletion, rollback, user transition, retirement owner

Use stable identifiers so evaluation reports, traces, incidents, contracts, risk records, release artifacts, and user-facing disclosures refer to the same system and version. Inventory fields should be populated automatically where authoritative metadata exists, but an automatically discovered endpoint cannot supply purpose, affected parties, acceptable consequences, or a risk owner.

Inventory quality is operational. Detect unregistered systems through procurement, repositories, cloud accounts, API traffic, expense data, identity systems, and integration catalogs. Provide a simple registration path and distinguish experimentation from production, but do not create a “pilot” label that permits indefinite use without review.

NIST AI RMF 1.0 makes inventory, clearly defined roles, executive responsibility, monitoring, stakeholder input, third-party risk, and safe decommissioning part of its cross-cutting GOVERN function (NIST, 2023). It is a voluntary US risk-management framework and is being revised; use the named version rather than treating “NIST aligned” as a permanent property.

Classify consequences before crediting controls

Section titled “Classify consequences before crediting controls”

Risk classification should determine the depth of evidence, review, approval, monitoring, external input, and retirement planning required. Begin with the use as proposed, before assuming every control works. Consider:

  • severity, scale, duration, and reversibility of plausible harm;
  • people, rights, property, services, infrastructure, organizations, and ecosystems affected;
  • whether a person can recognize, challenge, correct, or recover from the outcome;
  • data sensitivity, delegated authority, external effects, operating horizon, and propagation paths;
  • exposure: number and vulnerability of users, frequency, geography, and adversarial access;
  • foreseeable misuse, correlated failure, concentration, and dependency risk;
  • novelty, uncertainty, evidence quality, and how quickly capability or context can change; and
  • applicable legal, regulatory, contractual, sector, and internal-policy requirements.

Call this inherent risk: the risk before crediting the controls relied on in the current decision. Then identify treatments and evidence of their effectiveness. What remains is residual risk, including measurement uncertainty, untested conditions, control dependencies, and the circumstances that would invalidate the assessment.

Do not conceal a non-compensable harm inside one aggregate score. A low average likelihood does not make an irreversible rights violation or cross-tenant disclosure acceptable. Numerical matrices can support comparison, but multiplying speculative severity and probability values can create false precision. Use scenarios, consequence bands, ranges, uncertainty, and explicit decision rules where the evidence does not support a precise number.

The classification applies to a defined version and scope. A low-risk internal drafting assistant can become a higher-risk system when it gains customer data, write authority, automatic memory, a new population, or a path into a regulated decision.

A risk register should be a decision tool, not a warehouse of generic concerns. For each material risk, record:

Risk scenario and affected parties:
Cause, event, and consequence:
System and lifecycle scope:
Inherent consequence and exposure:
Existing controls and control owners:
Evidence of control design and effectiveness:
Dependencies, assumptions, and uncertainty:
Residual risk and invalidation conditions:
Treatment: avoid / reduce / transfer / share / accept:
Action owner, resources, due date, and verification:
Decision authority and escalation path:
Monitoring signals and review triggers:
Linked incidents, complaints, exceptions, and obligations:

“Mitigate with guardrails” is not a treatment. Name the control, where it is enforced, the threat or failure it addresses, its known limitations, its owner, and the evidence required. Security, reliability, evaluation, observability, product, legal, and operational work can all contribute controls; governance connects them to the risk decision without redefining their implementation.

Avoid and retire are valid treatments. If a task cannot be made acceptably safe, if the organization lacks evidence or operational capacity, or if the use conflicts with an obligation, the correct decision may be not to build, not to release, to narrow the scope, or to decommission the system.

Risk acceptance is an authorized decision that the remaining risk is justified for a stated scope and period. It should include:

  • the system version, use, users, affected parties, jurisdictions, and operating limits;
  • the benefit or necessity being pursued and feasible alternatives considered;
  • the evidence reviewed, material uncertainty, dissent, and unresolved gaps;
  • residual-risk scenarios, likelihood or uncertainty, severity, and distribution of impact;
  • controls, monitoring, incident readiness, compensating measures, and stop conditions;
  • the accountable risk owner and an acceptor with authority proportional to the consequence;
  • an effective date, expiry, review triggers, and required follow-up work; and
  • a reasoned decision record that can be inspected later.

Silence is not acceptance. A risk is not accepted because a ticket sits in a backlog, a launch date passes, a provider offers the feature, a user clicks “I agree,” or no committee objected. Nor can an executive accept away a binding legal duty, another person’s rights, or a risk outside their delegated authority.

The acceptor should not be rewarded only for shipping while bearing none of the consequence. High-consequence or disputed decisions may require independent technical, domain, legal, security, safety, or affected-stakeholder challenge. Independence is a control against shared assumptions and incentives, not a ceremonial second signature.

OpenAI’s 2026 Frontier Governance Framework connects systemic-risk assessment to an explicit acceptance determination, documented justification, safety margins, named organizational responsibility, external expert input, reporting, and framework change control (OpenAI, 2026). Anthropic’s RSP v3 similarly documents a responsible officer, risk reports, noncompliance reporting, external review criteria, procedural review, board approval, and a public change log (Anthropic, 2026). These are valuable current implementation examples for frontier-model risks, not universal prescriptions for company structure or risk thresholds.

At minimum, distinguish these functions even when a small organization combines them in a few people:

Function Accountable question
System or use owner Is this use valuable, supported, operated, and retired responsibly?
Risk owner Is the named risk understood, treated, monitored, and escalated across boundaries?
Control owner Is the control implemented, operated, evidenced, maintained, and repaired?
Evidence owner Are evaluations, assurance artifacts, and limitations valid and current for the decision?
Independent challenger What assumptions, conflicts, missing parties, or weak evidence could change the decision?
Risk-acceptance authority Is proceeding with this residual risk within my mandate and justified by the record?
Incident and remediation owner Who contains impact, communicates, corrects state, and closes resulting actions?
Obligation owner Which external or internal requirements apply, and how is fulfillment evidenced?

A RACI chart can communicate participation but does not supply authority, competence, time, access, or budget. Define who can stop a launch, restrict a capability, require evidence, approve an exception, notify affected parties, contact regulators, and retire the system. Define the escalation path when owners disagree or a decision exceeds their mandate.

Separate creation from assurance when consequence warrants it. The team that built a system holds essential context but also shares assumptions and delivery incentives. Independence can come from another internal function, domain expert, affected-party review, commissioned evaluator, auditor, or regulator depending on the claim. Document conflicts and information access; a reviewer who sees only a polished summary cannot challenge omitted evidence.

Protect good-faith escalation. Provide more than one route for reporting suspected noncompliance or hidden risk, prohibit retaliation, track investigation and remediation, and make unresolved material concerns visible to the appropriate authority. A culture that punishes bad news defeats even a well-designed process.

Policy has to reach engineering and operations. For each requirement, specify:

Policy statement and rationale:
Systems, actors, data, decisions, and lifecycle stages in scope:
Required, prohibited, and discretionary behavior:
Decision owner and exception authority:
Implementation controls and enforcement points:
Evidence and quality gates:
Monitoring, incident, and reporting signals:
Exception conditions, compensating controls, and maximum duration:
Review triggers, owner, version, and effective date:

Some requirements can become deterministic policy checks, authorization rules, schema constraints, release gates, retention jobs, or deployment configuration. Others require contextual human judgment. State which is which. A prose rule that says “use human review for sensitive actions” is incomplete until the organization defines sensitive, identifies the reviewer, provides relevant evidence, and determines what happens when review is unavailable.

Policy exceptions are controlled decisions, not side channels. Record scope, reason, owner, risk, compensating controls, expiry, and exit criteria. Monitor their actual use. Repeated exceptions may reveal a bad policy, an underfunded control, or a business process that has moved outside the approved operating claim.

Version policy and preserve what applied when a decision or effect occurred. Changing a threshold after a failed gate is a policy decision requiring an owner and rationale, not routine test maintenance.

Build an obligation register, not a “compliant” label

Section titled “Build an obligation register, not a “compliant” label”

Legal, regulatory, contractual, professional, sector, and internal-policy obligations differ by jurisdiction, role, system category, intended use, affected population, and date. A useful obligation record names:

  • the issuing authority or counterparty and canonical source;
  • whether the item is law, regulation, adopted standard, contract, voluntary framework, guidance, draft, or internal policy;
  • jurisdiction, role, affected system and use, effective dates, and applicability rationale;
  • the concrete requirement, prohibition, notice, record, assessment, reporting, or response duty;
  • owner, implementation control, evidence, review cadence, and known interpretation question; and
  • dependencies and overlap with existing management systems.

Do not collapse these fields into “EU compliant,” “NIST certified,” or “follows ISO.” NIST AI RMF is voluntary. ISO/IEC 42001:2023 is a published international management-system standard; certification, where used, has a defined organizational scope and does not prove that every product outcome is safe or lawful (ISO, 2023). The EU AI Act is binding law with role-, use-, and date-specific duties; among other requirements for in-scope high-risk systems, it establishes lifecycle risk-management and quality-management obligations (Regulation (EU) 2024/1689). Qualified counsel must determine applicability; this Fieldbook does not.

Reuse existing information-security, privacy, quality, safety, procurement, records, and incident processes where they genuinely satisfy the AI requirement. Do not create a parallel governance bureaucracy merely to rename the same control. Conversely, do not assume an existing certification covers model behavior, delegated authority, prompt injection, evaluation validity, or affected-party remedies when it does not.

Govern third parties and shared responsibility

Section titled “Govern third parties and shared responsibility”

Providers and integrators can supply valuable evidence: system cards, evaluation reports, security attestations, data terms, incident notices, change logs, uptime commitments, subprocessor lists, and deletion behavior. Assess whether each artifact covers the version, region, feature, deployment mode, and claim you rely on.

The deployer still owns its choices of use, data, prompts, retrieval, memory, interfaces, authority, approval, user experience, monitoring, and recovery. A provider cannot evaluate every downstream context. Contract language allocates duties and remedies; it does not make a control effective.

For material dependencies, govern:

  • supported and prohibited use, data processing, training and retention terms;
  • model and API change notice, version stability, deprecation, and rollback;
  • evidence access, audit or assessment rights, incident notification, and cooperation;
  • subcontractors, data locations, security, availability, capacity, and concentration;
  • intellectual-property, confidentiality, export, sector, and jurisdiction constraints;
  • graceful degradation, provider substitution, data portability, deletion, and exit testing; and
  • the party responsible for each control at every handoff.

Reassess when the provider changes a model, safety behavior, interface, pricing, region, retention term, or service dependency. “Same API name” does not mean same governed system.

Preserve a decision trail, not model thoughts

Section titled “Preserve a decision trail, not model thoughts”

Auditability means reconstructing what the organization knew, required, decided, and caused. Preserve:

  • the system, configuration, policy, obligation, and evidence versions;
  • the decision question, eligible options, criteria, and authority;
  • material inputs, verified facts, uncertainty, dissent, and excluded evidence;
  • the selected treatment, approval or rejection, conditions, and timestamp;
  • external effects, incidents, complaints, overrides, exceptions, and follow-up actions; and
  • changes to the decision basis and why reassessment did or did not occur.

Do not substitute model-generated rationales or hidden chain-of-thought for this record. They may be incomplete, unfaithful, sensitive, unavailable, or irrelevant to the actual policy and state transitions. Record observable inputs, accepted transitions, policy decisions, evidence, and effects. Protect the trail with access, retention, integrity, privacy, and deletion rules; “log everything forever” creates another governance failure.

Assurance asks whether evidence justifies a claim for a named audience and decision. Self-assessment, peer review, independent testing, red teaming, audit, conformity assessment, and certification have different scopes. State who performed the work, against which criteria and version, with what access, sampling, independence, limitations, and expiration. Singapore’s AI Verify program illustrates the useful distinction between technical tests and process checks while explicitly warning that framework output does not guarantee a system is safe or free from bias (IMDA).

Include affected people and contestability

Section titled “Include affected people and contestability”

People affected by a system may reveal harms, operating conditions, accessibility barriers, cultural assumptions, and recourse failures that developers and owners do not see. Decide whose input is needed during design, evaluation, deployment, monitoring, incident response, and review. Participation should be early enough to change the decision and supported with understandable information; collecting comments after commitment is not meaningful challenge.

Governance owns whether notice, explanation, correction, human review, challenge, appeal, remedy, and support must exist, who responds, and what evidence is retained. Product & UX owns how people encounter and use those mechanisms. Legal duties and suitable mechanisms vary by context.

The OECD AI Principles, updated in 2024, ground accountability in actors’ roles, context, and ability to act, and connect it to lifecycle traceability, systematic risk management, transparency, and the ability to challenge outcomes (OECD Accountability, Transparency). They are an active non-binding intergovernmental recommendation, not an implementation specification.

Approval is not permanent. Define material-change and periodic-review triggers, including:

  • new models, providers, fine-tuning, prompts, policies, thresholds, or context assembly;
  • new data classes, memory behavior, interfaces, authority, destinations, or integrations;
  • new user populations, languages, jurisdictions, sectors, scale, or operating horizons;
  • changed capabilities, evaluations, control performance, attack methods, or external research;
  • incidents, near misses, complaints, appeals, audit findings, or unexplained drift;
  • provider terms, dependencies, laws, standards, contracts, or voluntary commitments; and
  • expired exceptions, missing owners, control failures, or unavailable recovery paths.

Not every change needs the same review. Define which evidence can be reused, which gates rerun, which owners notified, and which changes require new risk acceptance. Preserve the rationale when a trigger is assessed as immaterial.

Retirement is a governed lifecycle stage. Remove authority, credentials, traffic, integrations, scheduled work, stored memory, shadow copies, and user entry points; preserve required records; fulfill deletion and notification duties; migrate users and artifacts; and verify that the system no longer causes effects. An endpoint marked deprecated while background tasks and credentials remain active is not retired.

Choice Benefit Cost or risk
Central minimum governance Consistency, shared expertise, comparable records Bottlenecks and weak domain context
Domain-owned decisions Contextual expertise and faster action Inconsistent standards and local incentives
Detailed risk taxonomy Better coverage and comparison Classification overhead and false precision
Simple consequence tiers Usable routing and proportional process Important differences may be collapsed
Independent assurance Challenges assumptions and conflicts Cost, delay, limited access, ceremonial review
Broad transparency Scrutiny, trust, and affected-party challenge Privacy, security, misuse, and trade-secret exposure
Extensive decision records Reconstruction and accountability Staleness, sensitive-data concentration, documentation burden
Strict no-exception policy Clear minimums Hidden workarounds and inability to handle legitimate edge cases
Controlled exceptions Adaptability and visible residual risk Normalization of bypasses if ownership and expiry are weak
Provider standardization Lower integration and assurance cost Concentration, correlated failure, and exit difficulty

Proportionality does not mean optional governance for small teams. It means the evidence, independence, ceremony, and authority match the consequence. A concise record with one empowered owner can govern a low-consequence internal tool better than a large committee with no operational connection.

A review group can offer advice but cannot stop launch, require evidence, fund remediation, narrow scope, or retire the system.

The registry lists a provider and model name but omits purpose, data, authority, integrations, affected people, versions, incidents, and owners.

A red–amber–green cell replaces a description of who can be harmed, how, under which conditions, with what uncertainty, and through which effects.

The risk record credits a guardrail, approval, or provider filter without an owner, failure behavior, current evidence, or dependency analysis.

The launch proceeds because a ticket was not answered. No person with the required mandate accepted the residual risk.

Accountable in name, powerless in practice

Section titled “Accountable in name, powerless in practice”

A product manager owns the risk on paper but cannot access evidence, change the release, restrict authority, obtain resources, or escalate independently.

A waiver has no expiry, compensating control, usage monitoring, or exit criterion and quietly becomes the normal path.

A vendor certificate or system card is treated as proof that the deployer’s data, retrieval, permissions, workflows, users, and outcomes are safe and compliant.

The team meets a documentation or conformity duty and concludes that every foreseeable harm is controlled, including risks outside the rule’s scope.

Model-generated reasoning is stored as the decision record while authoritative state, policy checks, evidence versions, approvals, and actual effects are missing.

The system gains write authority, a new population, or a new provider but continues under a risk assessment for the earlier read-only pilot.

The organization either publishes nothing useful or releases sensitive data and security details because “transparency” was not tied to a stakeholder and purpose.

Maintain a governance decision record for every material development, deployment, expansion, exception, continued-operation, and retirement decision:

Decision, system ID, version, and lifecycle transition:
Owner, users, affected parties, purpose, and operating claim:
System composition, authority, data, effects, scale, and jurisdictions:
Supported, excluded, and foreseeable-misuse conditions:
Applicable obligations and voluntary commitments:
Inherent-risk scenarios and consequence classification:
Controls, owners, dependencies, and effectiveness evidence:
Residual risk, uncertainty, dissent, and invalidation conditions:
Alternatives and risk treatments considered:
Decision authority, outcome, rationale, and timestamp:
Conditions, monitoring, stop rules, exceptions, and expiry:
Material-change and periodic-review triggers:
Disclosure, challenge, incident, remediation, and retirement paths:
Evidence links and follow-up actions:

Use one review question:

If this decision produces harm, can we show who had the authority and evidence to decide, what uncertainty and affected parties were considered, which controls were relied on, and what was required when those assumptions failed?

If the answer is a committee name, a model card, or “the policy says so,” the decision is not yet governable.

  • Does the inventory describe the assembled system, use, affected parties, versions, authority, data, dependencies, risk, owners, evidence, and exit path?
  • Are experiments, limited releases, production systems, suspended systems, and retired systems distinguishable and discoverable?
  • Is inherent risk classified before controls are credited, using concrete scenarios, consequence, exposure, reversibility, uncertainty, and obligations?
  • Does every material risk have a risk owner, control owners, treatment, evidence, monitoring, due dates, and an escalation path?
  • Is residual risk explicit about uncertainty, dependencies, untested conditions, and events that invalidate the assessment?
  • Are risk acceptances reasoned, authorized, scoped, time-bounded, monitored, and reopened on material change?
  • Can the acceptor stop, narrow, delay, or retire the system and obtain resources proportional to the consequence?
  • Are build, review, evidence, control, acceptance, incident, and obligation functions distinguished, with conflicts and combined roles visible?
  • Can personnel report noncompliance or hidden risk through protected routes to an authority outside the immediate delivery chain?
  • Does each policy requirement map to scope, owners, enforceable controls or explicit judgment, evidence, monitoring, exceptions, and versions?
  • Are exceptions authorized, compensated, observed, expiring, and analyzed for policy or control debt?
  • Does the obligation register distinguish law, regulation, standard, contract, framework, guidance, draft, and internal policy by jurisdiction, role, version, and date?
  • Are provider evidence and contracts checked against the actual version, region, feature, claim, and deployment context?
  • Are change notice, incident cooperation, dependency concentration, data portability, substitution, and exit tested for material third parties?
  • Can a decision be reconstructed from authoritative facts, policy and evidence versions, accepted transitions, approvals, actual effects, and follow-up actions?
  • Does each assurance artifact state criteria, scope, performer, independence, access, method, limitations, validity period, and decision supported?
  • Are affected parties and domain experts involved early enough to change material decisions, with usable challenge and remedy paths where required?
  • Do model, data, authority, integration, population, jurisdiction, scale, incident, evidence, provider, and policy changes trigger proportionate reassessment?
  • Does retirement revoke effects and access, handle data and records, transition users and work, and verify that hidden execution has stopped?
  • Are governance workload, overdue actions, expired decisions, open exceptions, unowned systems, control failures, and repeated dissent visible to leadership?
  • OpenAI, “Practices for Governing Agentic AI Systems” — core early agent-specific research supporting lifecycle roles, baseline responsibilities, suitability, constraints, legibility, monitoring, and accountability. The 2023 paper is explicitly preliminary and does not define a current universal standard.
  • OpenAI, “Frontier Governance Framework” — OpenAI frontier-model governance framework supporting connected risk identification, analysis, acceptance, mitigation, reporting, external input, named responsibility, and controlled framework updates. It applies to OpenAI’s specified frontier-model and regulatory scope rather than ordinary application governance in general.
  • Anthropic, “Responsible Scaling Policy, version 3.0” — Anthropic voluntary governance policy supporting named ownership, risk reports, independent review criteria, protected noncompliance reporting, procedural review, board approval, and versioned public change. Its catastrophic-risk scope and organization-specific commitments are not generalized.
  • NIST AI 100-1, “Artificial Intelligence Risk Management Framework 1.0” and NIST’s current AI RMF page — final voluntary US framework used for cross-cutting lifecycle governance, inventory, risk tolerance, roles, executive responsibility, stakeholder input, third-party management, monitoring, and decommissioning. Version 1.0 is being revised as of this chapter’s review.
  • ISO/IEC 42001:2023, “Artificial intelligence management system” — published international management-system standard used for policies, objectives, processes, risk treatment, and continual improvement. The public summary does not expose the full normative text, and certification has a defined scope rather than proving every system safe or lawful.
  • Regulation (EU) 2024/1689, especially Articles 9, 16, and 17 — binding EU law used as a concrete example of lifecycle risk management, quality management, documentation, monitoring, corrective action, and conformity duties for systems and actors in scope. Applicability varies by role, use, jurisdiction, and phased date and requires qualified legal interpretation.
  • OECD AI Principles: Accountability, Transparency and explainability, and Robustness, security and safety — active non-binding intergovernmental principles, updated in 2024, supporting role- and context-based accountability, traceability, systematic lifecycle risk management, disclosure, challenge, and safe retirement. They do not prescribe an organizational implementation.
  • Singapore IMDA, “AI Verify” — national public program and open-source testing ecosystem used to distinguish technical tests, process checks, governance claims, and the limits of assurance. Toolkit or self-assessment output is not treated as independent certification or proof of safety.