Security, Privacy, and Guardrails

Agentic systems combine a component that interprets natural language with private data, external content, credentials, software interfaces, and the ability to act. This creates useful flexibility, but it also creates a security problem: data can look like instructions, instructions can arrive through data, and a persuasive model output can cross into a real effect unless another component enforces the boundary.

Prompt injection is important, but it is not the whole threat model. Ordinary broken access control, insecure defaults, secret exposure, vulnerable dependencies, injection into downstream interpreters, weak tenant isolation, and missing audit records remain capable of compromising the system. AI-specific controls must extend established application and infrastructure security, not replace it.

Thesis

Treat every model input, output, retrieved item, memory, interface result, and delegated message as untrusted data at a security boundary. Security comes from limiting reachable data and effects, mediating every protected operation outside model judgment, and layering controls so one manipulated decision cannot become an unacceptable consequence.

A guardrail can reduce risk, but “we have guardrails” is not a security property. Name the threat, protected asset, trust boundary, enforcement point, failure behavior, evidence, and residual risk. Controls that only ask the same model to behave safely are probabilistic influence, not complete mediation.

Begin with assets, actors, and reachable effects

Threat-model the assembled system, not a chat box. Identify:

• assets: user and enterprise data, secrets, identities, funds, files, code, infrastructure, model weights, prompts, policies, memory, logs, and reputation;
• actors: end users, administrators, developers, operators, model and infrastructure providers, integration owners, external content authors, compromised services, insiders, and attackers;
• entry points: prompts, uploads, webpages, email, retrieval collections, memory, feedback, model outputs, callbacks, APIs, plugins, MCP servers, Skills, package registries, and supply-chain updates;
• principals and authority: whose identity each operation uses, on whose behalf it acts, which tenant and purpose apply, and who can delegate or approve;
• trust boundaries: browser, model provider, application, sandbox, execution proxy, data store, queue, external service, tenant, region, and administrative plane; and
• sinks: places where untrusted influence becomes consequential—data disclosure, network request, file or database write, code execution, payment, message, permission change, deployment, or durable memory.

Trace credible paths from source to sink. A poisoned document is low impact when the system can only summarize public text. The same document becomes high impact when its instructions can cause private records to be fetched and transmitted to an attacker-controlled URL. OpenAI’s current prompt-injection article uses a similar source–sink framing and argues that input filtering alone cannot contain fully developed social-engineering-style attacks (OpenAI, 2026).

Classify threats by affected security property and consequence, not by whether they use a “jailbreak.” Include confidentiality, integrity, availability, authenticity, accountability, privacy, user safety, financial loss, and loss of control. Record assumptions about provider behavior, model visibility, tenant isolation, and human review; an assumption is not a control.

Prompt injection crosses the instruction–data boundary

Direct prompt injection arrives through the user or an API caller. Indirect injection is embedded in content the system retrieves or observes: a webpage, document, email, issue, repository, image text, Tool result, memory, or another agent’s message. Persistent injection survives when manipulated content is stored in memory, an artifact, a summary, or a trusted collection and influences later runs.

The core difficulty is semantic. Models are designed to interpret natural language, and no delimiter, message label, escaping convention, or phrase such as “ignore instructions below” guarantees that text will be treated only as data. Model training and classifiers can improve resistance, but adaptive attackers can make repeated attempts and combine manipulation with legitimate-looking task context.

Design for attacks that sometimes succeed:

1. label provenance and trust so the system and model can distinguish governing instructions from evidence;
2. exclude secrets, broad private data, and capabilities the task does not need;
3. prevent untrusted content from redefining goals, authority, approval, policies, or destinations;
4. validate proposed effects against the user’s actual request and current state;
5. isolate code, files, network, identity, and tenants through non-model controls;
6. require explicit confirmation for concrete consequential transmissions or actions where appropriate; and
7. monitor, test, and respond to novel attack paths.

OpenAI describes layered defenses including model training, monitoring, sandboxing, link and data-transmission controls, confirmation, and red teaming (Understanding prompt injections). Anthropic similarly separates defenses across the model, the external content it consumes, and the environment in which it acts, and states that model-layer protection cannot stand alone (Anthropic, 2026). These are current provider-published accounts, not proof that either provider or any specific model is immune.

Minimize the blast radius before detecting the attack

The most dependable control is often to make a compromised decision less powerful. Apply least privilege to each run and operation:

• expose only eligible interfaces and data collections;
• prefer read-only or propose-only modes before write access;
• scope identity by tenant, resource, operation, purpose, destination, amount, environment, and time;
• use short-lived delegated credentials rather than ambient user or service credentials;
• cap records, recipients, funds, requests, tokens, runtime, child work, and rate;
• separate high-consequence capabilities so one compromised path does not reach all of them; and
• require a new authorization when material parameters or state change.

Hiding an interface from the model reduces accidental selection but does not authorize anything. Every protected operation must be checked at the execution boundary even if the request arrives through a compromised prompt, direct API, stale queue item, child agent, or forged callback.

Avoid confused-deputy behavior: the system must not use its own broader authority merely because an untrusted party asks it to act. Carry the initiating principal, delegated scope, tenant, purpose, and user intent to the downstream enforcement point. A child workflow, plugin, Skill, MCP server, or external agent cannot receive authority the parent does not possess and should receive only the subset it needs.

Keep credentials outside model-visible context

The model should propose an operation without seeing the secret used to perform it. At the execution boundary:

1. authenticate the initiating principal and calling component;
2. authorize the concrete operation and current parameters;
3. resolve a narrowly scoped credential from a secret manager or workload identity;
4. invoke the destination through a controlled client or proxy;
5. prevent the response from reflecting secrets into model context; and
6. record the policy decision and effect without logging the credential.

Do not place API keys in system prompts, Tool descriptions, memory, files reachable by an untrusted runtime, environment dumps, or trace attributes. Rotate and revoke credentials, separate development and production identities, and detect anomalous use. A secret redacted after it reaches the model may already have been transmitted to a provider or retained in telemetry.

Network egress is part of secret protection. An attacker needs a destination as well as data. Use destination allowlists where the task is bounded, block loopback and cloud metadata services, resolve DNS and redirects under policy, restrict protocols and ports, and inspect whether sensitive data is being sent to a new party. OpenAI’s link-safety work illustrates that even a URL fetch can exfiltrate data through path or query parameters (OpenAI, 2026).

Isolate code and open-ended interfaces

Shells, browsers, code interpreters, file editors, and computer-use systems expose broad effect surfaces. Run them in an environment whose boundary does not depend on the generated command being benign:

• ephemeral container, VM, or process sandbox appropriate to the threat;
• allowlisted mounts with separate read and write scopes;
• no host sockets, credential directories, metadata endpoints, or unrelated user files;
• unprivileged identity, syscall and process restrictions, quotas, and time limits;
• controlled network proxy and DNS behavior;
• disposable working state and verified artifact export; and
• isolation between tenants, runs, and untrusted code.

Sandboxing is not a universal guarantee. Kernel, runtime, browser, proxy, and configuration vulnerabilities remain; a sandbox may contain filesystem effects while still allowing data exfiltration; and a mounted repository may itself contain secrets. Threat-model the sandbox implementation, patch it, test escape paths, and make its guarantees explicit.

Anthropic’s Claude Code material uses filesystem and network isolation to contain damage even if prompt injection succeeds (Anthropic, 2025). The transferable lesson is the non-model boundary, not the product configuration or a claim of perfect isolation.

Prefer domain-specific operations over unrestricted commands when the task permits. issue_refund(order_id, amount, reason) can enforce ownership, limits, policy, and idempotency more reliably than a shell script with database credentials. Open-ended interfaces remain useful for coding and investigation, but their data, network, credential, lifetime, and export boundaries must match the consequence.

Treat model output as tainted input

Structured output narrows ambiguity; it does not make content trustworthy. Validate syntax, type, length, range, enumeration, resource ownership, state version, destination, and business invariants before use. Then encode or parameterize for the downstream interpreter.

Never concatenate model output directly into:

• SQL, shell, HTML, templates, regular expressions, or query languages;
• file paths, URLs, headers, email recipients, or access-control rules;
• code that executes outside an appropriate sandbox; or
• policy, memory, retrieval indexes, or audit records without validation and provenance.

Use parameterized queries, safe APIs, canonical path checks, output encoding, schema validation, and explicit allowlists. A model that generates correct-looking JSON can still request an unauthorized resource. A content-safety classifier can approve text that becomes command injection in another interpreter.

When a model output becomes context for another model, retain its origin and trust level. Multi-agent consensus does not sanitize attacker-controlled content, especially when agents share models, sources, memory, or permissions.

Engineer data privacy through the lifecycle

Privacy is not only an output filter. Map data from collection through model input, provider processing, retrieval, memory, logs, evaluation, support, export, backup, and deletion. For each data class record:

• purpose and lawful or policy basis where applicable;
• subject, owner, tenant, geography, residency, and sensitivity;
• who may access it and on whose behalf;
• which model, provider, region, subcontractor, or external service receives it;
• whether it may be retained, used for training, cached, embedded, summarized, or joined with other data;
• retention, deletion, correction, export, and legal-hold behavior; and
• evidence that downstream copies and derived records follow the decision.

Collect and disclose only what the task requires. Minimize before model submission, not only before display. Use field selection, pseudonymous identifiers, tokenization, protected references, local processing, or aggregation when the model does not need raw identity. Do not retrieve an entire mailbox or customer record merely because the credential permits it.

Embeddings, summaries, labels, prompts, outputs, memories, and traces can remain personal or confidential data even when they are “derived.” Deleting the source row while leaving searchable vectors, evaluation cases, caches, and backups may not satisfy the promised deletion behavior. Record lineage so correction and deletion can propagate where required.

Privacy rules vary by jurisdiction, sector, contract, age group, and purpose. NIST AI 600-1 provides cross-sector risk-management considerations for data privacy, provenance, value-chain dependencies, and generative-AI lifecycle risks, but it is voluntary US risk-management material rather than legal advice or a universal compliance checklist (NIST, 2024).

Use guardrails according to what they can enforce

“Guardrail” covers controls with very different assurance. Separate at least:

Guardrail type	Useful for	Important limitation
Deterministic validation	Schemas, limits, allowlists, state predicates, signatures	Cannot judge every semantic or contextual harm
Authorization and policy engine	Principal, resource, action, purpose, destination, and approval decisions	Depends on correct identity, policy, and complete mediation
Model or classifier check	Semantic content, relevance, suspicious instructions, possible disclosure	Probabilistic, adaptive to attacks, subject to drift and false decisions
Input transformation	File parsing, canonicalization, malware scanning, content separation	Transformation can lose provenance and cannot make language trustworthy
Output handling	Encoding, DLP, citation and format checks, sink-specific validation	Too late if an earlier side effect or provider disclosure already occurred
Sandbox and capability boundary	Limits filesystem, process, network, data, and effect reach	Implementation vulnerabilities and overly broad configuration remain
Human approval	Contextual accountability for concrete high-impact decisions	Fatigue, poor evidence, manipulation, and ambiguous approval scope
Monitoring and rate control	Detects abuse and limits repeated attempts or consumption	Usually reduces or discovers harm rather than proving correctness

Define whether a guardrail blocks, redacts, narrows, asks for confirmation, quarantines, logs, or only scores. State its timeout and failure behavior. A security check that times out should not silently become allow. Conversely, failing closed on every low-risk classifier outage can make availability an easy denial-of-service target.

Order matters. A parallel input classifier can reduce latency, but a consequential operation must not commit before the blocking check finishes. An output filter cannot retract data already sent through a URL or Tool call. Apply controls immediately before the sink they protect and re-check material state changes.

Calibrate model-based guardrails on representative benign, malicious, ambiguous, multilingual, obfuscated, and adaptive cases. Measure attack success, false accepts, false rejects, abstentions, utility loss, latency, and cost across repeated trials. Version the model, prompt, threshold, and policy. Never use the monitored model’s self-assessment as the only evidence that it resisted an attack.

Make approval a security ceremony, not a click

Approval should bind a person with relevant authority to a concrete operation, target, data disclosure, amount, destination, state version, time, and consequence. Show:

• what will happen and which system will receive it;
• the exact sensitive fields or external effects;
• why the action is needed and which evidence supports it;
• whether it is reversible and how it will be verified; and
• what changed since any earlier approval.

Do not let untrusted content write the approval explanation unchallenged. Generate critical fields from structured state and execution policy, distinguish model commentary from verified facts, and make suspicious destinations or scope changes salient. Avoid repeated low-value prompts that train users to approve reflexively.

Approval does not repair an insecure operation. Users cannot reasonably inspect hidden data flows, encoded payloads, thousands of file changes, or a broad shell command. Use deterministic constraints and scoped interfaces first; reserve human judgment for decisions people can actually evaluate.

Secure the AI supply chain

The system depends on more than a model endpoint. Inventory and govern:

• models, weights, providers, adapters, system prompts, policies, and inference configuration;
• frameworks, SDKs, containers, browsers, interpreters, extensions, and native libraries;
• Tool and MCP servers, plugins, Skills, scripts, templates, registries, and manifests;
• training, fine-tuning, retrieval, evaluation, feedback, and memory data;
• build pipelines, package sources, artifact stores, deployment identities, and update channels; and
• external APIs, hosted search, vector stores, monitoring, and human operations vendors.

Verify publisher and ownership, pin or approve versions, scan dependencies and artifacts, protect build and release identities, record provenance and licenses, monitor advisories, and test updates before widening access. Treat descriptions and manifests from a registry as claims, not permission. Review nested scripts and transitive network behavior rather than approving a friendly package name.

The UK NCSC secure-AI development publication, developed with CISA and agencies from 17 other countries, covers secure design, development, deployment, operation, supply chains, assets, infrastructure, monitoring, and responsible disclosure (NCSC, 2023). It is broad lifecycle material, not certification that a component is secure.

Data supply chains need the same discipline. Preserve provenance, access control, review status, and transformation history. Separate trusted policy material from user content. Detect poisoning, unauthorized additions, mass changes, unusual retrieval influence, and feedback loops that let an attacker promote generated content into future authority.

Test controls as an adaptive adversary

Security evaluation should exercise end-to-end effects, not only whether the model says a forbidden phrase. Build tests around threat scenarios:

• direct, indirect, multilingual, encoded, visual, multi-turn, and persistent prompt injection;
• data exfiltration through URLs, images, DNS, errors, logs, files, messages, and allowed third parties;
• privilege escalation, confused-deputy use, cross-tenant access, stale approval, and delegated-authority expansion;
• malicious Tool results, MCP servers, Skills, dependencies, callbacks, and compromised child agents;
• unsafe model output reaching SQL, shell, browser, templates, code, paths, and policy engines;
• sandbox escape, mounted-secret discovery, metadata-service access, network redirect, and artifact smuggling;
• retrieval, memory, feedback, evaluation, and configuration poisoning;
• repeated adaptive attacks, rate-limit evasion, resource exhaustion, and denial of wallet; and
• control outage, timeout, version drift, failover, rollback, and incident containment.

Start with deterministic unit and integration tests for enforcement points. Add adversarial simulations and qualified human red teams for semantic and chained attacks. Test the same source against several sinks and the same sink through several sources. An isolated prompt-injection benchmark can miss the product path that turns manipulation into harm.

OWASP’s 2025 LLM Top 10 is a useful community checklist spanning prompt injection, sensitive disclosure, supply chain, poisoning, improper output handling, excessive agency, vector weaknesses, misinformation, and unbounded consumption (OWASP, 2025). It is an awareness taxonomy, not a complete threat model, control standard, or evidence that checking ten boxes secures an application.

Red-team safely: isolate targets and tenants, use synthetic or authorized data, prevent real external effects, coordinate with providers, preserve evidence, and follow responsible disclosure. Re-run tests after model, prompt, policy, interface, data, dependency, and sandbox changes. Security claims decay as both systems and attackers change.

Operate the controls

Assign owners for threat models, identities, data classes, policies, sandboxes, supply-chain assets, guardrails, vulnerabilities, and exceptions. Monitor:

• denied and allowed high-risk operations;
• unusual data access, destinations, credential use, and cross-tenant attempts;
• prompt-injection and exfiltration signals without treating detectors as ground truth;
• changes to models, prompts, policies, permissions, integrations, data, and guardrail versions;
• sandbox violations, egress denials, output-validation failures, and repeated approvals;
• false-positive impact, user overrides, exception age, and control availability; and
• discovered vulnerabilities, patch state, and residual exposure.

The Observability and Incident Response chapter owns detection pipelines, evidence handling, and coordinated response. This chapter defines the security signals and containment controls they operate. Preserve enough evidence for investigation without turning security logs into another ungoverned store of prompts, secrets, and personal data.

Trade-offs

Choice	Benefit	Cost or risk
Broad context and credentials	More tasks succeed without interruption	Larger prompt-injection and disclosure blast radius
Narrow per-task delegation	Stronger least privilege and attribution	More policy, identity, and integration work
Model-based semantic guardrails	Flexible coverage of open-ended content	Probabilistic errors, drift, evasion, cost, and latency
Deterministic allowlists	Clear assurance for bounded operations	Limited flexibility and maintenance burden
Strict network isolation	Strong exfiltration containment	Blocks legitimate discovery and external dependencies
User confirmation	Adds contextual control and accountability	Friction, fatigue, manipulation, and weak review at scale
Full security logging	Richer investigation evidence	Sensitive-data concentration and insider/access risk
Minimal logging	Lower privacy exposure	Weaker detection, attribution, and forensics
Automatic memory	Personalization and continuity	Persistent injection, scope leakage, correction, and deletion burden
Frequent component updates	Faster fixes and capability improvements	Supply-chain drift and insufficient regression testing

Failure modes

Prompt wording as a security boundary

The system tells the model to ignore malicious instructions but gives it unrestricted private data, credentials, and network access.

Guardrail equals authorization

A classifier approves a request, so the executor skips principal, resource, tenant, purpose, and current-state checks.

Installed means trusted

A plugin, MCP server, Skill, model, or connector becomes callable with broad authority because it appears in a registry or was added by an administrator.

Read-only interface with a write-capable identity

The model sees only a read operation, but a compromised integration or generated query can reach credentials that permit mutation.

Secrets inside context

Credentials are embedded in prompts or Tool output and later appear in model output, traces, memory, provider logs, or attacker-controlled network requests.

Approval generated by the attacker

An injected document supplies a plausible explanation for a dangerous action; the interface presents the explanation without verified target, data, or effect details.

Sanitized text, dangerous sink

Content passes a toxicity or injection classifier and is then concatenated into SQL, shell, HTML, a URL, or an access-control expression.

Sandbox with open egress

Filesystem writes are contained, but the process can read mounted credentials or private source and send them anywhere.

Provenance lost through summarization

Untrusted external content is summarized and stored as trusted memory, allowing a persistent injection to cross future sessions.

Cross-tenant retrieval before authorization

Similarity search retrieves private candidates across tenants and filters them after ranking or after model exposure.

Delete the source, retain every derivative

The application removes the primary record but leaves embeddings, prompts, traces, evaluation cases, caches, backups, and memories searchable.

One-shot red-team success

The system blocks a small fixed attack set once and declares the control secure without adaptive attempts, end-to-end sinks, benign false positives, or repeated trials.

Fieldbook guidance

Maintain a security and privacy design record for each consequential capability:

Operating claim, users, and consequence level:
Assets and unacceptable outcomes:
Actors, adversaries, and assumptions:
Input sources, trust labels, transformations, and persistent stores:
Data classes, purpose, providers, regions, retention, and deletion path:
Principals, delegated authority, credentials, and approval scope:
Reachable capabilities, sinks, destinations, and external effects:
Source-to-sink attack paths:
Deterministic enforcement points and fail behavior:
Model-based guardrails, calibration, and known error rates:
Sandbox, filesystem, process, tenant, and network boundaries:
Supply-chain inventory, provenance, versions, and update policy:
Monitoring, containment controls, and evidence policy:
Adversarial tests, repeated-trial results, and residual risk:
Owner, exception authority, compensating controls, and expiry:

Use one concise rule during review:

Assume the model can be persuaded. What can it see, what can it cause, and which non-model boundary still prevents an unacceptable result?

If the answer depends only on the model refusing, narrow the data or effect surface or add an enforceable control.

Checklist

• Does the threat model cover the assembled application, conventional software risks, model-specific risks, supply chain, operators, and external services?
• Are protected assets, actors, principals, tenants, trust boundaries, input sources, and consequential sinks explicit?
• Have source-to-sink paths been tested for direct, indirect, persistent, and multi-agent prompt injection?
• Does each run receive only the data, interfaces, destinations, credentials, time, and effect limits it needs?
• Are authentication, authorization, approval, and credential resolution enforced outside model judgment at access time?
• Are credentials absent from model context, untrusted files, traces, memory, errors, and generated code?
• Are network egress, redirects, metadata services, DNS, protocols, and data-bearing URLs controlled?
• Are shells, browsers, code, files, and open-ended interfaces isolated according to their real threat and tenant model?
• Is model output validated and encoded for each downstream sink rather than trusted because it is structured?
• Do provenance and trust labels survive retrieval, extraction, summarization, memory, delegation, and logging?
• Are data purpose, minimization, provider flow, region, retention, correction, deletion, and derived copies documented and implemented?
• Does each guardrail name its threat, enforcement point, action, timeout, failure behavior, evidence, and known limits?
• Are model-based guardrails calibrated across benign and adaptive malicious cases, languages, obfuscation, repeated trials, and version changes?
• Does approval show verified operation, target, data, destination, consequence, state, and changes since prior approval?
• Are models, prompts, policies, dependencies, containers, integrations, Skills, MCP servers, data, and update channels inventoried and governed?
• Are tenant separation, delegated authority, child work, callbacks, and stale queued requests enforced end to end?
• Do security tests verify actual data disclosure and external effects, not only model text?
• Are red-team environments authorized, isolated, non-destructive, and connected to remediation and regression cases?
• Are control health, denials, overrides, unusual access, destinations, exceptions, vulnerabilities, and residual exposure observable?

References and how they are used

• OpenAI, “Designing AI agents to resist prompt injection” and “Understanding prompt injections” — OpenAI security articles used for a social-engineering and source–sink threat model, layered defenses, constrained effects, confirmation, monitoring, and ongoing red teaming. They describe OpenAI’s current approach and do not establish universal immunity.
• OpenAI, “Keeping your data safe when an AI agent clicks a link” — OpenAI security-engineering article used for URL-based data exfiltration risk and destination-aware controls. The product mechanism is one layer and should not be generalized into a complete egress policy.
• Anthropic, “How we contain Claude across products” — Anthropic engineering and incident article used for separate model, content, and environment defenses, traditional security foundations, isolation, egress controls, and the limits of model-layer protection. Reported product and benchmark rates are not generalized here.
• Anthropic, “Beyond permission prompts: making Claude Code more secure and autonomous” — Anthropic implementation article used for filesystem and network isolation around a coding agent. It is a product-specific sandbox design, not proof that every local or container boundary is secure.
• NIST AI 600-1, “Generative Artificial Intelligence Profile” — final voluntary US cross-sector risk-management profile used for lifecycle risk, privacy, provenance, supply-chain, measurement, and incident considerations. It is not a regulation or an agent-specific control standard.
• UK NCSC et al., “Guidelines for Secure AI System Development” — international public secure-AI development publication developed with CISA and agencies from 17 other countries, supporting secure-by-design ownership across design, development, deployment, operation, supply chain, infrastructure, monitoring, and disclosure. Version 1.0 is broad lifecycle material rather than certification criteria.
• OWASP, “Top 10 for LLM Applications 2025” — open-community awareness taxonomy used to check coverage of prompt injection, disclosure, supply chain, poisoning, output handling, excessive agency, retrieval, misinformation, and consumption risks. It is neither exhaustive nor a substitute for a product threat model and tested controls.