How to Secure AI Agents: Design for a Compromised Model
If prompt injection cannot be prevented, how do you ship AI agents safely? The answer is the same one we used for SQL injection and memory bugs: contain what you cannot eliminate. Here is how.

There is currently no reliable way to stop a language model from following instructions hidden in the content it reads. That sounds like it makes AI agents indefensible. It does not.
We have been here before. Before parameterized queries were universal, we contained SQL injection with least privilege and monitoring. Before memory-safe languages, we contained memory corruption with exploit mitigations. Security came from the architecture around the vulnerable component. LLM applications are the same.
The core principle: assume the model is compromised
Design as if every model output could be controlled by an attacker. If any untrusted content reaches the model’s context (a web page, an email, a document, a tool result, or a user message), then everything the model does next, including its tool calls, may be attacker-chosen.
Five practical rules follow:
- The model is never the only thing standing between an attacker and something sensitive.
- Authorization decisions are made by deterministic code using the authenticated user’s identity, never by the model.
- Every tool is safe to call with any parameters the model might produce.
- Model output is validated and encoded before it reaches any interpreter or renderer.
- High-impact actions require human confirmation from someone who can see exactly what will happen.
Shrink the blast radius
Least privilege for tools. Prefer narrow, purpose-built tools over general ones. A get_my_recent_orders() tool is far safer than run_sql(query). A create_draft_reply() tool is safer than send_email(to, subject, body).
Bind tools to the user’s identity. Tools should get the user from the server-side session, not from a parameter the model fills in:
# FIXED: the model supplies no identity at all
def make_tools(session):
def get_my_subscription() -> dict:
return billing_api.get_subscription(
account_id=session.account_id, # from the server-side session
auth=session.user_scoped_token) # user-scoped, not a service account
return {"get_my_subscription": get_my_subscription}
Make confirmation meaningful. Show the exact action in an interface the model did not write: “Send email to records@attacker.example with 5 attachments,” not a model-written “Archive your emails?” And reserve confirmation for consequential actions, or users will click “yes” without reading.
Sandbox anything that runs code. Agents that execute code or browse should run in disposable containers or VMs with no production network access, no ambient credentials, and egress controls.
Break the lethal trifecta
An agent with private data, exposure to untrusted content, and the ability to communicate externally can be made to leak data. Make sure no single agent context has all three:
- Processes untrusted content and can communicate out? Keep private data away from it.
- Has private data and reads untrusted content? Remove outbound channels: no network tools, no external image rendering, no writing where an attacker can read.
- Has private data and can communicate out? Never let it read untrusted content.

Architectural patterns that give real guarantees
Action selector. The model picks from a fixed menu of predefined actions and never sees their results. Injection can at most choose the wrong harmless action.
Plan-then-execute. The model writes its plan before seeing untrusted content. Tool outputs can corrupt data flowing through the plan, but cannot add new steps. “Summarize my latest email and send it to my manager” fixes the recipient before any email is read.
Dual LLM. Proposed by Simon Willison in 2023, this pattern splits the work between two models. A privileged model with tools never sees untrusted content directly. A quarantined model with no tools processes the untrusted content and returns results as symbolic variables that code, not the privileged model, substitutes into the final output.
Capability-based systems such as CaMeL go further, tracking where every piece of data came from and enforcing policies on what tainted data is allowed to do.
Input and output defenses: useful layers, not walls
On the input side, instruction hierarchies, clear labeling of untrusted content, stripping hidden text like invisible Unicode and HTML comments, and injection classifiers all lower attack success rates.
On the output side:
- Render safely. Disallow raw HTML, sanitize Markdown, and restrict image and link destinations with a strict CSP.
- Validate tool calls in code. Check recipients against an allowlist, enforce limits (a refund cap belongs in the tool, not the system prompt), and reject anything outside policy.
RAG and MCP need the same discipline
Retrieval-augmented generation must be permission-aware: filter documents by what the current user is allowed to see before they reach the model, and keep tenants strictly isolated.
MCP and tool integrations need vetting, version pinning, and least privilege. Review tool descriptions, which can carry hidden instructions, and alert when a server’s tools change.
Operate it like any other security-critical system
- Red team continuously, with automated suites and human testers.
- Log prompts, tool calls, and decisions (without secrets), and alert on anomalies such as unusual tool usage or outbound requests.
- Have an incident plan for AI features, including a way to disable tools quickly.
- Use governance frameworks like the OWASP Top 10 for LLM Applications and NIST’s AI Risk Management Framework to organize the program.
A pre-launch checklist
Before shipping an LLM feature, confirm that:
- Tools are narrow and bound to the user’s identity.
- Authorization is enforced in code, not prompts.
- No context combines all three legs of the lethal trifecta.
- Consequential actions need clear human confirmation.
- Output is rendered safely and tool calls are validated.
- Retrieval respects user permissions.
- Monitoring and a kill switch exist.
Key takeaways
- Design every LLM feature to stay safe when the model is fooled.
- Least-privilege tools, identity binding, meaningful confirmation, and sandboxing limit the blast radius.
- Break the lethal trifecta so no context can read secrets, read attacker content, and send data out.
- Action-selector, plan-then-execute, dual LLM, and capability patterns give stronger guarantees than prompts.
- Treat input filters as helpful layers, and keep the real guarantees in deterministic code.
Go deeper with the book
This article is adapted from Web Security in the Age of AI: Attacks, Defenses, Tools, and Bug Hunting for Modern Web Applications and LLM Systems by Rupak (see Chapter 24, “Defending LLM Applications and Agentic Systems”).
The book pairs every attack with its fix across 27 chapters, from core vulnerabilities and professional tools to bug bounty hunting, attacking and defending LLM applications, and lessons from real breaches. It includes hands-on labs, review questions, a full testing checklist, a security headers reference, and an LLM red teaming playbook.
Related reading
- Prompt Injection Explained: How Attackers Hijack LLM Apps
- Broken Access Control and IDOR: The #1 Web Security Risk







