Introduction
AI agents no longer only generate answers. They call tools, modify systems, communicate externally, and act inside operational environments.
This shifts the architectural question. The challenge is no longer only whether an agent can complete a task, but how much authority the surrounding system should grant it.
Trust should not be inferred from fluency, benchmark performance, or apparent competence. It should be granted incrementally, for specific actions, based on evidence and enforced boundaries.
Your New Intern Has Production Credentials
Imagine hiring a technically impressive intern and, on the first day, giving them access to the production repository, CI/CD pipeline, cloud credentials, customer data, and incident channels.
They may write excellent code and solve difficult tasks faster than expected. None of that would justify unrestricted access on day one.
Organizations usually separate capability from authority:
Capability asks: Can this person perform the action?
Authority asks: Should this person be allowed to perform it?
New team members typically begin with limited access, supervised work, and reversible tasks. Responsibility increases after their work has been reviewed and their judgment tested in context.
The same architectural principle should apply to AI agents.
The analogy is not about intelligence, personhood, or malicious intent. It is about delegated authority. An agent may be highly capable and still behave unpredictably under ambiguous instructions, unfamiliar inputs, or incomplete context.
Distrust does not mean expecting the agent to betray the system. It means that demonstrated competence alone is not enough to justify authority.
Trust Is Scoped to Actions, Not Agents
Calling an agent “trusted” says little about how it should operate.
The practical unit of trust is the path between an agent’s decision and its production consequences. That path may be advisory, mediated, or direct.
At the advisory level, the agent produces recommendations, drafts, or proposed code changes but cannot alter production.
At the mediated level, it initiates changes through enforced controls such as code review, policy checks, approval gates, transaction limits, or constrained workflows.
At the direct level, it can change production systems without prior approval, potentially affecting deployed software, customers, finances, or business continuity.
These levels should not be assigned to the agent as a whole. The same coding agent may edit an isolated branch, require approval before merging, and remain prohibited from deploying to production. Each action creates a different path to business impact and requires a separate trust decision.
This contextual view is consistent with NIST’s treatment of AI trustworthiness, which depends on the system’s context of use rather than a single global property.
The closer an agent’s decisions move toward direct production impact, the stronger the required controls and evidence must become.
Architecture Is Where Trust Becomes Real
A trust decision has no operational value unless the system enforces it.
Prompts can instruct an agent to act cautiously, request approval, or avoid resources. They influence behaviour but do not define actual authority. If connected credentials permit unrestricted access, the system still depends on the model choosing not to use it.
Architecture narrows or removes that choice.
Read-only access is not sufficient when sensitive information remains reachable. An agent may still read secrets, expose them in logs, or pass them to another process. Credentials should remain outside the execution environment wherever possible, supported by redaction, restricted mounts, egress controls, and policy-enforced limits on protected files and environment variables. Shell-command filtering can add another barrier, but should not be the primary security boundary.
Narrowly scoped credentials limit resources. Isolated environments contain execution. Separate proposal and execution paths prevent the acting agent from authorizing its own changes. Network restrictions, transaction limits, and external policy checks constrain what can happen after an incorrect decision.
This matches Anthropic’s containment model, which places hard boundaries around filesystems, credentials, processes, and networks, and OpenAI’s sandbox guidance, which separates model-directed execution from trusted control-plane functions.
A coding agent on an isolated branch may still introduce defective code. The architecture ensures that the defect cannot reach production without crossing additional boundaries.
The purpose of trust architecture is not to prevent every model failure. It is to prevent model failure from automatically becoming system failure.
Blast Radius Determines the Trust Threshold
Not every agent action requires the same level of trust.
Drafting an internal summary is not equivalent to emailing a customer. Editing an isolated branch is not equivalent to deploying production code. Recommending a refund is not equivalent to issuing one.
The difference is the blast radius: reversibility, recovery cost, downstream dependencies, regulatory exposure, and the number of systems or people affected by an incorrect action.
An agent may receive broad autonomy in a contained environment while remaining tightly restricted in production. Reversible, limited-impact actions may justify fewer controls. Irreversible actions affecting customers, finances, or infrastructure require stronger evidence, narrower permissions, and additional boundaries.
Threat modeling supports this decision by identifying sensitive resources, trust boundaries, failure paths, and consequences. Its purpose here is not to predict every attack, but to determine where autonomy becomes operationally dangerous.
The principle aligns with NIST’s zero-trust guidance: access should be granular, continually evaluated, and limited to the minimum privileges required for a particular request.
The question is not simply whether the agent performs well, but what happens when it does not.
Authority Should Grow Through Evidence
Once the blast radius is understood, greater autonomy should be justified through evidence rather than confidence.
Evidence should come from progressively more realistic conditions. An agent may first be evaluated offline, then observed in shadow mode, granted read-only production access, allowed to execute under supervision, and eventually permitted to act autonomously within defined boundaries.
This is not a universal maturity model. Its purpose is to keep authority proportional to what has been demonstrated.
Relevant evidence extends beyond task completion. It may include unauthorized-action rates, intervention frequency, recovery after errors, behaviour under ambiguous instructions, response to tool failures, and performance under adversarial conditions.
Anthropic’s guidance on agent evaluations emphasizes that agents operate over multiple turns, use tools, modify state, and adapt to intermediate results. A successful final response may therefore hide an unsafe trajectory or incorrect external state.
A verifier or adversarial agent can check logic, generate failure cases, and attempt to break the workflow. However, it is not fully independent evidence when both agents share the same model family, tools, data, or assumptions. Automated verification should be combined with deterministic tests, trace inspection, red teaming, and human review where justified. NIST’s AI RMF Playbook similarly recommends deployment-like evaluation, monitoring, and recurring red-team exercises.
Authority can increase only inside boundaries supported by the evidence. Strong performance in one workflow does not justify broader access elsewhere.
Human Approval Is a Boundary, Not a Safety Model
Human approval is valuable at consequential decision points. It is less effective as a generic safeguard for an otherwise overpowered system.
Three cases require different responses.
Permitted but consequential actions should pause before execution. Production deployments, large refunds, external communication, and financial commitments may be legitimate, but their impact justifies explicit authorization. Google Cloud, OpenAI, and LangGraph all describe review checkpoints before sensitive actions execute.
Forbidden actions should be blocked automatically, not converted into approval requests. An agent without production access should not request an exception through the same workflow. Filesystem restrictions, credential isolation, network controls, and external policy enforcement establish boundaries outside the model’s discretion.
Uncertainty or repeated failure should transfer control before the agent improvises beyond its demonstrated competence. Retry limits and failure thresholds provide explicit escalation points.
Repeated confirmation prompts do not necessarily strengthen oversight. Anthropic reports that users approved roughly 93% of Claude Code permission prompts and paid less attention as approvals accumulated. Stronger sandboxing later reduced those prompts by 84%.
Routine actions should proceed within enforced boundaries. Consequential actions should pause for authorization. Forbidden actions should not execute.
Trust Must Remain Revocable
Authority should not become permanent because an agent performed reliably in the past.
The evidence supporting trust applies to a specific deployment configuration: model, instructions, tools, credentials, memory, environment, and task. Changing any of these variables may introduce behaviour that previous evaluations did not cover.
A model upgrade may alter tool use. A newly connected service may create failure paths. Broader credentials may turn a contained mistake into a production incident. Even an unchanged agent may encounter tasks or data outside the conditions under which its authority was justified.
Trust therefore requires continuing operational evidence. Actions and outcomes should remain observable, blocked attempts should be recorded, and material changes should trigger renewed evaluation. The architecture must preserve ways to reduce permissions, interrupt execution, roll back changes, or disable the workflow.
This does not mean trust expires after a fixed period. It means supporting evidence can become outdated. NIST’s AI RMF Playbook recommends post-deployment monitoring, change management, incident response, recovery, override mechanisms, and decommissioning when systems exceed risk tolerances.
Revocation is not merely an incident-response mechanism. It is part of the original trust design.
Trust that cannot be reduced, withdrawn, or re-evaluated is permanent access.
Trust Is an Architecture Decision
AI agents provide capability. Architecture determines how much of that capability can reach production, under which conditions, and with what consequences.
Trust is not a permanent label attached to an agent. It is a series of scoped decisions: which actions may be delegated, which boundaries must remain enforced, what evidence justifies greater autonomy, and when that authority should be withdrawn.
A trustworthy system does not depend on the model always making the correct decision. It limits the impact of incorrect decisions, preserves meaningful approval points, blocks forbidden actions, and keeps execution observable and recoverable.
This principle is reflected in current production guidance. Anthropic describes containment as hard technical boundaries, while OpenAI separates automatic guardrails from human approval for sensitive side effects.
The relevant question is not:
Is this agent trustworthy?
It is:
Which actions can this system safely delegate, under which conditions, based on what evidence, and with what recovery path?
Trust is not something an AI model possesses. It is the authority an architecture is prepared to grant—and the evidence, boundaries, and consequences that justify that decision.