Skip to content

Where an AI agent should stop

A practical approval model for agents that read, recommend, and change business records.

Giving an agent access to a tool is a decision about authority. Reading a project record, proposing a change, and applying that change are different responsibilities. A useful implementation makes those differences explicit before connecting the agent to live work.

Consider an agent that reviews a project board after a meeting. It finds a possible change to a delivery date. The right output might be a proposed update with its source, not an immediate edit. The following is a design example, not a report of a client deployment.

Define the action before the agent

Start with a narrow job: propose updates to existing project tasks from a confirmed decision record. Specify which projects it can read, which fields it may propose changing, and who can authorize each change. Exclude actions the job does not need, such as deleting tasks or sending messages to clients.

OWASP identifies excessive functionality, permissions, and autonomy as sources of excessive agency. Its guidance recommends limiting tools and permissions, requiring approval for consequential actions, and enforcing authorization outside the model. Those controls are a useful starting point for this design.

Reference: OWASP: Excessive Agency

Make approval specific

A request to approve should show the exact change, its destination, the evidence behind it, and its likely consequence. A general instruction to improve a project board is not authorization to change every date on it.

Keep approval attached to a specific proposal. If the target record or proposed values change before execution, return the proposal for review. A previous approval should not silently carry over to a different action.

Example approval record
FieldWhat it establishes
Target and current revisionThe exact record being changed
Before and after valuesThe complete proposed change
Evidence referenceWhere the decision came from
Approver and scopeWho authorized this action
Expiry and outcomeWhether approval is still valid and what happened

Keep evidence separate from instructions

Meeting notes, email, and retrieved documents are inputs to review. Their contents should not be able to expand tool permissions or override approval requirements. If a document tells an agent to ignore its restrictions, that text remains untrusted input.

Use the application to check identities, allowed fields, and permissions on every write. A sentence in a prompt is not an access-control mechanism. Show uncertainty when the evidence does not establish a decision; do not manufacture a confident recommendation to fill the gap.

Reference: OWASP: Excessive Agency

Verify the result, including partial failure

For a change spanning several systems, decide what success means in each system. Updating a task board does not prove that the delivery schedule or customer record was updated. Report completed, failed, and untouched actions separately.

Use a stable operation identifier to prevent unintended duplicates where the receiving system supports it. After an uncertain response, check the destination before retrying. Record enough to investigate an action without copying whole private documents into logs.

Some actions cannot be reversed cleanly. In those cases, design a compensating action or an escalation path rather than presenting rollback as a universal safety net.

Test the boundary, not just the happy path

Before a pilot, test contradictory notes, a changed destination record, expired approval, an unavailable approver, a permission failure, and an interrupted write. Include a case where the correct result is no action.

Then test whether a reviewer can understand the proposal without reconstructing the entire conversation. Approval that requires too much investigation encourages superficial review.

A useful agent reduces effort within a defined authority boundary. Measure completed cases, human review time, corrections, and failures before expanding the pilot.

Sources

OWASP: Excessive Agency

Security guidance informing the permission and authorization controls. The workflow example and approval record above are proposed applications.

Working on a similar operating problem?

Related services