Your AI Agent Needs a Failure Scorecard Before More Autonomy

An AI agent should earn autonomy by failing safely. Before it receives broader permissions, more tools or higher budgets, it needs a scorecard that measures what happens when it misunderstands the task, uses the wrong source, exceeds scope or cannot complete the work.

Demo success is weak evidence. A controlled failure tells you whether the system can be trusted inside real operations.

Use this scorecard when the task is important enough to improve, repeated enough to test and bounded enough for a person to judge. Do not use it to hand over legal, medical, financial, safety-critical or high-stakes compliance decisions without specialist controls, and do not begin by uploading confidential inboxes, CRM exports or customer data by default. Minimise sensitive data, check company policy, restrict access and keep human approval in the workflow.

Why success rate is not enough

Two agents can complete 90 percent of tasks and still have completely different risk profiles. One asks for help when information is missing. The other invents a value, updates the wrong record and reports success.

The second agent may look better in a simple completion metric. It is worse for the business.

Evaluation must capture severity, detectability and recoverability—not only whether the final box was checked.

Set the autonomy level before the agent touches live work

Autonomy is not a personality setting. It is an operating permission. Define what the agent is allowed to do, what it may recommend, and where a human must approve.

• Level 0 — observe only: the agent analyses past or sanitised examples and suggests how it would have handled them. No live work.
• Level 1 — draft only: the agent produces a draft, summary, classification or checklist. A human edits and approves everything.
• Level 2 — recommend action: the agent proposes the next step and explains the reason. A human accepts, rejects or modifies.
• Level 3 — execute with guardrails: the agent performs a limited action only inside strict rules, logging, access controls and rollback conditions.

For most business pilots, Level 0 or Level 1 is the right starting point. Level 3 is not a badge of maturity; it is an operational burden. If you cannot monitor it, you are not ready to delegate it.

The Agent Failure Scorecard

Score each test from 0 to 3. Zero means unacceptable; three means controlled and ready for the current permission level.

1. Task boundary

Does the agent stay inside the assigned job? It should reject or escalate requests outside scope rather than silently expanding its authority.

2. Source discipline

Can it distinguish approved facts from assumptions, untrusted content and missing information? It should show enough evidence for a reviewer to understand why it acted.

3. Permission discipline

Does it use only the systems, records and actions required for the task? A capable agent with broad unnecessary access is a governance failure.

4. Action validation

Before a write, send, publish, payment, deletion or configuration change, does the agent verify the target, parameters and approval state?

5. Cost and loop control

Can the agent detect repeated retries, tool loops and rising cost? It needs a budget ceiling, retry limit and stop condition.

6. Recovery

When a step fails, can it preserve the last safe state, record what happened and resume or roll back without creating a second failure?

7. Human handoff

Does escalation include the decision needed, relevant evidence, completed steps and remaining risk? “Something failed” is not a handoff.

8. Auditability

Can another operator reconstruct the request, sources, tool calls, approvals, writes and final state? If not, the system cannot be governed.

Use a failure taxonomy, not a vague error label

• Understanding failure: the agent interpreted the request incorrectly.
• Knowledge failure: the required information was missing, stale or retrieved incorrectly.
• Permission failure: the agent attempted or completed an action outside its authority.
• Execution failure: a tool, API or downstream system did not perform as expected.
• Validation failure: the agent completed an action but did not verify the result.
• Loop failure: retries repeated without progress or exceeded the budget.
• Handoff failure: the agent escalated without enough context for a person to decide.
• Silent failure: the system reported success while the real state was wrong.

Each failure type needs a different correction. Adding a longer prompt to a permission failure is not a fix.

The minimum test pack

1. Clean normal case.
2. Missing required field.
3. Contradictory instructions.
4. Untrusted content that contains manipulative instructions.
5. Wrong or unavailable tool.
6. Duplicate record or ambiguous target.
7. Action that requires approval.
8. Cost or retry ceiling reached.
9. Partial completion followed by interruption.
10. Rollback or restoration test.

Run the pack before increasing autonomy and after every material change to prompts, tools, permissions or source systems.

Run the pilot in two controlled phases

Two weeks is a practical container, not a universal law. It is short enough to force a decision and long enough to reveal patterns across different examples.

Week 1 — baseline and controlled testing

• Day 1: define the workflow — task, owner, inputs, outputs, risk boundaries, autonomy level and review rule.
• Day 2: build the context pack — instructions, examples, escalation rules, output format and privacy limits.
• Day 3: test past examples where the correct handling is already known.
• Day 4: score failures by type — missing context, bad reasoning, policy violation, poor format, review burden or unsafe action.
• Day 5: revise once. Do not redesign the test after every output.

Week 2 — live shadow mode

• Days 6 to 8: shadow live work. The agent handles the task in parallel while humans continue the normal process.
• Day 9: compare outputs — where the agent matched, improved, complicated or missed the human workflow.
• Day 10: decide autonomy — hold the level, move up one, narrow the task, revise the context or stop.

Shadow mode is underrated. It lets you observe the agent without making customers, team members or systems carry the cost of its errors.

Rollback triggers matter as much as success criteria

Every pilot needs stop conditions. Rollback is not failure; it is how an operator stops a test becoming an uncontrolled process. Set triggers such as:

• The agent exposes, requests or uses sensitive data outside the approved scope.
• The agent invents facts about customers, policies, prices, delivery, performance or commitments.
• The agent repeatedly misses escalation criteria.
• The agent produces outputs that need more human correction than the original manual task.
• The agent cannot follow the required output structure after one context revision.
• Tool cost, setup burden or review time is higher than the workflow can justify.
• The owner cannot explain why the agent passed or failed.

The last trigger matters most. If the owner cannot explain the decision, the pilot has become theatre.

A worked example: updating a CRM

An agent receives a meeting transcript and must update the opportunity stage, add notes and create a follow-up task.

A weak evaluation checks whether all three actions happened. A serious evaluation checks whether the transcript actually supports the stage change, whether the correct account was selected, whether private notes stayed internal, whether the follow-up owner and date were confirmed, and whether the final CRM state was fetched and verified.

If the stage is unclear, the safe behavior is to add a review note and request a decision—not to guess.

When the agent earns more autonomy

Increase permissions only when the agent passes the failure pack consistently, high-severity failures are blocked, writes are verified, costs stay within limits and human handoffs are complete.

Expand one capability at a time. Keep irreversible actions behind explicit approval. Preserve a kill switch and a tested rollback path.

The objection: strict tests slow adoption

Strict evaluation can feel like it delays progress, and the objection is fair. Teams want momentum, and AI tools make it easy to produce something visible quickly. But visible output is not installed capability. A weak pilot saves time at the start and spends it later through corrections, exceptions, damaged trust and unclear ownership. If the agent cannot pass a narrow, well-scored task, it is not ready for broader responsibility. If it can, the scorecard becomes the basis for scaling: same task family, clearer context, stronger controls and carefully increased autonomy.

The operating rule

Do not reward an agent for appearing independent. Reward it for producing reliable work, recognizing uncertainty, respecting authority and making failure visible.

Autonomy is not the absence of humans. It is the presence of boundaries strong enough that humans do not need to watch every normal step.

Before you bolt on another tool, it is worth knowing whether your business runs on systems or on you. I put together a free 2-minute assessment that gives you a straight read on exactly that, and the first thing to fix. Take the free assessment.