Do Not Invite an AI Agent Without a Scorecard

Do not add an AI agent to a team channel until you know what kind of mistake it is allowed to make. The wrong question is “Which agent should we use?” The operator question is “What failure is acceptable, who catches it, and when must the agent stop?”

A teammate-style agent can feel useful because people can tag it, delegate work, and move on. The risk is that the team starts treating it like a coworker before management has built the scorecard. This article gives you a lightweight Agent Performance SOP: golden tasks, pass/fail rubrics, human handoff rules, permission limits, and a weekly reliability review.

The tool choice comes after the failure map

The supplied Claude Tag announcement describes Claude joining Slack as a team member with access to the channels and tools a team chooses, so people can tag it and delegate tasks. That is not just a prompt interface. It moves AI into the operating rhythm of a team.

That does not make the agent dangerous by default. It makes the manager responsible for the work boundary. A human assistant is not hired with “try to be useful” as the only instruction. They get responsibilities, examples, approval limits, and escalation rules. Agents need the same thing, only more explicit.

The hidden mistake is evaluating agents by impressive demos. Demos reward clean situations. Operations punish the team when the agent mishandles an edge case, uses the wrong context, exposes sensitive information, or takes a confident action without authority.

Practical takeaway: before comparing agents, write down the failures you can accept. If you cannot name acceptable failure, you are not ready to delegate.

Define acceptable failure before you write the prompt

Acceptable failure is a mistake that does not damage the customer, the company, the data, or the decision. It may create rework, but it does not create uncontrolled risk.

For example, an agent drafting a first version of an internal meeting summary may miss a nuance. That is recoverable if a human reviews the summary before it becomes a decision record. An agent sending a final pricing promise to a customer without approval is different. The same writing skill becomes a business risk because the output leaves the company and creates expectation.

Use this failure classification before testing any agent:

• Green failure: The agent can be wrong, and the cost is limited to internal rework. Examples: rough drafts, first-pass classification, brainstorming options.
• Amber failure: The agent can assist, but a human must approve before the output affects a customer, vendor, financial record, legal position, or public statement.
• Red failure: The agent should not act. The task involves confidential judgement, irreversible action, regulated advice, account access, payment approval, hiring decisions, termination decisions, or sensitive personal data.

This is not legal or compliance advice. It is an operating filter. Your company policy, contracts, industry rules, and data controls still matter.

Practical takeaway: every agent task should have a failure color before it has a prompt.

The Agent Performance SOP

The Agent Performance SOP is a pre-rollout routine for any AI agent that will participate in team work. It is for founders, operations leads, marketing managers, support managers, agency owners, and technical leads who want to test agent reliability before real delegation.

Use it when an agent will read team context, draft outputs for other people, summarize conversations, recommend actions, or trigger follow-up work. Do not use it as a substitute for security, legal, compliance, or procurement review when those are required.

Required inputs

• Workflow name: The specific workflow where the agent may be used. Example: inbound support triage, sales follow-up drafting, campaign brief review, weekly project update.
• Task owner: The human responsible for the workflow result.
• Agent boundary: What the agent may read, draft, recommend, or trigger.
• Data boundary: What the agent may not receive by default, including confidential documents, personal data, customer secrets, credentials, private financials, and restricted internal conversations.
• Golden task set: 20 representative tasks that reflect normal work, messy work, and edge cases.
• Pass/fail rubric: The scoring rule for each task.
• Escalation map: When the agent must hand work to a human.
• Review cadence: A weekly reliability review owned by a named person.

Expected output

The SOP should produce a clear go, limited-go, or no-go decision for one agent inside one workflow. It should not produce a vague opinion such as “the agent seems good.” The useful output is operational: what the agent may do, under what limits, with which human approvals.

Practical takeaway: the SOP does not certify an agent as generally reliable. It certifies one agent for one workflow under named constraints.

Step 1: Build a 20-task golden set

A golden task is a realistic task with a known acceptable answer. It lets you judge the agent against your business standard, not against the agent’s confidence.

Build 20 tasks before giving the agent broad access. The tasks should come from the workflow you want to improve. Do not fill the set with easy examples. Easy examples test presentation. Messy examples test operating reliability.

Use this mix:

• 8 normal tasks: Common work the agent will see often.
• 4 incomplete-context tasks: Tasks where the right answer is to ask a clarifying question or refuse to assume.
• 4 policy-boundary tasks: Tasks involving data limits, approval limits, or authority limits.
• 2 adversarial tasks: Tasks where the request conflicts with the workflow rule, such as asking the agent to bypass approval.
• 2 edge-case tasks: Rare but plausible cases that would cause confusion or business risk.

Imagine a support workflow. A normal task may ask the agent to categorize a customer email. An incomplete-context task may omit the product plan or order status. A policy-boundary task may include private customer information that should not be repeated in a public-facing draft. An adversarial task may ask the agent to issue a refund without approval. An edge-case task may include a customer threatening public escalation if they do not receive an immediate answer.

Practical takeaway: the golden set should include the work you hope the agent handles and the work you fear it will mishandle.

Step 2: Score each task with a pass/fail rubric

A pass/fail rubric prevents the loudest person in the room from becoming the evaluation system. The agent either meets the business standard or it does not.

Use these five checks for each golden task:

• Instruction fit: Did the agent complete the requested task without drifting into extra action?
• Context discipline: Did it use only the provided or permitted context?
• Accuracy standard: Did it avoid unsupported claims, false details, and invented facts?
• Boundary respect: Did it follow data, approval, and authority limits?
• Escalation behavior: Did it ask for help when the task crossed a rule or lacked enough information?

For each task, mark one of three outcomes:

• Pass: The output is usable under the workflow rule with no material correction.
• Pass with review: The output is useful, but a human must adjust or approve it before use.
• Fail: The output creates unacceptable rework, risk, false confidence, data exposure, or unauthorized action.

Do not average away red failures. A single failure involving sensitive data, unauthorized action, or customer-impacting false information should block rollout for that task category until the workflow is redesigned.

Practical takeaway: an agent can be useful at drafting and still unfit for decisions. Score the task, not the personality of the output.

Step 3: Write the escalation map before deployment

An escalation map tells the agent when to stop and who should take over. Without it, the agent may continue generating plausible work exactly when the business needs a human decision.

Create escalation rules in plain language:

• If the request involves money, contract terms, pricing promises, refunds, payroll, legal language, compliance, account access, or sensitive personal data, stop and route to the task owner.
• If required information is missing, ask a clarifying question instead of filling the gap.
• If the user asks the agent to bypass a policy, state the boundary and route to the owner.
• If the output will be sent outside the company, label it as a draft until a human approves it.
• If the agent is uncertain, it must say what is uncertain and what input is needed.

The map also needs named owners. “Escalate to a human” is too vague. Use roles: support lead, account owner, finance approver, legal reviewer, project manager, or founder. The agent should not decide who has authority if the workflow has not already defined it.

Practical takeaway: escalation is not a failure of automation. It is the control that keeps automation inside the business boundary.

Step 4: Limit access before you test access

Agent access should start narrow. The supplied announcement describes a choice around channels and tools. That choice is an operating control point, not an admin detail.

Do not give broad workspace access because it is convenient. Start with the minimum channels, documents, and tools required for the golden task set. If a task needs customer context, consider whether a sanitized export, limited record, or approved summary can be used instead of raw private data. Check company policy before placing confidential, customer, employee, financial, or regulated data into any AI system.

Use this access rule:

• Read access: Only to the channels or documents needed for the tested workflow.
• Draft access: Allowed for low-risk outputs, but external-facing work stays labeled as draft until approved.
• Action access: Disabled by default unless the task is low risk, reversible, logged, and approved by the workflow owner.
• Admin access: Not granted for normal agent work.

The access question is not “Can the agent do more?” It is “What is the smallest permission set that lets it do the tested job?”

Practical takeaway: permissions are part of performance. An agent with unnecessary access is not more capable; it is harder to govern.

Step 5: Run a weekly reliability review

The first test is not the end of evaluation. It is the baseline. Agents operate inside changing workflows, changing documents, changing team habits, and changing expectations.

Run a weekly review for any active agent workflow. Keep it short and focused:

1. Review failed tasks: What did the agent get wrong, and was the failure green, amber, or red?
2. Review escalations: Did the agent stop at the right moments, or did it continue when it should have handed off?
3. Review human overrides: Where did people repeatedly correct the same issue?
4. Update golden tasks: Add new edge cases from the week’s real work.
5. Adjust access: Remove permissions that were not needed. Add permissions only when the task set proves the need.
6. Decide status: Continue, restrict, revise instructions, redesign workflow, or pause use.

Make the review owned by the workflow manager, not the software buyer. The person accountable for the business result should decide whether the agent is reliable enough for that workflow.

Practical takeaway: reliability is a management routine, not a one-time setup screen.

A mini-walkthrough: sales follow-up drafting

Suppose a team wants an agent to draft follow-up messages after sales calls. The wrong rollout is to add the agent to the sales channel and tell the team to tag it when they need help. That creates inconsistent inputs, inconsistent quality, and unclear approval rules.

The SOP version looks different:

1. Workflow: Draft follow-up emails after sales calls.
2. Allowed job: Produce a first draft based on approved call notes and permitted sales context.
3. Not allowed: Promise pricing, discounts, delivery dates, custom terms, or legal language.
4. Golden tasks: Include normal discovery calls, unclear buyer intent, missing budget details, a prospect asking for a discount, and a request involving sensitive internal information.
5. Rubric: The draft must reflect only provided context, ask for missing information when needed, avoid invented commitments, and mark the message as draft.
6. Escalation: Route pricing, legal, or unusual delivery promises to the account owner.
7. Review: Sales manager checks failed drafts weekly and updates examples.

The agent may still be useful. But now its usefulness is bounded. It drafts. It does not negotiate. It assists the rep. It does not become the account owner.

Practical takeaway: the cleanest agent workflows separate drafting from authority.

The objection: testing feels like delay

The objection is understandable. A 20-task evaluation feels like friction when the tool is available and the team wants to try it.

But skipping the scorecard does not remove work. It moves the work into production, where errors are harder to see and more expensive to fix. People will still evaluate the agent. They will just do it through scattered complaints, private corrections, and lost trust.

A small evaluation suite is faster than uncontrolled adoption because it creates a shared language. The team can say, “This passed normal drafting but failed policy-boundary tasks,” instead of arguing about whether the agent is “good.”

This is the operating principle behind serious Business Systems & Operations: define the decision standard before the tool enters the workflow. The same logic applies across practical AI in Practice work. AI is the engine. The operator is the architect.

Practical takeaway: testing is not delay. It is how you stop the team from outsourcing judgement to a tool they have not managed yet.

Agent rollout checklist

Use this checklist before inviting an AI agent into a team workspace or assigning it recurring work.

• Workflow named: The agent is assigned to one workflow, not general productivity.
• Owner named: A human owns the business result and the weekly review.
• Failure colors assigned: Green, amber, and red task categories are written down.
• 20 golden tasks built: Normal, incomplete-context, policy-boundary, adversarial, and edge-case tasks are included.
• Rubric written: Instruction fit, context discipline, accuracy, boundary respect, and escalation behavior are scored.
• Escalation map approved: The agent knows when to stop and which role takes over.
• Access minimized: Channels, documents, and tools are limited to the tested workflow.
• External outputs controlled: Customer-facing, vendor-facing, or public outputs require human approval unless explicitly approved as low risk.
• Sensitive data protected: Confidential uploads are avoided by default, and company policy is checked before private data is used.
• Weekly review scheduled: Failures, escalations, access, and new golden tasks are reviewed.
• Rollout status decided: Go, limited-go, or no-go is recorded for the specific workflow.

If you cannot complete the checklist, do not compensate with a better prompt. The missing piece is not wording. It is management.

Common questions before rollout

Can an AI agent be treated like a junior employee?

Only in the narrow sense that it needs scope, examples, review, and escalation rules. Do not treat it as accountable. A human owner remains responsible for the workflow result.

How many tasks are enough for an agent evaluation?

Start with 20 golden tasks for one workflow. That is enough to expose common failure patterns without turning evaluation into a long research project. Add new tasks from real failures during weekly reviews.

Should the agent ever take actions automatically?

Only after the action is low risk, reversible, logged, permissioned, and tested. Drafting and recommending should come before acting.

Start with the scorecard, then choose the agent

Teammate language is easy to understand. A teammate can be tagged. A teammate can answer. A teammate can take work from your queue.

But in operations, the name matters less than the control system. If the agent has no golden tasks, no pass/fail rule, no escalation map, and no review owner, you have not hired help. You have introduced unmanaged variation into the workflow.

Pick one workflow this week. Write 20 golden tasks. Mark the green, amber, and red failures. Then test the agent against the work before you invite it into the team’s daily rhythm.

Where does your business actually stand?

Before you bolt on another tool, it is worth knowing whether your business runs on systems or on you. I put together a free 2-minute assessment that gives you a straight read on exactly that, and the first thing to fix. Take the free assessment.