A Leaderboard Can’t Tell You Which Model Should Touch Your Refunds

Leaderboards can tell you which model looked strong in a test. They cannot tell you which model should touch your refund queue, codebase, research brief, or sales copy. The operating mistake is choosing a favorite model and building the business around it.

The better move is model routing: assign the right model to the right task based on risk, eval quality, latency, availability, privacy, and cost. Model loyalty is fragile. Workflow rules are safer.

The best default model is a fragile operating policy

A single default model feels clean because it removes choice. It also hides risk. If every AI task goes to the same model, your copy drafts, research summaries, support replies, and code suggestions inherit the same cost exposure, latency pattern, failure modes, and availability risk.

The latest AI news cycle shows why this is no longer a theoretical issue: GLM-5.2 drew attention for coding and agent work at a reported lower cost, Claude Fable and Mythos were described as restricted for a period, GPT-5.6 was described as moving through slower access, and analysts expected partial routing rather than a full switch from one model family to another. The point is not that one model permanently won. The point is that the default can become stale, expensive, slow, or unavailable while the workflow still needs to run. Read the source roundup here.

That is the operator lesson. Do not rebuild your company around whichever model is loud this week. Build routes. One model may be good enough for first-draft ad copy. Another may be better for source-grounded research. Another may be acceptable for low-risk classification. Another may be too limited in access to trust as the only path for a business-critical workflow.

For more tool analysis in this operating style, keep the Tools & Teardowns mindset: a tool is only as useful as the workflow it can survive.

Model routing is task assignment, not model fandom

Model routing means you decide which model handles which task before the prompt is sent. The decision can be manual at first. It can become automated later. The important part is that the route is based on business rules, not loyalty.

A practical routing policy answers five questions:

• What is the task? Draft, classify, summarize, research, code, compare, extract, respond, or decide.
• What happens if the output is wrong? Internal brainstorming is different from customer-facing advice, production code, or policy-sensitive support.
• How will quality be judged? A model should not be trusted for a workflow until the team knows what a good answer looks like.
• Which constraints matter? Latency, cost, availability, privacy, access control, and human review requirements.
• What is the fallback? If the preferred model is unavailable, slow, costly, or weak on the task, what happens next?

The underrated point: routing is not mainly a technical decision. It is an operating decision. The marketing lead may own copy routes. The engineering lead may own coding routes. The support manager may own customer response routes. Operations may own the fallback policy. A developer can build the switch. The operator must define when the switch should move.

The Model Router Scorecard

Use this scorecard when your team is choosing a model for a recurring workflow, not for one-off experimentation. It is for founders, operators, marketers, agency owners, developers, and team leads who need a defensible routing rule instead of a leaderboard screenshot.

Required inputs: task description, sample inputs, expected output, current human standard, available models, privacy constraints, cost sensitivity, speed requirement, and reviewer owner.

Score each factor from 1 to 5. A score of 1 means weak fit. A score of 5 means strong fit. Do not average blindly. Some factors are gates.

1. Task risk. How costly is a wrong output? A draft subject line is low risk. A customer refund answer, contract summary, or code change is higher risk. If task risk is 4 or 5, human approval is mandatory before external use.
2. Evidence requirement. Does the model need to use supplied sources only, reason from internal documents, or produce claims that must be verified? If the task needs factual precision, reject any model that invents unsupported details in testing.
3. Eval confidence. Can your team judge the output reliably? If you cannot tell whether the answer is good, the workflow is not ready for automation. Use the model for drafting or comparison only.
4. Output stability. Does the model produce the same structure and level of completeness across repeated runs? Stable formatting matters for automation. Creative variance may help ideation but can break handoffs.
5. Latency fit. Does the task need a fast response? A live support assist has different timing needs from a weekly research memo. Slow can be acceptable if quality matters more than speed.
6. Availability and access. Can the team actually use the model when the workflow runs? If access is limited, tiered, delayed, or uncertain, it should not be the only route for business-critical tasks.
7. Cost fit. Is the model worth using at the task volume? Cost should matter after quality and risk gates are passed. Cheap wrong answers are still expensive when humans must repair them.
8. Data sensitivity. Does the task involve customer data, confidential company information, private code, legal content, financial records, or personal information? Minimize inputs, remove sensitive fields where possible, and follow company policy before sending private data to any AI system.
9. Review path. Who approves the output, and what are they checking? If there is no named reviewer for high-risk tasks, the model should not produce final customer-facing or production-facing work.

Decision rule: choose the cheapest and fastest model only after it passes the minimum quality, availability, privacy, and review gates for that task. For high-risk work, eval confidence and review path outrank cost. For low-risk internal drafts, cost and speed can matter more. For automated workflows, output stability becomes a gate, not a preference.

Expected output: a one-line routing rule for each workflow. For example: support classification can use a lower-cost model if it matches policy labels during testing; customer response drafting uses a stronger model; final sending stays with a human. That is a routing policy, not model fandom.

Common failure to avoid: scoring the model instead of the workflow. A model can look impressive in a benchmark and still be the wrong choice for your messy inputs, tone rules, approval path, or data restrictions. Impressive is easy. Reliable is the work.

The 30-minute eval SOP for real work

Do not spend a week comparing models before you know whether a workflow deserves AI. Run a focused 30-minute eval first. The goal is not to crown a universal winner. The goal is to decide which model, if any, is safe enough for a specific task route.

Who it is for: the owner of a recurring task in copy, coding, research, or support.

When to use it: before choosing a default model, changing providers, reducing model cost, adding automation, or expanding AI access to a team.

Required inputs: three real but sanitized task examples, the current human-approved output standard, two or three candidate models, scoring criteria, and one reviewer who knows the work.

1. Minutes 0-5: define the task boundary. Write one sentence that says exactly what the model is allowed to do. Example: draft a first response to a support ticket using the supplied policy notes. Not: handle support.
2. Minutes 5-8: remove sensitive data. Strip customer names, private identifiers, confidential deal terms, secret keys, internal credentials, and anything the model does not need. If the task cannot be tested safely without sensitive data, pause and check company policy.
3. Minutes 8-12: prepare three test packets. Use one easy case, one normal case, and one edge case. The edge case is where weak routing is exposed. For support, that might be an angry customer asking for an exception. For research, it might be a source that contains conflicting claims.
4. Minutes 12-20: run the same task through each candidate model. Use the same input and constraints. Do not improve the prompt for one model and not the others. Keep the comparison fair enough to support a decision.
5. Minutes 20-26: score against task-specific criteria. For copy, check claim accuracy, offer clarity, tone fit, and whether the model invents proof. For coding, check whether the suggestion is minimal, testable, and clear about risk. For research, check whether it separates source facts from interpretation. For support, check policy accuracy, escalation judgment, tone, and whether it makes promises the company cannot keep.
6. Minutes 26-30: write the route. Choose one of four outcomes: approved for drafting, approved for internal assist only, approved for automated low-risk steps, or rejected for this workflow. Add the fallback model and the human review point.

Quality check: the winning route must be understandable to someone who was not in the eval. If the rule says only that Model A is better, it is not ready. If it says Model A drafts refund replies under policy notes, Model B handles classification, and a support lead approves exceptions, it is usable.

Copy, coding, research, and support need different gates

Each workflow needs its own pass/fail criteria. A model that writes persuasive copy may be weak at source-grounded research. A model that explains code clearly may be too slow or costly for bulk ticket labeling. Treat each task as a separate route.

Copy tasks

Pass criteria: the output matches the offer, avoids unsupported claims, respects the brand voice, and gives a human editor a usable first draft. The model does not need to be brilliant. It needs to stop creating cleanup work.

Eval example: give each model the same product notes, audience, offer, exclusions, and forbidden claims. Ask for three ad variants. Reject the model if it invents features, guarantees outcomes, or changes the offer to make the copy sound stronger.

Coding tasks

Pass criteria: the output is testable, limited in scope, and clear about assumptions. For code-related work, the model should not be treated as the final authority. It is a drafting and reasoning assistant unless your engineering process has tests, review, and rollback.

Eval example: give a small bug description, relevant sanitized code context, expected behavior, and constraints. Compare whether the models suggest focused changes or rewrite the world. The safer route is often the model that makes the fewest unnecessary edits.

Research tasks

Pass criteria: the model uses supplied material, separates facts from interpretation, flags missing evidence, and refuses to fill gaps with confident fiction. Research failures are dangerous because they often sound polished.

Eval example: provide two short source excerpts with a mild contradiction. Ask for an executive brief that labels supported facts, uncertainty, and recommended next questions. Reject any model that smooths over the contradiction without telling you.

Support tasks

Pass criteria: the model follows policy, keeps the tone calm, escalates exceptions, and does not offer refunds, discounts, timelines, or commitments beyond the supplied rules. Customer-facing AI needs a tighter leash than internal brainstorming.

Eval example: test one simple ticket, one frustrated customer, and one policy exception. The best route may split the job: one model classifies the ticket, another drafts the reply, and a human approves anything involving money, account access, safety, legal risk, or sensitive data.

A support workflow before and after routing

Imagine a team reviewing support tickets every day. The weak approach is to send every ticket to the strongest-looking model and ask it to answer customers. That feels efficient until the model apologizes for the wrong issue, offers an unsupported exception, or handles a sensitive account problem without escalation.

The routed approach splits the workflow:

1. Trigger: a new support ticket arrives.
2. Input filter: remove unnecessary personal data before AI processing where possible.
3. Classification route: a lower-cost model labels the ticket type and urgency if it passed the classification eval.
4. Policy lookup: the workflow supplies only the relevant policy notes or approved response rules.
5. Drafting route: a stronger model drafts the reply for complex or emotional tickets.
6. Human gate: a support lead approves messages involving refunds, account access, complaints, exceptions, or sensitive data.
7. Fallback: if the preferred model is unavailable or slow, the ticket goes to human handling or a pre-approved backup model, not to an untested replacement.

The output is not an AI agent pretending to run support. The output is a controlled support assist workflow with known routes, known review points, and known failure boundaries. This is the difference between experimentation and operations. For the wider operating layer around this kind of setup, see Business Systems & Operations.

The objection: why not just use the strongest model?

The objection is reasonable. If one model appears strongest, why complicate the stack?

Because strength is not one thing. A model that is strong but unavailable is not your operating answer. A model that is accurate but too costly for bulk low-risk work is not your default answer. A model that is fast but weak on evidence-heavy tasks is not your research answer.

The correction is to stop asking which model is best and start asking which model is approved for this route. That language changes the decision. It forces the team to define the task, the risk, the eval, the fallback, and the review owner.

This also protects you from benchmark theater. Benchmarks can be useful signals, but they are not your customer inbox, your codebase, your product facts, your brand constraints, or your internal permissions. Your workflow is the real benchmark.

Model routing rules you can adopt this week

Start small. Do not build a complex router before you have a simple policy. Use these rules as a first operating layer:

• One workflow, one route card. Every recurring AI workflow should have a written model route, fallback, and review point.
• No high-risk final outputs without human approval. This includes sensitive customer messages, production code, legal-style claims, financial claims, medical claims, and anything involving confidential data.
• Lower-cost models earn low-risk work first. Use them for classification, extraction, summarization drafts, formatting, and internal prep only after they pass your eval.
• Premium models must justify the spend. Use stronger or more expensive models where the task risk, reasoning demand, or review burden makes them worth it.
• Availability is a gate. If access is limited or unreliable, the model can be a specialist, not the only path for a critical workflow.
• Data minimization is default. Send the least sensitive input that can complete the task. Check company policy before using private documents, customer records, inbox data, CRM exports, internal code, or confidential files.
• Re-eval on change. Re-run the 30-minute eval when a model changes, access changes, pricing pressure changes, prompt patterns change, or the workflow becomes customer-facing.

If your team is still early in AI adoption, this is where AI in Practice matters: useful AI is not a pile of prompts. It is a set of operating rules that make the output safe enough to use.

FAQ

How many models should a small team test?

Start with two or three candidate models per workflow. More than that creates comparison noise. Your goal is a routing decision, not a research project.

Should model routing be automated?

Not at first. Write the route manually, run it with human review, then automate only the parts that are stable, low-risk, and easy to evaluate.

What if the cheaper model is almost as good?

Use it where the failure cost is low and the review path is clear. Do not move sensitive, customer-facing, or production-facing work to a cheaper model just because a small eval looked acceptable.

The next step is simple: choose one recurring task, collect three sanitized examples, run the 30-minute eval, and write one routing rule. If the rule cannot name the task, model, fallback, reviewer, and failure boundary, the workflow is not ready yet.

Where does your business actually stand?

Before you bolt on another tool, it is worth knowing whether your business runs on systems or on you. I put together a free 2-minute assessment that gives you a straight read on exactly that, and the first thing to fix. Take the free assessment.