Design for the Rate Limit or Inherit the Outage

A queue that had cleared every morning stopped clearing. The workflow still showed as running. The connection to the external service still tested fine. And the operations lead who went looking for the failure found a log full of ordinary activity, and one gap where a batch of items should have been.

Traced backwards, the sequence was unremarkable. Each item in the queue triggered a call to an external AI service. Under a heavier morning load, the service began rejecting some of those calls, because the workflow had reached the capacity available to it. The workflow caught each rejection, recorded it as handled, and moved on to the next item. The items whose calls had been rejected left the queue along with the ones that succeeded.

That work had been accepted from a customer, removed from the queue, and left uncompleted. The limit prevented the intended call; the workflow’s handling determined what happened to the work afterwards. This is the outage — and it arrived while every component behaved exactly as it had been built to behave.

What the workflow reached

An external AI or API dependency has finite capacity available to your workflow. A rate limit is the service enforcing that boundary: the point past which it stops accepting your calls for some period. These limits take different forms. Some are documented and tier-dependent, some move with demand, and some are simply enforced at the moment you meet them.

Resilience here begins with a modest change of assumption — treat immediate capacity as finite, rather than assuming every intended call will be accepted at the moment you send it.

The mistake is not encountering the limit. It is reaching a known class of capacity constraint without having decided what happens to the work next.

The error handler stopped one decision short

The workflow in that incident did have error handling. It caught the rejection and wrote it down. The missing piece was a decision about the item whose call had just been rejected.

An error can tell you why the call did not complete. It does not decide what your workflow should do with the unfinished work.

These are two separate pieces of design, and the second is the one that tends to go missing — because catching the error feels like handling it. A caught exception produces a tidy log line and a workflow that appears to be coping. The item it was holding is a separate question, and one that has to be answered deliberately.

Four dispositions for the unfinished work

When capacity is unavailable, the work in hand has to go somewhere. There are four places it can go, and each fits a different situation:

• Retry — attempt the same work again after capacity may have become available
• Queue — preserve the work for automatic processing when capacity becomes available
• Defer — preserve it for a later scheduled or deliberately resumed attempt
• Degrade — continue with a deliberately reduced but still valid outcome that avoids the blocked call

Retry is one response to a capacity limit, not the default answer to every rate-limit event. It is the familiar one, and the reason it disappoints is mechanical: retrying immediately does not create capacity. Without a deliberate retry condition — a wait, a bound on attempts, and a named destination for work that still fails after them — the workflow may simply hit the same boundary again while the original work remains unresolved. Where the completion status of a call is genuinely uncertain, a retry also has to avoid repeating a side effect that already happened.

Queue and defer are close enough to be worth separating. Queue when the work should resume automatically as capacity becomes available. Defer when the workflow deliberately moves that work to a later processing point — a schedule, a trigger, or an action somebody resumes.

Degrade is the one worth defining carefully, because it sits close to a different decision entirely. To degrade is to skip or reduce a capacity-dependent step while still producing an explicitly valid reduced outcome: an optional enrichment step is unavailable, so the record is created without the enrichment and marked as lacking it. Degrading does not mean switching providers or models to preserve equivalent service. That is a continuity decision — a separate question from what to do with the work when the capacity available to you runs out.

A confirmation the customer is waiting on and an overnight enrichment pass belong in different classes, and they earn different answers.

Whatever the disposition, unresolved work stays visible

All four answer the same question: what happens to the work. They leave a second question open, and it applies to every one of them equally — can anybody tell that this work is still unresolved?

Whatever disposition you choose, unresolved work must remain visible until it is completed, deliberately cancelled, or otherwise resolved. A queued item with no surface showing that it is waiting looks exactly like a completed one. A deferred item that only exists inside a scheduler will be rediscovered by the customer who was waiting for it.

A graceful failure preserves the work state and makes the unresolved action visible. Quietly dropping work is not resilience — it is an outage with better manners. Any response that loses the work, leaves its status unknowable, or hides an unresolved action fails that test, however calmly the workflow appears to continue.

The Capacity Envelope

The envelope is a short description of one workflow at its capacity boundary. It is not a quota sheet: it records what your workflow does when capacity blocks the intended call, in four fields, in this order.

The capacity signal

What tells this workflow that the intended external call cannot proceed because capacity is unavailable? In the morning-queue incident, that signal was present from the first rejected call. It was caught, it was logged, and it was treated as the end of the matter. A signal that reaches only the log has been detected but not connected to anything.

The disposition

For this class of work: retry, queue, defer, or degrade? Each class of work gets its own answer, and writing them down separately is what stops one reflex being applied to everything the workflow touches.

The preserved state

What has to survive so the work can continue correctly later? Usually the item itself, its position in the sequence, and whatever the workflow had already done to it. In the incident above, the item left the queue and took with it any record that it had been accepted — which is why recovery began with customer complaints.

The unresolved-work surface

Where does unfinished work remain visible until it is resolved? A list somebody opens, a count on an operational view, a notification with a name attached to it — the form matters less than the fact that unresolved work has somewhere to sit where a person will see it. If the honest answer to this field is that unresolved work would sit in the logs, the surface has not been chosen yet.

Fill it in for one workflow

Take one workflow that depends on an external AI or API call — the one whose failure would reach a customer first. Write the four fields for it: the signal, the disposition, the preserved state, the surface.

Leave one of the fields undefined, and the capacity limit will expose whatever behaviour the workflow already has there. The purpose of the envelope is to make that behaviour deliberate before the limit is reached.

Before you bolt on another tool, it is worth knowing whether your business runs on systems or on you. I put together a free 2-minute assessment that gives you a straight read on exactly that, and the first thing to fix. Take the free assessment.