A queue that had cleared every morning stopped clearing. The workflow still showed as running. The connection to the external service still tested fine. And the operations lead who went looking for the failure found a log full of ordinary activity, and one gap where a batch of items should have been.<\/p>\n
Traced backwards, the sequence was unremarkable. Each item in the queue triggered a call to an external AI service. Under a heavier morning load, the service began rejecting some of those calls, because the workflow had reached the capacity available to it. The workflow caught each rejection, recorded it as handled, and moved on to the next item. The items whose calls had been rejected left the queue along with the ones that succeeded.<\/p>\n
That work had been accepted from a customer, removed from the queue, and left uncompleted. The limit prevented the intended call; the workflow’s handling determined what happened to the work afterwards. That lost work was the outage, and it happened while every component behaved exactly as it had been built to behave.<\/p>\n
An external AI or API dependency has finite capacity available to your workflow. A rate limit is the service enforcing that boundary: the point past which it stops accepting your calls for some period. These limits take different forms. Some are documented and depend on your tier. Some move with demand, and some are simply enforced at the moment you meet them.<\/p>\n
Resilience here begins with a modest change of assumption: treat immediate capacity as finite, and do not count on every intended call being accepted at the moment you send it.<\/p>\n
The mistake is not encountering the limit. It is reaching a known class of capacity constraint without having decided what happens to the work next.<\/strong><\/p>\n The workflow in that incident did have error handling. It caught the rejection and wrote it down. The missing piece was a decision about the item whose call had just been rejected.<\/p>\n An error can tell you why the call did not complete; what your workflow should do with the unfinished work is a decision you still have to make.<\/strong><\/p>\n These are two separate pieces of design, and the second is the one that tends to go missing, because catching the error feels like handling it. A caught exception produces a tidy log line and a workflow that appears to be coping. The item it was holding is a separate question, and one that has to be answered deliberately.<\/p>\n When capacity is unavailable, the work in hand has to go somewhere. There are four places it can go, and each fits a different situation:<\/p>\n Retry is one response to a capacity limit, not the default answer to every rate-limit event. It is the familiar one, and the reason it disappoints is mechanical: retrying immediately does not create capacity. Without a deliberate retry condition (a wait, a bound on attempts, and a named destination for work that still fails after them), the workflow may simply hit the same boundary again while the original work remains unresolved. Where the completion status of a call is genuinely uncertain, a retry also has to avoid repeating a side effect that already happened.<\/p>\n Queue and defer are easy to confuse. Queue when the work should resume automatically as capacity becomes available. Defer when the workflow deliberately moves that work to a later processing point: a schedule, a trigger, or an action somebody resumes.<\/p>\n Degrade needs a careful definition because it is easily confused with a different decision. To degrade is to skip or reduce a capacity-dependent step while still producing an explicitly valid reduced outcome: an optional enrichment step is unavailable, so the record is created without the enrichment and marked as lacking it. Degrading does not mean switching providers or models to preserve equivalent service. That is a continuity decision, a separate question from what to do with the work when the capacity available to you runs out.<\/p>\n A confirmation the customer is waiting on and an overnight enrichment pass belong in different classes, and they earn different answers.<\/p>\n All four answer the same question: what happens to the work. They leave a second question open, and it applies to every one of them equally: can anybody tell that this work is still unresolved?<\/p>\n Whatever disposition you choose, unresolved work must remain visible until it is completed, deliberately cancelled, or otherwise resolved. A queued item with no surface showing that it is waiting looks exactly like a completed one. A deferred item that only exists inside a scheduler will be rediscovered by the customer who was waiting for it.<\/p>\n A graceful failure preserves the work state and makes the unresolved action visible. Dropping work silently is still an outage.<\/strong> Any response that loses the work, leaves its status unknowable, or hides an unresolved action fails that test, however calmly the workflow appears to continue.<\/p>\n The envelope is a short description of one workflow at its capacity boundary. It leaves quotas aside and records what your workflow does when capacity blocks the intended call, in four fields, in this order.<\/p>\n What tells this workflow that the intended external call cannot proceed because capacity is unavailable? In the morning-queue incident, that signal was present from the first rejected call. It was caught, it was logged, and it was treated as the end of the matter. A signal that reaches only the log has been detected but not connected to anything.<\/p>\n For this class of work: retry, queue, defer, or degrade? Each class of work gets its own answer, and writing them down separately is what stops one reflex being applied to everything the workflow touches.<\/p>\n What has to survive so the work can continue correctly later? Usually the item itself, its position in the sequence, and whatever the workflow had already done to it. In the incident above, the item left the queue and took with it any record that it had been accepted, which is why recovery began with customer complaints.<\/p>\n Where does unfinished work remain visible until it is resolved? The form matters less than the fact that unresolved work has somewhere to sit where a person will see it: a list somebody opens, a count on an operational view, or a notification with a name attached to it. If the honest answer to this field is that unresolved work would sit in the logs, the surface has not been chosen yet.<\/p>\n Take one workflow that depends on an external AI or API call, the one whose failure would reach a customer first. Write the four fields for it: the signal, the disposition, the preserved state, the surface.<\/p>\n Leave one of the fields undefined, and the capacity limit will expose whatever behaviour the workflow already has there. The purpose of the envelope is to make that behaviour deliberate before the limit is reached.<\/p>\n Before you bolt on another tool, it is worth knowing whether your business runs on systems or on you. I put together a free 2-minute assessment that gives you a straight read on exactly that, and the first thing to fix. Take the free assessment<\/a>.<\/p>\nThe error handler stopped one decision short<\/h2>\n
Four dispositions for the unfinished work<\/h2>\n
\n
Whatever the disposition, unresolved work stays visible<\/h2>\n
The Capacity Envelope<\/h2>\n
The capacity signal<\/h3>\n
The disposition<\/h3>\n
The preserved state<\/h3>\n
The unresolved-work surface<\/h3>\n
Fill it in for one workflow<\/h2>\n