Recovering an automation after an outage
Recover interrupted automation by reconciling completed actions, resolving uncertain outcomes and replaying only work that still needs doing.
Suppose an order-processing workflow stops halfway through a morning batch. Some customers received confirmations. Some orders reached the warehouse. A few requests timed out after leaving your system, so the local log cannot tell you whether they succeeded.
Restarting the worker restores processing capacity. It does not resolve those uncertain orders. Before replaying the batch, establish which business actions happened and which remain outstanding.
Contain the backlog before restarting it
Pause the affected workflow’s dispatch while you investigate. Continue recording incoming work where possible, with a clear distinction between queued jobs and jobs already attempted. Otherwise, recovery traffic and new traffic can act on the same records at once.
Record the affected period, workflow version and external services involved. Preserve job identifiers, attempt logs and destination record IDs. A screenshot of an error count will not tell you which orders need attention later.
Assign someone to decide when processing can resume. For a small operation, that may be the same person doing the repair. The decision still needs an owner, especially when sales, support and operations each see a different part of the incident.
Reconcile against the destination
For each affected item, compare the intended action with evidence from the receiving system. A successful invoice creation should have an invoice ID or another durable reference. A local timeout is evidence that the response was missing, not proof that the invoice was never created.
Sort the work into confirmed complete, confirmed outstanding and unresolved. Keep unresolved items out of an indiscriminate replay. Query the destination using a stable business reference where it supports that lookup. If the receiving system offers neither a reliable lookup nor duplicate protection, an operator may need to review the record.
This is why idempotent automation needs an identity for the logical action that survives process restarts. An identifier generated for a new attempt cannot establish whether an earlier attempt already completed that action.
Check what a retry actually guarantees
Duplicate delivery is a documented possibility in Amazon SQS standard queues. A queue message appearing again does not, by itself, mean its previous business action failed. The consumer still needs to recognise work already completed.
External APIs have their own rules. Stripe’s API v1 idempotency documentation says a repeated key can return the saved response, including a server error, and keys can be removed once they are at least 24 hours old. Reusing a key after it has been pruned can create a new request. A recovery procedure must check the relevant API’s retention and retry behaviour rather than assuming duplicate protection lasts indefinitely.
Keep the original action identity and request parameters. Do not create a fresh key merely to make an ambiguous request go through. If the original action might already have happened, resolve that uncertainty first.
Replay a small, reviewed set
Start with a few confirmed outstanding items. Check the receiving system after those complete, then increase the replay rate within its capacity limits. Preserve dependencies: an order must exist before an operation that requires its destination ID can succeed.
Some pending work may no longer be appropriate. A customer might have cancelled an order during the outage, or a reminder might now be out of date. Recheck the business conditions before taking an external action. Sending yesterday’s queued message today is still a new action with a current consequence.
For workflows with several stages, resume from recorded progress where that progress is trustworthy. The checkpoint design in durable pipelines helps avoid repeating completed stages. Keep failed replay items and their reasons in a review queue rather than repeatedly sending the same broken input.
Close the incident with a count you can explain
Account for the affected items. Each should have a confirmed outcome: completed before recovery, completed during recovery, intentionally skipped or assigned for further investigation. A drained queue is insufficient evidence if messages disappeared without completing their business actions.
Record the remaining exceptions, their owners and the evidence used to resolve uncertain outcomes. Keep that record separate from routine worker health checks.
Then test the failure you just encountered. Interrupt a job after the destination accepts an action but before your worker records success. Restart it and inspect the destination for duplicates. The recovery procedure is ready when the team can follow it using the records the system actually retains.
