Idempotent Automation: Running the Same Job Twice Without Doing the Work Twice

Sooner or later, every automation runs the same job twice. A network call times out and gets retried, but the first attempt had actually succeeded. A webhook provider delivers the same event again because it did not see your acknowledgement. A process restarts halfway through a batch and starts the batch over.

None of this is exotic. It is the normal behaviour of systems that talk over networks. The only question is whether running twice produces one result or two, and that is a property you have to design in, because it does not happen on its own.

What idempotent means in practice

An operation is idempotent if doing it twice leaves the system in the same state as doing it once. Setting a field to a value is idempotent. Adding one to a counter is not. “Create an invoice for order 4417” is not idempotent by default; run it twice and there are two invoices.

The distinction matters most for the actions with side effects outside your system: sending an email, charging a card, creating a record in someone else’s API, posting a message. Those are exactly the ones that cannot be quietly cleaned up afterwards, and exactly the ones that retry logic touches most.

Retries make the problem urgent

Retries are the reason this matters. Adding retry logic is the correct response to transient failures, and it is also what turns a latent duplicate risk into a real one.

The dangerous case is the ambiguous failure. A request to create a record times out. Did it fail before the record was created, or after, with only the response lost? From the caller’s side the two are indistinguishable. Retrying is wrong in the second case if the operation is not idempotent, and giving up is wrong in the first. Without a way to tell the difference, every choice is a gamble.

Idempotency removes the gamble. If the operation is safe to repeat, the ambiguous failure stops being a dilemma: retry, and either the first attempt is recognised or the second one does the work.

Give every unit of work a stable key

The core technique is an idempotency key: a value that identifies one logical piece of work and stays the same across every attempt at it.

The key has to come from the work, not the attempt. A random identifier generated on each retry defeats the purpose, because each attempt looks new. A key derived from the thing being processed, such as the order identifier plus the action, is the same every time, so a second attempt can be recognised as a repeat.

Where the receiving system supports it, pass the key along. Many payment and messaging APIs accept one and will return the original result for a repeated key instead of acting again. Where it does not, the check has to live on your side: record the key and its outcome before acting, and look it up before doing the work.

Check and act have to be one step

A common attempt at this is to read first and write after: look for an existing record, and create one if none is found. It works in testing and fails under concurrency, because two runs can both read “none found” before either writes.

The check and the claim must be a single atomic operation. In practice that means a database uniqueness constraint on the key, so the second writer is rejected by the database rather than by application logic that can race. Insert the key first, and only the run that wins the insert performs the action. The constraint is doing the real work; the application code just reacts to it.

Inbound events deserve the same treatment

Retries from your own code are half of it. The other half is events arriving from outside. Webhook providers generally guarantee delivery at least once, which is a polite way of saying duplicates will happen, and often out of order.

The handler should do three things. Acknowledge quickly, so the provider stops resending. Record the event’s unique identifier and discard any it has already processed. And make the processing itself safe to repeat, because the record-and-discard check narrows the window for duplicates but does not close it entirely.

Order matters too. If a “paid” event can arrive before the matching “created” event, processing them strictly as received produces nonsense. Handlers that apply state transitions based on the current record, rather than assuming a sequence, tolerate this far better.

Partial completion is the hard case

Single steps are the easy version. The difficult one is a multi-step job that fails in the middle, where steps one and two succeeded and step three did not.

A restart-from-the-top policy re-runs the first two. If they are idempotent, that is wasteful but harmless. If they are not, it is a duplicate. The better design makes each stage a checkpoint, so completed work is not repeated at all, which is the approach in durable pipelines. Idempotency and checkpointing solve neighbouring problems: checkpoints avoid redoing work, and idempotency makes it safe when redoing happens anyway. A reliable system wants both.

The same discipline applies at larger scale. In an outreach state machine, the question “has this lead already received step three?” is an idempotency check, and getting it wrong means someone receives the same email twice.

How to test it

This is one area where testing has to be deliberate, because the failures only appear on the unhappy path. Run every important step twice and assert the end state matches a single run. Run it twice concurrently. Kill the process between the action and the record of the action, restart, and confirm the outcome is still single. Replay a captured webhook payload. If those pass, the system will survive the retries it is going to get.

When it is not worth the effort

An automation that only writes to a spreadsheet row, or whose duplicate is trivially visible and trivially deleted, does not need this machinery. The effort belongs on steps where a duplicate costs money, trust, or hours of cleanup. Most automations have a few such steps and many harmless ones, and the skill is telling them apart.

We build automation that is safe to retry. See how we approach workflow automation, or read about choosing between no-code and custom automation.