Idempotent webhooks: building integrations that tolerate retries

Most modern integrations run on webhooks. The payment provider tells you a charge succeeded, the shipping carrier tells you a parcel moved, the CRM tells you a deal closed, and your application reacts. On a good day this is elegant. On a normal day, webhooks arrive twice, arrive in the wrong order, arrive hours late after the sender's outage, or do not arrive at all. Integrations that assume each event is delivered exactly once, promptly and in order will eventually double-ship an order or leave a paid invoice marked unpaid. The fix is a handful of design rules we apply to every webhook handler and automation we build.

Rule one: verify, store, acknowledge

A webhook endpoint should do as little as possible before returning a response. Senders typically wait only a few seconds and retry on timeouts, so slow handlers cause duplicate deliveries. Our handlers follow the same three steps:

1. Verify the signature using the shared secret, and reject anything that fails or whose timestamp is too old, to prevent forged and replayed requests.
2. Store the raw payload, headers and event identifier in an inbox table.
3. Return a success response immediately, and process the event from a queue.

app.post("/webhooks/payments", async (req, res) => {
  if (!verifySignature(req.rawBody, req.headers["x-signature"], secret)) {
    return res.status(400).end();
  }
  const event = JSON.parse(req.rawBody);
  await db.webhookInbox.insertIgnoreDuplicate({
    provider: "payments",
    eventId: event.id,
    payload: event,
  });
  await queue.add("process-webhook", { provider: "payments", eventId: event.id });
  res.status(200).end();
});

Storing the raw payload first has a second benefit: if processing fails because of a bug, the event is not lost. Fix the bug, replay from the inbox, and nothing needs to be requested again from the sender.

Rule two: deduplicate on the event identity

Nearly every provider includes a unique event identifier. A unique constraint on provider and event identifier in the inbox table turns duplicate deliveries into harmless no-ops at the database level. That handles exact duplicates.

It does not handle semantic duplicates, where two different events describe the same business fact, such as a payment succeeded event and a charge updated event that both indicate the invoice is paid. For those, the processing logic itself must be idempotent: check whether the invoice is already paid before marking it, and use idempotency keys when your handler calls other APIs in turn. Deduplication at the edge reduces work; idempotency in the logic guarantees correctness.

Assume every event will arrive twice and some will never arrive. Design for that, and the ordinary days take care of themselves.

Rule three: do not trust the order

Events can arrive out of order, especially after retries. A subscription cancelled event can land before the subscription updated event that preceded it. If the handler blindly applies each event's state, the older update overwrites the cancellation and the customer keeps being billed.

There are two reliable approaches:

• Compare versions or timestamps. Store the last applied event's timestamp or version on the record, and ignore any incoming event older than it.
• Treat the webhook as a signal, not a source of truth. On receiving any event about an object, fetch the current state of that object from the provider's API and apply it. The event only tells you something changed; the API tells you what is true now.

We prefer the second approach wherever the provider's API allows it and rate limits permit. It is slightly slower and uses more API calls, but it removes an entire class of ordering bugs.

Rule four: reconcile on a schedule

Even with perfect handlers, some events never arrive. Senders have outages, endpoints get misconfigured during deploys, and signing secrets get rotated on one side but not the other. The only protection against missing events is a periodic reconciliation job that compares your records against the provider's.

For payments, that might mean fetching every charge from the last 48 hours each hour and confirming that each one matches an order in the correct state. For a CRM sync, a nightly comparison of records updated in the last day. Differences are corrected automatically where the rules are clear and flagged for a human where they are not. For one subscription business, the first reconciliation run found 37 customers whose cancellations had never been processed, all traced back to a single afternoon when the webhook secret had been rotated incorrectly.

Rule five: make it observable and replayable

An integration you cannot see into is one you cannot trust. We give every integration a small admin view showing recent events, their processing status, errors and retry counts, with the ability to replay a single event or a time range. We alert when the failure rate for an event type rises, and, just as important, when events stop arriving entirely during hours when they normally flow. Silence is often the first sign of a broken integration.

These patterns apply equally to the automations we build that react to webhooks, whether they trigger an AI agent, update a spreadsheet or post to a team channel. Our workflow automation projects use the same inbox and reconciliation design, and our API development work applies it in reverse when clients need to send reliable webhooks to their own customers. For payment flows specifically, payment gateway integration is where getting this right matters most.

Integrations that hold up on bad days

If your integrations work most of the time and need manual fixing the rest, we can make them dependable. Tell us which systems are involved and we will quote a fixed price within 24 hours. Send us the details.