Background jobs and queues that survive failure

Almost every application we build ends up with a queue. Sending email, generating PDFs, syncing orders to an ERP, resizing images, calling an AI model: anything slow or unreliable belongs outside the web request. Moving work to a queue makes pages faster, but it also moves failures out of sight. A failed web request shows an error to someone. A failed job fails quietly, perhaps retries a few times, and then disappears into a log nobody reads. Weeks later, finance asks why forty invoices never reached the accounting system. These are the patterns we use so background work fails loudly, recovers safely and never silently vanishes.

Assume every job will run twice

Most queue systems guarantee at-least-once delivery. A worker can finish a job and crash before acknowledging it, and the job will be delivered again. Deploys, network blips and timeouts cause the same effect. If a job charges a card, sends an email or creates a record in another system, running it twice is a real problem.

The fix is to make handlers idempotent: running them twice has the same effect as running them once. Common techniques:

• Check state before acting. If the invoice is already marked as synced, return early.
• Use idempotency keys with external APIs that support them, derived from the job's business identity rather than generated randomly.
• Use database unique constraints to make duplicate inserts fail harmlessly.
• Record completion in the same transaction as the side effect, where both are in your database.

Idempotency is the foundation for everything else. Once handlers are safe to repeat, retries stop being scary.

Retries with backoff and limits

Transient failures are the norm when calling third-party services. A good retry policy distinguishes between failures worth retrying, such as timeouts, rate limits and server errors, and failures that will never succeed, such as validation errors or missing records. Retrying the second kind wastes capacity and delays the alert.

For retryable failures, use exponential backoff with jitter, so a struggling downstream service is not hammered by every worker at the same moment:

class SyncInvoiceToErp implements ShouldQueue
{
    public $tries = 8;
    public $timeout = 60;

    public function backoff(): array
    {
        return [10, 30, 90, 300, 900, 1800, 3600];
    }

    public function handle(ErpClient $erp): void
    {
        if ($this->invoice->erp_synced_at) {
            return;
        }
        $erp->createInvoice($this->invoice, idempotencyKey: "inv-{$this->invoice->id}");
        $this->invoice->update(['erp_synced_at' => now()]);
    }
}

Eight attempts spread over a couple of hours covers most outages of a partner API without human intervention.

Dead-letter queues and visible failure

When a job exhausts its retries, it must go somewhere a human will see it. A dead-letter queue or failed jobs table holds the payload, the error and the attempt history. What matters is what happens next:

1. An alert fires when the number of failed jobs rises, grouped by job type so that one broken integration produces one alert, not four hundred.
2. Someone owns each job type and is responsible for triage.
3. Failed jobs can be retried from an admin screen once the cause is fixed, individually or in bulk.
4. Failed jobs older than a set period are reviewed, not silently pruned.

A job that fails silently is worse than one that never ran, because everyone believes it did.

The dual-write problem and the outbox

A subtle bug appears when code saves a record and dispatches a job in the same request. If the database transaction commits but the dispatch fails, the job never runs. If the dispatch succeeds but the transaction rolls back, the job runs against data that does not exist. Both happen more often than you would expect under load.

The transactional outbox pattern fixes this. Instead of dispatching directly, the code writes a row to an outbox table in the same transaction as the business data. A separate process reads the outbox and dispatches jobs, marking rows as sent. Either both the data and the intent to process it exist, or neither does. Many frameworks offer a simpler version: dispatching jobs only after the surrounding transaction commits. Turn that on by default.

Timeouts, long jobs and fairness

A few operational details prevent most queue incidents we are called in to fix:

• Timeouts on everything. Every job has a maximum runtime, and every outbound HTTP call inside it has its own shorter timeout. A job hanging on a slow API can hold a worker indefinitely.
• Split long jobs. A job that processes 50,000 rows should become a coordinator that dispatches batches of a few hundred, each retryable on its own.
• Separate queues by priority. Password reset emails should not wait behind a nightly export. Run dedicated workers for urgent queues.
• Graceful shutdown. Workers should finish their current job on deploy rather than being killed mid-task.

Monitoring the queue itself

Beyond failed jobs, three metrics tell you whether the queue is healthy: queue depth, the age of the oldest waiting job, and throughput per job type. Oldest job age is the most useful single signal. A queue with 10,000 jobs processing quickly is fine; a queue with 12 jobs where the oldest has waited 40 minutes means workers are stuck or starved. We alert on age thresholds per queue, set according to how quickly that work needs to happen.

For one client, adding these patterns to an existing order-sync pipeline took about three weeks. In the quarter before, the operations team had reconciled missing orders by hand roughly twice a week. In the quarter after, there were two failed-job alerts, both caused by the ERP being down for maintenance, and both cleared automatically once it came back.

Building reliable background work

Queues are part of every custom web application we build, and these patterns are standard in our Laravel and Django projects alike. For existing systems where jobs go missing, a focused review usually finds the gaps within a few days.

Stop losing work in the background

If your team is reconciling data by hand because background jobs cannot be trusted, we can help. Describe your setup and we will send a fixed-price proposal within 24 hours. Talk to us.