← Docs

Recipe: Queue + worker design

A durable job queue backed by Upstash KV with at-least-once delivery, exponential backoff, and dead-letter routing.

Architecture

Jobs are pushed into a sorted set keyed by scheduled epoch. A single-threaded worker polls the next ready slice, claims items atomically via Lua, and executes handlers with a configurable concurrency cap. Completed jobs are removed; failures are retried with jittered backoff or moved to a dead-letter set after N attempts.

Key layout

  • queue:pending — sorted set (score = run_at)
  • queue:jobs:{jobId} — hash with payload, attempts, status
  • queue:dlq — dead-letter set
  • queue:inflight — set of claimed job IDs

Claim script

-- Lua (atomic claim)
local now = tonumber(ARGV[1])
local limit = tonumber(ARGV[2])
local jobs = redis.call('ZRANGEBYSCORE',
  'queue:pending', 0, now, 'LIMIT', 0, limit)
for _, id in ipairs(jobs) do
  redis.call('ZREM', 'queue:pending', id)
  redis.call('SADD', 'queue:inflight', id)
end
return jobs

Worker loop

The worker runs a tight loop: claim → execute → ack or retry. Concurrency is bounded by a semaphore. Backoff is min(2^n + jitter, 3600) seconds. A heartbeat goroutine periodically re-claims inflight jobs whose lease expired, preventing silent stalls.

Edge cases

  • Duplicate delivery is possible — handlers must be idempotent.
  • Lease expiry defaults to 30 s; tune per job weight.
  • DLQ entries are retained for 7 days then pruned.
← Back to docs