Background Jobs Primer

Resque Background-Job Primer

Resque is a Redis-backed library for creating background jobs, placing them on multiple queues, and processing them later. In Meridian we lean on Resque to absorb spiky workloads, defer slow third-party calls, and keep request latency below our 200ms p95 budget. This primer walks through the worker lifecycle, queue topology, and the failure-recovery patterns we ship in production.

1.Worker Lifecycle

Every Resque worker forks a child process per job. The parent stays resident, polling Redis via BLPOP across its registered queues. When work arrives the parent forks, the child executes theperform method, and exits. This forking model gives us bulletproof memory isolation at the cost of ~50ms of overhead per job.

2.Queue Topology

We run four named queues in priority order: critical, high, default, and low. Workers are launched with an explicit queue list, and the order matters: QUEUE=critical,high drains critical before touching high. Avoid the wildcard QUEUE=* in production because it sorts queue names alphabetically, which defeats priority routing entirely.

3.Failure Recovery

Failed jobs land in the failed queue with full backtraces. Pair Resque with resque-retry for exponential backoff and resque-scheduler for delayed enqueue. The minimal worker definition below shows the retry hooks we ship by default.

class SendDigestJob
  extend Resque::Plugins::Retry

  @queue = :high
  @retry_limit = 5
  @retry_delay = 30

  def self.perform(user_id, digest_type)
    user = User.find(user_id)
    DigestMailer.send(user, digest_type).deliver_now
  end

  def self.on_failure_log(exception, *args)
    Rails.logger.error "[resque] #{exception.message} args=#{args}"
  end
end