Recipe / Queues
BullMQ is a Redis-backed job queue for Node.js that gives Meridian workloads durable retries, scheduled jobs, and per-tenant concurrency. This primer walks through the producer/worker split, shows the connection shape, and explains why every Meridian background hop (embeddings, image generation, webhook fan-out) should land on a queue instead of a synchronous handler.
HTTP handlers must answer within seconds. Anything slower (model inference, batch embeddings, S3 uploads) belongs on a queue so the request can return immediately and the worker fleet can scale independently. BullMQ persists every job in Redis, so a crashed worker resumes mid-stream instead of losing customer data.
A producer calls queue.add with a job name plus a payload. A worker subscribes to the same queue name and receives jobs in the order Redis hands them over. Concurrency is set per worker process and can be tuned live from the Meridian admin console.
npm install bullmq ioredis
# Worker file
import { Queue, Worker } from "bullmq";
const connection = { host: "127.0.0.1", port: 6379 };
const emailQueue = new Queue("email", { connection });
await emailQueue.add("welcome", { to: "user@example.com" }, {
attempts: 5,
backoff: { type: "exponential", delay: 1000 },
removeOnComplete: 1000,
});
new Worker("email", async (job) => {
await sendEmail(job.data.to);
}, { connection, concurrency: 25 });Set attempts to at least 3 and use exponential backoff so transient upstream failures self-heal. Always design job handlers to be idempotent: use the job id as a dedupe key in your database so a retried job never double-charges a customer or double-sends an email.