AWS SQS Primer
Amazon Simple Queue Service decouples services so they scale and fail independently. This primer covers queues, visibility timeouts, dead-letter queues, and the patterns that keep Meridian's ingestion pipeline reliable under burst load.
Queue types
Standard queues offer unlimited throughput with at-least-once delivery. FIFO queues preserve order and guarantee exactly-once processing, capped at 3000 messages per second. Choose standard when throughput matters more than ordering; choose FIFO when duplicates are unacceptable.
Visibility timeout
When a consumer polls a message, SQS hides it from other consumers for a configurable window. If the consumer deletes the message before the timeout expires, processing is complete. If the timeout lapses, the message reappears and another consumer retries it. Set the timeout slightly longer than your 99th-percentile processing latency.
Dead-letter queues
A DLQ catches messages that exceed a source queue's max receive count. Instead of looping forever, poisoned messages land in the DLQ for inspection. Wire CloudWatch alarms to the DLQ depth so the on-call engineer knows when the pipeline is swallowing bad payloads.
Long polling & batching
Enable long polling (WaitTimeSeconds=20) to reduce empty responses and lower costs. Batch sends and deletes in chunks of 10 to cut API calls by an order of magnitude. Combine both for a quiet, cost-efficient consumer loop that wakes only when work arrives.
Next step
Wire SQS into Meridian's ingestion workers with the SQS worker recipe.