Area 08 · Backend · Concept 12 of 20

Message Queues

Pass work between services reliably (RabbitMQ, SQS, Kafka).

11–20 · Production levelPhase 0 + 5 · Weeks 1–2, 18–20≈ 1 hour

The idea

A message queue lets one service hand work to another without both being available at the same moment. The producer publishes a message; the queue stores it durably; consumers take it, process it and acknowledge it. Unacknowledged messages are redelivered, so work is not lost when a worker crashes.

Different tools fit different needs: SQS and Cloud Tasks for simple managed work queues, RabbitMQ for flexible routing, Kafka for high-volume event streams that many consumers read and replay. For small apps, Redis-based queues are often enough.

Design consumers to be idempotent, because at-least-once delivery means a message can arrive twice.

Why it matters

Queues decouple slow AI work from user requests and absorb traffic bursts without dropping work.

Picture it

Words to know

  • Producer / consumerWho sends messages / who processes them.
  • At-least-onceMessages may be delivered more than once, never zero times.
  • AckAcknowledgement that a message was processed.

Try it without code

Find your events

  1. List events in your app (document uploaded, chat finished, feedback given).
  2. For each, list who should react to it.

Use it in code

Publish and consume with AWS SQS via boto3.

python
# pip install boto3
import json, boto3
sqs = boto3.client("sqs", region_name="eu-west-1")
QUEUE = sqs.get_queue_url(QueueName="doc-ingest")["QueueUrl"]

def publish(doc_id):
    sqs.send_message(QueueUrl=QUEUE, MessageBody=json.dumps({"doc_id": doc_id}))

def consume_forever(handle):
    while True:
        resp = sqs.receive_message(QueueUrl=QUEUE, MaxNumberOfMessages=5, WaitTimeSeconds=20)
        for m in resp.get("Messages", []):
            handle(json.loads(m["Body"]))                     # must be idempotent
            sqs.delete_message(QueueUrl=QUEUE, ReceiptHandle=m["ReceiptHandle"])   # ack
PRACTICE · TICK IT AFTER THIS

Publish a document.uploaded event from your API and consume it in a worker that runs ingestion.

Common mistakes

  • Acknowledging before processing finishes.
  • Non-idempotent consumers causing duplicate work.

Go deeper