The idea
A message queue lets one service hand work to another without both being available at the same moment. The producer publishes a message; the queue stores it durably; consumers take it, process it and acknowledge it. Unacknowledged messages are redelivered, so work is not lost when a worker crashes.
Different tools fit different needs: SQS and Cloud Tasks for simple managed work queues, RabbitMQ for flexible routing, Kafka for high-volume event streams that many consumers read and replay. For small apps, Redis-based queues are often enough.
Design consumers to be idempotent, because at-least-once delivery means a message can arrive twice.
Queues decouple slow AI work from user requests and absorb traffic bursts without dropping work.
Picture it
Words to know
- Producer / consumerWho sends messages / who processes them.
- At-least-onceMessages may be delivered more than once, never zero times.
- AckAcknowledgement that a message was processed.
Try it without code
Find your events
- List events in your app (document uploaded, chat finished, feedback given).
- For each, list who should react to it.
Use it in code
Publish and consume with AWS SQS via boto3.
# pip install boto3
import json, boto3
sqs = boto3.client("sqs", region_name="eu-west-1")
QUEUE = sqs.get_queue_url(QueueName="doc-ingest")["QueueUrl"]
def publish(doc_id):
sqs.send_message(QueueUrl=QUEUE, MessageBody=json.dumps({"doc_id": doc_id}))
def consume_forever(handle):
while True:
resp = sqs.receive_message(QueueUrl=QUEUE, MaxNumberOfMessages=5, WaitTimeSeconds=20)
for m in resp.get("Messages", []):
handle(json.loads(m["Body"])) # must be idempotent
sqs.delete_message(QueueUrl=QUEUE, ReceiptHandle=m["ReceiptHandle"]) # ackPublish a document.uploaded event from your API and consume it in a worker that runs ingestion.
Common mistakes
- Acknowledging before processing finishes.
- Non-idempotent consumers causing duplicate work.