The idea
Some AI work takes minutes: ingesting a 300-page PDF, running an agent, generating a report. Holding a web request open that long leads to timeouts and lost work. Instead, the API puts a job on a queue and returns a job ID immediately; workers process jobs; the client polls or gets a webhook or push event when done.
Queues (Redis with RQ, Celery, SQS, Cloud Tasks) give you retries, concurrency control (respecting provider rate limits), and isolation: a burst of uploads does not slow down chat.
Store job status and results in your database, make jobs idempotent, and set timeouts and dead-letter handling for jobs that keep failing.
It is the backbone of any AI feature that takes longer than a few seconds.
Picture it
Words to know
- WorkerA process that pulls jobs from a queue and runs them.
- Dead-letter queueWhere jobs go after too many failures.
Try it without code
Find your slow paths
- List requests in your app that take over 5 seconds.
- For each, write what the user sees while it runs.
Use it in code
Enqueue ingestion with RQ and poll status.
# pip install rq redis fastapi ; run a worker with: rq worker ingest
from fastapi import FastAPI
from redis import Redis
from rq import Queue
app = FastAPI()
q = Queue("ingest", connection=Redis())
def ingest(doc_id: str): # lives in a module the worker can import
... # parse, chunk, embed, store
return {"doc_id": doc_id, "chunks": 120}
@app.post("/documents/{doc_id}/ingest", status_code=202)
def start(doc_id: str):
job = q.enqueue(ingest, doc_id, job_timeout=900, retry=None)
return {"job_id": job.id}
@app.get("/jobs/{job_id}")
def status(job_id: str):
job = q.fetch_job(job_id)
return {"status": job.get_status(), "result": job.result}Move PDF ingestion in your RAG app into a worker with a progress percentage the UI can poll.
Common mistakes
- Doing long work inside the request handler.
- Unlimited worker concurrency that trips provider rate limits.