Area 05 · AI System Design · Concept 15 of 20

Async Job Pipelines

Put long AI tasks in a queue and run them in workers, outside the web request.

11–20 · Production levelPhase 6 · Weeks 21–23≈ 1.5 hours

The idea

Some AI work takes minutes: ingesting a 300-page PDF, running an agent, generating a report. Holding a web request open that long leads to timeouts and lost work. Instead, the API puts a job on a queue and returns a job ID immediately; workers process jobs; the client polls or gets a webhook or push event when done.

Queues (Redis with RQ, Celery, SQS, Cloud Tasks) give you retries, concurrency control (respecting provider rate limits), and isolation: a burst of uploads does not slow down chat.

Store job status and results in your database, make jobs idempotent, and set timeouts and dead-letter handling for jobs that keep failing.

Why it matters

It is the backbone of any AI feature that takes longer than a few seconds.

Picture it

Words to know

  • WorkerA process that pulls jobs from a queue and runs them.
  • Dead-letter queueWhere jobs go after too many failures.

Try it without code

Find your slow paths

  1. List requests in your app that take over 5 seconds.
  2. For each, write what the user sees while it runs.

Use it in code

Enqueue ingestion with RQ and poll status.

python
# pip install rq redis fastapi ; run a worker with:  rq worker ingest
from fastapi import FastAPI
from redis import Redis
from rq import Queue

app = FastAPI()
q = Queue("ingest", connection=Redis())

def ingest(doc_id: str):            # lives in a module the worker can import
    ...                             # parse, chunk, embed, store
    return {"doc_id": doc_id, "chunks": 120}

@app.post("/documents/{doc_id}/ingest", status_code=202)
def start(doc_id: str):
    job = q.enqueue(ingest, doc_id, job_timeout=900, retry=None)
    return {"job_id": job.id}

@app.get("/jobs/{job_id}")
def status(job_id: str):
    job = q.fetch_job(job_id)
    return {"status": job.get_status(), "result": job.result}
PRACTICE · TICK IT AFTER THIS

Move PDF ingestion in your RAG app into a worker with a progress percentage the UI can poll.

Common mistakes

  • Doing long work inside the request handler.
  • Unlimited worker concurrency that trips provider rate limits.

Go deeper