Area 08 · Backend · Concept 13 of 20

Background Jobs

Run slow tasks (embeddings, reports, emails) in workers, outside the request.

11–20 · Production levelPhase 0 + 5 · Weeks 1–2, 18–20≈ 1 hour

The idea

Any task that takes more than a second or two, or that can fail and retry, belongs in a background job: embedding a document, generating a weekly report, sending emails, re-indexing, running nightly evals. The request enqueues the job and returns; a worker runs it.

Job frameworks (Celery, RQ, Dramatiq, arq, or cloud tasks) add retries with backoff, scheduling, timeouts, result storage and monitoring. Persist job status in your database so the UI can show progress and errors.

Scheduled jobs (cron-like) handle periodic work like re-syncing permissions or refreshing indexes.

Why it matters

It keeps your API fast and makes slow AI work reliable and observable.

Words to know

  • JobA unit of background work with arguments and status.
  • SchedulerStarts jobs on a timetable.

Try it without code

Sort your tasks

  1. List everything your app does per request.
  2. Move anything over 2 seconds to a "background" column.

Use it in code

Celery task with retries, plus a periodic schedule.

python
# pip install celery redis ; run: celery -A tasks worker -B
from celery import Celery

app = Celery("tasks", broker="redis://localhost:6379/0", backend="redis://localhost:6379/1")

@app.task(bind=True, autoretry_for=(TimeoutError,), retry_backoff=True, max_retries=5)
def embed_document(self, doc_id: str):
    # load doc, chunk, embed, upsert; update documents.status as you go
    return {"doc_id": doc_id, "status": "indexed"}

app.conf.beat_schedule = {
    "nightly-evals": {"task": "tasks.run_evals", "schedule": 24 * 3600},
}

@app.task
def run_evals():
    ...
PRACTICE · TICK IT AFTER THIS

Move document embedding and nightly evals into background jobs with status shown in your UI.

Common mistakes

  • Passing large objects as job arguments instead of IDs.
  • No visibility into failed jobs.

Go deeper