The idea
Any task that takes more than a second or two, or that can fail and retry, belongs in a background job: embedding a document, generating a weekly report, sending emails, re-indexing, running nightly evals. The request enqueues the job and returns; a worker runs it.
Job frameworks (Celery, RQ, Dramatiq, arq, or cloud tasks) add retries with backoff, scheduling, timeouts, result storage and monitoring. Persist job status in your database so the UI can show progress and errors.
Scheduled jobs (cron-like) handle periodic work like re-syncing permissions or refreshing indexes.
It keeps your API fast and makes slow AI work reliable and observable.
Words to know
- JobA unit of background work with arguments and status.
- SchedulerStarts jobs on a timetable.
Try it without code
Sort your tasks
- List everything your app does per request.
- Move anything over 2 seconds to a "background" column.
Use it in code
Celery task with retries, plus a periodic schedule.
# pip install celery redis ; run: celery -A tasks worker -B
from celery import Celery
app = Celery("tasks", broker="redis://localhost:6379/0", backend="redis://localhost:6379/1")
@app.task(bind=True, autoretry_for=(TimeoutError,), retry_backoff=True, max_retries=5)
def embed_document(self, doc_id: str):
# load doc, chunk, embed, upsert; update documents.status as you go
return {"doc_id": doc_id, "status": "indexed"}
app.conf.beat_schedule = {
"nightly-evals": {"task": "tasks.run_evals", "schedule": 24 * 3600},
}
@app.task
def run_evals():
...Move document embedding and nightly evals into background jobs with status shown in your UI.
Common mistakes
- Passing large objects as job arguments instead of IDs.
- No visibility into failed jobs.