Area 05 · AI System Design · Concept 14 of 20

Batch Inference

Send big offline jobs as a batch at a lower price when nobody is waiting for the answer.

11–20 · Production levelPhase 6 · Weeks 21–23≈ 45 min

The idea

Not every request needs an answer in seconds. Classifying a million tickets, embedding a document archive, generating eval judgements overnight: these can wait hours. Providers offer batch APIs that accept a file of requests, process them within a window (often up to 24 hours) and charge a large discount.

Design for it: write requests as JSONL with your own IDs, submit, poll for completion, download results and join them back by ID. Handle partial failures by resubmitting only the failed items.

Self-hosted, the same idea means running large offline jobs at maximum batch size when interactive traffic is low.

Why it matters

It can cut the cost of offline workloads roughly in half with almost no code change.

Words to know

  • custom_idYour identifier for each request, used to match results.

Try it without code

Find batch candidates

  1. List every model call in your system.
  2. Mark which ones a user waits for.
  3. The rest are batch candidates.

Use it in code

Submit a batch and collect results.

python
import time, anthropic
client = anthropic.Anthropic()
tickets = {"t1": "I was charged twice", "t2": "Export button broken"}

batch = client.messages.batches.create(requests=[
    {"custom_id": tid, "params": {"model": "claude-haiku-5-5", "max_tokens": 5,
     "messages": [{"role": "user", "content": f"Label BILLING, BUG or HOWTO: {text}"}]}}
    for tid, text in tickets.items()])

while client.messages.batches.retrieve(batch.id).processing_status != "ended":
    time.sleep(60)
for res in client.messages.batches.results(batch.id):
    if res.result.type == "succeeded":
        print(res.custom_id, res.result.message.content[0].text)
PRACTICE · TICK IT AFTER THIS

Run your eval judges as a batch job and compare cost with the synchronous run.

Common mistakes

  • Using batch for user-facing requests.
  • Losing the mapping from results back to records.

Go deeper