The idea
Not every request needs an answer in seconds. Classifying a million tickets, embedding a document archive, generating eval judgements overnight: these can wait hours. Providers offer batch APIs that accept a file of requests, process them within a window (often up to 24 hours) and charge a large discount.
Design for it: write requests as JSONL with your own IDs, submit, poll for completion, download results and join them back by ID. Handle partial failures by resubmitting only the failed items.
Self-hosted, the same idea means running large offline jobs at maximum batch size when interactive traffic is low.
It can cut the cost of offline workloads roughly in half with almost no code change.
Words to know
- custom_idYour identifier for each request, used to match results.
Try it without code
Find batch candidates
- List every model call in your system.
- Mark which ones a user waits for.
- The rest are batch candidates.
Use it in code
Submit a batch and collect results.
import time, anthropic
client = anthropic.Anthropic()
tickets = {"t1": "I was charged twice", "t2": "Export button broken"}
batch = client.messages.batches.create(requests=[
{"custom_id": tid, "params": {"model": "claude-haiku-5-5", "max_tokens": 5,
"messages": [{"role": "user", "content": f"Label BILLING, BUG or HOWTO: {text}"}]}}
for tid, text in tickets.items()])
while client.messages.batches.retrieve(batch.id).processing_status != "ended":
time.sleep(60)
for res in client.messages.batches.results(batch.id):
if res.result.type == "succeeded":
print(res.custom_id, res.result.message.content[0].text)Run your eval judges as a batch job and compare cost with the synchronous run.
Common mistakes
- Using batch for user-facing requests.
- Losing the mapping from results back to records.