The idea
An LLM call spends seconds waiting on the network. In synchronous code, a worker waiting on one call can do nothing else. With async code, await hands control back to an event loop while waiting, so one process can have hundreds of calls in flight.
In Python: define coroutines with async def, call them with await, and run several concurrently with asyncio.gather or TaskGroup. Use async libraries (httpx, AsyncAnthropic, asyncpg); a blocking call like time.sleep or requests.get inside async code freezes everything.
FastAPI runs async def endpoints on the event loop and plain def endpoints in a thread pool, so both work, but mixing blocking calls into async endpoints is a classic bug.
AI backends are I/O-bound. Async is how you serve many users and run parallel retrieval, judges and tool calls.
Words to know
- CoroutineA function defined with async def.
- Event loopThe scheduler that runs coroutines while others wait.
Try it without code
Feel the difference
- Run the example below.
- Compare the time for sequential and concurrent calls.
Use it in code
Five model calls, sequential vs concurrent.
import asyncio, time, anthropic
client = anthropic.AsyncAnthropic()
async def ask(q):
m = await client.messages.create(model="claude-haiku-5-5", max_tokens=50,
messages=[{"role": "user", "content": q}])
return m.content[0].text
async def main():
qs = [f"One fun fact about the number {i}" for i in range(5)]
t = time.perf_counter()
for q in qs: await ask(q)
print(f"sequential {time.perf_counter() - t:.1f}s")
t = time.perf_counter()
await asyncio.gather(*(ask(q) for q in qs))
print(f"concurrent {time.perf_counter() - t:.1f}s")
asyncio.run(main())Make your RAG pipeline run query rewriting and keyword search concurrently and measure the latency saved.
Common mistakes
- requests or time.sleep inside async functions.
- Unbounded gather that trips rate limits; use a Semaphore.