Area 08 · Backend · Concept 03 of 20

Async and Await

Handle many slow calls (like LLM requests) at the same time without blocking.

01–10 · The basePhase 0 + 5 · Weeks 1–2, 18–20≈ 1.5 hours

The idea

An LLM call spends seconds waiting on the network. In synchronous code, a worker waiting on one call can do nothing else. With async code, await hands control back to an event loop while waiting, so one process can have hundreds of calls in flight.

In Python: define coroutines with async def, call them with await, and run several concurrently with asyncio.gather or TaskGroup. Use async libraries (httpx, AsyncAnthropic, asyncpg); a blocking call like time.sleep or requests.get inside async code freezes everything.

FastAPI runs async def endpoints on the event loop and plain def endpoints in a thread pool, so both work, but mixing blocking calls into async endpoints is a classic bug.

Why it matters

AI backends are I/O-bound. Async is how you serve many users and run parallel retrieval, judges and tool calls.

Words to know

  • CoroutineA function defined with async def.
  • Event loopThe scheduler that runs coroutines while others wait.

Try it without code

Feel the difference

  1. Run the example below.
  2. Compare the time for sequential and concurrent calls.

Use it in code

Five model calls, sequential vs concurrent.

python
import asyncio, time, anthropic
client = anthropic.AsyncAnthropic()

async def ask(q):
    m = await client.messages.create(model="claude-haiku-5-5", max_tokens=50,
                                     messages=[{"role": "user", "content": q}])
    return m.content[0].text

async def main():
    qs = [f"One fun fact about the number {i}" for i in range(5)]
    t = time.perf_counter()
    for q in qs: await ask(q)
    print(f"sequential {time.perf_counter() - t:.1f}s")
    t = time.perf_counter()
    await asyncio.gather(*(ask(q) for q in qs))
    print(f"concurrent {time.perf_counter() - t:.1f}s")

asyncio.run(main())
PRACTICE · TICK IT AFTER THIS

Make your RAG pipeline run query rewriting and keyword search concurrently and measure the latency saved.

Common mistakes

  • requests or time.sleep inside async functions.
  • Unbounded gather that trips rate limits; use a Semaphore.

Go deeper