Area 05 · AI System Design · Concept 06 of 20

Streaming Responses

Send tokens to the user as they are generated, so the app feels fast.

01–10 · The basePhase 6 · Weeks 21–23≈ 1 hour

The idea

A 400-token answer can take 5–10 seconds to generate. Waiting for all of it feels broken; showing words as they arrive feels fast. Streaming APIs return a sequence of events (text deltas, tool call pieces, a final usage event) over one connection.

Your backend relays these to the browser, usually with Server-Sent Events. The UI appends text as it arrives. Handle cancellation (user closes the tab, so stop the upstream call), errors mid-stream and the final usage event for cost logging.

Streaming changes how you validate: you cannot check the full output before showing it. For structured outputs, stream to a progress UI and validate at the end.

Why it matters

It is the single biggest improvement to perceived speed in chat-style products.

Words to know

  • DeltaA small piece of new text in a stream.
  • SSEServer-Sent Events: one-way HTTP streaming to the browser.

Try it without code

Compare feel

  1. Ask a chat app for a long answer and watch it stream.
  2. Imagine the same wait with a spinner and no text.

Use it in code

Stream from the model to the browser with FastAPI and SSE.

python
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import anthropic, json

app = FastAPI()
client = anthropic.AsyncAnthropic()

@app.get("/chat")
async def chat(q: str):
    async def events():
        async with client.messages.stream(model="claude-sonnet-5-5", max_tokens=800,
                                          messages=[{"role": "user", "content": q}]) as stream:
            async for text in stream.text_stream:
                yield f"data: {json.dumps({'text': text})}\n\n"
            final = await stream.get_final_message()
            yield f"data: {json.dumps({'done': True, 'usage': final.usage.model_dump()})}\n\n"
    return StreamingResponse(events(), media_type="text/event-stream")
PRACTICE · TICK IT AFTER THIS

Stream answers into a simple HTML page with EventSource and log time to first token.

Common mistakes

  • Proxies buffering the stream (disable buffering).
  • Not cancelling upstream when the user disconnects.

Go deeper