Area 05 · AI System Design · Concept 19 of 20

Context Window Management

Decide what goes in the window: trim, summarise, retrieve only what’s needed.

11–20 · Production levelPhase 6 · Weeks 21–23≈ 45 min

The idea

At system level, context is a shared budget split between system prompt, tools, history, retrieved data and room for the answer. Write the budget down per feature, for example 1,500 system, 2,000 tools, 3,000 history, 4,000 retrieval, 1,000 answer.

Enforce it with code: trim history by tokens, summarise when over budget, cap retrieval by tokens rather than chunk count, and load tools dynamically. Log the actual breakdown per request so you can see which part grows.

Smaller contexts are cheaper, faster and often more accurate. Use a large window when the task needs it, not by default.

Why it matters

It controls cost, latency and quality at once, across every feature.

Words to know

  • Context budgetPlanned token allocation for each part of the prompt.

Try it without code

Budget one feature

  1. Pick one feature.
  2. Allocate tokens to each prompt part.
  3. Check a real request against the plan.

Use it in code

Fill retrieval up to a token budget instead of a fixed k.

python
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")

def fill_to_budget(chunks, budget_tokens):
    picked, used = [], 0
    for c in chunks:                       # chunks are already ranked
        n = len(enc.encode(c))
        if used + n > budget_tokens:
            break
        picked.append(c); used += n
    return picked, used

chunks = ["short chunk " * 20, "long chunk " * 400, "another " * 50]
print(fill_to_budget(chunks, budget_tokens=1000)[1])
PRACTICE · TICK IT AFTER THIS

Log the token breakdown (system, tools, history, retrieval, output) for 50 requests and find the biggest consumer.

Common mistakes

  • Fixed k chunks regardless of chunk size.
  • No room left for the answer.

Go deeper