The idea
At system level, context is a shared budget split between system prompt, tools, history, retrieved data and room for the answer. Write the budget down per feature, for example 1,500 system, 2,000 tools, 3,000 history, 4,000 retrieval, 1,000 answer.
Enforce it with code: trim history by tokens, summarise when over budget, cap retrieval by tokens rather than chunk count, and load tools dynamically. Log the actual breakdown per request so you can see which part grows.
Smaller contexts are cheaper, faster and often more accurate. Use a large window when the task needs it, not by default.
It controls cost, latency and quality at once, across every feature.
Words to know
- Context budgetPlanned token allocation for each part of the prompt.
Try it without code
Budget one feature
- Pick one feature.
- Allocate tokens to each prompt part.
- Check a real request against the plan.
Use it in code
Fill retrieval up to a token budget instead of a fixed k.
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
def fill_to_budget(chunks, budget_tokens):
picked, used = [], 0
for c in chunks: # chunks are already ranked
n = len(enc.encode(c))
if used + n > budget_tokens:
break
picked.append(c); used += n
return picked, used
chunks = ["short chunk " * 20, "long chunk " * 400, "another " * 50]
print(fill_to_budget(chunks, budget_tokens=1000)[1])Log the token breakdown (system, tools, history, retrieval, output) for 50 requests and find the biggest consumer.
Common mistakes
- Fixed k chunks regardless of chunk size.
- No room left for the answer.