The idea
The context window is the model’s working memory for one call. It holds the system prompt, any tools you define, the conversation history, retrieved documents and the answer being generated. If the total goes over the limit, the API rejects the call or you must cut something.
Windows have grown from a few thousand tokens to hundreds of thousands or more. Bigger is not free: you pay for every input token on every call, latency rises with prompt length, and quality on details far back in a long context is not guaranteed.
The output limit is separate and usually much smaller. A model with a large window might still cap a single reply at a few thousand or tens of thousands of tokens, set by the max_tokens parameter you send.
Every design choice in RAG and agents (how many chunks, how much history, when to summarise) is a decision about what to put in this window.
Words to know
- Context windowMaximum input + output tokens for one call.
- max_tokensThe cap you set on how long the reply may be.
- TruncationCutting old messages or text so the request fits.
Try it without code
Fill a window on purpose
- Start a new chat and paste a very long document, then keep chatting for many turns.
- Ask the model to quote the first sentence you sent.
- Note whether the app silently dropped or summarised early history.
Use it in code
Keep a chat under a token budget by dropping the oldest turns first.
import anthropic
client = anthropic.Anthropic()
MODEL, BUDGET = "claude-sonnet-5-5", 8000
def fit(system, messages):
while True:
n = client.messages.count_tokens(model=MODEL, system=system, messages=messages).input_tokens
if n <= BUDGET or len(messages) <= 1:
return messages, n
messages = messages[2:] # drop the oldest user+assistant pair
history = [{"role": "user", "content": "hello " * 3000},
{"role": "assistant", "content": "Hi!"},
{"role": "user", "content": "What did I say first?"}]
history, used = fit("You are concise.", history)
print(len(history), "messages kept,", used, "tokens")Add the trimming function to a simple chat loop and print the remaining budget after every turn.
Common mistakes
- Dropping the system prompt or tool definitions when trimming.
- Assuming a big window means the model uses all of it equally well.
- Leaving max_tokens so small that answers get cut off mid-sentence.