Budgeting the Context Window: A Token Accounting Method

Measure before you budget

Most teams have never counted what actually goes into a turn. The estimate people give from memory is consistently wrong, and usually wrong in the same direction — tool schemas are far larger than anyone expects.

Start by measuring one real conversation at turn one, turn five and turn fifteen, broken down by component. The shape of that table tells you where the problem is before you change anything.

A typical result: a fixed cost of two to four thousand tokens that you pay on every call regardless, and a conversational component that has quietly grown past everything else by turn fifteen.

def audit(messages, tools, system) -> dict:
    n = lambda x: count_tokens(json.dumps(x) if not isinstance(x, str) else x)
    fixed  = n(system) + sum(n(t) for t in tools)
    convo  = sum(n(m) for m in messages if m["role"] != "tool")
    tools_ = sum(n(m) for m in messages if m["role"] == "tool")
    total  = fixed + convo + tools_
    return {"fixed": fixed, "conversation": convo, "tool_results": tools_,
            "total": total, "share_fixed": round(fixed / total, 2)}

# turn  1 -> fixed 3,100  convo   180  tools     0   total  3,280
# turn  5 -> fixed 3,100  convo   940  tools 6,200   total 10,240
# turn 15 -> fixed 3,100  convo 2,800  tools 31,400  total 37,300

The three budgets, with numbers

Take your usable window — the model's context minus the space you need for the response — and divide it deliberately rather than by accident.

A workable starting split for a single-purpose agent on a large window: fixed context capped at around ten per cent, retrieved context up to thirty per cent, conversation and tool results forty per cent, and a reserve of twenty per cent that you never plan to use. The reserve is not waste; it is what absorbs an unexpectedly large tool return without the turn collapsing.

The exact percentages matter less than having them written down and enforced in code. A budget nobody checks is a comment.

Fixed context is the highest-leverage cut

Fixed context is paid on every call, so a saving there multiplies by every turn of every conversation you will ever run. It is also where the most waste sits, because tool schemas grow silently as the tool set grows.

Two moves. Trim the tool set — an agent with thirty tools is usually three agents wearing one coat, and splitting improves selection accuracy at the same time as it cuts tokens. And tighten schemas: nested object parameters serialise enormously, and flattening them often saves hundreds of tokens per tool with no loss of capability.

Cutting fixed context from 3,100 to 1,800 tokens does not sound dramatic. Over a fifteen-turn conversation it is nearly twenty thousand tokens, on every conversation.

Enforce the budget at the boundary

Budgets fail when they live in a design document. Put the check where context enters — at the tool dispatch boundary — and it holds automatically.

The rule: no single tool result may exceed its share, and when you truncate you say so. Silent truncation is worse than no truncation, because the model reasons over a partial set believing it is complete.

BUDGET = {"fixed": 0.10, "retrieved": 0.30, "conversation": 0.40}
USABLE = 180_000                       # window minus response headroom
MAX_TOOL_RESULT = int(USABLE * BUDGET["retrieved"] / 4)   # ~4 live results

def clamp(payload: dict) -> dict:
    text = json.dumps(payload)
    if count_tokens(text) <= MAX_TOOL_RESULT:
        return payload
    return {
        "ok": payload.get("ok", True),
        "truncated": True,
        "note": (f"Result exceeded the {MAX_TOOL_RESULT} token budget and was "
                 f"cut. Narrow the query - filter by date or add a limit."),
        "partial": json.loads(text[: MAX_TOOL_RESULT * 3]  + "}") ,
    }

Compact the conversation on a trigger, not a timer

When the conversational budget is breached, compact: replace the oldest exchanges with a summary that preserves decisions and constraints while dropping payloads.

Trigger on the measured budget rather than a turn count, because turns vary enormously in size — three turns of large tool results can exceed twenty turns of conversation.

The thing to protect is decisions. A compaction that loses "the customer chose the express option" has broken the agent in a way that is hard to spot and easy to blame on the model. Summarise the narrative; carry the facts forward verbatim, and promote the durable ones into session state where compaction cannot touch them.

Budgets are a cost model too

The same numbers tell you what a conversation costs, which is the calculation most teams do only after the first surprising bill.

The important observation is that input tokens dominate and they compound. A fifteen-turn conversation does not send its context once; it sends a growing context fifteen times. Total input is the sum over turns, not the size of the final turn — which is why a 37,000-token final turn implies roughly a quarter of a million input tokens across the conversation.

That is the arithmetic that makes fixed-context trimming worth an afternoon, and it is what prompt caching acts on. Both are covered in the cost post later in this series.

# Input is re-sent every turn, so cost is the SUM over turns.
turns  = [3_280, 6_100, 10_240, 18_900, 37_300]
total_in = sum(turns)                 # ~75,800 for five sampled turns

# Trim fixed context by 1,300 tokens and you save 1,300 x turns,
# on every conversation, forever.