Context Engineering: Budgeting, Caching and Compacting the Window

From prompt engineering to context engineering

Prompt engineering is about the words in one message. Context engineering is about everything the model sees on a turn — system instructions, tool definitions, loaded skills, retrieved documents, memory, the conversation so far, and the tool results piling up inside it — and about deciding, per turn, what deserves to be there. In an agent that runs for forty turns, the prompt you wrote is a rounding error; the tool results are the context.

Two constraints drive the discipline. Attention is finite: models get measurably worse at following instructions as the window fills, well before the hard limit. And tokens are money and latency: every turn re-sends the whole window, so a 100k-token context on turn forty has cost you four million input tokens cumulatively.

The layers and their budgets

Think of the window as layers with different lifetimes. Stable layers (system prompt, tool schemas, skill index) change rarely and should be cached. Semi-stable layers (loaded skill bodies, retrieved documents, memory) change per task. Volatile layers (conversation turns, tool results) change every turn and are where compaction happens.

Prompt caching on each provider

All three providers let you pay once for a stable prefix and reuse it across calls, cutting input cost by roughly 75–90% on the cached portion and improving time-to-first-token. The mechanics differ and interviewers like to hear that you know them. Claude uses explicit cache_control breakpoints on content blocks, with a minimum cacheable length and a short TTL that refreshes on use. Gemini has explicit context caches you create as objects with a TTL and reference by name, plus implicit caching on recent models. OpenAI caches automatically for prompts over a threshold length when the prefix matches — no API change, but you must keep the stable content first.

The rule that applies to all three: order the context from most stable to most volatile. System prompt, then tools, then skills, then documents, then conversation. One dynamic token in the prefix — a timestamp, a request ID — and the cache misses on every call.

from anthropic import Anthropic

client = Anthropic()
reference = open("handbook.md").read()   # large, stable

SYSTEM = [
    {"type": "text", "text": "You are a support engineer. Follow the handbook exactly."},
    {
        "type": "text",
        "text": reference,
        # Everything up to and including this block is cached (5-min TTL,
        # refreshed on each hit; a 1-hour TTL is available).
        "cache_control": {"type": "ephemeral"},
    },
]

def ask(messages: list[dict]) -> str:
    resp = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=1000,
        system=SYSTEM,
        messages=messages,                      # volatile part after the breakpoint
    )
    u = resp.usage
    print(u.cache_read_input_tokens, "read from cache,",
          u.cache_creation_input_tokens, "written to cache")
    return resp.content[0].text

# Tools placed before the system prompt are cached too. Up to 4 breakpoints
# let you cache system, tools and the conversation-so-far separately.

Trimming tool results before they hit the window

The cheapest context win is at the tool boundary. A search tool that returns 30 results with full snippets, a SQL tool that returns 2,000 rows, a file reader that returns a 4,000-line file — each one dumps tens of thousands of tokens into the window that the model will scan once and drag around forever. Design tools to return what the model needs for the next decision: top-5 results with one-line snippets and an offset for more; row counts and a sample with a flag for the full set; a file with line ranges.

Then, after the model has used a tool result, consider replacing it in history with a short placeholder ("[read 412 lines of auth.py]"). The model already extracted what it needed; the raw content is dead weight on every subsequent turn.

Compaction for long-running agents

When the window passes a threshold — 60–70% is a common trigger — summarise the conversation so far into a structured state document and start a fresh context containing the stable layers plus that summary. The summary must preserve what the agent needs to continue: the goal, decisions made and why, open items, file paths and identifiers touched, and anything the user explicitly asked to be remembered. It must drop the noise: raw tool output, superseded drafts, exploration that led nowhere.

Claude Code, Gemini CLI and Codex all implement some form of this; the Claude Agent SDK and OpenAI Agents SDK expose hooks for it. When you build your own, generate the summary with a dedicated prompt and a fixed schema, and keep the last two or three raw turns verbatim so the agent does not lose the immediate thread.

• •Trigger: token count crosses a threshold, or a natural phase boundary (task finished, new task begins)
• •Keep: goal, plan, decisions + rationale, open questions, identifiers (paths, IDs, URLs), user instructions
• •Drop: raw tool results, superseded drafts, dead ends, pleasantries
• •Verify: an eval that compacts mid-task and checks the agent still finishes correctly — compaction bugs are silent otherwise