The idea
An LLM has no memory between calls. Short-term memory is simply everything you resend each time: the system prompt, the conversation so far, and the tool calls and results from this run.
It grows quickly. Tool results (web pages, file contents, query results) can be thousands of tokens each, and every step resends all of them. That drives up cost and latency and can push important details out of focus.
Manage it deliberately: truncate or summarise large tool results before appending, keep only what later steps need, and use a scratchpad (a notes tool or plan) for key facts.
Most agent cost and many agent failures come from an overstuffed context.
Words to know
- Working memoryAnother name for what is in the context window right now.
Try it without code
Measure context growth
- Run an agent task with logging of input tokens per step.
- Plot or list them.
- Spot which tool results caused the biggest jumps.
Use it in code
Trim tool results before they enter the context.
def compact_result(text, limit=3000):
if len(text) <= limit:
return text
head, tail = text[:limit * 2 // 3], text[-limit // 3:]
return f"{head}\n...[{len(text) - limit} chars omitted]...\n{tail}"
# in the loop:
# results.append({"type": "tool_result", "tool_use_id": b.id, "content": compact_result(raw)})
# and log msg.usage.input_tokens after every step to watch growthLog input tokens per step for 3 agent runs, then add result trimming and compare totals.
Common mistakes
- Appending entire web pages as tool results.
- Assuming the model "remembers" past sessions.