AI Agent Memory Architectures: Short-Term, Long-Term, and Checkpointing — What Interviews Actually Test

Why "just put it in the prompt" stops working

An agent's only native memory is its context window — and the context window is a whiteboard, not a filing cabinet. It is wiped between sessions, it overflows within one long session, and everything on it is re-read (and re-billed) on every model call. A 100-turn support conversation at full history means paying prefill for 100 turns of tokens on turn 101, with the relevant fact from turn 3 competing for attention against 96 turns of noise.

Production agent memory is therefore a hierarchy, exactly like CPU caches: small-fast-expensive at the top (the context window), large-slow-cheap at the bottom (durable stores), with explicit policies deciding what moves between layers. Interviewers want to hear that hierarchy named and each layer justified.

Layer 1 — working memory: the context window, managed

The first layer is what stays in the prompt: the system instructions, the last N turns verbatim, and a running summary of everything older. The standard pattern is summarize-and-truncate: when the transcript approaches a token budget, an LLM call compresses the oldest turns into a summary block that replaces them. The trade-off to state out loud: summaries are lossy and irreversible — a detail dropped by the summarizer is gone unless a lower layer holds the raw transcript.

A detail that separates senior answers: token budgets should reserve headroom for tool results, which can be huge and arrive mid-turn. Teams that budget only for conversation history get context-overflow crashes the first time a tool returns a 40KB JSON payload.

Layer 2 — episodic and semantic memory: the retrieval store

Facts worth remembering across sessions — user preferences, prior decisions, entity facts — move into a store the agent queries at the start of (or during) a session. Two common shapes: a vector store over past conversation chunks (episodic recall: "what did we discuss about the pricing migration?") and a structured key-value or document store of extracted facts (semantic memory: preferred_language=Python, deploy_day=Friday).

The interview-grade insight is that writing this memory is the hard half. Naive designs embed every turn, which retrieves noise forever. Better designs run an extraction step — after a session or significant turn, an LLM decides what durable facts are worth writing, deduplicates against existing memory, and updates rather than appends when facts change. Memory quality is a write-path property, not a read-path property.

Layer 3 — checkpointing: surviving the crash

Working memory and retrieval stores cover forgetting; checkpointing covers dying. A multi-step agent that crashes on step 7 of 10 must not repeat steps 1–6 — some had side effects. The pattern (LangGraph calls its implementation a checkpointer) is to persist the full graph state — messages, intermediate variables, next node — after every step, keyed by thread ID, into Postgres/Redis/SQLite.

Resume then means: load the latest checkpoint for the thread, and continue from the recorded position. The same mechanism gives you human-in-the-loop for free — an approval gate is just a checkpoint the graph pauses at until a human resumes it — and time-travel debugging, since every historical state is replayable. When an interviewer asks "what happens when your orchestrator restarts mid-run?", checkpointing is the answer they are waiting to hear.

# LangGraph: durable state via a Postgres checkpointer
from langgraph.checkpoint.postgres import PostgresSaver
from langgraph.graph import StateGraph

builder = StateGraph(AgentState)
# ... add nodes and edges ...

with PostgresSaver.from_conn_string(DB_URL) as checkpointer:
    graph = builder.compile(checkpointer=checkpointer)

    config = {"configurable": {"thread_id": "ticket-4812"}}
    graph.invoke({"messages": [user_msg]}, config)
    # process dies here — no problem:
    # a new process with the same thread_id resumes
    # from the last persisted step, not from scratch.

The failure modes interviewers probe

Stale memory: the user changed their deployment day, but semantic memory still says Friday. Fix: extraction updates keyed facts instead of appending, and retrieval prefers recency on conflict.

Memory poisoning: a tool result or malicious input gets written as a durable "fact" and influences every future session. Fix: provenance tags on memory entries, and never letting retrieved memory override system-level policy.

Cost creep: retrieval stuffing 20 memory chunks into every prompt quietly doubles token spend. Fix: rank and cap injected memory, and measure answer quality against a no-memory baseline — memory must earn its tokens.

Checkpoint bloat: full-state snapshots of long threads grow unboundedly. Fix: snapshot deltas per step with periodic compaction, exactly like a write-ahead log.

Go deeper

The AI Agent Orchestration case study on this site designs the full durable-run architecture around this memory stack — including the interviewer pressure round. Pair it with the LangGraph tutorial for the hands-on version, and the LLM Inference Serving case study to understand why prefix caching makes long-context memory layouts cheaper than they look.