A retry loop re-runs the work. Durability refuses to.
There is a retry decorator on every tool and a try/except around the loop, so the agent is resilient. Then the pod gets rescheduled forty minutes into a sixty-minute migration task, and you find out what you actually built. A retry decorator protects one call against one transient failure. It has nothing to say about the process dying between step eleven and step twelve, and nothing at all to say about the fourteen writes that already committed when it did.
Durable execution makes a different claim. A retry loop re-executes work and hopes the result comes out the same. Durable execution records the outcome of every step that cannot be recomputed, and on recovery it replays the decisions it already made instead of making them again. The model call that cost eleven cents and took nine seconds is not repeated — its recorded result is handed back to the same line of code that asked for it, and the process fast-forwards to the point where it died.
Agents are the worst possible shape for a retry loop, which is why this is now a standard system design question. They are long-running, expensive per step, and full of side effects nobody wrote a compensating action for. A twenty-step research agent that restarts from zero has not retried. It has re-billed.
A checkpoint is an event log, not a snapshot of your object
The first instinct is to serialise the agent: pickle the object, write it to Redis, load it back. That works in a demo and fails in two specific ways. Half the state is not serialisable — open HTTP clients, generators, the provider SDK’s own session — and the half that is gets captured at an arbitrary moment, which may be the middle of a tool call that has already fired.
The durable version is an append-only log of what happened, with the loop reconstructed from it. Temporal states the rule plainly: the event history is the source of truth, and the workflow function must make the same API calls in the same sequence given the same input. Four event kinds carry an agent: a message appended, a tool call issued with its canonicalised arguments, a tool result, and a compaction. Everything else — token counts, the current step number, whether the budget is exhausted — is derivable, and derivable state does not belong in a checkpoint.
The field that decides whether recovery is correct is the pending one. A call-issued event with no matching result is the crash window: the tool may have run, may have half-run, may never have been reached. No amount of logging closes that window, because the gap between the side effect and the record of it is where the process died. You cannot learn what happened. You can only make re-running it harmless, which is the next problem and the real one.
# The checkpoint is the LOG. The agent object is rebuilt from it, never stored.
from dataclasses import dataclass
from typing import Any, Literal
import hashlib, json
EventKind = Literal["msg", "call_issued", "call_result", "compacted"]
@dataclass(frozen=True)
class Event:
seq: int
kind: EventKind
payload: dict[str, Any]
def canonical(args: dict) -> str:
# Stable bytes for the same logical call: sorted keys, no incidental spacing.
return json.dumps(args, sort_keys=True, separators=(",", ":"))
def step_key(run_id: str, seq: int, tool: str, args: dict) -> str:
# DERIVED, not random. Identical on every replay -- see the idempotency section.
h = hashlib.sha256(canonical(args).encode()).hexdigest()[:16]
return f"{run_id}.{seq}.{tool}.{h}"
def rebuild(log: list[Event]) -> dict:
"""Pure function of the log. No clock, no network, no randomness."""
state: dict[str, Any] = {"messages": [], "pending": None}
for e in sorted(log, key=lambda x: x.seq):
if e.kind == "msg":
state["messages"].append(e.payload)
elif e.kind == "call_issued":
state["pending"] = e.payload # issued; outcome unknown
elif e.kind == "call_result":
state["pending"] = None
state["messages"].append(e.payload)
elif e.kind == "compacted":
state["messages"] = e.payload["summary_messages"]
return state
# Note what is absent: step counts, token totals, "is_done". All derivable.
# A checkpoint that stores derived values is a checkpoint that can disagree
# with itself after a code change.Replay only works if the loop is deterministic
This is the part that surprises people, and it is the reason durable execution feels hostile at first. If recovery works by re-running your function against a recorded history, then your function has to produce the same sequence of calls every time. When a replayed command does not match the history, Temporal fails the execution with a non-determinism error, and the two documented causes are changing the code of a running workflow and intrinsic non-determinism — branching inline on the local clock or on a random number.
So ordinary Python becomes illegal inside the loop. Reading time.time() to decide whether to compact, calling random.random() to jitter a retry, reading a mutable global for the budget: all three diverge on replay. The SDKs hand you recorded substitutes — a workflow clock and a seeded workflow random — precisely so those values come back identical. And everything genuinely external, the model call above all, moves into an activity: recorded once, returned from history on replay, never re-executed.
A language model at non-zero temperature is non-deterministic by construction, so this is not a theoretical concern — it is the single largest source of divergence in an agent, and it is why "the model call is an activity" is not an optimisation. The OpenAI Agents SDK integration draws the line explicitly: model calls are always routed through activities, activity_as_tool() is for anything doing HTTP, database or file I/O, and a plain function_tool runs inside the workflow and must stay deterministic. Two subtler ones in the same document are worth memorising, because they are the kind of thing that only breaks in production: the on_approval_request callback runs in workflow context and must be deterministic, and a callable MCP tool_filter is evaluated during replay, so it cannot touch I/O, the clock, randomness, or mutable global state.
# The rule: same API calls, same sequence, same input. Anything that can
# differ between runs is an Activity, not a line in the loop.
from datetime import timedelta
from temporalio import activity, workflow
@activity.defn
async def call_model(messages: list[dict]) -> dict:
# Non-deterministic by construction: temperature, model version, retries.
# Recorded once; on replay the RESULT comes back, the API is not called.
return await provider.messages.create(model="...", messages=messages)
@workflow.defn
class ResearchAgent:
@workflow.run
async def run(self, task: str) -> str:
messages = [{"role": "user", "content": task}]
# WRONG inside workflow code -- each of these diverges on replay:
# if time.time() - started > 300: compact(messages)
# if random.random() < 0.1: messages = reorder(messages)
# budget = GLOBAL_TOKEN_BUDGET
#
# RIGHT -- the SDK records these, so replay sees the same values:
deadline = workflow.now() + timedelta(minutes=30)
jitter = workflow.random().random()
for seq in range(40):
reply = await workflow.execute_activity(
call_model, messages,
# The OpenAI plugin's ModelActivityParameters defaults
# start_to_close_timeout to 60s. That is wrong for a reasoning
# model on a long prompt. Set it on purpose.
start_to_close_timeout=timedelta(minutes=5),
)
messages.append(reply)
if not reply.get("tool_calls"):
return reply["text"]
if workflow.now() > deadline:
return "deadline reached"
return "step budget exhausted"Idempotency keys, and where the key has to come from
Replay protects you from re-deciding. It does nothing about re-doing. The only defence against the crash window is that executing the same step twice produces the same single effect, and that means a key the remote system can recognise.
Stripe’s contract is the clearest published version of the semantics, and worth knowing in detail because every other vendor is a variation on it. For v1 the status code and body of the first request bearing a key are saved once endpoint execution begins, and retries with that key return the saved response — including 500s. Keys may be up to 255 characters. The idempotency layer compares the incoming parameters against the original request and errors if they differ, which is a feature: it is what stops a replan that changed the amount from quietly reusing the grant for the old one. Only POST accepts a key; GET and DELETE are idempotent by definition.
Two consequences specific to agents. First, the key must be derived, not generated — a uuid4() at the call site is a fresh key on every replay, which is the same as having no key. Derive it from the run id, the step sequence and a hash of the canonicalised arguments, so the second attempt at step nine presents exactly what the first one did. Second, keys get pruned after at least 24 hours, and a reused key after pruning generates a new request. An agent that pauses three days waiting on a human approval is outside that window entirely: the key minted on Monday protects nothing on Thursday. Long pauses need your own dedupe table, keyed on the approval, not the vendor’s.
# Derived key -- identical across replays, different across logical steps.
def issue_refund(run_id: str, seq: int, order_id: str, amount_cents: int):
key = step_key(run_id, seq, "refund",
{"order": order_id, "amount_cents": amount_cents})
return stripe.Refund.create(
charge=order_id, amount=amount_cents,
idempotency_key=key, # <= 255 chars, and STABLE
)
# What NOT to do, and why it looks fine in testing:
# idempotency_key=str(uuid.uuid4()) # new key each replay == no key
# idempotency_key=f"{run_id}:refund" # same key for a RE-PLANNED amount
# # -> vendor errors, or worse, doesn't
# For anything that outlives the vendor's 24h window -- a durable pause on a
# human approval, a retry three days later -- own the dedupe yourself.
def once(conn, key: str, fn):
"""Claim the key first, then act. The row IS the record of intent."""
claimed = conn.execute(
"INSERT INTO effects(key, state) VALUES (%s,'pending') "
"ON CONFLICT (key) DO NOTHING RETURNING key", (key,)
).fetchone()
if not claimed:
row = conn.execute("SELECT state, result FROM effects WHERE key=%s",
(key,)).fetchone()
if row.state == "done":
return row.result # already happened; return the result
raise InFlight(key) # someone else is mid-flight: back off
result = fn() # crash here -> row stays 'pending'
conn.execute("UPDATE effects SET state='done', result=%s WHERE key=%s",
(result, key))
return result
# A 'pending' row that never resolves is the crash window made visible. That is
# the point: a stuck row is a reconciliation job, not a silent double charge.The history is the state, and the history grows
Durability has a bill, and for agents it arrives in an unusual currency. Because the history is the state, a long-running agent accumulates one. Temporal caps a workflow execution at 51,200 events or 50 MB, and warns at 10,240 events or 10 MB. Pending activities, child workflows, signals and cancellation requests each default to a limit of 2,000 per execution, and hitting one fails the workflow task. The escape hatch is continue-as-new: end the current execution and start a fresh one carrying forward a condensed input.
Look at what that actually asks for. To continue as new you must compress a long trajectory into a short state that is sufficient to carry on — which is exactly the compaction problem from earlier in this series, now with a second, harder budget attached. Your agent is simultaneously bounded by the model’s context window, measured in tokens, and by the durable history, measured in events and megabytes. They are different units and they fill at different rates. A chatty loop with tiny messages hits the event ceiling first; a RAG agent stuffing 40 KB documents into each observation hits the size ceiling first.
And there is a trap worth stating outright: the compaction itself must be a recorded event, not a computed one. If the loop decides to summarise based on a live token count, two replays of the same history can compact at different points and diverge. Record the decision, record the summary, replay both.
What the frameworks give you, and the defaults that hand it back
LangGraph exposes the tradeoff as a durability setting with three values, and the gap between them is the whole guarantee. With exit, state is persisted only when execution completes; performance is best and you cannot recover from a mid-run failure or interrupt at all. With async, changes are persisted while the next step executes, leaving a small window in which a crash loses the checkpoint. With sync, each change is written before the next step starts, at some performance cost. Only the last one means what "durable" implies, so set it explicitly rather than inheriting whatever the version you installed happens to default to.
The resume semantics are the other thing to internalise. Resumed code does not continue from the line where it stopped: the run replays from the start of the node where execution halted, and for a subgraph, from the parent node that called it. The guidance follows directly — wrap each side-effecting operation in its own task, so its recorded result is pulled from the checkpointer rather than re-executed, and make the effect idempotent anyway, because a task that started but never finished will re-run.
If you are running the OpenAI Agents SDK on Temporal, the plugin does most of the wiring: OpenAIAgentsPlugin registers the model activity, configures the Pydantic data converter and propagates tracing context. Read the restrictions before you design around it. SQLiteSession is unsupported because it is a local file. LocalShellTool and ComputerTool are unsupported. Streaming requires a streaming_topic and is incompatible with use_local_activity. And one that is a security finding rather than a limitation: do not pass secrets through an MCP factory_argument, because it is recorded in workflow history — durable state is a log, and a log of your credentials is a durable leak.
# LangGraph: one task per side effect, and durability set on purpose.
from langgraph.func import task
@task
def send_invoice(customer: str, cents: int, key: str) -> str:
return billing.create_invoice(customer, cents, idempotency_key=key)
@task
def notify(customer: str, invoice_id: str, key: str) -> None:
mailer.send(customer, invoice_id, idempotency_key=key)
def bill_node(state):
# On resume this function restarts from the line BELOW, not from wherever
# it stopped. Each task's recorded result is fetched from the checkpointer;
# anything not wrapped in a task simply runs a second time.
inv = send_invoice(state["customer"], state["cents"],
state["key"] + ".inv").result()
notify(state["customer"], inv, state["key"] + ".mail").result()
return {"invoice_id": inv}
# Two side effects, two tasks. Putting both in one task means a crash between
# them replays the invoice as well as the email.
# durability="exit" persisted only at completion. Fastest; no mid-run
# recovery and no interrupts at all.
# durability="async" persisted while the next step runs. Small crash window
# where the checkpoint is lost.
# durability="sync" persisted before the next step starts. Some overhead.
# The only one that matches what "durable" promises.
graph.invoke(initial_state, config, durability="sync")When a retry loop really is enough
Durable execution is not free, and the cost is not the server. It is a programming model that rejects ordinary code, a replay story you now owe on every deploy — changing the code of a running workflow is one of the two documented causes of non-determinism errors, which is why versioning and patching APIs exist — and a new class of incident where the engine is fine and the history is poisoned. Adopting it for a task that finishes in eight seconds is how a team ends up maintaining a workflow engine to run a for loop.
A workable boundary. If the whole task completes in well under a minute and every write is already idempotent, a retry loop plus an idempotency key is the right answer; restarting from zero costs a rounding error. If the task runs for minutes to hours, fans out across tools, or has irreversible effects partway through, durability earns its keep because a restart from zero is money. And if the task pauses on something outside the process — a human approval, a vendor callback, a batch window — you need durability regardless of runtime, because the pause can outlive the pod, the deploy, and the connection.
The in-between case is real and under-used: checkpoint to a row. One table keyed by run id, holding the event log and a step cursor, with the loop rebuilding from it on start. That is maybe eighty lines, it survives a restart, and it gives you the thing that actually matters — a resume that does not redo paid work — without inheriting a determinism contract. Reach for the engine when you need timers, signals, fan-out with per-branch retries, and visibility across thousands of concurrent runs.
The default, and the interview answer
Default to this: an append-only event log as the checkpoint with nothing derived stored in it, every external call behind an activity boundary, idempotency keys derived from run id plus step plus a hash of canonicalised arguments, sync durability, one task per side effect, compaction recorded as an event rather than computed, and your own dedupe table for anything that pauses longer than a vendor keeps keys.
The interview answer is three moves. A retry loop re-executes and hopes; durable execution replays recorded outcomes, so the eleven-cent model call that already succeeded is never paid for twice — which is why the model call has to be an activity and not a line in the loop. Replay imposes determinism, so the clock, the randomness and the mutable globals in your loop become illegal, and a non-zero-temperature model is the biggest divergence source in the system. And durability does not eliminate the crash window between a side effect and its record; nothing does. It converts it into a problem you can fix with a derived idempotency key, and the key has to be derived, because uuid4() at the call site is a new key on every replay and therefore no key at all.
The one-line version for a design doc: checkpoint the decisions, make the effects idempotent, and keep the clock out of the loop.
Work the same ideas as a full interview case study, with pressure questions and model answers.