Fan-out is a capacity decision wearing a speed decision’s clothes
Twenty documents, one summarizer agent each. Sequentially it takes eleven minutes; wrapped in asyncio.gather, forty seconds. Then on a Tuesday it takes nine minutes again, because most of the wall clock now goes on collecting 429s and sleeping them off. Nothing in your code changed. The speed-up was never bounded by your concurrency — it was bounded by the tokens per minute your organization may spend, and you had been under that ceiling by accident.
The part worth internalising: a sequential agent loop is self-throttling. One request in flight, the next turn blocked on the last, and you get backpressure for free without ever using the word. Fan-out deletes that property. Dispatch N branches and you have handed rate-limit management to the provider, who manages it by refusing you — and the refusal is not localised, because the retries you fire are competing with the forty-nine siblings still running.
Three decisions come before the library: how many branches run at once, what happens to the siblings when one fails, and what you return when some succeed. Every default is wrong for agents.
Pick the failure semantics before you pick the primitive
The two obvious primitives in Python are not interchangeable, and each difference is documented in a line. asyncio.gather with the default return_exceptions=False propagates the first raised exception immediately to the awaiting task — and, per the docs, other awaitables in the sequence will not be cancelled and will continue to run. For an agent fan-out that is the worst available default: an exception at the top while twenty branches keep burning tokens nobody will read. With return_exceptions=True, exceptions are treated the same as successful results and aggregated in the result list.
asyncio.TaskGroup, added in 3.11, inverts it. The first time a task fails with anything other than CancelledError, the remaining tasks in the group are cancelled, no further tasks can be added, and the exceptions are combined into an ExceptionGroup you catch with except*. The docs state the comparison outright: TaskGroup provides stronger safety guarantees than gather, because gather will not cancel the remaining scheduled tasks.
Map that onto what the product needs. All-or-nothing — a missing branch invalidates the answer — is TaskGroup, and the cancellation is the feature. Best-effort — forty-seven of fifty summaries is usable — is gather with return_exceptions=True. Quorum, where the first k good answers are all you want, is asyncio.wait with FIRST_COMPLETED and an explicit cancel of the losers. Choosing by taste rather than semantics is how you pay for work you then throw away.
# Three failure semantics. Pick one deliberately; the defaults are not neutral.
import asyncio
# 1. ALL-OR-NOTHING. One failure cancels the siblings. 3.11+.
async def all_or_nothing(items):
async with asyncio.TaskGroup() as tg: # cancels remaining on failure
tasks = {i: tg.create_task(run_branch(i)) for i in items}
return {i: t.result() for i, t in tasks.items()}
# caller: try/except* ExceptionGroup -- errors arrive grouped, not one by one
# 2. BEST-EFFORT. Exceptions become values. Nothing is cancelled.
async def best_effort(items):
out = await asyncio.gather(*(run_branch(i) for i in items),
return_exceptions=True)
ok = [(i, r) for i, r in zip(items, out) if not isinstance(r, Exception)]
bad = [(i, r) for i, r in zip(items, out) if isinstance(r, Exception)]
return ok, bad # bad is NOT optional output
# 3. QUORUM. First k good answers; cancel the rest instead of paying for them.
async def quorum(items, k):
pending = {asyncio.create_task(run_branch(i)) for i in items}
done_ok = []
while pending and len(done_ok) < k:
done, pending = await asyncio.wait(pending,
return_when=asyncio.FIRST_COMPLETED)
done_ok += [t.result() for t in done if not t.exception()]
for t in pending:
t.cancel() # the whole point of quorum
return done_ok[:k]
# gather's default -- return_exceptions=False -- raises at the top while the
# siblings keep running. That is a bill with no corresponding output.Bound the concurrency, then find out which number actually binds
A semaphore around the branch body is the whole implementation, and where it goes matters: acquire before the request, hold it for the entire branch including its retries, release in a finally. A semaphore released before a retry sleep is not a concurrency limit, it is a suggestion.
Then establish which of the three ceilings is lowest, because only one is yours: your semaphore, the HTTP client’s connection pool, the provider’s rate limit. People assume the client pool protects them. It does not — the OpenAI Python SDK ships DEFAULT_CONNECTION_LIMITS as Limits(max_connections=1000, max_keepalive_connections=100), so the transport will open a thousand sockets while your quota tops out two orders of magnitude earlier. If you have not written a limiter, you do not have one.
Framework caps have their own failure mode: being silently ignored. LangGraph takes max_concurrency as a top-level key of the config dict, and an open issue documents the docs example nesting it under configurable, where it does nothing — on a forty-branch Send fan-out, a top-level cap of 4 peaked at 4 concurrent tasks while the nested spelling peaked at 40, identical to passing no config, with no warning raised. A cap that is ignored looks exactly like a cap that is working, right up until the traffic arrives.
# A bounded pool. The semaphore covers the retries, not just the first attempt.
import asyncio, random
class Pool:
def __init__(self, width: int, floor: int = 2):
self.sem = asyncio.Semaphore(width)
self.width, self.floor = width, floor
self._extra = 0 # permits withdrawn by backpressure
async def narrow(self):
"""A 429 is news about the POOL, not about one branch."""
if self.width - self._extra > self.floor:
self._extra += 1
await self.sem.acquire() # hold a permit out of circulation
async def run(self, coro_factory, attempts: int = 4):
async with self.sem: # held across every retry below
for n in range(attempts):
try:
return await coro_factory()
except RateLimited as e:
await self.narrow()
delay = e.retry_after or 0.5 * 2 ** n # header is a MINIMUM
await asyncio.sleep(delay + random.uniform(0, delay * 0.3))
raise Exhausted()
# Three ceilings. Know which one is lowest.
# yours -> Semaphore(width)
# transport -> openai-python: Limits(max_connections=1000, keepalive=100)
# provider -> RPM / ITPM / OTPM, and this is almost always the real one
# LangGraph: max_concurrency is TOP-LEVEL config. Nested under "configurable"
# it is silently ignored -- 40 Send branches ran 40-wide with no warning.The limit you hit is tokens, not the number you were watching
Everyone budgets requests per minute, because it is the easy number. The limit that bites a fan-out is tokens. The Claude API sets limits per model class across three dimensions — RPM, input tokens per minute, output tokens per minute — on a token bucket, so capacity is continuously replenished rather than reset at fixed intervals. And the docs spell out the consequence that catches fan-outs: a rate of 60 requests per minute might be enforced as 1 request per second, so a sixty-wide burst gets refused while your per-minute total is nowhere near the ceiling.
The counting rules decide your arithmetic. On Claude, input tokens per minute counts input_tokens plus cache_creation_input_tokens, and cache_read_input_tokens does not count — which makes prompt caching a rate-limit strategy and not merely a cost one, since fifty branches sharing a cached prefix charge it roughly once. Output tokens per minute is evaluated on tokens actually generated; max_tokens does not factor in. OpenAI goes the other way: the limit is the larger of max_tokens and the estimate from your request’s character count, with advice to set max_tokens as close to your expected response size as possible. A lazy max_tokens=4096 on fifty branches reserves two hundred thousand tokens per minute you will not use.
So admission control belongs in tokens, not request slots. Estimate the branch’s input cost before dispatch, charge it against a local budget, and correct that budget from headers you already receive: anthropic-ratelimit-input-tokens-remaining, rounded to the nearest thousand, or x-ratelimit-remaining-tokens. A fan-out that reads its own headers degrades; one that does not, fails.
# Admission control in TOKENS. Request slots are the wrong unit.
import time
class TokenBudget:
"""Local mirror of the provider's bucket, corrected by its own headers."""
def __init__(self, tpm: int):
self.tpm, self.avail, self.t = tpm, float(tpm), time.monotonic()
def _refill(self):
now = time.monotonic()
self.avail = min(self.tpm, self.avail + (now - self.t) * self.tpm / 60)
self.t = now # continuous, like a token bucket
async def admit(self, est: int):
while True:
self._refill()
if est <= self.avail:
self.avail -= est # charge BEFORE dispatch, not after
return
await asyncio.sleep((est - self.avail) * 60 / self.tpm)
def reconcile(self, headers):
"""Truth lives in the response, not in your estimate."""
for h in ("anthropic-ratelimit-input-tokens-remaining",
"x-ratelimit-remaining-tokens"):
if h in headers:
self.avail = min(self.avail, float(headers[h]))
self.t = time.monotonic()
def estimate(prompt: str, max_out: int) -> int:
# Claude ITPM: input_tokens + cache_creation_input_tokens.
# cache_read_input_tokens does NOT count -> a shared cached
# prefix across 50 branches is charged roughly once.
# OpenAI TPM: the LARGER of max_tokens and the char-count estimate, so a
# lazy max_tokens=4096 x 50 branches reserves ~200k TPM unused.
return len(prompt) // 4 + max_outRetry is how a rate limit becomes an outage
Fan-out and retry multiply, and the multiplication usually happens twice without anyone choosing it. The OpenAI SDK defaults to DEFAULT_MAX_RETRIES of 2, an initial delay of 0.5s and a cap of 8.0s; wrap that in your own retry loop and fifty branches become a hundred and fifty requests against a quota that just said it was full. The docs are blunt: if you run your own retry loop, disable SDK retries or count them in your limits, so retries do not multiply. They also note that failed requests still count toward your per-minute limits — the sentence that kills the instinct to resend harder.
Treat retry-after as a minimum and add a small random delay — without jitter, fifty branches refused in the same second retry in the same second, and you have built a thundering herd with extra steps. Read the error before retrying at all: a Claude spend-cap 429 carries no retry-after header and reports error.details.error_code of enforced_spend_limit_reached, so retrying can only fail. OpenAI’s slow_down 429 asks for the opposite of a retry — follow retry-after, reduce your rate, then increase gradually, no more than 50% every 15 minutes once you are at a million input tokens per minute.
The fan-out-specific rule falls out of that. A 429 is information about the pool, not about the branch that received it. The branch retries; the pool narrows. If one refusal does not shrink the concurrency limit for every sibling, you will spend the next minute discovering the same fact forty-nine more times.
Partial failure is a product decision, so name it in a type
Forty-seven of fifty branches succeeded. There are three things you can do and two are defensible. Fail the whole task, which is honest and sometimes required. Return forty-seven and say so. Or return forty-seven as though it were fifty — which nobody chooses and which ships constantly, because it is what you get when the result type has no room for a gap.
So give it room. A fan-out returns a value, the failures, and a coverage number, and every downstream consumer reads coverage before value. This matters doubly when the fan-in step is itself a model call: hand an LLM forty-seven summaries with no note of the three missing and it writes a confident synthesis of the corpus, because nothing in its context says the corpus is incomplete. That is silent partial success — the failure mode with no error anywhere in the trace — and the fix is one field.
Side effects need one more thing. A retried branch that writes — files a ticket, sends an email, charges a card — needs an idempotency key derived from the branch identity rather than the attempt, or your retry strategy is a duplicate-work generator. Read-only fan-out is cheap to retry; write fan-out is not, which is the real reason to keep them in separate stages.
Fan-in is where parallel work quietly becomes wrong
Every framework concedes the same point about shared state, which is a good sign it is load-bearing. Google’s ADK runs each sub-agent of a ParallelAgent in its own branch and documents that there is no automatic sharing of conversation history or state between these branches during execution: communication must be explicit, concurrent access to shared context needs careful handling such as locks, and the order of results may not be deterministic. LangGraph’s Send fans out one worker per item at runtime, the worker does not receive the full graph state, and a key several branches write needs a reducer declared on it — Annotated[list[str], operator.add] — or the updates conflict.
Non-determinism in the merge is the subtle cost. Concatenate in completion order and identical inputs produce a different answer every run, your evals stop being reproducible, and you spend a day blaming the model’s temperature for an ordering bug. Sort by a stable branch key before merging. It costs nothing and it is the last layer where you still control determinism.
And the rule from the topology post holds exactly: reads parallelize, writes do not. Fan out the retrieval, the summarisation, the independent checks; keep one writer composing the answer. A fan-out with a single merge point is a pattern you can reason about. A fan-out where every branch mutates shared state is a distributed system you did not mean to build and will not debug.
# Fan-in: a typed envelope, a deterministic order, one writer.
from dataclasses import dataclass, field
@dataclass
class FanOut:
value: list # what came back
failures: dict = field(default_factory=dict) # branch_key -> reason
@property
def coverage(self) -> float:
n = len(self.value) + len(self.failures)
return len(self.value) / n if n else 0.0
def merge(results: list[tuple[str, str]], failures: dict) -> FanOut:
# STABLE ORDER. Completion order makes identical inputs give different
# answers, and your evals stop being reproducible for no model reason.
return FanOut(value=[v for _, v in sorted(results, key=lambda r: r[0])],
failures=failures)
def merge_prompt(fo: FanOut) -> str:
missing = ", ".join(sorted(fo.failures)) or "none"
# Tell the merging model what is ABSENT. Omit this and it writes a
# confident synthesis of a corpus it never saw -> silent partial success.
return (f"Coverage: {fo.coverage:.0%}. Missing branches: {missing}.
"
f"Synthesise ONLY from the sections below; do not infer the gaps.
"
+ "
---
".join(fo.value))
# Side effects: key retries by BRANCH identity, never by attempt.
def idempotency_key(run_id: str, branch_key: str) -> str:
return f"{run_id}:{branch_key}" # not uuid4() per attempt
# ADK: no automatic state sharing between ParallelAgent branches; result order
# is not deterministic. LangGraph: a key two branches write needs a
# reducer -- Annotated[list[str], operator.add] -- or updates conflict.The default, and the interview answer
Default to a bounded pool whose width comes from your token limit rather than your core count, admission control in tokens, best-effort semantics with an explicit coverage field, retry-after plus jitter against one shared pool that narrows on every 429, and a deterministic merge in a single writer. Reach for TaskGroup when a missing branch genuinely invalidates the answer, and for quorum when you are buying redundancy rather than coverage.
The interview answer is three sentences. Fan-out is a capacity decision rather than a speed decision: a sequential loop is self-throttling, and parallel dispatch hands rate-limit management to the provider, who enforces it by refusing you — and the limit that binds is tokens per minute against a continuously replenishing bucket, not requests, which is why a sixty-wide burst gets refused inside a sixty-per-minute budget. Choose failure semantics before the primitive, since gather does not cancel siblings on the first exception and TaskGroup does, and make partial failure a typed result with a coverage field or the merging model will confidently synthesise a corpus it never received. And a 429 is news about the pool rather than the branch, so the branch retries with jitter while the pool narrows for everyone.
The one-line version for a design doc: parallelism is bought in tokens, and partial failure is a field, not an exception.
Work the same ideas as a full interview case study, with pressure questions and model answers.