System Design: Multi-Agent AI Orchestration Platform (like LangGraph / AutoGPT swarms)

Version one: receive the goal, loop — call the LLM, parse a tool call from the response, execute it, append the result to the context, repeat until the model says done. It works for five-minute demos and fails in every dimension at once: the server restarts and the run evaporates; the context window fills on long tasks; the model loops on a failing tool forever; and one run can burn an unbounded number of tokens.

Name the root cause out loud: all state lives in one process's memory and one model's context window, and the only judge of 'am I done, am I stuck, have I spent too much' is the agent itself. Everything that follows moves state and control outside the loop.

Split thinking from doing. A planner agent turns the goal into a DAG of subtasks — nodes with explicit dependencies and a declared tool profile each — persisted in a durable store (a Postgres task-graph table). An orchestrator walks the DAG, dispatching ready subtasks to worker agents through a queue: each worker is a narrow, stateless LLM call scoped to one subtask and one tool allowlist.

Decomposition buys three things worth naming: specialization (a cheap model for summarizing, a strong one for code), bounded context (each worker sees only its subtask, not the whole trajectory), and resumability — if step 4 of 9 fails, you retry step 4 from the persisted graph, not the whole run.

Every worker still runs a tool-call loop, but each call now passes through middleware the agent cannot reason its way around: a budget gate decrementing shared counters (max tool calls per agent, max tokens and wall-clock per run) that kills the run with a partial result when exhausted; an allowlist check so the research agent physically cannot execute code; and an approval gate that pauses the run for a human before high-risk actions — sending email, spending money, deleting data.

Tool outputs are attack surface, not just data: a scraped web page can carry prompt-injection text aimed at your agent. Filter and sanitize tool results — and redact PII — before they re-enter any prompt. The design principle to state plainly: guardrails live outside the agent's judgment, in code, because the component you are guarding against is the decision-maker itself.

Agents never share raw context windows. Workers write structured artifacts — a summary, a JSON result, a citation list — into the task graph, and downstream agents read those. Prompts stay small, every intermediate result is auditable, and a run-scoped vector store adds semantic memory for what does not fit in the graph, with long-term memory persisting learnings across runs.

Before the orchestrator marks a subtask complete, a supervisor/critic agent reviews the artifact against the subtask's spec — catching hallucinated citations, off-scope actions, and format drift. Cheap insurance: one extra small-model call per subtask, versus a final deliverable assembled from unchecked intermediate garbage.

At ~10K concurrent runs averaging ~8 subtasks and ~3 tool calls each, you are juggling on the order of ~240K LLM calls in flight — synchronous fan-out is dead on arrival. The orchestrator becomes a durable-execution engine (Temporal-style, or a queue-backed state machine): every transition persisted, workers stateless and horizontally scaled, per-tenant queue fairness so one enormous run cannot starve everyone else.

Users watch progress over an SSE stream — plan created, subtask 3/8 running, tokens spent — which doubles as the approval-gate UI. Every model and tool call is logged with inputs and outputs, making any run replayable step by step. Durable graph, guarded loop, artifact memory, full audit: that is the closing summary.