System Design: OpenAI/Gemini Platform

Version one: a web server receives POST /v1/chat/completions, loads the model onto a GPU, runs generation, returns the text. It demos beautifully and fails immediately: loading tens of gigabytes of weights per request takes minutes, one GPU serves one request at a time, and there is no auth, no safety filter, no metering — three things a public LLM API cannot ship without.

The first fix is warm pools: inference servers that load weights once at startup and stay resident, fronted by a gateway that authenticates, applies per-org rate limits, and streams tokens back over SSE as they generate — because a user watching a 30-second generation needs the first token in well under a second, not the last.

A real request path has stages: gateway → safety policy (prompt-level abuse and jailbreak screening) → request router → dispatcher. The dispatcher is the interesting box — it knows which model versions live on which GPU clusters, picks a target by requested model and current load, and keeps warm pools sized per model tier so a burst on the small fast model never evicts the large one.

This tiering is also the latency story: small models answer in well under ~300ms and absorb most traffic; large-context requests route to bigger clusters with a ~1s budget. A telemetry bus taps every hop — latency, token counts, errors — because at millions of requests/day you debug from aggregates, not logs.

Training shares nothing with serving except the artifact it produces. The pipeline is a batch factory: a data lake feeds preprocessing (dedup, filtering, tokenization), which feeds a distributed training run — on the order of ~2K accelerators for weeks on a trillion-token corpus, orchestrated by Kubernetes or Slurm over a high-bandwidth interconnect fabric.

At that scale, hardware failure is a schedule item, not an exception: with thousands of GPUs running for weeks, node failures arrive routinely. Checkpointing the full training state to durable storage on a regular cadence is what converts a dead node from 'restart a multi-million-dollar run' into 'lose the minutes since the last checkpoint'. Checkpoint frequency is itself a trade-off — snapshot time steals training throughput.

The model registry is the only door between the planes: training publishes versioned weights with eval scores attached, and serving deploys exclusively from it. New versions roll out as canaries — a small slice of traffic on the new weights while automated checks compare quality metrics, latency, and refusal rates against the incumbent — with automated rollback wired to the same signals.

This is where fine-tuning slots in without a new architecture: POST /v1/fine-tunes runs a small training job against an org's uploaded files, and the resulting adapter weights land in the same registry, deployed through the same canary gate, scoped to that tenant. One promotion path for every model that ever serves traffic.

Every request already emits token counts onto the telemetry bus; a usage pipeline aggregates them into an OLAP store (~1 KB/request, on the order of a GB/day of logs) that powers GET /v1/usage and per-token billing. Metering must survive cancelled streams — bill the tokens actually generated — which is why it hangs off telemetry emitted during generation, not a post-hoc reconciliation.

The admin console closes the loop for humans: rollout controls, per-org quotas and key management, safety overrides, and incident response. Enterprise requirements — tenant isolation, regional data residency, retention policies — show up here as routing policy (pin an org's traffic to a region's pools) rather than as new machinery. The final diagram is the lane architecture: gateway → safety → router → dispatcher → pools, with training → registry → deployment feeding it from behind.