System Design: Windsurf / Codex

Version one is an IDE plugin that sends the active file to a large LLM on every pause in typing and inlines whatever comes back. It produces impressive demos and three immediate failures: round-trips to a large model take seconds against a ~200ms feel-threshold; the model suggests functions that do not exist because it cannot see the rest of the repo; and at ~2M daily developers triggering ~200 completions each — on the order of 400M requests/day — the GPU bill is ruinous.

Each failure names a subsystem: a latency budget forces model tiers, missing context forces repo indexing, and cost forces caching and ranking. The rest of the design is those three, in order.

Completion quality is mostly retrieval quality. A repo indexer parses the codebase into an AST and symbol graph and embeds code chunks into a sharded vector index — this runs per commit, off the hot path. Per keystroke, the plugin assembles a cheap local bundle (active file, cursor window, recent edits, open buffers, git diff) and ships it to a context service that merges in retrieved snippets: the callee's signature from another file, the matching test, a similar usage elsewhere.

A prompt builder ranks all of it into the model's context window under a hard token budget — nearest-scope first, retrieval filling the remainder. The split to narrate: indexing is expensive and asynchronous, assembly is cheap and synchronous, and the ~200ms budget is spent almost entirely on assembly plus inference.

One model cannot serve both a ghost-text suggestion (needs <200ms) and 'explain this module' (can take ~1s). Split the fleet: a small distilled model handles inline completions from warm pools with streaming output, a large model serves chat and multi-file edits, and a compact on-device model provides last-mile suggestions when the network disappears — the resilience answer interviewers ask for by name.

Caching attacks the volume: identical context prefixes recur constantly (every developer in a repo shares its skeleton), so completion caching on context hashes and server-side prefix reuse cut both latency and GPU cost. At ~20K requests/s peak, every cached hit is a GPU-second saved.

Users grade every suggestion for free: accepted, rejected, accepted-then-immediately-edited. That telemetry (~400M events/day, ~5K/s) flows into a pipeline feeding a ranking service — generate a handful of candidates, score them with a lightweight learned ranker on context features and historical acceptance, and show the winner. Acceptance rate becomes the platform's north-star metric, and ranker updates ship through offline eval against logged sessions before touching production.

This loop is what separates a product from an API wrapper: the model may be a commodity, but the ranking signal — which suggestions this codebase's developers actually keep — compounds and is yours alone.

Source code is the crown jewels, so a policy engine wraps the whole flow: secret scanning redacts keys and tokens from context bundles before they leave the IDE, tenant isolation keeps one company's embeddings out of another's retrieval, context bundles are stored ephemerally (~24h) with zero-retention contracts for enterprises, and every access lands in an audit log.

Close on the converged shape — plugin → context service → prompt builder → tiered inference, with the indexer feeding retrieval, telemetry feeding ranking, and policy wrapped around everything. Then name the failure mode you designed away: cloud unreachable degrades to the local model behind a circuit breaker. Never silence, just smaller suggestions.