System Design Arena
System Design: Retrieval-Augmented Generation (RAG) Pipeline at Scale
A 2026-standard AI system design interview question: design the ingestion, indexing, and query-time retrieval pipeline that grounds an LLM's answers in a company's private documents, with millions of chunks and sub-second latency.
Case Study
RAG Pipeline at Scale
Same conversation, diagrams, and wrap-up you expect—now framed with clearer scaffolding and iconography.
Prompt
Design a RAG system that lets an LLM answer questions grounded in a company's internal documents (docs, tickets, wikis) — millions of documents, updated continuously, with citations and sub-second retrieval at query time.
Interview snapshot
- • Topic: RAG Pipeline at Scale
- • Expected depth: 45 - 60 minutes
- • Focus areas: APIs, scale estimation, resilient architecture
- • Wrap-up: risk, monitoring, disaster recovery
Key takeaways
- • Ingest heterogeneous documents (PDF, HTML, Markdown, tickets) continuously, not just in a nightly batch.
- • Query-time retrieval latency under ~200 ms at the 99th percentile, out of an overall answer budget of a few seconds.
- • POST /api/documents - registers a new/updated document, enqueues it for parsing and chunking.
🎙️ Interview mode
Practice this like a real interview
Don't read the answer first. Work the framework against the bare prompt, then compare against what the interviewer expected at each step.
Scorecard
What the interviewer is evaluating
Staff-level signal
Does the candidate treat retrieval quality as the product (chunking, hybrid search, reranking, evals) instead of assuming "vector DB = done"?
🧠 Staff engineer judgment
Do not fine-tune the model to fix retrieval problems. If the right chunk is not in the context, no amount of prompting rescues the answer — fix search first.
Calibration
The common (bad) answer
Docs → embeddings → vector DB Query → similar chunks → LLM → answer
❌ Why this scores poorly: The tutorial diagram. It ships a demo, then fails in production on chunking, stale indexes, permission leaks, and no way to measure whether answers got better or worse.
✓ What a strong answer adds
- Chunking strategy by document structure, with overlap — the highest-leverage knob.
- Hybrid retrieval (BM25 + vectors) with a reranker; pure vectors miss exact terms.
- Incremental ingestion with per-document versioning — no full reindex per edit.
- Permission-aware retrieval: filter at query time by the caller’s ACLs, before the LLM sees anything.
- Retrieval evals (golden questions) so changes are measured, not vibed.
Build it up
Step-by-Step Walkthrough
RAG looks like “just call the vector DB” until you draw it. The interview is won in the two pipelines around the lookup: how documents get in, and what happens between retrieval and the final answer.
The naive loop: stuff context into the prompt
User asks a question; you embed it, cosine-search a vector store, paste the top chunks into the prompt, and let the LLM answer. This is the demo everyone builds in an afternoon.
It works until documents update, chunks come back irrelevant, or the model answers from the wrong tenant's data. Each failure points at a missing subsystem.
Treat ingestion as a real pipeline
Documents need parsing, chunking (semantic boundaries beat fixed windows), embedding, and upserting — with versioning so an updated document replaces its stale chunks instead of coexisting with them.
Run it async off a queue: ingestion latency is invisible to users, but ingestion correctness is everything to answer quality.
?The question that filters seniors
“What happens when a document is edited?” — if chunk IDs are content-addressed and carry a doc version, you delete-by-version then upsert. Without that, your index accumulates ghosts forever.
Retrieval is a ranking problem, not a lookup
Pure vector similarity misses exact identifiers and rare terms; BM25 misses paraphrases. Production systems run hybrid retrieval and then a cross-encoder reranker over the merged candidates.
Filter by ACL and tenant before ranking, not after — trimming post-hoc leaks existence information and wrecks your top-k.
Close the loop: grounding, caching, evals
Attach citations to every claim so answers are auditable; cache full answers keyed on (query embedding, corpus version) for repeat questions; and run an offline eval set through the pipeline on every change to chunking, embeddings, or prompts.
The eval harness is what makes this a system instead of a demo — without it you cannot tell whether yesterday's “improvement” silently degraded answers.
!Do not skip the failure mode
Say what happens when retrieval returns nothing relevant: the system should say “I don't know” with a confidence threshold, not hallucinate. Naming the abstention path is worth more than another box on the diagram.
Blueprint
Architecture Diagram
The narrated components above, laid out as an actual flow — so you can see how a request moves through the system, not just read a list of pieces.
Core Architecture
Ingestion pipeline
parse -> chunk (overlapping windows) -> embed -> upsert into vector index, all queue-driven so a slow parse doesn't block the rest of the batch.
Vector index
ANN structure (HNSW/IVF-PQ) sharded by tenant or document collection, with metadata filtering (permissions, freshness) applied at query time.
Hybrid retrieval
vector search + BM25 keyword search run in parallel, results merged and passed through a cross-encoder re-ranker for precision.
Generation
top-k re-ranked chunks assembled into a grounded prompt with citation markers; a lightweight groundedness/hallucination check runs on the output before it's returned.
Freshness
change-data-capture or webhook from the source system triggers re-embedding of just the changed document, not a full re-index.
Flow Diagram
How the API maps to this flow
- 1
POST /api/documents— registers a new/updated document, enqueues it for parsing and chunking. - 2
POST /api/query { question, filters? }— runs hybrid retrieval + re-rank + generation, returns answer with citations. - 3
DELETE /api/documents/{id}— tombstones a document so its chunks are excluded from future retrieval and eventually purged. - 4
GET /api/query/{id}/feedback— captures thumbs up/down for retrieval quality monitoring and future re-ranker training.
?But why does this actually hold up at scale?
10M documents x ~8 chunks/doc average = 80M vectors; at 768 dimensions in float32, raw vectors alone are ~245 GB — product-quantization compression is close to mandatory at this scale.
Video walkthrough
How to Build a Scalable RAG System for AI Apps
ByteMonk · 15 min
Ingestion, chunking, embedding storage and query-time retrieval as one production pipeline.
Work the prompt yourself first — guided practice grades your own answer step by step. Watching someone else design it feels like progress and teaches you far less.
Interview flow
Dialogue timeline
Interviewer
Why not just fine-tune the model on the company's documents instead of retrieval?
Candidate
Fine-tuning bakes knowledge into weights — it's slow to update, doesn't cite sources, and the model can still hallucinate confidently. RAG keeps knowledge in an external, freshly-updatable index and forces the model to ground its answer in retrieved passages it can cite, which is what compliance and trust require here.
Interviewer
Walk me through the ingestion pipeline.
Candidate
Documents land in a queue, get parsed (PDF/HTML/markdown to plain text), chunked into overlapping windows (roughly 300-500 tokens with 10-15% overlap so answers spanning a chunk boundary aren't lost), embedded with a text-embedding model, and upserted into a vector index alongside metadata — source, permissions, last-updated timestamp.
Interviewer
How do you serve a query in under a second against millions of chunks?
Candidate
Approximate nearest-neighbor search (HNSW or IVF-PQ) instead of brute-force cosine similarity — sub-linear lookup at the cost of a small recall hit. I'd also do hybrid retrieval: combine the ANN vector search with a keyword/BM25 pass and re-rank the merged candidates, because pure embedding search misses exact-match terms like error codes or product SKUs.
Interviewer
What happens after retrieval — how do you stop the model from hallucinating anyway?
Candidate
Pass only the top re-ranked chunks into the prompt with explicit citation markers, instruct the model to answer only from provided context and say 'I don't know' otherwise, and — for anything user-facing — run a cheap groundedness check that verifies each claim in the answer actually appears in a cited chunk before returning it.
Scoping
Requirements & trade-offs
Functional Requirements
- —Ingest heterogeneous documents (PDF, HTML, Markdown, tickets) continuously, not just in a nightly batch.
- —Answer natural-language questions grounded in retrieved passages, with inline citations back to source documents.
- —Respect source-level permissions — a user should never see retrieval results from documents they can't access.
- —Support incremental re-indexing when a source document is edited or deleted (no stale answers).
Non-Functional Requirements
- —Query-time retrieval latency under ~200 ms at the 99th percentile, out of an overall answer budget of a few seconds.
- —Index scales to tens of millions of chunks without a full rebuild for every update.
- —Recall is tunable — the product can trade a little accuracy for a lot of speed at peak load.
- —Groundedness: the system should be able to say 'I don't know' rather than hallucinate when retrieval comes up empty.
Blueprint
Architecture modules
Module 1
API Endpoints
- •POST /api/documents - registers a new/updated document, enqueues it for parsing and chunking.
- •POST /api/query { question, filters? } - runs hybrid retrieval + re-rank + generation, returns answer with citations.
- •DELETE /api/documents/{id} - tombstones a document so its chunks are excluded from future retrieval and eventually purged.
- •GET /api/query/{id}/feedback - captures thumbs up/down for retrieval quality monitoring and future re-ranker training.
Module 2
Core Architecture
- •Ingestion pipeline: parse -> chunk (overlapping windows) -> embed -> upsert into vector index, all queue-driven so a slow parse doesn't block the rest of the batch.
- •Vector index: ANN structure (HNSW/IVF-PQ) sharded by tenant or document collection, with metadata filtering (permissions, freshness) applied at query time.
- •Hybrid retrieval: vector search + BM25 keyword search run in parallel, results merged and passed through a cross-encoder re-ranker for precision.
- •Generation: top-k re-ranked chunks assembled into a grounded prompt with citation markers; a lightweight groundedness/hallucination check runs on the output before it's returned.
- •Freshness: change-data-capture or webhook from the source system triggers re-embedding of just the changed document, not a full re-index.
Module 3
Back-of-the-Envelope
- •10M documents x ~8 chunks/doc average = 80M vectors; at 768 dimensions in float32, raw vectors alone are ~245 GB — product-quantization compression is close to mandatory at this scale.
- •100 queries/sec x (vector search + BM25 + re-rank of ~50 candidates) — re-ranking is usually the latency bottleneck, not the ANN lookup itself.
- •Ingestion: 50K new/updated docs/day x 8 chunks x 1 embedding call ~ 400K embedding calls/day, batched to amortize embedding-model throughput.
- •Permission filtering at query time on a sharded index needs the shard key to include tenant/ACL group, or every query fans out to every shard.
Module 4
Failure Modes & Guardrails
- •Stale index: a document is deleted in the source system but its chunks still serve retrieval results — tombstone immediately, purge asynchronously, never the reverse order.
- •Permission leakage: filtering ACLs in application code after retrieval, instead of in the index query itself, risks a document's content leaking into a re-ranker prompt even if it's stripped from the final answer.
- •Chunking too small loses context (answers span chunk boundaries); too large dilutes embedding relevance and blows the prompt budget — this is a real tuning knob, not a one-time decision.
- •Silent hallucination: without a groundedness check, a fluent answer with no real citation support looks identical to a correct one — this is the failure mode that erodes user trust fastest.
Module 5
Design Playbook
- •Open by asking about data freshness requirements and permission model — both reshape the architecture more than the vector database choice does.
- •Name the ANN algorithm and its recall/latency tradeoff explicitly (HNSW vs. IVF-PQ) rather than saying 'use a vector database' and moving on.
- •Bring up hybrid (vector + keyword) retrieval unprompted — pure embedding search missing exact-match terms is a well-known real-world failure interviewers listen for.
- •Discuss groundedness/citation verification as a first-class pipeline stage, not an afterthought bolted onto the prompt.
- •Close with monitoring: retrieval recall on a golden query set, citation-accuracy spot checks, and query latency percentiles — how do you know the system is still working next month?