System Design Arena
Design a Web Crawler
Frontier scheduling, politeness, dedup at billions-of-URLs scale, and trap avoidance — the classic pipeline-design interview.
Case Study
Web Crawler
Same conversation, diagrams, and wrap-up you expect—now framed with clearer scaffolding and iconography.
Prompt
Design a web crawler that downloads billions of pages for a search index, refreshing them on a schedule, while staying polite to every host.
Interview snapshot
- • Topic: Web Crawler
- • Expected depth: 45 - 60 minutes
- • Focus areas: APIs, scale estimation, resilient architecture
- • Wrap-up: risk, monitoring, disaster recovery
Key takeaways
- • Fetch pages from seed URLs, extract links, discover the web graph.
- • Throughput: ~10K pages/sec sustained to cover 10B pages in ~2 weeks.
- • URL Frontier: priority front-queues + per-host back-queues + timing wheel.
🎙️ Interview mode
Practice this like a real interview
Don't read the answer first. Work the framework against the bare prompt, then compare against what the interviewer expected at each step.
Scorecard
What the interviewer is evaluating
Staff-level signal
Does the candidate see that the URL frontier must satisfy priority AND per-host politeness simultaneously — and structure it as two levels?
🧠 Staff engineer judgment
Politeness is a hard constraint, not a nice-to-have — one crawler that hammers a small site is a production incident and a reputation problem. Design the frontier around it.
Calibration
The common (bad) answer
Queue of URLs → workers fetch → push found links back
❌ Why this scores poorly: One global queue means whichever host is popular gets hammered (impolite), important pages wait behind junk (no priority), and the same URL enqueues forever (no dedup).
✓ What a strong answer adds
- Two-level frontier: priority front-queues → per-host back-queues + timing wheel.
- Seen-URL set (Bloom + exact store) checked before enqueue.
- Robots.txt cached per host; per-host budgets against traps.
- Durable inter-stage queues: crash = resume, not restart.
- Refresh scheduling by observed change rate × importance.
Video walkthrough
Design a Web Crawler — System Design Interview
Hello Interview · ex-Meta staff engineer · 1h 5m
Politeness, dedup and the frontier queue, with the scale math done on screen.
Work the prompt yourself first — guided practice grades your own answer step by step. Watching someone else design it feels like progress and teaches you far less.
Interview flow
Dialogue timeline
Interviewer
What does success look like?
Candidate
Fetch billions of pages, extract links to discover more, refresh pages on a freshness schedule, and never overload any single website — politeness is a hard constraint, not a courtesy.
Interviewer
Sketch the pipeline.
Candidate
A loop: URL frontier → fetcher → parser → dedup → frontier again. Each stage scales independently; the frontier — the prioritized queue of what to fetch next — is the heart of the design.
Interviewer
Why is the frontier hard? It's just a queue.
Candidate
Because it must satisfy two orthogonal orders: priority (important/stale pages first) and politeness (≤1 concurrent request per host, with per-host delay). A single global queue can't do both.
Interviewer
So how do you structure it?
Candidate
Two-level: priority front-queues feed per-host back-queues. A host is assigned to one back-queue; a timing wheel releases each host only after its politeness delay. Fetchers pull whichever host is ready — priority within, rate-limits across.
Interviewer
How do you avoid fetching the same URL twice at 10B scale?
Candidate
Normalize the URL, hash it, check a seen-set. 10B×8-byte hashes = 80GB — a sharded hash store, or a Bloom filter front (a few GB) with the exact store behind it. False positives just skip a page: acceptable.
Interviewer
What about content dedup — mirrors and boilerplate?
Candidate
SimHash/shingling on parsed text: near-duplicate pages collapse to one canonical. That's a separate check from URL dedup and saves the index, not the fetch.
Interviewer
Crawler traps: calendars generating infinite URLs.
Candidate
Budgets per host — max pages, max depth; URL-pattern heuristics; and diminishing returns detection: if new-content rate from a host collapses, back off. Robots.txt cached per host with TTL is table stakes.
Interviewer
How does refresh work — you can't recrawl 10B pages daily.
Candidate
Per-page revisit schedules driven by observed change rate and importance: news hourly, static docs monthly. The frontier's priority score blends freshness debt with page rank — crawl budget goes where change happens.
Scoping
Requirements & trade-offs
Functional Requirements
- —Fetch pages from seed URLs, extract links, discover the web graph.
- —Respect robots.txt and per-host rate limits — always.
- —Refresh pages by importance and observed change rate.
- —Out of scope: indexing/ranking, JS rendering (mention as extension).
Non-Functional Requirements
- —Throughput: ~10K pages/sec sustained to cover 10B pages in ~2 weeks.
- —Politeness: never more than 1 in-flight request per host, configurable delay.
- —Restartable: a crashed crawl resumes, never restarts.
- —Storage-efficient dedup at 10B-URL scale.
Blueprint
Architecture modules
Module 1
Components
- •URL Frontier: priority front-queues + per-host back-queues + timing wheel.
- •Fetcher fleet: async HTTP, DNS cache, robots.txt cache.
- •Parser: link extraction, canonicalization, SimHash for near-dupes.
- •Seen-URL store: Bloom filter + sharded exact store; Content store: WARC-style blobs.
Module 2
Back-of-the-Envelope
- •10B pages / 14 days ≈ 8.3K fetches/sec sustained.
- •Average page 100KB → ~830MB/sec ingest ≈ 1PB per crawl cycle.
- •10K/sec at ~1 sec/fetch → ~10K concurrent connections → ~50 async fetcher nodes.
- •Frontier holds ~1B queued URLs → disk-backed queues, RAM heads only.
Module 3
System Diagram Notes
- •Frontier (partitioned by host-hash) → Fetchers → Parsers → back to Frontier.
- •Each stage connected by durable queues — crash-restartable by design.
- •Seen-set consulted before enqueue, not after fetch.
- •Metrics loop: per-host success/change rates tune priorities.
Module 4
Design Playbook
- •Draw the loop first; name the frontier as the interesting component.
- •Politeness × priority as two orthogonal orderings is THE insight interviewers grade.
- •Do the dedup math out loud — 80GB of hashes sounds big until you shard it.
- •Traps and budgets show production scar tissue.
- •Close with refresh scheduling: crawl budget follows change rate.