Design a Web Crawler

Design a web crawler that downloads billions of pages for a search index, refreshing them on a schedule, while staying polite to every host.

What clarifying questions would you ask the interviewer?