System Design Lab
A web crawler is throughput-bound at the fetcher and memory-bound at the "seen" set, with politeness capping how fast you may go.
Change the target crawl rate, how many pages you intend to fetch, the fetcher worker pool, average page size, how many distinct domains you span, and the dedup tolerance. The design moves from a single-threaded loop to a worker pool, a politeness-aware scheduler, a Bloom-filter "seen" set, and a partitioned frontier with sharded content storage.
Guided walkthrough
Reason about it one step at a time
Takeaway
Normal evolution scenarios
Click left to right for the intended demo path. Each card changes the workload inputs.
Recommended shape
Current architecture path
Seeds / Frontier
Seed URLs starting URLs that prime the crawl before discovered links take over
URL frontier holds URLs waiting to be fetched, ordered by priority and per-domain queues
Scheduler
Scheduler picks the next fetchable URL while honoring robots.txt and per-domain rate limits
DNS cache caches host-to-IP lookups so resolution does not become the hidden bottleneck
Fetchers
Fetcher pool concurrent I/O-bound workers that download pages from the open web
Parser / dedup
Parser parses fetched pages and extracts outgoing links to feed back to the frontier
Seen set rejects already-seen URLs and near-duplicate content, often via a Bloom filter at scale
Content store
Content store durable store of downloaded page bodies for indexing and analysis
Store shards spreads page bodies across nodes once one store can no longer hold the corpus
Bottlenecks
Fetch throughput vs target
Frontier / queue backlog
Seen-set memory
DNS lookup pressure
Content storage
Why this changes
Decision tradeoffs
Source-backed rules
These are the durable system-design claims behind the model. The exact slider thresholds are deliberately labeled as teaching assumptions.
A crawler must enforce politeness with per-host rate limits and robots.txt
The URL frontier must avoid hammering any one host; per-domain queues and back-off enforce a polite delay between requests to the same server.
Manning & Schütze, IR (Stanford NLP)DNS resolution is a hidden bottleneck and must be cached
DNS lookups are a well-known bottleneck for a high-throughput crawler, so a custom cache (and asynchronous resolution) is standard practice.
Manning & Schütze, IR (Stanford NLP)A Bloom filter tests set membership in tiny space at the cost of false positives
Bloom filters answer "have I seen this URL?" using a few bits per element and never produce false negatives, making them the standard seen-set at billions of URLs.
Bloom, CACM 1970A scalable crawler partitions the frontier and seen-set across machines
The "Web Crawling" survey describes partitioning the URL frontier and duplicate-detection state across machines so a crawler can scale toward the whole web.
Olston & Najork (Google Research)Teaching assumptions
- A fetcher worker is modeled as I/O bound; throughput scales with worker count until politeness or DNS caps it.
- Single-node throughput, DNS, memory, and storage budgets are conservative teaching numbers, not vendor limits.
- Seen-set memory uses ~64 bytes/URL for an exact hash set and ~1.5 bytes/URL for a Bloom filter at the chosen tolerance.