System Design Lab
A recommendation system is a funnel: cheap retrieval narrows millions of items to hundreds, then an expensive ranker scores what survives.
Change users, catalog size, request rate, candidates retrieved per request, ranking-model latency, embedding dimensions, and feature count. The design moves from a popularity list to collaborative filtering, then two-tower embeddings with ANN retrieval, then a ranking model backed by a feature store, and finally real-time features and full multi-stage serving at scale.
Guided walkthrough
Reason about it one step at a time
Takeaway
Normal evolution scenarios
Click left to right for the intended demo path. Each card changes the workload inputs.
Recommended shape
Current architecture path
Clients
Client requests a ranked list and emits clicks, watches, and other interactions
Serving API
Serving API fans a request through retrieval, ranking, and re-ranking under a latency budget
Re-ranker applies business rules, diversity, and freshness to the ranked list before returning it
Retrieval
Two-tower model embeds the user query so similar item vectors can be found by distance
ANN search finds approximate nearest item vectors over millions of items in milliseconds
Ranking
Ranking model scores each candidate with a heavy model using many user-item features
Feature store serves precomputed and live features for each user-item pair at low latency
Stores
Item index holds item embeddings and metadata, sharded as the catalog grows
Feature DB durable store of batch and streaming features behind the online feature store
Training loop
Interaction log captures impressions and engagement off the hot path for training and live features
Training jobs rebuilds embeddings and the ranking model from logged interactions
Bottlenecks
Retrieval scan cost
Ranking compute
Feature-store load
Item index memory
End-to-end latency budget
Why this changes
Decision tradeoffs
Source-backed rules
These are the durable system-design claims behind the model. The exact slider thresholds are deliberately labeled as teaching assumptions.
Sampling-Bias-Corrected Neural Modeling (Yi et al., 2019)
Separate user and item towers produce embeddings whose dot product approximates relevance, so retrieval becomes a nearest-neighbor search over precomputed item vectors.
Google ResearchANN search makes nearest-neighbor retrieval scale to billions of vectors
Approximate nearest neighbor libraries (ScaNN, FAISS) and vector databases trade a little recall for orders-of-magnitude faster similarity search, which is what lets retrieval scan a huge catalog in milliseconds.
ScaNN (Google Research)A feature store serves the same features online and offline
A feature store provides low-latency online feature lookups for serving and consistent historical features for training, avoiding training/serving skew.
FeastLarge-scale recommendation is staged: candidate generation then ranking
The canonical pattern narrows a huge item set with a cheap retrieval stage before a more expensive ranking stage scores the survivors, so heavy compute only touches a few hundred items.
System Design PrimerTeaching assumptions
- Retrieval cost is modeled as a per-request scan that ANN turns from linear in catalog size into roughly logarithmic, so brute force only survives on small catalogs.
- Ranking throughput per server scales down with model latency and feature count; the numbers are conservative teaching budgets, not vendor benchmarks.
- Feature-store load counts one lookup per feature per candidate per request, the worst case before batching and caching.