System Design Lab
A feature store is two stores wearing one schema: the hard part is computing the same feature identically for training and serving.
Change serving lookups, feature and entity counts, the online latency target, training rows, and streaming freshness, and toggle streaming features and point-in-time correctness. The design evolves from features computed inline, to an offline batch store, to an online/offline split with parity, to point-in-time training joins, and finally to streaming materialization at scale.
Guided walkthrough
Reason about it one step at a time
Takeaway
Normal evolution scenarios
Click left to right for the intended demo path. Each card changes the workload inputs.
Recommended shape
Current architecture path
Models
Serving model asks for a feature vector by entity key on every prediction
Training job requests historical features joined to labels to train models
Serving + registry
Serving API fetches feature vectors at low latency for online inference
Feature registry single source of truth for feature definitions and transform logic
Online store
Online store serves the latest feature values keyed by entity in single-digit ms
Materialization
Batch job recomputes features on a schedule and writes them to both stores
Stream job updates features from an event stream for seconds-level freshness
Offline
Offline store durable historical feature values for training and backfills
Point-in-time join joins each label to feature values as of its timestamp to stop leakage
Bottlenecks
Online lookup load
Online latency pressure
Online store size
Freshness vs lag budget
Training join cost
Why this changes
Decision tradeoffs
Source-backed rules
These are the durable system-design claims behind the model. The exact slider thresholds are deliberately labeled as teaching assumptions.
A feature store splits a low-latency online store from a scalable offline store
Feast materializes features into an online store for serving and keeps an offline store for historical retrieval, so the same feature definitions back both paths.
Feast DocsOnline/offline parity is the core problem a feature store solves
Michelangelo computes features once and shares them across training and serving so the model sees the same values in both, avoiding training/serving skew.
Uber MichelangeloTraining joins must be point-in-time correct to avoid label leakage
Features must be joined to labels as of the prediction timestamp; using values from after the label leaks future information and inflates offline metrics.
Feast Point-in-Time JoinsIn-memory key-value stores serve online features at sub-millisecond latency
Point lookups by entity key from an in-memory store meet tight serving budgets that a warehouse scan cannot.
Redis DocsTeaching assumptions
- Online lookups are modeled as point reads of one feature row per entity key; each fetch returns the full feature vector.
- Single-node read, latency, storage, and join budgets are conservative teaching numbers, not vendor limits.
- Online store size approximates entities x features x ~32 bytes; real systems add TTLs, indexes, and replication overhead.