System Design Lab
Real-time model serving is a throughput-vs-tail-latency problem: batching and GPUs buy capacity, but every gain spends part of your p99 budget.
Change request rate, per-request model latency, the batch window, how many replicas and how many models you host, and your p99 target. Toggle dynamic batching and GPU acceleration. The design moves from a single model endpoint to dynamic batching, GPU-backed autoscaling replicas, a model registry with canary rollout, and finally multi-model packing at high throughput.
Guided walkthrough
Reason about it one step at a time
Takeaway
Normal evolution scenarios
Click left to right for the intended demo path. Each card changes the workload inputs.
Recommended shape
Current architecture path
Clients
Client sends real-time feature vectors and waits for a prediction under a tight deadline
Gateway
Inference gateway routes each request to the right model and version and enforces timeouts
Traffic split sends a slice of traffic to a new version for canary and A/B comparison
Model servers
Dynamic batcher collects concurrent requests within a window into one padded batch
Server replicas parallel model-server instances the gateway load-balances across
Registry + GPU
Model registry stores versioned model artifacts and the mapping of names to deployed versions
GPU pool pooled GPUs the replicas schedule onto for high-throughput inference
Async
Metrics + latency collects per-model latency and accuracy metrics off the hot path
Shadow traffic mirrors live requests to candidate models for evaluation without serving their output
Bottlenecks
Serving capacity
p99 latency budget
Accelerator efficiency
Replica fleet pressure
Multi-model packing
Why this changes
Decision tradeoffs
Source-backed rules
These are the durable system-design claims behind the model. The exact slider thresholds are deliberately labeled as teaching assumptions.
Dynamic batching trades latency for accelerator throughput
Triton coalesces individual inference requests into a batch within a configurable delay so the GPU processes many requests per kernel launch, raising throughput at the cost of added queueing latency.
NVIDIA Triton Inference ServerA serving system should version models and serve them behind a stable name
TF Serving loads versioned model artifacts from a registry/model store and can serve multiple versions of a model so traffic can move between versions without redeploying the server.
TensorFlow ServingKServe provides canary rollout and autoscaling for inference
KServe exposes a model-serving control plane with traffic splitting for canary rollouts and request-driven autoscaling (scale to and from zero) so replicas track load.
KServeTail latency, not the mean, governs interactive serving SLAs
In fan-out and high-throughput systems the 99th-percentile response time dominates user-visible latency, so capacity and batching must be sized against p99 rather than average latency.
The Tail at Scale (Dean & Barroso)Teaching assumptions
- Per-replica capacity is a teaching approximation: CPU sustains a small fixed ops/s, a GPU replica clears several times more, and batching multiplies throughput only when traffic is concurrent enough to fill a batch. Batch gain rises with both the window and the model latency, because heavier models amortize more kernel-launch and memory-transfer overhead per batched call.
- p99 is modeled as model compute time plus the batch window plus a queueing penalty that grows as utilization approaches saturation; the penalty is divided by a square-root-staffing factor in the replica count to approximate M/M/c rather than M/M/1, so spreading the same load over more replicas genuinely lowers the tail. Real tails also depend on GC, padding, and cold starts.
- Hosting many models is treated as needing either more replicas or model packing once distinct models exceed what one replica can hot-load without thrashing.