System Design Lab

LLM inference is autoregressive: the KV cache grows per token and dominates GPU memory, so batching and memory layout decide throughput.

Change request rate, prompt and output lengths, model size, GPU count, and the time-to-first-token target. Toggle continuous batching with paged attention and tensor parallelism. The design moves from one GPU serving one request at a time, to a KV cache with batching, to continuous batching plus paged attention, to tensor-parallel sharding for big models, and finally prefill/decode disaggregation at scale.

Read the full tutorial

Guided walkthrough

Reason about it one step at a time

Normal evolution scenarios

Click left to right for the intended demo path. Each card changes the workload inputs.

Workload

These are inputs, not preset architecture stages.

Request rate New generation requests arriving per second. Each runs autoregressively until its output is complete. Avg prompt tokens Input length processed in the prefill phase; sets the starting size of each KV cache. Avg output tokens Tokens generated one at a time in the decode phase; the KV cache grows for each. Model size Parameter count. Weights at 16-bit take ~2 bytes each and must fit in GPU memory before any KV cache. GPU count GPUs available to serve the model. More GPUs add memory and compute, via replicas or tensor-parallel shards. p99 time-to-first-token Latency budget from request arrival to the first streamed token; the prefill phase must fit inside it. Continuous batching + paged attention In-flight batching admits and retires requests every step; paged attention stores the KV cache in fixed pages to avoid fragmentation. Tensor-parallel sharding Split each layer across GPUs so a model larger than one GPU can run, at the cost of per-step collective communication.

Recommended shape

Current architecture path

Clients

Client sends a prompt and reads generated tokens as they stream back

Gateway

Inference gateway authenticates, queues requests, and streams tokens back over the connection

Scheduler

Scheduler admits and retires requests every decode step to keep the GPU busy

Prefill pool runs the compute-heavy prefill so long prompts do not block decode

GPU workers

GPU worker generates tokens autoregressively for the current batch

Paged KV cache stores attention keys and values in fixed pages to avoid fragmentation

Model + async

Model shards splits each layer across GPUs so a large model fits and runs

Metrics stream collects throughput and latency off the hot path for autoscaling

Bottlenecks

KV cache pressure

Per-GPU weight footprint (falls as you shard)

Decode throughput vs capacity

Batch slot saturation

Time-to-first-token

Why this changes

Decision tradeoffs

KV cache management

Continuous batching

Tensor-parallel sharding

Prefill/decode split

GPU autoscaling

Token streaming

Source-backed rules

These are the durable system-design claims behind the model. The exact slider thresholds are deliberately labeled as teaching assumptions.

Verified rule

PagedAttention stores the KV cache in fixed pages to eliminate fragmentation

Borrowing OS virtual-memory paging, vLLM stores the KV cache in non-contiguous fixed-size blocks, cutting waste from fragmentation and raising serving throughput 2-4x.

vLLM (PagedAttention)

Verified rule

vLLM combines continuous batching with paged attention for high-throughput serving

The documentation describes continuous (in-flight) batching and PagedAttention as the core mechanisms that keep the GPU busy across requests of different lengths.

vLLM Docs

Verified rule

TensorRT-LLM shards large models across GPUs with tensor and pipeline parallelism

For models too large for one GPU, TensorRT-LLM splits each layer across devices and supports multi-GPU and multi-node parallelism plus disaggregated serving.

NVIDIA TensorRT-LLM

Verified rule

Triton Inference Server schedules and batches inference requests in production

Triton documents dynamic and in-flight batching, scheduling, and multi-backend serving used to run LLM inference at scale behind a single endpoint.

NVIDIA Triton

Teaching assumptions

Per-GPU memory, token throughput, and batching efficiency are conservative teaching numbers for one modern data-center GPU, not vendor benchmarks.
KV cache size is approximated as proportional to sequence length and sub-linear in model scale; modern frontier models use Grouped-Query Attention (GQA/MQA) to share key/value heads, which shrinks the KV cache by roughly 8x and is folded into the coefficient here.
Prefill is modeled as the compute-bound first-token phase and decode as the memory-bound token loop; their relative cost varies by hardware and kernel.