System Design Lab

A RAG system is two pipelines — offline ingestion and an online retrieve-then-generate query path — and each force scales a different stage.

Change query rate, corpus size, chunk size, how many chunks you retrieve, the model context window, and embedding dimensions, then toggle reranking and hybrid search. The design moves from stuffing the prompt to a vector index with ANN retrieval, to chunking and reranking, to hybrid search plus caching, and finally a multi-tenant, continuously-ingesting platform.

Read the full tutorial

Guided walkthrough

Reason about it one step at a time

Normal evolution scenarios

Click left to right for the intended demo path. Each card changes the workload inputs.

Workload

These are inputs, not preset architecture stages.

Query rate Online questions per second hitting the retrieve-then-generate path. Documents indexed Source documents in the corpus; each is chunked and embedded into the vector index. Chunk size Tokens per chunk. Smaller chunks sharpen retrieval but multiply vectors and shrink context coverage. Top-k retrieved Chunks pulled from the vector index per query before assembly and (optional) reranking. Context window Model context budget in tokens; retrieved chunks plus the prompt must fit inside it. Embedding dimensions Vector width. Higher dimensions can lift recall but enlarge the index and slow ANN search. Index freshness target How quickly a newly written document must become retrievable; tighter targets force streaming ingestion. Rerank retrieved chunks Run a cross-encoder over the top-k to reorder by relevance; higher quality, real latency and GPU cost. Hybrid (keyword + vector) search Combine BM25 keyword matches with vector similarity to catch exact terms dense vectors miss.

Recommended shape

Current architecture path

Clients

Client sends natural-language questions and renders grounded answers

RAG API

RAG API orchestrates embed, retrieve, rerank, assemble, and generate per query

Response cache returns cached answers for repeated or similar questions before any retrieval

Retrieval

Query embedder embeds the question with the same model used at ingestion

Vector index approximate-nearest-neighbour search returns the top-k candidate chunks

Keyword index inverted index that catches exact terms dense vectors miss, for hybrid search

Rerank + assemble

Reranker reorders candidates by query-chunk relevance before they enter the context

Context assembler packs the best chunks plus the prompt to fit within the context window

Generation

LLM generator generates the answer conditioned on the assembled context

Ingestion

Chunk + embed splits new documents into chunks and embeds them off the query path

Index writer upserts chunk vectors into the index to keep retrieval fresh

Bottlenecks

Vector index scale

ANN query load

Rerank load

Context window fit

Ingestion / freshness

Why this changes

Decision tradeoffs

Chunking strategy

Vector index / ANN

Hybrid search

Reranking

Context assembly

Ingestion / freshness

Source-backed rules

These are the durable system-design claims behind the model. The exact slider thresholds are deliberately labeled as teaching assumptions.

Verified rule

RAG conditions generation on retrieved passages instead of memorised weights

The original RAG paper retrieves relevant passages with a dense retriever and feeds them to a seq2seq generator, separating knowledge (the index) from the parametric model.

Lewis et al., 2020 (arXiv:2005.11401)

Verified rule

Billion-scale similarity search needs an ANN index, not brute force

FAISS provides quantised and graph-based indexes so nearest-neighbour search over very large vector sets stays fast and memory-feasible.

FAISS

Verified rule

HNSW gives logarithmic-time approximate nearest-neighbour search

Hierarchical Navigable Small World graphs trade a tunable recall/latency knob for high-recall search that scales to large indexes, the default in most vector databases.

Malkov & Yashunin, 2016 (arXiv:1603.09320)

Verified rule

Vector databases manage embeddings, metadata filtering, and updates

A production vector store handles index building, upserts for freshness, and metadata filters for multi-tenant isolation, beyond raw nearest-neighbour math.

pgvector

Teaching assumptions

Average documents split into a few chunks; total vectors are documents times chunks-per-doc scaled by chunk size.
Freshness is modeled as new/changed docs (a small daily share of the corpus) being embedded and upserted, not a full-corpus re-embed; a tighter target forces a streaming pipeline.
ANN, rerank, embedding, and generation per-node budgets are conservative teaching numbers, not vendor limits.
Cache hit rate and recall are approximated; real systems tune HNSW parameters and rerankers empirically.