System Design Lab

A chat backend is a connection problem first: millions of live sockets must be tracked, then a message fanned out to everyone who should receive it.

Change concurrent connections, message rate, group size, the online ratio, history retention, devices per user, and regions. The design moves from a single socket server to a gateway fleet with a session registry, a routing and fan-out tier with a presence service, durable per-user inboxes, and finally cross-region routing with offline push.

Read the full tutorial

Guided walkthrough

Reason about it one step at a time

Normal evolution scenarios

Click left to right for the intended demo path. Each card changes the workload inputs.

Workload

These are inputs, not preset architecture stages.

Concurrent connections Live WebSocket/long-lived TCP sockets held open at once. This sets the gateway memory floor. Messages sent Inbound messages per second before group amplification. Each one may fan out to many recipients. Average group size Recipients per message. A 1:1 chat is 2; a large group multiplies every send into many deliveries. Online ratio Share of recipients currently connected. The rest must be written to a durable inbox for later. History retention How long messages are kept so offline and multi-device users can sync them later. Devices per user Active devices per account. Every delivery is multiplied so each device stays in sync. Regions Regions terminating connections close to users; cross-region sends must be routed between them. End-to-end encryption Encrypt on the device so the server only routes ciphertext; it cannot fan out by reading content. Read receipts Track delivered/read state per recipient and device, adding a return event for every message.

Recommended shape

Current architecture path

Clients

Client holds a long-lived WebSocket to send and receive messages in real time

Connection gateway

WS gateway terminates persistent connections and keeps each live socket in memory

Session registry maps each online user/device to the gateway node that holds its socket

Routing + presence

Router / fan-out expands a send to every group member and routes each to the right gateway

Presence tracks who is online and propagates delivered/read receipts

Message store

Message store persists every message so history survives restarts and late syncs

Per-user inbox queues messages per recipient device until it reconnects and pulls them

Offline push

Push service sends APNs/FCM notifications to wake offline devices so they sync

Bottlenecks

Connection memory

Message fan-out rate

Inbox + history storage

Presence + receipt cost

Cross-region routing

Why this changes

Decision tradeoffs

Connection / session layer

Message routing

Group fan-out

Offline inbox

Presence service

End-to-end encryption

Source-backed rules

These are the durable system-design claims behind the model. The exact slider thresholds are deliberately labeled as teaching assumptions.

Verified rule

A WebSocket is a persistent, full-duplex connection the server must keep open

Each chat client holds one long-lived connection, so the gateway tier is sized by how many open sockets it can keep in memory, not by request rate.

MDN Web Docs

Verified rule

Publish/subscribe decouples senders from the many recipients of a message

Fan-out to a group is naturally a pub/sub problem: one publish is delivered to every subscriber, which is how a router expands a send to many members.

Google Cloud Pub/Sub

Verified rule

A durable message queue lets offline recipients receive messages later

A per-user inbox queue holds messages for disconnected or multi-device users until each device reconnects and drains its queue.

AWS SQS

Verified rule

End-to-end encryption means the server routes ciphertext it cannot read

With E2EE the server can fan out and store messages but cannot inspect content, so server-side features must work on ciphertext and metadata only.

Signal Protocol

Teaching assumptions

Each live socket is charged a fixed memory budget (kernel plus app buffers plus session state); real costs vary by stack.
Fan-out rate is messages per second times average group size times devices per user; presence and receipts add return events on top.
Single-node gateway, routing, inbox-write, and presence budgets are conservative teaching numbers, not vendor limits.