System Design Lab
A notification system is a fan-out pipeline whose hardest job is decoupling fast producers from slow, unreliable providers.
Change the request rate, fan-out per notification, channels per notification, user base, provider latency, retry attempts, and per-user send cap. The design moves from a direct synchronous send to a durable queue with channel workers, multi-channel routing, dedup plus rate-limit plus preference filtering, and finally high-volume retries with a dead-letter queue and delivery analytics.
Guided walkthrough
Reason about it one step at a time
Takeaway
Normal evolution scenarios
Click left to right for the intended demo path. Each card changes the workload inputs.
Recommended shape
Current architecture path
Producers
Producers services and triggers that ask for a notification to be sent
Ingest API
Ingest API accepts requests, validates them, and returns fast without blocking on a send
Preference store holds per-user channel choices, opt-outs, and quiet-hour windows
Queue + workers
Message queue durable buffer so producers never block on slow providers
Channel workers pull messages, apply dedup and rate-limit, and dispatch per channel
Provider adapters
Push adapter sends mobile push through Apple and Google gateways
Email adapter sends email through a transactional email provider
SMS adapter sends SMS through a telephony provider
Retry + analytics
Retry + DLQ retries failed sends with backoff and parks the unsendable in a dead-letter queue
Delivery analytics collects delivery, bounce, and failure events for dashboards off the hot path
Bottlenecks
Queue backlog pressure
Provider-bound throughput
Fan-out amplification
Retry / DLQ pressure
Dedup / rate-limit load
Why this changes
Decision tradeoffs
Source-backed rules
These are the durable system-design claims behind the model. The exact slider thresholds are deliberately labeled as teaching assumptions.
A queue decouples fast producers from slow consumers
Placing a durable queue between producers and the senders that call slow providers smooths bursts and stops a provider stall from blocking the request path.
AWS β Queue-Based Load LevelingFailed deliveries belong in a dead-letter queue after retries
After a bounded number of failed processing attempts, messages are moved to a dead-letter queue so poison messages cannot block the main queue and can be inspected later.
AWS SQS Dead-Letter QueuesExponential backoff with jitter avoids retry storms
Retrying failed provider calls with exponentially growing, jittered delays prevents synchronized retries from overwhelming an already struggling provider.
AWS Architecture BlogAPNs and FCM are the gateways for mobile push
Mobile push is delivered through provider gateways (APNs for Apple, FCM for Android) rather than directly to devices, so a push adapter must speak each gateway protocol.
Apple APNs DocumentationTeaching assumptions
- Each channel-send is modeled as one external provider call; total sends = requests x fan-out x channels.
- Provider concurrency is bounded, so the achievable send rate scales as worker concurrency / provider latency; latency, not CPU, is the binding constraint.
- Dedup and rate-limit are approximated as a per-user counter lookup on every candidate send; preference filtering is a per-user read before dispatch.