System Design Lab
A payment ledger is bound by correctness, not QPS: money must never be created, lost, or double-charged.
Change payment rate, idempotency-key TTL, external provider latency, total ledger entries, the reconciliation window, and regions. The design adds correctness mechanisms first — idempotency keys, a double-entry append-only ledger, ACID writes, an async PSP state machine — and only partitions by account once integrity is locked down.
Guided walkthrough
Reason about it one step at a time
Takeaway
Normal evolution scenarios
Click left to right for the intended demo path. Each card changes the workload inputs.
Recommended shape
Current architecture path
Clients
Client submits payments and retries on timeout, sometimes resending the same request
Payment API
Payment API validates payment requests and owns the payment state machine
Idempotency layer stores idempotency keys so a retried request is processed exactly once
Ledger
Ledger service records balanced debits and credits as the immutable source of truth
Datastore
ACID store commits balanced entries atomically and durably
Account shards partitions the ledger by account so writes scale without breaking balance
Async + audit
PSP adapter authorizes and captures through the slow external provider with retries and a DLQ
Reconciliation re-fetches provider records and re-balances them against the ledger
Audit log keeps an append-only event trail of every state transition for compliance
Bottlenecks
Ledger write load
Idempotency-store load
PSP-bound async backlog
Ledger storage growth
Reconciliation lag
Why this changes
Decision tradeoffs
Source-backed rules
These are the durable system-design claims behind the model. The exact slider thresholds are deliberately labeled as teaching assumptions.
Idempotency keys make a retried payment request safe to replay
Stripe lets clients attach an idempotency key so retries from network failures return the original result instead of creating a second charge.
Stripe DocsCapture funds asynchronously because the provider is slow and can fail
A PaymentIntent moves through requires_action, processing, and succeeded states; treating capture as an async state machine tolerates provider latency and failures.
Stripe DocsA double-entry, append-only ledger is the immutable source of truth
Modeling money as balanced postings to accounts keeps the ledger auditable and self-checking, so balances are derived rather than mutated in place.
martinfowler.comEvent sourcing keeps a full, replayable audit trail of state changes
Persisting every state transition as an immutable event gives a complete audit log and lets balances be rebuilt by replaying history.
martinfowler.comTeaching assumptions
- Correctness is the binding constraint; single-node throughput and storage budgets are conservative teaching numbers, not vendor limits.
- Each payment is modeled as one balanced double-entry write; the idempotency store sees roughly one lookup per attempt including retries.
- PSP-bound backlog is approximated from payment rate times provider latency, i.e. the in-flight captures waiting on the external provider.