System Design Lab
An LLM agent is a reason-act loop whose cost, latency, and blast radius all grow with the number of steps per task.
Tune concurrent sessions, tool calls and steps per task, LLM latency, context size, and how many tools are registered. The design grows from a single tool call into a reason-act loop, then adds memory and planning, parallel tool execution behind a sandbox, and finally multi-agent orchestration with full tracing at scale.
Guided walkthrough
Reason about it one step at a time
Takeaway
Normal evolution scenarios
Click left to right for the intended demo path. Each card changes the workload inputs.
Recommended shape
Current architecture path
Clients
Client submits a task and streams the agent run back as it progresses
Agent runtime
Agent loop drives the decide-call-observe loop and enforces the step budget per task
Planner breaks a goal into sub-steps and sequences or fans out the work
Tool execution
Tool registry holds tool schemas and routes the model to the right tool from many
Sandbox runs untrusted tool code and side effects in an isolated worker pool
Memory + LLM
Conversation keeps the working context and applies summarization or truncation
Vector memory embeds and retrieves facts across tasks beyond the context window
LLM gateway batches, rate-limits, and retries the many model calls every step makes
Observability
Tracing records each step, tool call, and token cost off the hot path
Guardrails validates outputs, retries failures, and enforces cost and loop limits
Bottlenecks
LLM call volume
Sequential task latency
Sandbox execution load
Context window pressure
Tool selection complexity
Why this changes
Decision tradeoffs
Source-backed rules
These are the durable system-design claims behind the model. The exact slider thresholds are deliberately labeled as teaching assumptions.
Reason-and-act interleaving makes agents iterate, not answer once
ReAct interleaves reasoning traces with actions so the model decides, acts, observes, and repeats β the loop structure that all step-count cost and latency follow from.
ReAct (Yao et al.)Models can be taught to call tools/APIs as part of generation
Toolformer shows the model itself learning when and how to invoke external tools, motivating a tool registry the model selects from rather than hard-coded calls.
Toolformer (Schick et al.)Tool-use APIs define a registry of tool schemas the model picks among
Production tool calling passes JSON tool schemas to the model; too many crowded into one prompt degrade selection, which is why routing matters as the registry grows.
Anthropic tool use docsMulti-step agent runs need tracing across each LLM and tool span
Orchestration guidance treats per-step tracing, retries, and guardrails as first-class, because a multi-step run is much harder to debug than a single request.
OpenAI Agents SDK docsTeaching assumptions
- Sequential task latency assumes steps run one after another; parallel tools shorten wall-clock but not total LLM calls.
- Per-lane LLM and per-worker sandbox budgets are conservative teaching numbers, not provider limits.
- Context pressure compares the working window against a comfortable single-task budget; long-term recall is modeled as a separate vector path.