Learning outcomes
- Explain the operating model behind Model Serving Capacity Planning.
- Evaluate trade-offs and failure modes for Model Serving Capacity Planning.
- Apply Model Serving Capacity Planning to a production decision.
Mental model
Capacity planning begins with workload and SLO distributions, then models the bottleneck resource and validates it under representative load.
Learning outcomes
- Explain the mechanism and ownership boundaries behind Model Serving Capacity Planning.
- Compare the main design alternatives and their operational trade-offs.
- Diagnose common failures and select evidence for a production decision.
Theory
Measure prompt and output lengths, concurrency, arrival bursts, model variants, cache demand, time to first token, decode rate, failure reserve, and deployment overhead. Size for tail objectives and failure scenarios, not average requests.
Trade-offs
High utilization lowers unit cost but increases queue sensitivity and failure impact. Spare capacity improves tail latency and resilience but carries idle cost.
Failure modes and misconceptions
Using requests per second without tokens; uniform synthetic prompts; ignoring warmup and model load; no failure reserve; extrapolating across hardware; and sizing from mean latency.
Decision scenario
A service has predictable daytime load and sharp launch bursts. Build a capacity model that includes KV memory, queueing, warm replicas, and one-worker failure.
Why are prompt and output token distributions more useful than request count alone?
Primary sources
vllm-paperkubernetes-resource-management
Evidence assessment
Theory and decision mastery
Decision scenario
A production team must adopt Model Serving Capacity Planning while meeting quality, latency, security, and operating constraints.
Which decision process is most defensible?
Relationships
Model Serving Capacity Planning builds on Inference Serving Architecture.
Model Serving Capacity Planning informs governed production decisions and review evidence.
Inference Autoscaling and Backpressure builds on serving-capacity-planning.
Primary sources
- Efficient Memory Management for Large Language Model Serving with PagedAttention — SOSP / arXiv, verified 2026-07-16
- Kubernetes resource management for Pods and containers — Kubernetes, verified 2026-07-21