Model Serving Capacity Planning

Mental model

Capacity planning begins with workload and SLO distributions, then models the bottleneck resource and validates it under representative load.

Learning outcomes

• Explain the mechanism and ownership boundaries behind Model Serving Capacity Planning.
• Compare the main design alternatives and their operational trade-offs.
• Diagnose common failures and select evidence for a production decision.

Theory

Measure prompt and output lengths, concurrency, arrival bursts, model variants, cache demand, time to first token, decode rate, failure reserve, and deployment overhead. Size for tail objectives and failure scenarios, not average requests.

Trade-offs

High utilization lowers unit cost but increases queue sensitivity and failure impact. Spare capacity improves tail latency and resilience but carries idle cost.

Failure modes and misconceptions

Using requests per second without tokens; uniform synthetic prompts; ignoring warmup and model load; no failure reserve; extrapolating across hardware; and sizing from mean latency.

Decision scenario

A service has predictable daytime load and sharp launch bursts. Build a capacity model that includes KV memory, queueing, warm replicas, and one-worker failure.

Primary sources

• vllm-paper
• kubernetes-resource-management