Reference architecture

Model Serving Capacity Planning

A workload-first method for sizing accelerators, KV cache, admission, batching, redundancy, autoscaling, and failure reserve.

18 minVerified 2026-07-214 primary sources
A governed production AI reference architecture with observable, secured service boundaries.

Architecture

A workload-first method for sizing accelerators, KV cache, admission, batching, redundancy, autoscaling, and failure reserve.

Intent and policy
Validated request
Controlled execution
Independent verification
Telemetry and response
Conceptual teaching model synthesized from:Efficient Memory Management for Large Language Model Serving with PagedAttentionvLLM V1 scheduler at v0.10.2Kubernetes resource management for Pods and containersHorizontal Pod Autoscaling

The architecture separates intent, policy, state, execution, evidence, and response. Each boundary has one accountable owner and an observable contract. Probabilistic components may propose or classify; deterministic software owns permissions, budgets, state transitions, and release conditions.

Architecture claim
Representative test
Production signal
Decision gate
Review or rollback
Conceptual teaching model synthesized from:Efficient Memory Management for Large Language Model Serving with PagedAttentionvLLM V1 scheduler at v0.10.2Kubernetes resource management for Pods and containersHorizontal Pod Autoscaling

Decisions

| Decision | Required evidence | Review trigger | |---|---|---| | Model token-work distributions, not requests per second alone. | Versioned test, trace, or architecture record | Material workload, provider, policy, or data change | | Size to tail objectives and defined worker failures. | Versioned test, trace, or architecture record | Material workload, provider, policy, or data change | | Use admission and backpressure while replicas warm. | Versioned test, trace, or architecture record | Material workload, provider, policy, or data change | | Choose replication or partitioning from memory fit and interconnect evidence. | Versioned test, trace, or architecture record | Material workload, provider, policy, or data change |

Alternatives and trade-offs

A centralized control plane simplifies policy and evidence collection but can become a latency and availability dependency. Distributed enforcement reduces central bottlenecks but requires consistent identity, versions, and audit semantics. Prefer the smallest boundary that can enforce the invariant and prove the outcome.

Failure modes

  • Uniform prompts hide queue and cache behavior.
  • Cold replicas are counted as ready capacity.
  • Retry storms amplify overload.
  • More GPUs are assumed to reduce latency.

Operational checklist

  • [ ] Prompt, output, concurrency, and burst distributions are measured.
  • [ ] Load tests include cancellation and mixed priority classes.
  • [ ] One-worker and one-zone failure reserve is explicit.
  • [ ] Queue deadlines and load shedding are configured.
  • [ ] Scale-down drains active work safely.

Connected practice

Sources

  • vllm-paper
  • vllm-v0102-scheduler
  • kubernetes-resource-management
  • kubernetes-hpa