Continuous Batching and Admission Control

Mental model

Continuous batching chooses work at each engine step; admission control limits what may enter before overload destroys every request's objective.

Learning outcomes

• Explain the mechanism and ownership boundaries behind Continuous Batching and Admission Control.
• Compare the main design alternatives and their operational trade-offs.
• Diagnose common failures and select evidence for a production decision.

Theory

Separate queued, admitted, running, and preempted state. Budget tokens and cache, prioritize classes explicitly, cap per-tenant concurrency, reject early when deadlines cannot be met, and observe starvation.

Trade-offs

Larger dynamic batches raise throughput but may delay short interactive requests. Aggressive admission maximizes offered load while creating unbounded queues and timeout waste.

Failure modes and misconceptions

No queue deadline; first-come fairness across unlike workloads; admitting beyond cache capacity; retry storms; hidden preemption; and reporting rejected work as availability failure.

Decision scenario

Interactive chat and batch summarization share one engine. Design admission classes and scheduling policies that prevent either workload from starving the other.

Primary sources

• vllm-v0102-scheduler
• sre-book