Mental model
Autoscaling changes future capacity; backpressure protects the system during the delay between overload and usable replicas.
Learning outcomes
- Explain the mechanism and ownership boundaries behind Inference Autoscaling and Backpressure.
- Compare the main design alternatives and their operational trade-offs.
- Diagnose common failures and select evidence for a production decision.
Theory
Scale on queue delay, admitted token work, cache pressure, and SLO risk rather than CPU alone. Use bounded queues, deadlines, retry budgets, load shedding, warm capacity, and stabilization windows.
Trade-offs
Fast scaling reacts to bursts but can oscillate and trigger expensive model loads. Conservative scaling is stable but needs larger warm reserve and earlier shedding.
Failure modes and misconceptions
Scaling from average CPU; unbounded queues; synchronized retries; counting cold replicas as ready; no scale-down drain; and ignoring accelerator allocation time.
Decision scenario
Traffic triples in thirty seconds while new model replicas take four minutes to load. Define backpressure and scaling behavior for the gap.
Why cannot autoscaling replace bounded queues and load shedding?
Primary sources
kubernetes-hpasre-book