Mental model
An inference service is a queueing system wrapped around expensive stateful execution. Requests compete for memory and compute while the serving layer protects deadlines, fairness, quality, and capacity.
Theory
Admission control rejects or degrades work that cannot meet policy or capacity. Routing selects a model revision, deployment, and priority class. The scheduler groups compatible work while balancing batch efficiency against waiting time. Prefill processes input tokens in parallel; decode repeatedly generates tokens and holds per-sequence KV-cache state.
Autoscaling requires workload-relevant signals. CPU alone may not reveal accelerator memory pressure or queued tokens. Useful signals include pending requests, queued tokens, active sequences, KV-cache utilization, prefill load, decode throughput, and deadline misses. Scaling also has cold-start and model-loading delay, so headroom and load shedding remain necessary.
Alternatives and trade-offs
Managed APIs minimize platform work but limit engine control. Dedicated deployments improve isolation and predictable capacity. Shared fleets increase utilization but require fairness and tenant controls. Larger batches improve throughput while increasing queue delay; quantization reduces memory and may change quality.
Failure modes and misconceptions
Average latency hides deadline failures. Autoscaling after saturation may react too late. Unlimited queues convert overload into timeouts. Mixing long and short requests without scheduling policy creates head-of-line blocking. Retrying timed-out generations can amplify overload.
Knowledge check
Which scaling signal best captures work waiting for accelerator execution, and why is CPU utilization insufficient?
Decision scenario
A chat service separates interactive and batch queues, limits input and output tokens, tracks queued-token seconds, reserves headroom for priority traffic, and sheds batch work before interactive p95 latency breaches its target.
Learning outcomes
- Explain Inference Serving Architecture as a system mechanism rather than a slogan.
- Compare its alternatives, trade-offs, and production failure modes.
- Apply the concept to a decision and identify evidence that would validate it.
Trade-offs
Using Inference Serving Architecture can improve capability or control, but it also introduces cost, latency, complexity, and failure modes that must be measured against an explicit objective.