Inference Serving Architecture

Learning outcomes

Trace routing queueing batching and execution
Relate cache pressure to throughput and latency
Select scaling signals for inference workloads

Mental model

An inference service is a queueing system wrapped around expensive stateful execution. Requests compete for memory and compute while the serving layer protects deadlines, fairness, quality, and capacity.

Authenticate and admit

Route model and priority

Queue and batch

Allocate KV memory

Run prefill and decode

Stream finalize and measure

Conceptual teaching model synthesized from:Efficient Memory Management for Large Language Model Serving with PagedAttention vLLM inference and serving engine Horizontal Pod Autoscaling

Theory

Admission control rejects or degrades work that cannot meet policy or capacity. Routing selects a model revision, deployment, and priority class. The scheduler groups compatible work while balancing batch efficiency against waiting time. Prefill processes input tokens in parallel; decode repeatedly generates tokens and holds per-sequence KV-cache state.

Autoscaling requires workload-relevant signals. CPU alone may not reveal accelerator memory pressure or queued tokens. Useful signals include pending requests, queued tokens, active sequences, KV-cache utilization, prefill load, decode throughput, and deadline misses. Scaling also has cold-start and model-loading delay, so headroom and load shedding remain necessary.

Alternatives and trade-offs

Managed APIs minimize platform work but limit engine control. Dedicated deployments improve isolation and predictable capacity. Shared fleets increase utilization but require fairness and tenant controls. Larger batches improve throughput while increasing queue delay; quantization reduces memory and may change quality.

Failure modes and misconceptions

Average latency hides deadline failures. Autoscaling after saturation may react too late. Unlimited queues convert overload into timeouts. Mixing long and short requests without scheduling policy creates head-of-line blocking. Retrying timed-out generations can amplify overload.

Knowledge check

Reflect before revealing the guide

Which scaling signal best captures work waiting for accelerator execution, and why is CPU utilization insufficient?

Decision scenario

A chat service separates interactive and batch queues, limits input and output tokens, tracks queued-token seconds, reserves headroom for priority traffic, and sheds batch work before interactive p95 latency breaches its target.

Learning outcomes

Explain Inference Serving Architecture as a system mechanism rather than a slogan.
Compare its alternatives, trade-offs, and production failure modes.
Apply the concept to a decision and identify evidence that would validate it.

Trade-offs

Using Inference Serving Architecture can improve capability or control, but it also introduces cost, latency, complexity, and failure modes that must be measured against an explicit objective.

Evidence assessment