Activity v1

Inference Batching and Queueing Simulator

Model deterministic arrival rate batch size service time cache pressure throughput utilization and tail-latency trade-offs.

completion

beginner

seed 11

Inference Batching and Queueing Simulator: Baseline

Explore the core controls with stable inputs and visible assumptions.

Scenario

Assumptions

The simulation is deterministic and intentionally simplifies provider and hardware behavior.

Failure injection

Stable baseline with no injected production fault.

Decision and rationale

Requests per second36

Maximum batch8

Batch service time (ms)140

KV cache pressure35

Queue state

Stable

Capacity46.0 req/s

Utilization78%

Queue delay176 ms

Estimated p95489 ms

Scheduler pressure78%

M/M/1-inspired teaching model with batching and cache penalties. Real serving systems require measured arrival distributions, token lengths, prefill/decode separation, and hardware traces.

0 saved attempts

Expected outcomes

Relate utilization and queueing to tail latency
Balance batch efficiency against waiting time

Connected concepts

Inference Serving Architecture Inference Engine Architecture Latency and Throughput

Apply this lab in an architecture

Cost and Reliability Engineering Model Serving Capacity Planning

Back to labs