Activity v1

Inference Batching and Queueing Simulator

Model deterministic arrival rate batch size service time cache pressure throughput utilization and tail-latency trade-offs.

completion
beginner
seed 11

Inference Batching and Queueing Simulator: Baseline

Explore the core controls with stable inputs and visible assumptions.

Assumptions

The simulation is deterministic and intentionally simplifies provider and hardware behavior.

Failure injection

Stable baseline with no injected production fault.

GPU VRAM & KV-Cache Memory Budget Estimator

Simulate model weights, KV-cache growth, PagedAttention block allocation, and GPU OOM thresholds.

⚠️ OOM WARNING
87.72 GB / 80 GB (100%)
Model & Infrastructure Configuration
Context Length16,384 tokens
Concurrency (Batch Size)16 requests
CUDA Out Of Memory (OOM) Threshold Exceeded!

The required VRAM (87.72 GB) exceeds the target cluster capacity (80 GB) by Math.abs(-7.72) GB. Apply INT8/FP4 quantization, reduce batch size, or upgrade GPU cluster topology.

VRAM Memory Breakdown (1x A100 (80GB))87.72 GB / 80 GB
16G
30.4G
39.32G
Model Weights (16 GB)
KV Cache (30.4 GB)
Activations (39.32 GB)
CUDA Buffer (2 GB)
Weights VRAM
16 GB
FP16 precision
KV Cache VRAM
30.4 GB
GQA attention
Concurrency
16 reqs
Batch size
VRAM Headroom
-7.72 GB
Available space
KV Cache Memory Sizing Formula
Memory_KV = 2 × Layers × Heads × HeadDim × Context × Batch × BytesPerElement

Multi-Head Latent Attention (MLA) reduces KV cache memory footprint by up to 93% by compressing Key-Value matrices into a single low-rank latent vector prior to attention matrix multiplication.

0 saved attempts

Expected outcomes

  • Relate utilization and queueing to tail latency
  • Balance batch efficiency against waiting time

Connected concepts

Inference Serving ArchitectureInference Engine ArchitectureLatency and Throughput

Apply this lab in an architecture

Cost and Reliability EngineeringModel Serving Capacity Planning
Back to labs