Open Studio
Back to Visual Labs
Lab 18 · Inference Virtual Memory Simulator
vLLM SOSP '23 Paper Implementation

PagedAttention & KV-Cache Virtual Memory Simulator

Explore how operating system virtual memory paging transforms high-throughput LLM inference. Simulate dynamic physical block allocation, watch logical-to-physical block table translations ($L_i \to P_k$), observe Prefix Caching with Copy-on-Write (CoW), and inspect real-time mathematical comparisons between naive contiguous over-allocation and zero-waste PagedAttention.

Paper: PagedAttention PaperGuide: Speculative ServingPostmortem: VRAM FragmentationSystem: vLLM
Workload Preset
Block Size (B)
2048 KB / block
Prefix Caching (CoW)
ACTIVE
Concurrency Multiplier
2.2×

Higher batch throughput in identical VRAM

Prefix Cache Savings
8.00 MB

Saved by deduplicating system prompts

Contiguous Waste
43.7%

Over-reserved for ungenerated tokens

Paged Fragmentation
20.8%

Bounded strictly by tail block slack (< 3.5%)

Legacy Paradigm
HuggingFace / FasterTransformer

Contiguous Static Pre-Allocation

Total Reserved
39.5 MB

Allocators must reserve contiguous virtual memory for the request's maximum possible sequence length (max_tokens) upfront. Ungenerated slots remain locked and unusable by concurrent requests.

Used: 22.3 MB (56.3%)Wasted: 17.3 MB (43.7%)
Over-allocated Waste
Internal Fragmentation
43.7%
Reserved tokens not reached
External Fragmentation
~18.5%
Contiguity requirement holes
vLLM SOSP '23
Virtual Memory Paging

PagedAttention On-Demand Allocation

Physical Pool Allocated
18.0 MB

KV-cache is partitioned into fixed-size physical blocks (16 tokens). Blocks are allocated on-demand as tokens decode. Non-contiguous blocks are stitched via a logical-to-physical block table, eliminating all external fragmentation.

Active Stored: 14.3 MBTail Slack: 3.75 MB (20.8%)
Internal Frag.
20.8%
Only in tail block
External Frag.
0.0%
Eliminated by paging
Prefix Saved
8.0 MB
Deduplicated VRAM
Mathematical Derivation (8B FP16 KV-Cache Baseline)

Memory per token is calculated as 2 × n_layers (32) × n_heads (32) × head_dim (128) × 2 bytes = 128 KB/token. In contiguous allocation, reserving 100 max tokens allocates 12.8 MB upfront. With PagedAttention (block size 16), only 2048 KB is allocated at a time. Internal fragmentation is bounded by (B - (tokens % B)) < 16 tokens per sequence, yielding a sustained 2.2× concurrency boost.

GPU VRAM Physical Memory Block Pool (64 Blocks · 128 MB)

Free (ref=0)
Dedicated
Shared (Prefix Cache)
Physical Block Inspector: P_0
SHARED (refCount = 3)
Base Offset: 0x000000 (0.0 MB mark)
Occupancy
16 / 16 tokens

Slack: 0 tokens (0.00 MB unconsumed)

Mapped Sequences
req-1, req-2, req-3

Shared across multiple sequences

Prefix Cache Key
sys-coding-agent-v1:block-0

Immutable prompt hash

Token Slots in Block P_0 (16 slots):
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

Sequence Block Tables (Logical L_i → Physical P_k Translation)

4 Active Sequences
req-1Code Copilot #1 (Auth Engine)
Generating (46/78 tok)
Sys Prompt: 32 tok•User Prompt: 14 tok•Generated: 0 tok
Context Fill: 46 tokensMax Target: 78 tokens (59%)
Virtual Page Table Mapping (Logical Block → Physical Block):
→
→
req-2Code Copilot #2 (Refactor Task)
Generating (40/64 tok)
Sys Prompt: 32 tok•User Prompt: 8 tok•Generated: 0 tok
Context Fill: 40 tokensMax Target: 64 tokens (63%)
Virtual Page Table Mapping (Logical Block → Physical Block):
→
→
req-3Code Copilot #3 (Test Generator)
Generating (52/84 tok)
Sys Prompt: 32 tok•User Prompt: 20 tok•Generated: 0 tok
Context Fill: 52 tokensMax Target: 84 tokens (62%)
Virtual Page Table Mapping (Logical Block → Physical Block):
→
→
→
req-4Doc Summarizer (RFC-9110)
Generating (40/90 tok)
Sys Prompt: 16 tok•User Prompt: 24 tok•Generated: 0 tok
Context Fill: 40 tokensMax Target: 90 tokens (44%)
Virtual Page Table Mapping (Logical Block → Physical Block):
→
→

PagedAttention Memory Manager Event Stream

Latest 15 memory events
8:42:30 AM[config-changed]Initialized simulator with 'shared-system-prompt' preset (64 blocks, 16 tokens/block)
8:42:30 AM[request-added]Added request req-4 ("Doc Summarizer (RFC-9110)") with 40 prompt tokens across 3 blocks
8:42:30 AM[block-allocated]Allocated P_8 for req-4 (L_2)
8:42:30 AM[block-allocated]Allocated P_7 for req-4 (L_1)
8:42:30 AM[block-allocated]Allocated P_6 as Prefix Cache anchor for sys-doc-summarizer:block-0
8:42:30 AM[request-added]Added request req-3 ("Code Copilot #3 (Test Generator)") with 52 prompt tokens across 4 blocks
8:42:30 AM[block-allocated]Allocated P_5 for req-3 (L_3)
8:42:30 AM[block-allocated]Allocated P_4 for req-3 (L_2)
8:42:30 AM[prefix-hit]Prefix Cache Hit: Request req-3 shared Block P_1 (refCount=3) for sys-coding-agent-v1:block-1
8:42:30 AM[prefix-hit]Prefix Cache Hit: Request req-3 shared Block P_0 (refCount=3) for sys-coding-agent-v1:block-0
8:42:30 AM[request-added]Added request req-2 ("Code Copilot #2 (Refactor Task)") with 40 prompt tokens across 3 blocks
8:42:30 AM[block-allocated]Allocated P_3 for req-2 (L_2)
8:42:30 AM[prefix-hit]Prefix Cache Hit: Request req-2 shared Block P_1 (refCount=2) for sys-coding-agent-v1:block-1
8:42:30 AM[prefix-hit]Prefix Cache Hit: Request req-2 shared Block P_0 (refCount=2) for sys-coding-agent-v1:block-0
8:42:30 AM[request-added]Added request req-1 ("Code Copilot #1 (Auth Engine)") with 46 prompt tokens across 3 blocks
1

Prefix Caching & CoW

Load the Copilot (Prefix) preset. Observe how requests #1, #2, and #3 share identical system prompt blocks (P_0 and P_1) with a refCount = 3 badge, completely eliminating duplicate VRAM usage.

Verifies Multi-Tenant Deduplication
2

Block Size Trade-Offs (16 vs 32)

Toggle Block Sizebetween 16 and 32 tokens. At 16 tokens, internal fragmentation is minimized to < 3.5% tail slack. At 32 tokens, block table lookup overhead is halved, but average tail waste increases proportionally to $B / 2$.

Verifies Bounded Slack Invariant
3

Contiguous Waste vs Paged Concurrency

Examine the Memory Efficiency Meters. In contiguous allocation, 60%–85% of VRAM is wasted waiting for ungenerated tokens. PagedAttention eliminates external fragmentation completely, delivering a sustained 2.5×–4× concurrency gain.

Verifies 3× Concurrency Multiplier