Inside Local LLM Inference: GGUF Quantization, FlashInfer Kernels & RadixTree KV Reuse

Executive Summary

Local LLM inference engines—such as llama.cpp, Ollama, and SGLang—have transformed edge and private server deployment by achieving high-throughput model execution on consumer GPUs and multi-core CPUs.

This 20-diagram interactive system breakdown deconstructs the core architectural innovations powering local LLM serving across 5 specialized chapters:

1. GGUF Quantization Formats (Q4_K_M, IQ4_XS, K-quants)

GGUF is a single-file binary format engineered for zero-copy memory mapping (mmap) and heterogeneous execution across CPU and GPU hardware. K-quant formats partition 256 consecutive weight elements into 16 sub-blocks with 6-bit sub-block scale deltas, reducing memory bandwidth by 75% compared to FP16. IQ4_XS leverages calibration dataset activations (imatrix) to construct non-linear 256-entry vector codebooks, matching FP16 perplexity on complex reasoning benchmarks.

2. FlashInfer & SIMD CUDA Kernels

High-throughput local serving requires dequantizing weights directly in hardware registers during matrix multiplication. SIMD vector intrinsics (AVX2, AVX-512, ARM Neon) load packed 4-bit bytes into 512-bit registers, executing bitwise shift and mask operations to unpack 16 weights simultaneously into FP16 ALUs. FlashInfer fuses INT4/INT8 KV-cache dequantization directly into page-table attention kernels, eliminating intermediate FP16 memory allocations and saturating GPU memory bandwidth.

3. RadixTree KV Reuse & Prefix Caching

Multi-turn conversation sessions and agentic workflows frequently reuse system prompts and shared context histories. SGLang RadixAttention organizes KV cache pages into a RadixTree hierarchy. Prompt tokens are matched against existing tree branches, allowing multi-turn requests to bypass up to 90% of prefill matrix multiplications. When GPU VRAM nears saturation, the RadixTree manager evicts unreferenced leaf nodes based on LRU timestamps.

4. Speculative Decoding & Verification Loops

Speculative decoding overcomes memory-bandwidth bottlenecks in autoregressive generation by generating candidate tokens with a fast draft model and verifying them in parallel. A lightweight 3B draft model generates K=5 candidate tokens sequentially. The 70B target model evaluates all 5 candidate tokens in a single parallel forward pass, accelerating token generation throughput by 2x-3x.

5. NUMA-Aware Memory & Offloading Strategies

When model parameter sizes exceed GPU VRAM, local engines partition layer execution across CPU RAM and VRAM. Model layers are partitioned dynamically across available devices. CPU host memory is allocated via cudaHostAlloc page-locked memory, ensuring maximum PCIe DMA transfer throughput. Multi-socket CPU systems bind thread worker pools to local NUMA nodes (numactl), preventing cross-socket interconnect bottlenecks.