Executive Summary
Local LLM inference engines—such as llama.cpp, Ollama, and SGLang—have transformed edge and private server deployment by achieving high-throughput model execution on consumer GPUs and multi-core CPUs.
This 20-diagram interactive system breakdown deconstructs the core architectural innovations powering local LLM serving across 5 specialized chapters:
1. GGUF Quantization Formats (Q4_K_M, IQ4_XS, K-quants)
GGUF is a single-file binary format engineered for zero-copy memory mapping (mmap) and heterogeneous execution across CPU and GPU hardware. K-quant formats partition 256 consecutive weight elements into 16 sub-blocks with 6-bit sub-block scale deltas, reducing memory bandwidth by 75% compared to FP16. IQ4_XS leverages calibration dataset activations (imatrix) to construct non-linear 256-entry vector codebooks, matching FP16 perplexity on complex reasoning benchmarks.
2. FlashInfer & SIMD CUDA Kernels
High-throughput local serving requires dequantizing weights directly in hardware registers during matrix multiplication. SIMD vector intrinsics (AVX2, AVX-512, ARM Neon) load packed 4-bit bytes into 512-bit registers, executing bitwise shift and mask operations to unpack 16 weights simultaneously into FP16 ALUs. FlashInfer fuses INT4/INT8 KV-cache dequantization directly into page-table attention kernels, eliminating intermediate FP16 memory allocations and saturating GPU memory bandwidth.
3. RadixTree KV Reuse & Prefix Caching
Multi-turn conversation sessions and agentic workflows frequently reuse system prompts and shared context histories. SGLang RadixAttention organizes KV cache pages into a RadixTree hierarchy. Prompt tokens are matched against existing tree branches, allowing multi-turn requests to bypass up to 90% of prefill matrix multiplications. When GPU VRAM nears saturation, the RadixTree manager evicts unreferenced leaf nodes based on LRU timestamps.
4. Speculative Decoding & Verification Loops
Speculative decoding overcomes memory-bandwidth bottlenecks in autoregressive generation by generating candidate tokens with a fast draft model and verifying them in parallel. A lightweight 3B draft model generates K=5 candidate tokens sequentially. The 70B target model evaluates all 5 candidate tokens in a single parallel forward pass, accelerating token generation throughput by 2x-3x.
5. NUMA-Aware Memory & Offloading Strategies
When model parameter sizes exceed GPU VRAM, local engines partition layer execution across CPU RAM and VRAM. Model layers are partitioned dynamically across available devices. CPU host memory is allocated via cudaHostAlloc page-locked memory, ensuring maximum PCIe DMA transfer throughput. Multi-socket CPU systems bind thread worker pools to local NUMA nodes (numactl), preventing cross-socket interconnect bottlenecks.
Continue Reading
All Blog PostsBalancing Cost, Latency, Reliability, and Security
Operational trade-offs must be measured at the successful user outcome, inside the same security and quality boundary.
Claude 3.5 Sonnet vs. GPT-4o: Enterprise Deployment Teardown
An architectural and operational teardown comparing Claude 3.5 Sonnet and GPT-4o across latency, prompt caching, structured output guarantees, and enterprise privacy.
ColBERTv2 Paper Breakdown: Multi-Vector Late Interaction, MaxSim Scoring, and Residual Compression
Definitive retrieval-systems teardown of ColBERTv2 (Santhanam et al., arXiv:2112.01488): bridging single-vector bi-encoder compression loss and cross-encoder latency via token-level MaxSim late interaction, centroid-residual b-bit quantization, and PLAID inverted-list pruning.
