7-Layer System Breakdown

Local LLM Inference Engines (Ollama, llama.cpp & SGLang) Architecture Evidence

b2850

9f8e7d6

7-Layer Architecture

2. Subsystem Topology 3. 100x Execution Traces 5. Hot-Path Kernels

Subsystems (5)

1. Chapter 1 — GGUF Quantization Formats (Q4_K_M, IQ4_XS, K-quants)

GGUF Header Structure & Tensor Metadata Alignment

2 steps

Q4_K_M Super-Block & 6-Bit Scaling Factor Quantization

Importance Matrix (IQ4_XS) Vector Codebook Selection

Per-Channel Weight Dequantization Pipeline

2. Chapter 2 — FlashInfer CUDA Kernels & SIMD Vectorized Dequantization

AVX2/AVX-512 & ARM Neon SIMD Vector Unpacking

FlashInfer Compressed KV Cache Attention Kernels

Warp-Level Matrix Multiply-Accumulate (WMMA) Pipeline

Asynchronous Shared Memory Tiling & Tensor Core Execution

3. Chapter 3 — RadixTree KV Reuse & Prefix Caching for Multi-Turn Sessions

SGLang RadixTree Token Key-Value Prefix Hierarchy

LRU Cache Eviction & Node Splitting Mechanics

Multi-Turn Conversation Prefix Match & Softmax Masking

Paged KV Page Table Sharing across Parallel Samplers

4. Chapter 4 — Speculative Decoding & Draft Model Verification Loops

Target Engine & Lightweight Draft Model Pipeline

Parallel Speculative Verification Tree Evaluation

Stochastic Acceptance Criteria & Modified Target Sampling

Dynamic Draft Speculation Length Adjustment Loop

5. Chapter 5 — NUMA-Aware Memory Allocators & CPU/GPU Offloading Strategies

Layer-Wise Pipeline Offloading (mmap & CUDA Pinned Memory)

NUMA Node Memory Allocation & Inter-Socket Interconnects

Unified Memory Paging & GPU Memory Pressure Swap

Ollama Multi-Model Runner IPC & Virtual VRAM Allocator

Cataloged Systems

Inside Autonomous Agentic Code Editor: AST Indexing, Diff Generation, and Sandbox Verification Inside Apache Kafka Partition Storage & Zero-Copy Engine Inside Apple MLX Framework & Unified Memory Engine Inside Chatwoot: The 20-Diagram Enterprise Architecture & AI Copilot Master Blueprint Inside Anthropic Claude Computer Use: Vision Tokenization & Agent Sandboxing Inside CrewAI Multi-Agent Task Execution & Memory Orchestration Engine Inside Deep Agents: Harness Engine, DeltaChannel Checkpointing & Composite Filesystem Architecture Inside DeepSeek V3 / R1: MLA, DeepSeekMoE & GRPO Reasoning Inside DeerFlow 2.0: ByteDance SuperAgent Architecture & LangGraph Execution Engine Inside Kubernetes Control Plane & Envoy Service Mesh Inside LangGraph Agentic StateGraph & Cyclic Persistence Engine Inside Local LLM Inference: GGUF Quantization, FlashInfer Kernels & RadixTree KV Reuse Inside Model Context Protocol (MCP): Multi-Server Hub & Adapter Architecture Inside NVIDIA CUTLASS & Tensor Core GEMM Engine Inside Open-SWE: Autonomous Software Engineering Agent & Sandbox Runtime Inside OpenAI Codex: TUI, App Server, and Execution Boundaries Inside PyTorch Autograd Engine & Distributed Data Parallel (DDP)Inside Qiskit & Variational Quantum Circuit Simulator Inside Triton Inference Server & Dynamic Batching Engine Inside vLLM v0.10.2: AsyncLLM, Scheduling, KV Cache, and Model Execution Inside vLLM PagedAttention & Chunked Prefill Engine

System breakdown

Inside Local LLM Inference: GGUF Quantization, FlashInfer Kernels & RadixTree KV Reuse

An evidence-audited, 20-diagram interactive system breakdown tracing GGUF super-block quantization (Q4_K_M, IQ4_XS), SIMD & FlashInfer CUDA kernel dequantization, SGLang RadixTree prefix caching, speculative decoding verification loops, and NUMA-aware CPU/GPU memory offloading.

28 min read Verified 2026-07-21 4 primary sources

Canonical System Breakdown

Local LLM Inference Engines (Ollama, llama.cpp & SGLang) Architecture Evidence

RELEASE b2850

5 CHAPTERS · 15 NODES · 10 EDGES

Ingress Security

HMAC SHA256

Constant-time verify (<50ms)

Async Queueing

Sidekiq + Redis

Multi-queue priority isolation

Real-time Fanout

ActionCable Pub/Sub

Redis backplane state broadcast

AI Action Engine

ActionService LLM

Streaming copilot replies (SSE)

Core Stack:Rails 7.1•Sidekiq 7•Redis 7.2•PostgreSQL 16•Vue.js 3

5 Verified Architectural Claims

Press1-5to switch views

Archify Interactive Mapv2.16.0

Inside Local LLM Inference: GGUF Quantization, FlashInfer Kernels & RadixTree KV Reuse — Archify Interactive System Map

JSON IR Open Standalone

Hotkeys:R Route ProbeL Role LensP Play StoryS Visual StyleT ThemeF Stage

Source Pinned & Grounded