Reference architecture

Quantization Frontiers & Hardware Execution Guide

A production guide for FP8, INT4, AWQ, GPTQ, and Unsloth model quantization, scaling factors, and Tensor Core GEMM kernel execution on H100 and A100 GPUs.

18 minVerified 2026-08-072 primary sources
A governed production AI reference architecture with observable, secured service boundaries.

Architecture

Model quantization reduces LLM memory bandwidth requirements by converting 16-bit floating-point weights (FP16/BF16) into lower-precision 8-bit (FP8) or 4-bit (INT4) formats.

High-precision FP16 weights
Per-block scale factor quantization
Packed INT4 / FP8 memory transfer
Dequantization in SRAM registers
Tensor Core GEMM matrix multiply
Output activation store
Conceptual teaching model synthesized from:vLLM V1 GPU model runner at v0.10.2vLLM AsyncLLM engine at v0.10.2

The Quantization Frontiers Guide evaluates per-tensor vs. per-channel scaling, FP8 E4M3 vs. E5M2 data formats, and Tensor Core hardware execution characteristics.

Baseline FP16 perplexity evaluation
AWQ per-channel quantization scale
Zero-point activation alignment
Perplexity degradation threshold check
Production VRAM & latency cutover
Conceptual teaching model synthesized from:vLLM V1 GPU model runner at v0.10.2vLLM AsyncLLM engine at v0.10.2

Quantization Format Comparison

| Format | Bits per Weight | Data Layout | H100 Native Support | VRAM Reduction | Perplexity Impact | |---|---|---|---|---|---| | FP16 / BF16 | 16 | Sign + Exponent + Mantissa | Yes | 1.0x (Baseline) | 0.00% (Baseline) | | FP8 (E4M3) | 8 | 1-4-3 (Weights & Act) | Yes (4th Gen Tensor Core) | 2.0x | < 0.05% | | FP8 (E5M2) | 8 | 1-5-2 (Gradients) | Yes (4th Gen Tensor Core) | 2.0x | < 0.10% | | INT4 AWQ | 4 | Activation-aware Group | No (Requires Dequant Kernel) | 3.5x | < 0.25% | | INT4 GPTQ | 4 | Second-order Hessian | No (Requires Dequant Kernel) | 3.6x | < 0.40% |

Scaling Factors & Dequantization

Quantization scales floating point values into discrete integer levels using block or per-channel scaling factors:

Quantized_Weight = Round( Weight / Scale ) + ZeroPoint
Dequantized_Weight = ( Quantized_Weight - ZeroPoint ) * Scale

In AWQ (Activation-aware Weight Quantization), salient weight channels are protected based on activation magnitude, preserving reasoning accuracy while compressing weights to 4 bits.

Decisions

| Decision | Required evidence | Review trigger | |---|---|---| | Adopt FP8 (E4M3) for H100 inference clusters to double KV-cache and batch throughput. | Benchmark showing < 0.1 perplexity loss on target evaluation set | Deployment on NVIDIA H100 / L40S hardware | | Use INT4 AWQ for edge or single-GPU VRAM constrained deployments. | Memory profiling showing 3.5x reduction under target context window | VRAM exhaustion on target GPU hardware | | Avoid 4-bit quantization on small models (< 3B parameters). | Perplexity evaluation showing > 1.5 point degradation | Workload deploying small sub-3B models |

Alternatives and trade-offs

FP8 provides native hardware GEMM execution on NVIDIA Hopper GPUs without dequantization latency penalties. INT4 AWQ saves more VRAM but requires unpacking 4-bit weights into FP16 in GPU SRAM before matrix multiplication.

Failure modes

  • Applying FP8 without per-tensor scaling factors leading to activation overflow.
  • Quantizing small fine-tuned models (< 3B parameters) resulting in reasoning collapse.
  • Misaligned memory layouts causing memory bank conflicts in CUDA dequantization kernels.

Operational checklist

  • [ ] Perplexity degradation is verified against baseline FP16 on golden evaluation datasets.
  • [ ] GPU architecture supports target quantization kernels (NVIDIA Hopper for native FP8).
  • [ ] VRAM savings translate directly to increased batch size or context window capacity.
  • [ ] Dequantization kernel overhead is measured under batch size 1 latency constraints.

Connected practice

Sources

  • vllm-v0102-model-runner
  • vllm-v0102-async-engine