AI Hardware Accelerators & Compute Architecture

Architecture

Scalable AI engineering requires matching model workload characteristics to underlying accelerator hardware topologies.

HBM3e high-bandwidth memory

SRAM L2 cache buffer

Streaming Multiprocessor register file

Tensor Core GEMM matrix execution

NVLink cross-GPU interconnect

Host PCIe host interface

Conceptual teaching model synthesized from:vLLM V1 GPU model runner at v0.10.2 vLLM AsyncLLM engine at v0.10.2

The AI Hardware Accelerators Guide evaluates memory bandwidth (HBM3e), FP8 Tensor Core throughput, NVLink interconnect topologies, and Roofline model arithmetic intensity limits across NVIDIA H100 SXM, AMD MI300X, and Google TPU v5p.

Compute arithmetic intensity (FLOPs/Byte)

Identify memory-bandwidth bound regime

Profile Tensor Core MAC utilization

Tune CUDA block tile sizes

Verify FLOPs efficiency ceiling

Conceptual teaching model synthesized from:vLLM V1 GPU model runner at v0.10.2 vLLM AsyncLLM engine at v0.10.2

Hardware Specs Comparison

| Accelerator | VRAM Capacity | Memory Bandwidth | FP8 Tensor TFLOPs | Interconnect Bandwidth | Primary Bottleneck | |---|---|---|---|---|---| | NVIDIA H100 SXM | 80 GB HBM3 | 3.35 TB/s | 1,979 TFLOPs | 900 GB/s (NVLink 4) | Memory Capacity | | NVIDIA H200 SXM | 141 GB HBM3e | 4.80 TB/s | 1,979 TFLOPs | 900 GB/s (NVLink 4) | Compute Bound | | AMD MI300X | 192 GB HBM3 | 5.30 TB/s | 2,610 TFLOPs | 896 GB/s (Infinity Fabric) | Software Ecosystem | | Google TPU v5p | 95 GB HBM3 | 2.76 TB/s | 459 TFLOPs (BF16) | 4,800 Gbps (ICI 3D Torus) | Inter-Chip Latency |

Roofline Model & Arithmetic Intensity

A workload's execution performance is constrained by either memory bandwidth or compute FLOPs capacity based on its Arithmetic Intensity:

Arithmetic Intensity = Total Floating Point Operations (FLOPs) / Total Memory Transferred (Bytes)

Memory-Bound Regime (Low Batch Size Generation): Arithmetic intensity is low ($\text < 100$). Performance is capped by HBM bandwidth.
Compute-Bound Regime (Prefill & Training): Arithmetic intensity is high ($\text > 200$). Performance is capped by Tensor Core matrix multiplication speed.

Decisions

| Decision | Required evidence | Review trigger | |---|---|---| | Select NVIDIA H200 or AMD MI300X for large LLM inference (70B+ parameters) at low batch sizes. | Memory profiling showing > 4.5 TB/s bandwidth utilization | Deploying 70B+ model inference | | Use NVLink 8-GPU SXM topology for tensor parallel workloads. | Interconnect profiling showing < 5% All-Reduce communication stall | Scaling model across multiple GPUs | | Profile arithmetic intensity on target workloads before selecting GPU cluster size. | Roofline analysis identifying memory vs compute bottleneck | Infrastructure capacity planning |

Alternatives and trade-offs

NVIDIA H100/H200 provides the most mature software ecosystem (CUDA, vLLM, TensorRT-LLM). AMD MI300X offers 192GB VRAM per GPU allowing single-node 70B inference, but requires ROCm kernel tuning. Google TPU v5p excels at large-scale pod pre-training using Megablox and JAX.

Failure modes

Deploying low-batch inference workloads on compute-optimized GPUs without speculative decoding or PagedAttention.
Running tensor parallel inference over PCIe connections instead of high-bandwidth NVLink/Infinity Fabric interconnects.
Ignoring VRAM capacity limits causing out-of-memory (OOM) crashes during long-context prefill sequences.

Operational checklist

[ ] Workload arithmetic intensity is plotted on a Roofline model graph.
[ ] Inter-GPU communication uses high-bandwidth NVLink / Infinity Fabric bridges.
[ ] VRAM allocation accounts for model weights, KV-cache, and activation buffers.
[ ] Tensor Core GEMM tiling sizes align with hardware warp boundaries.

Connected practice

Labs: /labs/agent-loop-tool-selection
System breakdowns: /systems/inside-vllm

Sources

vllm-v0102-model-runner
vllm-v0102-async-engine