GPU Hardware Architecture, CUDA Optimization & LLM Inference Infrastructure

Architecture & Hardware Microarchitecture

Building high-throughput, low-latency AI platforms requires aligning software algorithms with modern GPU hardware microarchitectures (NVIDIA Hopper H100/H200, Blackwell B200, AMD Instinct CDNA3).

The execution hierarchy routes data from high-bandwidth memory (HBM3e) down to Streaming Multiprocessors (SMs), where Tensor Cores execute fused matrix multiply-accumulate ($\text$) operations.

1. Hardware Comparison Matrix: H100 vs. H200 vs. Blackwell B200

2. Mathematical Performance Formulations: TFLOPS, AIFLOPS, TPS, TTFT, TPOT & MBU

Understanding production LLM inference performance requires isolating the Prefill Stage (Compute-Bound) from the Decode Stage (Memory-Bandwidth-Bound):

3. CUDA Kernel Optimization: Shared Memory & Warp Primitives

To achieve peak TFLOPS and MBU, GPU kernel engineers optimize low-level CUDA assembly:

Key Optimization Rules:

1. Shared Memory Bank Conflict Elimination: Shared memory is divided into 32 banks (4-byte or 8-byte width). Padding array dimensions (e.g. __shared__ float tile[32][33]) eliminates bank conflicts.
2. Warp Shuffle Register Communication: Use __shfl_sync(mask, val, srcLane) to swap data directly between warp threads without writing to shared memory.
3. Async Memory Pipeline (TMA): Use Tensor Memory Accelerator (TMA) hardware instructions (cp.async) to copy data directly from HBM to Shared Memory asynchronously, bypassing CPU/register overhead.

4. AI Infrastructure Engineering Competency Matrix

| Role Title | Core Competencies | Key Tools & Tech | Primary KPI Target | |---|---|---|---| | GPU Kernel Engineer | Custom CUDA C++, Triton, PTX assembly, CUTLASS, Shared memory bank conflict tuning | CUDA C++, PyTorch Triton, Nsight Compute, CUTLASS | Achieved TFLOPS / MBU (> 80%) | | LLM Inference Systems Engineer | vLLM, TensorRT-LLM, SGLang, PagedAttention, KV-cache quantization, continuous batching | vLLM, TensorRT-LLM, Triton Server, Python, C++ | P99 TTFT (< 200ms) & TPOT (< 15ms) | | AI Cluster Infrastructure Architect | Slurm, Kubernetes GPU Operators, InfiniBand NDR 400G, RoCEv2, NVLink Switch fabric | Slurm, K8s, Helm, Prometheus, DCGM, Terraform | Cluster GPU Utilization (> 85%) |

Decisions

| Decision | Required evidence | Review trigger | |---|---|---| | Deploy H200/B200 GPUs for memory-bound LLM decode serving. | Benchmark analysis confirming MBU $> 75%$ on 70B+ parameter models | TPOT exceeding 25ms SLA on H100 | | Use PagedAttention virtual memory block allocation for KV-cache. | VRAM memory audit showing zero external fragmentation | KV-cache Out-of-Memory crashes under batch load | | Enforce TMA asynchronous memory copies in custom CUDA kernels. | Nsight Compute trace showing zero memory stall cycles | Kernel compute pipeline idle time $> 15%$ |

Alternatives and trade-offs

H100 GPUs provide maximum FP8 compute TFLOPS for prompt prefill processing but can be memory-bandwidth-bound during single-stream decoding. H200 GPUs increase memory bandwidth to 4.8 TB/s, reducing TPOT by 1.4x at a higher hardware acquisition cost.

Failure modes

• Shared memory bank conflicts degrading GEMM kernel throughput by 4x.
• KV-cache VRAM fragmentation causing out-of-memory errors under large concurrency.
• Inter-GPU All-Reduce bottlenecks on non-NVLink PCIe interconnects.

Operational checklist

• [ ] Custom CUDA kernels verified with Nsight Compute for bank conflicts.
• [ ] Inference server batch size tuned to balance TTFT prefill latency and TPOT decode latency.
• [ ] NVLink interconnect bandwidth verified via nvidia-smi topo -m.
• [ ] Prometheus DCGM exporter monitoring GPU VRAM utilization and temperature metrics.

Connected practice

• System breakdowns: /systems/inside-vllm
• System breakdowns: /systems/inside-nvidia-cutlass

Sources

• vllm-paper
• langgraph-repo
• anthropic-effective-agents
• anthropic-trustworthy-agents