DeepSeek-V3 & R1 Paper Breakdown

A commit-pinned, mathematical and architectural breakdown of DeepSeek-V3 (671B total parameters, 37B active per token) and DeepSeek-R1 reasoning model.

1. Multi-Head Latent Attention (MLA)

Traditional Multi-Head Attention (MHA) stores full key and value tensors for every head in the KV cache, leading to severe memory bandwidth bottlenecks during generation.

MLA compresses the KV cache into a low-rank latent vector c_t^ per token:

c_t^{KV} = W^{DKV} h_t

Where:

• h_t is the hidden state vector at token position t.
• W^ is the down-projection matrix into latent dimension d_c (where d_c is much smaller than n_h * d_h).

During generation, the compressed latent vector c_t^ is cached directly instead of full key/value matrices. Keys and values are reconstructed on-the-fly using up-projection matrix W^:

K_t^C = W^{UK} c_t^{KV}

Memory Cache Efficiency Gains

| Attention Variant | KV Cache Bytes Per Token / Layer | Max Concurrency (8x H100 80GB) | |---|---|---| | Standard MHA (Llama 3 70B) | ~1,280 bytes | ~128 concurrent streams | | Grouped Query Attention (GQA, 8 groups) | ~160 bytes | ~1,024 concurrent streams | | DeepSeek MLA (Compressed Latent) | ~57.6 bytes | ~2,880 concurrent streams |

2. Auxiliary-Loss-Free MoE Load Balancing

Standard Mixture-of-Experts (MoE) models use an auxiliary load balancing loss penalty to prevent gate collapse (all tokens routing to the same expert). However, auxiliary losses degrade model performance because they force sub-optimal expert assignments.

DeepSeek-V3 introduces bias-adjusted dynamic routing:

s_{i,t} = TopK(Softmax(h_t^T w_i + b_i), K)

Where b_i is a dynamic bias term updated online based on expert utilization:

• If expert i is overloaded, b_i is decremented.
• If expert i is underutilized, b_i is incremented.

This guarantees perfect load distribution across all 256 routed experts without adding loss constraints to the gradient graph.

3. DualPipe Overlapped Compute-Communication Pipeline

To scale across 2,048 H800 GPUs, DeepSeek-V3 introduces DualPipe, an overlapping scheme that hides inter-node All-to-All communication latency behind Tensor Core GEMM matrix multiplication.

Execution Schedule

GPU 0: [Forward Chunk A] ==> [Dispatch A (All-to-All)] ==> [Forward Chunk B]
                              | (Overlapped Network Transmit) |
GPU 1:                        +============================> [Recv Chunk A]

By partitioning each micro-batch into forward and backward chunks, DualPipe achieves near-zero communication bubble overhead during training.