paper breakdown

Ring Attention Paper Breakdown: Blockwise Distributed Transformers for Near-Infinite Context

Definitive distributed-systems and arithmetic-intensity teardown of Ring Attention (Liu et al., arXiv:2310.01889): overcoming single-GPU HBM limits via ring-topology P2P KV block circulation, overlapped communication and FlashAttention compute, and exact online softmax rescaling.

16 min readVerified 2026-09-292 primary sources
Technical publication illustration.

Ring Attention Paper Breakdown

A distributed-systems and hardware-interconnect breakdown of Ring Attention with Blockwise Transformers for Near-Infinite Context (Liu, Zaharia, & Abbeel, ICLR 2024 / arXiv:2310.01889), the foundational sequence-parallelism architecture that scales exact Transformer context lengths linearly with the number of GPUs (L_total = N_hosts * c) with zero communication overhead when compute dominates interconnect transfer.

1. The Single-GPU HBM Context Wall

Even though FlashAttention eliminates the O(L^2) intermediate attention matrix in GPU High Bandwidth Memory (HBM), a single GPU's HBM capacity still bounds maximum sequence length L because the activations, Queries (Q), Keys (K), Values (V), and feed-forward intermediates scale as O(L * d) per layer:

text(3 lines)
1Single-GPU Activation + KV Footprint ( Model, 32 Layers, BF16, L = 1,000,000 tokens):
2 Q, K, V + MLP Activations > 500 GB >> 80 GB (H100 HBM3 capacity)

Tensor Parallelism (Megatron-LM style) splits head dimensions d across up to 8 intra-node GPUs, but requires synchronous AllReduce barriers twice per layer and cannot scale sequence length across hundreds of multi-node hosts.

2. Ring Topology & Asynchronous Double-Buffered KV Circulation

Ring Attention partitions a sequence of length L = N_h * c into N_h contiguous token blocks of length c, assigning block i (Q_i, K_i, V_i in R^{c x d}) to device rank_i arranged in a logical ring topology:

text(6 lines)
1 [Rank 0: Holds Q_0] --- Send(K,V) ---> [Rank 1: Holds Q_1]
2 ^ |
3 | Send(K,V)
4 Send(K,V) v
5 [Rank 3: Holds Q_3] <-- Send(K,V) --- [Rank 2: Holds Q_2]

The N_h-Step Ring Execution Schedule

Notice a key algebraic property of blockwise attention: to compute the output block O_i for local queries Q_i, rank_i never needs to move Q_i or O_i off-chip. It only needs to stream each remote (K_j, V_j) block past Q_i in N_h ring stages (s = 0, 1, ..., N_h - 1):

text(7 lines)
1At Ring Step s (on Device Rank i):
2 Active KV Block in Buffer A : (K_j, V_j) where j = (i - s) mod N_h
3 Concurrent Compute Stream : FlashAttention_Update(Q_i, K_j, V_j, m_i, l_i, O_i)
4 Concurrent NCCL P2P Stream : Async_Send(K_j, V_j -> Rank (i+1) mod N_h)
5 Async_Recv(K_{j-1}, V_{j-1} <- Rank (i-1) mod N_h into Buffer B)
6 End of Step s : Swap pointers (Buffer A <-> Buffer B)

Because O_i, running row-max m_i in R^c, and running normalizer l_i in R^c stay pinned in rank_i's local SRAM/HBM, each (K_j, V_j) block is discarded immediately after its inner FlashAttention tile pass completes!

3. Cross-Device Online Softmax Accumulator Math

Ring Attention relies on the associativity and commutativity of Online Softmax Rescaling. Across ring steps s = 0 ... N_h - 1, device i maintains three state tensors for its c query tokens:

  • m_i^{(s)} in R^{c x 1}: Running row-wise maximum logit seen across all KV blocks visited so far.
  • l_i^{(s)} in R^{c x 1}: Running row-wise exponential sum normalizer.
  • O_i^{(s)} in R^{c x d}: Running unnormalized attention output accumulator.

When remote block (K_j, V_j) arrives at step s:

text(8 lines)
11. Local Block Score : S_{i,j} = (Q_i * K_j^T) / sqrt(d) in R^{c x c}
22. Block Row Max : m_tilde = rowmax(S_{i,j}) in R^{c x 1}
33. Global Running Max : m_i^{(s)} = max( m_i^{(s-1)}, m_tilde ) in R^{c x 1}
44. Rescaling Factor : alpha = exp( m_i^{(s-1)} - m_i^{(s)} ) in (0, 1]^c
55. Local Block Exponentials : P_{i,j} = exp( S_{i,j} - m_i^{(s)} ) in R^{c x c}
66. Running Normalizer : l_i^{(s)} = alpha (*) l_i^{(s-1)} + rowsum(P_{i,j})
77. Running Output Rescale : O_i^{(s)} = diag(alpha) * O_i^{(s-1)} + P_{i,j} * V_j

After all N_h ring steps complete (s = N_h - 1), device i performs a single final division O_i = O_i^{(N_h - 1)} / l_i^{(N_h - 1)}, producing bit-for-bit identical mathematical outputs to monolithic single-GPU attention over length L = N_h * c without ever materializing more than one (K_j, V_j) block in memory.

4. Zero-Overhead Overlapping Condition & Roofline Analysis

When is the P2P ring communication latency T_comm completely hidden behind the blockwise FlashAttention compute time T_comp?

Let:

  • c = block length (tokens per host)
  • d = hidden dimension
  • F = effective Tensor Core FLOP/s per GPU (e.g., 600 TFLOPS in BF16 on H100)
  • B = unidirectional P2P interconnect bandwidth in Bytes/s (e.g., 450 GB/s NVLink or 50 GB/s InfiniBand)

Compute vs. Communication Time per Ring Step

  1. Compute Time (T_comp): Computing Q_i K_j^T (2 * c^2 * d FLOPs) and P_{i,j} V_j (2 * c^2 * d FLOPs) requires 4 * c^2 * d FLOPs (plus causal/backward passes):
text(2 lines)
1T_comp = (4 * c^2 * d) / F
  1. Communication Time (T_comm): Sending both K_j and V_j (2 * c * d elements in 2-byte FP16/BF16 = 4 * c * d Bytes) over bandwidth B:
text(2 lines)
1T_comm = (4 * c * d) / B
  1. Zero-Overhead Condition (T_comp >= T_comm):
text(2 lines)
1(4 * c^2 * d) / F >= (4 * c * d) / B ===> c >= F / B

Notice that d cancels out completely! As long as the per-GPU block length c exceeds the hardware ridge ratio F / B, communication is 100% hidden behind computation:

| Interconnect Tier | Effective F (BF16) | Link Bandwidth B | Critical Block Size c_min = F / B | Practical Block Size c | Communication Overhead | |---|---|---|---|---|---| | Intra-Node NVLink 4.0 (H100) | 600 TFLOPS | 450 GB/s | 1,334 tokens | c = 4,096 | 0.0% (Compute-Bound) | | Multi-Node 400GbE / NDR IB | 600 TFLOPS | 50 GB/s | 12,000 tokens | c = 16,384 | 0.0% (Compute-Bound) | | Standard 100GbE RoCEv2 | 312 TFLOPS (A100) | 12.5 GB/s | 24,960 tokens | c = 32,768 | 0.0% (Compute-Bound) |

5. Causal Masking Imbalance & Striped / ZigZag Attention

In autoregressive causal attention (m >= n), a naive contiguous block assignment causes severe compute load imbalance:

  • At step s = 0, all N_h ranks compute lower-triangular diagonal blocks (50% active FLOPs).
  • At step s > 0, ranks where block index i < j (Q_i precedes K_j in the sequence) are 100% masked out (0% active FLOPs) while ranks with i > j must compute full dense c x c blocks (100% active FLOPs).

Production Ring Attention deployments (such as Striped Attention and Megatron Context Parallelism ZigZag Partitioning) assign tokens to rank i either modulo-striped (token t -> rank (t mod N_h)) or symmetrically paired from the front and back of the sequence ({block_i, block_{2*N_h - 1 - i}}), guaranteeing that every GPU computes the exact same number of unmasked causal tokens at every single ring step.