CPU Cache Alignment & NUMA Nodes

Learning outcomes

• Eliminate false sharing in multi-threaded data structures
• Enforce NUMA memory affinity for high-performance workloads

Mental model

CPUs do not read RAM byte-by-byte; CPUs fetch memory in fixed 64-byte Cache Lines into L1, L2, and L3 caches. In multi-socket servers, NUMA (Non-Uniform Memory Access) means accessing local RAM connected to the CPU socket is significantly faster than accessing remote RAM across inter-socket interconnects.

Theory

• False Sharing: Occurs when two independent threads running on different CPU cores update distinct variables that happen to reside within the same 64-byte cache line. The MESI cache coherency protocol continuously invalidates and bounces the cache line back and forth between cores, degrading performance.
• Cache Padding: Padding struct members to 64-byte boundaries ensures distinct variables reside on separate cache lines.
• NUMA Binding (numactl): Pinning application process memory and execution threads to the same NUMA node to guarantee local RAM access.

// C example: Eliminating False Sharing via Cache Line Alignment
#include <stdio.h>
#include <pthread.h>

// Struct with 64-byte alignment to prevent False Sharing
struct alignas(64) ThreadCounter {
    long count;
    char padding[56]; // Pad struct to fill 64-byte cache line
};

struct ThreadCounter counters[2];

void* worker(void* arg) {
    long id = (long)arg;
    for (long i = 0; i < 100000000; i++) {
        counters[id].count++;
    }
    return NULL;
}

Alternatives and trade-offs

• Unaligned Data Structures: Compact memory usage; vulnerable to false sharing performance penalties under concurrent multi-threaded writes.
• Cache-Line Aligned Structs (alignas(64)): Eliminates false sharing bounce; slightly increases struct memory footprint.

Failure modes and misconceptions

1. False Sharing Cache Bouncing: Multi-threaded counters placed side-by-side in an unpadded array cause throughput drops up to 20x due to continuous MESI cache line invalidation loops.
2. Cross-NUMA Memory Access Penalties: Running latency-sensitive memory stores across arbitrary CPU sockets without NUMA binding causes remote memory access latency spikes.

Decision scenario

Align thread counters to 64-byte cache line boundaries and pin database process memory using numactl --membind to eliminate false sharing and remote NUMA latency.

Learning outcomes

• Structure C/C++ data structures to align with CPU 64-byte cache lines.
• Detect and eliminate multi-threaded False Sharing using cache line padding.
• Bind memory allocations and execution threads to local NUMA nodes.

Trade-offs

Hardware cache line alignment and NUMA pinning deliver maximum multi-threaded throughput, but require explicit memory layout design and CPU topology awareness.

Primary sources

• Linux Kernel Kernel.org Official Architecture & Systems Documentation — Linux Kernel Organization, verified 2026-07-23