NVMe Storage & I/O Schedulers

Learning outcomes

• Select optimal I/O schedulers for NVMe SSD storage arrays
• Bypass OS page cache using O_DIRECT for database engines

Mental model

Legacy spinning hard drives relied on complex single-queue kernel I/O schedulers (CFQ) to reorder read head movements. Modern NVMe SSDs utilize the Multi-Queue Block Layer (blk-mq), providing up to 64,000 submission/completion hardware queues running in parallel across CPU cores.

Theory

• NVMe (Non-Volatile Memory Express): High-speed storage protocol over PCIe buses supporting 64,000 parallel queues with 64,000 commands per queue.
• I/O Schedulers:
  • none: Bypasses software queue scheduling entirely; passes I/O requests directly to hardware. Recommended default for fast NVMe SSDs.
  • mq-deadline: Prevents I/O starvation by setting hard execution deadline limits for reads and writes.
  • kyber: Target-latency scheduler designed for fast flash storage.
• Direct I/O (O_DIRECT): Bypasses the Linux page cache, transferring data directly between application RAM buffers and NVMe hardware storage controllers.

// C example: Opening file with O_DIRECT for database zero-copy I/O
#define _GNU_SOURCE
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>

int main() {
    // Open file with O_DIRECT to bypass OS page cache
    int fd = open("db_wal.log", O_WRONLY | O_CREAT | O_DIRECT, 0644);
    
    // Memory buffer MUST be aligned to 512-byte / 4096-byte sector boundaries
    void *buffer;
    posix_memalign(&buffer, 4096, 4096);
    
    write(fd, buffer, 4096);
    close(fd);
    free(buffer);
    return 0;
}

Alternatives and trade-offs

• Buffered I/O (Page Cache): Default mode; caches file blocks in RAM, fast reads, but risks data loss during power failures unless explicitly fsync()ed.
• Direct I/O (O_DIRECT): Complete application control over disk writes; bypasses RAM cache, requires memory-aligned buffers (4KB boundary alignment).

Failure modes and misconceptions

1. Unaligned Memory Buffers with O_DIRECT: Passing unaligned heap memory pointers to write() with O_DIRECT causes system call failures returning EINVAL (Invalid argument).
2. Using Legacy Single-Queue Schedulers on NVMe: Setting legacy cfq or bfq schedulers on NVMe storage throttles multi-core throughput down to single-queue bottlenecks.

Decision scenario

Configure the none I/O scheduler and use O_DIRECT with sector-aligned memory buffers for high-performance database WAL logging on NVMe storage drives.

Learning outcomes

• Structure high-speed storage access using NVMe Multi-Queue hardware layers.
• Select optimal I/O schedulers (none, mq-deadline, kyber) for NVMe SSD arrays.
• Implement zero-page-cache file writes using O_DIRECT and aligned memory buffers.

Trade-offs

NVMe multi-queue architecture and O_DIRECT eliminate page cache overhead, but demand rigorous 4KB buffer alignment and application-managed memory buffers.

Primary sources

• Linux Kernel Kernel.org Official Architecture & Systems Documentation — Linux Kernel Organization, verified 2026-07-23