Mental model
Legacy spinning hard drives relied on complex single-queue kernel I/O schedulers (CFQ) to reorder read head movements. Modern NVMe SSDs utilize the Multi-Queue Block Layer (blk-mq), providing up to 64,000 submission/completion hardware queues running in parallel across CPU cores.
Application issues Storage Write
Bypass Page Cache via O_DIRECT (Optional)
blk-mq assigns Hardware Submission Queue
Execute via NVMe PCIe Controller
Hardware Completion Interrupt fires
Theory
- NVMe (
Non-Volatile Memory Express): High-speed storage protocol over PCIe buses supporting 64,000 parallel queues with 64,000 commands per queue. - I/O Schedulers:
none: Bypasses software queue scheduling entirely; passes I/O requests directly to hardware. Recommended default for fast NVMe SSDs.mq-deadline: Prevents I/O starvation by setting hard execution deadline limits for reads and writes.kyber: Target-latency scheduler designed for fast flash storage.
- Direct I/O (
O_DIRECT): Bypasses the Linux page cache, transferring data directly between application RAM buffers and NVMe hardware storage controllers.
c(21 lines)
1// C example: Opening file with O_DIRECT for database zero-copy I/O
2#define _GNU_SOURCE
3#include <fcntl.h>
4#include <unistd.h>
5#include <stdio.h>
6#include <stdlib.h>
7
8int main() {
9 // Open file with O_DIRECT to bypass OS page cache
10 int fd = open("db_wal.log", O_WRONLY | O_CREAT | O_DIRECT, 0644);
11
12 // Memory buffer MUST be aligned to 512-byte / 4096-byte sector boundaries
13 void *buffer;
14 posix_memalign(&buffer, 4096, 4096);
15
16 write(fd, buffer, 4096);
17 close(fd);
18 free(buffer);
19 return 0;
20}
Alternatives and trade-offs
- Buffered I/O (Page Cache): Default mode; caches file blocks in RAM, fast reads, but risks data loss during power failures unless explicitly
fsync()ed. - Direct I/O (
O_DIRECT): Complete application control over disk writes; bypasses RAM cache, requires memory-aligned buffers (4KB boundary alignment).
Failure modes and misconceptions
- Unaligned Memory Buffers with
O_DIRECT: Passing unaligned heap memory pointers towrite()withO_DIRECTcauses system call failures returningEINVAL(Invalid argument). - Using Legacy Single-Queue Schedulers on NVMe: Setting legacy
cfqorbfqschedulers on NVMe storage throttles multi-core throughput down to single-queue bottlenecks.
Reflect before revealing the guide
Decision scenario
Configure the none I/O scheduler and use O_DIRECT with sector-aligned memory buffers for high-performance database WAL logging on NVMe storage drives.
Learning outcomes
- Structure high-speed storage access using NVMe Multi-Queue hardware layers.
- Select optimal I/O schedulers (
none,mq-deadline,kyber) for NVMe SSD arrays. - Implement zero-page-cache file writes using
O_DIRECTand aligned memory buffers.
Trade-offs
NVMe multi-queue architecture and O_DIRECT eliminate page cache overhead, but demand rigorous 4KB buffer alignment and application-managed memory buffers.