Linux I/O Multiplexing & epoll

Learning outcomes

• Build O(1) scalable event loops using Linux epoll
• Distinguish edge-triggered from level-triggered I/O modes

Mental model

To handle tens of thousands of concurrent TCP sockets without spawning a thread per socket, Linux uses I/O multiplexing. epoll registers interest in socket file descriptors in a kernel Red-Black tree and returns ready sockets via a ready list in $O(1)$ time.

Theory

• select() / poll(): Copies file descriptor sets between user and kernel space on every call; performs linear $O(N)$ scanning over all descriptors. Fails at scale.
• epoll (epoll_create, epoll_ctl, epoll_wait): Maintains in-kernel Red-Black tree of watched descriptors. Operates in $O(1)$ time relative to total watched sockets.
• Level-Triggered (LT): Default mode; epoll_wait() continues notifying as long as unread data remains in the socket buffer.
• Edge-Triggered (ET): epoll_wait() notifies ONCE when new data arrives. Requires reading the socket until EAGAIN or EWOULDBLOCK.

// C snippet: Creating an epoll event loop
#include <sys/epoll.h>
#include <unistd.h>
#include <stdio.h>

#define MAX_EVENTS 64

int main() {
    int epoll_fd = epoll_create1(0);
    struct epoll_event ev, events[MAX_EVENTS];

    // Watch socket_fd for read events (Level Triggered)
    ev.events = EPOLLIN;
    ev.data.fd = 0; // STDIN
    epoll_ctl(epoll_fd, EPOLL_CTL_ADD, 0, &ev);

    // Wait for events
    int nfds = epoll_wait(epoll_fd, events, MAX_EVENTS, 5000);
    printf("Active file descriptors: %d\n", nfds);

    close(epoll_fd);
    return 0;
}

Alternatives and trade-offs

• Thread-Per-Socket: Simple synchronous code; severe RAM overhead (1MB stack per thread) and CPU context switching degrade performance past 1,000 threads.
• epoll Event Loop (Uvicorn / Node.js / Nginx): Handles 100,000+ concurrent connections on a single thread with minimal RAM footprint.

Failure modes and misconceptions

1. Edge-Triggered Starvation & Deadlock: In Edge-Triggered (EPOLLET) mode, failing to loop read() until receiving EAGAIN leaves remaining buffer data unread, causing the socket to hang indefinitely.
2. Blocking Sockets in epoll: Registering blocking file descriptors in an epoll event loop blocks the entire event loop thread if a read operation blocks. Sockets MUST be configured with O_NONBLOCK.

Decision scenario

Build non-blocking high-concurrency network servers using Linux epoll in Level-Triggered mode to achieve $O(1)$ event multiplexing without socket starvation bugs.

Learning outcomes

• Compare $O(N)$ file descriptor polling (select/poll) with $O(1)$ epoll.
• Distinguish Level-Triggered (LT) from Edge-Triggered (ET) event notification modes.
• Configure non-blocking file descriptors (O_NONBLOCK) for event loop processing.

Trade-offs

epoll enables scalable $O(1)$ socket event loops for high-concurrency servers, but requires asynchronous non-blocking event handling state machines.

Primary sources

• Linux Kernel Kernel.org Official Architecture & Systems Documentation — Linux Kernel Organization, verified 2026-07-23