Concept lesson

Performance Profiling & Flamegraphs

CPU flamegraphs, off-CPU profiling, stack sampling, and bottleneck identification.

lesson
Freshness: current15 min read
Mastery
not started · 0%

Learning outcomes

  • Generate and analyze CPU flamegraphs using perf
  • Identify off-CPU blocking latency bottlenecks

Mental model

A Flamegraph is a visual representation of sampled stack traces. The X-axis spans the full sample population (wider boxes indicate functions consuming more CPU time), while the Y-axis shows call stack depth from top to bottom.

Sample stack traces at 99Hz via Linux perf
Aggregate stack trace samples into frequency fold format
Render SVG Flamegraph (X=Sample Width, Y=Stack Depth)
Identify wide plateau boxes consuming CPU
Optimize hot paths in application code
Conceptual teaching model synthesized from:Linux Kernel Kernel.org Official Architecture & Systems Documentation

Theory

  • On-CPU Profiling: Samples active CPU instruction pointers at regular intervals (e.g. 99 Hertz). Identifies CPU-bound algorithm bottlenecks.
  • Off-CPU Profiling: Traces time spent by threads sleeping, blocked on disk I/O, waiting for locks, or waiting on network sockets.
  • perf (Linux Performance Counters): The standard kernel profiling utility for capturing hardware performance counters and stack traces.
# Capture CPU stack traces across all CPUs at 99Hz for 30 seconds
perf record -F 99 -a -g -- sleep 30

# Collapse stack traces and generate Flamegraph SVG
perf script | stackcollapse-perf.pl | flamegraph.pl > cpu_flamegraph.svg

# Profile Off-CPU blocking time using eBPF offcputime
offcputime-bpfcc -df 30 > offcpu.stacks
flamegraph.pl --color=io offcpu.stacks > offcpu_flamegraph.svg

Alternatives and trade-offs

  • Instrumented Micro-benchmarks: High precision for isolated functions; distorts runtime behavior and cannot capture kernel locks.
  • perf Flamegraph Sampling: Low overhead (under 1% CPU impact); provides immediate visual identification of hot code paths across user and kernel space.

Failure modes and misconceptions

  1. Missing Frame Pointers (-fomit-frame-pointer): Compiling C/C++ applications with -fomit-frame-pointer destroys stack frame pointers, resulting in broken, truncated flamegraph stacks ([unknown]). Always compile with -fno-omit-frame-pointer.
  2. Ignoring Off-CPU Latency: Focusing exclusively on CPU Flamegraphs fails to reveal why an application is slow when threads spend 95% of their time sleeping on database I/O or mutex contention. Use Off-CPU flamegraphs for I/O bottlenecks.
Reflect before revealing the guide

Decision scenario

Generate CPU and Off-CPU Flamegraphs using perf and eBPF tools to visually diagnose application performance bottlenecks in production.

Learning outcomes

  • Sample kernel and application stack traces using perf.
  • Interpret Flamegraph X-axis (sample width) and Y-axis (call stack depth).
  • Generate Off-CPU Flamegraphs to isolate I/O blocking and lock contention.

Trade-offs

Flamegraph sampling provides clear visual identification of system bottlenecks, but requires compiling binaries with frame pointers (-fno-omit-frame-pointer).

Evidence assessment

Theory and decision mastery

not-started · 0%
theory0%
decision0%
activityNot mapped
projectNot mapped
1. What is the core architectural principle governing flamegraphs performance profiling?
2. What primary operational trade-off must be managed when configuring flamegraphs performance profiling?
3. Which failure mode is most commonly observed when flamegraphs performance profiling is misconfigured?

Decision scenario

You are designing a high-concurrency production system requiring reliable execution of flamegraphs performance profiling under heavy traffic load.

Which architectural decision ensures maximum resilience, scalability, and system stability?

Primary sources