Python GIL vs Multiprocessing

Learning outcomes

• Analyze GIL thread lock contention
• Offload CPU tasks across multi-core processes

Mental model

The CPython Global Interpreter Lock (GIL) is a mutual exclusion lock preventing multiple OS threads from executing Python bytecode in parallel on multiple CPU cores.

Theory

For I/O-bound tasks (network requests, DB queries, file reads), threads yield the GIL during socket waits, making asyncio or threading highly efficient. For CPU-bound tasks (image processing, data transformation, matrix math), multiple Python threads contend for the GIL and run slower than single-threaded code. CPU workloads require multiprocessing or ProcessPoolExecutor to spawn separate Python interpreter instances on distinct CPU cores.

import concurrent.futures
import math

def compute_heavy_factors(n: int) -> int:
    count = 0
    for i in range(1, int(math.sqrt(n)) + 1):
        if n % i == 0:
            count += 1
    return count

def run_parallel_cpu_jobs(numbers: list[int]) -> list[int]:
    with concurrent.futures.ProcessPoolExecutor() as executor:
        results = list(executor.map(compute_heavy_factors, numbers))
    return results

Alternatives and trade-offs

• asyncio: Best for high-concurrency network I/O; low memory footprint (~kBs per coroutine).
• threading: Good for legacy blocking I/O calls; shares memory space but limited by GIL for CPU.
• multiprocessing: True multi-core CPU parallelism; high IPC serialization cost (pickle) and memory overhead per process.

Failure modes and misconceptions

1. Threading for CPU math: Using Python threading.Thread for matrix calculation does not utilize multiple CPU cores due to GIL lock contention.
2. IPC Pickle Overhead: Passing huge objects across ProcessPoolExecutor triggers slow pickle serialization.

Decision scenario

Combine asyncio for non-blocking HTTP network handling with ProcessPoolExecutor for offloading heavy image or data transformation tasks across CPU cores.

Learning outcomes

• Explain CPython GIL bytecode lock constraints across OS threads.
• Select between AsyncIO coroutines, threading, and multiprocessing.
• Optimize IPC data transfers for multi-process Python execution.

Trade-offs

Multiprocessing achieves true multi-core CPU parallelism, but incurs inter-process communication overhead and higher memory consumption per process worker.

Primary sources

• Python 3.12 AsyncIO Event Loop & Asynchronous I/O Specification — Python Software Foundation, verified 2026-07-22