Context Managers & Generators

Learning outcomes

• Build custom context managers with try/finally
• Stream large datasets lazily with generators

Mental model

Generators stream data items lazily on demand without allocating full arrays in RAM, while Context Managers enforce strict resource acquisition and release (RAII) patterns around execution blocks.

Theory

Context managers implement __enter__() and __exit__(exc_type, exc_val, exc_tb). If an exception occurs inside the with block, __exit__ receives the exception details and can choose to suppress it by returning True. The @contextlib.contextmanager decorator simplifies this using a yield generator inside a try...finally block.

import time
from contextlib import contextmanager
from typing import Generator

@contextmanager
def execution_timer(label: str) -> Generator[None, None, None]:
    start = time.perf_counter()
    try:
        yield
    finally:
        elapsed = (time.perf_counter() - start) * 1000
        print(f"[{label}] Completed in {elapsed:.2f}ms")

def stream_large_dataset(limit: int) -> Generator[dict, None, None]:
    for i in range(limit):
        yield {"id": i, "payload": f"record_{i}"}

def run_example():
    with execution_timer("Processing Pipeline"):
        for record in stream_large_dataset(5):
            print(f"Processed: {record['id']}")

Alternatives and trade-offs

• Lists / Materialized Arrays: Fast indexed access, but consumes massive memory for large datasets (O(N) RAM).
• Generators (yield): O(1) memory complexity, producing items on demand during iteration.

Failure modes and misconceptions

1. Swallowing exceptions accidentally: Returning a truthy value (return True) from __exit__ swallows exceptions raised in with blocks.
2. Re-using single-use generators: Generators exhaust their state once iterated.

Decision scenario

Use @contextmanager with try...finally blocks for managing temporary files, database transactions, or lock resource lifecycle boundaries. Use yield generators for processing multi-gigabyte data files.

Learning outcomes

• Author custom context managers using __enter__/__exit__ and @contextmanager.
• Implement memory-efficient yield generators for large data pipelines.
• Ensure guaranteed resource cleanup across exception pathways.

Trade-offs

Generators provide O(1) memory efficiency for large stream processing, but generator objects are single-use and cannot be indexed directly.

Primary sources

• Python 3.12 AsyncIO Event Loop & Asynchronous I/O Specification — Python Software Foundation, verified 2026-07-22