Mental model
Generators stream data items lazily on demand without allocating full arrays in RAM, while Context Managers enforce strict resource acquisition and release (RAII) patterns around execution blocks.
Theory
Context managers implement __enter__() and __exit__(exc_type, exc_val, exc_tb). If an exception occurs inside the with block, __exit__ receives the exception details and can choose to suppress it by returning True. The @contextlib.contextmanager decorator simplifies this using a yield generator inside a try...finally block.
Alternatives and trade-offs
- Lists / Materialized Arrays: Fast indexed access, but consumes massive memory for large datasets (O(N) RAM).
- Generators (
yield): O(1) memory complexity, producing items on demand during iteration.
Failure modes and misconceptions
- Swallowing exceptions accidentally: Returning a truthy value (
return True) from__exit__swallows exceptions raised inwithblocks. - Re-using single-use generators: Generators exhaust their state once iterated.
Decision scenario
Use @contextmanager with try...finally blocks for managing temporary files, database transactions, or lock resource lifecycle boundaries. Use yield generators for processing multi-gigabyte data files.
Learning outcomes
- Author custom context managers using
__enter__/__exit__and@contextmanager. - Implement memory-efficient
yieldgenerators for large data pipelines. - Ensure guaranteed resource cleanup across exception pathways.
Trade-offs
Generators provide O(1) memory efficiency for large stream processing, but generator objects are single-use and cannot be indexed directly.