Message Queues & Celery Task Pools

Mental model

Message queues decouple HTTP request-response cycles from slow background tasks (e.g. video processing, email generation, PDF rendering). Web apps push task payloads into a queue broker (Redis), while worker processes execute tasks asynchronously.

Theory

Celery connects to a Broker (Redis / RabbitMQ) for task transport and a Result Backend (Redis / PostgreSQL) to record task completion states (PENDING, STARTED, SUCCESS, FAILURE).

Alternatives and trade-offs

• FastAPI BackgroundTasks: Runs in-process on the Uvicorn worker process; lost if process crashes mid-task, unsuitable for heavy CPU loads.
• Celery + Redis: Out-of-process distributed scaling, task persistence, retries; requires maintaining extra worker infrastructure.

Failure modes and misconceptions

1. Default worker_prefetch_multiplier = 4 Starvation: When tasks take minutes to complete, a worker prefetching 4 long tasks starves other idle workers. Set worker_prefetch_multiplier = 1 for long-running tasks.
2. Missing task_acks_late: Default early acknowledgment removes messages from the broker before execution; if the worker container crashes mid-task, the message is permanently lost.

Decision scenario

Use Celery with Redis broker and task_acks_late=True for processing expensive asynchronous tasks (PDF generation, bulk emails, audio transcription) to ensure task durability and worker load balancing.

Learning outcomes

• Structure out-of-process background task execution with Celery and Redis.
• Configure prefetch multipliers and late acknowledgment for long-running tasks.
• Implement task retries with exponential backoff and jitter.

Trade-offs

Celery enables reliable out-of-process task execution, but introduces broker operational management and serialization overhead.