Message Queues & Celery Task Pools

Learning outcomes

• Offload long-running computations to background Celery workers
• Manage task result backends and worker prefetches

Mental model

Message queues decouple HTTP request-response cycles from slow background tasks (e.g. video processing, email generation, PDF rendering). Web apps push task payloads into a queue broker (Redis), while worker processes execute tasks asynchronously.

Theory

Celery connects to a Broker (Redis / RabbitMQ) for task transport and a Result Backend (Redis / PostgreSQL) to record task completion states (PENDING, STARTED, SUCCESS, FAILURE).

# Celery worker configuration
from celery import Celery

celery_app = Celery(
    "tasks",
    broker="redis://localhost:6379/0",
    backend="redis://localhost:6379/1"
)

celery_app.conf.update(
    task_serializer="json",
    accept_content=["json"],
    result_serializer="json",
    worker_prefetch_multiplier=1, # Prevent long-running task starvation
    task_acks_late=True,          # Ack only after execution completes
)

@celery_app.task(name="generate_pdf_report", bind=True, max_retries=3)
def generate_pdf_report(self, user_id: str):
    try:
        # Long-running PDF generation logic
        return {"user_id": user_id, "status": "completed"}
    except Exception as exc:
        raise self.retry(exc=exc, countdown=10)

Alternatives and trade-offs

• FastAPI BackgroundTasks: Runs in-process on the Uvicorn worker process; lost if process crashes mid-task, unsuitable for heavy CPU loads.
• Celery + Redis: Out-of-process distributed scaling, task persistence, retries; requires maintaining extra worker infrastructure.

Failure modes and misconceptions

1. Default worker_prefetch_multiplier = 4 Starvation: When tasks take minutes to complete, a worker prefetching 4 long tasks starves other idle workers. Set worker_prefetch_multiplier = 1 for long-running tasks.
2. Missing task_acks_late: Default early acknowledgment removes messages from the broker before execution; if the worker container crashes mid-task, the message is permanently lost.

Decision scenario

Use Celery with Redis broker and task_acks_late=True for processing expensive asynchronous tasks (PDF generation, bulk emails, audio transcription) to ensure task durability and worker load balancing.

Learning outcomes

• Structure out-of-process background task execution with Celery and Redis.
• Configure prefetch multipliers and late acknowledgment for long-running tasks.
• Implement task retries with exponential backoff and jitter.

Trade-offs

Celery enables reliable out-of-process task execution, but introduces broker operational management and serialization overhead.

Primary sources

• FastAPI Framework Architecture & Dependency Injection Specification — Tiangolo, verified 2026-07-22