Mental model
FastAPI BackgroundTasks runs functions in-process after sending the HTTP response, whereas dedicated queues (Celery, ARQ, Redis Queue) offload tasks to out-of-process distributed worker nodes.
Theory
Native BackgroundTasks are lightweight and ideal for fast, non-critical post-request operations (sending a confirmation email, writing an audit log). However, because BackgroundTasks run within the web server process memory, if the web server crashes or restarts, unexecuted background tasks are permanently lost. Heavy computational tasks or tasks requiring persistent retries must be offloaded to dedicated distributed task queues like Celery or ARQ backed by Redis/RabbitMQ.
Alternatives and trade-offs
- FastAPI
BackgroundTasks: Zero infrastructure overhead, fast to implement. Lost if process crashes; shares web worker CPU/RAM. - ARQ / Celery + Redis: Guaranteed persistence, retries, exponential backoff, rate limiting, and isolated multi-node worker pools.
Failure modes and misconceptions
- Long-running CPU tasks in
BackgroundTasks: Running a 5-minute video encoding job in nativeBackgroundTasksblocks the web worker process and risks HTTP server timeouts. - Task persistence assumptions: Native
BackgroundTaskshave no persistent queue or retry mechanism.
Decision scenario
Use native BackgroundTasks for fast (~10-50ms) non-critical post-response operations. Offload mission-critical background jobs requiring persistent retries or heavy CPU processing to Celery/ARQ workers backed by Redis.
Learning outcomes
- Select between native in-process
BackgroundTasksand distributed task queues. - Prevent web server memory and CPU starvation from background task offloading.
- Architect resilient task processing pipelines with persistent retry guarantees.
Trade-offs
Native BackgroundTasks require zero external broker infrastructure, but lack persistence and retry guarantees if the web process restarts.