PythonIntermediate

celery

Distributed task queue for async and scheduled background jobs in Python.

asyncqueuetasksbackgrounddistributed

The Problem

Web requests need to return fast. Sending emails, resizing images, calling slow APIs, or running ML inference inside a request handler blocks the server and degrades UX. You need a way to hand off work to a separate process and respond immediately.

Mental Model

Celery is like a post office. Your web app drops tasks into a mailbox (broker — usually Redis or RabbitMQ). Worker processes pick them up, execute them, and optionally drop results in a result backend. The web app never waits.

60-second start

Define and run your first background task

pip install celery redis
from celery import Celery
app = Celery('tasks', broker='redis://localhost:6379/0')
@app.task
def send_email(to, subject):
    # your email logic
    print(f'Sending {subject} to {to}')
# In your web handler:
send_email.delay('user@example.com', 'Welcome!')
celery -A tasks worker --loglevel=info

Use it when

Sending emails or notifications, processing uploads, running scheduled cron-style jobs, chaining multi-step workflows, fan-out parallel processing.

Skip it when

Simple scripts that don't serve web traffic — just use asyncio or threading. If you need sub-millisecond latency, Celery's overhead is too high. For tiny projects, consider RQ (simpler) or FastAPI BackgroundTasks.

Alternatives

RQSimpler Redis-backed queue, no broker config, great for smaller projects.

DramatiqMore reliable at-least-once semantics, simpler API, growing popularity.

arqAsync-first, asyncio-native, minimal overhead. Good for FastAPI/aiohttp stacks.

HueyLightweight, single-file, Redis-backed. Easy to embed in small Django/Flask apps.

Ready to explore the full API?

See every class, function, and relationship in an interactive graph.