Celery Python Library Cheatsheet

Celery is a powerful distributed task queue system for Python that enables you to distribute the execution of tasks across multiple workers. It is widely used for handling asynchronous and distributed processing in applications. Whether you are a beginner or an experienced developer, having a handy cheatsheet can save you time and help you navigate Celery’s features efficiently. In this cheatsheet, we’ll cover essential concepts, commands, and configurations to make your Celery experience smoother.

Installation:

pip install celery

Basic Concepts:

1. Task:

A task in Celery is a unit of work that can be executed asynchronously. Tasks are defined using the @task decorator.

from celery import Celery

app = Celery('myapp')

@app.task
def add(x, y):
    return x + y

2. Worker:

A Celery worker is a separate process that executes tasks. Start a worker with:

celery -A your_project_name worker --loglevel=info

3. Task Result:

Celery supports task results, allowing you to get the result of a completed task. This is enabled by configuring a result backend (such as Redis or a database).

app = Celery('myapp', backend='rpc://', broker='pyamqp://guest@localhost//')

result = add.delay(4, 4)
print(result.get())

4. Configuration:

Celery can be configured using a configuration module or directly in code.

# Configuration in code
app.conf.update(
    result_backend='rpc://',
    broker_url='pyamqp://guest@localhost//',
)

# Configuration in a separate module
# celeryconfig.py
result_backend = 'rpc://'
broker_url = 'pyamqp://guest@localhost//'

Command-Line Interface:

1. Starting a Worker:

celery -A your_project_name worker --loglevel=info

2. Running a Task:

celery -A your_project_name call your_task_name

3. Inspecting Workers:

celery -A your_project_name inspect active

4. Monitoring:

Celery Flower is a real-time web-based monitoring tool.

pip install flower
flower -A your_project_name --port=5555

Advanced Features:

1. Periodic Tasks (Celery Beat):

Celery Beat is a scheduler that enables periodic tasks. Start it with:

celery -A your_project_name beat --loglevel=info

2. Chord:

Chord is a powerful feature for grouping tasks and executing a callback when all tasks are complete.

from celery import group, chord

# Define tasks
@task
def add(x, y):
    return x + y

@task
def multiply(x, y):
    return x * y

# Create a chord
header = [add.s(4, 4), multiply.s(8, 8)]
result = chord(header)(sum.s())

# result.get() will return the sum of all tasks

3. Retrying Tasks:

Celery supports automatic retrying of failed tasks.

@app.task(bind=True, max_retries=3)
def my_task(self, *args, **kwargs):
    try:
        # Task logic here
    except Exception as exc:
        # Retry the task
        raise self.retry(exc=exc)

This cheatsheet covers some fundamental aspects of Celery, helping you get started with task execution, worker management, and more advanced features. Remember to explore the official Celery documentation for more in-depth information and examples. With Celery, you can efficiently handle background tasks, distribute workloads, and build scalable and responsive applications.

FAQ