Django Celery Integration — Core Concepts

Why Django needs Celery

HTTP requests have a natural time budget — typically under a few seconds. Operations like sending email, processing images, generating reports, or calling third-party APIs can easily exceed that. Running them inside a view blocks the web server thread, delays the user’s response, and risks timeouts.

Celery moves these operations to separate worker processes that run independently of the web server. The web process enqueues a message describing the task, and a Celery worker picks it up and executes it asynchronously.

Architecture overview

The Django-Celery setup has three components:

• Django (producer) — creates task messages and publishes them
• Message broker — stores task messages until workers consume them (Redis or RabbitMQ)
• Celery workers (consumers) — execute the tasks

Django and Celery share the same codebase but run as separate processes. Django serves HTTP requests; Celery workers run in the background.

Setting up the integration

The standard setup involves creating a Celery application instance alongside your Django project:

# project/celery.py
import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')

app = Celery('project')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

# project/__init__.py
from .celery import app as celery_app
__all__ = ('celery_app',)

The autodiscover_tasks() call scans all installed apps for tasks.py files. Configuration uses the CELERY namespace in Django settings:

# settings.py
CELERY_BROKER_URL = 'redis://localhost:6379/0'
CELERY_RESULT_BACKEND = 'redis://localhost:6379/1'
CELERY_TASK_SERIALIZER = 'json'
CELERY_ACCEPT_CONTENT = ['json']

Defining and calling tasks

Tasks are functions decorated with @shared_task:

# myapp/tasks.py
from celery import shared_task

@shared_task
def send_welcome_email(user_id):
    user = User.objects.get(id=user_id)
    send_email(to=user.email, template='welcome')

Call tasks from views or anywhere else:

# In a view
def register(request):
    user = create_user(request.POST)
    send_welcome_email.delay(user.id)  # Returns immediately
    return redirect('registration_complete')

.delay() is shorthand for .apply_async(). The task runs in a worker process, not in the web request. Notice we pass user.id (an integer), not the User object — task arguments must be serializable to JSON.

Tracking results

If you need to check whether a task succeeded or retrieve its return value:

result = send_welcome_email.delay(user.id)
print(result.id)        # UUID of the task
print(result.status)    # PENDING, STARTED, SUCCESS, FAILURE
print(result.get(timeout=10))  # Wait up to 10s for result

Result tracking requires a result backend (configured via CELERY_RESULT_BACKEND). For fire-and-forget tasks, disable it to reduce overhead: CELERY_TASK_IGNORE_RESULT = True.

Periodic tasks with Celery Beat

Celery Beat is a scheduler that sends tasks to the queue at defined intervals:

# settings.py
from celery.schedules import crontab

CELERY_BEAT_SCHEDULE = {
    'daily-report': {
        'task': 'reports.tasks.generate_daily_report',
        'schedule': crontab(hour=8, minute=0),
    },
    'cleanup-every-hour': {
        'task': 'maintenance.tasks.cleanup_temp_files',
        'schedule': 3600.0,  # Every 3600 seconds
    },
}

Run the beat scheduler alongside workers:

celery -A project beat --loglevel=info
celery -A project worker --loglevel=info

In production, run beat as a single instance — multiple beat processes will duplicate scheduled tasks.

Error handling and retries

Tasks fail. Networks time out, services go down, databases lock. Celery supports automatic retries:

@shared_task(bind=True, max_retries=3, default_retry_delay=60)
def call_payment_api(self, order_id):
    try:
        process_payment(order_id)
    except PaymentGatewayTimeout as exc:
        raise self.retry(exc=exc)

bind=True gives the task access to self, which provides the retry() method. Each retry re-enqueues the task with exponential backoff available through retry_backoff=True.

Common misconception

Many developers start Celery tasks inside signal handlers or model save() methods. This runs the task dispatch before the database transaction commits — the worker might try to fetch a record that doesn’t exist yet. Always use transaction.on_commit() to dispatch tasks after the transaction is confirmed.

The one thing to remember: Django-Celery integration offloads slow work from web requests to background workers — pass serializable arguments, handle failures with retries, and always dispatch tasks after transaction commit.