How to add Celery, Celerybeat and Redis to a Django website.

/
django, celerybeat and redis logo and a celery

Install

Redis-server

sudo apt install redis-server

Python packages

In your python environment

pip install redis celery[redis] django-celery-beat

Celery

Add to settings.py

INSTALLED_APPS += [
    'django_celery_beat',
]

and

CELERY_BROKER_URL = 'redis://localhost:6379'
CELERY_RESULT_BACKEND = 'redis://localhost:6379'

to /project/webbapp/__init__.py add

from .celery import app as celery_app

__all__ = ("celery_app",)

Add a celery.py to /project/webbapp/ with

import os

from celery import Celery

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "webbapp.settings")

app = Celery("webbapp")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()

To run management commands

As an example to run the wagtail management command publish_scheduled you can do the following.

Create an app task

In your app add a tasks.py and add this code

from celery import shared_task
from celery.utils.log import get_task_logger
from django.core.management import call_command

logger = get_task_logger(__name__)

@shared_task
def publish_scheduled():
    call_command("publish_scheduled", )
    logger.info("The publish_scheduled task just ran.")

Celery beat

Add your tasks to settings.py

from celery.schedules import crontab

CELERY_BEAT_SCHEDULE = {
    'birdapp657_publish_scheduled': {
        'task': 'birdapp657.tasks.publish_scheduled',
        'schedule': crontab(minutes='1,31'),  # Schedule the task to run twice every hour
    }
}

Development

redis-server
redis-cli ping
celery -A webapp worker -l info
celery -A webapp beat -l info -S django

Production

sudo nano /etc/redis/redis.conf

change "supervised no" to "supervised systemd"

sudo systemctl enable redis-server.service
sudo systemctl daemon-reload
sudo apt install supervisor

In /etc/supervisor/conf.d add two new files

celery.conf

[program:celery]
directory=/project/webapp
command=/project/webbapp/env/bin/celery -A webapp worker --loglevel=INFO
autostart=true
autorestart=true
startsecs=10
stopwaitsecs = 600

celerybeat.conf

[program:celerybeat]
directory=/project/webapp
command=/project/webbapp/env/bin/celery -A webbapp beat -S django --loglevel=INFO
autostart=true
autorestart=true
startsecs=10

Run the two commands

sudo supervisorctl start all
sudo supervisorctl status all