How to add Celery, Celerybeat and Redis to a Django website.
Install
Redis-server
sudo apt install redis-server
Python packages
In your python environment
pip install redis celery[redis] django-celery-beat
Celery
Add to settings.py
INSTALLED_APPS += [
'django_celery_beat',
]
and
CELERY_BROKER_URL = 'redis://localhost:6379'
CELERY_RESULT_BACKEND = 'redis://localhost:6379'
to /project/webbapp/__init__.py add
from .celery import app as celery_app
__all__ = ("celery_app",)
Add a celery.py to /project/webbapp/ with
import os
from celery import Celery
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "webbapp.settings")
app = Celery("webbapp")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()
To run management commands
As an example to run the wagtail management command publish_scheduled you can do the following.
Create an app task
In your app add a tasks.py and add this code
from celery import shared_task
from celery.utils.log import get_task_logger
from django.core.management import call_command
logger = get_task_logger(__name__)
@shared_task
def publish_scheduled():
call_command("publish_scheduled", )
logger.info("The publish_scheduled task just ran.")
Celery beat
Add your tasks to settings.py
from celery.schedules import crontab
CELERY_BEAT_SCHEDULE = {
'birdapp657_publish_scheduled': {
'task': 'birdapp657.tasks.publish_scheduled',
'schedule': crontab(minutes='1,31'), # Schedule the task to run twice every hour
}
}
Development
redis-server
redis-cli ping
celery -A webapp worker -l info
celery -A webapp beat -l info -S django
Production
sudo nano /etc/redis/redis.conf
change "supervised no" to "supervised systemd"
sudo systemctl enable redis-server.service
sudo systemctl daemon-reload
sudo apt install supervisor
In /etc/supervisor/conf.d add two new files
celery.conf
[program:celery]
directory=/project/webapp
command=/project/webbapp/env/bin/celery -A webapp worker --loglevel=INFO
autostart=true
autorestart=true
startsecs=10
stopwaitsecs = 600
celerybeat.conf
[program:celerybeat]
directory=/project/webapp
command=/project/webbapp/env/bin/celery -A webbapp beat -S django --loglevel=INFO
autostart=true
autorestart=true
startsecs=10
Run the two commands
sudo supervisorctl start all
sudo supervisorctl status all
References:
https://docs.celeryq.dev/en/stable/getting-started/introduction.html#installation
https://docs.celeryq.dev/en/stable/getting-started/backends-and-brokers/redis.html
https://docs.celeryq.dev/en/stable/django/first-steps-with-django.html
https://docs.celeryq.dev/en/stable/userguide/daemonizing.html
https://github.com/celery/celery/blob/master/extra/supervisord/celeryd.conf
https://github.com/celery/celery/blob/master/extra/supervisord/celerybeat.conf