How to add Celery, Celerybeat and Redis to a Django website.
Install
Redis-server
sudo apt install redis-server
Python packages
In your python environment
pip install redis celery[redis] django-celery-beat
Celery
Add to settings.py
INSTALLED_APPS += [
'django_celery_beat',
]
and
CELERY_BROKER_URL = 'redis://localhost:6379'
CELERY_RESULT_BACKEND = 'redis://localhost:6379'
Add a celery.py to /project/webbapp/ with
import os
from celery import Celery
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "webbapp.settings")
app = Celery("webbapp")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()
to /project/webbapp/__init__.py add
from .celery import app as celery_app
__all__ = ("celery_app",)
To run management commands
As an example to run the wagtail management command publish_scheduled you can do the following.
Create an app task
In your app add a tasks.py and add this code
from celery import shared_task
from celery.utils.log import get_task_logger
from django.core.management import call_command
logger = get_task_logger(__name__)
@shared_task
def publish_scheduled():
call_command("publish_scheduled", )
logger.info("The publish_scheduled task just ran.")
Celery beat
Add your tasks to settings.py
from celery.schedules import crontab
CELERY_BEAT_SCHEDULE = {
'birdapp657_publish_scheduled': {
'task': 'birdapp657.tasks.publish_scheduled',
'schedule': crontab(minute='1,31'), # Schedule the task to run twice every hour
}
}
Development
redis-server
redis-cli ping
celery -A webapp worker -l info
celery -A webapp beat -l info -S django
Production
Add a /etc/systemd/system/celery.service
[Unit]
Description=Celery Worker Service
After=network.target redis.target
[Service]
Type=forking
User=webuser
Group=webuser
WorkingDirectory=/project/webbapp
ExecStart=/project/webbapp/venv/bin/celery -A myapp.celery_app worker --loglevel=INFO
ExecStop=/project/webbapp/venv/bin/celery worker stopwait
ExecReload=/project/webbapp/venv/bin/celery worker restart
Restart=always
[Install]
WantedBy=multi-user.target
/etc/systemd/system/celerybeat.service
[Unit]
Description=Celery Beat Service
After=network.target redis.target
[Service]
Type=simple
User=webuser
Group=webuser
WorkingDirectory=/project/webbapp
ExecStart=/project/webbapp/venv/bin/celery -A \
myapp.celery_app beat \
--loglevel=INFO
Restart=always
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl start celery.service celerybeat.service
sudo systemctl enable celery.service celerybeat.service
sudo systemctl status celery.service
sudo systemctl status celerybeat.service
References:
https://docs.celeryq.dev/en/stable/getting-started/introduction.html#installation
https://docs.celeryq.dev/en/stable/getting-started/backends-and-brokers/redis.html
https://docs.celeryq.dev/en/stable/django/first-steps-with-django.html
https://docs.celeryq.dev/en/stable/userguide/daemonizing.html
https://docs.celeryq.dev/en/stable/userguide/daemonizing.html#usage-systemd