How to Run Background Jobs in Python Without Redis

1. Why Slow Work Hurts a Web Page

A visitor signs up on your site. Your code saves the account, sends a welcome email and builds a PDF receipt. The email server takes four seconds to answer. The visitor stares at a spinning page for four seconds, and some of them leave.

Slow work inside a web request causes three problems. The visitor waits. One of your few web processes is busy and cannot serve anyone else. If the email server fails, the whole signup fails with it.

The fix is to answer the visitor first and do the slow work afterwards, in a separate program. That program is a background worker. This guide explains how workers operate, compares the libraries and builds a working queue that needs no Redis server.

2. How a Background Worker Works

A background job system has three parts.

• Your web app writes a short note that describes a job, such as "send the welcome email to user 42."
• A queue stores the notes in order until something handles them.
• A worker is a second program that runs all the time. It takes one note from the queue, does the job and takes the next one.

The web app and the worker run the same code from the same project. They are two processes with two different start commands. The web process answers visitors. The worker process answers the queue.

Good candidates for a worker include sending email, resizing images, calling slow APIs, building reports, importing large files and running machine learning models.

3. Job Queue Libraries Compared

The queue needs a place to live. Most tutorials reach for Redis, a separate database server. Redis is fast and proven, but it is one more service to run, secure and pay for. Small apps often do not need it. Several libraries can keep the queue in a database you already have.

A queue in SQLite handles far more work than most people expect. A worker on a small server can process several jobs per second. Thousands of emails a day is a light load. Move to Redis when you need many workers on several servers, or tens of jobs per second all day.

The rest of this guide uses Huey with SQLite. Huey is small, has been maintained for over a decade and needs one file for its queue.

4. Build a Queue with Huey and SQLite

Add Huey to your requirements.txt.

huey

Create a file named tasks.py. It sets up the queue and defines one job. The queue file lives in the app's data folder, so it survives restarts and new versions.

# tasks.py
import os
import smtplib
from email.message import EmailMessage

from huey import SqliteHuey

data_dir = os.environ.get("APP_DATA_DIR", ".")
huey = SqliteHuey(filename=os.path.join(data_dir, "jobs.db"))


@huey.task(retries=3, retry_delay=60)
def send_welcome_email(address):
    msg = EmailMessage()
    msg["Subject"] = "Welcome"
    msg["From"] = os.environ["SMTP_FROM"]
    msg["To"] = address
    msg.set_content("Thanks for signing up.")
    with smtplib.SMTP_SSL(os.environ["SMTP_HOST"], 465, timeout=30) as smtp:
        smtp.login(os.environ["SMTP_USER"], os.environ["SMTP_PASSWORD"])
        smtp.send_message(msg)
    print(f"Sent the welcome email to {address}", flush=True)

The @huey.task line changes what happens when you call the function. A call no longer sends the email. It writes a note to the queue and returns at once. The options tell Huey to try a failed job three more times, one minute apart.

Call the job from your web app like any other function.

# app.py
from flask import Flask, request

from tasks import send_welcome_email

app = Flask(__name__)


@app.post("/signup")
def signup():
    address = request.form["email"]
    # ... save the account ...
    send_welcome_email(address)   # returns in about a millisecond
    return "Thanks. Check your inbox."

Now start the worker. Huey installs a program named huey_consumer.py. Give it the location of your queue object, which is the file name, a dot and the variable name.

huey_consumer.py tasks.huey --workers 2

The worker prints a line when it starts and a line for each job. Run your web app in a second terminal and sign up. The page answers at once, and the worker sends the email a moment later.

Run jobs on a timetable too

Huey can also start jobs by the clock. The worker checks the timetable once a minute.

from huey import crontab


@huey.periodic_task(crontab(minute="0", hour="7"))
def send_daily_summary():
    ...

5. Use Huey with Django

Huey ships with a Django add-on. Add it to your settings and point it at a SQLite file in the data folder.

# settings.py
import os

INSTALLED_APPS = [
    # ...
    "huey.contrib.djhuey",
]

HUEY = {
    "huey_class": "huey.SqliteHuey",
    "filename": os.path.join(os.environ.get("APP_DATA_DIR", BASE_DIR), "jobs.db"),
    "consumer": {"workers": 2},
}

Put jobs in a tasks.py file inside any of your Django apps. The add-on finds them by that name.

# shop/tasks.py
from huey.contrib.djhuey import db_task

from .models import Order


@db_task(retries=2, retry_delay=30)
def build_receipt(order_id):
    order = Order.objects.get(pk=order_id)
    # ... create the PDF and save it ...

Use db_task for jobs that touch the database. It closes the database connection after each job. Pass the order's id, not the order object. The worker then loads fresh data when the job runs.

Start the worker with a management command.

python manage.py run_huey

6. Write a Worker with No Library

A worker is only a loop. Seeing one without a library makes every library easier to understand. This script takes jobs from a table in your own SQLite database.

# worker.py
import os
import signal
import sqlite3
import time

running = True


def stop(signum, frame):
    global running
    running = False   # finish the job in hand, then leave the loop


signal.signal(signal.SIGTERM, stop)

db = sqlite3.connect(os.path.join(os.environ["APP_DATA_DIR"], "app.db"), timeout=30)
db.execute("PRAGMA journal_mode=WAL")

print("Worker started", flush=True)
while running:
    row = db.execute(
        "SELECT id, payload FROM jobs WHERE status = 'waiting' ORDER BY id LIMIT 1"
    ).fetchone()
    if row is None:
        time.sleep(2)
        continue
    job_id, payload = row
    try:
        handle(payload)   # your own function
        db.execute("UPDATE jobs SET status = 'done' WHERE id = ?", (job_id,))
    except Exception as error:
        db.execute("UPDATE jobs SET status = 'failed', error = ? WHERE id = ?", (str(error), job_id))
        print(f"Job {job_id} failed: {error}", flush=True)
    db.commit()
print("Worker stopped", flush=True)

Three details matter. The loop sleeps when the queue is empty, so it does not burn CPU. It catches errors, so one bad job cannot stop the worker. It handles the SIGTERM signal, so a restart lets the current job finish first.

7. Run a Worker on Cloudilio

On a VPS, you would write a systemd unit for the worker and keep it in step with your web app by hand. On Cloudilio, a worker is one setting.

1. Publish your app. Include tasks.py and add huey to requirements.txt.
2. Open the Tasks page. Find the Background worker card.
3. Enter the worker command. Use the same command you ran on your computer, such as huey_consumer.py tasks.huey --workers 2.
4. Press Save. The worker starts within a few seconds. The card shows Running.
5. Watch the logs. Open the Logs page. Lines from the worker start with [worker].

Cloudilio then keeps the worker in step with your website.

• Same code. When you publish a new version, the worker restarts on the new version after it goes live.
• Same secrets. The worker gets the secrets from your Secrets page. When you change one, the worker restarts with the new value.
• Same data. The worker reads and writes the same /app/data folder as your website. A queue file saved there is shared by both.
• Its own memory. The worker has its own memory limit, equal to your website's. A heavy job cannot take memory from your pages.
• Automatic restarts. If the worker crashes, Cloudilio starts it again. The card warns you when it keeps stopping.
• No public address. Nothing on the internet can reach the worker. It only takes work from your queue.

Each app can have one worker. To do more work at once, raise the number of threads in the worker's own options, such as --workers 4 for Huey.

Use Celery or RQ with Redis from another provider

Cloudilio does not include a Redis server today. Celery and RQ still work with a Redis database from another provider. Save its address as a secret named REDIS_URL, read it in your code and set the worker command.

celery -A tasks worker --loglevel=INFO --concurrency=2

8. Write Jobs That Survive Restarts

Workers restart. A new version goes live, a secret changes or a job uses too much memory. These habits keep your jobs correct through every restart.

• Make jobs safe to run twice. A job can be stopped after it did its work and before it was marked as done. Before sending an email, check a "sent" flag. Set the flag in the same step that sends.
• Pass ids, not objects. A job should load the newest data when it runs. Data copied into the queue an hour ago can be stale.
• Keep jobs short. When Cloudilio stops a worker, it asks first and waits ten seconds before forcing it. Celery uses that time to finish the job in hand. Huey stops at once, and the job it was running does not finish. Split a job that takes ten minutes into many small jobs.
• Record finished work. Mark each record as done in your own database. After a restart, a short query finds the records a stopped job missed, and you can queue them again.
• Set retries with a delay. Email servers and APIs fail for short periods. Three retries a minute apart handle most of those failures.
• Give every outside call a timeout. A call with no timeout can hang forever and block the worker.
• Turn on WAL mode in SQLite. Your website and your worker both use the database. WAL mode lets one write while the other reads. Our SQLite guide covers the settings.
• Print what happens. A line for each finished or failed job turns your logs into a history you can search.

9. Questions People Ask

Can I run Python background tasks without Redis?

Yes. Huey can keep its queue in a SQLite file, and Django Q2 can keep its queue in your Django database. Both need only a worker process next to your web app.

Should I use a background worker or a scheduled task?

Use a worker when your website creates the work, such as an email after each signup. Use a scheduled task when the clock creates the work, such as a report every morning. Our scheduling guide covers the second case.

Can I start a thread in my web app instead?

You can, but the work is lost when the web process restarts, and web servers such as Gunicorn restart their processes often. A queue on disk keeps every job until a worker finishes it.

Is SQLite fast enough for a job queue?

For most small and medium apps, yes. A single worker can handle several jobs per second. Move to Redis when you need many workers on several servers.

How do I see what my worker is doing?

Read its output. On Cloudilio, the worker's lines appear on the Logs page with [worker] in front. Print a line when each job starts, finishes or fails.