Job Queue

noun

say it: “JOB KYOO” /dʒɑb kju/

In Python, a separate program, such as Celery or RQ, that stores background jobs, runs them in worker processes apart from the web server, and retries them when they fail.

What is a job queue in Python?

A job queue is a separate program for background work: it keeps each job in a store of its own (often Redis), runs it in workers apart from the web server, and tries it again when it fails. Celery and RQ are the two Python ones. Use one for work that must happen even if the server restarts; FastAPI's BackgroundTasks keeps its jobs only in memory.

from redis import Redis
from rq import Queue

queue = Queue(connection=Redis())
queue.enqueue(send_confirmation, "maya@example.com")   # a worker runs it later, even after a restart

In a sentence

  • “Confirmation emails moved to a job queue once losing one on a restart wasn't acceptable.”
  • “RQ is a small job queue that keeps its jobs in Redis.”

Level: Freelancer · More Python terms at this level

Updated

In other subjects