Skip to content

Celery

Celery runs Python tasks away from the page request. A broker such as RabbitMQ or Redis holds messages, and a Python worker does the work.

Visit Celery

Which part holds the message and which part makes the file?

Your app sends a message to a broker, the service that holds and delivers work. A Python worker reads the message and makes the file. If you want Celery to store return values and task states, configure a result backend, a separate role that can use its own storage.

For a report page, also save the report status and location in your app so the page can tell the user what actually exists.

Does a task retry automatically when the worker stops?

Celery can be configured to retry selected failures. A retry sends another message with the same task ID. By default a worker acknowledges receipt just before running the task. Late acknowledgement moves that confirmation after the task returns, but some worker-child failures still acknowledge by default. Check your broker and worker-loss settings before expecting redelivery.

Use a stable report ID to check for an existing file before producing another. A retry policy handles selected errors. It cannot establish whether an interrupted attempt already saved the file.

What do you pay for and keep if you change brokers?

Celery is open-source BSD software. The services that run the worker, broker and optional result store can cost money. Keep the Python task code, task names, queue routing, retry and acknowledgement settings, and the app’s own result records.

When changing brokers, check which messages still need work and plan how to stop sending to the old queue. Verify the replacement’s delivery and time limits before assuming it behaves the same way.

How would you trace a failed attempt and a stopped worker?

Use paper cards for a report, broker and worker. Do not start a process, connect a broker or create a real report.

  1. Follow a chosen retry
    Write that a temporary read failure happens before the report is saved. Draw another message with the same task ID if your policy retries that failure.
  2. Separate a crash case
    Draw an early acknowledgement followed by a stopped worker before it saves anything. Expect no automatic rerun for that already acknowledged delivery.
  3. Inspect the user’s result
    If the report file and app status are both missing, leave the request incomplete and mark it for review. A worker receiving the card alone is no evidence of a saved file.