Demystifying Celery with FastAPI

Let’s face it: as backend engineers, we all know that not every task in our API should run during a request-response cycle. Whether it’s sending an email, processing a big report, or syncing with an external API, these tasks don't need to bog down our main app. That's where task queues come in — and specifically, Celery.

In this blog, I’ll break down what Celery is, how it works, why message queues are crucial for modern apps, and how to integrate Celery with FastAPI for asynchronous background task processing.

---

What is Celery?

At its core, Celery is a framework that allows you to run tasks asynchronously in the background. It lets you offload time-consuming operations so your app can stay responsive, which is critical for providing a smooth user experience.

Why use a task queue like Celery?

Before diving into how Celery fits into FastAPI, let’s quickly explore the concept of message-queues. Imagine you’re running a restaurant — you, the waiter, don’t want to be stuck in the kitchen chopping vegetables or making soup. Instead, you take the customer’s order and pass it off to the kitchen staff, who work on it while you attend to other customers. This is essentially what a message queue does for your app: it lets tasks like data processing or API calls happen in the background, while your app continues to handle incoming requests.

When tasks are queued up, the system can process them in order without making the user wait. This means that heavy tasks don’t block the rest of the app — every task has its time.

---

Core Concepts of Celery and Task Queues

1. Producer: This is your FastAPI app that sends tasks to the queue (Redis, RabbitMQ, etc.).

2. Broker: The queue itself. Celery supports Redis and RabbitMQ as brokers. It's where tasks wait until a worker is ready to process them.

3. Worker: The backend employee — these are processes running Celery that pick up tasks from the broker and execute them.

4. Result Backend (optional): Once the worker finishes the task, it might store the result in a result backend, typically a database or Redis.

So, you’re basically telling your app, “Don’t worry about that heavy task right now. I’ll deal with it in the background while you handle the next customer (request).”

---

Setting Up Celery with FastAPI + Redis

Let’s set things up. Here’s how you can integrate Celery into your FastAPI app and get background tasks running with Redis as the broker.

1. Install the Requirements

To get started, you’ll need to install FastAPI, Celery, Redis, and a few other packages

python3 -m venv env 
source env/bin/activate ## For linux
.env\Scripts\activate ## For windows
pip install fastapi celery redis uvicorn

2. Directory Structure

We’ll use the following structure:

project/
├── app.py              # FastAPI app
├── celery_worker.py    # Celery worker
├── tasks.py            # Task definitions
└── requirements.txt

3. Create Celery Tasks

Let’s define a task in tasks.py. In this example, we’ll simulate generating a report for a user (it’ll sleep for a few seconds to mimic a time-consuming task):

from celery import Celery

celery_app = Celery(
    "worker",
    broker="redis://localhost:6379/0", # change this to your redis url
    backend="redis://localhost:6379/1" # change this to your redis url
)

@celery_app.task()
def generate_report(user_id: int):
    import time, random
    time.sleep(5)  # Simulating a long task
    return {
        "user_id": user_id,
        "status": "completed",
        "report_id": random.randint(1000, 9999)
    }

4. FastAPI Integration

Now, let’s set up FastAPI to send tasks to Celery. In app.py, you’ll create an endpoint that triggers the background task:

from fastapi import FastAPI
from tasks import generate_report

app = FastAPI()

@app.post("/start-report/{user_id}")
def start_report(user_id: int):
    task = generate_report.delay(user_id)  # Send task to Celery
    return {"task_id": task.id, "status": "processing"}

5. Running the Celery Worker

Now, let’s set up a Celery worker to start processing these tasks. Run the worker in a separate terminal:

celery -A celery_worker.celery_app worker --loglevel=info

---

Real-World Scenario: On-Demand Report Generation

Let’s put this into context. Suppose you're building a dashboard for a logistics company. Managers request detailed shipment reports for the last quarter. These reports take a minute or two to generate because they’re based on large datasets and involve multiple API calls.

Instead of holding up the frontend and keeping the user waiting, you can trigger generate_report(user_id) in the background. You immediately return a task ID to the user, and they can periodically check the status of the report generation via another endpoint:

@app.get("/task-status/{task_id}")
def task_status(task_id: str):
    task = generate_report.AsyncResult(task_id)
    if task.state == "SUCCESS":
        return {"status": "completed", "result": task.result}
    else:
        return {"status": task.state}

This allows the API to stay responsive while Celery works on the task in the background. No more waiting on a loading screen — everything happens asynchronously.

---

When Not to Use Celery

While Celery is powerful, it’s not for every project. Here are a few scenarios where you might want to skip Celery:

• Small tasks: If your tasks are super fast (e.g., logging), it’s overkill.

• Simple background tasks: FastAPI has a built-in BackgroundTasks feature for simple background operations.

• Limited infrastructure: Running Celery requires a broker (Redis, RabbitMQ) and workers, so make sure your infrastructure can handle it.

If your tasks are straightforward, Celery might add unnecessary complexity.

---

Final Thoughts

Celery is a robust tool for handling asynchronous background tasks in Python. By integrating it with FastAPI, you can keep your web server fast and responsive while offloading lengthy processes. It’s an essential tool for any backend engineer who’s serious about building scalable and maintainable systems.

And there you have it — asynchronous tasks that won’t block your app’s performance. A task queue that does the heavy lifting so you don’t have to!

If you’re ready to level up your backend architecture and implement retry logic, task prioritization, or even scheduled tasks, Celery can be your best friend. Drop your questions in the comments or reach out if you need help extending this into something more complex.

Happy coding!