When a single Python process is no longer enough — long-running jobs, scheduled tasks, fan-out workloads — FlowNode shows how to wire a clean Celery + Redis topology that scales horizontally without coupling business logic to the broker.
It's the reference setup I reach for whenever I need background processing behind an API: webhook fan-out, ETL batches, scheduled reports, retry-on-failure pipelines.
flowchart LR
Producer[API / Producer] -->|enqueue| Broker[(Redis Broker)]
Broker --> Worker1[Worker 1]
Broker --> Worker2[Worker 2]
Broker --> WorkerN[Worker N ...]
Worker1 --> Result[(Redis Result Backend)]
Worker2 --> Result
WorkerN --> Result
Beat[Celery Beat] -->|schedule| Broker
style Broker fill:#DC382D,color:#fff
style Result fill:#DC382D,color:#fff
style Worker1 fill:#37814A,color:#fff
style Worker2 fill:#37814A,color:#fff
style WorkerN fill:#37814A,color:#fff
- Multi-worker scaling: spin up N workers via
docker compose --scale worker=N - Scheduled jobs: Celery Beat for cron-like recurring tasks
- Retries with exponential backoff: declarative
autoretry_for+retry_backoff - Task chains and groups: orchestrate dependent tasks
- asyncio compatibility: async-ready task entrypoints
- Result inspection: Redis result backend with TTL
- Containerized: full stack with
docker compose up
git clone https://github.com/franamaro-dev/FlowNode.git
cd FlowNode
docker compose up --buildSubmit a sample task:
docker compose exec api python -c "from app.tasks import sample; sample.delay('hello')"Inspect logs:
docker compose logs -f worker| Concern | Tool |
|---|---|
| Task queue | Celery 5 |
| Broker / Result | Redis 7 |
| API producer | FastAPI |
| Async | asyncio |
| Container | Docker + docker-compose |
| Testing | pytest, celery.contrib.testing |
.
├── app/
│ ├── tasks/ # Celery task definitions
│ ├── api/ # FastAPI producer endpoints
│ ├── celery_app.py # Celery app factory
│ └── config.py
├── docker-compose.yml # api + worker + redis + beat
├── Dockerfile
└── requirements.txt
- Flower dashboard integration
- Prometheus metrics exporter
- Dead-letter queue handler
- Distributed tracing with OpenTelemetry
- Kubernetes manifests (Deployment + HPA)
MIT © Francisco Amaro Prieto
Built by Francisco Amaro — Backend Engineer & SOC L1 Analyst LinkedIn · Email