Skip to content

README.md

Latest commit

 

History

History
132 lines (94 loc) · 3.58 KB

README.md

File metadata and controls

132 lines (94 loc) · 3.58 KB

FlowNode

High-throughput distributed task engine — Celery + Redis + asyncio.

License: MIT Python 3.11+ Celery Redis Docker CI

What it solves

When a single Python process is no longer enough — long-running jobs, scheduled tasks, fan-out workloads — FlowNode shows how to wire a clean Celery + Redis topology that scales horizontally without coupling business logic to the broker.

It's the reference setup I reach for whenever I need background processing behind an API: webhook fan-out, ETL batches, scheduled reports, retry-on-failure pipelines.

Architecture

flowchart LR
    Producer[API / Producer] -->|enqueue| Broker[(Redis Broker)]
    Broker --> Worker1[Worker 1]
    Broker --> Worker2[Worker 2]
    Broker --> WorkerN[Worker N ...]
    Worker1 --> Result[(Redis Result Backend)]
    Worker2 --> Result
    WorkerN --> Result
    Beat[Celery Beat] -->|schedule| Broker

    style Broker fill:#DC382D,color:#fff
    style Result fill:#DC382D,color:#fff
    style Worker1 fill:#37814A,color:#fff
    style Worker2 fill:#37814A,color:#fff
    style WorkerN fill:#37814A,color:#fff
Loading

Features

  • Multi-worker scaling: spin up N workers via docker compose --scale worker=N
  • Scheduled jobs: Celery Beat for cron-like recurring tasks
  • Retries with exponential backoff: declarative autoretry_for + retry_backoff
  • Task chains and groups: orchestrate dependent tasks
  • asyncio compatibility: async-ready task entrypoints
  • Result inspection: Redis result backend with TTL
  • Containerized: full stack with docker compose up

Quickstart

git clone https://github.com/franamaro-dev/FlowNode.git
cd FlowNode
docker compose up --build

Submit a sample task:

docker compose exec api python -c "from app.tasks import sample; sample.delay('hello')"

Inspect logs:

docker compose logs -f worker

Tech stack

Concern Tool
Task queue Celery 5
Broker / Result Redis 7
API producer FastAPI
Async asyncio
Container Docker + docker-compose
Testing pytest, celery.contrib.testing

Project structure

.
├── app/
│   ├── tasks/         # Celery task definitions
│   ├── api/           # FastAPI producer endpoints
│   ├── celery_app.py  # Celery app factory
│   └── config.py
├── docker-compose.yml # api + worker + redis + beat
├── Dockerfile
└── requirements.txt

Roadmap

  • Flower dashboard integration
  • Prometheus metrics exporter
  • Dead-letter queue handler
  • Distributed tracing with OpenTelemetry
  • Kubernetes manifests (Deployment + HPA)

License

MIT © Francisco Amaro Prieto

Built by Francisco Amaro — Backend Engineer & SOC L1 Analyst LinkedIn · Email