Chronos Engine: A Distributed Task Scheduler

Chronos Engine is a powerful, asynchronous, and resilient task scheduling system built with Python. It is designed to handle a large volume of scheduled tasks by leveraging a tiered bucketing strategy with PostgreSQL, a persistent data store with MongoDB, and a high-throughput message queue with Apache Kafka for scalable task execution.

This architecture is event-driven and designed for scalability and fault tolerance, making it suitable for applications that need to reliably execute tasks at specific times.

---

🏛️ Architecture Overview

The lifecycle of a scheduled task (a "campaign") follows this path:

1. API Ingestion: A new campaign is scheduled via a POST request to the FastAPI api service.
2. Data Storage & Bucketing:
   • The full campaign details are saved permanently in MongoDB.
   • A reference to the task is placed into the appropriate time-based "bucket" in PostgreSQL (yearly, monthly, weekly, daily, or hourly) based on its scheduled_for time.
3. Worker Processing: A single, consolidated workers process runs in the background, containing multiple concurrent tasks:
   • Bucket Workers: Periodically, tasks are moved from long-term buckets to shorter-term buckets as their execution time approaches (e.g., from daily_bucket to hourly_bucket).
   • Dispatcher & Publisher: When a task is due, it's moved to a kafka_outbox table and reliably published as a message to a Kafka topic. This uses the Transactional Outbox pattern for resilience.
4. Task Execution:
   • A team of scalable campaign_executor consumers listens to the Kafka topic.
   • When a message is received, the consumer fetches the full details from MongoDB and executes the task.
   • Kafka's consumer group protocol automatically load balances the tasks among all available consumer instances.

---

🚀 Getting Started

Follow these instructions to get the entire application stack running locally.

Prerequisites

• Docker
• Docker Compose

Installation & Setup

1. Clone the repository:

   git clone [https://github.com/wailbentafat/chronos-engine.git](https://github.com/wailbentafat/chronos-engine.git)
   cd chronos-engine

2. Configuration: All necessary environment variables are pre-configured in the docker-compose.yml file using a YAML anchor (&app-env) for consistency. Key variables include database connection strings and Kafka broker addresses.

3. Build and Run the Application: This single command will build your application's Docker image and start all the services (API, workers, databases, Kafka) in the background.

   docker-compose up --build -d

   On the very first run, the application will automatically:

   • Create all the necessary tables and indexes in the PostgreSQL database.
   • Create the campaign_tasks topic in Kafka with 3 partitions for scalability.

4. Check that all services are running:

   docker-compose ps

   You should see all containers (api, workers, campaign_executor, postgres, mongo, kafka, zookeeper) with a running or Up status.

---

🛠️ How to Use

Schedule a New Campaign

To schedule a new task, send a POST request to the /campaigns endpoint.

• Endpoint: POST /campaigns
• Body: A JSON object with business_id, name, and a scheduled_for timestamp in ISO 8601 format (UTC is recommended, indicated by a Z).

Example using curl:

This schedules a campaign to run in the near future.

curl -X 'POST' \
  'http://localhost:8000/campaigns' \
  -H 'Content-Type: application/json' \
  -d '{
    "business_id": "biz_123_test",
    "name": "My First Scheduled Campaign",
    "scheduled_for": "2025-06-09T14:30:00Z"
  }'