Local ETL for Modern Data Platform

This project provides a fully local, containerized ETL environment using modern data stack tools.
It includes:

• Airflow for orchestration
• dbt for transformations
• PostgreSQL as a warehouse simulator
• DuckDB (Python) for lightweight processing
• MinIO as an S3-compatible data lake

---

Getting Started

Start the full stack with:

docker-compose up -d

This builds the Airflow image (including Python dependencies) and launches all services.

---

Airflow

• UI available at: http://localhost:9080
• To retrieve the autogenerated login credentials:

docker exec <container-name> bash -c "cat /opt/airflow/simple_auth_manager_passwords.json.generated"

Use the displayed username/password to log in.

---

MinIO (Data Lake)

• MinIO console: http://localhost:9000
• Upload CSV files to the csv bucket.

You can use either the UI or the AWS CLI.
Configure the CLI using the credentials in docker-compose.yml:

aws configure

Upload files to the expected folder structure:

aws --endpoint-url http://localhost:9000 s3 cp <file>.csv s3://csv/leads/<YYYY>/<MM>/<DD>/<file>.csv
aws --endpoint-url http://localhost:9000 s3 cp <file>.csv s3://csv/sales/<YYYY>/<MM>/<DD>/<file>.csv

---

PostgreSQL Warehouse

• Adminer UI: http://localhost:8080
• Database credentials are defined in the postgres service inside docker-compose.yml.