Open-source ML competition platform — two tracks, one backend

Judge untrusted ML models in sandboxed Docker pipelines, or hand every team a live cloud VM
and grade what they build on it as a black box. Real-time verdicts, anonymous leaderboards, zero downtime.

---

Architecture · MLOps Grading · Reliability · Infrastructure

---

Battle-Tested

Evalda has run two production competitions — most recently AINS 4.0, an overnight event where both tracks ran simultaneously for eight continuous hours.

Metric	DataQuest 2026	AINS 4.0
Tracks	Data science	Data science + MLOps
Duration	7 hours	8 hours (overnight)
Submissions judged	1,452	349 file submissions + 92 VM grade runs
Teams	41	34
Peak throughput	400 submissions/hour	67/hour sustained all night
Cloud VMs provisioned	—	15 across 2 AWS accounts, 0 failures, 0 orphans
Unplanned downtime	0	0

Full numbers and failure-mode breakdowns: docs/reliability.md.

---

Two Tracks

Data Science — judge the artifact

Teams submit a .zip with a Python solution, a trained model, and a requirements.txt. The system:

1. Validates the zip in a security sandbox (path traversal, zip bombs, symlinks, extension allowlist)
2. Extracts and sanitizes requirements.txt (blocks malicious pip options)
3. Installs dependencies in an isolated container with outbound-only internet
4. Runs the model in a fully sandboxed container (no network, no labels, resource-capped)
5. Scores the predictions in a trusted process outside all containers
6. Streams real-time progress to the participant via WebSocket
7. Updates the anonymous leaderboard with blind-hour support

Every container runs with all capabilities dropped, no privilege escalation, memory and CPU limits, and tmpfs with noexec. Ground-truth labels never enter any container. See docs/architecture.md.

MLOps — judge the running system

Each team gets a dedicated AWS VM, baked from a Packer + Ansible image and stamped out by Terraform, pre-loaded with a deliberately broken or naive starting point: an unserved model, a dead Airflow pipeline, a single-threaded inference script. Their job is to make it survive the grader.

The grader treats the VM as a black box — it only calls the team's endpoints and measures correctness and behavior on its own side of the wire. Grade runs are seeded and replayable for dispute resolution. VM slots are claimed atomically (no double-provisioning), balanced across multiple AWS accounts by vCPU budget, and reaped on TTL. See docs/mlops-grading.md.

Three challenges ship with the platform — serve-sla, scale-out, and toxic-pie — each a self-contained folder (AMI recipe, participant repo, API contracts, grader). Adding a challenge means adding a folder.

flowchart LR
classDef proxy fill:#2C3E50,stroke:#fff,stroke-width:2px,color:#fff;
classDef api fill:#059669,stroke:#fff,stroke-width:2px,color:#fff;
classDef worker fill:#D97706,stroke:#fff,stroke-width:2px,color:#fff;
classDef docker fill:#2496ED,stroke:#fff,stroke-width:2px,color:#fff;
classDef db fill:#3ECF8E,stroke:#fff,stroke-width:2px,color:#111;
classDef cache fill:#DC382D,stroke:#fff,stroke-width:2px,color:#fff;
classDef cloud fill:#844FBA,stroke:#fff,stroke-width:2px,color:#fff;

    N[Nginx]:::proxy --> F(FastAPI):::api
    F -->|DS queue| C{DS Worker}:::worker
    F -->|MLOps queue| M{MLOps Worker}:::worker
    C -->|4-phase pipeline| D[[Sandbox Containers]]:::docker
    M -->|Terraform + boto3| V[(Team VMs on AWS)]:::cloud
    M -->|grader sandbox| G[[Grader Container]]:::docker
    G -->|HTTP, black box| V
    D --> S[(Supabase)]:::db
    M --> S

    R[(Redis)]:::cache
    F <--> R
    C <--> R
    M <--> R

    subgraph Core Backend
        F
        C
        M
        R
    end

Loading

---

Tech Stack

Component	Technology	Role
Frontend	Next.js 16, shadcn/ui, TanStack Query	Submission UI, VM control panel, leaderboards, admin panel
Backend API	FastAPI, 4 uvicorn workers	Auth, rate limiting, intake, WebSocket streaming
Task queues	Celery (separate DS + MLOps workers)	Judging pipeline / VM provisioning + grading
Containers	Docker (socket-mounted, sibling containers)	Sandboxed execution: 4-phase DS pipeline + MLOps grader
Image builds	Packer + Ansible	Challenge AMIs, baked once per challenge
Provisioning	Terraform + boto3	Per-team VMs, per-VM state, readiness watching, TTL reaping
Database	Supabase Postgres + RLS	Profiles, teams, submissions, VM slots, challenge schedule
Auth	Supabase Auth (JWT)	Whitelist-gated registration, token verification
Storage	Supabase Storage	Submission zips (private, 50MB, zip-only)
Cache / broker	Redis 7.2	Brokers, verdict streams, rate limits, leaderboard cache
Reverse proxy	Nginx	SSL termination, request filtering

---

Documentation

Document	Description
architecture.md	System design, trust boundaries, security model, lessons learned
submission-workflow.md	The data science submission lifecycle, step by step with sources
mlops-grading.md	VM lifecycle, black-box grading, multi-account balancing
infrastructure.md	The Packer / Ansible / Terraform / boto3 toolchain
reliability.md	Production numbers from both events
challenge-serve-sla.md	Challenge design: model serving under an SLA
challenge-scale-out.md	Challenge design: replicated serving with consistent shared state
challenge-toxic-pie.md	Challenge design: repairing a broken Airflow pipeline

---

Quick Start

Prerequisites

• Docker and Docker Compose
• Node.js 18+
• A Supabase project (free tier works)
• For the MLOps track only: an AWS account, Terraform, and Packer

Backend

cd backend
cp .env.example .env
# Fill in Supabase credentials, Redis password, admin account
# (MLOPS_* variables are only needed if you run the MLOps track)

docker compose up --build

On first run the system seeds teams and the whitelist from data/*_teams.json, creates the admin account, and builds the sandbox and judge images.

Frontend

cd frontend
npm install
cp .env.local.example .env.local
npm run dev

Database

supabase db push

Migrations create the profiles/teams/whitelist tables, both tracks' submission tables, the MLOps VM and challenge-schedule tables, RLS policies, and the atomic RPCs (score updates, VM slot claims) locked to service_role.

Teams

Copy backend/data/datasc_teams.example.json (and mlops_teams.example.json) and fill in your rosters. Only whitelisted emails can register; users are auto-linked to their team on signup.

MLOps challenges (optional)

Each challenge under backend/packer/ bakes its own AMI:

cd backend
./generate-AMIs.sh   # or run packer build per challenge

Model binaries and datasets are not tracked in this repo — each challenge repo ships a train.py that regenerates them deterministically before baking.

---

Project Structure

Evalda/
├── backend/
│   ├── main.py                    # FastAPI app, lifespan, startup
│   ├── app/src/
│   │   ├── routers/               # Thin HTTP/WS endpoints (datasc_*, mlops_*)
│   │   ├── services/              # Business logic per track
│   │   ├── mlops/
│   │   │   ├── challenges/        # Per-challenge graders (contract, oracle, load)
│   │   │   └── shared/            # Sandbox launcher, Terraform/boto3 infra, worker
│   │   ├── auth/                  # JWT verification, rate limiting, WS guards
│   │   ├── models/  db/  settings/
│   │   └── ...
│   ├── sandbox/  judge/           # DS verify + runner containers
│   ├── packer/                    # Challenge AMI recipes + participant repos
│   ├── terraform/                 # Per-challenge VM provisioning
│   ├── template/                  # Participant solution template (DS track)
│   └── tests/                     # pytest suite (MLOps grader, infra, auth)
├── frontend/                      # Next.js 16 app (config-driven, per-track pages)
├── supabase/                      # Migrations, RLS, RPCs, seed schedule
└── docs/                          # Everything linked above

---

Adapting for Your Competition

The event-specific surface is deliberately small:

1. Branding and copy — one config object per track in frontend/lib/competition.ts drives the landing pages, timelines, and stats. No event names are hardcoded in components.
2. DS scoring — backend/app/src/services/scorer.py and backend/judge/runner.py.
3. Submission format — backend/sandbox/verify.py (allowlists, required files).
4. MLOps challenges — add a folder under backend/packer/ + backend/app/src/mlops/challenges/; grader difficulty is tuned entirely via MLOPS_* env vars.
5. Schedule — competition windows and the sequential challenge unlock are env vars + one seed table.

---

Acknowledgments

Evalda was built for DataQuest 2026 (DataOverflow, by the IEEE INSAT SB CS Chapter and ACM INSAT) and extended with the MLOps track for AINS 4.0.

Security audit and penetration testing by Salah Chafai, whose findings directly shaped the RLS policies, RPC permissions, and WebSocket hardening. And thanks to Ossama Ferjani for one very well-timed phone call.

---

License

MIT