|
2 | 2 |
|
3 | 3 | A CLI-first platform for running large-scale scrapers on your own infrastructure. |
4 | 4 |
|
5 | | -## Current Version: v0.4.x ✅ |
| 5 | +## Current Version: v0.6.x |
6 | 6 |
|
7 | | -- Apify-compatible REST API with input validation (Zod schemas) |
8 | | -- Docker-based Actor execution |
9 | | -- CLI for development and deployment (`crc init`, `crc dev`, `crc push`, `crc run`, `crc call`, `crc logs`, `crc status`) |
10 | | -- Datasets, Key-Value Stores, Request Queues |
11 | | -- Basic web dashboard |
12 | | -- Security config validation at startup |
13 | | -- User-scoped resources (IDOR protection) |
14 | | -- Environment variable support for Actor runs |
15 | | -- Cron scheduling, retry policies, run timeouts |
16 | | -- Webhooks with delivery tracking and retry logic |
17 | | -- Resource limits (memory caps per container) |
18 | | -- Prometheus metrics and health checks |
19 | | -- Graceful shutdown (SIGTERM/SIGINT handling) |
20 | | -- Database backup and restore utilities |
| 7 | +- Everything from v0.5.0 plus: |
| 8 | +- DigitalOcean App Platform deployment (API + Dashboard) |
| 9 | +- Single-command DO setup script (PG, Redis, App Platform, Runner Droplet) |
| 10 | +- One-click deploy templates for Railway, Render, DigitalOcean |
| 11 | +- Platform-agnostic runner auto-scaler (DO provider, extensible) |
| 12 | +- Runner heartbeat system (CPU, memory, disk metrics via Redis) |
| 13 | +- Dashboard path-routing for reverse proxy deployments |
| 14 | +- Scraper deployment tooling (deploy any scraper from a monorepo) |
21 | 15 |
|
22 | 16 | --- |
23 | 17 |
|
@@ -59,17 +53,41 @@ A CLI-first platform for running large-scale scrapers on your own infrastructure |
59 | 53 | | Graceful shutdown | ✅ Shipped — SIGTERM/SIGINT handlers in API and runner | |
60 | 54 | | Backup & restore | ✅ Shipped — backup:create, backup:restore, backup:list scripts | |
61 | 55 |
|
62 | | -## v0.5.0 - Polish & Scaling |
| 56 | +## v0.5.0 ✅ - Security & Polish |
| 57 | + |
| 58 | +| Feature | Status | |
| 59 | +|---------|--------| |
| 60 | +| Auth middleware | ✅ Shipped — JWT + API key authentication | |
| 61 | +| User-scoped resources | ✅ Shipped — IDOR protection | |
| 62 | +| Input validation | ✅ Shipped — Zod schemas on all endpoints | |
| 63 | +| SSRF protection | ✅ Shipped — webhook URL validation | |
| 64 | +| Security config validation | ✅ Shipped — blocks insecure defaults in production | |
| 65 | + |
| 66 | +## v0.6.0 ✅ - Cloud Deployment & Auto-Scaling |
| 67 | + |
| 68 | +| Feature | Status | |
| 69 | +|---------|--------| |
| 70 | +| DigitalOcean deployment | ✅ Shipped — App Platform (API + Dashboard) + Runner Droplet | |
| 71 | +| Setup automation | ✅ Shipped — `bash deploy/digitalocean/setup.sh` provisions everything | |
| 72 | +| One-click deploy | ✅ Shipped — Railway, Render, DigitalOcean buttons | |
| 73 | +| Runner auto-scaler | ✅ Shipped — queue-based, platform-agnostic (DO provider) | |
| 74 | +| Runner heartbeat | ✅ Shipped — CPU/memory/disk metrics via Redis | |
| 75 | +| Dashboard path routing | ✅ Shipped — works behind path-stripping reverse proxies | |
| 76 | +| SSL for managed DBs | ✅ Shipped — auto-enables for production Postgres | |
| 77 | +| Auto-migrations | ✅ Shipped — `run_command` runs migrations before API starts | |
| 78 | + |
| 79 | +## v0.7.0 - Scale & Operate |
63 | 80 |
|
64 | | -**Priority:** Improve developer experience and prepare for multi-worker deployments. |
| 81 | +**Priority:** Production operations for 100+ scrapers. |
65 | 82 |
|
66 | 83 | | Feature | Description | |
67 | 84 | |---------|-------------| |
68 | | -| Actor versioning | Complete the versioning workflow (table exists, workflow in progress) | |
| 85 | +| GHCR image registry | Push actor images to GitHub Container Registry for fast runner pull | |
| 86 | +| DO Spaces integration | Managed S3 storage instead of self-hosted MinIO | |
| 87 | +| Scheduling UI | Create and manage cron schedules from the dashboard | |
| 88 | +| Actor versioning | Complete the versioning workflow (table exists, needs UI) | |
69 | 89 | | API key scopes | Read-only vs full access keys | |
70 | | -| Improved dashboard | Better UX for UI users | |
71 | | -| Multi-worker runner scaling | Scale horizontally for parallel execution | |
72 | | -| Docs improvements | More guides and examples | |
| 90 | +| Hetzner scaler provider | Scale runners on Hetzner Cloud (cheaper than DO) | |
73 | 91 |
|
74 | 92 | --- |
75 | 93 |
|
|
0 commit comments