|
| 1 | +# Roadmap |
| 2 | + |
| 3 | +A CLI-first platform for running large-scale scrapers on your own infrastructure. |
| 4 | + |
| 5 | +## Current Version: v0.1.0 ✅ |
| 6 | + |
| 7 | +- Apify-compatible REST API |
| 8 | +- Docker-based Actor execution |
| 9 | +- CLI for deployment (`crc push`, `crc run`, `crc logs`) |
| 10 | +- Datasets, Key-Value Stores, Request Queues |
| 11 | +- Basic web dashboard |
| 12 | + |
| 13 | +--- |
| 14 | + |
| 15 | +## v0.2.0 - CLI & Developer Experience |
| 16 | + |
| 17 | +**Priority:** Make the CLI the best way to work with Crawlee Cloud. |
| 18 | + |
| 19 | +| Feature | Description | |
| 20 | +|---------|-------------| |
| 21 | +| `crc init` | Scaffold new Actor projects from templates | |
| 22 | +| `crc dev` | Local development mode with hot reload | |
| 23 | +| `crc status` | Check run status and resource usage | |
| 24 | +| Input validation | Validate inputs before running | |
| 25 | +| Better errors | Actionable hints for common issues | |
| 26 | + |
| 27 | +## v0.3.0 - Production Scraping at Scale |
| 28 | + |
| 29 | +**Priority:** Run large scraping jobs reliably. |
| 30 | + |
| 31 | +| Feature | Description | |
| 32 | +|---------|-------------| |
| 33 | +| Cron scheduling | Schedule runs with cron expressions | |
| 34 | +| Retry policies | Automatic retries with configurable backoff | |
| 35 | +| Run timeouts | Kill stuck runs automatically | |
| 36 | +| Webhooks | HTTP callbacks on run completion | |
| 37 | +| Multi-worker | Scale horizontally for parallel execution | |
| 38 | +| Resource limits | Memory/CPU caps per run | |
| 39 | + |
| 40 | +## v0.4.0 - Reliability & Operations |
| 41 | + |
| 42 | +**Priority:** Production-grade stability. |
| 43 | + |
| 44 | +| Feature | Description | |
| 45 | +|---------|-------------| |
| 46 | +| Metrics | Prometheus endpoints | |
| 47 | +| Health checks | API and runner health monitoring | |
| 48 | +| Graceful shutdown | Complete in-flight runs before stopping | |
| 49 | +| Run history | Auto-cleanup old runs and data | |
| 50 | +| Backup & restore | Database backup utilities | |
| 51 | + |
| 52 | +## v0.5.0 - Polish |
| 53 | + |
| 54 | +| Feature | Description | |
| 55 | +|---------|-------------| |
| 56 | +| Actor versioning | Deploy and rollback specific versions | |
| 57 | +| API key scopes | Read-only vs full access keys | |
| 58 | +| Improved dashboard | Better UX for UI users | |
| 59 | +| Docs improvements | More guides and examples | |
| 60 | + |
| 61 | +--- |
| 62 | + |
| 63 | +## Non-Goals |
| 64 | + |
| 65 | +To keep focus, these are explicitly **not** on the roadmap: |
| 66 | + |
| 67 | +- ❌ Web IDE for editing Actors |
| 68 | +- ❌ Multi-tenant workspaces |
| 69 | +- ❌ Complex RBAC/permissions |
| 70 | +- ❌ Built-in proxy rotation (use your own) |
| 71 | + |
| 72 | +--- |
| 73 | + |
| 74 | +## Contributing |
| 75 | + |
| 76 | +Have ideas? [Open an issue on GitHub](https://github.com/crawlee-cloud/crawlee-cloud/issues)! |
| 77 | + |
| 78 | +The best contributions are CLI improvements, bug fixes, and documentation. |
0 commit comments