Roadmap

[josecelano]

This EPIC tracks the overall development roadmap for the Torrust Tracker Deployer project.

📋 Living Documentation

• 📖 Complete Roadmap: docs/roadmap.md
• ❓ Q&A Clarifications: docs/roadmap-questions.md
• 🛠️ Feature Process: docs/features/README.md

🎯 Roadmap Overview

The roadmap is organized into 12 main sections with incremental delivery:

1. Main app scaffolding - Console commands, logging, presentation layer ✅ COMPLETED
2. Hetzner provider - Additional infrastructure provider support ✅ COMPLETED
3. Application commands - Service deployment with incremental slicing ✅ COMPLETED
4. Docker image - Official containerized deployer ✅ COMPLETED
5. Console commands - Status and testing capabilities ✅ COMPLETED
6. HTTPS support - SSL/TLS for all services ✅ COMPLETED
7. Backup support - Database and configuration backups ✅ COMPLETED
8. Verbosity levels - User-facing output control with -v, -vv, -vvv flags
9. Extend deployer usability - validate and render commands for partial adoption
10. Improve usability (UX) - DNS reminders, service URLs, purge command
11. Improve AI agent experience - agentskills.io, documentation headers, dry-run mode
12. JSON output format support - Machine-readable output for automation and AI agents

🔑 Key Insights

Target Users

• Primary: Developers wanting simple deployment without infrastructure knowledge
• Secondary: System administrators comfortable with the deployer's approach

Technical Approach

• Configuration: TOML files + environment variables (aligned with Torrust ecosystem)
• Architecture: Clear DDD layering (presentation → application → domain)
• Deployment: Service-based incremental slicing (hello-world → Tracker → MySQL → Prometheus → Grafana)
• Testing: Focus on E2E tests, expanding with each service addition

Strategic Decisions

• MVP Scope: Basic deployer with Hetzner provider support
• Service Slicing: Deploy fully working stacks incrementally, not deployment stages
• State Management: JSON persistence with simple locking mechanisms
• Error Handling: Detailed messages with verbosity levels, user-friendly guidance

Development Process

• Team Size: 1 Rust developer
• Dependencies: Minimal external team dependencies (some Tracker project coordination)
• Feature Workflow: Document in docs/features/ → Create issue → Link as child of this EPIC

📊 Progress Tracking

Child issues will be created for each major feature and linked to this EPIC. Progress can be tracked through:

• Individual feature completion
• Roadmap document checkbox updates
• Integration test expansion

🔗 Related Resources

• Development Principles
• UX Research
• Contributing Guidelines

---

Note: This is a living roadmap. The linked documents will be updated as development progresses and new insights are gained. Below you have a copy of docs/roadmap.md, which is the source of truth.

Roadmap

1. Scaffolding for main app ✅ COMPLETED

Epic Issue: #2 - Scaffolding for main app

• 1.1 Setup logging - Issue #3 ✅ Completed
• 1.2 Create command torrust-tracker-deployer destroy - EPIC #8, EPIC #9, EPIC #10 ✅ Completed
• 1.3 Refactor extract shared code between testing and production for app bootstrapping ✅ Completed
• 1.4 Improve command to use better abstraction to handle presentation layer ✅ Completed - EPIC #102
• 1.5 Create command torrust-tracker-deployer create - EPIC #34 ✅ Completed
• 1.6 Create command torrust-tracker-deployer provision (UI layer) - Issue #174 ✅ Completed
• 1.7 Create command torrust-tracker-deployer configure (UI layer) - Issue #180 ✅ Completed
• 1.8 Create command torrust-tracker-deployer test (UI layer) - Issue #188 ✅ Completed

2. Add new infrastructure provider: Hetzner ✅ COMPLETED

Epic Issue: #205 - Add Hetzner Provider Support ✅ Completed

• 2.1 Add Hetzner provider support (Phase 1: Make LXD Explicit) ✅ Completed
• 2.2 Add Hetzner provider support (Phase 2: Add Hetzner) ✅ Completed

3. Continue adding more application commands ✅ COMPLETED

Epic Issue: #216 - Implement ReleaseCommand and RunCommand with vertical slices

Note: These are internal app layer commands. The approach is to slice by functional services - we fully deploy a working stack from the beginning and incrementally add new services.

• 3.1 Finish ConfigureCommand ✅ Completed - Epic #16
• 3.2 Implement ReleaseCommand and RunCommand with vertical slices - Epic #216

Strategy: Build incrementally with working deployments at each step.

• 3.2.1 Hello World slice (scaffolding) - #217 ✅ Completed
• 3.2.2 Torrust Tracker slice - #220 ✅ Completed
• 3.2.3 MySQL slice - #232 ✅ Completed
• 3.2.4 Prometheus slice - #238 ✅ Completed
• 3.2.5 Grafana slice - #246 ✅ Completed

4. Create a docker image for the deployer ✅ COMPLETED

• 4.1 Create docker image for the deployer - Issue #264 ✅ Completed

5. Add extra console app commands ✅ COMPLETED

• 5.1 torrust-tracker-deployer show - Issue #241 ✅ Completed
• 5.2 torrust-tracker-deployer test ✅ Completed
• 5.3 torrust-tracker-deployer list - Issue #260 ✅ Completed

6. Add HTTPS support ✅ COMPLETED

Research Complete: Issue #270 - Caddy evaluation successful, production deployment verified

• 6.1 Add HTTPS support with Caddy for all HTTP services - Issue #272 ✅ Completed
  • Implemented Caddy TLS termination proxy
  • Added HTTPS support for HTTP tracker
  • Added HTTPS support for tracker API
  • Added HTTPS support for Grafana

7. Add backup support ✅ COMPLETED

Epic Issue: #309 - Add backup support

• 7.1 Research database backup strategies - Issue #310 ✅ Completed
  • Investigated SQLite and MySQL backup approaches
  • Recommended maintenance-window hybrid approach (container + crontab)
  • Built and tested POC with 58 unit tests
  • Documented findings in docs/research/backup-strategies/
• 7.2 Implement backup support - Issue #315 ✅ Completed
  • Added backup container templates (Dockerfile, backup.sh) - Published to Docker Hub
  • Added backup service to Docker Compose template with profile-based enablement
  • Extended environment configuration schema with backup settings
  • Deployed backup artifacts via Ansible playbooks
  • Installed crontab for scheduled maintenance-window backups (3 AM daily)
  • Supports: MySQL dumps, SQLite file copy, config archives
  • Backup retention cleanup (configurable days, default 7)
  • Note: Volume management is out of scope - user provides a mounted location
  • Implementation Details: Phase 1-4 completed (container, service integration, crontab scheduling, documentation)

8. Add levels of verbosity

• 8.1 Add levels of verbosity as described in the UX research
  • Implement -v, -vv, -vvv flags for user-facing output
  • See docs/research/UX/ for detailed UX research

9. Extend deployer usability

Add new commands to allow users to take advantage of the deployer even if they do not want to use all functionalities. This enables partial adoption of the tool.

These commands complete a trilogy of "lightweight" entry points:

• register - For users with pre-provisioned instances
• validate - For users who only want to validate a deployment configuration
• render - For users who only want to build artifacts and handle deployment manually

This makes the deployer more versatile for different scenarios and more AI-agent friendly (dry-run commands provide feedback without side effects).

• 9.1 Implement validate command ✅ Completed in 272847e3
  • Validate deployment configuration without executing any deployment steps
  • See feature specification: docs/features/config-validation-command/
  • User documentation: docs/user-guide/commands/validate.md
• 9.2 Implement artifact generation command (✅ Completed in 37cbe240) - Issue #326
  • Command name: render - Generates deployment artifacts without provisioning infrastructure
  • Dual input modes: --env-name (from Created state environment) or --env-file (from config file)
  • Requires --instance-ip parameter for Ansible inventory generation
  • Generates all 8 service artifacts: OpenTofu, Ansible, Docker Compose, Tracker, Prometheus, Grafana, Caddy, Backup
  • Output to user-specified directory via --output-dir parameter (prevents conflicts with provision artifacts)
  • No remote operations - purely local artifact generation
  • Use cases: Preview before provisioning, manual deployment workflows, configuration inspection
  • User documentation: docs/user-guide/commands/render.md
  • Manual testing guide: docs/e2e-testing/manual/render-verification.md
  • All templates always rendered (no conditional logic)
  • Specification: docs/issues/326-implement-artifact-generation-command.md

10. Improve usability (UX)

Minor changes to improve the output of some commands and overall user experience.

• 10.1 Add DNS setup reminder in provision command output ✅ Completed
  • Display reminder when any service has a domain configured
  • Issue: #332
  • Implemented in PR #333
• 10.2 Improve run command output with service URLs ✅ Completed
  • Show service URLs immediately after services start
  • Include hint about show command for full details
  • Issue: #334
  • Implemented in PR #337
• 10.3 Add DNS resolution check to test command - Issue #336 ✅ Completed
  • Verify configured domains resolve to the expected instance IP
  • Advisory warning only (doesn't fail tests) - DNS is decoupled from service tests
  • See spec: docs/issues/336-add-dns-resolution-check-to-test-command.md
• 10.4 Add purge command to remove local environment data - Issue #322 ✅ Completed
  • Removes data/{env}/ and build/{env}/ for destroyed environments
  • Allows reusing environment names after destruction
  • Users don't need to know internal storage details
  • Added confirmation prompt with --force flag
  • Added comprehensive user documentation

11. Improve AI agent experience

Add features and documentation that make the use of AI agents to operate the deployer easier, more efficient, more reliable, and less prone to hallucinations.

Context: We assume users will increasingly interact with the deployer indirectly via AI agents (GitHub Copilot, Cursor, etc.) rather than running commands directly. This section ensures AI agents have the best possible experience when working with the deployer.

• 11.1 Consider using agentskills.io for AI agent capabilities
  • Agent Skills is an open format for extending AI agent capabilities with specialized knowledge and workflows
  • Developed by Anthropic, adopted by Claude Code, OpenAI Codex, Amp, and others
  • Provides progressive disclosure: metadata at startup, instructions on activation, resources on demand
  • Skills can bundle scripts, templates, and reference materials
  • See issue: #274
  • See spec: docs/issues/274-consider-using-agentskills-io.md
• 11.2 Add AI-discoverable documentation headers to template files ✅ Completed
  • Templates generate production config files (docker-compose, tracker.toml, Caddyfile, etc.)
  • Documentation is moving from templates to Rust wrapper types (published on docs.rs)
  • Problem: AI agents in production only see rendered output, not the source repo
  • Solution: Add standardized header to templates with links to repo, wrapper path, and docs.rs
  • Enables AI agents to find documentation even when working with deployed configs
  • See draft: docs/issues/drafts/add-ai-discoverable-documentation-headers-to-templates.md
• 11.3 Provide configuration examples and questionnaire for AI agent guidance ✅ Completed
  • Problem: AI agents struggle with the many valid configuration combinations
  • Questionnaire template: structured decision tree to gather all required user information
  • Example dataset: real-world scenarios mapping requirements to validated configs
  • Covers: provider selection, database type, tracker protocols, HTTPS, monitoring, etc.
  • Benefits: few-shot learning for agents, reduced hallucination, training/RAG dataset
  • See draft: docs/issues/drafts/provide-config-examples-and-questionnaire-for-ai-agents.md
• 11.4 Add dry-run mode for all commands
  • Allow AI agents (and users) to preview what will happen before executing operations
  • Particularly valuable for destructive commands (destroy, stop)
  • Flag: --dry-run shows planned actions without executing
  • Reduces risk when AI agents operate autonomously

12. Add JSON output format support

Add machine-readable JSON output format (--json flag) for selected commands to improve automation and AI agent integration. Initial phase focuses on commands where structured output provides the most value.

Context: JSON output enables programmatic parsing, making it easier for scripts and AI agents to extract specific information (like IP addresses, service URLs, environment names) without parsing human-readable text.

Phase 1 - High-Value Commands:

• 12.1 Add JSON output to create command

  • Rationale: Contains info about where to find more detailed information (paths, configuration references)
  • Structured output helps automation track environment artifacts

• 12.2 Add JSON output to provision command

  • Rationale: Contains the provisioned instance IP address - critical for automation workflows
  • Easier to parse and extract IP than regex matching console output

• 12.3 Add JSON output to show command

  • Rationale: Contains the instance IP and comprehensive environment state
  • Structured format makes it simple to query specific fields programmatically

• 12.4 Add JSON output to run command

  • Rationale: Contains the list of enabled services and their URLs
  • Allows automation to verify which services are running and how to access them

• 12.5 Add JSON output to list command

  • Rationale: Shows full environment names without truncation, enabling unambiguous identification
  • Table format truncates long names - JSON provides complete information

Future Enhancement: JSON output can be extended to all commands (configure, release, test, destroy, validate, render, purge) based on user demand and use cases.

---

Deferred Features

Features considered valuable but out of scope for v1. We want to release the first version and wait for user acceptance before investing more time. These can be revisited based on user feedback.

Feature	Rationale	Notes
MCP (Model Context Protocol) server	Native AI integration without shell commands	Would let AI agents call deployer as MCP tools directly
Structured error format for AI agents	Already improving errors in section 10	Could formalize with error codes, fix suggestions in JSON