Production Deployment Implementation

[FL-AntoineDurand]

Production Deployment Implementation

Overview

Implement production deployment for Holistix Forge on Ubuntu VPS. This issue tracks the complete production setup based on our Production Deployment Guide.

Status: 📋 Ready to start after Issue #30
Priority: High
Related Documentation: PRODUCTION_DEPLOYMENT.md

Context

Our development environment is designed with production parity in mind, achieving 85% reusability of existing scripts and 90% architecture reuse. The main work involves:

• ❌ Removing dev container layer (direct install on Ubuntu VPS)
• ⚠️ Implementing new DNS architecture (after Issue feat(infra): simplify DNS infrastructure by replacing PowerDNS with CoreDNS file plugin #30)
• ✏️ Replacing mkcert with Let's Encrypt SSL
• ➕ Adding systemd service management
• ➕ Implementing security hardening
• ➕ Setting up operational monitoring and backups

Dependencies

• Issue feat(infra): simplify DNS infrastructure by replacing PowerDNS with CoreDNS file plugin #30 - DNS architecture simplification (CoreDNS file plugin - BLOCKING)

Implementation Tasks

Phase 1: Core Infrastructure

• Provision Ubuntu 24.04 VPS (4 vCPU, 8GB RAM, 100GB SSD minimum)
• Configure SSH hardening and firewall (ufw)
• Configure DNS at domain registrar (NS records)
• Install core dependencies (Node.js, PostgreSQL, Nginx, Docker)
• Setup Let's Encrypt SSL with wildcard certificate

Phase 2: Script Adaptation

⚠️ BLOCKED: Waiting for Issue #30 DNS architecture decision

• Wait for Issue feat(infra): simplify DNS infrastructure by replacing PowerDNS with CoreDNS file plugin #30 completion
• Create scripts/production/setup-production.sh
• Create production DNS setup script (based on feat(infra): simplify DNS infrastructure by replacing PowerDNS with CoreDNS file plugin #30 outcome)
• Create systemd service files for all services
• Adapt create-env.sh for production (Let's Encrypt paths, strong passwords)
• Create scripts/production/deploy.sh for deployment automation
• Create backup and restore scripts
• Document production procedures

Phase 3: Deployment & Testing

• Build application artifacts (npx nx run-many --target=build --all --configuration=production)
• Run production setup script
• Create production environment
• Deploy artifacts to VPS
• Start and verify all services
• Test full stack functionality (DNS, HTTPS, API, frontend, gateways, containers)
• Perform security audit

Phase 4: Operations Setup

• Configure Grafana alerts (pool exhaustion, disk usage, memory, SSL expiry, API errors)
• Setup external uptime monitoring
• Test and verify backup/restore procedures
• Create operational runbooks
• Perform load testing
• Document disaster recovery plan
• Setup CI/CD integration (optional)

---

Success Criteria

• Production deployment works end-to-end
• All services auto-start on boot
• Monitoring and alerts configured
• Backups tested and working
• Security audit passed
• Load testing completed
• Operational documentation complete

VPS Requirements

• Ubuntu 24.04 LTS
• 4 vCPU, 8GB RAM, 100GB SSD (minimum)
• Static public IP
• Owned domain name with DNS registrar access
• DNS provider API access for Let's Encrypt

Security Checklist

• Firewall configured (ufw)
• SSH hardened (no root, key-only)
• Strong database passwords (32+ characters)
• Limited database user privileges
• SSL/TLS enforced (Let's Encrypt)
• Rate limiting configured (Nginx)
• Security headers added (X-Frame-Options, CSP, etc.)
• Automated security updates enabled

Monitoring Checklist

• System metrics (CPU, RAM, disk, network)
• Application metrics (API requests, response times)
• Database metrics (connections, queries)
• Gateway pool status
• Container health
• Log aggregation (Loki)
• Distributed tracing (Tempo)
• External uptime monitoring

Backup Strategy

What to backup: PostgreSQL databases, organization data (org-data/), Nginx configs, environment files, SSL certificates

Retention: Daily (7 days), Weekly (4 weeks), Monthly (12 months)

References

• Production Deployment Guide
• Local Development Guide
• DNS Complete Guide
• Gateway Architecture
• System Architecture
• GitHub Issue #30 - DNS Simplification (BLOCKING)

Notes

• 85% of development scripts are reusable with minor modifications
• 90% of architecture transfers directly to production
• Main differences are simplifications (no dev container, simpler DNS after feat(infra): simplify DNS infrastructure by replacing PowerDNS with CoreDNS file plugin #30)
• Production will be simpler than dev after DNS changes
• Development environment designed with production parity from the start

---

Labels: enhancement, production, deployment, infrastructure
Milestone: Production Ready
Assignees: Core team
Blocked by: #30