Functionome Server Maintenance (Nginx + FastAPI + Elasticsearch Docker)

[tmushayahama]

Routine maintenance to apply OS updates, validate Nginx/app health, clean Docker artifacts, ensure Elasticsearch disk usage is controlled, reboot, and verify full recovery.

---

Checklist

Pre-Maintenance

• Confirm maintenance window + notify users
• Record baseline: uptime, disk, memory, service status, versions
• Backup/confirm access to Nginx + Elasticsearch configs
• Confirm Elasticsearch data persistence + snapshot/backup exists
• Confirm monitoring/alerts expectations during reboot

OS Updates

• Apply OS/security updates and cleanup obsolete packages
• Reboot

Nginx

• Validate config and reload/restart if updated
• Confirm TLS/certs not near expiry (if applicable)
• Smoke test reverse proxy to FastAPI and review errors
  @dustine32 notes

FastAPI / App

• Review recent logs for errors/warnings
• Add health endpoint, we current don't have
• Verify background jobs/workers

Docker Cleanup

• Remove stopped containers and unused images (orphan/dangling)
• Prune unused networks
• Review volumes and remove only confirmed-unused ones
• Confirm Docker disk usage improved

Elasticsearch (Docker)

• Confirm container running and cluster health OK
• Review index sizes/growth and enforce retention (delete/close old indices if needed)
• Check shard allocation and disk watermarks

Logs & Disk Hygiene

• Confirm log rotation functioning and logs are within retention/size policy
• Remove/archive oversized logs if needed

Reboot & Verification

• Reboot server
• Verify SSH access restored
• Confirm Nginx, FastAPI, Docker, Elasticsearch auto-started
• Smoke test external traffic path + Elasticsearch queries
• Review post-reboot logs for critical errors

Closeout

• Document what changed (updates applied, Docker cleanup, indices removed)
• Record before/after disk usage
• Note follow-ups (retention tuning, alerts, capacity)

---

Acceptance Criteria

• OS patched and stable
• Nginx/FastAPI/Elasticsearch healthy after reboot
• Docker cleanup completed without data loss
• Disk usage reduced or confirmed within safe limits
• Maintenance actions documented

cc @dustine32 @mugitty