enhancement: implement automated instance lifecycle management and rotation

[senomorf]

Problem Statement

Oracle Cloud free tier users often reach instance limits (E2: 2/2 instances, A1: 4/4 OCPUs) and need manual intervention to free capacity for new deployments. This creates operational overhead and reduces automation effectiveness.

Proposed Solution

Implement optional automated instance lifecycle management with intelligent rotation capabilities, building upon Issue #64's limit-aware automation.

Key Features

1. Automated Instance Rotation

• Configurable via environment variable: AUTO_ROTATE_INSTANCES=true/false
• Age-based rotation: Terminate oldest instances when limits reached
• Safety checks: Verify instance health and uptime before termination
• Configurable minimum age: Prevent rotation of recently created instances

2. Instance Lifecycle Management

• Health monitoring: Check instance status and resource utilization
• Smart selection: Choose optimal instances for termination based on criteria:
  • Instance age (oldest first)
  • Resource utilization (lowest utilization first)
  • Instance type priority (configurable preferences)
• Graceful shutdown: Proper instance termination with cleanup

3. Enhanced Configuration Options

# Instance Lifecycle Management
AUTO_ROTATE_INSTANCES=true                    # Enable automatic rotation
INSTANCE_MIN_AGE_HOURS=24                     # Minimum age before rotation eligible
ROTATION_STRATEGY="oldest_first"              # "oldest_first", "least_utilized"
HEALTH_CHECK_ENABLED=true                     # Verify instance health before rotation

Implementation Approach

Phase 1: Core Lifecycle Management

• Create scripts/instance-lifecycle.sh with basic rotation logic
• Implement instance age and health checking functions
• Add safety checks and confirmation mechanisms
• Integration with existing limit detection from Issue enhancement: implement intelligent limit-aware automation to prevent futile E2 instance creation attempts #64

Phase 2: Smart Selection Algorithms

• Implement multiple rotation strategies (age-based, utilization-based)
• Add instance health scoring and selection optimization
• Configurable rotation policies with user preferences
• Enhanced logging and audit trail for rotation decisions

Phase 3: Advanced Features

• Backup and snapshot creation before rotation
• Instance replacement scheduling (create new before terminating old)
• Multi-region rotation support
• Integration with external monitoring systems

Benefits

• Reduced manual intervention: Automatic capacity management
• Improved automation reliability: Continuous operation without user input
• Resource optimization: Intelligent instance selection for rotation
• Operational visibility: Detailed logging and notification of lifecycle events
• Safety-first approach: Multiple verification steps before any destructive operations

Safety Considerations

• Opt-in only: Feature disabled by default, requires explicit enablement
• Comprehensive logging: All rotation decisions and actions logged
• Rollback capabilities: Ability to recover from rotation issues
• User notifications: Telegram alerts for all lifecycle management actions
• Minimum age protection: Prevent accidental rotation of recently created instances

Files to Create/Modify

• New: scripts/instance-lifecycle.sh - Core lifecycle management functionality
• Modify: scripts/launch-parallel.sh - Integration with rotation logic
• Modify: scripts/utils.sh - Additional helper functions for instance management
• Modify: scripts/notify.sh - Lifecycle management notifications

Success Metrics

• Automated capacity recovery when limits reached
• Zero manual intervention required for standard operations
• Proper instance selection based on configured strategy
• 100% safety record with no unintended terminations
• Clear operational visibility through logs and notifications

This enhancement transforms the Oracle Instance Creator from intelligent limit detection to full autonomous lifecycle management, providing a complete hands-off experience for Oracle free tier automation.

Related Issues

• Builds upon enhancement: implement intelligent limit-aware automation to prevent futile E2 instance creation attempts #64 (limit-aware automation)
• Complements existing state management from previous implementations

[claude]

Claude encountered an error —— View job

---

I'll analyze this and get back to you.

[claude]

Claude finished @senomorf's task —— View job • claude/issue-71-20250831-2035 • Create PR ➔

---

Implementing Automated Instance Lifecycle Management ✅

I've successfully implemented the automated instance lifecycle management and rotation system as requested in issue #71. This transforms the Oracle Instance Creator from intelligent limit detection to full autonomous instance management.

Task Breakdown

Phase 1: Analysis and Core Setup

• Analyze existing codebase and architecture
• Create core lifecycle management script (scripts/instance-lifecycle.sh)
• Implement instance age and health checking functions
• Add safety checks and configuration validation

Phase 2: Integration and Enhancement

• Integrate with existing limit detection logic
• Modify scripts/launch-parallel.sh for rotation integration
• Enhance scripts/utils.sh with lifecycle helper functions
• Update scripts/notify.sh for lifecycle notifications

Phase 3: Configuration and Testing

• Add comprehensive configuration options
• Implement rotation strategies (oldest_first, least_utilized)
• Add validation and testing
• Create documentation and usage examples

✨ Implementation Highlights

🔧 Core Features Delivered:

• Automated Instance Rotation: Terminates oldest instances when limits reached
• Multiple Strategies: oldest_first and least_utilized selection algorithms
• Safety-First Approach: Multiple validation steps prevent accidental terminations
• Comprehensive Configuration: Environment variables for fine-tuned control

🛡️ Safety Mechanisms:

• Minimum Age Protection: Configurable 24-hour default minimum age
• Health Verification: Checks instance state before any operations
• Dry Run Mode: Test rotation logic without making actual changes
• Explicit Opt-in: Feature disabled by default (AUTO_ROTATE_INSTANCES=false)

🔗 Seamless Integration:

• Limit-Aware: Works with existing cached limit detection from Issue enhancement: implement intelligent limit-aware automation to prevent futile E2 instance creation attempts #64
• Performance Optimized: Respects existing 93% performance improvements
• Parallel Execution: Integrates with current A1.Flex + E2.1.Micro workflow
• Telegram Notifications: Comprehensive lifecycle event alerts

📋 Configuration Options:

AUTO_ROTATE_INSTANCES="true"             # Enable automatic rotation
INSTANCE_MIN_AGE_HOURS="24"              # Minimum age before eligible  
ROTATION_STRATEGY="oldest_first"         # Selection strategy
HEALTH_CHECK_ENABLED="true"              # Verify health before rotation
DRY_RUN="false"                          # Test mode

🧪 Testing & Validation:

• Created comprehensive test suite (tests/test_instance_lifecycle.sh)
• Validates configuration, age calculations, safety checks, and error handling
• Integrated with existing validation system (scripts/validate-config.sh)
• Full syntax validation passed for all scripts

📖 Documentation:

• Updated CLAUDE.md with lifecycle management commands and configuration
• Created detailed documentation in docs/INSTANCE_LIFECYCLE_MANAGEMENT.md
• Added usage examples, troubleshooting, and best practices

🚀 Usage Examples

Basic Auto-Rotation:

export AUTO_ROTATE_INSTANCES=true
./scripts/launch-parallel.sh

Testing & Validation:

./scripts/instance-lifecycle.sh dry-run  # Test without changes
./scripts/instance-lifecycle.sh stats    # View rotation statistics
./tests/test_instance_lifecycle.sh       # Run test suite

Advanced Configuration:

export AUTO_ROTATE_INSTANCES=true
export ROTATION_STRATEGY="least_utilized"
export INSTANCE_MIN_AGE_HOURS=48
./scripts/launch-parallel.sh

This implementation provides complete hands-off Oracle Cloud free tier automation with intelligent capacity management, building perfectly upon the existing architecture while maintaining all performance optimizations and safety standards.

📤 Ready for Review

All changes have been committed and pushed to the feature branch.

--- | Branch: claude/issue-71-20250831-2035