Skip to content

OPERATIONS.md

Latest commit

 

History

History
228 lines (185 loc) · 5.09 KB

OPERATIONS.md

File metadata and controls

228 lines (185 loc) · 5.09 KB

Reputation System Operations

Related Documentation:

Table of Contents

  1. Performance Standards
  2. Monitoring
  3. Logging
  4. Testing
  5. Deployment
  6. Maintenance

1. Performance Standards

Response Times

  • Snapshot listing: < 200ms
  • Snapshot creation: < 500ms
  • Workflow start: < 100ms
  • Workflow status: < 200ms
  • Strategy child workflow status: < 100ms
  • Reputation data: < 1s for 10k records

Long-Running Operations

  • Use Temporal workflows for async operations
  • Main workflow timeout: 30 minutes
  • Child workflow timeout: 15 minutes per strategy
  • Automatic retries for transient failures
  • Progress tracking via Temporal events
  • Strategy execution in parallel

2. Monitoring

System Metrics

  • CPU usage
  • Memory utilization
  • Disk I/O
  • Network traffic
  • Database connections
  • Queue lengths

Application Metrics

  • Request latency
  • Error rates
  • Rate limit usage
  • Resource utilization
  • Strategy execution times
  • Score calculation times

Temporal Metrics

  • Snapshot workflow execution time
  • Snapshot workflow success/failure rates
  • Strategy child workflow execution times
  • Strategy child workflow success/failure rates
  • Activity execution time
  • Activity success/failure rates
  • Workflow queue size
  • Worker task processing rate
  • Temporal service health
  • Concurrent child workflows per snapshot
  • Latest successful workflow age
  • Failed workflow retry rates
  • Strategy execution success rates
  • Workflow execution latency
  • Child workflow completion time
  • Strategy parameter validation time

Onchain Metrics

  • Gas usage
  • Transaction success rates
  • Commitment times
  • Verification times
  • Smart contract events

3. Logging

Required Logs

  • All API requests and responses
  • Authentication attempts
  • Rate limit hits
  • Error conditions
  • Performance metrics
  • Strategy execution details
  • Onchain transactions
  • Workflow execution events
  • Child workflow status changes
  • Strategy parameter validation results
  • Merkle tree generation steps
  • Commitment process steps
  • Score verification attempts

Log Format

{
  "timestamp": "2025-06-01T12:00:00Z",
  "level": "INFO",
  "request_id": "uuid",
  "endpoint": "/api/v1/snapshots",
  "method": "POST",
  "status": 201,
  "duration_ms": 150,
  "client_ip": "1.2.3.4",
  "user_id": "staff_123",
  "strategy": "token-weighted-voting",
  "workflow_id": "work_123",
  "child_workflow_id": "work_124",
  "event_type": "workflow_started",
  "event_details": {
    "snapshot_id": "snap_123",
    "strategy_count": 3,
    "parameters": {
      "start_block": 0,
      "end_block": 12345678
    }
  }
}

Log Levels

  • ERROR: System errors, failed operations
  • WARN: Potential issues, retries
  • INFO: Normal operations, state changes
  • DEBUG: Detailed execution information
  • TRACE: Verbose debugging information

4. Testing

Required Tests

  • Snapshot CRUD operations
  • Reputation calculation accuracy
  • DDI JWT validation
  • External API integration (Explorer, Proposal API, DDI, File Storage)
  • Workflow execution scenarios
  • Rate limiting and throttling
  • Error handling and recovery

Test Data

  • Mock blockchain data for consistent testing
  • Sample reputation calculations
  • Various snapshot configurations
  • Edge cases and error scenarios

Test Environments

  • Development: Local development
  • Staging: Pre-production testing
  • Production: Live environment

5. Deployment

Requirements

  • Node.js runtime
  • PostgreSQL database
  • Redis for caching
  • Ethereum node access
  • DDI platform integration
  • File storage system

Configuration

  • Environment variables
  • Database connections
  • External service endpoints
  • Rate limiting rules
  • Logging levels
  • Monitoring thresholds

Health Checks

  • API endpoint availability
  • Database connectivity
  • External service status
  • Blockchain node status
  • File storage access
  • Memory usage
  • Disk space

6. Maintenance

Regular Tasks

  • Log rotation
  • Database backups
  • Performance monitoring
  • Security updates
  • Dependency updates
  • Configuration review

Emergency Procedures

  • Service degradation
  • Data corruption
  • Security incidents
  • Blockchain issues
  • External service failures