Skip to content

GETTING_STARTED.md

Latest commit

 

History

History
313 lines (227 loc) · 8.21 KB

GETTING_STARTED.md

File metadata and controls

313 lines (227 loc) · 8.21 KB

SpaceWatch Enhancement Complete! 🎉

Your SpaceWatch application has been successfully enhanced with AWS S3 and Azure Blob Storage style metrics and logging - focused exclusively on storage operations.

🔐 Multi-Tenant Support

IMPORTANT: SpaceWatch now supports multi-tenant usage! Each request must provide its own DigitalOcean Spaces credentials.

Required Headers for All Requests:

  • X-Spaces-Key: Your DigitalOcean Spaces access key
  • X-Spaces-Secret: Your DigitalOcean Spaces secret key

Optional Headers:

  • X-Log-Bucket: Your bucket for access logs
  • X-Metrics-Bucket: Your bucket for metrics
  • X-Log-Prefix: Prefix for access logs (default: "")
  • X-Metrics-Prefix: Prefix for metrics (default: "spacewatch-metrics/")

Authentication:

  • User endpoints (chat, tools, metrics for user buckets) - No API key required, uses Spaces credentials
  • Admin endpoints (/admin/api/*, /metrics/operations, /logs/operations, /stats) - Requires X-API-Key header

See README.md for detailed multi-tenant usage examples.

What's New

1. Real-Time Storage Operations Dashboard 📊

A new dashboard panel displays live storage metrics (auto-refreshes every 30 seconds):

  • Operations/sec - Current request rate to your storage
  • Total Operations - Count of operations in last 5 minutes
  • Error Rate - Percentage of failed operations
  • Data Transfer/sec - Bandwidth utilization
  • Latency P50/P95/P99 - Performance percentiles
  • Operation Breakdown - Detailed breakdown by operation type

Health Status Indicator:

  • 🟢 Healthy - Error rate < 5%
  • 🟡 Degraded - Error rate 5-10%
  • 🔴 Unhealthy - Error rate > 10%

2. New API Endpoints 🔌

/metrics/operations (Admin Only)

Get detailed storage operation analytics:

curl -H "X-API-Key: your_key" http://localhost:8000/metrics/operations

Returns:

  • Current 5-minute metrics
  • 1-hour and 24-hour trends
  • Per-bucket analytics
  • Operation type breakdown

/logs/operations (Admin Only)

Query storage operation logs (S3-style):

# Get all operations
curl -H "X-API-Key: your_key" http://localhost:8000/logs/operations

# Filter by operation type
curl -H "X-API-Key: your_key" http://localhost:8000/logs/operations?operation_type=LIST_OBJECTS

# Find slow operations (>1 second)
curl -H "X-API-Key: your_key" http://localhost:8000/logs/operations?min_duration_ms=1000

# Filter by bucket
curl -H "X-API-Key: your_key" http://localhost:8000/logs/operations?bucket=my-bucket

Enhanced /health

Now includes comprehensive storage metrics:

curl http://localhost:8000/health

3. Structured Logging 📝

Every storage operation is logged with detailed metrics:

Normal Operations:

2024-01-15 10:30:45 [INFO] STORAGE_OP: LIST_OBJECTS | GET /tools/list-all | status=200 | duration=45.23ms | bytes=2048 | bucket=my-bucket

Slow Operations (>5 seconds):

2024-01-15 10:35:12 [WARNING] SLOW_OPERATION: QUERY_LOGS - 5234.56ms - bucket=logs-bucket status=200

Failed Operations:

2024-01-15 10:40:30 [ERROR] STORAGE_OP_ERROR: LIST_OBJECTS | GET /tools/list-all | duration=156.78ms | error=Bucket not found

4. Request Correlation 🔗

Every API response includes monitoring headers:

X-Request-Duration-Ms: 45.23
X-Request-Id: req-1705318245123
X-Operation-Type: LIST_OBJECTS

Use these for debugging and request tracing across your systems.

How to Use

1. Configure SpaceWatch

First Time Setup:

Before starting the application, run the interactive setup wizard:

# Install dependencies
pip install -r requirements.txt

# Run interactive setup
python setup.py

The setup wizard will guide you through configuring:

  • AI Agent URL and API Key (required)
  • Application API Key for security (optional)
  • Default DigitalOcean Spaces region (optional)

Already Configured?

If you already have a .env file, you can skip the setup and go directly to starting the application.

2. Start the Application

# Start the server
uvicorn main:app --host 0.0.0.0 --port 8000 --reload

3. Access the Dashboard

Open your browser to:

http://localhost:8000

You'll see the new Storage Operations Health panel at the top of the dashboard showing real-time metrics!

4. Monitor Your Storage

The dashboard will automatically:

  • Track all storage operations
  • Calculate latency percentiles
  • Monitor error rates
  • Display operation breakdown
  • Auto-refresh every 30 seconds

5. Query Metrics via API

Use the new API endpoints to integrate with your monitoring systems:

# Get current storage metrics
curl http://localhost:8000/health | jq '.storage_metrics'

# Get detailed operation analytics
curl http://localhost:8000/metrics/operations | jq .

# View recent operation logs
curl http://localhost:8000/logs/operations?limit=10 | jq .

What Metrics Are Tracked?

Operation Types

  • LIST_BUCKETS - Bucket discovery operations
  • LIST_OBJECTS - Object listing in buckets
  • ANALYZE_STORAGE - Storage analysis (top largest, summaries)
  • QUERY_METRICS - Metrics retrieval
  • QUERY_LOGS - Log queries
  • AI_QUERY - AI assistant queries

Performance Metrics

  • Latency Percentiles (P50, P95, P99)
  • Average Latency
  • Operations per Second
  • Bytes per Second

Reliability Metrics

  • Error Rate (%)
  • Success Rate (%)
  • Failed Operation Count

Per-Bucket Analytics

  • Operation count
  • Error rate
  • Bytes transferred
  • Average latency

Comparison with AWS/Azure

Your SpaceWatch now has similar metrics to:

AWS S3 CloudWatch:

✅ Request metrics (AllRequests, GetRequests, PutRequests)
✅ Error metrics (4xxErrors, 5xxErrors)
✅ Latency metrics (FirstByteLatency, TotalRequestLatency)
✅ Data transfer metrics (BytesDownloaded, BytesUploaded)

Azure Blob Storage Analytics:

✅ Transaction metrics
✅ Success/Error rates
✅ Latency percentiles
✅ Availability indicators
✅ Egress metrics

Configuration Options

Adjust Metrics Retention

Default: 10,000 operations

# In main.py
MAX_STORAGE_METRICS_SAMPLES = 20000  # Increase to 20k

Change Slow Operation Threshold

Default: 5 seconds

# In main.py, record_storage_operation function
if duration_ms > 3000:  # Change to 3 seconds
    logger.warning(...)

Customize Metrics Window

Default: 5 minutes

# In get_storage_metrics function
recent_window = 600  # Change to 10 minutes

Documentation

Full documentation is available in these files:

  1. METRICS_GUIDE.md - Comprehensive guide to all metrics features
  2. IMPROVEMENTS_SUMMARY.md - Detailed technical summary
  3. README.md - Updated quick start guide

Example Use Cases

1. Performance Monitoring

# Check if latency is acceptable
curl http://localhost:8000/health | jq '.storage_metrics | {p50, p95, p99}'

2. Error Detection

# Monitor error rate
curl http://localhost:8000/health | jq '.storage_metrics.error_rate_percent'

3. Capacity Planning

# Track operation trends
curl http://localhost:8000/metrics/operations | jq '.trends'

4. Debugging

# Find slow operations
curl http://localhost:8000/logs/operations?min_duration_ms=1000 | jq .

No Breaking Changes ✅

All existing features continue to work:

  • ✅ AI chat assistant
  • ✅ Bucket and object management
  • ✅ Access log analysis
  • ✅ Metrics snapshots
  • ✅ Storage analytics
  • ✅ IP tracking

The new features are additive only - everything you had before still works!

Next Steps

  1. Start the application and check out the new dashboard
  2. Monitor your storage operations in real-time
  3. Integrate with external monitoring using the new API endpoints
  4. Set up alerts based on error rates or latency thresholds
  5. Analyze operation patterns to optimize your storage usage

Support

For questions or issues:

  1. Check METRICS_GUIDE.md for detailed documentation
  2. Review IMPROVEMENTS_SUMMARY.md for technical details
  3. Check the application logs for any errors

Enjoy your enhanced storage observability! 🚀

Your SpaceWatch application now provides enterprise-grade storage monitoring similar to AWS S3 CloudWatch and Azure Blob Storage Analytics!