feat(connectivity_check): add automated monitoring with direct Datadog integration

[sai-praveen-os]

JIRA: PLATEXP-11106

Add Direct Datadog Monitoring to connectivity_check Module

Problem

The connectivity_check module currently supports on-demand connectivity testing via script invocation. We need automated scheduled monitoring with near real-time metrics for proactive alerting. CloudWatch-based approaches have 15-20 minute delays, which is too slow for critical services.

Solution

Extend the module with optional monitoring capabilities using direct Datadog integration via @racker/janus-core. This provides metrics in seconds rather than minutes, enabling fast alerting on connectivity failures.

Changes

lambda/index.ts:

• Added direct Datadog integration using @racker/janus-core stats library
• Publishes metrics with connectivity. prefix
• Metrics: status (gauge), latency (timing/distribution), success.count (counter), error.count (counter)
• Comprehensive tagging: env, endpoint, host, protocol, critical
• Additional custom tags via METRIC_TAGS environment variable
• Metrics published via HTTP directly to Datadog (no CloudWatch intermediary)

lambda/package.json:

• Added @racker/janus-core dependency for Datadog integration and logging

lambda/tsconfig.json:

• TypeScript configuration for compilation to CommonJS

scripts/build-lambda.sh:

• Automated build script for creating lambda.zip
• Handles npm authentication via GitHub PAT
• Compiles TypeScript to JavaScript
• Packages dependencies

.github/workflows/build-lambda.yml:

• GitHub Actions workflow to build lambda.zip on commits
• Authenticates to GitHub Packages for private @racker/* dependencies
• Commits built lambda.zip to repository for Terraform consumption

main.tf:

• Uses pre-built lambda.zip (no build during Terraform apply)
• Configurable timeout (default: 60s for multiple endpoint checks)
• OpenTelemetry layer for observability
• Environment variables: DATADOG_API_KEY, ENVIRONMENT, METRIC_TAGS

variables.tf:

• Added monitoring configuration: enable_monitoring, monitoring_schedule, monitoring_targets
• Added Datadog configuration: datadog_api_key, environment, metric_tags
• Configurable Lambda timeout and memory

monitoring.tf:

• EventBridge rule for scheduled Lambda invocation (default: 1 minute)
• Lambda permissions for EventBridge invocation
• No CloudWatch alarms (alerting handled in Datadog)

Usage Examples

Basic Monitoring Setup

module "connectivity_monitor" {
  source = "git@github.com:RSS-Engineering/terraform//modules/connectivity_check?ref=<commit>"
  
  function_name      = "connectivity-check-primary-dev"
  subnet_ids         = ["subnet-xxx"]
  security_group_ids = ["sg-xxx"]
  
  enable_monitoring   = true
  monitoring_schedule = "rate(1 minute)"
  
  monitoring_targets = [
    {
      host     = "identity.api.rackspacecloud.com"
      port     = 443
      protocol = "https"
      path     = "/v2.0"
      critical = true
    },
    {
      host     = "api.example.com"
      port     = 443
      protocol = "https"
      critical = false
    }
  ]
  
  datadog_api_key = var.datadog_api_key
  environment     = "dev"
  metric_tags     = "service:janus,team:platform"
}

Monitoring with Custom Timeout

module "connectivity_monitor" {
  source = "git@github.com:RSS-Engineering/terraform//modules/connectivity_check?ref=<commit>"
  
  # ... other config ...
  
  timeout = 120  # For checking many endpoints
}

Metrics Published to Datadog

All metrics use the connectivity. prefix:

Metric	Type	Description	Tags
connectivity.endpoint.status	gauge	1 = up, 0 = down	env, endpoint, host, protocol, critical, [custom tags]
connectivity.endpoint.latency	timing (distribution)	Response time in milliseconds (creates .avg, .min, .max, .95percentile, etc.)	env, endpoint, host, protocol, critical, [custom tags]
connectivity.endpoint.success.count	counter	Successful checks	env, endpoint, host, protocol, critical, [custom tags]
connectivity.endpoint.error.count	counter	Failed checks	env, endpoint, host, protocol, critical, [custom tags]

Note: The latency metric uses stats.timing() which creates a distribution metric with automatic aggregations (.avg, .min, .max, .median, .95percentile, etc.). Custom tags from metric_tags variable are appended to all metrics.

Key Features

• Near real-time metrics: Published to Datadog in seconds (vs 15-20 minutes with CloudWatch)
• Direct integration: No CloudWatch intermediary, reducing complexity and cost
• Flexible scheduling: Configurable check frequency (default: 1 minute)
• Comprehensive tagging: Easy filtering and aggregation in Datadog
• Custom metric tags: Add service, team, or other tags via metric_tags variable
• Critical flag: Enables different alert severities in Datadog monitors
• Pre-built Lambda: GitHub Actions builds lambda.zip, avoiding npm auth issues during Terraform apply
• Backward compatible: All monitoring features are optional

Implementation Notes

Lambda Module System: The Lambda uses CommonJS (require()) instead of ES modules due to @racker/janus-core@12.4.0 being CommonJS-only with no ES module support. ES modules were attempted but failed at runtime with directory import errors. This is an internal implementation detail that doesn't affect module consumers.

Latency Metrics: The connectivity.endpoint.latency metric uses stats.timing() which creates a Datadog distribution metric. This automatically generates aggregated metrics like .avg, .min, .max, .median, .95percentile, etc. Use connectivity.endpoint.latency.avg in Datadog queries.

Build Process

1. Developer commits changes to lambda/ directory
2. GitHub Actions workflow triggers on push
3. Workflow authenticates to GitHub Packages using JANUS_GITHUB_PAT
4. TypeScript compiled to JavaScript
5. Dependencies installed and packaged
6. lambda.zip committed back to repository
7. Terraform uses pre-built lambda.zip (no build during apply)

Testing

Tested in janus-infrastructure dev environment:

• Lambda executes successfully every minute
• Metrics published to Datadog within seconds
• All endpoints checked correctly (TCP and HTTPS)
• Tags applied correctly for filtering
• Dashboard created successfully

Breaking Changes

None - all monitoring features are opt-in via enable_monitoring flag.