Skip to content

INTEGRATION_TESTS.md

Latest commit

 

History

History
196 lines (134 loc) · 5.18 KB

INTEGRATION_TESTS.md

File metadata and controls

196 lines (134 loc) · 5.18 KB

Integration Testing Guide

This document describes how to run integration tests against a real incident.io instance.

Overview

The project includes two types of tests:

  1. Unit tests - Fast, mocked tests that run on every commit (109 tests)
  2. Integration tests - Real API calls to incident.io that validate actual behavior

Integration tests are optional and require a real incident.io API key.

Local Integration Testing

Prerequisites

  1. Access to an incident.io workspace (test or development environment recommended)
  2. An API key from incident.io API settings
  3. An alert source ID from your incident.io dashboard

Setup

Set the required environment variables:

export INCIDENT_IO_API_KEY="your-test-api-key-here"
export INCIDENT_IO_ALERT_SOURCE_ID="your-alert-source-id-here"

Or create a .env file (already gitignored):

INCIDENT_IO_API_KEY=your-test-api-key-here
INCIDENT_IO_ALERT_SOURCE_ID=your-alert-source-id-here

Running Tests

Run only the integration tests:

npm run test:integration

Or run all tests including integration:

npm test

If INCIDENT_IO_API_KEY or INCIDENT_IO_ALERT_SOURCE_ID is not set, integration tests will be skipped automatically.

CI Integration Testing

CI Setup

  1. Go to your repository Settings → Secrets and variables → Actions
  2. Add repository secrets:
    • INCIDENT_IO_API_KEY - Your test API key
    • INCIDENT_IO_ALERT_SOURCE_ID - Your alert source ID
  3. Use a test workspace (not production)

Running in GitHub Actions

Integration tests run via a manual workflow trigger:

  1. Go to Actions → Integration Tests
  2. Click Run workflow
  3. Optionally select a test severity level
  4. Click Run workflow button

The workflow will:

  • Run API integration tests (tests/incident-api.test.ts)
  • Test the action end-to-end with various configurations
  • Test template variable expansion
  • Create real test alerts in your incident.io instance

What Gets Tested

The integration workflow tests:

  1. API Client Tests

    • Connection validation
    • Alert creation with minimal fields
    • Alert creation with all optional fields
    • Different severity levels
    • Deduplication behavior
    • Metadata handling
    • Error handling
    • Rate limiting

  2. Action end-to-end Tests

    • Minimal alert creation
    • Full-featured alert with templates and custom fields
    • Custom template variables

Test Alerts in incident.io

All test alerts are clearly marked:

  • Titles start with [E2E Test] or [Integration Test]
  • Contain metadata identifying them as test alerts
  • Use unique deduplication keys to avoid interference

You can clean up test alerts in your incident.io dashboard after running tests.

Best Practices

For Development

  • Use a test or development incident.io workspace, not production
  • Integration tests create real alerts - they're not free operations
  • Run integration tests before major releases to validate API contracts

For CI

  • Store INCIDENT_IO_API_KEY as a repository secret
  • Use a dedicated test workspace or sandbox environment
  • Integration tests should run on:
    • Manual trigger (current setup)
    • Before releases (recommended)
    • After significant API client changes

API Key Security

  • Never commit API keys to the repository
  • Use .env files locally (already in .gitignore)
  • Use GitHub secrets for CI/CD
  • Use a test workspace API key with limited permissions if possible

Troubleshooting

Integration Tests Are Skipped

If you see "Skipping integration test - INCIDENT_IO_API_KEY or INCIDENT_IO_ALERT_SOURCE_ID not set":

  1. Verify the environment variables are set:

    echo $INCIDENT_IO_API_KEY
echo $INCIDENT_IO_ALERT_SOURCE_ID

  2. Ensure they are exported if running in a shell

  3. In CI, verify both secrets exist in repository settings

API Connection Failures

If tests fail with connection errors:

  1. Verify the API key is valid
  2. Check your incident.io workspace is accessible
  3. Verify network connectivity to api.incident.io
  4. Check incident.io service status

Rate Limiting

If you hit rate limits:

  1. Wait a few minutes before re-running
  2. The client automatically retries with exponential backoff
  3. Consider reducing test frequency

Test Alerts in Production

If test alerts accidentally reach production:

  1. They're clearly marked with [E2E Test] or [Integration Test] prefixes
  2. They contain "test": true in custom fields for easy filtering
  3. Review your API key - ensure it's for a test workspace

Future Enhancements

Potential improvements for the integration testing setup:

  1. Automatic cleanup - Delete test alerts after test completion
  2. PR testing - Optionally run integration tests on PRs
  3. Scheduled runs - Daily integration tests to catch API changes
  4. Test repository - Separate repository testing realistic workflow scenarios
  5. Performance benchmarks - Track alert creation latency over time

Related Documentation