Skip to content

Add validation assertions for GitHub Actions job telemetry data#3176

Draft
Claude wants to merge 4 commits intomainfrom
claude/better-asserts-for-github-actions
Draft

Add validation assertions for GitHub Actions job telemetry data#3176
Claude wants to merge 4 commits intomainfrom
claude/better-asserts-for-github-actions

Conversation

@Claude
Copy link
Contributor

@Claude Claude AI commented Mar 15, 2026

Enhanced the job-exporter-global test to validate the structure and completeness of exported OpenTelemetry signals beyond mere endpoint verification.

Changes

  • HTTP server enhancement: Modified Node.js test server to capture and persist request bodies to signal-specific files (/tmp/traces.txt, /tmp/logs.txt, /tmp/metrics.txt)

  • Trace validation (JSON protocol only):

    • Verifies at least one span with github.actions.type=job attribute
    • Verifies at least one span with github.actions.type=step attribute
    • Ensures exactly one root span (no parent) per trace
  • Log validation (JSON protocol only):

    • Confirms logs contain spanId and traceId for trace correlation
  • Metrics validation (JSON protocol only):

    • Validates presence of job-level and step-level metrics

All validations run conditionally (protocol == 'http/json' only) with graceful handling when signals are not exported. Each validation outputs diagnostic counts before asserting requirements.

- run: |
    cat /tmp/traces.txt | jq -s '.' > /tmp/traces.json
    job_span_count=$(cat /tmp/traces.json | jq '[.[] | .resourceSpans[]?.scopeSpans[]?.spans[]? | select(.attributes[]? | select(.key == "github.actions.type" and .value.stringValue == "job"))] | length')
    [ "$job_span_count" -ge 1 ] || { echo "Expected at least 1 job span"; exit 1; }
  if: ${{ matrix.exporter_traces == 'otlp' && matrix.protocol == 'http/json' }}
Original prompt

This section details on the original issue you should resolve

<issue_title>Better asserts for github actions for jobs</issue_title>
<issue_description>we should assert that all are of the same trace, and all have a parent that exists except one, ...</issue_description>

Comments on the Issue (you are @claude[agent] in this section)

@plengauer In the exporter global test, write the content of every single received request to a big file per signal (one for traces, one for logs, one for metrics). Add conditional steps, that if a certain signal is exported, and not using protobuf but json, add additional checks which exit with non zero exit code if failed. These checks are signal specific. For traces, check whether there is one span that has github.actions.type set to job as traceparent, one with the attribute set to step. None should have empty parents. For logs check that there is one that has a span and trace id set. For metrics, simply check that there are job and step and action metrics. Check the job level instrumentations to learn what metrics to expect.

Co-authored-by: plengauer <100447901+plengauer@users.noreply.github.com>
@Claude Claude AI changed the title [WIP] Add better asserts for GitHub actions jobs Add validation assertions for GitHub Actions job telemetry data Mar 15, 2026
@Claude Claude AI requested a review from plengauer March 15, 2026 10:37
Copy link
Owner

@plengauer plengauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot please address comments

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Better asserts for github actions for jobs

3 participants