feat: E2E coverage instrumentation with CI pipeline for all dynamic plugins

[gustavolira]

Summary

Adds Istanbul-based E2E coverage collection for dynamic plugins with a CI pipeline that automatically builds instrumented OCI images when source.json changes — and skips builds when the image already exists.

Scope: frontend plugins only. Coverage collection uses window.__coverage__ which is only available for browser-executed code. Backend plugin coverage would require a different collection mechanism (global.__coverage__ from the Node.js process) and is out of scope for this PR.

How it works

1. source.json changes (push to main) trigger the build-instrumented-plugins workflow
2. For each workspace with e2e-tests/, the workflow checks if an instrumented OCI image already exists for the current repo-ref
3. If not, it runs instrument-plugin.sh which: clones upstream at the exact ref, builds the plugin (backstage-cli + janus-cli export-dynamic), then applies nyc instrument on the final webpack output
4. The instrumented bundle is published as an OCI artifact to ghcr.io
5. During E2E tests with E2E_COLLECT_COVERAGE=1, the _coverageCollector auto-fixture (in e2e-test-utils PR #95) collects window.__coverage__ from the browser after each test
6. report-coverage.sh merges per-test coverage JSONs via nyc merge, generates lcov via nyc report
7. upload-coverage.sh uploads lcov to Codecov with per-workspace flags (e2e-tech-radar, e2e-topology, etc.) for dashboard filtering

Key design decisions

• Post-webpack instrumentation: babel-plugin-istanbul gets stripped by webpack's module federation during export-dynamic-plugin. Applying nyc instrument after the final webpack build preserves Istanbul __coverage__ in the browser runtime
• Caching by source.json ref: OCI images are tagged with ref-<sha-first-12>. If the ref hasn't changed, the build is skipped entirely — no CI overhead
• Cross-repo Codecov attribution: Coverage uploads use the upstream repo's SHA and slug from source.json, so coverage appears on the correct repo in Codecov
• Parallel matrix: All 18 workspaces with E2E tests build in parallel via GitHub Actions matrix strategy
• nyc CLI for merge/report: Coverage merging and lcov generation use nyc merge + nyc report instead of custom code — nyc is already a pipeline dependency and handles lcov edge cases better
• testInfo.project.outputDir for coverage path: The auto-fixture writes coverage JSONs to <outputDir>/coverage/, avoiding CWD mismatch issues between Playwright workers and the main process

Known limitations

• Overlays and patches are not applied: The instrumented build clones upstream source directly. Production builds apply overlays from workspaces/*/plugins/*/overlay/ and patches from workspaces/*/patches/*.patch. Most workspaces have no overlays/patches, so the instrumented build matches production in the majority of cases. Tracking as a follow-up.
• Single frontend plugin per workspace: The build discovers the first frontend-plugin in metadata. Workspaces with multiple frontend plugins only instrument the first one.
• Missing injection integration: This PR builds instrumented OCI artifacts and provides the collection/upload mechanism, but the logic to swap production plugins with instrumented ones during E2E execution is a follow-up in e2e-test-utils.

Files

File	Purpose
.github/workflows/build-instrumented-plugins.yaml	CI workflow: detect changed workspaces, check OCI cache, build and publish
scripts/instrument-plugin.sh	Build instrumented dynamic plugin from upstream source
scripts/report-coverage.sh	Merge per-test coverage JSONs (nyc merge), generate lcov (nyc report), upload to Codecov
scripts/upload-coverage.sh	Upload lcov to Codecov with cross-repo attribution and flags

Related PRs

• rhdh-e2e-test-utils PR #95 — _coverageCollector auto-fixture that collects window.__coverage__ per test

Usage

# Manual: instrument a specific plugin
./scripts/instrument-plugin.sh tech-radar

# Run E2E with coverage collection
E2E_COLLECT_COVERAGE=1 ./run-e2e.sh -w tech-radar

# Merge + upload (called automatically by run-e2e.sh)
./scripts/report-coverage.sh tech-radar

# Upload only (requires lcov already generated)
CODECOV_TOKEN=xxx ./scripts/upload-coverage.sh tech-radar

Test plan

• Verify workflow detects changed source.json files correctly
• Verify OCI image existence check skips unnecessary builds
• Run ./scripts/instrument-plugin.sh tech-radar locally — verify __coverage__ in output
• Verify force-rebuild workflow_dispatch input bypasses cache
• Verify Codecov flags produce per-plugin dashboard filtering

Ref: RHIDP-13411

[gustavolira]

/review
-i
--pr_reviewer.require_score_review=true
--pr_reviewer.require_can_be_split_review=true
--pr_reviewer.num_code_suggestions="2"

[subhashkhileri]

Review:

the Playwright integration can be significantly simplified by replacing custom code with nyc CLI. See inline comments.

Also: Coverage collection (E2E_COLLECT_COVERAGE=1) should only be enabled on the e2e-ocp-helm PR check job — nightly uses released OCI refs (not instrumented builds), so coverage would produce empty data.

[subhashkhileri]

Review: Instrumentation Pipeline

Following up on the Playwright integration review — this covers the instrumentation workflow and build script.

The post-webpack instrumentation approach (nyc instrument after export-dynamic-plugin) is the right technique for module federation. However, there are significant gaps between how this workflow builds plugins vs. how the existing production workflow does it.

[subhashkhileri]

Proposal: Reuse the existing build pipeline instead of a separate instrumentation workflow

The current PR introduces scripts/instrument-plugin.sh (205 lines) and .github/workflows/build-instrumented-plugins.yaml (271 lines) — a parallel pipeline that re-clones, re-builds, and re-packages plugins from scratch. This duplicates the production build pipeline and misses overlays, patches, and edge cases that the production pipeline already handles.

Proposed approach

Instead of rebuilding from source, instrument the already-published production image. Add an instrument job to publish-release-branch-workspace-plugins.yaml that runs after the export job:

  instrument:
    needs: export
    if: # only for workspaces that have e2e-tests/
    runs-on: ubuntu-latest
    strategy:
      matrix:
        workspace: # detect workspaces with e2e-tests/ (similar to current detect-workspaces job)
      fail-fast: false
    permissions:
      contents: read
      packages: write
    steps:
      - uses: actions/checkout@v6

      - name: Resolve published image ref
        id: meta
        run: |
          # Read source.json + metadata to find the frontend plugin's image name and tag
          # Use the SAME naming/tagging as production: ghcr.io/{repo}/{plugin}:{tag}
          # The production image was just pushed by the export job

      - name: Log in to GHCR
        uses: docker/login-action@v4
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Setup Node.js
        uses: actions/setup-node@v6

      - name: Instrument and publish coverage image
        run: |
          PROD_IMAGE="${{ steps.meta.outputs.image-ref }}"
          COVERAGE_IMAGE="${{ steps.meta.outputs.coverage-image-ref }}"

          # 1. Extract plugin bundle from production image
          CONTAINER=$(podman create "$PROD_IMAGE")
          mkdir -p .instrumented
          podman cp "$CONTAINER:/opt/app-root/src/" .instrumented/plugin/
          podman rm "$CONTAINER"

          # 2. Instrument JS files with Istanbul
          npx --yes nyc instrument .instrumented/plugin/ .instrumented/instrumented/ --source-map

          # 3. Build new image from instrumented files
          cat > .instrumented/Containerfile <<EOF
          FROM scratch
          COPY instrumented/ /opt/app-root/src/
          EOF
          podman build -t "$COVERAGE_IMAGE" -f .instrumented/Containerfile .instrumented/

          # 4. Push with -coverage suffix, same tag as production
          podman push "$COVERAGE_IMAGE"

What this eliminates

File	Lines	Reason
scripts/instrument-plugin.sh	205	No need to re-clone, re-build, re-export from source
.github/workflows/build-instrumented-plugins.yaml	271	No separate workflow needed

What this fixes

1. Overlays and patches are included — the production image already has them applied
2. All plugins discovered — not just the first frontend-plugin match
3. Same toolset — Podman throughout, matching the production pipeline (no ORAS)
4. Same tag format — bs_1.49.4__1.32.0 with -coverage suffix on the image name
5. Same triggers — runs whenever production images are published (push to release branches)
6. No pnpm/npm detection needed — the plugin is already built

Coverage image naming

Production: ghcr.io/{repo}/{plugin}:{tag}
Coverage: ghcr.io/{repo}/{plugin}-coverage:{tag}

Same tag, -coverage suffix on the image name. This makes it trivial to swap in the coverage image during E2E runs — just append -coverage to the image name.

Note on image structure

The podman cp + podman build FROM scratch approach needs to match whatever directory layout the rhdh-cli package command produces. The exact path inside the container (e.g., /opt/app-root/src/) needs verification against the actual production image. The Containerfile above is illustrative — the actual paths should be confirmed by inspecting a published production image with podman inspect or skopeo inspect.

An alternative simpler approach: instead of rebuilding from a Containerfile, use podman commit after modifying the extracted files in-place, or use buildah to add/replace layers. The goal is to match the production image format exactly.

[subhashkhileri]

Move coverage logic out of run-e2e.sh

The ~30 lines of coverage code added to run-e2e.sh (nyc merge, nyc report, multi-workspace warning, per-workspace upload loop) should move into a self-contained script — e.g. rename scripts/upload-coverage.sh → scripts/report-coverage.sh and have it handle the full pipeline (merge → report → upload).

run-e2e.sh then reduces to:

if [[ "${E2E_COLLECT_COVERAGE:-}" == "1" ]]; then
    "$SCRIPT_DIR/scripts/report-coverage.sh" "${E2E_WORKSPACES[@]}"
fi

Keeps run-e2e.sh focused on test orchestration, makes the coverage script independently re-runnable (useful for debugging uploads without re-running tests), and avoids splitting the logic across two files.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[psrna]

@gustavolira @kadel Are there any other areas where you don't have consensus on the approach? Has all the requested changes/feedback been addressed? What is still outstanding? Thanks!

[kadel]

@gustavolira @kadel Are there any other areas where you don't have consensus on the approach? Has all the requested changes/feedback been addressed? What is still outstanding? Thanks!

I think that we have consensus on overall approach. But the code is not ready to be merged. There are still some issues with code.

@psrna I'm happy to help with the review. But I think that someone who knows more about test setup in this repo (like @subhashkhileri or @zdrapela) should have the final "ack"

[kadel]

Wouldn't it be safer to pin to a specific version, rather than always using latest?

[gustavolira]

Pinned codecov-cli to version 9.0.4

[kadel]

This was previously raised by @subhashkhileri
COVERAGE_OUTPUT_DIR is never defined or exported in this script

[gustavolira]

@kadel this file has been removed.T he coverage merge and lcov generation is now handled by the Playwright CoverageReporter (in e2e-coverage/coverage-reporter.ts), which runs as part of the test process itself, no separate shell script needed. The COVERAGE_OUTPUT_DIR issue no longer applies.

[kadel]

If this was run with multiple workspaces, and they were previously merged into one coverage data, won't this mean the merged coverage data gets uploaded into each workspace?
Does Codecov handle this correctly? Does it ignore file paths that don't belong to the source repo? Have you tested a multi-workspace upload to verify?

[gustavolira]

This was indeed broken for multi-workspace runs.
Upload is handled separately by upload-coverage.sh, which takes a single workspace argument and uploads the lcov for that workspace only. For multi-workspace runs, the caller is responsible for invoking upload once per workspace with workspace-scoped coverage data.

[kadel]

why is failed upload non-fatal? Isn't the whole point of this workflow to submit coverage report to codecov?

[gustavolira]

You're right, now the script runs under set -euo pipefail, so any failure in codecov upload-process now terminates the script with a non-zero exit code

[subhashkhileri]

1). I think we might not need build-instrumented-plugins.yaml as a separate workflow. It triggers on source.json changes to main, but the instrumented images are actually consumed during the PR flow , after /publish and before E2E tests. The existing auto-publish-pr.yaml already has everything we need: published-exports gives us the exact image refs just pushed, and workspace metadata tells us which are frontend plugins.

What if we just add an instrument job in that existing flow?

/publish → export → instrument (new) → add-completion-comment → "/test e2e-ocp-helm"
                        │
                        └─ reads published-exports, filters for frontend plugins if required,
                           instruments and, pushes with __coverage tag

No separate triggers, no GHCR existence checks, no workspace detectio, the publish flow already handles all of that.

2). On naming, can we use tag suffix instead of image name suffix? The PR creates separate GHCR packages (plugin-coverage:tag). Since we already use composite tags like bs_1.49.4__1.5.0, appending __coverage makes the image swap a simple.

3). Nothing actually consumes these instrumented images yet. We should update rhdh-e2e-test-utils to handle the PR image swap for coverage images when E2E_COLLECT_COVERAGE=1. Might be worth getting that wired up before merging so we can validate the full pipeline end-to-end.

[gustavolira]

1). I think we might not need build-instrumented-plugins.yaml as a separate workflow. It triggers on source.json changes to main, but the instrumented images are actually consumed during the PR flow , after /publish and before E2E tests. The existing auto-publish-pr.yaml already has everything we need: published-exports gives us the exact image refs just pushed, and workspace metadata tells us which are frontend plugins.

What if we just add an instrument job in that existing flow?

/publish → export → instrument (new) → add-completion-comment → "/test e2e-ocp-helm"
                        │
                        └─ reads published-exports, filters for frontend plugins if required,
                           instruments and, pushes with __coverage tag

No separate triggers, no GHCR existence checks, no workspace detectio, the publish flow already handles all of that.

2). On naming, can we use tag suffix instead of image name suffix? The PR creates separate GHCR packages (plugin-coverage:tag). Since we already use composite tags like bs_1.49.4__1.5.0, appending __coverage makes the image swap a simple.

3). Nothing actually consumes these instrumented images yet. We should update rhdh-e2e-test-utils to handle the PR image swap for coverage images when E2E_COLLECT_COVERAGE=1. Might be worth getting that wired up before merging so we can validate the full pipeline end-to-end.

Thanks for the suggestions, @subhashkhileri .

1. Merging into auto-publish-pr.yaml: These workflows serve different purposes. build-instrumented-plugins.yaml pre-builds instrumented images from production OCI artifacts on main — it's a cache-warming step so coverage images are ready before any E2E run. auto-publish-pr.yaml builds PR-specific images from the overlay branch. Merging them would mean coverage images are only available after /publish on a PR, losing the pre-caching benefit. The whole point is that instrumentation is applied to the exact artifact shipped to users, not rebuilt from source during PR flow.
2. Tag suffix vs. image name suffix: Keeping coverage images as separate GHCR packages (plugin-name-coverage:tag) makes registry management cleaner — easier to list, delete, or set permissions independently. A tag suffix (tag__coverage) would mix them into the same package. Either approach works, but separate packages give us better isolation
3. Wiring up consumption before merging: Agreed this is needed, but it's already scoped as a follow-up — the PR description lists "Missing injection integration" as a known limitation and points to e2e-test-utils PR #95. The instrumented images need to exist before the consumer logic can be validated end-to-end. Merging the build pipeline first avoids a circular dependency.

CC: @psrna @kadel