Skip to content

feat(ci): add progress output and job summary to e2e action#207

Open
hanthor wants to merge 1 commit into
mainfrom
fix/e2e-output-improvements
Open

feat(ci): add progress output and job summary to e2e action#207
hanthor wants to merge 1 commit into
mainfrom
fix/e2e-output-improvements

Conversation

@hanthor
Copy link
Copy Markdown
Member

@hanthor hanthor commented Jun 2, 2026

Summary

Two improvements to make e2e failures easier to diagnose without downloading artifacts:

1. Live progress output in CI log

Adds --format progress to the behave command. Previously all formatters wrote to files, leaving the CI log nearly empty during the test run. Now the log shows live dot-notation as tests execute:

SSSSS.SSSS.F.SSS...

(S = skipped, . = passed, F = failed)

2. GitHub Actions job summary

Adds a "Summarise results" step (if: always()) that runs on the GitHub Actions runner after tests complete. It:

  • Prints a compact failure digest to the CI log with the exact scenario, step name, and error:
================================================================
E2E FAILED: 51 passed / 2 failed / 29 skipped (total 82)

Failed scenarios:
  ❌  [GNOME app launch smoke tests] Files file manager launches successfully
       step : the Files file manager launches successfully
       error: ASSERT FAILED: Visible Files window not found. Visible frames: []
  ❌  [System health smoke checks] No failed systemd units at boot
       step : No failed systemd units at boot
       error: ASSERT FAILED: Failed systemd units detected: ['fwupd-refresh.service ...']
================================================================
  • Writes a GitHub Actions job summary (markdown table) visible directly in the Actions UI:
✅ Passed ❌ Failed ⏭️ Skipped Total
51 2 29 82

With a failures table showing Feature / Scenario / Failed Step / Error, and an SLA violations table.

No artifact download needed to see what broke.

Verified against real artifacts from run 26737615278.

- Add '--format progress' to the behave command so the CI log shows
  live pass/fail/skip dots as tests run (previously silent while all
  formatters wrote to files only)
- Add 'Summarise results' step (if: always()) that:
  - Prints a compact failure digest to the CI log with the exact
    scenario, step name, and error message for each failure
  - Writes a GitHub Actions job summary table (markdown) with
    pass/fail/skip counts, a failures table, and SLA violation table
  so failures are visible directly in the Actions UI without downloading
  any artifacts

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant