Add nightly quality workflow with GitHub Pages dashboard #393
+189
−0
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,189 @@ | ||
| # Proposal: Nightly Quality Dashboard for Public Repository | ||
|
|
||
|
|
||
| ## Option A: GitHub Actions + GitHub Pages | ||
|
|
||
| ### Summary | ||
|
|
||
| Run nightly CI jobs via GitHub Actions on a cron schedule. A final job collects results and generates a static HTML dashboard, deployed to GitHub Pages. | ||
|
|
||
| ### How It Works | ||
|
|
||
| 1. A nightly job triggers at midnight | ||
| 2. Parallel jobs run: CodeQL, Coverage, Clang-Tidy | ||
| 3. `deploy_dashboard` job runs after all others | ||
| 4. `generate_dashboard.py` produces a single-page HTML dashboard with job statuses | ||
| 5. Coverage HTML report is included as a sub-page | ||
| 6. Result is deployed to GitHub Pages | ||
|
Comment on lines
+14
to
+17
Contributor
There was a problem hiding this comment. include it into the existing sphinx build in //docs/sphinx |
||
|
|
||
| ### User Access | ||
|
|
||
| - Dashboard URL: `https://<org>.github.io/<repo>/` | ||
| - Coverage report: `https://<org>.github.io/<repo>/coverage/index.html` | ||
| - Enable in repo Settings → Pages → Source: "GitHub Actions" | ||
|
|
||
| ### Cost | ||
|
|
||
| - **$0** — GitHub Actions provides unlimited minutes for public repos; GitHub Pages is free for public repos (100 GB bandwidth/month, 1 GB storage) | ||
|
|
||
| ### Pros | ||
|
Comment on lines
+26
to
+29
Contributor
There was a problem hiding this comment. don´t know if this is true for organistations
Contributor
Author
There was a problem hiding this comment. You maybe mean private repos but for public repos its free |
||
|
|
||
| - Completely free, no external services needed | ||
| - Self-contained within the repository (workflow + script in-repo) | ||
| - Dashboard is publicly accessible with a stable URL | ||
| - Coverage report included alongside status | ||
| - Simple to maintain — just Python + standard GitHub Actions | ||
| - `workflow_dispatch` allows manual re-runs | ||
| - Concurrency control prevents overlapping runs | ||
|
|
||
| ### Cons | ||
|
|
||
| - **No historical trends** — only shows the latest run (GitHub Pages is overwritten each deploy) | ||
| - GitHub Pages has a soft limit of 10 deploys/hour | ||
| - If all jobs fail, the dashboard still deploys (by design), but coverage artifact may be missing | ||
| - Single-page dashboard — limited interactivity | ||
| - No notifications on failure (need to add separately) | ||
|
|
||
| ### Recommendations | ||
|
|
||
| 1. **Add failure notifications** — Add a step to post to Slack/email/GitHub Issues on failure | ||
| 2. **Historical data** — Consider committing a JSON log to a `gh-pages` branch to build trend charts | ||
| 3. **Badge** — Add a workflow status badge to `README.md`: | ||
| ```markdown | ||
|  | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Option B: GitHub Actions + GitHub Actions Workflow Badges Only (Minimal) | ||
|
|
||
| ### Summary | ||
|
|
||
| Skip the custom dashboard entirely. Rely on GitHub's built-in workflow status badges and the Actions tab for visibility. | ||
|
|
||
| ### How It Works | ||
|
|
||
| 1. Same `nightly_quality.yml` (without the `deploy_dashboard` job) | ||
| 2. Add status badges to `README.md` for each job | ||
| 3. Anyone visits the repo or Actions tab to see results | ||
|
|
||
| ### User Access | ||
|
|
||
| - Actions tab: `https://github.com/<org>/<repo>/actions/workflows/nightly_quality.yml` | ||
| - Badges visible on the repo landing page (README) | ||
|
|
||
| ### Cost | ||
|
|
||
| - **$0** | ||
|
|
||
| ### Pros | ||
|
|
||
| - Zero maintenance — no dashboard code to maintain | ||
| - GitHub provides filtering, logs, re-run buttons natively | ||
| - Historical runs are preserved indefinitely in the Actions tab | ||
| - Badges auto-update | ||
|
|
||
| ### Cons | ||
|
|
||
| - No unified "at-a-glance" dashboard page | ||
| - No coverage visualization (must download artifacts manually) | ||
| - Badges only show pass/fail for the entire workflow, not per-job granularity | ||
| - Less polished for external stakeholders | ||
|
|
||
| --- | ||
|
|
||
| ## Option C: GitHub Actions + Third-Party Dashboard (Codecov, Grafana Cloud, etc.) | ||
|
|
||
| ### Summary | ||
|
|
||
| Use GitHub Actions for CI, but publish metrics to a dedicated dashboard service. | ||
|
|
||
| ### How It Works | ||
|
|
||
| 1. Same nightly workflow runs tests + coverage | ||
| 2. Upload coverage to **Codecov** (free for public repos) | ||
| 3. Optionally push metrics to **Grafana Cloud** free tier (10k metrics, 50 GB logs) | ||
| 4. Dashboard lives on the third-party service | ||
|
|
||
| ### User Access | ||
|
|
||
| - Codecov: `https://codecov.io/gh/<org>/<repo>` | ||
| - Grafana: custom dashboard URL | ||
|
|
||
| ### Cost | ||
|
|
||
| - **$0** for Codecov (public repos) and Grafana Cloud free tier | ||
|
|
||
| ### Pros | ||
|
|
||
| - Rich historical trends, graphs, and PR annotations (Codecov) | ||
| - Professional-looking dashboards with no custom code | ||
| - Codecov integrates with PR checks (coverage diff on every PR) | ||
| - Grafana supports alerting | ||
|
|
||
| ### Cons | ||
|
|
||
| - External dependency — service outages affect visibility | ||
| - Data lives outside your control | ||
| - Grafana free tier has retention limits (14 days for logs) | ||
| - More complex setup (API keys, secrets) | ||
| - Team must adopt another tool/login | ||
|
|
||
| --- | ||
|
|
||
| ## Option D: GitHub Actions + GitHub Pages with Historical Trends | ||
|
|
||
| ### Summary | ||
|
|
||
| Enhanced version of Option A. Persist historical results in a JSON file on the `gh-pages` branch and render trend charts in the dashboard. | ||
|
|
||
| ### How It Works | ||
|
|
||
| 1. Same nightly workflow | ||
| 2. Before deploying, fetch previous `history.json` from `gh-pages` branch | ||
| 3. Append current run results, keep last N entries (e.g., 90 days) | ||
| 4. Dashboard includes a JavaScript chart (Chart.js or similar) showing pass/fail over time | ||
| 5. Deploy updated dashboard + history to GitHub Pages | ||
|
|
||
| ### User Access | ||
|
|
||
| - Same as Option A: `https://<org>.github.io/<repo>/` | ||
|
|
||
| ### Cost | ||
|
|
||
| - **$0** | ||
|
|
||
| ### Pros | ||
|
|
||
| - All benefits of Option A, plus historical trend visualization | ||
| - Still fully self-contained, no external services | ||
| - Stakeholders can see quality trajectory over time | ||
|
|
||
| ### Cons | ||
|
|
||
| - More complex to implement (JSON history management, chart rendering) | ||
| - Risk of data loss if `gh-pages` branch is force-pushed | ||
| - More JavaScript in the dashboard (maintenance burden) | ||
| - Still limited compared to dedicated tools (no PR annotations, no alerting) | ||
|
|
||
| --- | ||
|
|
||
| ## Comparison Matrix | ||
|
|
||
| | Criteria | A (Pages) | B (Badges) | C (3rd Party) | D (Pages+History) | | ||
| |-------------------------|--------------|------------|----------------|--------------------| | ||
| | Cost | Free | Free | Free | Free | | ||
| | Setup effort | Low | Minimal | Medium | Medium | | ||
| | Maintenance | Low | None | Low | Medium | | ||
| | Historical trends | No | Via Actions tab | Yes | Yes | | ||
| | Coverage visualization | Yes | No (artifact only) | Yes (Codecov) | Yes | | ||
| | External dependency | No | No | Yes | No | | ||
| | At-a-glance dashboard | Yes | No | Yes | Yes | | ||
| | PR integration | No | Badges only | Yes (Codecov) | No | | ||
| | Alerting | Manual | None | Yes (Grafana) | Manual | | ||
|
|
||
| --- | ||
|
|
||
| ## Recommendation | ||
|
|
||
| **Start with Option A** — it's virtualised, free, and low-maintenance. | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This document is only describing different ways of creating a dashboard.
For me at first a quality.md should be created which describes how do run quality tools like:
Based on this we can then derive which metrics need to be displayed on the dashboard.
For the Dashboard we have already an existing sphinx build on read the docs where we could integrate our dashboard: https://score-communication.readthedocs.io/en/latest/
For the report then maybe start with passed/failed tests, continue with coverage and then maybe static code analysis.
Since we already have a sphinx build it maybe would be beneficial if we could get the results as rst files then we could directly include them into our build
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is not the quality.md, the quality.md is done by Ambuj in different PR https://cc-github.bmwgroup.net/oss/eclipse-score-communication/pull/215
this doc is only for how we can see/manage the quality KPIs on Score