From ad787a205a52625925ea8f4d5373dac471088cac Mon Sep 17 00:00:00 2001 From: komalmahale Date: Fri, 8 May 2026 14:27:41 +0200 Subject: [PATCH] Add S-CORE quality maintenance proposal --- S-CORE_Quality_Maintenance_Proposal.md | 268 +++++++++++++++++++++++++ 1 file changed, 268 insertions(+) create mode 100644 S-CORE_Quality_Maintenance_Proposal.md diff --git a/S-CORE_Quality_Maintenance_Proposal.md b/S-CORE_Quality_Maintenance_Proposal.md new file mode 100644 index 000000000..50e7df7e7 --- /dev/null +++ b/S-CORE_Quality_Maintenance_Proposal.md @@ -0,0 +1,268 @@ +# S-CORE Quality Maintenance Proposal + +**Author:** Mahale Komal +**Date:** May 6, 2026 +**For:** Ahmed & Jan + +--- + +## Summary + +This paper describes how S-CORE can define, track, and enforce key quality KPIs. + +Examples of these KPIs are: +- 100% line coverage +- No clang-tidy issues +- No CodeQL issues +- No compiler warnings + +The key question is not only which targets we want to achieve, but also how we want to steer and sustain them as a team. + +I suggest a **hybrid quality plan** for S-CORE. + +This means: +- Run **fast checks on every PR** so developers get quick feedback. +- Run **full quality checks at night** so we still catch deep issues. +- Show results in a **shared quality dashboard** so everyone can track quality. + +**Proposed Model:** +- **PR checks (fast):** Build, unit tests, formatting, and basic lint. +- **Nightly checks (full):** CodeQL, full clang-tidy, sanitizers, coverage, and static analysis. +- **Reporting:** Export quality numbers to a shared dashboard. + +--- + +## Problem Statement + +The team needs a clear way to manage software quality. + +Today, we can define quality targets, but that is not enough. We also need a practical way to: +- See the current status of these KPIs +- Take action when quality goes down +- Stop important regressions from being merged + +Ahmed also made an important point. Some quality rules may need to be enforced directly in CI instead of only being shown in reports. + +For this reason, this proposal should be treated as a decision paper. It should compare the available options and suggest one approach for the team to discuss. + +--- + +## Options Considered + +### Option 1: Dashboard-Based Monitoring + +In this model, the team tracks quality KPIs through a shared dashboard. The dashboard provides visibility into metrics such as coverage, static analysis findings, security findings, and compiler warnings. + +**Pros** +- Clear visibility of quality status +- Supports trend monitoring over time +- Encourages transparency and team ownership +- Useful for reporting and management review + +**Cons** +- Relies on manual team reaction +- Does not prevent low-quality code from being merged +- Loses effectiveness if the dashboard is not reviewed regularly + +### Option 2: CI Enforcement + +In this model, quality gates are enforced directly in the CI pipeline. If code does not meet the agreed thresholds, the build or merge process fails. + +**Pros** +- Enforces quality standards automatically +- Prevents regressions from being merged +- Makes expectations clear and measurable +- Reduces dependence on manual follow-up + +**Cons** +- Can slow down development if thresholds are too strict from the beginning +- May create friction if the current codebase is not ready for full enforcement +- Requires careful rollout and baseline definition + +### Option 3: Combined Approach + +In this model, dashboard reporting and CI enforcement are used together. The dashboard provides visibility and trend monitoring, while CI enforces the most critical quality gates. + +**Pros** +- Balances visibility with enforcement +- Supports both team awareness and process control +- Helps track improvement over time while preventing regressions +- More robust than using only one approach + +**Cons** +- Requires more setup effort +- Needs agreement on which KPIs are monitored only and which are enforced in CI + +--- + +## Pros + +### 1. Faster PR Feedback +- PR checks can finish in about 5 to 10 minutes. +- Developers get results quickly. +- This helps teams work faster. + +### 2. Better Quality Coverage +- Nightly CodeQL can find security problems. +- Nightly full clang-tidy can find more code warnings. +- Nightly sanitizers can find memory leaks and race conditions. +- Coverage trend can be tracked in the dashboard. + +### 3. Clear Visibility for Teams +- A shared dashboard shows quality trends. +- Everyone can see the same data. +- Teams can spot problems early. + +### 4. Proven Approach +- Many teams use this model: fast PR checks + full nightly checks. +- SPP already follows a similar dashboard-based idea. + +--- + +## Cons + +### 1. Some Issues Are Found Later +- A PR may pass fast checks but fail nightly checks. +- Some issues may be found the next day. +- We need a clear rule for how fast these failures must be fixed. + +### 2. Dashboard Setup Needs Effort +- We need to connect CI results to a dashboard solution. +- Dashboard design and setup will take time. +- We may reuse ideas from SPP. + +### 3. Merge Risk Can Increase +- If full checks are nightly, a bug can be merged before night run. +- We need a quick fix process for nightly failures. + +### 4. Team Ownership Is Needed +- We must define who handles nightly failures. +- We must define expected fix time. + +### 5. More CI Config to Maintain +- PR and nightly jobs are different. +- We need good documentation so nothing is missed. + +--- + +## Implementation Roadmap + +### Phase 1 : Start Nightly Quality Jobs +- [ ] Create a nightly workflow for CodeQL and full clang-tidy. +- [ ] Create a nightly workflow for sanitizers (TSAN, ASAN, LSAN). +- [ ] Export coverage result from nightly job. +- [ ] Define fix-time rule for nightly failures. + +### Phase 2 : Add Shared Dashboard +- [ ] Decide which numbers to show: coverage, warnings, test pass rate, defects. +- [ ] Choose the dashboard tool. Options can include Grafana, a static HTML dashboard, SonarQube Community Edition, or Power BI if a company license is available. +- [ ] Push nightly data to the selected dashboard. +- [ ] Share dashboard with S-CORE teams. + +--- + +## Questions + +1. Which checks must run on every PR and block merge? +2. Which checks can run only at night? +3. What is the maximum allowed PR CI time? +4. Is full CodeQL needed on PR, or is nightly enough? +5. Is full clang-tidy needed on PR, or nightly only? +6. Which quality numbers should we show in the dashboard? +7. Who will handle nightly failures, and how fast should they be fixed? +8. Can we create an S-CORE dashboard like SPP? + +--- + +## Dashboard Tool Options + +If Power BI is not preferred, we can use one of these options: + +### Option A: Grafana +- Open source and widely used +- Good for trend charts and team dashboards +- Needs a data source and some setup effort + +### Option B: Static HTML Dashboard +- Low-cost and simple +- CI can generate and publish a dashboard after nightly jobs +- Good for basic KPI reporting + +### Option C: SonarQube Community Edition +- Free and self-hosted +- Good for code quality and coverage reporting +- May not cover all KPI types without extra integration + +### Option D: Power BI +- Good reporting and visualization +- Desktop is free, but cloud sharing usually needs a paid license +- Useful only if company licensing is already available + +--- + +## Recommendation + +Use the hybrid model, which corresponds to **Option 3: Combined Approach**. + +Why: +- Fast PR checks keep developer feedback quick. +- Nightly full checks keep code quality high. +- Cost is lower than running everything on every PR. +- A shared dashboard gives clear quality visibility. + +This recommendation does not assume that reporting alone is sufficient. Instead, it combines: +- **Visibility** through dashboard reporting +- **Control** through CI enforcement for the most critical checks + +For example: +- CI can enforce no compiler warnings, no critical CodeQL issues, and no new clang-tidy violations. +- The dashboard can track broader trends such as overall coverage progress and quality status over time. + +This approach is practical and can be started step by step. + +--- + +**Next Steps:** + +1. Agree on the final list of KPIs. +2. Decide which KPIs must block merges in CI and which KPIs are tracked through reporting. +3. Run a short pilot: nightly CodeQL + coverage. +4. Review results with team. +5. Select the dashboard tool and start rollout. + +--- + +## One-Page Version + +### Goal +Keep S-CORE quality high, but keep PR feedback fast and CI cost under control. + +### Simple Plan +- Run fast checks on every PR (build, tests, formatting). +- Run full quality checks at night (CodeQL, full clang-tidy, sanitizers, coverage). +- Show quality trends in a shared dashboard. + +### Why This Plan +- Developers get quick PR feedback. +- Deep quality issues are still checked every day. +- CI cost is lower than running all checks on every PR. +- Quality status is visible for all teams. + +### Risks +- Some issues are found at night, not immediately on PR. +- We need clear ownership for nightly failures. +- Dashboard setup needs some initial effort. + +### What Is Already in S-CORE (Observed) +- PR build and test checks are present. +- Formatting checks are present. +- Coverage and sanitizer workflows exist but are not PR-blocking checks. + +### What Is Missing (Observed) +- No dedicated CodeQL GitHub Actions workflow. +- No nightly scheduled quality workflow. +- No KPI export/dashboard flow to a shared reporting tool. + + +### Recommendation +Start with the hybrid model now, run a short pilot, and then finalize targets with Jan based on CI time, cost, and team capacity.