Add --report CLI option#300
Conversation
mohit-sheth
force-pushed
the
summary-analysis
branch
2 times, most recently
from
739f441 to
81775fd
Compare
|
/hold |
|
I like the stand alone option to run against But I do not think adding this as an integrated part of the orion run is necessary, we already have all that data there and we can arrange the output to include it. Currently when passing If we work on improving this, the integrated report is not necessary. I opened #307 this PR that already improves on the output of the table, we can use it to improve the summary output also. |
|
@chentex Thanks for reviewing this #307 improves things by splitting a very wide table into groups of 5 metrics. That’s definitely better 👍🏾 than one huge table. But with configs like node-density-cni (which has 25+ metrics...and we want to add more. @jtaleric has configs with 50+ metrics iirc) you still end up with 5 or more tables, each repeating the same time, uuid, and version columns, plus the inline ····· markers. To figure out what actually regressed, you still have to scan through all those tables looking for dotted lines and percentage changes. This is useful for perfscale engineers but it'll be time consuming for external team members The I think #307 and this PR complement each other nicely and can land independently As the number of metrics keeps growing, having both views available (plus --viz #290 that got merged recently should make the output a lot easier to read and act on |
mohit-sheth
force-pushed
the
summary-analysis
branch
from
81775fd to
6ddbf78
Compare
|
I like the overall idea, but I don't think that the implementation should be this way. The existing collapse already condenses the output data. For And for |
|
Hi @mohit-sheth I moved everything to a new PR #308 if you see the output there, you can see that it already provides a summary of what has failed making it easier to assess what was wrong with it. |
| data = {} | ||
| for path in file_paths: | ||
| if not os.path.exists(path): | ||
| print(f"Warning: file not found: {path}", file=sys.stderr) |
There was a problem hiding this comment.
Let's please continue using logger as other files do in place of print.
mohit-sheth
force-pushed
the
summary-analysis
branch
3 times, most recently
from
99e1489 to
1f62f63
Compare
|
/lgtm |
|
/unhold |
… module Signed-off-by: Mohit Sheth <msheth@redhat.com>
mohit-sheth
force-pushed
the
summary-analysis
branch
from
1f62f63 to
e160078
Compare
|
/lgtm |
afcollins
left a comment
afcollins
left a comment
There was a problem hiding this comment.
Looks good to me.
Unit tests would be good to add with new code. No failures in my tests suite.
$ orion --report output_payload-node-density-cni.json
╔════════════════════════════════════════════════════╗
║ Orion Regression Report ║
╚════════════════════════════════════════════════════╝
─── payload-node-density-cni ─── REGRESSION DETECTED
Regression(s) found :
--------------------------------------------------
Test: payload-node-density-cni:
Changepoint at: 4.22.0-0.nightly-2026-03-11-034211
Previous version: 4.22.0-0.nightly-2026-03-10-194001
Build: https://...ci-main-aws-4.22-nightly-x86-payload-control-plane-6nodes/2031577334372372480
Affected Metrics
+---------------------+---------+---------------------+-------------------+
| Metric | Value | Percentage change | Labels |
+=====================+=========+=====================+===================+
| podReadyLatency_P99 | 14500 | 358.82% | [Jira: PerfScale] |
+---------------------+---------+---------------------+-------------------+
Other PRs in payload (40):
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: afcollins The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
5 participants
Type of change
Description
--report file1.json,file2.json): reads pre-existing orion JSON output files and generates a consolidated regression report without running any analysis--report as a flag): appends a summary report after normal orion analysisorion/report.pymodule with functions to parse orion JSON output, extract changepoints with regressed metrics, and render a formatted report with per-workload pass/regression/skip status, metric tables, and PR relevance sortingProw PR: Add deferred Orion summary report step for consolidated regression reporting openshift/release#75575