This application is intended to be used with cx1e2e: https://github.com/cxpsemea/cx1e2e
cx1e2e-tracker has two main use-cases:
- providing a historical context for successful test durations
- allowing comparison of two specific test reports
It does so by consuming the JSON-format reports generated by cx1e2e.
The following parameters are relevant for historical mode:
cx1e2e-tracker.exe:
-avg-pct uint
Output a warning when test duration exceeds historical average by this percentage (default 15)
-history string
Cx1e2e-tracker history file generated by this program (default "cx1e2e-history.json")
-max-pct uint
Output a warning when test duration exceeds historical maximum by this percentage (default 15)
-report string
Cx1e2e JSON report file generated by cx1e2e (default "cx1e2e_result.json")
A normal invocation could look like:
cx1e2e-tracker.exe -report cx1e2e_result.json -history cx1e2e-history.json
In this mode, the history file (cx1e2e-history.json) is generated and, in subsequent executions, read & updated to contain a running tally of the average and maximum durations for sucessfull (PASS) tests in the report file (cx1e2e_result.json). When test results listed in the report have durations exceeding the average or maximum durations in the history file, a warning is generated. This can be used to identify tests that are still passing successfully, but are taking longer than expected.
Example output:
[WARNING][2025-04-02 13:03:53.652] Passing test examples\group\create.yaml - Create Group: /e2e-group-parent/e2e-group-subgroup2 duration 0.481069 exceeded historical max by 26%
[WARNING][2025-04-02 13:03:53.652] Passing test examples\group\create.yaml - Create Group: /e2e-group-parent/e2e-group-subgroup2 duration 0.481069 exceeded historical average by 26%
[WARNING][2025-04-02 13:03:53.652] Passing test examples\group\update.yaml - Update Group: /e2e-group-parent/e2e-group-subgroup2 duration 0.484639 exceeded historical max by 17%
[WARNING][2025-04-02 13:03:53.652] Passing test examples\group\update.yaml - Update Group: /e2e-group-parent/e2e-group-subgroup2 duration 0.484639 exceeded historical average by 17%
[WARNING][2025-04-02 13:03:53.652] Passing test examples\group\delete.yaml - Delete Group: e2e-group-parent duration 0.583426 exceeded historical max by 21%
[WARNING][2025-04-02 13:03:53.652] Passing test examples\group\delete.yaml - Delete Group: e2e-group-parent duration 0.583426 exceeded historical average by 21%
cx1e2e-tracker.exe:
-cmp-pct uint
When comparing two reports, hide tests that have durations that differ by less than this percentage (default 15)
-compare string
Cx1e2e JSON report file generated by cx1e2e to compare against (default "cx1e2e_result2.json")
-report string
Cx1e2e JSON report file generated by cx1e2e (default "cx1e2e_result.json")
A normal invocation could look like:
cx1e2e-tracker.exe -report cx1e2e_result.json -compare cx1e2e_result2.json
In this mode, the test details from both reports are compared, and a table is printed showing the tests, the PASS/FAIL/SKIP status and duration for both reports, along with a percentage difference between the test durations.
Example output:
Test Test ID .\report1.json .\report2.json Difference
examples\query\create.yaml - Create Application 1 PASS (0.39s) PASS (0.58s) +49.35%
examples\query\create.yaml - Create Preset 3 PASS (2.12s) PASS (2.48s) +17.27%
examples\query\create.yaml - Create Scan 4 PASS (63.00s) PASS (124.58s) +97.76%
examples\query\create.yaml - Read Result 5 PASS (1.22s) PASS (1.54s) +25.79%
examples\query\project.yaml - Create Scan 7 PASS (63.00s) PASS (122.90s) +95.08%
examples\query\project.yaml - Read Result 8 PASS (1.00s) PASS (1.35s) +34.87%
examples\query\app.yaml - Create Query 10 SKIP (0.00s) PASS (165.16s) -
examples\query\app.yaml - Create Scan 11 SKIP (0.00s) PASS (124.61s) -
examples\query\app.yaml - Read Result 12 SKIP (0.00s) PASS (1.29s) -
examples\query\app.yaml - Delete Query 13 SKIP (0.00s) PASS (33.80s) -
examples\query\corp.yaml - Create Query 14 PASS (131.57s) PASS (162.97s) +23.87%
The output omits tests which were skipped in both reports.