Skip to content

fix: clear Reliable/Unreliable consistency label with both metrics (fixes #149)#150

Merged
nerdCopter merged 2 commits into
masterfrom
fix/issue-149-consistency-warning-message
May 21, 2026
Merged

fix: clear Reliable/Unreliable consistency label with both metrics (fixes #149)#150
nerdCopter merged 2 commits into
masterfrom
fix/issue-149-consistency-warning-message

Conversation

@nerdCopter
Copy link
Copy Markdown
Owner

@nerdCopter nerdCopter commented May 21, 2026

Artifacts: https://github.com/nerdCopter/BlackBox_CSV_Render/actions/runs/26253231908


Fixes #149 — the consistency warning in Optimal P Estimation always blamed CV>40% even when the actual failure was Consistency<70%, causing confusing messages like CV=31.9% — unreliable (>40%).

  • Both metrics always shownReliable/Unreliable label followed by Consistency=XX% (≥70%), CV=XX% (≤40%), regardless of which condition failed
  • Unambiguous threshold notation≥/≤ makes the pass/fail direction self-evident for both metrics
  • CV=N/A when insufficient samples (coefficient_of_variation is None)
  • Consistent across console and PNG — same verbiage in both outputs, same color logic (orange when unreliable)
  • Removes Consistency=XX% from the compact header line — no longer duplicated
  • No warning icon — plain Reliable: / Unreliable: prefix, consistent in both states

Before

Roll: Td=22ms (target ...), Noise=LOW, Consistency=69%
  ⚠ Low consistency (Consistency=69% (<70%)) — unreliable

After

Roll: Td=22ms (target ...), Noise=LOW
  Unreliable: Consistency=69% (≥70%), CV=31.9% (≤40%)

Test plan

  • Clippy clean, cargo fmt applied
  • Ran all 18 ./input/*HELIO* files with --step --estimate-optimal-p
  • All three failure modes exercised across test files:
    • Consistency-only: 20260103_160240.01 Roll — Consistency=69% (≥70%), CV=31.9% (≤40%)
    • CV-only: 20260503_140158_master.01 Roll — Consistency=90% (≥70%), CV=90.1% (≤40%)
    • Both: 20260503_112224_PR1139_hover-test Roll — Consistency=54% (≥70%), CV=49.3% (≤40%)
  • All 18 PNGs generated without error

🤖 Generated with Claude Code

nerdCopter and others added 2 commits May 21, 2026 15:36
The warning message always attributed inconsistency to CV>40% regardless
of which condition actually failed. When consistency% was the trigger
(e.g. CV=31.9% but only 62% of samples within ±1 SD), the message
incorrectly implied CV exceeded the threshold.

Apply Option B (issue #149) to both console output and PNG label:
pattern-match (cv_failed, consistency_failed) and emit only the
threshold(s) that were actually crossed:
  - CV only:          "CV=42.3% (>40%)"
  - Consistency only: "Consistency=62% (<70%)"
  - Both:             "CV=42.3% (>40%) and Consistency=62% (<70%)"
  - Neither:          "insufficient samples (need >= 2)"

Flagging logic in is_consistent() is unchanged; only diagnostic text fixed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the per-condition failure match with a single always-shown
reliability line in both console output and PNG legend:

  Reliable:   Consistency=82% (≥70%), CV=25.1% (≤40%)
  Unreliable: Consistency=69% (≥70%), CV=31.9% (≤40%)

Changes:
- Remove Consistency from the compact header line (now in reliability line)
- Always emit both metrics regardless of which condition failed
- Use ≥/≤ threshold notation for unambiguous pass/fail direction
- Show CV=N/A when insufficient samples (coefficient_of_variation is None)
- Drop TD_SAMPLES_MIN_FOR_STDDEV import from plot_step_response (unused)
- Consistent plain-text prefix: no warning icon on either reliable or unreliable

Closes #149.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 21, 2026

📝 Walkthrough

Walkthrough

Console output and Optimal P legend are refactored to report both Consistency and Coefficient of Variation metrics together with their respective thresholds, replacing the prior single-metric CV-only message with dual-metric reporting based on the is_consistent() flag.

Changes

Unified Reliability and Consistency Reporting

Layer / File(s) Summary
Dual-metric reliability reporting: console and legend
src/plot_functions/plot_step_response.rs, src/data_analysis/optimal_p_estimation.rs
Import TD_CONSISTENCY_MIN_THRESHOLD constant; refactor console output to emit an always-present "Reliable:" or "Unreliable:" line displaying both Consistency and CV with thresholds derived from td_stats.is_consistent(); refactor Optimal P legend to compute cv_str and threshold-annotated cons_str, then set label and color based on reliability flag, replacing prior CV-only threshold attribution.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

  • nerdCopter/BlackBox_CSV_Render#145: Both PRs modify Optimal P estimation console and legend reporting logic to format Consistency and CV metrics using the same threshold values and is_consistent() determination.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: replacing misleading consistency labels with clear Reliable/Unreliable indicators showing both CV and consistency metrics.
Linked Issues check ✅ Passed The PR successfully addresses issue #149 by replacing the misleading single-metric CV warning with always-showing-both-metrics (Consistency=XX% and CV=XX%), clearly indicating pass/fail thresholds and reliability status.
Out of Scope Changes check ✅ Passed All changes are scoped to fixing the misleading consistency warning message: reformatting console output and PNG legend text in the two affected files without altering core is_consistent() logic.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/issue-149-consistency-warning-message

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
src/plot_functions/plot_step_response.rs (1)

633-667: ⚡ Quick win

Consider extracting duplicated reliability formatting logic.

This reliability formatting block (lines 635-649 specifically) is duplicated in optimal_p_estimation.rs lines 517-531. Both format CV and Consistency strings identically using the same constants and thresholds.

Consider extracting this into a helper method on TdStatistics, e.g., format_reliability_metrics() that returns (cv_str, cons_str), to maintain DRY and ensure future changes stay synchronized.

♻️ Suggested refactoring approach

Add to TdStatistics in optimal_p_estimation.rs:

impl TdStatistics {
    /// Format reliability metrics with thresholds for display
    pub fn format_reliability_metrics(&self) -> (String, String) {
        let cv_str = self.coefficient_of_variation.map_or_else(
            || "CV=N/A".to_string(),
            |cv| {
                format!(
                    "CV={:.1}% (≤{:.0}%)",
                    cv * 100.0,
                    TD_COEFFICIENT_OF_VARIATION_MAX * 100.0,
                )
            },
        );
        let cons_str = format!(
            "Consistency={:.0}% (≥{:.0}%)",
            self.consistency * 100.0,
            TD_CONSISTENCY_MIN_THRESHOLD * 100.0,
        );
        (cv_str, cons_str)
    }
}

Then in this file, replace lines 635-649 with:

-                        let cv_str = analysis.td_stats.coefficient_of_variation.map_or_else(
-                            || "CV=N/A".to_string(),
-                            |cv| {
-                                format!(
-                                    "CV={:.1}% (≤{:.0}%)",
-                                    cv * 100.0,
-                                    TD_COEFFICIENT_OF_VARIATION_MAX * 100.0,
-                                )
-                            },
-                        );
-                        let cons_str = format!(
-                            "Consistency={:.0}% (≥{:.0}%)",
-                            analysis.td_stats.consistency * 100.0,
-                            TD_CONSISTENCY_MIN_THRESHOLD * 100.0,
-                        );
+                        let (cv_str, cons_str) = analysis.td_stats.format_reliability_metrics();

Apply similar change in optimal_p_estimation.rs.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/plot_functions/plot_step_response.rs` around lines 633 - 667, Extract the
duplicated CV/Consistency formatting into a helper on TdStatistics (e.g., pub fn
format_reliability_metrics(&self) -> (String, String)) that returns the cv_str
and cons_str using TD_COEFFICIENT_OF_VARIATION_MAX and
TD_CONSISTENCY_MIN_THRESHOLD; then replace the duplicated blocks in
plot_step_response.rs (the block building cv_str/cons_str and cons_label) and
optimal_p_estimation.rs to call TdStatistics::format_reliability_metrics(), keep
the existing is_consistent() check to select label text
("Reliable"/"Unreliable") and color
(COLOR_OPTIMAL_P_TEXT/COLOR_OPTIMAL_P_WARNING), and push the same PlotSeries as
before. Ensure the new helper is added where TdStatistics is defined so both
modules can use it.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/plot_functions/plot_step_response.rs`:
- Around line 633-667: Extract the duplicated CV/Consistency formatting into a
helper on TdStatistics (e.g., pub fn format_reliability_metrics(&self) ->
(String, String)) that returns the cv_str and cons_str using
TD_COEFFICIENT_OF_VARIATION_MAX and TD_CONSISTENCY_MIN_THRESHOLD; then replace
the duplicated blocks in plot_step_response.rs (the block building
cv_str/cons_str and cons_label) and optimal_p_estimation.rs to call
TdStatistics::format_reliability_metrics(), keep the existing is_consistent()
check to select label text ("Reliable"/"Unreliable") and color
(COLOR_OPTIMAL_P_TEXT/COLOR_OPTIMAL_P_WARNING), and push the same PlotSeries as
before. Ensure the new helper is added where TdStatistics is defined so both
modules can use it.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: ad9e54e2-7cce-4add-9d4b-95d567815a37

📥 Commits

Reviewing files that changed from the base of the PR and between 00d4b60 and f79f39b.

📒 Files selected for processing (2)
  • src/data_analysis/optimal_p_estimation.rs
  • src/plot_functions/plot_step_response.rs
📜 Review details
🧰 Additional context used
📓 Path-based instructions (1)
src/**/*.rs

📄 CodeRabbit inference engine (AGENTS.md)

src/**/*.rs: All constants go in src/constants.rs — no hardcoded magic numbers in function code
Axis indices (0=Roll, 1=Pitch, 2=Yaw) must use the src/axis_names.rs module with AXIS_COUNT, AXIS_NAMES, and axis_name() function instead of creating redundant axis index constants
ALWAYS use Firmware revision header to detect firmware type, NEVER use Firmware type
Firmware revision patterns: Betaflight uses Betaflight 4.x.x or Betaflight YYYY.MM, Emuflight uses EmuFlight 0.x.x, INAV uses INAV x.x.x

Files:

  • src/plot_functions/plot_step_response.rs
  • src/data_analysis/optimal_p_estimation.rs
🔇 Additional comments (3)
src/plot_functions/plot_step_response.rs (1)

15-15: LGTM!

src/data_analysis/optimal_p_estimation.rs (2)

506-506: LGTM!


515-537: LGTM!

@nerdCopter nerdCopter merged commit f1d13b7 into master May 21, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: misleading CV consistency warning message — blames CV>40% when actual failure is consistency<70%

1 participant