refactor: move validate_metrics_value to utilities/monitoring

[hmeir]

What this PR does / why we need it:

Move validate_metrics_value from tests/observability/utils.py to
utilities/monitoring.py for cross-directory reuse.
Now accepts str | int expected_value (int 0 for absent metrics).
Add unit tests. Update all observability callers.

Which issue(s) this PR fixes:

Special notes for reviewer:

Depends on PR #4972 (infra changes) — merge that first.
get_metrics_value returns int 0 when metric is absent vs str "0"
when emitted with value 0. expected_value=0 (int) matches absent only.

jira-ticket:

NONE

Summary by CodeRabbit

• Refactor

  • Reorganized metrics validation utilities for improved maintainability.
  • Centralized monitoring-related helper functions into dedicated modules.

• Tests

  • Added comprehensive test coverage for metrics validation.
  • Introduced alert verification testing utilities.

[coderabbitai]

Warning

Review limit reached

@hmeir, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 58 minutes. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 607bdff1-9dd8-445b-9be0-776888d3366a

📥 Commits

Reviewing files that changed from the base of the PR and between b1c300f and 6598193.

📒 Files selected for processing (13)

• conftest.py
• tests/conftest.py
• tests/observability/metrics/conftest.py
• tests/observability/metrics/test_aaq_metrics.py
• tests/observability/metrics/test_general_metrics.py
• tests/observability/metrics/test_metrics.py
• tests/observability/metrics/test_migration_metrics.py
• tests/observability/metrics/test_ssp_metrics.py
• tests/observability/metrics/test_vms_metrics.py
• tests/observability/upgrade/test_upgrade_observability.py
• tests/observability/utils.py
• utilities/monitoring.py
• utilities/unittests/test_monitoring.py

📝 Walkthrough

Walkthrough

This PR refactors validate_metrics_value from the test-local tests/observability/utils.py to the shared utilities/monitoring.py library, updates eight test modules to import from the new location, adds unit tests validating the implementation, and introduces a new alert verification helper in test utilities.

Changes

Metrics Validation Refactoring

Layer / File(s)	Summary
Move validate_metrics_value to utilities and refactor test utils utilities/monitoring.py, tests/observability/utils.py	validate_metrics_value is relocated to utilities/monitoring.py with datetime and type-checking imports. The function polls metrics via TimeoutSampler until a match or timeout, logging per-sample timestamps. tests/observability/utils.py removes the old function and introduces verify_no_listed_alerts_on_cluster, which queries Prometheus alert states and asserts no provided alerts are firing (respecting the allowlist).
Unit tests for validate_metrics_value implementation utilities/unittests/test_monitoring.py	TestValidateMetricsValue adds four test cases covering string "0" matching, integer 0 matching from absent metrics, type mismatch assertion (absent metric vs. string "0" expectation), and timeout when values never match.
Update test imports across observability test suite tests/observability/metrics/conftest.py, tests/observability/metrics/test_aaq_metrics.py, tests/observability/metrics/test_general_metrics.py, tests/observability/metrics/test_metrics.py, tests/observability/metrics/test_migration_metrics.py, tests/observability/metrics/test_ssp_metrics.py, tests/observability/metrics/test_vms_metrics.py, tests/observability/upgrade/test_upgrade_observability.py	All test files consistently update their imports to pull validate_metrics_value from utilities.monitoring instead of tests.observability.utils. Test logic remains unchanged; only the import source is modified.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Suggested labels

new-tests

---

Caution

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

• Ignore

❌ Failed checks (1 error)

Check name	Status	Explanation	Resolution
Stp Link Required	❌ Error	Newly added test file utilities/unittests/test_monitoring.py lacks required STP/RFE/Jira link in module or class docstring.	Add STP, RFE, or Jira link to module docstring (e.g., """Unit tests for monitoring module\n\nSTP: """) or class docstring.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The pull request title accurately and concisely describes the main change: moving the validate_metrics_value function from one module to another. It is 61 characters, well under the 120-character limit, and clearly communicates the refactoring effort.
Description check	✅ Passed	The pull request description covers all required template sections with substantive content: explains the rationale for the move, notes the function signature change (str | int support), mentions dependency on PR #4972, and includes special notes for reviewers about the int vs str semantics. All sections are populated.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.66%. Comparing base (a2dc143) to head (6598193).
⚠️ Report is 36 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4987      +/-   ##
==========================================
- Coverage   98.67%   98.66%   -0.02%     
==========================================
  Files          25       25              
  Lines        2487     2463      -24     
==========================================
- Hits         2454     2430      -24     
  Misses         33       33              

Flag	Coverage Δ
utilities	98.66% <100.00%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[openshift-virtualization-qe-bot-5]

Report bugs in Issues

Welcome! 🎉

This pull request will be automatically processed with the following features:

🔄 Automatic Actions

• Reviewer Assignment: Reviewers are automatically assigned based on the OWNERS file in the repository root
• Size Labeling: PR size labels (XS, S, M, L, XL, XXL) are automatically applied based on changes
• Issue Creation: A tracking issue is created for this PR and will be closed when the PR is merged or closed
• Branch Labeling: Branch-specific labels are applied to track the target branch
• Auto-verification: Auto-verified users have their PRs automatically marked as verified
• Labels: Enabled categories: branch, can-be-merged, cherry-pick, has-conflicts, hold, needs-rebase, size, verified, wip

📋 Available Commands

PR Status Management

• /wip - Mark PR as work in progress (adds WIP: prefix to title)
• /wip cancel - Remove work in progress status
• /hold - Block PR merging (approvers only)
• /hold cancel - Unblock PR merging
• /verified - Mark PR as verified
• /verified cancel - Remove verification status
• /reprocess - Trigger complete PR workflow reprocessing (useful if webhook failed or configuration changed)
• /regenerate-welcome - Regenerate this welcome message

Review & Approval

• /lgtm - Approve changes (looks good to me)
• /approve - Approve PR (approvers only)
• /assign-reviewers - Assign reviewers based on OWNERS file
• /assign-reviewer @username - Assign specific reviewer
• /check-can-merge - Check if PR meets merge requirements

Testing & Validation

• /retest tox - Run Python test suite with tox
• /retest build-container - Rebuild and test container image
• /retest verify-bugs-are-open - verify-bugs-are-open
• /retest all - Run all available tests

Container Operations

• /build-and-push-container - Build and push container image (tagged with PR number)
  • Supports additional build arguments: /build-and-push-container --build-arg KEY=value

Cherry-pick Operations

• /cherry-pick <branch> - Schedule cherry-pick to target branch when PR is merged
  • Multiple branches: /cherry-pick branch1 branch2 branch3

Label Management

• /<label-name> - Add a label to the PR
• /<label-name> cancel - Remove a label from the PR

✅ Merge Requirements

This PR will be automatically approved when the following conditions are met:

1. Approval: /approve from at least one approver
2. LGTM Count: Minimum 2 /lgtm from reviewers
3. Status Checks: All required status checks must pass
4. No Blockers: No wip, hold, has-conflicts labels and PR must be mergeable (no conflicts)
5. Verified: PR must be marked as verified

📊 Review Process

Approvers and Reviewers

Approvers:

• dshchedr
• myakove
• rnetser
• vsibirsk

Reviewers:

• OhadRevah
• RoniKishner
• albarker-rh
• dshchedr
• geetikakay
• hmeir
• rlobillo
• rnetser
• vsibirsk

Available Labels

• hold
• verified
• wip
• lgtm
• approve

AI Features

• Cherry-Pick Conflict Resolution: Enabled (claude/claude-opus-4-6[1m])

💡 Tips

• WIP Status: Use /wip when your PR is not ready for review
• Verification: The verified label is removed on new commits unless the push is detected as a clean rebase
• Cherry-picking: Cherry-pick labels are processed when the PR is merged
• Container Builds: Container images are automatically tagged with the PR number
• Permission Levels: Some commands require approver permissions
• Auto-verified Users: Certain users have automatic verification and merge privileges

For more information, please refer to the project documentation or contact the maintainers.

[OhadRevah]

/lgtm

[openshift-virtualization-qe-bot-3]

/retest all

Auto-triggered: Files in this PR were modified by merged PR #5012.

Overlapping files

tests/observability/metrics/test_general_metrics.py
tests/observability/metrics/test_metrics.py

[openshift-virtualization-qe-bot-3]

/retest all

Auto-triggered: Files in this PR were modified by merged PR #4934.

Overlapping files

tests/conftest.py

[openshift-virtualization-qe-bot-3]

/retest all

Auto-triggered: Files in this PR were modified by merged PR #4954.

Overlapping files

conftest.py
tests/observability/metrics/test_vms_metrics.py

[openshift-virtualization-qe-bot-3]

/retest all

Auto-triggered: Files in this PR were modified by merged PR #4985.

Overlapping files

tests/conftest.py

[openshift-virtualization-qe-bot-3]

/retest all

Auto-triggered: Files in this PR were modified by merged PR #5086.

Overlapping files

conftest.py