Flaky test report: committed-code failures on 2026-03-24

[andrross]

Committed-Code Test Failures — Past 12 Hours (as of 2026-03-24 ~22:45 UTC)

7 distinct test failures across 6 builds, all against committed code (Timer or Post Merge Action on main). Historical patterns include all build types.

---

1. MixedClusterClientYamlTestSuiteIT — 310_match_bool_prefix

• Recent build: #73223 (Timer)
• First seen: Mar 2024
• Total unique builds affected: 448
• Pattern: Persistently high flake for 2 years. Fails in every month without exception, typically 5–15 builds/month with periodic spikes (Sep 2024: 185 builds, Jan 2025/2026: ~20 builds). No improvement trend.

2. MixedClusterClientYamlTestSuiteIT — strict_allow_templates

• Recent build: #73215 (Timer)
• First seen: Jun 2024
• Total unique builds affected: 141
• Pattern: Moderate flake with periodic spikes. Spiked hard in Sep 2024 (66 builds) and Jan 2026 (14 builds), otherwise 1–5 builds/month.

3. AzureBlobStoreRepositoryTests.testWriteRead

• Recent build: #73222 (Post Merge Action)
• First seen: Apr 2024
• Total unique builds affected: 75
• Pattern: Worsening. Rate has roughly tripled over the past year — from 1–3 builds/month in 2024 to 5–9 builds/month in late 2025 through early 2026.

4. ClusterDisruptionIT.testAckedIndexing

• Recent build: #73190 (Timer)
• First seen: Apr 2024
• Total unique builds affected: 34
• Pattern: Stable low-moderate flake. Very rare in 2024 (3 total), picked up to 1–5 builds/month since mid-2025. No significant change recently.

5. SegmentReplicationWithNodeToNodeIndexShardTests

• Recent build: #73191 (Post Merge Action)
• Failed methods: classMethod, testPrimaryPromotionWithConcurrentTranslogRecovery
• First seen: Jun 2024
• Total unique builds affected: 39
• Pattern: Accelerating. Was sporadic (1 build/month) through early 2025, then increased in mid-2025. March 2026 is the worst month yet with 7 builds affected.

6. WarmIndexSegmentReplicationIT.testNodeDropWithOngoingReplication

• Recent build: #73194 (Timer)
• First seen: Mar 2025
• Total unique builds affected: 10
• Pattern: Stable, rare. About 1 year old, hitting roughly once a month with no change in pattern.

7. NodeJoinLeftIT.testClusterStabilityWhenDisconnectDuringSlowNodeLeftTask

• Recent build: #73232 (Timer)
• First seen: Jun 2025
• Total unique builds affected: 8
• Pattern: Stable, rare. About 9 months old, hitting roughly once every 1–2 months.

---

Summary

Test	Recent Build	First Seen	Builds	Trend
310_match_bool_prefix	#73223	Mar 2024	448	Persistently high, no improvement
strict_allow_templates	#73215	Jun 2024	141	Moderate with periodic spikes
AzureBlobStoreRepositoryTests	#73222	Apr 2024	75	Worsening
SegmentReplicationWithNodeToNode...	#73191	Jun 2024	39	Accelerating
ClusterDisruptionIT	#73190	Apr 2024	34	Stable low-moderate
WarmIndexSegmentReplicationIT	#73194	Mar 2025	10	Stable, rare
NodeJoinLeftIT	#73232	Jun 2025	8	Stable, rare

The two tests most worth attention are AzureBlobStoreRepositoryTests.testWriteRead (gradually worsening) and SegmentReplicationWithNodeToNodeIndexShardTests (accelerating sharply in March 2026).

Data sourced from the gradle-check-* indices on metrics.opensearch.org.