Flaky test report: committed-code failures on 2026-03-27

[andrross]

Flaky test report: committed-code failures on 2026-03-27

Summary

11 test failures were detected across committed-code builds (Timer/Post Merge Action) in the past 24 hours, covering 10 distinct test methods across 6 builds. None of the locally-reproducible failures reproduced with the original seed, confirming these are non-deterministic (flaky) failures.

Failing Tests

1. AzureBlobStoreRepositoryTests.testSnapshotAndRestore

• Build: 73350 (seed: 3DC2D440A4A83FB8)
• Reproduced locally: Skipped (requires Docker for azure-fixture)
• First failure: 2024-03-28
• Unique builds affected: 134
• Pattern: Stable flake. Fails consistently across all months since March 2024, typically 2-19 builds/month. The most prolific flaky test in this set. No improvement trend.

2. ClusterRerouteIT.testDelayWithALargeAmountOfShards

• Build: 73353 (seed: 8DA18342FC5337F8)
• Reproduced locally: No
• First failure: 2024-03-25
• Unique builds affected: 115
• Pattern: Was very active in early 2024 (35 builds in April, 36 in June), then largely subsided. Resurfaced at low levels since late 2025 (2-6 builds/month). Improving overall.

3. AzureBlobStoreRepositoryTests.testRequestStats

• Build: 73350 (seed: 3DC2D440A4A83FB8)
• Reproduced locally: Skipped (requires Docker for azure-fixture)
• First failure: 2024-04-02
• Unique builds affected: 111
• Pattern: Stable flake. Consistent 2-16 builds/month since April 2024. No improvement trend.

4. FullRollingRestartIT.testFullRollingRestart_withNoRecoveryPayloadAndSource

• Build: 73375 (seed: F0F31BE72316BD5C)
• Reproduced locally: No
• First failure: 2024-10-11
• Unique builds affected: 99
• Pattern: Bursty. Massive spike in July 2025 (47 builds) and August 2025 (24 builds), then quiet until February 2026 (17 builds) and March 2026 (8 builds). Worsening — appears to be in an active burst.

5. RecoveryWhileUnderLoadIT.testRecoverWhileUnderLoadWithReducedAllowedNodes

• Build: 73365 (seed: 32CCAC04D40896B)
• Reproduced locally: No
• First failure: 2024-04-03
• Unique builds affected: 81
• Pattern: Bursty. Major spike in June-August 2025 (29+21+9 builds), quiet September-January, then resurfaced in February-March 2026 (4+13 builds). Worsening — currently in an active burst.

6. AzureBlobStoreRepositoryTests.testDeleteBlobs

• Build: 73365 (seed: 32CCAC04D40896B)
• Reproduced locally: Skipped (requires Docker for azure-fixture)
• First failure: 2024-04-05
• Unique builds affected: 59
• Pattern: Stable low-level flake. 1-8 builds/month since April 2024. Slightly worsening in recent months (4-5 builds/month in late 2025 through March 2026).

7. RemoteShrinkIndexIT.testCreateShrinkIndex

• Build: 73359 (seed: 501AA23F75523CE1)
• Reproduced locally: No
• First failure: 2024-03-27
• Unique builds affected: 55
• Pattern: Bursty. Massive spike in November 2025 (41 builds), otherwise very quiet (0-3 builds/month). Currently quiet — only 1 build in March 2026.

8. Netty4HttpRequestSizeLimitIT.testDoesNotLimitExcludedRequests

• Build: 73350 (seed: 3DC2D440A4A83FB8)
• Reproduced locally: No
• First failure: 2025-04-29
• Unique builds affected: 36
• Pattern: Stable low-level flake. 1-5 builds/month since April 2025. No clear trend.

9. SmokeTestMultiNodeClientYamlTestSuiteIT (histogram profiler)

• Build: 73375 (seed: F0F31BE72316BD5C)
• Reproduced locally: Skipped (YAML REST test — cannot filter by parameterized test name)
• First failure: 2024-08-07
• Unique builds affected: 14
• Pattern: Rare flake. Sporadic failures across months, 0-2 builds/month. Stable at low levels.

10. IngestFromKafkaIT.testDynamicUpdateKafkaParams

• Build: 73372 (seed: 572E2D6246ECD3F4)
• Reproduced locally: No
• First failure: 2026-02-03
• Unique builds affected: 3
• Pattern: New flake. Only 3 occurrences since February 2026. Too early to determine trend.

Summary Table

#	Test	Build	Unique Builds	First Failure	Reproduced	Trend
1	AzureBlobStoreRepositoryTests.testSnapshotAndRestore	73350	134	2024-03-28	Skipped (Docker)	Stable
2	ClusterRerouteIT.testDelayWithALargeAmountOfShards	73353	115	2024-03-25	No	Improving
3	AzureBlobStoreRepositoryTests.testRequestStats	73350	111	2024-04-02	Skipped (Docker)	Stable
4	FullRollingRestartIT.testFullRollingRestart_withNoRecoveryPayloadAndSource	73375	99	2024-10-11	No	Worsening
5	RecoveryWhileUnderLoadIT.testRecoverWhileUnderLoadWithReducedAllowedNodes	73365	81	2024-04-03	No	Worsening
6	AzureBlobStoreRepositoryTests.testDeleteBlobs	73365	59	2024-04-05	Skipped (Docker)	Stable
7	RemoteShrinkIndexIT.testCreateShrinkIndex	73359	55	2024-03-27	No	Bursty
8	Netty4HttpRequestSizeLimitIT.testDoesNotLimitExcludedRequests	73350	36	2025-04-29	No	Stable
9	SmokeTestMultiNodeClientYamlTestSuiteIT (histogram profiler)	73375	14	2024-08-07	Skipped (YAML)	Stable
10	IngestFromKafkaIT.testDynamicUpdateKafkaParams	73372	3	2026-02-03	No	New

Methodology

• Failures were collected from the gradle-check-* index in the OpenSearch metrics cluster, filtered to Timer runs on main and Post Merge Action runs.
• Historical flake rates were computed across all build types (including PR builds) using monthly date histograms with cardinality aggregation on build_number.
• Local reproduction was attempted using the exact seed from the failing build's Jenkins console log.
• Azure repository tests were skipped because they require Docker (azure-fixture container).
• The YAML REST test was skipped because parameterized test names cannot be filtered via --tests.