Skip to content

Flaky test fixed#2147

Closed
Derrick2000 wants to merge 3 commits intoapache:mainfrom
Derrick2000:flaky
Closed

Flaky test fixed#2147
Derrick2000 wants to merge 3 commits intoapache:mainfrom
Derrick2000:flaky

Conversation

@Derrick2000
Copy link
Copy Markdown

Dear Maintainers,

Our Nondex tool has detected a potential flaky test at org.apache.cxf.tools.wadlto.jaxrs.JAXRSContainerTest.testThrows. This flakiness is caused by an ordering issue when iterating over a HashMap and using assertArrayEquals in the unit test.

The issue with map.entrySet() lies in the fact that HashMap does not guarantee any specific iteration order for its entries. This lack of deterministic ordering introduces an element of randomness to the test execution, causing unreliable test results. Here are some reasons why this can happen:

Non-Deterministic Ordering in HashMap: HashMap inherently has no fixed order for its elements, and this order can change across different runs or Java versions. As a result, iterating over map.entrySet() may yield entries in a different sequence each time, depending on factors like hash code distribution and hash collisions.

Use of assertArrayEquals: The assertArrayEquals method in JUnit (and similar testing frameworks) checks for exact order in array elements. When the test iterates over a HashMap and collects entries into an array for comparison, any change in iteration order will cause assertArrayEquals to fail, even if the contents of the array are logically the same.

Impact on Test Consistency: This ordering inconsistency means that the test can pass or fail unpredictably, depending on the iteration order of the HashMap. Such flaky behavior can cause false negatives, where a test fails not because of a bug in the code under test, but due to the non-deterministic nature of HashMap ordering.

Concurrency and Threading Issues: If there are multiple threads accessing or modifying the HashMap during the test (intentionally or unintentionally), this can further exacerbate flakiness. Any modification to the map’s state while iterating can lead to ConcurrentModificationException or silent changes in iteration order.

Thus, we have suggested:

  1. HashMap to LinkedHashMap
  2. assertArrayEquals to assertEquals with Set collections

assertEquals(2, cc.getServiceClassNames().size());

final Map<String, Class<?>[]> methods = new HashMap<>();
final Map<String, Class<?>[]> methods = new LinkedHashMap<>();
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no dependency on ordering anywhere in the test (AFAICT) @Derrick2000 please clarify where the flakyness was observed

@gnodet
Copy link
Copy Markdown
Contributor

gnodet commented Mar 10, 2026

Closing — no evidence of actual test flakiness was provided, and the maintainer's question about where the flakiness was observed went unanswered.

@gnodet gnodet closed this Mar 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants