Skip to content

LUCENE-14611: Reduce memory pressure in TestTermInSetQuery.testDuel#16300

Open
lsh1215 wants to merge 5 commits into
apache:mainfrom
lsh1215:lucene-14611-testduel-oom
Open

LUCENE-14611: Reduce memory pressure in TestTermInSetQuery.testDuel#16300
lsh1215 wants to merge 5 commits into
apache:mainfrom
lsh1215:lucene-14611-testduel-oom

Conversation

@lsh1215

@lsh1215 lsh1215 commented Jun 27, 2026

Copy link
Copy Markdown

Description

This reduces avoidable memory pressure in TestTermInSetQuery.testDuel.

The test is checking query equivalence, not query-cache behavior. It still uses the randomized test-framework newSearcher(reader), but disables query caching for this test so repeated equivalence checks do not retain cached doc-id sets across iterations.

For the no-score comparison path, the test now collects matching doc IDs with the existing test-framework FixedBitSetCollector and compares the resulting bit sets. This avoids materializing full sorted TopDocs when only doc-id equality matters.

This is intentionally different from prior range-cap attempts such as #15084 and #15118: it does not reduce randomized input sizes or change the stress shape. Instead, it removes the pressure sources that are outside the behavior this test is meant to validate.

Local investigation

I could not reproduce the OOM on my machine with the original issue command, likely due to local environment differences. To make the failure mode visible locally, I used the same stress shape with the test JVM constrained to a 16m heap.

With the original code, that 16m squeeze reproduced:

java.lang.OutOfMemoryError: Java heap space

Heap dump inspection, jstat, and JFR heap profiling pointed to avoidable pressure from the randomized query cache retaining doc-id sets and from the no-score branch materializing sorted TopDocs.

Verification

Ran with JDK 26.

./gradlew tidy
BUILD SUCCESSFUL

./gradlew :lucene:core:compileTestJava
BUILD SUCCESSFUL

./gradlew :lucene:core:test --tests "org.apache.lucene.search.TestTermInSetQuery.testDuel" -Ptests.filter=@Nightly -Ptests.jvms=1 -Ptests.seed=B03A5F38917C1431 -Ptests.useSecurityManager=false -Ptests.gui=true -Ptests.file.encoding=UTF-8 -Ptests.vectorsize=128 -Ptests.forceintegervectors=true
:lucene:core:test (SUCCESS): 1 test

./gradlew :lucene:core:test --tests "org.apache.lucene.search.TestTermInSetQuery" -Ptests.jvms=1 -Ptests.seed=B03A5F38917C1431 -Ptests.useSecurityManager=false -Ptests.gui=true -Ptests.file.encoding=UTF-8 -Ptests.vectorsize=128 -Ptests.forceintegervectors=true
:lucene:core:test (SUCCESS): 15 tests, 1 skipped

./gradlew :lucene:core:test --tests "org.apache.lucene.search.TestTermInSetQuery.testSkipperOptimizationGapAssumption" -Ptests.asserts=true -Ptests.file.encoding=UTF-8 -Ptests.gui=true -Ptests.jvmargs= -Ptests.jvms=4 -Ptests.seed=A8777D0B61FEFFBE -Ptests.vectorsize=128
:lucene:core:test (SUCCESS): 1 test

I also re-ran the local 16m squeeze with the inner testDuel loop temporarily raised from 100 to 10000; with this fix applied, it completed successfully instead of reaching the Java heap OOM state observed with the original code.

Fixes #14611.

@github-actions github-actions Bot added this to the 11.0.0 milestone Jun 27, 2026
@lsh1215 lsh1215 marked this pull request as ready for review June 27, 2026 03:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TestTermInSetQuery.testDuel occasionally OOMs

1 participant