Cardinality aggregation dynamic pruning changes

[rishabhmaurya]

Changes to experiment with Dynamic pruning for cardinality aggregation described in opensearch-project#11959.

Here is the breakdown of algorithm -

1. Check for all preconditions on when this optimization can be enabled -

• 1. Only enabled when Cardinality Aggregation is the only aggregation.
• 2. The field is a low cardinality field.
• 3. Field type is one of Keyword, Numeric?
• 4. Other?

2. Once preconditions are met, while collectors are created and picked for a given segment, create a DynamicPruningCollectorWrapper to wrap the collector with optimization.

3. DynamicPruningCollectorWrapper will enumerate all the terms for the given field and creates a DisjunctionWithDynamicPruningScorer similar to DisjunctionScorer in lucene in conjunction with the parent query. DisjunctionWithDynamicPruningScorer scorer should have following capabilities in addition to what DisjunctionScorer have -

• 1. #removeAllDISIsOnCurrentDoc() - it removes all the DISIs for subscorer pointing to current doc. This is helpful in dynamic pruning for Cardinality aggregation, where once a term is found, it becomes irrelevant for rest of the search space, so this term's subscorer DISI can be safely removed from list of subscorer to process.
• 2. #removeAllDISIsOnCurrentDoc() breaks the invariant of Conjuction DISI i.e. the docIDs of all sub-scorers should be ess than or equal to current docID iterator is pointing to. When we remove elements from priority, it results in heapify action, which modifies the top of the priority queye, which represents the current docID for subscorers here. To address this, we are wrapping the iterator with SlowDocIdPropagatorDISI which keeps the iterator pointing to last docID before #removeAllDISIsOnCurrentDoc() is called and updates this docID only when next() or advance() is called.

4. When collection of document will start and DynamicPruningCollectorWrapper is used, it will collect all the documents at once by iterating over all the document from the query created in step 3.

5. Dynamic pruning step when collecting a document - when a match is found, all the terms for a given document will be enumerated and collected for cardinality computation. Once done, the subscorer DISI corresponding to each of these terms collector can be safely removed from the DisjunctionWithDynamicPruningScorer by calling removeAllDISIsOnCurrentDoc(). Once all docs are collector, we can straightaway throw CollectionTerminatedException for early termination of query.

Note: to be used only for prototype and reference purpose, not intended to merge to main. It may contain a lot of bugs and definitely doesn't cover all preconditions.

Description

[Describe what this change achieves]

Related Issues

Resolves opensearch-project#11959

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[rishabhmaurya]

we should probably add an assertion on how many times collector.collect gets called, which should be 2 when dynamic pruning is applied vs 5 when its not applied?

[sandeshkr419]

I have added test utilities (CountingAggregator) in opensearch-project#11643 with which we can assert the count of collect() calls once the changes are in. I'm asserting similar things in my PR.