Add an rb_group_elements_by_source function

[chriswheeldon-peakon]

Summary

This PR is a proposal for a new set returning function, rb_group_elements_by_source(bitmaps roaringbitmap[]) → TABLE (sources int[], members roaringbitmap), that, given an array of input bitmaps returns a table, where each row contains an array of indexes into the source bitmap array and a single bitmap with the common set of elements that were contained in those source bitmaps.

For example,

select sources, rb_to_array(members) from rb_group_elements_by_source(ARRAY[
  rb_build(ARRAY[1,2,3,4,5,6,7,8,9,10]),
  rb_build(ARRAY[1,2,3,4,5,11,12]),
  rb_build(ARRAY[1,2,3,13,14,15])
]) order by sources::text;
 sources | rb_to_array  
---------+--------------
 {1,2,3} | {1,2,3}
 {1,2}   | {4,5}
 {1}     | {6,7,8,9,10}
 {2}     | {11,12}
 {3}     | {13,14,15}

i.e.

• members 1, 2 and 3 appeared in all three source bitmaps
• members 4 and 5 only appeared in source bitmaps 1 and 2.
• members 6 to 10 only appeared in the first source bitmap
• etc.

For our case, we need to perform this grouping on between 10 and 200 bitmaps, each containing on the order ~100,000 elements. We've tried, therefore, to focus on finding a performant implementation and, though our use-case is likely very specific to our domain and problem space, we have tried to extract the computationally expensive part out into what we hope is a generic enough and potentially generally useful enough function to make it suitable for inclusion in this library.

I haven't yet implemented a 64-bit version, as I was hoping to get some feedback on this before proceeding with that.

Many thanks for this library and for taking the time to look over this proposal 🙇

Implementation Details

The function builds a min-heap over the input bitmaps iterators and a hashtable mapping a given combination of input array indices, represented as a bitmask, to a bitmap. It then repeatedly pops from the heap to find all the bitmaps that contain a given element (i.e. a k-way merge), updating the bitmask as it goes. When no more iterators exist that return that element we look up the bitmask in the hashtable, inserting if it does not exist, and then add the value to the corresponding bitmap. We then continue popping off the heap for the next element.

[ChenHuajun]

Thank you for this excellent proposal and the detailed explanation! This function addresses a very interesting use case for multi-set analysis.
I am generally supportive of merging this feature. Hope you can further polish this feature—for instance, adding 64-bit support, supplementing documentation and test coverage.
Looking forward to your updates!

[chriswheeldon-peakon]

Thanks a lot @ChenHuajun!

In terms of the 64-bit implementation. I'm just thinking about the amount of code duplication this could entail and whether putting it behind a C++ implementation using generics may allow it to be a lot more concise (we could probably also simplify some of the algorithmic code here using the C++ stdlib).

How would you feel about introducing C++ into the project? If you are against then I'm happy to keep going with a C implementation.