Currently Pipnn consume more memory than Vamana, we need to explore options to reduce its mem footprint:
- reduce mem consumption by review/profiling/fix code issues in current implementation.
- use split build (partition + extract knn -> write knn graph on disk -> hashprune/final prune)
- like vamana, chunk build dataset and merge
Currently Pipnn consume more memory than Vamana, we need to explore options to reduce its mem footprint: