Skip to content

Reduce Pipnn Memory Consumption #1030

Description

@SeliMeli

Currently Pipnn consume more memory than Vamana, we need to explore options to reduce its mem footprint:

  1. reduce mem consumption by review/profiling/fix code issues in current implementation.
  2. use split build (partition + extract knn -> write knn graph on disk -> hashprune/final prune)
  3. like vamana, chunk build dataset and merge

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Fields

No fields configured for issues without a type.

Projects

Status
Todo

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions