Skip to content

Conversation

@Akhil-Pathivada
Copy link
Contributor

@Akhil-Pathivada Akhil-Pathivada commented Dec 23, 2025

Overview

Adds Disk-based vector search support for OSS OpenSearch (requires 2.17+).

Features

  • Disk-based mode with 6 compression levels (1x, 2x, 4x, 8x, 16x, 32x)
  • Auto engine selection based on compression level (Lucene: 1x/4x, FAISS: 2x/8x/16x/32x)
  • Rescoring support with configurable oversample factor
  • Conditional UI - disk/in-memory options shown contextually
  • Version validation - requires OpenSearch 2.17+

Changes

Backend (config.py, oss_opensearch.py)

  • Added on_disk, compression_level, oversample_factor configuration
  • Implemented CompressionLevel class with engine mapping
  • Added resolved_engine property for automatic engine selection
  • Extracted _build_vector_field_mapping() for disk vs in-memory modes
  • Applied rescoring for both disk-based and in-memory quantization modes

Frontend (models.py, dbCaseConfigs.py)

  • Added 3 new enum types for disk-based parameters
  • Created 5 OSS-specific UI inputs with conditional display logic
  • Reused AWS inputs for common parameters (no duplication)

Technical Notes

  • Mutual exclusivity: Disk compression and in-memory quantization cannot coexist
  • Backward compatible: Default on_disk=False preserves existing behavior
  • No AWS impact: AWS OpenSearch configurations unchanged

Screenshots

Screenshot 2025-12-24 at 01 39 24 Screenshot 2025-12-24 at 01 39 47

@Akhil-Pathivada Akhil-Pathivada force-pushed the feature/oss-opensearch-disk-based-mode branch 2 times, most recently from 50866e1 to f854146 Compare December 23, 2025 17:56
@Akhil-Pathivada Akhil-Pathivada force-pushed the feature/oss-opensearch-disk-based-mode branch from f854146 to 53960ce Compare December 23, 2025 20:04
@Akhil-Pathivada Akhil-Pathivada marked this pull request as ready for review December 23, 2025 20:11
@Akhil-Pathivada
Copy link
Contributor Author

/assign @alwayslove2013

@sre-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Akhil-Pathivada, alwayslove2013
To complete the pull request process, please assign xuanyang-cn after the PR has been reviewed.
You can assign the PR to them by writing /assign @xuanyang-cn in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@alwayslove2013 alwayslove2013 merged commit 67f3c98 into zilliztech:main Dec 24, 2025
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants