feat: integrate OMEGA adaptive early termination into zvec

[driPyf]

Closes #300

Greptile Summary

This PR introduces a new OMEGA index type to zvec, integrating adaptive early termination on top of HNSW. Instead of adding a separate search engine, the implementation keeps HNSW as the underlying graph traversal path and adds a learned query-time stopping policy that decides whether the current search state is already sufficient for a target recall. The implementation spans a new omega algorithm directory, OMEGA-aware searcher/streamer/index support, DB-layer training orchestration, Python bindings, benchmark integration, and Python workflow tests.

Key changes:

• OMEGA is now exposed as a first-class index type with dedicated index params and query params in both the core interfaces and Python bindings.
• OmegaSearcher and OmegaStreamer integrate OMEGA with the existing HNSW search loop through hook callbacks, so online search remains HNSW-based while query-time stop decisions come from OMEGALib.
• The offline pipeline now includes held-out query generation, ground-truth collection, search-trace collection, model training, and persistence of omega_model/ artifacts such as model.txt, threshold_table.txt, interval_table.txt, gt_collected_table.txt, and gt_cmps_all_table.txt.
• Runtime behavior still falls back to plain HNSW when no model is available or the dataset is below min_vector_threshold.
• Python workflow tests now cover insert → optimize/train → online OMEGA query, as well as the fallback-to-HNSW path when OMEGA is inactive.

Issues found:

• The current integration makes OMEGALib a required build dependency, so environments without working OpenMP or LightGBM toolchains will fail at configure time rather than transparently building without OMEGA.
• The Python workflow tests validate that the OMEGA query path runs and that fallback results match HNSW, but they do not directly assert that adaptive early termination actually triggered inside OMEGALib.
• The current query-time validation is intentionally behavioral rather than log- or metric-driven, so regressions in “OMEGA active but not stopping early” may not be caught by Python tests alone.

Confidence Score: 4/5

• Likely safe to merge after normal build validation. The overall design is coherent: HNSW remains the underlying search engine, OMEGA is layered as query-time control, fallback behavior is preserved, and the offline training loop is integrated into the DB lifecycle rather than living in an external script.
• The remaining risks are mainly around build environment requirements and the fact that Python tests validate end-to-end behavior more strongly than internal early-stop activation.
• Pay close attention to src/core/algorithm/omega/omega_searcher.cc, src/core/algorithm/omega/omega_streamer.cc, and src/db/training/omega_training_coordinator.cc — these files define the runtime activation/fallback logic, the HNSW hook integration, and the offline training lifecycle that make the feature work end to end.

Important Files Changed

Filename	Overview
src/core/algorithm/omega/omega_searcher.cc	OMEGA-aware searcher that loads persisted model artifacts, decides whether OMEGA should be active for a query, and falls back to plain HNSW when the model is unavailable or the dataset is below threshold.
src/core/algorithm/omega/omega_streamer.cc	Streaming/search-time integration layer that wires OMEGA into the HNSW loop through hooks, supports both inference and training modes, and persists the searcher-side OMEGA params alongside the index.
src/core/algorithm/omega/omega_context.h	Extends HnswContext with OMEGA-specific per-query state such as target_recall, training_query_id, and collected training outputs.
src/core/interface/indexes/omega_index.cc	Framework-facing OMEGA index wrapper that routes build/load/search lifecycle through the correct OMEGA-aware streamer/searcher path.
src/db/training/omega_training_coordinator.cc	Coordinates held-out query reuse, training-data collection, retraining flow, and model output management under omega_model/.
src/db/training/omega_model_trainer.cc	Bridges zvec-collected training records into OMEGALib’s C++ training API and writes the full set of OMEGA model artifacts.
src/db/index/segment/segment.cc	Adds OMEGA auto-training and retraining hooks into the segment optimize/build lifecycle.
src/db/index/column/vector_column/engine_helper.hpp	DB-to-core translation layer for OMEGA index params and runtime searcher config.
src/binding/python/model/param/python_param.cc	Python bindings for OmegaIndexParam, OmegaQueryParam, and optimize-time retraining options.
python/tests/test_collection.py	Python workflow tests covering OMEGA training, OMEGA query execution, and fallback-to-HNSW behavior.
thirdparty/CMakeLists.txt	Build integration for OMEGALib and its dependency wiring into zvec.
CMakeLists.txt	Top-level build wiring for OMEGA-related core libraries and tools.

Sequence Diagram

sequenceDiagram
    participant User
    participant Collection
    participant Segment
    participant OmegaTrainingCoordinator
    participant OMEGALib
    participant OmegaSearcher
    participant HNSW

    Note over User,Collection: Offline build / optimize
    User->>Collection: insert(docs)
    Collection->>Segment: build / persist vector index
    User->>Collection: optimize()
    Collection->>Segment: collect held-out queries and traces
    Segment->>OmegaTrainingCoordinator: training records + gt data
    OmegaTrainingCoordinator->>OMEGALib: train model
    OMEGALib-->>OmegaTrainingCoordinator: model + auxiliary tables
    OmegaTrainingCoordinator-->>Segment: persist omega_model/

    Note over User,Collection: Online query
    User->>Collection: query(VectorQuery, OmegaQueryParam)
    Collection->>OmegaSearcher: search(...)
    OmegaSearcher->>OmegaSearcher: should_use_omega()

    alt model available and threshold satisfied
        OmegaSearcher->>HNSW: search with OMEGA hooks
        HNSW->>OMEGALib: update SearchContext during traversal
        OMEGALib-->>HNSW: stop / continue
        HNSW-->>OmegaSearcher: results
    else fallback
        OmegaSearcher->>HNSW: plain HNSW search
        HNSW-->>OmegaSearcher: results
    end

    OmegaSearcher-->>Collection: results
    Collection-->>User: query results

Loading

[richyreachy]

use lower case function name

[richyreachy]

should telll whether the result is loaded successfully.

[feihongxu0824]

Hi @driPyf , thanks for the PR! Could you please resolve the merge conflicts with the main branch? Once that's done, I'll continue with the review. Thanks!

[feihongxu0824]

Also, could you please submit a PR to zvec-web beforehand? That PR would provide a more intuitive view of Omega's features and help users better understand how to perceive this capability.

[feihongxu0824]

CMake Warning at thirdparty/omega/OMEGALib/CMakeLists.txt:30 (message):
  OMEGA: OpenMP not found, building LightGBM without OpenMP


CMake Error at thirdparty/omega/OMEGALib/lightgbm/CMakeLists.txt:27 (cmake_minimum_required):
  CMake 3.28 or higher is required.  You are running version 3.26.5

cmake version needs update to 3.28?

[feihongxu0824]

after updating cmake version to 3.28, macOS local build still fail, env:

Apple clang version 16.0.0 (clang-1600.0.26.6)
Target: arm64-apple-darwin24.1.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

the error msg:

alias declarations are a C++11 extension
no template named 'function' in namespace 'std'
thread_local unknown type name
delegating constructors are permitted only in C++11

It seems that C++11/17 was not specified when compiling Eigen.