From 22ac284865ec08991b219b9e6cdd6de1fbd12492 Mon Sep 17 00:00:00 2001 From: Edgar Babajanyan Date: Wed, 6 May 2026 21:13:07 -0700 Subject: [PATCH 1/2] README: document mmap vectors, disk-backed chunks, storage architecture Add storage architecture section showing the on-disk layout and how vectors, HNSW indices, and chunk metadata are persisted. Update the feature list with mmap vectors, redb chunk store, and incremental HNSW indexing. Co-Authored-By: Claude Opus 4.6 (1M context) --- README.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/README.md b/README.md index e3005e9..09af2c9 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,9 @@ Embedded vector + full-text search engine for [Captain](https://runcaptain.com). - **Full-text search** via Tantivy (BM25) with precomputed bitset faceting (microsecond facets) - **Vector search** via USearch HNSW (mmap-backed, disk-persistent) - **Hybrid search** via Reciprocal Rank Fusion (RRF, k=60) +- **Memory-mapped vector storage**. Raw vectors live on disk, not in RAM. Zero-copy reads via mmap. +- **Disk-backed chunk metadata** via redb (pure Rust embedded DB). Handles millions of documents without loading them all into memory. +- **Incremental HNSW indexing**. Adding vectors appends to the index — no full rebuild required. - **Named vector spaces** ... run multiple embedding models on the same collection - **One-click model upgrades** with background re-embedding and atomic swap - **Parent-child documents** with relationship-aware scoring (TAMS video search compatible) @@ -375,6 +378,29 @@ docker run -p 8080:80 --gpus all ghcr.io/huggingface/text-embeddings-inference \ MTEB and MMEB are different benchmarks on different scales. MTEB scores are 0-100 (text tasks). MMEB scores are 0-1 (cross-modal retrieval). They cannot be compared directly. +## Storage architecture + +Compass keeps vector data and chunk metadata on disk, not in RAM. + +``` +data/{collection}/ +├── collection.json # Collection metadata (name, dims, spaces) +├── relationships.bin # Parent-child + sibling graph +├── tantivy/ # BM25 inverted index (disk-backed) +└── vectors/ + ├── {space}.index # USearch HNSW graph (mmap on read) + ├── {space}.bin # Raw f32 vectors (mmap via MmapVectors) + └── {space}.keymap # HNSW key → chunk ID mapping +``` + +**Vectors**: Stored in a flat `[u32 dims][u32 count][f32...]` file, memory-mapped at query time. Adding vectors appends to the file and remaps — no full rewrite. At 1M vectors × 768 dims this is ~3GB on disk, near-zero RSS. + +**HNSW index**: Built incrementally via USearch `.add()` + `.save()`. Loaded via `.load()` for mutation or `.view()` for read-only mmap. The graph structure is separate from the raw vectors. + +**Chunk metadata**: Persisted via redb (pure Rust, ACID, MVCC). Point lookups by chunk ID during search result assembly. Batch inserts during ingestion. + +**Ingestion path**: New vectors are appended to the mmap file, inserted into the HNSW graph incrementally, and chunk metadata is written to redb — all without cloning existing data. + ## Throughput and scaling **Query throughput.** USearch HNSW serves around 15k QPS per instance on a 16-core box at p99 < 50ms for top-10 retrieval. For very high QPS workloads, shard collections across multiple Compass instances behind a load balancer. From 34ec1b97800de031023f8f1d36b4bdbbc0dee881 Mon Sep 17 00:00:00 2001 From: Edgar Babajanyan Date: Fri, 8 May 2026 12:45:22 -0700 Subject: [PATCH 2/2] Open-source readiness: community files, release automation, changelog MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add standard OSS governance files: - CODE_OF_CONDUCT.md (Contributor Covenant v2.1) - SECURITY.md (vulnerability disclosure policy) - .github/ISSUE_TEMPLATE/ (bug report, feature request, config) - .github/PULL_REQUEST_TEMPLATE.md - .github/CODEOWNERS - docker-compose.yml for local quickstart Add release automation: - .github/workflows/release.yml — triggered on v* tags - Builds linux-amd64, linux-arm64, macos-arm64 binaries - Pushes Docker image to ghcr.io - Creates GitHub Release with artifacts Update README: - Add CI, license, and Rust version badges - Add Contributing and Security sections with links Update CHANGELOG: - Add mmap vector storage, disk-backed chunks, incremental HNSW from PR #1 to [Unreleased] section - Document MSRV bump to 1.88 and Dockerfile changes Co-Authored-By: Claude Opus 4.6 (1M context) --- .github/CODEOWNERS | 9 +++ .github/ISSUE_TEMPLATE/bug_report.yml | 49 ++++++++++++ .github/ISSUE_TEMPLATE/config.yml | 5 ++ .github/ISSUE_TEMPLATE/feature_request.yml | 23 ++++++ .github/PULL_REQUEST_TEMPLATE.md | 16 ++++ .github/workflows/release.yml | 87 ++++++++++++++++++++++ CHANGELOG.md | 6 ++ CODE_OF_CONDUCT.md | 38 ++++++++++ README.md | 12 +++ SECURITY.md | 24 ++++++ docker-compose.yml | 18 +++++ 11 files changed, 287 insertions(+) create mode 100644 .github/CODEOWNERS create mode 100644 .github/ISSUE_TEMPLATE/bug_report.yml create mode 100644 .github/ISSUE_TEMPLATE/config.yml create mode 100644 .github/ISSUE_TEMPLATE/feature_request.yml create mode 100644 .github/PULL_REQUEST_TEMPLATE.md create mode 100644 .github/workflows/release.yml create mode 100644 CODE_OF_CONDUCT.md create mode 100644 SECURITY.md create mode 100644 docker-compose.yml diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS new file mode 100644 index 0000000..2ba1566 --- /dev/null +++ b/.github/CODEOWNERS @@ -0,0 +1,9 @@ +# Default owners for everything +* @runcaptain/eng + +# Vector backends +crates/compass-index-api/ @runcaptain/eng +crates/compass-vector-gpu/ @runcaptain/eng + +# CI and release +.github/ @runcaptain/eng diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml new file mode 100644 index 0000000..8a1c3a3 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -0,0 +1,49 @@ +name: Bug Report +description: Report a bug in Compass +labels: ["bug"] +body: + - type: textarea + id: description + attributes: + label: Description + description: What happened? What did you expect to happen? + validations: + required: true + - type: textarea + id: reproduce + attributes: + label: Steps to reproduce + description: Minimal steps or curl commands to reproduce the issue. + placeholder: | + 1. Create collection: curl -X POST localhost:4001/collections ... + 2. Ingest data: curl -X POST localhost:4001/collections/test/ingest ... + 3. Search: curl -X POST localhost:4001/collections/test/search ... + validations: + required: true + - type: input + id: version + attributes: + label: Compass version + description: Output of the /health endpoint or git commit SHA. + placeholder: "v0.2.0 / abc1234" + validations: + required: true + - type: dropdown + id: os + attributes: + label: Operating system + options: + - Linux (x86_64) + - Linux (aarch64) + - macOS (Apple Silicon) + - macOS (Intel) + - Docker + - Other + validations: + required: true + - type: textarea + id: logs + attributes: + label: Relevant logs + description: Paste any error messages or stack traces. + render: shell diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 0000000..f6dbf1f --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,5 @@ +blank_issues_enabled: true +contact_links: + - name: Security vulnerability + url: mailto:security@runcaptain.com + about: Report security issues privately via email — do not open a public issue. diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml new file mode 100644 index 0000000..6b951c1 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.yml @@ -0,0 +1,23 @@ +name: Feature Request +description: Suggest a new feature or improvement +labels: ["enhancement"] +body: + - type: textarea + id: problem + attributes: + label: Problem + description: What problem does this feature solve? + validations: + required: true + - type: textarea + id: solution + attributes: + label: Proposed solution + description: How should this work? Include API examples if relevant. + validations: + required: true + - type: textarea + id: alternatives + attributes: + label: Alternatives considered + description: What other approaches did you consider? diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 0000000..a9bcf20 --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -0,0 +1,16 @@ +## Summary + + + +## Test plan + + + +- [ ] `cargo test` passes +- [ ] `cargo clippy` clean +- [ ] `cargo fmt` clean +- [ ] Tested with Docker build (if applicable) + +## Changelog + + diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml new file mode 100644 index 0000000..e03d684 --- /dev/null +++ b/.github/workflows/release.yml @@ -0,0 +1,87 @@ +name: Release + +on: + push: + tags: + - "v*" + +env: + CARGO_TERM_COLOR: always + +jobs: + build: + name: Build (${{ matrix.target }}) + runs-on: ${{ matrix.os }} + strategy: + matrix: + include: + - target: x86_64-unknown-linux-gnu + os: ubuntu-24.04 + artifact: compass-linux-amd64 + - target: aarch64-unknown-linux-gnu + os: ubuntu-24.04 + artifact: compass-linux-arm64 + - target: aarch64-apple-darwin + os: macos-14 + artifact: compass-macos-arm64 + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + with: + targets: ${{ matrix.target }} + - uses: Swatinem/rust-cache@v2 + - if: matrix.target == 'aarch64-unknown-linux-gnu' + run: | + sudo apt-get update + sudo apt-get install -y gcc-aarch64-linux-gnu cmake pkg-config libssl-dev + echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc" >> $GITHUB_ENV + - if: matrix.os == 'ubuntu-24.04' && matrix.target == 'x86_64-unknown-linux-gnu' + run: | + sudo apt-get update + sudo apt-get install -y cmake pkg-config libssl-dev + - run: cargo build --release --target ${{ matrix.target }} -p compass + - run: | + mkdir -p dist + cp target/${{ matrix.target }}/release/compass dist/${{ matrix.artifact }} + - uses: actions/upload-artifact@v4 + with: + name: ${{ matrix.artifact }} + path: dist/${{ matrix.artifact }} + + docker: + name: Docker image + runs-on: ubuntu-24.04 + permissions: + contents: read + packages: write + steps: + - uses: actions/checkout@v4 + - uses: docker/login-action@v3 + with: + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + - uses: docker/build-push-action@v6 + with: + context: . + push: true + tags: | + ghcr.io/${{ github.repository }}:${{ github.ref_name }} + ghcr.io/${{ github.repository }}:latest + + release: + name: GitHub Release + needs: [build, docker] + runs-on: ubuntu-24.04 + permissions: + contents: write + steps: + - uses: actions/checkout@v4 + - uses: actions/download-artifact@v4 + with: + path: dist + merge-multiple: true + - uses: softprops/action-gh-release@v2 + with: + generate_release_notes: true + files: dist/* diff --git a/CHANGELOG.md b/CHANGELOG.md index ff048f4..8098e07 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added +- **Memory-mapped vector storage.** New `MmapVectors` module (`search/mmap_vectors.rs`) replaces `Vec>` with a flat mmap-backed file. Zero-copy reads via `bytemuck::cast_slice`, append-only writes, near-zero RSS for vector data regardless of dataset size. File format: `[u32 dims][u32 count][f32...]`. +- **Disk-backed chunk metadata.** New `ChunkStore` module (`search/chunk_store.rs`) backed by redb (pure Rust embedded DB). Replaces in-memory `HashMap` with persistent, ACID-compliant storage. Point lookups, batch inserts, full scans for rebuild. +- **Incremental HNSW indexing.** Ingest path now loads the existing USearch index via `.load()`, appends new vectors with `.add()`, and saves — instead of cloning all vectors and rebuilding from scratch. Falls back to full rebuild when the `Arc` cannot be unwrapped. +- **New dependencies:** `memmap2` (mmap), `bytemuck` (zero-copy cast), `redb` (embedded DB). - **Cargo workspace layout.** Source moved from a single crate to a workspace under `crates/`. Splits: `compass` (umbrella + binary), `compass-index-api` (stable trait surface), `compass-vector-gpu` (optional GPU backend). - **Vector index trait abstraction.** New `compass_index_api::VectorIndex` trait abstracts over CPU and GPU backends. The existing USearch path is wrapped in `UsearchHnswIndex`; new backends bind to the trait without touching call sites. - **Optional GPU backend (`--features gpu`).** New `compass-vector-gpu` crate integrates NVIDIA cuVS for CAGRA→HNSW build acceleration. CPU-side search after conversion. Linux + CUDA 12+ only. See `ARCHITECTURE.md` for build prerequisites. @@ -25,6 +29,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), - **Cargo manifest** is now a workspace root with shared `[workspace.dependencies]` and `[workspace.package]` metadata. Per-crate manifests inherit version, edition, and rust-version. - **Compiler profile** adds `lto = "thin"` and `codegen-units = 1` to release builds for better optimization. Adds debug symbols to bench builds. +- **MSRV bumped to 1.88** from 1.82. Required by `time@0.3.47` (edition 2024) and `icu_collections@2.2.0`. +- **Dockerfile** upgraded to `rust:latest` + `debian:trixie-slim` for glibc compatibility with newer Rust toolchains. ### Deprecated diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 0000000..87e8d56 --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,38 @@ +# Contributor Covenant Code of Conduct + +## Our Pledge + +We as members, contributors, and leaders pledge to make participation in our +community a harassment-free experience for everyone, regardless of age, body +size, visible or invisible disability, ethnicity, sex characteristics, gender +identity and expression, level of experience, education, socio-economic status, +nationality, personal appearance, race, caste, color, religion, or sexual +identity and orientation. + +## Our Standards + +Examples of behavior that contributes to a positive environment: + +* Using welcoming and inclusive language +* Being respectful of differing viewpoints and experiences +* Gracefully accepting constructive criticism +* Focusing on what is best for the community +* Showing empathy towards other community members + +Examples of unacceptable behavior: + +* The use of sexualized language or imagery, and sexual attention or advances of any kind +* Trolling, insulting or derogatory comments, and personal or political attacks +* Public or private harassment +* Publishing others' private information without explicit permission +* Other conduct which could reasonably be considered inappropriate in a professional setting + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported to the project team at **conduct@runcaptain.com**. All complaints will +be reviewed and investigated promptly and fairly. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org/), version 2.1. diff --git a/README.md b/README.md index 09af2c9..a068d51 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,9 @@ # Compass +[![CI](https://github.com/runcaptain/compass/actions/workflows/ci.yml/badge.svg)](https://github.com/runcaptain/compass/actions/workflows/ci.yml) +[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE) +[![Rust](https://img.shields.io/badge/rust-1.88%2B-orange.svg)](https://www.rust-lang.org) + Embedded vector + full-text search engine for [Captain](https://runcaptain.com). Single binary, zero external dependencies. Designed for high-throughput retrieval. Built for on-prem enterprise deployments where customer data cannot leave their VPC. ## What it does @@ -439,6 +443,14 @@ PUT /collections/:name/default-vector-space Switch default space GET /health Health check ``` +## Contributing + +See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, PR guidelines, and commit conventions. + +## Security + +To report a vulnerability, email **security@runcaptain.com**. See [SECURITY.md](SECURITY.md) for details. + ## License Apache 2.0. See [LICENSE](LICENSE). \ No newline at end of file diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..a70dbc2 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,24 @@ +# Security Policy + +## Supported Versions + +| Version | Supported | +| ------- | ------------------ | +| 0.2.x | Yes | +| < 0.2 | No | + +## Reporting a Vulnerability + +If you discover a security vulnerability in Compass, please report it +responsibly. **Do not open a public GitHub issue.** + +Email **security@runcaptain.com** with: + +1. A description of the vulnerability +2. Steps to reproduce +3. Potential impact +4. Suggested fix (if any) + +We will acknowledge your report within 48 hours and aim to release a fix +within 7 days for critical issues. You will be credited in the release notes +unless you prefer to remain anonymous. diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 0000000..7579c4c --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,18 @@ +services: + compass: + build: . + ports: + - "4001:4001" + volumes: + - compass-data:/app/data + environment: + PORT: 4001 + DATA_DIR: /app/data + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:4001/health"] + interval: 10s + timeout: 5s + retries: 3 + +volumes: + compass-data: