Open Sourcing Compass

.github/CODEOWNERS

@@ -0,0 +1,9 @@
+# Default owners for everything
+* @runcaptain/eng
+
+# Vector backends
+crates/compass-index-api/ @runcaptain/eng
+crates/compass-vector-gpu/ @runcaptain/eng
+
+# CI and release
+.github/ @runcaptain/eng

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -0,0 +1,49 @@
+name: Bug Report
+description: Report a bug in Compass
+labels: ["bug"]
+body:
+  - type: textarea
+    id: description
+    attributes:
+      label: Description
+      description: What happened? What did you expect to happen?
+    validations:
+      required: true
+  - type: textarea
+    id: reproduce
+    attributes:
+      label: Steps to reproduce
+      description: Minimal steps or curl commands to reproduce the issue.
+      placeholder: |
+        1. Create collection: curl -X POST localhost:4001/collections ...
+        2. Ingest data: curl -X POST localhost:4001/collections/test/ingest ...
+        3. Search: curl -X POST localhost:4001/collections/test/search ...
+    validations:
+      required: true
+  - type: input
+    id: version
+    attributes:
+      label: Compass version
+      description: Output of the /health endpoint or git commit SHA.
+      placeholder: "v0.2.0 / abc1234"
+    validations:
+      required: true
+  - type: dropdown
+    id: os
+    attributes:
+      label: Operating system
+      options:
+        - Linux (x86_64)
+        - Linux (aarch64)
+        - macOS (Apple Silicon)
+        - macOS (Intel)
+        - Docker
+        - Other
+    validations:
+      required: true
+  - type: textarea
+    id: logs
+    attributes:
+      label: Relevant logs
+      description: Paste any error messages or stack traces.
+      render: shell

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,5 @@
+blank_issues_enabled: true
+contact_links:
+  - name: Security vulnerability
+    url: mailto:security@runcaptain.com
+    about: Report security issues privately via email — do not open a public issue.

.github/ISSUE_TEMPLATE/feature_request.yml

@@ -0,0 +1,23 @@
+name: Feature Request
+description: Suggest a new feature or improvement
+labels: ["enhancement"]
+body:
+  - type: textarea
+    id: problem
+    attributes:
+      label: Problem
+      description: What problem does this feature solve?
+    validations:
+      required: true
+  - type: textarea
+    id: solution
+    attributes:
+      label: Proposed solution
+      description: How should this work? Include API examples if relevant.
+    validations:
+      required: true
+  - type: textarea
+    id: alternatives
+    attributes:
+      label: Alternatives considered
+      description: What other approaches did you consider?

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,16 @@
+## Summary
+
+<!-- What does this PR do? 1-3 bullet points. -->
+
+## Test plan
+
+<!-- How did you verify this works? -->
+
+- [ ] `cargo test` passes
+- [ ] `cargo clippy` clean
+- [ ] `cargo fmt` clean
+- [ ] Tested with Docker build (if applicable)
+
+## Changelog
+
+<!-- Which CHANGELOG.md section does this belong in? (Added / Changed / Fixed / Removed) -->

.github/workflows/release.yml

@@ -0,0 +1,87 @@
+name: Release
+
+on:
+  push:
+    tags:
+      - "v*"
+
+env:
+  CARGO_TERM_COLOR: always
+
+jobs:
+  build:
+    name: Build (${{ matrix.target }})
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        include:
+          - target: x86_64-unknown-linux-gnu
+            os: ubuntu-24.04
+            artifact: compass-linux-amd64
+          - target: aarch64-unknown-linux-gnu
+            os: ubuntu-24.04
+            artifact: compass-linux-arm64
+          - target: aarch64-apple-darwin
+            os: macos-14
+            artifact: compass-macos-arm64
+    steps:
+      - uses: actions/checkout@v4
+      - uses: dtolnay/rust-toolchain@stable
+        with:
+          targets: ${{ matrix.target }}
+      - uses: Swatinem/rust-cache@v2
+      - if: matrix.target == 'aarch64-unknown-linux-gnu'
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y gcc-aarch64-linux-gnu cmake pkg-config libssl-dev
+          echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc" >> $GITHUB_ENV
+      - if: matrix.os == 'ubuntu-24.04' && matrix.target == 'x86_64-unknown-linux-gnu'
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y cmake pkg-config libssl-dev
+      - run: cargo build --release --target ${{ matrix.target }} -p compass
+      - run: |
+          mkdir -p dist
+          cp target/${{ matrix.target }}/release/compass dist/${{ matrix.artifact }}
+      - uses: actions/upload-artifact@v4
+        with:
+          name: ${{ matrix.artifact }}
+          path: dist/${{ matrix.artifact }}
+
+  docker:
+    name: Docker image
+    runs-on: ubuntu-24.04
+    permissions:
+      contents: read
+      packages: write
+    steps:
+      - uses: actions/checkout@v4
+      - uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+      - uses: docker/build-push-action@v6
+        with:
+          context: .
+          push: true
+          tags: |
+            ghcr.io/${{ github.repository }}:${{ github.ref_name }}
+            ghcr.io/${{ github.repository }}:latest
+
+  release:
+    name: GitHub Release
+    needs: [build, docker]
+    runs-on: ubuntu-24.04
+    permissions:
+      contents: write
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/download-artifact@v4
+        with:
+          path: dist
+          merge-multiple: true
+      - uses: softprops/action-gh-release@v2
+        with:
+          generate_release_notes: true
+          files: dist/*

CHANGELOG.md

@@ -8,6 +8,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 
 ### Added
 
+- **Memory-mapped vector storage.** New `MmapVectors` module (`search/mmap_vectors.rs`) replaces `Vec<Vec<f32>>` with a flat mmap-backed file. Zero-copy reads via `bytemuck::cast_slice`, append-only writes, near-zero RSS for vector data regardless of dataset size. File format: `[u32 dims][u32 count][f32...]`.
+- **Disk-backed chunk metadata.** New `ChunkStore` module (`search/chunk_store.rs`) backed by redb (pure Rust embedded DB). Replaces in-memory `HashMap<u64, DocumentChunk>` with persistent, ACID-compliant storage. Point lookups, batch inserts, full scans for rebuild.
+- **Incremental HNSW indexing.** Ingest path now loads the existing USearch index via `.load()`, appends new vectors with `.add()`, and saves — instead of cloning all vectors and rebuilding from scratch. Falls back to full rebuild when the `Arc` cannot be unwrapped.
+- **New dependencies:** `memmap2` (mmap), `bytemuck` (zero-copy cast), `redb` (embedded DB).
 - **Cargo workspace layout.** Source moved from a single crate to a workspace under `crates/`. Splits: `compass` (umbrella + binary), `compass-index-api` (stable trait surface), `compass-vector-gpu` (optional GPU backend).
 - **Vector index trait abstraction.** New `compass_index_api::VectorIndex` trait abstracts over CPU and GPU backends. The existing USearch path is wrapped in `UsearchHnswIndex`; new backends bind to the trait without touching call sites.
 - **Optional GPU backend (`--features gpu`).** New `compass-vector-gpu` crate integrates NVIDIA cuVS for CAGRA→HNSW build acceleration. CPU-side search after conversion. Linux + CUDA 12+ only. See `ARCHITECTURE.md` for build prerequisites.
@@ -25,6 +29,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 
 - **Cargo manifest** is now a workspace root with shared `[workspace.dependencies]` and `[workspace.package]` metadata. Per-crate manifests inherit version, edition, and rust-version.
 - **Compiler profile** adds `lto = "thin"` and `codegen-units = 1` to release builds for better optimization. Adds debug symbols to bench builds.
+- **MSRV bumped to 1.88** from 1.82. Required by `time@0.3.47` (edition 2024) and `icu_collections@2.2.0`.
+- **Dockerfile** upgraded to `rust:latest` + `debian:trixie-slim` for glibc compatibility with newer Rust toolchains.
 
 ### Deprecated
 

CODE_OF_CONDUCT.md

@@ -0,0 +1,38 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, caste, color, religion, or sexual
+identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior:
+
+* The use of sexualized language or imagery, and sexual attention or advances of any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a professional setting
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the project team at **conduct@runcaptain.com**. All complaints will
+be reviewed and investigated promptly and fairly.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org/), version 2.1.

README.md

@@ -1,12 +1,19 @@
 # Compass
 
+[![CI](https://github.com/runcaptain/compass/actions/workflows/ci.yml/badge.svg)](https://github.com/runcaptain/compass/actions/workflows/ci.yml)
+[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)
+[![Rust](https://img.shields.io/badge/rust-1.88%2B-orange.svg)](https://www.rust-lang.org)
+
 Embedded vector + full-text search engine for [Captain](https://runcaptain.com). Single binary, zero external dependencies. Designed for high-throughput retrieval. Built for on-prem enterprise deployments where customer data cannot leave their VPC.
 
 ## What it does
 
 - **Full-text search** via Tantivy (BM25) with precomputed bitset faceting (microsecond facets)
 - **Vector search** via USearch HNSW (mmap-backed, disk-persistent)
 - **Hybrid search** via Reciprocal Rank Fusion (RRF, k=60)
+- **Memory-mapped vector storage**. Raw vectors live on disk, not in RAM. Zero-copy reads via mmap.
+- **Disk-backed chunk metadata** via redb (pure Rust embedded DB). Handles millions of documents without loading them all into memory.
+- **Incremental HNSW indexing**. Adding vectors appends to the index — no full rebuild required.
 - **Named vector spaces** ... run multiple embedding models on the same collection
 - **One-click model upgrades** with background re-embedding and atomic swap
 - **Parent-child documents** with relationship-aware scoring (TAMS video search compatible)
@@ -375,6 +382,29 @@ docker run -p 8080:80 --gpus all ghcr.io/huggingface/text-embeddings-inference \
 
 MTEB and MMEB are different benchmarks on different scales. MTEB scores are 0-100 (text tasks). MMEB scores are 0-1 (cross-modal retrieval). They cannot be compared directly.
 
+## Storage architecture
+
+Compass keeps vector data and chunk metadata on disk, not in RAM.
+
+```
+data/{collection}/
+├── collection.json                  # Collection metadata (name, dims, spaces)
+├── relationships.bin                # Parent-child + sibling graph
+├── tantivy/                         # BM25 inverted index (disk-backed)
+└── vectors/
+    ├── {space}.index                # USearch HNSW graph (mmap on read)
+    ├── {space}.bin                  # Raw f32 vectors (mmap via MmapVectors)
+    └── {space}.keymap               # HNSW key → chunk ID mapping
+```
+
+**Vectors**: Stored in a flat `[u32 dims][u32 count][f32...]` file, memory-mapped at query time. Adding vectors appends to the file and remaps — no full rewrite. At 1M vectors × 768 dims this is ~3GB on disk, near-zero RSS.
+
+**HNSW index**: Built incrementally via USearch `.add()` + `.save()`. Loaded via `.load()` for mutation or `.view()` for read-only mmap. The graph structure is separate from the raw vectors.
+
+**Chunk metadata**: Persisted via redb (pure Rust, ACID, MVCC). Point lookups by chunk ID during search result assembly. Batch inserts during ingestion.
+
+**Ingestion path**: New vectors are appended to the mmap file, inserted into the HNSW graph incrementally, and chunk metadata is written to redb — all without cloning existing data.
+
 ## Throughput and scaling
 
 **Query throughput.** USearch HNSW serves around 15k QPS per instance on a 16-core box at p99 < 50ms for top-10 retrieval. For very high QPS workloads, shard collections across multiple Compass instances behind a load balancer.
@@ -413,6 +443,14 @@ PUT    /collections/:name/default-vector-space         Switch default space
 GET    /health                                         Health check
 ```
 
+## Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, PR guidelines, and commit conventions.
+
+## Security
+
+To report a vulnerability, email **security@runcaptain.com**. See [SECURITY.md](SECURITY.md) for details.
+
 ## License
 
 Apache 2.0. See [LICENSE](LICENSE).

SECURITY.md

@@ -0,0 +1,24 @@
+# Security Policy
+
+## Supported Versions
+
+| Version | Supported          |
+| ------- | ------------------ |
+| 0.2.x   | Yes                |
+| < 0.2   | No                 |
+
+## Reporting a Vulnerability
+
+If you discover a security vulnerability in Compass, please report it
+responsibly. **Do not open a public GitHub issue.**
+
+Email **security@runcaptain.com** with:
+
+1. A description of the vulnerability
+2. Steps to reproduce
+3. Potential impact
+4. Suggested fix (if any)
+
+We will acknowledge your report within 48 hours and aim to release a fix
+within 7 days for critical issues. You will be credited in the release notes
+unless you prefer to remain anonymous.

docker-compose.yml

@@ -0,0 +1,18 @@
+services:
+  compass:
+    build: .
+    ports:
+      - "4001:4001"
+    volumes:
+      - compass-data:/app/data
+    environment:
+      PORT: 4001
+      DATA_DIR: /app/data
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:4001/health"]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+
+volumes:
+  compass-data: