Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
24814e9
feat(fixtures): add scalar fixtures
tobyhede May 27, 2026
d99cb65
feat(codegen): add domain generator
tobyhede May 27, 2026
e400463
feat(int4): add int4 domains
tobyhede May 27, 2026
b75f0ac
feat(lint): add domain lints
tobyhede May 27, 2026
a674ab2
test(matrix): add scalar matrix
tobyhede May 27, 2026
3afa5dc
style(codegen): use proper-cased role phrases in file-header briefs
tobyhede May 27, 2026
93de071
feat(aggregates): per-domain MIN/MAX for the encrypted-domain family
tobyhede May 27, 2026
ab887e9
style: apply cargo fmt + markdown linter fix
tobyhede May 27, 2026
25e7d0a
chore(int4): address review feedback
tobyhede May 28, 2026
5507776
ci(test-eql): scope permissions and disable credential persistence
tobyhede May 28, 2026
8a4c77e
style(tests): tighten Rust idioms in scalar_domains and matrix
tobyhede May 28, 2026
cd57ac2
fix(aggregates): retain composite-type MIN/MAX alongside per-domain a…
tobyhede May 28, 2026
ceb4a2f
test(int4): sync fixture assertions with 17-value generator
tobyhede May 28, 2026
537f264
docs(int4): sync encrypted-domain reference docs with generator
tobyhede May 28, 2026
eafa24f
refactor(test-matrix): slim ordered_numeric_matrix! to suite/scalar/e…
tobyhede May 28, 2026
b82939d
test(test-matrix): make matrix coverage reviewable and prove arms hav…
tobyhede May 28, 2026
ea0b822
refactor(test-matrix): share one cartesian-product driver across cate…
tobyhede May 28, 2026
31bf95a
refactor(test-matrix): drop dead assert_plan_uses_index, tighten exports
tobyhede May 28, 2026
e021622
refactor(test-matrix): drop redundant named format args, minor idioms
tobyhede May 28, 2026
7dfe2b1
test(test-matrix): cover ORDER BY NULLS FIRST/LAST for ordered domains
tobyhede May 28, 2026
80f8cfe
style(test-matrix): flush-left SQL continuations in order_by_nulls arm
tobyhede May 28, 2026
a746f22
feat(numeric): add encrypted-domain scalar type for numeric/Decimal
tobyhede May 29, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 69 additions & 0 deletions .github/workflows/macro-expand-eql.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
name: "Macro expand EQL"

# Regenerates the int4 matrix `cargo expand` snapshot and fails if it has
# drifted from the committed copy. This is a body-level fidelity backstop for
# the `ordered_numeric_matrix!` / `scalar_domain_matrix!` macros — the
# name-inventory snapshot (test-eql.yml `matrix-coverage` job) catches
# add/remove of whole arms; this catches changes *inside* the generated bodies.
#
# Non-blocking by design: it is NOT a required PR check. `cargo expand` needs a
# nightly toolchain, so it is isolated off the PR path.
# - nightly schedule (the backstop that flags a forgotten local regen)
# - manual workflow_dispatch
#
# The toolchain is pinned (nightly-2026-05-01) in lockstep with the
# `test:matrix:expand` mise task so the snapshot only moves when the macro
# moves, not when nightly reformats its expansion. Bump both together.
on:
schedule:
# 03:00 UTC daily
- cron: "0 3 * * *"

workflow_dispatch:

env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
MISE_VERBOSE: "1"

defaults:
run:
shell: bash -l {0}

permissions:
contents: read

jobs:
macro-expand:
name: "Macro expand drift (nightly)"
runs-on: ubuntu-latest
timeout-minutes: 30

steps:
- uses: actions/checkout@v6
with:
persist-credentials: false

- uses: jdx/mise-action@v4
with:
version: 2026.4.0
install: true
cache: true

- uses: Swatinem/rust-cache@v2
with:
workspaces: tests/sqlx
shared-key: sqlx-tests

# Pinned nightly — keep the date in lockstep with the `cargo +nightly-...`
# invocation in the `test:matrix:expand` mise task. rustfmt formats the
# expansion deterministically.
- name: Install pinned nightly + cargo-expand
run: |
rustup toolchain install nightly-2026-05-01 --profile minimal --component rustfmt
cargo binstall -y cargo-expand

- name: Regenerate and verify the matrix expansion snapshot
run: |
mise run test:matrix:expand
git diff --exit-code -- tests/sqlx/snapshots/int4_expanded.rs \
|| { echo "Expansion snapshot stale — run 'mise run test:matrix:expand' (needs the pinned nightly) and commit."; exit 1; }
69 changes: 68 additions & 1 deletion .github/workflows/test-eql.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,13 +29,18 @@ defaults:
run:
shell: bash -l {0}

permissions:
contents: read

jobs:
schema:
name: "JSON Schema validation"
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v6
with:
persist-credentials: false

- uses: jdx/mise-action@v4
with:
Expand All @@ -52,10 +57,61 @@ jobs:
run: |
mise run test:schema

codegen:
name: "Encrypted-domain codegen"
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v6
with:
persist-credentials: false

- uses: jdx/mise-action@v4
with:
version: 2026.4.0
install: true
cache: true

- name: Run codegen generator + drift tests
run: |
mise run test:codegen

matrix-coverage:
name: "Matrix coverage inventory"
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v6
with:
persist-credentials: false

- uses: jdx/mise-action@v4
with:
version: 2026.4.0
install: true
cache: true

- uses: Swatinem/rust-cache@v2
with:
workspaces: tests/sqlx
shared-key: sqlx-tests

# Regenerate the matrix test-name inventory with the SAME pinned feature
# set the local task uses (`--no-default-features`, scale excluded), then
# fail if it differs from the committed snapshot. A coverage change shows
# up as added/removed names in the PR diff — e.g. emptying `ord_domains`
# drops ~140 names, impossible to miss in review. No Postgres needed:
# `--list` only enumerates, the suite uses runtime queries.
- name: Regenerate and verify the matrix test-name inventory
run: |
mise run test:matrix:inventory
git diff --exit-code -- tests/sqlx/snapshots/int4_matrix_tests.txt \
|| { echo "Coverage inventory stale — run 'mise run test:matrix:inventory' and commit."; exit 1; }

test:
name: "Test & Validate EQL (Postgres ${{ matrix.postgres-version }})"
runs-on: ubuntu-latest-m
needs: schema
needs: [schema, codegen]

strategy:
fail-fast: false
Expand All @@ -64,11 +120,20 @@ jobs:

env:
POSTGRES_VERSION: ${{ matrix.postgres-version }}
# CS_* are required for `mise run test:sqlx` to regenerate the
# cipherstash-client-encrypted fixtures before the suite runs.
# This repository does not accept fork PRs, so the secrets-on-
# `pull_request` constraint that breaks the fork CI flow does not
# apply here — leave the env block unconditional.
CS_CLIENT_ACCESS_KEY: ${{ secrets.CS_CLIENT_ACCESS_KEY }}
CS_WORKSPACE_CRN: ${{ secrets.CS_WORKSPACE_CRN }}
CS_CLIENT_ID: ${{ secrets.CS_CLIENT_ID }}
CS_CLIENT_KEY: ${{ secrets.CS_CLIENT_KEY }}

steps:
- uses: actions/checkout@v6
with:
persist-credentials: false

- uses: jdx/mise-action@v4
with:
Expand Down Expand Up @@ -104,6 +169,8 @@ jobs:

steps:
- uses: actions/checkout@v6
with:
persist-credentials: false

- uses: jdx/mise-action@v4
with:
Expand Down
9 changes: 9 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -222,6 +222,15 @@ tests/sqlx/migrations/001_install_eql.sql
# Generated SQLx fixtures (regenerated via `mise run fixture:generate`,
# never commit — stale fixtures hide bugs)
tests/sqlx/fixtures/eql_v2_int4.sql
tests/sqlx/fixtures/eql_v2_numeric.sql

# Generated encrypted-domain SQL — regenerated by `tasks/build.sh` from
# tasks/codegen/types/<T>.toml on every build (or `mise run codegen:domain
# <T>` to refresh manually). Hand-written *_extensions.sql stays committed.
src/encrypted_domain/*/*_types.sql
src/encrypted_domain/*/*_functions.sql
src/encrypted_domain/*/*_operators.sql
src/encrypted_domain/*/*_aggregates.sql

# Large generated test data files
tests/ste_vec_vast.sql
Expand Down
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,11 @@ Each entry that ships in a published release links to the PR that introduced it.

## [Unreleased]

### Added

- **`numeric` encrypted-domain scalar type.** The `eql_v2_numeric`, `eql_v2_numeric_eq`, `eql_v2_numeric_ord`, and `eql_v2_numeric_ord_ore` domains join `int4` as the second ordered scalar generated from the codegen materializer (`tasks/codegen/types/numeric.toml`), mapping PostgreSQL `numeric` (Rust `rust_decimal::Decimal`, EQL cast `decimal`). Equality (`=` / `<>`) routes through the `hm` HMAC term and ordering (`<` `<=` `>` `>=`, plus `MIN` / `MAX`) through the `ob` ORE block term — both index-backed via `eql_v2.eq_term` / `eql_v2.ord_term` with no decryption, exactly like `int4`. Why: callers storing encrypted decimal/monetary values can now use a type-safe domain column with the full searchable-comparison surface instead of the untyped `eql_v2_encrypted` composite. The ORE encoding is order-preserving across the entire `Decimal` range, including `Decimal::MIN` / `Decimal::MAX`.
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Add the PR link and avoid overstating ordering readiness.

This entry is missing the required PR link suffix, and it currently reads as fully supporting ordered comparisons even though PR #242 documents an outstanding ordering comparator gap (issue #241). Please append the PR link and tighten wording to reflect the current limitation until #241 lands.

As per coding guidelines, "When making a user-facing change, write the CHANGELOG.md entry by picking the right section (Added / Changed / Deprecated / Removed / Fixed / Security), leading with the user-visible fact, then a short 'Why' explanation, then a PR link in parentheses. Match existing entries' tone and density - a single dense paragraph per entry, not a bullet list".

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@CHANGELOG.md` at line 25, Update the CHANGELOG.md entry for the `numeric`
encrypted-domain scalar to include the PR link suffix (append "(`#242`)") and
soften the ordering claim to reflect the outstanding comparator gap referenced
by issue `#241`; revise phrases that assert full ordered-comparison readiness
(e.g., "The ORE encoding is order-preserving across the entire `Decimal`
range...") to state that ordering is provided via the ORE term but a comparator
implementation/edge-case support is still outstanding (see issue `#241`), and
ensure the entry follows the project style: single dense paragraph, leading with
the user-visible change, a short "Why" explanation, then the PR link.

- **Per-domain `MIN` / `MAX` aggregates for the encrypted-domain family.** `eql_v2.min(eql_v2_<T>_ord)` / `eql_v2.max(eql_v2_<T>_ord)` (and the `_ord_ore` twin) are generated for every ord-capable scalar variant, giving type-safe extrema on domain-typed columns — comparison routes through the variant's `<` / `>` operator (ORE block term, no decryption). Why: the new domain types previously had no equivalent of the composite-type aggregates. The existing `eql_v2.min(eql_v2_encrypted)` / `eql_v2.max(eql_v2_encrypted)` aggregates are **retained** and continue to work on `eql_v2_encrypted` columns; the per-domain aggregates are additive and coexist with them.

## [2.3.1] — 2026-05-21

### Fixed
Expand Down
23 changes: 22 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ This is the **Encrypt Query Language (EQL)** - a PostgreSQL extension for search
- `src/operators/` - SQL operators for encrypted data comparisons
- `src/config/` - Configuration management functions
- `src/blake3/`, `src/hmac_256/`, `src/bloom_filter/`, `src/ore_*` - Index implementations
- `src/encrypted_domain/` - Encrypted-domain type families (jsonb-backed PostgreSQL domains, one per operator/index capability)
- `tasks/` - mise task scripts
- `tests/sqlx/` - Rust/SQLx test framework (PostgreSQL 14-17 support)
- `release/` - Generated SQL installation files
Expand All @@ -72,6 +73,25 @@ This is the **Encrypt Query Language (EQL)** - a PostgreSQL extension for search
- **Operators**: Support comparisons between encrypted and plain JSONB data
- **CipherStash Proxy**: Required for encryption/decryption operations

### Encrypted-Domain Types

`src/encrypted_domain/` holds **encrypted-domain type families** — jsonb-backed PostgreSQL domains, one domain per operator/index capability (`eql_v2_<T>` storage-only, `eql_v2_<T>_eq`, `eql_v2_<T>_ord`). `eql_v2_int4` (PR #225) is the reference scalar implementation; future scalar types such as `int8`, `bool`, `date`, `float`, `numeric`, and `timestamp` follow this materializer pattern. `jsonb` needs a separate design and is out of scope for the scalar materializer.

Adding a scalar encrypted-domain type is generated from a minimal manifest at `tasks/codegen/types/<T>.toml`: the filename supplies `<T>`, and the `[domain]` table maps each generated domain name to the fixed index terms it carries. Example: `int4_eq = ["hm"]`, `int4_ord = ["ore"]`. Term capabilities are fixed in `tasks/codegen/terms.py`: `hm` provides equality, and `ore` provides equality plus ordering. `mise run build` regenerates the scalar SQL surface into `src/encrypted_domain/<T>/` from every manifest at the start of every build; that surface includes supported comparison wrappers plus blockers for native `jsonb` operators that would otherwise be reachable through domain fallback. Use `mise run codegen:domain <T>` to refresh a single type manually while iterating on its manifest. The generated `*_types.sql` / `*_functions.sql` / `*_operators.sql` files are gitignored and never committed — the TOML manifest plus `tasks/codegen/terms.py` are the source of truth. Generated files carry an `AUTO-GENERATED — DO NOT EDIT` header; change the manifest or term catalog and rebuild, never hand-edit. Hand-written SQL beyond the fixed surface goes in `src/encrypted_domain/<T>/<T>_extensions.sql` with no auto-generated header and explicit `-- REQUIRE:` edges — that file IS committed. `text` and `jsonb` are out of scope for this scalar materializer.

**Adding a new encrypted-domain type: follow `docs/reference/encrypted-domain-implementation-spec.md`.** The mechanics are fixed for ordered scalar domains; the manifest only declares domain names and terms. New term behavior belongs in `tasks/codegen/terms.py` with tests, not in free-form TOML fields.

Regeneration is deterministic: identical manifest + term catalog produce byte-identical SQL. If `mise run build` produces unexpected output, the change is in the manifest, `tasks/codegen/terms.py`, or `tasks/codegen/templates.py` — not in random run-to-run variation.

Footguns the spec exists to prevent:

- **Blockers must never be `STRICT`.** A `STRICT` blocker lets PostgreSQL skip the body and return `NULL` on a `NULL` argument, silently bypassing the "operator not supported" exception.
- **No domain-over-domain** (`CREATE DOMAIN a AS b`). Operators resolve against the ultimate base type (`jsonb`), so a derived domain does not inherit the base domain's operator surface — blockers stop engaging.
- **No operator class on a domain.** Index through a functional index on the extractor (`eq_term` / `ord_term`), whose return type already carries a default opclass.
- **Inlinable functions** (extractors, comparison wrappers) need `LANGUAGE sql`, a single-statement `SELECT`, `IMMUTABLE`, and **no `SET` clause** — a pinned `search_path` disables inlining. No per-type allowlist edit: the `pin_search_path.sql` structural rule recognises encrypted-domain functions intrinsically and `tasks/test/splinter.sh` covers the converged extractor/wrapper names.
- **Blockers must be `LANGUAGE plpgsql`, not `LANGUAGE sql`.** The inverse of the rule above. A blocker exists to always raise, but a `LANGUAGE sql` body is inlinable and the planner can elide the call when the result is provably unused (dead `CASE` branch, folded predicate). `LANGUAGE plpgsql` is opaque to the planner, so the call — and its `RAISE` — survives. The generator in `tasks/codegen/templates.py` enforces this; don't "simplify" the rendered blockers to `LANGUAGE sql` even though the body is a single expression.
- **Build with `mise run clean && mise run build`** — a bare build can leave stale `release/*.sql`.

### Testing Infrastructure
- Tests are written in Rust using SQLx, located in `tests/sqlx/`
- Tests run against PostgreSQL 14, 15, 16, 17 using Docker containers
Expand Down Expand Up @@ -199,6 +219,7 @@ Prefer `LANGUAGE SQL` over `LANGUAGE plpgsql` unless you need procedural feature
- Exception handling (`BEGIN...EXCEPTION...END`)
- Complex control flow (loops, early returns)
- Dynamic SQL (`EXECUTE`)
- Functions that must remain opaque to the planner — typically blockers whose only job is to `RAISE`. `LANGUAGE sql` would be inlined and may be elided when the result is provably unused; `LANGUAGE plpgsql` is never inlined, so the body always runs. See the encrypted-domain footgun list above and the blocker renderers in `tasks/codegen/templates.py`.

## Release & changelog discipline

Expand All @@ -222,7 +243,7 @@ What does *not* need an entry:

Pick the right section (`Added` / `Changed` / `Deprecated` / `Removed` / `Fixed` / `Security`). Lead with the user-visible fact, then a short "Why." explanation, then a PR link in parentheses. Match the tone and density of existing entries — a single dense paragraph per entry, not a bullet list.

Example shape (real entry from `2.3.0`):
Example entry (real entry from `2.3.0`):

> **`=`, `<>`, `~~` (`LIKE`), `~~*` (`ILIKE`) on `eql_v2_encrypted` are now inlinable SQL functions.** The planner can structurally match these operators against the documented functional indexes (`eql_v2.hmac_256(col)` for equality, `eql_v2.bloom_filter(col)` for `LIKE`/`ILIKE`), so bare-form queries (`WHERE col = $1`) engage the index without per-query rewriting. Previously these operators wrapped multi-branch PL/pgSQL bodies that the planner could not inline, forcing seq scans on Supabase / managed Postgres installations that lack operator-class indexes. ([#193](...), [#196](...))

Expand Down
7 changes: 0 additions & 7 deletions docs/development/documentation-inventory.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,13 +77,6 @@ Generated: Mon 27 Oct 2025 11:39:50 AEDT
## src/crypto.sql


## src/encrypted/aggregates.sql

- CREATE FUNCTION eql_v2.min(a eql_v2_encrypted, b eql_v2_encrypted)
- CREATE AGGREGATE eql_v2.min(eql_v2_encrypted)
- CREATE FUNCTION eql_v2.max(a eql_v2_encrypted, b eql_v2_encrypted)
- CREATE AGGREGATE eql_v2.max(eql_v2_encrypted)

## src/encrypted/casts.sql

- CREATE FUNCTION eql_v2.to_encrypted(data jsonb)
Expand Down
Loading