Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/test-eql.yml
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,7 @@ jobs:
run: |
mise run test:matrix:inventory
git diff --exit-code -- tests/sqlx/snapshots/int4_matrix_tests.txt \
tests/sqlx/snapshots/int2_matrix_tests.txt \
|| { echo "Coverage inventory stale — run 'mise run test:matrix:inventory' and commit."; exit 1; }

test:
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ Each entry that ships in a published release links to the PR that introduced it.
### Added

- **`eql_v3` encrypted-domain schema, with the `int4` family as its first member.** Encrypted-domain type families now live in a new, additional `eql_v3` schema (the existing `eql_v2` schema is unchanged — it keeps the core types/operators and stays the documented public API). Four jsonb-backed domains for encrypted `int4` columns: `eql_v3.int4` (storage-only), `eql_v3.int4_eq` (`=` / `<>` via HMAC), and `eql_v3.int4_ord` / `eql_v3.int4_ord_ore` (also `<` `<=` `>` `>=` via ORE block terms). Supported comparisons resolve to inlinable wrappers; the native `jsonb` operator surface reachable through domain fallback is blocked (raises rather than silently mis-resolving). Each domain's `CHECK` requires the EQL envelope (`v`, `i`), the ciphertext (`c`), and the variant's index term(s), and pins the payload version (`VALUE->>'v' = '2'`, matching `eql_v2._encrypted_check_v`) — so a missing key or wrong-version payload is rejected on insert or cast rather than surfacing later at query time. Index via a functional index on the `eql_v3.eq_term` / `eql_v3.ord_term` extractors, not an operator class on the domain. The extractors still return the core `eql_v2.hmac_256` / `eql_v2.ore_block_u64_8_256` index-term types, which remain in `eql_v2` and are referenced cross-schema. Why: a type-safe, per-capability encrypted integer column instead of the untyped `eql_v2_encrypted`, namespaced under its own schema. This is the reference scalar implementation for the generated domain family. ([#239](https://github.com/cipherstash/encrypt-query-language/pull/239), supersedes [#225](https://github.com/cipherstash/encrypt-query-language/pull/225))
- **`eql_v3.int2` encrypted-domain type family.** Four jsonb-backed domains for encrypted `int2` columns — `eql_v3.int2` (storage-only), `eql_v3.int2_eq` (`=` / `<>` via HMAC), and `eql_v3.int2_ord` / `eql_v3.int2_ord_ore` (also `<` `<=` `>` `>=` via ORE block terms, with `MIN` / `MAX` aggregates) — generated from `tasks/codegen/types/int2.toml` by the same materializer as the `eql_v3.int4` reference. Index via a functional index on the `eql_v3.eq_term` / `eql_v3.ord_term` extractors, not an operator class on the domain. Why: a type-safe, per-capability encrypted `smallint` column, proving the scalar generator generalizes beyond the `int4` reference. ([#243](https://github.com/cipherstash/encrypt-query-language/pull/243))
- **Per-domain `MIN` / `MAX` aggregates for the encrypted-domain family.** `eql_v3.min(eql_v3.<T>_ord)` / `eql_v3.max(eql_v3.<T>_ord)` (and the `_ord_ore` twin) are generated for every ord-capable scalar variant, giving type-safe extrema on domain-typed columns — comparison routes through the variant's `<` / `>` operator (ORE block term, no decryption). The aggregates are declared `PARALLEL = SAFE` with a combine function (the state function itself — min/max are associative), so PostgreSQL can use partial/parallel aggregation on large `GROUP BY` workloads. Why: the new domain types previously had no equivalent of the composite-type aggregates. The existing `eql_v2.min(eql_v2_encrypted)` / `eql_v2.max(eql_v2_encrypted)` aggregates are **retained** and continue to work on `eql_v2_encrypted` columns; the per-domain aggregates are additive and coexist with them. ([#239](https://github.com/cipherstash/encrypt-query-language/pull/239))

## [2.3.1] — 2026-05-21
Expand Down
1 change: 1 addition & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ This project uses `mise` for task management. Common commands:
- Run SQLx tests directly: `mise run test:sqlx`
- Run SQLx tests in watch mode: `mise run test:sqlx:watch`
- Tests are located in `tests/sqlx/` using Rust and SQLx framework
- Regenerate the scalar matrix coverage snapshots: `mise run test:matrix:inventory` (no database required). These committed `tests/sqlx/snapshots/<T>_matrix_tests.txt` baselines pin the set of `scalars::<T>::*` test names so a silently dropped/renamed/`#[cfg]`-gated test fails CI's `matrix-coverage` job. When you add or remove matrix tests (or add a scalar type), regenerate and commit the affected snapshot in the same change. See `tests/sqlx/snapshots/README.md`.

### Build System
- Dependencies are resolved using `-- REQUIRE:` comments in SQL files
Expand Down
23 changes: 17 additions & 6 deletions docs/reference/encrypted-domain-generator.md
Original file line number Diff line number Diff line change
Expand Up @@ -377,12 +377,23 @@ The end-to-end shape from a generator perspective:
4. **Build picks it up automatically** — `tasks/build.sh` regenerates
before computing the `tsort` graph, so the new files appear in the
dependency walk via the `-- REQUIRE:` edges the generator emits.
5. **Baseline & test.** Create a hand-reviewed byte-parity baseline under
`tests/codegen/reference/<token>/` (each file marked `-- REFERENCE:` /
`// REFERENCE:`) so `test_against_reference.py` guards the new type — it
only covers types that have a baseline directory. Then run
`mise run test:codegen`, the relevant SQLx suites, and the PostgreSQL
matrix.
5. **Test.** Do **not** add a `tests/codegen/reference/<token>/` baseline.
`int4` is the sole golden master for the type-generic generator: the SQL
templates are pure token substitution and the only type-specific rendering
is `<token>_values.rs`, so a per-type baseline can only fail where `int4`'s
already would. Drift protection for the new type comes from the `int4`
reference (shared templates + `terms.py`), the committed `<token>_values.rs`
const guarded by the codegen staleness check, the `<token>` cases in
`test_scalars.py`, and the `ordered_numeric_matrix!` SQLx suite (behaviour,
not bytes). Run `mise run test:codegen`, the relevant SQLx suites, and the
PostgreSQL matrix.
6. **Snapshot the matrix inventory.** Run `mise run test:matrix:inventory`
and commit the new `tests/sqlx/snapshots/<token>_matrix_tests.txt` — the
sorted list of the type's `scalars::<token>::*` test names. CI's
`matrix-coverage` job `git diff --exit-code`s it (like `<token>_values.rs`)
to catch a silently dropped or renamed matrix test. The snapshot is a
committed test baseline, not gitignored generated SQL. See
`tests/sqlx/snapshots/README.md`.

Adding a new **term** is a bigger move — edit `terms.py`, add tests,
audit `splinter.sh` for a name collision, and update the reference
Expand Down
45 changes: 36 additions & 9 deletions docs/reference/encrypted-domain-implementation-spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,13 +83,21 @@ future migration.
- [ ] Put optional hand-written SQL in
`src/encrypted_domain/<T>/<T>_extensions.sql` with explicit
`-- REQUIRE:` edges. This file IS committed.
- [ ] Create a hand-reviewed byte-parity baseline under
`tests/codegen/reference/<T>/` — one file per generated SQL output plus
`<T>_values.rs`, each headed with the `-- REFERENCE:` / `// REFERENCE:`
marker. `tasks/codegen/test_against_reference.py` only guards types that
have a baseline directory, so without it the new type gets no
drift protection. The committed-fixture parity assertion is currently
`int4`-only; extend it to cover `<T>`.
- [ ] Do **not** add a `tests/codegen/reference/<T>/` baseline. `int4` is the
single golden master for the type-generic generator: the SQL templates are
pure token substitution and the only type-specific rendering is
`<T>_values.rs`, so a per-type baseline can only fail when `int4`'s already
would. Drift protection for the new type comes from the `int4` reference
(shared templates + `terms.py`), the committed `<T>_values.rs` const guarded
by the CI staleness check (`mise run codegen:domain <T>` + `git diff
--exit-code`) and the `<T>` cases in `tasks/codegen/test_scalars.py`, and
the `ordered_numeric_matrix!` SQLx suite (behaviour, not bytes).
- [ ] Run `mise run test:matrix:inventory` and commit the regenerated
`tests/sqlx/snapshots/<T>_matrix_tests.txt` — the sorted inventory of every
`scalars::<T>::*` test name in the `encrypted_domain` binary. CI diffs it
(same as `<T>_values.rs`); a stale snapshot fails the `matrix-coverage`
job with "Coverage inventory stale". This baseline is what catches a
silently dropped, renamed, or `#[cfg]`-gated matrix test. See §8.
- [ ] Run `mise run test:codegen`, the relevant SQLx suites, and the
PostgreSQL matrix before merging.

Expand Down Expand Up @@ -252,8 +260,9 @@ Cover each generated domain with SQLx tests appropriate to its terms:
- domain `CHECK` rejects non-object and under-populated payloads;
- real typed columns are tested, not only cast literals;
- generated ordered-domain twins remain byte-identical modulo type name
(verified by `tasks/codegen/test_against_reference.py` against the
hand-reviewed baseline in `tests/codegen/reference/<T>/`).
(the shared generator is anchored by the `int4` golden master in
`tests/codegen/reference/int4/` via `tasks/codegen/test_against_reference.py`;
new types add no baseline of their own — see §2).

For ordered numeric scalars this coverage is generated by the
`ordered_numeric_matrix!` convention wrapper in `tests/sqlx/src/matrix.rs`:
Expand All @@ -272,6 +281,24 @@ For ordered `int4`, keep the assertion that distinct plaintext values
produce distinct ORE blocks. Do not add assertions for term behavior that
the catalog does not promise.

### Matrix coverage inventory snapshot

The *set of test names* the matrix emits is itself guarded. `mise run
test:matrix:inventory` lists every test in the `encrypted_domain` binary
under a pinned feature set (`--no-default-features`, which deliberately
excludes the `scale` arm — see the task comment in `mise.toml`), greps it to
each `scalars::<T>::*` matrix, `LC_ALL=C sort`s for byte-stable ordering, and
writes one committed snapshot per scalar at
`tests/sqlx/snapshots/<T>_matrix_tests.txt`. The CI `matrix-coverage` job
regenerates with the same feature set and `git diff --exit-code`s every
snapshot; a divergence fails with "Coverage inventory stale". This is the
guard that catches a silently dropped, renamed, or `#[cfg]`-gated matrix
test — a behaviour the SQLx assertions above cannot see, because a deleted
test simply stops running. When you add a scalar you add a new snapshot;
when you add or remove matrix tests you regenerate and commit the affected
snapshot in the same change. The files are a committed test baseline, **not**
gitignored generated SQL. See `tests/sqlx/snapshots/README.md`.

## 9. Fixtures

Fixture generation should use real encrypted payloads produced through
Expand Down
27 changes: 18 additions & 9 deletions mise.toml
Original file line number Diff line number Diff line change
Expand Up @@ -51,12 +51,17 @@ cd tests/sqlx
sqlx migrate run

# Regenerate fixtures every run — they are not committed (see .gitignore).
# Generator encrypts via cipherstash-client directly; CS_* credentials must
# be present in the shell environment (CS_CLIENT_ACCESS_KEY +
# CS_WORKSPACE_CRN, or the legacy CS_CLIENT_ID/CS_CLIENT_KEY pair).
# fixture:generate:all enumerates every scalar manifest in
# tasks/codegen/types/ that declares a [fixture] table, so new scalar types
# are picked up automatically without editing this task.
# Generator encrypts via cipherstash-client directly, which needs BOTH a
# ZeroKMS auth credential (CS_CLIENT_ACCESS_KEY + CS_WORKSPACE_CRN, via
# AutoStrategy) AND a client key (CS_CLIENT_ID + CS_CLIENT_KEY, via
# EnvKeyProvider) in the shell environment. Auth and key material are
# separate roles — the two pairs are not alternatives.
echo "Regenerating SQLx fixtures..."
cd "{{config_root}}"
mise run fixture:generate eql_v2_int4
mise run fixture:generate:all

echo "Running Rust tests..."
cd tests/sqlx
Expand Down Expand Up @@ -94,7 +99,7 @@ mise exec python -- python -m pytest tasks/codegen -q
"""

[tasks."test:matrix:inventory"]
description = "Regenerate the int4 matrix test-name inventory snapshot (no database required)"
description = "Regenerate the int4/int2 matrix test-name inventory snapshots (no database required)"
dir = "{{config_root}}/tests/sqlx"
run = """
# Pin an explicit feature set so the inventory is deterministic regardless of
Expand All @@ -103,14 +108,18 @@ run = """
# of this default-feature inventory, covered instead by the scale gate + the
# family::mutations negative controls. `--list` enumerates the whole
# encrypted_domain binary (family::support, family::inlinability,
# family::mutations, scalars::int4); `grep '^scalars::int4'` scopes the
# snapshot to the matrix only, so landing other family tests never dirties it.
# `LC_ALL=C sort` makes ordering byte-stable across locales (a bare `sort` is
# locale-dependent and yields spurious CI diffs).
# family::mutations, scalars::int4, scalars::int2); the per-scalar `grep`
# scopes each snapshot to that matrix only, so landing other family tests
# never dirties it. `LC_ALL=C sort` makes ordering byte-stable across locales
# (a bare `sort` is locale-dependent and yields spurious CI diffs).
set -euo pipefail
mkdir -p snapshots
cargo test --no-default-features --test encrypted_domain -- --list |
sed -n 's/: test$//p' |
grep '^scalars::int4' |
LC_ALL=C sort > snapshots/int4_matrix_tests.txt
cargo test --no-default-features --test encrypted_domain -- --list |
sed -n 's/: test$//p' |
grep '^scalars::int2' |
LC_ALL=C sort > snapshots/int2_matrix_tests.txt
"""
9 changes: 9 additions & 0 deletions tasks/codegen/scalars.py
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,15 @@ def render_literal(self, value: str) -> str:
min_value=-2147483648,
max_value=2147483647,
),
"int2": ScalarKind(
token="int2",
rust_type="i16",
min_symbol="i16::MIN",
max_symbol="i16::MAX",
zero_symbol="0",
min_value=-32768,
max_value=32767,
),
}


Expand Down
18 changes: 18 additions & 0 deletions tasks/codegen/test_scalars.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,3 +62,21 @@ def test_require_scalar_unknown_raises():

def test_int4_registered_in_catalog():
assert "int4" in SCALAR_KINDS


def test_int2_kind_resolves_and_renders():
kind = require_scalar("int2")
assert kind.rust_type == "i16"
assert kind.numeric_value("MIN") == -32768
assert kind.numeric_value("MAX") == 32767
assert kind.numeric_value("ZERO") == 0
assert kind.render_literal("MIN") == "i16::MIN"
assert kind.render_literal("MAX") == "i16::MAX"
assert kind.render_literal("ZERO") == "0"
assert kind.render_literal("30000") == "30000"


def test_int2_kind_rejects_out_of_range():
kind = require_scalar("int2")
with pytest.raises(ScalarError, match="out of range"):
kind.numeric_value("40000")
19 changes: 19 additions & 0 deletions tasks/codegen/types/int2.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Encrypted-domain scalar manifest for int2.
# The filename supplies the type token. Each domain lists the index terms
# it carries; term capabilities are fixed in tasks/codegen/terms.py.

[domain]
int2 = []
int2_eq = ["hm"]
int2_ord_ore = ["ore"]
Comment thread
tobyhede marked this conversation as resolved.
int2_ord = ["ore"]

# Single source of truth for the int2 fixture plaintext list. Drives the
# generated tests/sqlx/src/fixtures/int2_values.rs const, shared by the fixture
# generator and the matrix oracle. Sentinels MIN/MAX/ZERO map to i16 named
# consts; the set MUST include MIN, MAX, and zero (matrix comparison pivots).
[fixture]
values = [
"MIN", "-30000", "-100", "-1", "ZERO", "1", "2", "5", "10", "17", "25",
"42", "50", "100", "250", "1000", "9999", "30000", "MAX",
]
29 changes: 29 additions & 0 deletions tasks/fixtures.toml
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,32 @@ cargo test --features fixture-gen --lib \
"fixtures::${fixture}::generate" \
-- --ignored --exact --nocapture
"""

["fixture:generate:all"]
description = "Regenerate every scalar SQLx fixture declared by a type manifest"
# Enumerates tasks/codegen/types/*.toml — the SAME manifests that
# `codegen:domain:all` drives — and regenerates the SQLx fixture for each type
# whose manifest declares a [fixture] table. This keeps the test fixtures in
# lockstep with the declared scalar types: adding a new scalar type (a new
# <T>.toml with a [fixture] table) is picked up automatically, so the test
# task never has to hand-list each fixture. Same prerequisites as
# `fixture:generate` (Postgres up + CS_* credentials).
dir = "{{config_root}}"
run = """
generated=0
for manifest in tasks/codegen/types/*.toml; do
# Guard the no-match case (glob stays literal under POSIX sh).
[ -e "$manifest" ] || continue
# Only types that declare a [fixture] table have a SQLx fixture generator.
grep -qE '^\\[fixture\\]' "$manifest" || continue
token=$(basename "$manifest" .toml)
echo "Generating fixture eql_v2_${token}..."
mise run fixture:generate "eql_v2_${token}"
generated=$((generated + 1))
done
if [ "$generated" -eq 0 ]; then
echo "No scalar manifests with a [fixture] table found in tasks/codegen/types/" >&2
exit 1
fi
echo "Regenerated ${generated} scalar fixture(s)."
"""
12 changes: 10 additions & 2 deletions tests/codegen/reference/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,13 @@
# Codegen reference

The SQL files under `<T>/` are the original, hand-written reference implementation for each encrypted-domain scalar type.
The SQL files under `int4/` are the original, hand-written reference implementation for the encrypted-domain scalar generator. `int4` is the **single golden master**: the generator in `tasks/codegen/` is type-generic — its SQL templates are pure token substitution, and the only type-specific rendering is the `<T>_values.rs` const — so one anchored type detects all template/term drift for every current and future scalar.

They are the parity baseline for the generator in `tasks/codegen/`. `tasks/codegen/test_against_reference.py` renders the generator's output and asserts it matches these files byte-for-byte. If the generator diverges, either it regressed (fix `tasks/codegen/`) or the reference is being updated deliberately (commit the new reference in the same PR).
`tasks/codegen/test_against_reference.py` renders the generator's output for `int4` and asserts it matches these files byte-for-byte. If the generator diverges, either it regressed (fix `tasks/codegen/`) or the reference is being updated deliberately (commit the new `int4` reference in the same PR).

## New scalar types do not add a reference

Adding a scalar type (`int2`, `int8`, …) does **not** add a `tests/codegen/reference/<T>/` directory. A per-type baseline would be redundant: the SQL is byte-identical to `int4` modulo the type token, so it can only fail when `int4`'s baseline already would. New types are guaranteed three other ways:

- the `int4` reference here anchors the shared generator (templates + `terms.py`);
- the committed `tests/sqlx/src/fixtures/<T>_values.rs` const is pinned by the CI staleness guard (`git diff --exit-code` after `mise run codegen:domain <T>`) and by the `<T>` cases in `tasks/codegen/test_scalars.py` (the only type-specific rendering, `i16::MIN` vs `i32::MIN`);
- the SQLx `ordered_numeric_matrix!` suite exercises the generated SQL's *behaviour* against a real database — a far stronger guarantee than a byte comparison.
Loading