Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
c1b2c3e
feat(eql_v2)!: collapse ste_vec ORE terms to single oc field
coderdan May 18, 2026
069697c
fix(ci): allowlist `eql_v2.ore_cllw` and `has_ore_cllw` in splinter
coderdan May 19, 2026
1be0646
feat(eql_v2)!: strict separation between root payload and sv-element …
coderdan May 19, 2026
70d803d
test(index_compare): migrate ore_cllw tests to typed entry API
coderdan May 19, 2026
3fab115
test(operator_compare): migrate to entry shape + new error wording
coderdan May 19, 2026
2e57831
refactor(hash): assume `hm` at root, drop coalesce fallback
coderdan May 19, 2026
11cb7b6
perf(ore_cllw): remove early exit + use bitmask in compare_term_bytes
coderdan May 19, 2026
e32999b
test: add `build_synthetic_ste_vec(id)` helper for v2.3-shape ste_vec…
coderdan May 19, 2026
a77aeef
test: rewrite tests/ste_vec.sql to v2.3-shape inline INSERTs
coderdan May 19, 2026
4872c10
feat(ste_vec_entry)!: tighten DOMAIN check to require s + c + hm
coderdan May 19, 2026
6bde3e3
test: re-enable ignored ste_vec tests + add `hm` to legacy fixtures
coderdan May 19, 2026
7b6ed5f
fix(ste_vec_entry)!: DOMAIN check is XOR(hm, oc), not AND(hm, c)
coderdan May 19, 2026
06d706a
chore(v2.3): codify XOR(hm, oc) contract end-to-end
coderdan May 19, 2026
ecb7ee6
docs(v2.3): address U-006 review feedback
coderdan May 20, 2026
2558bca
docs(changelog): add PR links to unlinked v2.3 entries
coderdan May 20, 2026
7a78645
fix(src): address Dan + CodeRabbit source-side review feedback
coderdan May 20, 2026
787074d
fix(fixtures): drop root c from ste_vec_vast + b3 from hm-backstop
coderdan May 20, 2026
5ea4a4f
test: address Dan's test-side review feedback
coderdan May 20, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 13 additions & 5 deletions CHANGELOG.md

Large diffs are not rendered by default.

23 changes: 10 additions & 13 deletions docs/reference/database-indexes.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,8 @@ For PostgreSQL to use an index on encrypted columns, **all** of these conditions

The encrypted data must contain the index term types that support the operation:

- **Equality queries** - Require `unique` index config (adds `hm` hmac_256 or `b3` blake3 terms)
- **Range queries** - Require `ore` index config (adds `ob` ore_block_u64_8_256 terms) **or** `ope` index config (adds `opf` ope_cllw_u64_65 / `opv` ope_cllw_var_8 terms)
- **Equality queries** - Require `unique` index config (adds `hm` hmac_256 terms)
- **Range queries** - Require `ore` index config on root scalars (adds `ob` ore_block_u64_8_256 terms), or `ste_vec` on encrypted JSON columns (adds `oc` ORE CLLW on sv elements — see [U-006](../upgrading/v2.3.md#u-006-ste_vec-ore-field-consolidation))
- **Pattern matching** - Typically scans (bloom filters don't use B-tree indexes)

**Example:**
Expand Down Expand Up @@ -179,7 +179,7 @@ SELECT * FROM events

The sort key now matches the functional index expression, so the planner streams rows out of the index in order — a plain Index Scan, no separate Sort node.

**Non-Block-ORE term types.** For columns carrying only `ore_cllw_u64_8`, `ore_cllw_var_8`, `opf` (OPE fixed-width), or `opv` (OPE variable-width) terms, the bare-form `<` / `>` operators no longer dispatch through `eql_v2.compare()` — they go straight to the Block ORE extractor, which raises on a missing `ob`. Either migrate the column configuration to `ore` (Block ORE), or rewrite range queries to the matching extractor form, e.g. `WHERE eql_v2.ore_cllw_u64_8(col) < eql_v2.ore_cllw_u64_8($1::jsonb)`. See [U-005](../upgrading/v2.3.md#u-005-range-operators-are-block-ore-only) for the migration notes.
**Non-Block-ORE term types.** For columns carrying only `oc` (sv-element ORE CLLW), the bare-form `<` / `>` operators no longer dispatch through `eql_v2.compare()` — they go straight to the Block ORE extractor, which raises on a missing `ob`. Either migrate the column configuration to `ore` (Block ORE), or rewrite range queries to the extractor form: `WHERE eql_v2.ore_cllw(e->'<selector>'::text) < eql_v2.ore_cllw($1::jsonb)`. See [U-005](../upgrading/v2.3.md#u-005-range-operators-are-block-ore-only) and [U-006](../upgrading/v2.3.md#u-006-ste_vec-ore-field-consolidation) for the migration notes.

### GROUP BY

Expand Down Expand Up @@ -276,10 +276,8 @@ CREATE INDEX ON users (encrypted_email eql_v2.encrypted_operator_class);

B-tree indexes **only work** with:
- `hm` (hmac_256) - for equality
- `b3` (blake3) - for equality
- `ob` (ore_block_u64_8_256) - for range queries
- `opf` (ope_cllw_u64_65) - for range queries (fixed-width OPE)
- `opv` (ope_cllw_var_8) - for range queries (variable-width OPE)
- `ob` (ore_block_u64_8_256) - for range queries on root scalars
- `oc` (ore_cllw) - for range queries on `ste_vec` elements (needs a custom comparator; tracked as #220)

They **do not work** with:
- `bf` (bloom_filter) - pattern matching
Expand Down Expand Up @@ -482,13 +480,12 @@ If you see `Seq Scan`, ensure:
**Check 1: Verify data has index terms**

```sql
-- Check if data contains hm (hmac_256) or b3 (blake3) for equality,
-- ob (ore) for range, or opf/opv (ope) for range
-- Check if data contains hm (hmac) for equality, ob (Block ORE)
-- for range queries on root scalars, or oc for range queries on
-- ste_vec elements.
SELECT encrypted_email::jsonb ? 'hm' AS has_hmac,
encrypted_email::jsonb ? 'b3' AS has_blake3,
encrypted_email::jsonb ? 'ob' AS has_ore,
encrypted_email::jsonb ? 'opf' AS has_ope_fixed,
encrypted_email::jsonb ? 'opv' AS has_ope_var
encrypted_email::jsonb ? 'ob' AS has_ore_block,
encrypted_email::jsonb ? 'oc' AS has_ore_cllw
FROM users LIMIT 1;
```

Expand Down
74 changes: 21 additions & 53 deletions docs/reference/schema/eql-payload-v2.3.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://cipherstash.com/schemas/eql/v2.3/payload.schema.json",
"title": "EQL v2.3 payload",
"description": "JSON Schema describing the on-the-wire EQL payload format as of release v2.3. Two top-level shapes are accepted, mutually exclusive: an `EncryptedPayload` (a single encrypted scalar value, with any combination of index terms) and an `SteVecPayload` (an STE vector for jsonb / containment queries). The encrypted root form is what `eql_v2.check_encrypted` validates and what `eql_v2_encrypted` columns hold.\n\nChanges since v2.2:\n- The `b3` (Blake3) index term is removed everywhere. STE vector element equality now uses `hm` directly.\n- `SteVecElement` may now carry an `hm` field (matches the root `EncryptedPayload`); the previous prohibition is lifted.\n- `opf` and `opv` are collapsed into a single `op` field. Width information that was previously implied by the field name is now carried on the value itself.\n- OPE (`op`) and ORE (`ob`, `ocf`, `ocv`) are now mutually exclusive within a single payload or `sv` element: a column is configured for one ordered scheme or the other, never both.",
"description": "JSON Schema describing the on-the-wire EQL payload format as of release v2.3. Two top-level shapes are accepted, mutually exclusive: an `EncryptedPayload` (a single encrypted scalar value, with any combination of index terms) and an `SteVecPayload` (an STE vector for jsonb / containment queries). The encrypted root form is what `eql_v2.check_encrypted` validates and what `eql_v2_encrypted` columns hold.\n\nChanges since v2.2:\n- The `b3` (Blake3) index term is removed everywhere. STE vector element equality now uses `hm` directly.\n- `SteVecElement` may now carry an `hm` field (matches the root `EncryptedPayload`); the previous prohibition is lifted.\n- `ste_vec` elements now carry a single ORE term `oc` (CLLW ORE). The pre-2.3 `ocf` / `ocv` split collapses into this one field: width information that was previously implied by the field name is now carried on the ciphertext via a leading domain-tag byte (`0x00` numeric, `0x01` string).\n- The two ordered schemes are scope-disjoint: `ob` (Block ORE) appears only on the root payload (scalar `eql_v2_encrypted` values), `oc` (CLLW ORE) appears only on `sv` elements.\n- Each `SteVecElement` carries exactly one of `hm` (boolean leaves / array / object roots) or `oc` (string / number leaves). They are mutually exclusive — cipherstash-suite never emits both on the same element. This is enforced by a `oneOf` constraint on `SteVecElement` and by the `eql_v2.ste_vec_entry` DOMAIN check in the EQL install.",

"oneOf": [
{ "$ref": "#/$defs/EncryptedPayload" },
Expand Down Expand Up @@ -46,25 +46,21 @@

"EncryptedPayload": {
"title": "Encrypted payload (scalar, database side)",
"description": "Shape stored in `eql_v2_encrypted` columns for a single encrypted scalar value. `v`, `c`, and `i.{t,c}` are required by `eql_v2.check_encrypted`. Any subset of the index-term fields may be present, depending on the configured indexes for the column. Mutually exclusive with `SteVecPayload`: must not carry `sv`. The OPE term (`op`) and the ORE terms (`ob`, `ocf`, `ocv`) are also mutually exclusive — see `OreOpeExclusion`.",
"description": "Shape stored in `eql_v2_encrypted` columns for a single encrypted scalar value. `v`, `c`, and `i.{t,c}` are required by `eql_v2.check_encrypted`. Any subset of the index-term fields may be present, depending on the configured indexes for the column. The orderable term at this scope is `ob` (Block ORE); `oc` lives only on `sv` elements. Mutually exclusive with `SteVecPayload`: must not carry `sv`.",
"type": "object",
"properties": {
"v": { "$ref": "#/$defs/Version" },
"k": { "type": "string", "const": "ct", "description": "Discriminator: identifies this as the scalar ciphertext form." },
"c": { "$ref": "#/$defs/Ciphertext" },
"i": { "$ref": "#/$defs/Ident" },

"hm": { "$ref": "#/$defs/IndexTerms/properties/hm" },
"bf": { "$ref": "#/$defs/IndexTerms/properties/bf" },
"ob": { "$ref": "#/$defs/IndexTerms/properties/ob" },
"ocf": { "$ref": "#/$defs/IndexTerms/properties/ocf" },
"ocv": { "$ref": "#/$defs/IndexTerms/properties/ocv" },
"op": { "$ref": "#/$defs/IndexTerms/properties/op" }
"hm": { "$ref": "#/$defs/IndexTerms/properties/hm" },
"bf": { "$ref": "#/$defs/IndexTerms/properties/bf" },
"ob": { "$ref": "#/$defs/IndexTerms/properties/ob" }
},
"required": ["v", "c", "i"],
"allOf": [
{ "not": { "required": ["sv"] } },
{ "$ref": "#/$defs/OreOpeExclusion" }
{ "not": { "required": ["sv"] } }
],
"additionalProperties": false
},
Expand All @@ -89,7 +85,7 @@

"SteVecElement": {
"title": "STE vector element",
"description": "One entry inside `sv`. Distinct from the root payload: carries a per-element selector `s` and an array flag `a`, but never an `i`, `v`, or nested `sv`. As of v2.3 element equality is matched by `s` plus `hm` (Blake3 / `b3` is removed); `hm` is now permitted at this level. The OPE term (`op`) and the ORE terms (`ob`, `ocf`, `ocv`) are mutually exclusive within an element — see `OreOpeExclusion`.",
"description": "One entry inside `sv`. Distinct from the root payload: carries a per-element selector `s` and an array flag `a`, but never an `i`, `v`, or nested `sv`. Every element carries exactly one of two equality / ordering terms — `hm` (HMAC-256) for boolean leaves and for the placeholder entries that represent array / object roots, or `oc` (CLLW ORE) for string and number leaves. They are mutually exclusive: cipherstash-suite emits one and only one per entry; an element with both terms or with neither is malformed.",
"type": "object",
"properties": {
"s": {
Expand All @@ -100,12 +96,9 @@
"type": "boolean",
"description": "Array marker. True when the selector points at a JSON array context."
},
"c": { "$ref": "#/$defs/Ciphertext" },
"hm": { "$ref": "#/$defs/IndexTerms/properties/hm" },
"ocf": { "$ref": "#/$defs/IndexTerms/properties/ocf" },
"ocv": { "$ref": "#/$defs/IndexTerms/properties/ocv" },
"op": { "$ref": "#/$defs/IndexTerms/properties/op" },
"ob": { "$ref": "#/$defs/IndexTerms/properties/ob" }
"c": { "$ref": "#/$defs/Ciphertext" },
"hm": { "$ref": "#/$defs/IndexTerms/properties/hm" },
"oc": { "$ref": "#/$defs/IndexTerms/properties/oc" }
},
"required": ["s", "c"],
"allOf": [
Comment thread
coderdan marked this conversation as resolved.
Expand All @@ -118,28 +111,17 @@
]
}
},
{ "$ref": "#/$defs/OreOpeExclusion" }
{
"description": "Exactly one of `hm` or `oc` must be present. `hm` covers boolean leaves and array / object root placeholders; `oc` covers string and number leaves. cipherstash-suite never emits both or neither on the same element.",
"oneOf": [
{ "required": ["hm"] },
{ "required": ["oc"] }
]
}
],
"additionalProperties": false
},

"OreOpeExclusion": {
"title": "OPE / ORE mutual exclusion",
"description": "A payload (or `sv` element) configured for ordered comparison must use OPE (`op`) or ORE (`ob` / `ocf` / `ocv`), never both. Encoded as `not (op AND any-ORE)`.",
"not": {
"allOf": [
{ "required": ["op"] },
{
"anyOf": [
{ "required": ["ob"] },
{ "required": ["ocf"] },
{ "required": ["ocv"] }
]
}
]
}
},

"IndexTerms": {
"title": "Index term field catalogue",
"description": "Catalogue of every deterministic / order-preserving index term emitted by Proxy in v2.3. Defined here so both the root payload and STE vector elements can $ref individual entries without duplicating descriptions.",
Expand All @@ -162,28 +144,14 @@

"ob": {
"title": "ORE block u64_8_256 (ob)",
"description": "Order-Revealing Encryption term used for ordered comparisons (`<`, `<=`, `>`, `>=`) via the custom `eql_v2.ore_block_u64_8_256` comparator. Stored as an array of hex-encoded ORE blocks. Mutually exclusive with `op` (OPE).",
"description": "Order-Revealing Encryption term used for ordered comparisons (`<`, `<=`, `>`, `>=`) on root scalars via the custom `eql_v2.ore_block_u64_8_256` comparator. Stored as an array of hex-encoded ORE blocks. Mutually exclusive with `oc`.",
"type": "array",
"items": { "type": "string", "pattern": "^[0-9a-f]+$" }
},

"ocf": {
"title": "ORE CLLW u64_8 fixed (ocf)",
"description": "CLLW Order-Revealing Encryption, fixed-width (numeric). Hex-encoded. Mutually exclusive with `op` (OPE).",
"type": "string",
"pattern": "^[0-9a-f]+$"
},

"ocv": {
"title": "ORE CLLW var_8 (ocv)",
"description": "CLLW Order-Revealing Encryption, variable-width (text). Hex-encoded. Mutually exclusive with `op` (OPE).",
"type": "string",
"pattern": "^[0-9a-f]+$"
},

"op": {
"title": "OPE CLLW (op)",
"description": "CLLW Order-Preserving Encryption term, hex-encoded. Sortable with native bytea ordering — used in environments without custom comparators. As of v2.3 a single field replaces the previous `opf` (fixed-width / numeric) and `opv` (variable-width / text) split; the value itself encodes any width information the comparator needs. Mutually exclusive with the ORE terms (`ob`, `ocf`, `ocv`).",
"oc": {
"title": "ORE CLLW (oc)",
"description": "CLLW Order-Revealing Encryption term for `ste_vec` elements. Hex-encoded; the decoded byte string starts with a domain-tag byte (`0x00` numeric, `0x01` string) followed by the CLLW ciphertext, which lets a single column hold mixed numeric and string values with a consistent total order. Mutually exclusive with `ob`. Comparison uses the CLLW per-byte protocol (`y + 1 == x`), so a custom comparator / opclass is required for index-driven ordered queries.",
"type": "string",
"pattern": "^[0-9a-f]+$"
}
Expand Down
Loading