Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ Each entry that ships in a published release links to the PR that introduced it.
- **`eql_v3` encrypted-domain schema, with the `int4` family as its first member.** Encrypted-domain type families now live in a new, additional `eql_v3` schema (the existing `eql_v2` schema is unchanged — it keeps the core types/operators and stays the documented public API). Four jsonb-backed domains for encrypted `int4` columns: `eql_v3.int4` (storage-only), `eql_v3.int4_eq` (`=` / `<>` via HMAC), and `eql_v3.int4_ord` / `eql_v3.int4_ord_ore` (also `<` `<=` `>` `>=` via ORE block terms). Supported comparisons resolve to inlinable wrappers; the native `jsonb` operator surface reachable through domain fallback is blocked (raises rather than silently mis-resolving). Each domain's `CHECK` requires the EQL envelope (`v`, `i`), the ciphertext (`c`), and the variant's index term(s), and pins the payload version (`VALUE->>'v' = '2'`, matching `eql_v2._encrypted_check_v`) — so a missing key or wrong-version payload is rejected on insert or cast rather than surfacing later at query time. Index via a functional index on the `eql_v3.eq_term` / `eql_v3.ord_term` extractors, not an operator class on the domain. The extractors return the searchable-encrypted-metadata index-term types `eql_v3.hmac_256` / `eql_v3.ore_block_u64_8_256`, which `eql_v3` owns directly (see the self-contained `eql_v3` schema entry below). Why: a type-safe, per-capability encrypted integer column instead of the untyped `eql_v2_encrypted`, namespaced under its own schema. This is the reference scalar implementation for the generated domain family. ([#239](https://github.com/cipherstash/encrypt-query-language/pull/239), supersedes [#225](https://github.com/cipherstash/encrypt-query-language/pull/225))
- **`eql_v3.int2` encrypted-domain type family.** Four jsonb-backed domains for encrypted `int2` columns — `eql_v3.int2` (storage-only), `eql_v3.int2_eq` (`=` / `<>` via HMAC), and `eql_v3.int2_ord` / `eql_v3.int2_ord_ore` (also `<` `<=` `>` `>=` via ORE block terms, with `MIN` / `MAX` aggregates) — generated from the `int2` row in `eql-scalars::CATALOG` by the same materializer as the `eql_v3.int4` reference. Index via a functional index on the `eql_v3.eq_term` / `eql_v3.ord_term` extractors, not an operator class on the domain. Why: a type-safe, per-capability encrypted `smallint` column, proving the scalar generator generalizes beyond the `int4` reference. ([#243](https://github.com/cipherstash/encrypt-query-language/pull/243))
- **`eql_v3.int8` encrypted-domain type family.** Four jsonb-backed domains for encrypted `int8` columns — `eql_v3.int8` (storage-only), `eql_v3.int8_eq` (`=` / `<>` via HMAC), and `eql_v3.int8_ord` / `eql_v3.int8_ord_ore` (also `<` `<=` `>` `>=` via ORE block terms, with `MIN` / `MAX` aggregates) — generated from the `int8` row in `eql-scalars::CATALOG` by the same materializer as the `eql_v3.int4` reference. Index via a functional index on the `eql_v3.eq_term` / `eql_v3.ord_term` extractors, not an operator class on the domain. Why: a type-safe, per-capability encrypted `bigint` column, extending the scalar generator across the full 64-bit integer width. ([#253](https://github.com/cipherstash/encrypt-query-language/pull/253))
- **`eql_v3.date` encrypted-domain type family.** Four jsonb-backed domains for encrypted `date` columns — `eql_v3.date` (storage-only), `eql_v3.date_eq` (`=` / `<>` via HMAC), and `eql_v3.date_ord` / `eql_v3.date_ord_ore` (also `<` `<=` `>` `>=` via ORE block terms, with `MIN` / `MAX` aggregates) — generated from the `date` row in `eql-scalars::CATALOG` by the same materializer as the `eql_v3.int4` reference. Plaintexts encrypt under the `date` cast and compare via the same ORE block terms as the integer scalars (ORE is plaintext-agnostic — dates order like integers). Index via a functional index on the `eql_v3.eq_term` / `eql_v3.ord_term` extractors, not an operator class on the domain. Why: the first **non-integer ordered** scalar encrypted-domain type — a type-safe, per-capability encrypted `date` column — proving the generator and SQLx test matrix generalize beyond fixed-width integers. ([#256](https://github.com/cipherstash/encrypt-query-language/pull/256))
- **Per-domain `MIN` / `MAX` aggregates for the encrypted-domain family.** `eql_v3.min(eql_v3.<T>_ord)` / `eql_v3.max(eql_v3.<T>_ord)` (and the `_ord_ore` twin) are generated for every ord-capable scalar variant, giving type-safe extrema on domain-typed columns — comparison routes through the variant's `<` / `>` operator (ORE block term, no decryption). The aggregates are declared `PARALLEL = SAFE` with a combine function (the state function itself — min/max are associative), so PostgreSQL can use partial/parallel aggregation on large `GROUP BY` workloads. Why: the new domain types previously had no equivalent of the composite-type aggregates. The existing `eql_v2.min(eql_v2_encrypted)` / `eql_v2.max(eql_v2_encrypted)` aggregates are **retained** and continue to work on `eql_v2_encrypted` columns; the per-domain aggregates are additive and coexist with them. ([#239](https://github.com/cipherstash/encrypt-query-language/pull/239))
- **Self-contained `eql_v3` schema + standalone `release/cipherstash-encrypt-v3.sql` installer.** The `eql_v3` encrypted-domain surface no longer depends on `eql_v2` at runtime: it now owns its own copies of the searchable-encrypted-metadata (SEM) index-term types — `eql_v3.hmac_256` and `eql_v3.ore_block_u64_8_256` (with its btree operator class) — so the `eql_v3.eq_term` / `eql_v3.ord_term` extractors return `eql_v3` types and no `eql_v2.<symbol>` appears anywhere in the v3 SQL. The whole v3 surface relocated under a single `src/v3/` tree (`src/v3/sem/` for the hand-written SEM types, `src/v3/scalars/` for the generated domain families). A new build variant ships the `eql_v3` schema on its own as `release/cipherstash-encrypt-v3.sql`, installable into a database with no `eql_v2` present; a CI gate greps that artifact and its dependency closure to keep it `eql_v2`-free. Why: a clean foundation for the per-scalar encrypted-domain model to stand alone, ahead of it replacing the `eql_v2_encrypted` composite column type. This is additive — a new schema and a new artifact — and leaves `eql_v2` byte-for-byte unchanged. ([#255](https://github.com/cipherstash/encrypt-query-language/pull/255))

Expand Down
5 changes: 5 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

144 changes: 132 additions & 12 deletions crates/eql-scalars/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,11 @@ pub enum ScalarKind {
Numeric,
Text,
Jsonb,
/// Calendar date (`chrono::NaiveDate`). Ordered like the integer kinds via
/// ORE, but string-backed (ISO-8601) at the catalog layer and with no i128
/// range — so it is *not* `is_int()` and the bounded-numeric accessors
/// panic for it, exactly like the other non-integer kinds.
Date,
}

impl ScalarKind {
Expand All @@ -50,6 +55,7 @@ impl ScalarKind {
ScalarKind::Numeric => "numeric",
ScalarKind::Text => "text",
ScalarKind::Jsonb => "jsonb",
ScalarKind::Date => "chrono::NaiveDate",
}
}

Expand All @@ -61,7 +67,7 @@ impl ScalarKind {
ScalarKind::I64 => "i64::MIN",
// Explicit (not `_`) so a future integer variant is a compile
// error here rather than silently hitting the panic.
ScalarKind::Numeric | ScalarKind::Text | ScalarKind::Jsonb => {
ScalarKind::Numeric | ScalarKind::Text | ScalarKind::Jsonb | ScalarKind::Date => {
panic!("min_symbol is only defined for integer kinds")
}
}
Expand All @@ -73,7 +79,7 @@ impl ScalarKind {
ScalarKind::I16 => "i16::MAX",
ScalarKind::I32 => "i32::MAX",
ScalarKind::I64 => "i64::MAX",
ScalarKind::Numeric | ScalarKind::Text | ScalarKind::Jsonb => {
ScalarKind::Numeric | ScalarKind::Text | ScalarKind::Jsonb | ScalarKind::Date => {
panic!("max_symbol is only defined for integer kinds")
}
}
Expand All @@ -83,7 +89,7 @@ impl ScalarKind {
pub const fn zero_symbol(self) -> &'static str {
match self {
ScalarKind::I16 | ScalarKind::I32 | ScalarKind::I64 => "0",
ScalarKind::Numeric | ScalarKind::Text | ScalarKind::Jsonb => {
ScalarKind::Numeric | ScalarKind::Text | ScalarKind::Jsonb | ScalarKind::Date => {
panic!("zero_symbol is only defined for integer kinds")
}
}
Expand All @@ -96,7 +102,7 @@ impl ScalarKind {
ScalarKind::I16 => i16::MIN as i128,
ScalarKind::I32 => i32::MIN as i128,
ScalarKind::I64 => i64::MIN as i128,
ScalarKind::Numeric | ScalarKind::Text | ScalarKind::Jsonb => {
ScalarKind::Numeric | ScalarKind::Text | ScalarKind::Jsonb | ScalarKind::Date => {
panic!("min_value is only defined for integer kinds")
}
}
Expand All @@ -109,7 +115,7 @@ impl ScalarKind {
ScalarKind::I16 => i16::MAX as i128,
ScalarKind::I32 => i32::MAX as i128,
ScalarKind::I64 => i64::MAX as i128,
ScalarKind::Numeric | ScalarKind::Text | ScalarKind::Jsonb => {
ScalarKind::Numeric | ScalarKind::Text | ScalarKind::Jsonb | ScalarKind::Date => {
panic!("max_value is only defined for integer kinds")
}
}
Expand Down Expand Up @@ -249,6 +255,10 @@ pub enum Fixture {
Numeric(&'static str),
Text(&'static str),
Jsonb(&'static str),
/// An ISO-8601 date string (`"1970-01-01"`). The catalog stays zero-dep, so
/// the string is parsed into a `chrono::NaiveDate` in the SQLx harness, not
/// here. Distinct by literal, like the other string-backed fixtures.
Date(&'static str),
}

impl Fixture {
Expand All @@ -264,7 +274,7 @@ impl Fixture {
Fixture::Max => Some(kind.max_value()),
Fixture::Zero => Some(0),
Fixture::Int(n) => Some(n),
Fixture::Numeric(_) | Fixture::Text(_) | Fixture::Jsonb(_) => None,
Fixture::Numeric(_) | Fixture::Text(_) | Fixture::Jsonb(_) | Fixture::Date(_) => None,
}
}

Expand All @@ -276,7 +286,7 @@ impl Fixture {
Fixture::Max => kind.max_symbol().to_string(),
Fixture::Zero => kind.zero_symbol().to_string(),
Fixture::Int(n) => n.to_string(),
Fixture::Numeric(s) | Fixture::Text(s) | Fixture::Jsonb(s) => {
Fixture::Numeric(s) | Fixture::Text(s) | Fixture::Jsonb(s) | Fixture::Date(s) => {
format!("{s:?}")
}
}
Expand Down Expand Up @@ -349,6 +359,7 @@ macro_rules! fixtures {
(text; $($s:literal),* $(,)?) => { &[$(Fixture::Text($s)),*] };
(numeric; $($s:literal),* $(,)?) => { &[$(Fixture::Numeric($s)),*] };
(jsonb; $($s:literal),* $(,)?) => { &[$(Fixture::Jsonb($s)),*] };
(date; $($s:literal),* $(,)?) => { &[$(Fixture::Date($s)),*] };
}

/// int4 fixture plaintexts — verbatim from `tasks/codegen/types/int4.toml`.
Expand All @@ -370,6 +381,20 @@ const INT8_FIXTURES: &[Fixture] = fixtures!(int i64;
Min, N(-5000000000), N(-100), N(-1), Zero, N(1), N(2), N(5), N(10), N(17),
N(25), N(42), N(50), N(100), N(250), N(1000), N(9999), N(5000000000), Max);

/// date fixture plaintexts — ISO-8601 (`YYYY-MM-DD`) strings, parsed into
/// `chrono::NaiveDate` in the SQLx harness (the catalog stays zero-dep). The
/// three temporal pivots MUST be present verbatim: `"1900-01-01"` (min_pivot),
/// `"1970-01-01"` (zero = `NaiveDate::default()`), and `"2099-12-31"`
/// (max_pivot) — the matrix fetches each one's ciphertext via
/// `fetch_fixture_payload`, which fails loudly if a row is absent. The interior
/// dates span varied years/months so range operators yield distinguishable
/// counts. All distinct.
const DATE_FIXTURES: &[Fixture] = fixtures!(date;
"1900-01-01", "1950-07-15", "1969-12-31", "1970-01-01", "1970-01-02",
"1980-02-29", "1991-11-09", "1999-12-31", "2000-01-01", "2004-02-29",
"2012-06-30", "2016-03-15", "2020-10-21", "2024-02-29", "2038-01-19",
"2099-12-31");

const INT4: ScalarSpec = ScalarSpec {
token: "int4",
kind: ScalarKind::I32,
Expand All @@ -391,13 +416,28 @@ const INT8: ScalarSpec = ScalarSpec {
fixtures: INT8_FIXTURES,
};

/// `date` — an ordered, non-integer scalar. Reuses `ORDERED_INT_DOMAINS` (the
/// four-domain ordered shape is identical to the integer scalars); only the
/// kind and fixtures differ.
///
/// Public (unlike the integer specs) because the SQLx harness reads
/// `DATE.fixtures` directly to parse the ISO strings into `chrono::NaiveDate`
/// at runtime — there is no `DATE_VALUES` const (chrono is not `const`-friendly
/// and `eql-scalars` stays zero-dep, so no typed slice is materialised here).
pub const DATE: ScalarSpec = ScalarSpec {
token: "date",
kind: ScalarKind::Date,
domains: ORDERED_INT_DOMAINS,
fixtures: DATE_FIXTURES,
};

/// The scalar catalog — the single source of truth. Order is significant (it
/// drives generation order). New types are appended as their SQL surface lands.
pub const CATALOG: &[ScalarSpec] = &[INT4, INT2, INT8];
pub const CATALOG: &[ScalarSpec] = &[INT4, INT2, INT8, DATE];

/// Materialise an integer scalar's fixtures into a typed `&'static` slice at
/// compile time. This is the **single-sourced** plaintext list the SQLx test
/// matrix reads as `ScalarType::FIXTURE_VALUES` and the fixture generator
/// matrix reads via `ScalarType::fixture_values()` and the fixture generator
/// encrypts — derived from the same `CATALOG` row that drives SQL generation,
/// so the oracle cannot drift from the fixture. (It replaces the old generated,
/// committed `tests/sqlx/src/fixtures/<T>_values.rs` — a Rust source of truth no
Expand Down Expand Up @@ -478,6 +518,7 @@ mod rust_tests {
assert!(!ScalarKind::Numeric.is_int());
assert!(!ScalarKind::Text.is_int());
assert!(!ScalarKind::Jsonb.is_int());
assert!(!ScalarKind::Date.is_int());
}

// Pin that the bounded-numeric accessors panic (with message) on non-int kinds.
Expand Down Expand Up @@ -522,6 +563,46 @@ mod rust_tests {
assert_eq!(ScalarKind::I64.min_value(), -9_223_372_036_854_775_808_i128);
assert_eq!(ScalarKind::I64.max_value(), 9_223_372_036_854_775_807_i128);
}

#[test]
fn date_maps_to_naive_date() {
// Ordered, non-integer kind: it carries a rust type but no i128 range,
// so it is not `is_int()` and the bounded accessors panic (below).
assert_eq!(ScalarKind::Date.rust_type(), "chrono::NaiveDate");
assert!(!ScalarKind::Date.is_int());
}

// The bounded-numeric accessors panic on Date exactly as on the other
// non-integer kinds — Date is not an integer kind.
#[test]
#[should_panic(expected = "min_symbol is only defined for integer kinds")]
fn min_symbol_panics_on_date() {
ScalarKind::Date.min_symbol();
}
Comment on lines +579 to +581
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a big deal but this might imply that min_symbol and max_symbol should be defined in a separate subtrait (can be defined by Int2 etc but not date).


#[test]
#[should_panic(expected = "max_symbol is only defined for integer kinds")]
fn max_symbol_panics_on_date() {
ScalarKind::Date.max_symbol();
}

#[test]
#[should_panic(expected = "zero_symbol is only defined for integer kinds")]
fn zero_symbol_panics_on_date() {
ScalarKind::Date.zero_symbol();
}

#[test]
#[should_panic(expected = "min_value is only defined for integer kinds")]
fn min_value_panics_on_date() {
ScalarKind::Date.min_value();
}

#[test]
#[should_panic(expected = "max_value is only defined for integer kinds")]
fn max_value_panics_on_date() {
ScalarKind::Date.max_value();
}
}

#[cfg(test)]
Expand Down Expand Up @@ -682,6 +763,10 @@ mod fixture_tests {
Fixture::Jsonb(r#"{"a":1}"#).numeric_value(ScalarKind::Jsonb),
None
);
assert_eq!(
Fixture::Date("1970-01-01").numeric_value(ScalarKind::Date),
None
);
}

#[test]
Expand Down Expand Up @@ -718,6 +803,10 @@ mod fixture_tests {
Fixture::Jsonb(r#"{"a":1}"#).render_literal(ScalarKind::Jsonb),
r#""{\"a\":1}""#
);
assert_eq!(
Fixture::Date("1970-01-01").render_literal(ScalarKind::Date),
"\"1970-01-01\""
);
}

#[test]
Expand All @@ -741,6 +830,11 @@ mod fixture_tests {
assert_eq!(NUMS, &[Fixture::Numeric("0.1"), Fixture::Numeric("-2.5")]);
const JSONS: &[Fixture] = fixtures!(jsonb; r#"{"a":1}"#);
assert_eq!(JSONS, &[Fixture::Jsonb(r#"{"a":1}"#)]);
const DATES: &[Fixture] = fixtures!(date; "1970-01-01", "2099-12-31");
assert_eq!(
DATES,
&[Fixture::Date("1970-01-01"), Fixture::Date("2099-12-31")]
);
}

#[test]
Expand Down Expand Up @@ -773,9 +867,33 @@ mod catalog_tests {
}

#[test]
fn catalog_has_int4_int2_int8_in_order() {
fn catalog_has_int4_int2_int8_date_in_order() {
let tokens: Vec<&str> = CATALOG.iter().map(|s| s.token).collect();
assert_eq!(tokens, vec!["int4", "int2", "int8"]);
assert_eq!(tokens, vec!["int4", "int2", "int8", "date"]);
}

/// The three temporal matrix pivots must be present verbatim in DATE's
/// fixture strings — `fetch_fixture_payload` fetches each one's ciphertext,
/// failing loudly if absent. The integer `fixtures_include_min_max_and_zero`
/// invariant filters `is_int()` and skips date, so this is its temporal
/// analogue.
#[test]
fn temporal_fixtures_include_pivot_plaintexts() {
let date = scalar("date");
let strings: Vec<&str> = date
.fixtures
.iter()
.filter_map(|f| match f {
Fixture::Date(s) => Some(*s),
_ => None,
})
.collect();
for pivot in ["1900-01-01", "1970-01-01", "2099-12-31"] {
assert!(
strings.contains(&pivot),
"date fixtures missing temporal pivot {pivot}"
);
}
}

#[test]
Expand Down Expand Up @@ -901,7 +1019,9 @@ mod invariant_tests {

fn distinct_key(f: Fixture, kind: ScalarKind) -> DistinctKey {
match f {
Fixture::Numeric(s) | Fixture::Text(s) | Fixture::Jsonb(s) => DistinctKey::Str(s),
Fixture::Numeric(s) | Fixture::Text(s) | Fixture::Jsonb(s) | Fixture::Date(s) => {
DistinctKey::Str(s)
}
_ => DistinctKey::Num(
f.numeric_value(kind)
.expect("sentinel/Int fixtures resolve to a number"),
Expand Down
Loading