From f025bf71707c95cdd690f392f6c463cae03992b1 Mon Sep 17 00:00:00 2001 From: InauguralPhysicist Date: Sun, 5 Jul 2026 15:29:51 -0500 Subject: [PATCH] fix(chr): byte-writing inverse of ord (1-255, loud errors); lib/utf8.eigs gains utf8_encode (#435) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit chr silently returned "" for anything outside 0-127 and silently truncated fractions — the silent-tolerance class. It is now the byte-writing partner of ord under the bytes-forever position (#416): any integer 1-255 emits that raw byte (chr of n == str_from_bytes of [n]); 0 raises (strings are NUL-terminated), as do negatives, fractions, >255 (message points at utf8_encode), and non-numbers. #435's premise was a misdiagnosis: high bytes were already constructible via str_from_bytes (#248). utf8_encode + utf8_from_codepoints land in lib/utf8.eigs on top of it, tested against the published vectors, every width boundary, and a from_codepoints∘codepoints round-trip. The false doc claims (chr 'emits the UTF-8 of a codepoint' in BUILTINS.md and the str_from_bytes C comment; 'decode-only' in STDLIB.md and the module header) are corrected. Both suites green: release + ASan/UBSan detect_leaks=1, 2519/2519, leak tally 0. Closes #435 Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 13 ++++++++++ docs/BUILTINS.md | 6 ++--- docs/STDLIB.md | 5 ++-- lib/utf8.eigs | 46 ++++++++++++++++++++++++++++------ src/builtins.c | 32 +++++++++++++++++------ tests/test_builtin_errors.eigs | 41 ++++++++++++++++++++++++++++++ tests/test_misc_builtins.eigs | 7 +++++- tests/test_utf8.eigs | 25 ++++++++++++++++++ 8 files changed, 153 insertions(+), 22 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 379f038..4343cd0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,20 @@ All notable changes to EigenScript are documented here. ## [Unreleased] +### Changed +- **`chr` is the byte-writing inverse of `ord`** (#435): `chr of n` now emits + the raw byte for any integer 1–255 (was: silent `""` for anything above + 127). Everything else raises loudly — 0 (strings are NUL-terminated), + negatives, fractions (previously truncated silently), values above 255 + (the error points at `utf8_encode` for codepoints), and non-numbers. + `chr of n` == `str_from_bytes of [n]` across the shared range. + ### Added +- **`utf8_encode` / `utf8_from_codepoints`** in `lib/utf8.eigs` (#435): the + encode half of the #416 module — codepoint(s) → UTF-8 byte string, built on + `str_from_bytes`. #435's premise was a misdiagnosis: high bytes were already + constructible via `str_from_bytes` (#248); encode had been omitted on that + mistaken belief. - **Parse errors carry `line:col`** (#407, first increment): the lexer already tracked columns; now the two parser diagnostics surface them. Human output reads `Parse error line 6:9: …`, the `--lint --json` `E002` element gains a diff --git a/docs/BUILTINS.md b/docs/BUILTINS.md index a2df19c..35f61a9 100644 --- a/docs/BUILTINS.md +++ b/docs/BUILTINS.md @@ -77,7 +77,7 @@ numeric fast paths used by reassignment and `unobserved` blocks. | `scan_int_tokens` | `scan_int_tokens of s` or `scan_int_tokens of [s, comment_marker]` | Token rows `[text, line, col, start, end, is_int, value]` | | `trim` | `trim of s` | Strip leading/trailing whitespace | | `str_replace` | `str_replace of [s, old, new]` | Replace all occurrences of old with new | -| `chr` | `chr of code` | Convert ASCII code to single character | +| `chr` | `chr of byte` | One-byte string from a byte value 1–255 (the writing inverse of `ord`). Raises outside 1–255 — including 0, since strings are NUL-terminated — and on fractions; for a Unicode *codepoint* use `utf8_encode` (lib/utf8.eigs). | | `join` | `join of [list, sep]` | Concatenate list elements with separator (C-backed, O(n)) | | `text_builder_new` | `text_builder_new of null` | Create a native growable text builder | | `text_builder_append` | `text_builder_append of [builder, value]` | Append one value as text | @@ -143,11 +143,11 @@ Compact typed arrays of doubles with O(1) indexed access. Iterable with ### Bytes ↔ values For serialization: reconstruct strings/floats from raw bytes (the inverse of an -`ord` loop / manual bit-packing), covering cases `chr` and 32-bit bitwise can't. +`ord` loop / manual bit-packing), covering cases 32-bit bitwise can't. | Name | Signature | Description | |------|-----------|-------------| -| `str_from_bytes` | `str_from_bytes of ` | Build a string from raw byte values (0–255). Unlike `chr` (which emits the UTF-8 of a *codepoint*), this writes the bytes verbatim, so it inverts an `ord`-over-bytes loop for any byte. Strings are NUL-terminated: a `0` byte ends the string — keep NUL-bearing binary in a buffer. | +| `str_from_bytes` | `str_from_bytes of ` | Build a string from raw byte values (0–255) — the list form of `chr` (`chr of n` == `str_from_bytes of [n]` for 1–255), inverting an `ord`-over-bytes loop. Strings are NUL-terminated: a `0` byte ends the string — keep NUL-bearing binary in a buffer. | | `f64_to_bytes` | `f64_to_bytes of x` | List of 8 ints: the big-endian IEEE-754 encoding of double `x` (network byte order, portable across host endianness). | | `f64_from_bytes` | `f64_from_bytes of ` | Decode a double from the first 8 big-endian IEEE-754 bytes. Inverse of `f64_to_bytes`. | diff --git a/docs/STDLIB.md b/docs/STDLIB.md index 05da13d..97c84fd 100644 --- a/docs/STDLIB.md +++ b/docs/STDLIB.md @@ -435,9 +435,8 @@ semantics when you need them, decoding UTF-8 over the byte primitives. | `utf8_at` | `utf8_at of [s, i]` | i-th codepoint (0-indexed), or -1 | | `utf8_char_at` | `utf8_char_at of [s, i]` | i-th character as a (multi-byte) string, or "" | | `utf8_validate` | `utf8_validate of s` | 1 if structurally valid UTF-8, else 0 | - -Decode-only for now: encoding needs to write bytes ≥ 0x80, which `chr` cannot -yet do (filed as #435). Reading raw bytes works, so decode/validate do. +| `utf8_encode` | `utf8_encode of cp` | Codepoint → UTF-8 byte string ("" if unencodable) | +| `utf8_from_codepoints` | `utf8_from_codepoints of cps` | List of codepoints → UTF-8 string (inverse of `utf8_codepoints`) | ### lib/pkg.eigs — Package Manager Runtime diff --git a/lib/utf8.eigs b/lib/utf8.eigs index 35eb078..ad9335e 100644 --- a/lib/utf8.eigs +++ b/lib/utf8.eigs @@ -15,12 +15,12 @@ # utf8_at of [s, i] # i-th codepoint (0-indexed), or -1 if out of range # utf8_char_at of [s, i] # i-th character as a (multi-byte) string, or "" # utf8_validate of s # 1 if structurally valid UTF-8, else 0 +# utf8_encode of cp # codepoint -> UTF-8 byte string ("" if unencodable) +# utf8_from_codepoints of cps # list of codepoints -> UTF-8 string # -# DECODE only — for now. Encoding a codepoint to bytes needs to write bytes -# >= 0x80, and `chr` currently caps at 127 (returns "" above it), so there is no -# way to construct a high byte in pure EigenScript. Filed upstream as #435; once -# `chr` (or a `byte` builtin) can emit 0..255, `utf8_encode` is a few lines. -# Reading raw bytes (`char_at`/`ord`) works, which is why decode/validate do. +# Byte writing comes from `str_from_bytes` (any byte 0..255 as a list) and +# `chr` (a single byte 1..255) — see #435 for the history: encode was +# originally omitted on the mistaken belief no high byte was constructible. # ---- _u8_lead_len: sequence length a lead byte begins (0 = not a lead) ---- define _u8_lead_len(b) as: @@ -141,5 +141,37 @@ define utf8_validate(s) as: i is i + nb return 1 -# utf8_encode is intentionally absent — see the header note and #435 (chr can't -# emit bytes >= 0x80 yet). Decoding is the load-bearing half at this scale. +# ---- utf8_encode: codepoint -> UTF-8 bytes as a string ---- +# Structural like the rest of the module: any codepoint in 1..0x10FFFF encodes +# (surrogates included — utf8_validate is structural and accepts them too). +# Returns "" for anything unencodable: cp > 0x10FFFF, cp < 0, and cp 0 +# (strings are NUL-terminated; NUL-bearing binary belongs in a buffer). +define utf8_encode(cp) as: + if cp < 1: + return "" + if cp > 1114111: + return "" + local bytes is [] + if cp < 128: + append of [bytes, cp] + if (cp >= 128) and (cp < 2048): + append of [bytes, 192 + (bit_shr of [cp, 6])] + append of [bytes, 128 + (bit_and of [cp, 63])] + if (cp >= 2048) and (cp < 65536): + append of [bytes, 224 + (bit_shr of [cp, 12])] + append of [bytes, 128 + (bit_and of [(bit_shr of [cp, 6]), 63])] + append of [bytes, 128 + (bit_and of [cp, 63])] + if cp >= 65536: + append of [bytes, 240 + (bit_shr of [cp, 18])] + append of [bytes, 128 + (bit_and of [(bit_shr of [cp, 12]), 63])] + append of [bytes, 128 + (bit_and of [(bit_shr of [cp, 6]), 63])] + append of [bytes, 128 + (bit_and of [cp, 63])] + return str_from_bytes of bytes + +# ---- utf8_from_codepoints: list of codepoints -> UTF-8 string ---- +# Inverse of utf8_codepoints. Unencodable codepoints contribute "" (dropped). +define utf8_from_codepoints(cps) as: + local out is "" + for c in cps: + out is out + (utf8_encode of c) + return out diff --git a/src/builtins.c b/src/builtins.c index ed34dc0..7b3cc8d 100644 --- a/src/builtins.c +++ b/src/builtins.c @@ -3359,11 +3359,27 @@ Value* builtin_secure_equals(Value *arg) { /* ==== BUILTIN: chr ==== */ /* chr of n → single-character string from ASCII code */ +/* chr of n → one-byte string, the byte-writing inverse of `ord` (which reads + * bytes 0..255). Strings are bytes (#416): chr writes a BYTE, not a Unicode + * codepoint — codepoint→UTF-8 encoding is lib/utf8.eigs `utf8_encode`. + * n must be an integer in 1..255; 0 raises because strings are NUL-terminated + * and cannot hold a NUL byte (keep NUL-bearing binary in a buffer). Anything + * else raises too — the old silent empty-string return for >127 hid every + * high-byte construction bug (#435). */ Value* builtin_chr(Value *arg) { - if (!arg || arg->type != VAL_NUM) return make_str(""); - int code = (int)arg->data.num; - if (code < 0 || code > 127) return make_str(""); - char buf[2] = { (char)code, '\0' }; + if (!arg || arg->type != VAL_NUM) { + runtime_error(0, "chr requires a number"); + return make_null(); + } + double num = arg->data.num; + if (num != (double)(long long)num || num < 1 || num > 255) { + if (num == 0) + runtime_error(0, "chr of 0: strings are NUL-terminated and cannot hold a NUL byte (use a buffer for binary with NULs)"); + else + runtime_error(0, "chr requires an integer byte in 1..255 (got %g); for a codepoint use utf8_encode (lib/utf8.eigs)", num); + return make_null(); + } + char buf[2] = { (char)(int)num, '\0' }; return make_str(buf); } @@ -4817,10 +4833,10 @@ Value* builtin_buf_from_list(Value *arg) { } /* str_from_bytes of → string of those raw bytes. - * Reconstructs a native string from its bytes (the inverse of an `ord` loop), - * which `chr` cannot do for bytes >= 128 (chr treats its arg as a Unicode - * codepoint and emits UTF-8). EigenScript strings are NUL-terminated, so a 0 - * byte ends the string; binary data that may contain NUL must stay in a buffer. + * Reconstructs a native string from its bytes (the inverse of an `ord` loop); + * the list form of scalar `chr` (chr of n == str_from_bytes of [n] for + * 1..255). EigenScript strings are NUL-terminated, so a 0 byte ends the + * string; binary data that may contain NUL must stay in a buffer. * Surfaced by tidelog's CBOR text-string decoder. */ Value* builtin_str_from_bytes(Value *arg) { int n = 0; diff --git a/tests/test_builtin_errors.eigs b/tests/test_builtin_errors.eigs index 174d176..d74c35f 100644 --- a/tests/test_builtin_errors.eigs +++ b/tests/test_builtin_errors.eigs @@ -226,6 +226,47 @@ catch e: c_hex3 is 1 assert of [c_hex3 == 1, "hex non-number raises"] +# ---- chr: byte writer 1..255 — everything else raises loudly (#435) ---- +# The old behavior returned "" silently for >127, hiding every high-byte bug. +assert of [(ord of (chr of 255)) == 255, "chr emits high bytes"] +c_chr1 is 0 +m_chr1 is "" +try: + chr of 256 +catch e: + c_chr1 is 1 + m_chr1 is e +assert of [c_chr1 == 1, "chr above 255 raises"] +assert of [(contains of [m_chr1, "1..255"]) == 1, "chr range message"] +assert of [(contains of [m_chr1, "utf8_encode"]) == 1, "chr message points at utf8_encode"] +c_chr2 is 0 +try: + chr of (0 - 1) +catch e: + c_chr2 is 1 +assert of [c_chr2 == 1, "chr negative raises"] +c_chr3 is 0 +try: + chr of 65.5 +catch e: + c_chr3 is 1 +assert of [c_chr3 == 1, "chr fraction raises (no silent truncation)"] +c_chr4 is 0 +m_chr4 is "" +try: + chr of 0 +catch e: + c_chr4 is 1 + m_chr4 is e +assert of [c_chr4 == 1, "chr of 0 raises (NUL unrepresentable)"] +assert of [(contains of [m_chr4, "NUL"]) == 1, "chr NUL message"] +c_chr5 is 0 +try: + chr of "A" +catch e: + c_chr5 is 1 +assert of [c_chr5 == 1, "chr non-number raises"] + # ---- nearest_in_range: wrong argument shape ---- c12 is 0 m12 is "" diff --git a/tests/test_misc_builtins.eigs b/tests/test_misc_builtins.eigs index 19a96c5..8902b0c 100644 --- a/tests/test_misc_builtins.eigs +++ b/tests/test_misc_builtins.eigs @@ -1,10 +1,15 @@ # Miscellaneous builtin coverage tests load_file of "lib/test.eigs" -# chr — ASCII code to character +# chr — byte value to one-byte string (the writing inverse of ord, #435) assert_eq of [chr of 65, "A", "chr 65 = A"] assert_eq of [chr of 48, "0", "chr 48 = 0"] assert_eq of [chr of 10, "\n", "chr 10 = newline"] +assert_eq of [ord of (chr of 128), 128, "chr 128 round-trips through ord"] +assert_eq of [ord of (chr of 233), 233, "chr 233 round-trips through ord"] +assert_eq of [ord of (chr of 255), 255, "chr 255 round-trips through ord"] +assert_eq of [len of (chr of 233), 1, "chr of a high byte is one byte"] +assert_eq of [chr of 233, str_from_bytes of [233], "chr n == str_from_bytes [n]"] # coalesce — return first non-null/non-empty assert_eq of [coalesce of ["hello", "default"], "hello", "coalesce non-empty"] diff --git a/tests/test_utf8.eigs b/tests/test_utf8.eigs index 1a9180f..ba570b3 100644 --- a/tests/test_utf8.eigs +++ b/tests/test_utf8.eigs @@ -33,6 +33,31 @@ check of ["lone lead byte (truncated)", utf8_validate of (substr of ["€", 0, 1 check of ["lone continuation byte", utf8_validate of (substr of ["€", 1, 1]), 0] check of ["truncated 2-of-3", utf8_validate of (substr of ["€", 0, 2]), 0] +# encoding — the same published vectors, driven backwards (#435) +check of ["encode ascii", utf8_encode of 65, "A"] +check of ["encode 2-byte (é)", utf8_encode of 233, "é"] +check of ["encode 3-byte (€)", utf8_encode of 8364, "€"] +check of ["encode 3-byte (中)", utf8_encode of 20013, "中"] +check of ["encode 4-byte (𐍈)", utf8_encode of 66376, "𐍈"] +# width boundaries: 0x7F/0x80, 0x7FF/0x800, 0xFFFF/0x10000, and the ceiling +check of ["0x7F is 1 byte", len of (utf8_encode of 127), 1] +check of ["0x80 is 2 bytes", len of (utf8_encode of 128), 2] +check of ["0x7FF is 2 bytes", len of (utf8_encode of 2047), 2] +check of ["0x800 is 3 bytes", len of (utf8_encode of 2048), 3] +check of ["0xFFFF is 3 bytes", len of (utf8_encode of 65535), 3] +check of ["0x10000 is 4 bytes", len of (utf8_encode of 65536), 4] +check of ["0x10FFFF is 4 bytes", len of (utf8_encode of 1114111), 4] +check of ["past the ceiling is empty", utf8_encode of 1114112, ""] +check of ["negative is empty", utf8_encode of (0 - 1), ""] +check of ["NUL is empty (strings can't hold it)", utf8_encode of 0, ""] +# every encoded boundary is structurally valid and round-trips +check of ["0x80 validates", utf8_validate of (utf8_encode of 128), 1] +check of ["0x10FFFF validates", utf8_validate of (utf8_encode of 1114111), 1] +check of ["0x10FFFF round-trips", utf8_at of [(utf8_encode of 1114111), 0], 1114111] +# list form: utf8_from_codepoints inverts utf8_codepoints +check of ["from_codepoints rebuilds the string", utf8_from_codepoints of (utf8_codepoints of s), s] +check of ["from_codepoints of empty", utf8_from_codepoints of ([]), ""] + if fails == 0: print of "UTF8_ALL_PASS" if fails > 0: