fix(chr): byte-writing inverse of ord (1–255, loud errors); utf8_encode lands (#435)

CHANGELOG.md

@@ -4,7 +4,20 @@ All notable changes to EigenScript are documented here.
 
 ## [Unreleased]
 
+### Changed
+- **`chr` is the byte-writing inverse of `ord`** (#435): `chr of n` now emits
+  the raw byte for any integer 1–255 (was: silent `""` for anything above
+  127). Everything else raises loudly — 0 (strings are NUL-terminated),
+  negatives, fractions (previously truncated silently), values above 255
+  (the error points at `utf8_encode` for codepoints), and non-numbers.
+  `chr of n` == `str_from_bytes of [n]` across the shared range.
+
 ### Added
+- **`utf8_encode` / `utf8_from_codepoints`** in `lib/utf8.eigs` (#435): the
+  encode half of the #416 module — codepoint(s) → UTF-8 byte string, built on
+  `str_from_bytes`. #435's premise was a misdiagnosis: high bytes were already
+  constructible via `str_from_bytes` (#248); encode had been omitted on that
+  mistaken belief.
 - **Parse errors carry `line:col`** (#407, first increment): the lexer already
   tracked columns; now the two parser diagnostics surface them. Human output
   reads `Parse error line 6:9: …`, the `--lint --json` `E002` element gains a

docs/BUILTINS.md

@@ -77,7 +77,7 @@ numeric fast paths used by reassignment and `unobserved` blocks.
 | `scan_int_tokens` | `scan_int_tokens of s` or `scan_int_tokens of [s, comment_marker]` | Token rows `[text, line, col, start, end, is_int, value]` |
 | `trim` | `trim of s` | Strip leading/trailing whitespace |
 | `str_replace` | `str_replace of [s, old, new]` | Replace all occurrences of old with new |
-| `chr` | `chr of code` | Convert ASCII code to single character |
+| `chr` | `chr of byte` | One-byte string from a byte value 1–255 (the writing inverse of `ord`). Raises outside 1–255 — including 0, since strings are NUL-terminated — and on fractions; for a Unicode *codepoint* use `utf8_encode` (lib/utf8.eigs). |
 | `join` | `join of [list, sep]` | Concatenate list elements with separator (C-backed, O(n)) |
 | `text_builder_new` | `text_builder_new of null` | Create a native growable text builder |
 | `text_builder_append` | `text_builder_append of [builder, value]` | Append one value as text |
@@ -143,11 +143,11 @@ Compact typed arrays of doubles with O(1) indexed access. Iterable with
 ### Bytes ↔ values
 
 For serialization: reconstruct strings/floats from raw bytes (the inverse of an
-`ord` loop / manual bit-packing), covering cases `chr` and 32-bit bitwise can't.
+`ord` loop / manual bit-packing), covering cases 32-bit bitwise can't.
 
 | Name | Signature | Description |
 |------|-----------|-------------|
-| `str_from_bytes` | `str_from_bytes of <list\|buffer>` | Build a string from raw byte values (0–255). Unlike `chr` (which emits the UTF-8 of a *codepoint*), this writes the bytes verbatim, so it inverts an `ord`-over-bytes loop for any byte. Strings are NUL-terminated: a `0` byte ends the string — keep NUL-bearing binary in a buffer. |
+| `str_from_bytes` | `str_from_bytes of <list\|buffer>` | Build a string from raw byte values (0–255) — the list form of `chr` (`chr of n` == `str_from_bytes of [n]` for 1–255), inverting an `ord`-over-bytes loop. Strings are NUL-terminated: a `0` byte ends the string — keep NUL-bearing binary in a buffer. |
 | `f64_to_bytes` | `f64_to_bytes of x` | List of 8 ints: the big-endian IEEE-754 encoding of double `x` (network byte order, portable across host endianness). |
 | `f64_from_bytes` | `f64_from_bytes of <list\|buffer>` | Decode a double from the first 8 big-endian IEEE-754 bytes. Inverse of `f64_to_bytes`. |
 

docs/STDLIB.md

@@ -435,9 +435,8 @@ semantics when you need them, decoding UTF-8 over the byte primitives.
 | `utf8_at` | `utf8_at of [s, i]` | i-th codepoint (0-indexed), or -1 |
 | `utf8_char_at` | `utf8_char_at of [s, i]` | i-th character as a (multi-byte) string, or "" |
 | `utf8_validate` | `utf8_validate of s` | 1 if structurally valid UTF-8, else 0 |
-
-Decode-only for now: encoding needs to write bytes ≥ 0x80, which `chr` cannot
-yet do (filed as #435). Reading raw bytes works, so decode/validate do.
+| `utf8_encode` | `utf8_encode of cp` | Codepoint → UTF-8 byte string ("" if unencodable) |
+| `utf8_from_codepoints` | `utf8_from_codepoints of cps` | List of codepoints → UTF-8 string (inverse of `utf8_codepoints`) |
 
 ### lib/pkg.eigs — Package Manager Runtime
 

lib/utf8.eigs

@@ -15,12 +15,12 @@
 #   utf8_at of [s, i]             # i-th codepoint (0-indexed), or -1 if out of range
 #   utf8_char_at of [s, i]        # i-th character as a (multi-byte) string, or ""
 #   utf8_validate of s            # 1 if structurally valid UTF-8, else 0
+#   utf8_encode of cp             # codepoint -> UTF-8 byte string ("" if unencodable)
+#   utf8_from_codepoints of cps   # list of codepoints -> UTF-8 string
 #
-# DECODE only — for now. Encoding a codepoint to bytes needs to write bytes
-# >= 0x80, and `chr` currently caps at 127 (returns "" above it), so there is no
-# way to construct a high byte in pure EigenScript. Filed upstream as #435; once
-# `chr` (or a `byte` builtin) can emit 0..255, `utf8_encode` is a few lines.
-# Reading raw bytes (`char_at`/`ord`) works, which is why decode/validate do.
+# Byte writing comes from `str_from_bytes` (any byte 0..255 as a list) and
+# `chr` (a single byte 1..255) — see #435 for the history: encode was
+# originally omitted on the mistaken belief no high byte was constructible.
 
 # ---- _u8_lead_len: sequence length a lead byte begins (0 = not a lead) ----
 define _u8_lead_len(b) as:
@@ -141,5 +141,37 @@ define utf8_validate(s) as:
         i is i + nb
     return 1
 
-# utf8_encode is intentionally absent — see the header note and #435 (chr can't
-# emit bytes >= 0x80 yet). Decoding is the load-bearing half at this scale.
+# ---- utf8_encode: codepoint -> UTF-8 bytes as a string ----
+# Structural like the rest of the module: any codepoint in 1..0x10FFFF encodes
+# (surrogates included — utf8_validate is structural and accepts them too).
+# Returns "" for anything unencodable: cp > 0x10FFFF, cp < 0, and cp 0
+# (strings are NUL-terminated; NUL-bearing binary belongs in a buffer).
+define utf8_encode(cp) as:
+    if cp < 1:
+        return ""
+    if cp > 1114111:
+        return ""
+    local bytes is []
+    if cp < 128:
+        append of [bytes, cp]
+    if (cp >= 128) and (cp < 2048):
+        append of [bytes, 192 + (bit_shr of [cp, 6])]
+        append of [bytes, 128 + (bit_and of [cp, 63])]
+    if (cp >= 2048) and (cp < 65536):
+        append of [bytes, 224 + (bit_shr of [cp, 12])]
+        append of [bytes, 128 + (bit_and of [(bit_shr of [cp, 6]), 63])]
+        append of [bytes, 128 + (bit_and of [cp, 63])]
+    if cp >= 65536:
+        append of [bytes, 240 + (bit_shr of [cp, 18])]
+        append of [bytes, 128 + (bit_and of [(bit_shr of [cp, 12]), 63])]
+        append of [bytes, 128 + (bit_and of [(bit_shr of [cp, 6]), 63])]
+        append of [bytes, 128 + (bit_and of [cp, 63])]
+    return str_from_bytes of bytes
+
+# ---- utf8_from_codepoints: list of codepoints -> UTF-8 string ----
+# Inverse of utf8_codepoints. Unencodable codepoints contribute "" (dropped).
+define utf8_from_codepoints(cps) as:
+    local out is ""
+    for c in cps:
+        out is out + (utf8_encode of c)
+    return out

src/builtins.c

@@ -3359,11 +3359,27 @@ Value* builtin_secure_equals(Value *arg) {
 
 /* ==== BUILTIN: chr ==== */
 /* chr of n → single-character string from ASCII code */
+/* chr of n → one-byte string, the byte-writing inverse of `ord` (which reads
+ * bytes 0..255). Strings are bytes (#416): chr writes a BYTE, not a Unicode
+ * codepoint — codepoint→UTF-8 encoding is lib/utf8.eigs `utf8_encode`.
+ * n must be an integer in 1..255; 0 raises because strings are NUL-terminated
+ * and cannot hold a NUL byte (keep NUL-bearing binary in a buffer). Anything
+ * else raises too — the old silent empty-string return for >127 hid every
+ * high-byte construction bug (#435). */
 Value* builtin_chr(Value *arg) {
-    if (!arg || arg->type != VAL_NUM) return make_str("");
-    int code = (int)arg->data.num;
-    if (code < 0 || code > 127) return make_str("");
-    char buf[2] = { (char)code, '\0' };
+    if (!arg || arg->type != VAL_NUM) {
+        runtime_error(0, "chr requires a number");
+        return make_null();
+    }
+    double num = arg->data.num;
+    if (num != (double)(long long)num || num < 1 || num > 255) {
+        if (num == 0)
+            runtime_error(0, "chr of 0: strings are NUL-terminated and cannot hold a NUL byte (use a buffer for binary with NULs)");
+        else
+            runtime_error(0, "chr requires an integer byte in 1..255 (got %g); for a codepoint use utf8_encode (lib/utf8.eigs)", num);
+        return make_null();
+    }
+    char buf[2] = { (char)(int)num, '\0' };
     return make_str(buf);
 }
 
@@ -4817,10 +4833,10 @@ Value* builtin_buf_from_list(Value *arg) {
 }
 
 /* str_from_bytes of <list|buffer of byte ints> → string of those raw bytes.
- * Reconstructs a native string from its bytes (the inverse of an `ord` loop),
- * which `chr` cannot do for bytes >= 128 (chr treats its arg as a Unicode
- * codepoint and emits UTF-8). EigenScript strings are NUL-terminated, so a 0
- * byte ends the string; binary data that may contain NUL must stay in a buffer.
+ * Reconstructs a native string from its bytes (the inverse of an `ord` loop);
+ * the list form of scalar `chr` (chr of n == str_from_bytes of [n] for
+ * 1..255). EigenScript strings are NUL-terminated, so a 0 byte ends the
+ * string; binary data that may contain NUL must stay in a buffer.
  * Surfaced by tidelog's CBOR text-string decoder. */
 Value* builtin_str_from_bytes(Value *arg) {
     int n = 0;

tests/test_builtin_errors.eigs

@@ -226,6 +226,47 @@ catch e:
     c_hex3 is 1
 assert of [c_hex3 == 1, "hex non-number raises"]
 
+# ---- chr: byte writer 1..255 — everything else raises loudly (#435) ----
+# The old behavior returned "" silently for >127, hiding every high-byte bug.
+assert of [(ord of (chr of 255)) == 255, "chr emits high bytes"]
+c_chr1 is 0
+m_chr1 is ""
+try:
+    chr of 256
+catch e:
+    c_chr1 is 1
+    m_chr1 is e
+assert of [c_chr1 == 1, "chr above 255 raises"]
+assert of [(contains of [m_chr1, "1..255"]) == 1, "chr range message"]
+assert of [(contains of [m_chr1, "utf8_encode"]) == 1, "chr message points at utf8_encode"]
+c_chr2 is 0
+try:
+    chr of (0 - 1)
+catch e:
+    c_chr2 is 1
+assert of [c_chr2 == 1, "chr negative raises"]
+c_chr3 is 0
+try:
+    chr of 65.5
+catch e:
+    c_chr3 is 1
+assert of [c_chr3 == 1, "chr fraction raises (no silent truncation)"]
+c_chr4 is 0
+m_chr4 is ""
+try:
+    chr of 0
+catch e:
+    c_chr4 is 1
+    m_chr4 is e
+assert of [c_chr4 == 1, "chr of 0 raises (NUL unrepresentable)"]
+assert of [(contains of [m_chr4, "NUL"]) == 1, "chr NUL message"]
+c_chr5 is 0
+try:
+    chr of "A"
+catch e:
+    c_chr5 is 1
+assert of [c_chr5 == 1, "chr non-number raises"]
+
 # ---- nearest_in_range: wrong argument shape ----
 c12 is 0
 m12 is ""

tests/test_misc_builtins.eigs

@@ -1,10 +1,15 @@
 # Miscellaneous builtin coverage tests
 load_file of "lib/test.eigs"
 
-# chr — ASCII code to character
+# chr — byte value to one-byte string (the writing inverse of ord, #435)
 assert_eq of [chr of 65, "A", "chr 65 = A"]
 assert_eq of [chr of 48, "0", "chr 48 = 0"]
 assert_eq of [chr of 10, "\n", "chr 10 = newline"]
+assert_eq of [ord of (chr of 128), 128, "chr 128 round-trips through ord"]
+assert_eq of [ord of (chr of 233), 233, "chr 233 round-trips through ord"]
+assert_eq of [ord of (chr of 255), 255, "chr 255 round-trips through ord"]
+assert_eq of [len of (chr of 233), 1, "chr of a high byte is one byte"]
+assert_eq of [chr of 233, str_from_bytes of [233], "chr n == str_from_bytes [n]"]
 
 # coalesce — return first non-null/non-empty
 assert_eq of [coalesce of ["hello", "default"], "hello", "coalesce non-empty"]

tests/test_utf8.eigs

@@ -33,6 +33,31 @@ check of ["lone lead byte (truncated)", utf8_validate of (substr of ["€", 0, 1
 check of ["lone continuation byte", utf8_validate of (substr of ["€", 1, 1]), 0]
 check of ["truncated 2-of-3", utf8_validate of (substr of ["€", 0, 2]), 0]
 
+# encoding — the same published vectors, driven backwards (#435)
+check of ["encode ascii", utf8_encode of 65, "A"]
+check of ["encode 2-byte (é)", utf8_encode of 233, "é"]
+check of ["encode 3-byte (€)", utf8_encode of 8364, "€"]
+check of ["encode 3-byte (中)", utf8_encode of 20013, "中"]
+check of ["encode 4-byte (𐍈)", utf8_encode of 66376, "𐍈"]
+# width boundaries: 0x7F/0x80, 0x7FF/0x800, 0xFFFF/0x10000, and the ceiling
+check of ["0x7F is 1 byte", len of (utf8_encode of 127), 1]
+check of ["0x80 is 2 bytes", len of (utf8_encode of 128), 2]
+check of ["0x7FF is 2 bytes", len of (utf8_encode of 2047), 2]
+check of ["0x800 is 3 bytes", len of (utf8_encode of 2048), 3]
+check of ["0xFFFF is 3 bytes", len of (utf8_encode of 65535), 3]
+check of ["0x10000 is 4 bytes", len of (utf8_encode of 65536), 4]
+check of ["0x10FFFF is 4 bytes", len of (utf8_encode of 1114111), 4]
+check of ["past the ceiling is empty", utf8_encode of 1114112, ""]
+check of ["negative is empty", utf8_encode of (0 - 1), ""]
+check of ["NUL is empty (strings can't hold it)", utf8_encode of 0, ""]
+# every encoded boundary is structurally valid and round-trips
+check of ["0x80 validates", utf8_validate of (utf8_encode of 128), 1]
+check of ["0x10FFFF validates", utf8_validate of (utf8_encode of 1114111), 1]
+check of ["0x10FFFF round-trips", utf8_at of [(utf8_encode of 1114111), 0], 1114111]
+# list form: utf8_from_codepoints inverts utf8_codepoints
+check of ["from_codepoints rebuilds the string", utf8_from_codepoints of (utf8_codepoints of s), s]
+check of ["from_codepoints of empty", utf8_from_codepoints of ([]), ""]
+
 if fails == 0:
     print of "UTF8_ALL_PASS"
 if fails > 0: