Settle the Unicode/text position: bytes-forever + lib/utf8.eigs (#416)#436
Merged
Conversation
…gs (#416) Retrofitting text semantics post-1.0 is the most expensive migration in language history, so this settles it deliberately. `str` is a byte string BY DEFINITION — now official in SPEC.md's new "Text" subsection (byte indexing, multibyte-safe concat/f-strings, byte-checked examples). Native UTF-8-by-construction is deliberately NOT adopted: it would ripple through the VM, JIT, AOT, and every tool for a bill this scale doesn't need. Character semantics, when you need them, come from the new lib/utf8.eigs — utf8_len / utf8_codepoints / utf8_at / utf8_char_at / utf8_validate, decoding UTF-8 over the byte primitives (char_at/ord). Tested against published vectors (U+00E9 2-byte, U+20AC 3-byte, U+10348 4-byte) and structural-validation cases built by slicing real multibyte chars; suite [50k]. Building it surfaced a real language gap, filed rather than worked around: `chr` returns "" for code > 127, so there is no way to construct a byte >= 0x80 in EigenScript — `utf8_encode` (and byte-output generally) is impossible until that lands. Filed #435; encode is omitted with a pointer, decode is the load-bearing half at this scale. Suite 2512/2512; ASan clean. Closes #416 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jul 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Retrofitting text semantics post-1.0 is the most expensive migration in language history — so this settles it deliberately, pre-1.0.
stris a byte string by definition — now official in SPEC.md's new "Text" subsection:len/char_at/[]/slicing index bytes, and UTF-8 source literals round-trip exactly through concat/f-strings/printing (byte-checked examples). Native UTF-8-by-construction is deliberately not adopted: it would ripple through the VM, JIT, AOT, and every tool for a bill this scale doesn't need.Character semantics come from the new
lib/utf8.eigs—utf8_len/utf8_codepoints/utf8_at/utf8_char_at/utf8_validate, decoding UTF-8 over the byte primitives. Tested against published vectors (U+00E9 2-byte, U+20AC 3-byte, U+10348 4-byte) and validation cases built by slicing real multibyte chars into truncated/lone-continuation sequences. Suite [50k], 16 checks.Forcing function surfaced a real gap (#435)
Building the module hit a wall:
chrreturns""for code > 127, so there is no way to construct a byte ≥ 0x80 in EigenScript.utf8_encode(and byte-output generally) is impossible until that's fixed. Filed as #435 (makechremit 0–255, symmetric withord) rather than worked around; encode is omitted with a pointer, decode is the load-bearing half.Gates: suite 2512/2512; ASan clean; SPEC + STDLIB doc-drift green.
Closes #416
🤖 Generated with Claude Code