feat(diagnostics): runtime-error carets + token-precise LSP ranges (#407 residual)

CHANGELOG.md

@@ -5,6 +5,26 @@ All notable changes to EigenScript are documented here.
 ## [Unreleased]
 
 ### Added
+- **Runtime-error carets + token-precise LSP ranges (#407 residual).**
+  Uncaught runtime errors now print the same one-line source excerpt +
+  `^` caret as parse errors, with the column attributed to the failing
+  token (the `[` of an out-of-range subscript, the operator of a type
+  mismatch, the undefined identifier, the `of` of a bad call). Mechanism:
+  a per-byte `cols[]` table beside the chunk's `lines[]` (stamped by
+  `compile_node`'s save/set/restore position cursor — zero dispatch-loop
+  cost, no OP_LINE/tape change), a refcounted per-unit source blob shared
+  by nested function chunks (so REPL lines, eval'd strings, and imported
+  modules all excerpt their own source), and uncaught-error printing
+  deferred from raise time to the dispatch loop's `CHECK_ERROR`, which
+  knows the failing instruction's offset. A `lines[off] == error_line`
+  guard suppresses the caret whenever the position can't be attributed
+  confidently (e.g. JIT-advanced ip), so a caret never points at the
+  wrong line. The `Error line N:` message and `  at fn (line N)` trace
+  lines are byte-unchanged. LSP diagnostics are now token-precise:
+  E002 (parse) and E003 (undefined name) squiggle exactly the offending
+  token instead of the whole line (0..1000 span remains only as the
+  no-position fallback). `examples/errors/*` pin runtime carets via a
+  new `# expect-caret:` directive; `tests/test_lsp.py` pins the ranges.
 - **The observer pair: value-channel raw-step signals (#422) + trajectory
   snapshots across call boundaries (#421).**
   - *#422*: relative normalization (`Δv/(1+|v|)`) erased exactly two

docs/DIAGNOSTICS.md

@@ -23,17 +23,27 @@ callers, Makefiles, and CI.
 
 ## Stack traces
 
-An uncaught runtime error (or uncaught `throw`) prints the call stack
-to stderr after the error line — every frame between the failure and
-the top level, innermost first:
+An uncaught runtime error (or uncaught `throw`) prints, to stderr: the
+error line, a one-line source excerpt with a `^` caret under the
+offending column (#407 residual — same format as parse errors), then
+the call stack — every frame between the failure and the top level,
+innermost first:
 
 ```
 Error line 6: index 99 out of range (list length 2)
+     6 | v is items[99]
+       |           ^
   at inner (line 6)
   at middle (line 8)
   at <module> (line 9)
 ```
 
+The excerpt block is additive: the `Error line N: ...` message and the
+`  at fn (line N)` trace lines are byte-unchanged, so tools that match
+on them keep working. The caret is suppressed (message + trace only)
+when the failing position can't be attributed confidently — e.g. inside
+JIT-compiled loops or when the unit's source isn't available.
+
 Caught errors print nothing; the handler decides.
 
 ## What `catch` binds
@@ -120,6 +130,8 @@ print of "reached"
 
 $ eigenscript type_err.eigs
 Error line 2: cannot apply '-' to list and num
+     2 | y is x - 5
+       |        ^
   at <module> (line 2)
 $ echo $?
 1
@@ -402,7 +414,10 @@ in the message (see "Runtime Error Types").
 
 The LSP server (`make lsp` → `src/eigenlsp`) surfaces the first syntax
 or parse error of each open document as a
-`textDocument/publishDiagnostics` squiggle at the failing line
-(`tests/test_lsp.py` pins the protocol behavior). For batch static
-checks, `eigenscript --lint file.eigs` runs the linter and
-`eigenscript --fmt` the formatter.
+`textDocument/publishDiagnostics` squiggle (`tests/test_lsp.py` pins the
+protocol behavior). Ranges are token-precise (#407 residual): parse
+errors (E002) and undefined-name errors (E003) squiggle exactly the
+offending token (`start`/`end` from the token's column and length);
+diagnostics without a tracked position fall back to a whole-line range.
+For batch static checks, `eigenscript --lint file.eigs` runs the linter
+and `eigenscript --fmt` the formatter.

docs/SPEC.md

@@ -902,13 +902,16 @@ validation
 -5
 ```
 
-An *uncaught* error prints the error followed by a stack trace —
-every frame between the failure and the top level, innermost first —
-then exits with code 1:
+An *uncaught* error prints the error, a one-line source excerpt with a
+`^` caret under the offending column, and a stack trace — every frame
+between the failure and the top level, innermost first — then exits
+with code 1:
 
 ```eigenscript skip
 # uncaught: stderr shows
 #   Error line 6: index 99 out of range (list length 2)
+#        6 | v is items[99]
+#          |           ^
 #     at inner (line 6)
 #     at middle (line 8)
 #     at <module> (line 9)

examples/errors/calling_a_number.eigs

@@ -1,5 +1,6 @@
 # ERROR DEMO — applying `of` to a non-function.
 # Only fn and builtin values are callable.
-# expect-error: Error line 5: cannot call num
+# expect-error: Error line 6: cannot call num
+# expect-caret: |        ^
 x is 5
 y is x of 10

examples/errors/index_out_of_range.eigs

@@ -1,6 +1,7 @@
 # ERROR DEMO — indexing past the end of a list.
 # Indexing is strictly bounds-checked (negative indices count from the
 # end, but only within -len..len-1).
-# expect-error: Error line 6: index 10 out of range (list length 3)
+# expect-error: Error line 7: index 10 out of range (list length 3)
+# expect-caret: |           ^
 items is [1, 2, 3]
 v is items[10]

examples/errors/string_plus_number.eigs

@@ -2,4 +2,5 @@
 # `+` requires both operands to be the same kind; convert explicitly
 # with `str of n` or `num of s`.
 # expect-error: cannot apply '+' to str and num
+# expect-caret: |                  ^
 msg is "count: " + 3

examples/errors/type_mismatch.eigs

@@ -1,4 +1,5 @@
 # ERROR DEMO — arithmetic on incompatible types.
 # There is no implicit coercion: list - num has no meaning.
-# expect-error: Error line 4: cannot apply '-' to list and num
+# expect-error: Error line 5: cannot apply '-' to list and num
+# expect-caret: |             ^
 x is [1, 2] - 5

examples/errors/undefined_variable.eigs

@@ -1,4 +1,5 @@
 # ERROR DEMO — reading a name that was never assigned.
 # Variables must be assigned before use; there is no implicit null.
-# expect-error: Error line 4: undefined variable 'total'
+# expect-error: Error line 5: undefined variable 'total'
+# expect-caret: |          ^
 print of total

fuzz/fuzz_eigenscript.c

@@ -63,7 +63,7 @@ int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
             /* Execute in a fresh child env so global bindings from one
              * input don't shadow builtins on the next. */
             Env *eval_env = env_new(g_global_env);
-            EigsChunk *chunk = compile_ast(ast, eval_env);
+            EigsChunk *chunk = compile_ast(ast, eval_env, source);
             if (chunk) {
                 Value *result = vm_execute(chunk, eval_env);
                 if (result) val_decref(result);

fuzz/fuzz_stdin.c

@@ -61,7 +61,7 @@ int main(void) {
     if (g_parse_errors == 0) {
         ASTNode *ast = parse(&tl);
         if (g_parse_errors == 0 && ast) {
-            EigsChunk *chunk = compile_ast(ast, global);
+            EigsChunk *chunk = compile_ast(ast, global, buf);
             if (chunk) {
                 Value *result = vm_execute(chunk, global);
                 if (result) val_decref(result);

src/builtins.c

@@ -641,8 +641,14 @@ Value* builtin_throw(Value *arg) {
         g_error_value = arg;
     }
     if (g_try_depth == 0) {
-        fprintf(stderr, "%s\n", g_error_msg);
-        vm_print_stack_trace(stderr);
+        /* #407 residual: defer to CHECK_ERROR for the caret excerpt when
+         * the VM is live — mirrors rt_error. */
+        if (eigs_current && eigs_current->vm && g_vm.frame_count > 0) {
+            g_error_print_pending = 1;
+        } else {
+            fprintf(stderr, "%s\n", g_error_msg);
+            vm_print_stack_trace(stderr);
+        }
     }
     free(msg);
     return make_null();
@@ -2729,7 +2735,7 @@ Value* builtin_load_file(Value *arg) {
     Env *target = g_load_env ? g_load_env : g_global_env;
     int saved_boundary = g_compile_module_boundary;
     g_compile_module_boundary = 1;                       /* #373 */
-    EigsChunk *lf_chunk = compile_ast(ast, target);
+    EigsChunk *lf_chunk = compile_ast(ast, target, source);
     g_compile_module_boundary = saved_boundary;
     if (g_parse_errors > 0) {
         g_parse_errors = saved_errors;
@@ -3683,7 +3689,7 @@ Value* builtin_eval(Value *arg) {
      * diagnostics ('break' outside a loop #337, un-encodable jumps) must
      * fail the eval instead of executing a placeholder chunk. */
     Env *target = g_builtin_call_env ? g_builtin_call_env : g_global_env;
-    EigsChunk *ev_chunk = compile_ast(ast, target);
+    EigsChunk *ev_chunk = compile_ast(ast, target, src);
     if (g_parse_errors > 0) {
         g_parse_errors = saved_errors;
         chunk_free(ev_chunk);

src/chunk.c

@@ -11,6 +11,35 @@
 
 /* ---- Chunk lifecycle ---- */
 
+/* #407: caret-excerpt source blob — see EigsSrcBuf in vm.h. */
+EigsSrcBuf *srcbuf_new(const char *text) {
+    if (!text) return NULL;
+    EigsSrcBuf *sb = xcalloc(1, sizeof(EigsSrcBuf));
+    sb->text = strdup(text);
+    sb->refcount = 1;
+    return sb;
+}
+
+void srcbuf_incref(EigsSrcBuf *sb) {
+    if (!sb) return;
+    if (__builtin_expect(g_vm_multithreaded, 0))
+        __atomic_add_fetch(&sb->refcount, 1, __ATOMIC_RELAXED);
+    else
+        sb->refcount++;
+}
+
+void srcbuf_decref(EigsSrcBuf *sb) {
+    if (!sb) return;
+    int rc;
+    if (__builtin_expect(g_vm_multithreaded, 0))
+        rc = __atomic_sub_fetch(&sb->refcount, 1, __ATOMIC_ACQ_REL);
+    else
+        rc = --sb->refcount;
+    if (rc > 0) return;
+    free(sb->text);
+    free(sb);
+}
+
 EigsChunk *chunk_new(const char *name) {
     EigsChunk *c = xcalloc(1, sizeof(EigsChunk));
     c->code_cap = 256;
@@ -22,6 +51,7 @@ EigsChunk *chunk_new(const char *name) {
     c->env_ic = xcalloc(c->const_cap, sizeof(EnvIC));
     c->lines_cap = 256;
     c->lines = xcalloc(c->lines_cap, sizeof(int));
+    c->cols = xcalloc(c->lines_cap, sizeof(int));
     c->fn_cap = 8;
     c->functions = xcalloc(c->fn_cap, sizeof(EigsChunk *));
     c->name = name ? strdup(name) : strdup("<module>");
@@ -62,6 +92,8 @@ void chunk_decref(EigsChunk *chunk) {
      * it and drops its owned parent ref). */
     env_decref(chunk->env_cache);
     free(chunk->lines);
+    free(chunk->cols);
+    srcbuf_decref(chunk->src);
     for (int i = 0; i < chunk->fn_count; i++)
         chunk_decref(chunk->functions[i]);   /* release creator refs */
     free(chunk->functions);
@@ -91,9 +123,11 @@ void chunk_emit(EigsChunk *chunk, uint8_t byte, int line) {
     if (chunk->lines_len >= chunk->lines_cap) {
         chunk->lines_cap *= 2;
         chunk->lines = realloc(chunk->lines, chunk->lines_cap * sizeof(int));
+        chunk->cols  = realloc(chunk->cols,  chunk->lines_cap * sizeof(int));
     }
     chunk->code[chunk->code_len] = byte;
     chunk->lines[chunk->lines_len] = line;
+    chunk->cols[chunk->lines_len]  = chunk->cur_col;
     chunk->code_len++;
     chunk->lines_len++;
 }

src/compiler.c

@@ -1572,7 +1572,15 @@ static void compile_node(Compiler *c, ASTNode *node) {
         return;
     }
     g_parse_depth++;
+    /* #407: position cursor for the per-byte cols[] table. Set to this
+     * node's column for everything it emits, restore on exit so a parent
+     * resuming after a child recursion stamps its own ops with its own
+     * column (post-order emission would otherwise leave the last child's
+     * column on the parent's opcode). */
+    int saved_col = c->chunk->cur_col;
+    if (node) c->chunk->cur_col = node->col;
     compile_node_inner(c, node);
+    c->chunk->cur_col = saved_col;
     g_parse_depth--;
 }
 
@@ -2000,6 +2008,8 @@ static void compile_node_inner(Compiler *c, ASTNode *node) {
         EigsChunk *fn_chunk = chunk_new(node->data.func.name);
         fn_chunk->param_count = node->data.func.param_count;
         fn_chunk->first_default = node->data.func.first_default;
+        fn_chunk->src = c->chunk->src;           /* #407: share the unit's blob */
+        srcbuf_incref(fn_chunk->src);
 
         Compiler fn_compiler;
         memset(&fn_compiler, 0, sizeof(fn_compiler));
@@ -2090,6 +2100,8 @@ static void compile_node_inner(Compiler *c, ASTNode *node) {
         EigsChunk *fn_chunk = chunk_new("<lambda>");
         fn_chunk->param_count = node->data.lambda.param_count;
         fn_chunk->first_default = node->data.lambda.param_count;  /* lambdas don't support defaults */
+        fn_chunk->src = c->chunk->src;           /* #407: share the unit's blob */
+        srcbuf_incref(fn_chunk->src);
 
         Compiler fn_compiler;
         memset(&fn_compiler, 0, sizeof(fn_compiler));
@@ -2646,8 +2658,12 @@ static void compile_block(Compiler *c, ASTNode **stmts, int count) {
 
 /* ---- Public API ---- */
 
-EigsChunk *compile_ast(ASTNode *ast, Env *env) {
+EigsChunk *compile_ast(ASTNode *ast, Env *env, const char *src) {
     EigsChunk *chunk = chunk_new("<module>");
+    /* #407: retain the unit's source for runtime-error caret excerpts.
+     * Owned copy — callers free their buffers while closures can keep
+     * chunks alive indefinitely. Nested fn chunks share the blob. */
+    chunk->src = srcbuf_new(src);
 
     Compiler compiler;
     memset(&compiler, 0, sizeof(compiler));

src/eigenlsp.c

@@ -740,14 +740,18 @@ static void send_diagnostics(Document *doc) {
         snprintf(full, sizeof(full), "syntax error: %s",
                  g_first_error_msg[0] ? g_first_error_msg : "invalid syntax");
 
+        /* #407 residual: token-precise range when the parser recorded the
+         * offending token's length; the old 0..1000 whole-line span only
+         * remains as the unknown-length fallback. */
+        int err_end = g_first_error_len > 0 ? err_col + g_first_error_len : 1000;
         strbuf_append(&sb, "{\"range\":{\"start\":{\"line\":");
         strbuf_append_fmt(&sb, "%d", err_line);
         strbuf_append(&sb, ",\"character\":");
         strbuf_append_fmt(&sb, "%d", err_col);
         strbuf_append(&sb, "},\"end\":{\"line\":");
         strbuf_append_fmt(&sb, "%d", err_line);
-        strbuf_append(&sb, ",\"character\":1000}},\"severity\":1,\"code\":\"E002\","
-                            "\"source\":\"eigenscript\",\"message\":");
+        strbuf_append_fmt(&sb, ",\"character\":%d}},\"severity\":1,\"code\":\"E002\","
+                            "\"source\":\"eigenscript\",\"message\":", err_end);
         json_escape_to(&sb, full);
         strbuf_append_char(&sb, '}');
     } else if (doc->ast) {
@@ -767,10 +771,16 @@ static void send_diagnostics(Document *doc) {
             if (ln < 0) ln = 0;
             int sev = strcmp(diags[i].severity, "error") == 0 ? 1 : 2;
             if (emitted++ > 0) strbuf_append_char(&sb, ',');
+            /* #407 residual: E-class diags carry the offending token's
+             * col/len (E003 = the undefined identifier) — publish a
+             * token-precise squiggle. len == 0 keeps the whole-line span
+             * (W-rules that haven't been migrated to positions yet). */
+            int dcol = diags[i].len > 0 ? diags[i].col : 0;
+            int dend = diags[i].len > 0 ? diags[i].col + diags[i].len : 1000;
             strbuf_append_fmt(&sb,
-                "{\"range\":{\"start\":{\"line\":%d,\"character\":0},"
-                "\"end\":{\"line\":%d,\"character\":1000}},\"severity\":%d,\"code\":\"%s\","
-                "\"source\":\"eigenscript\",\"message\":", ln, ln, sev, diags[i].code);
+                "{\"range\":{\"start\":{\"line\":%d,\"character\":%d},"
+                "\"end\":{\"line\":%d,\"character\":%d}},\"severity\":%d,\"code\":\"%s\","
+                "\"source\":\"eigenscript\",\"message\":", ln, dcol, ln, dend, sev, diags[i].code);
             json_escape_to(&sb, diags[i].message);
             strbuf_append_char(&sb, '}');
         }

src/eigenscript.c

@@ -47,15 +47,16 @@
  * for consumers that can't see the parser's stderr (the LSP, which turns
  * it into a publishDiagnostics squiggle). Reset at the top of tokenize().
  * g_first_error_line is 1-based and 0 when no error has been recorded. */
-void eigs_record_first_error_at(int line, int col, const char *msg) {
+void eigs_record_first_error_at(int line, int col, int len, const char *msg) {
     if (g_first_error_line) return;   /* keep only the first */
     g_first_error_line = line;
     g_first_error_col = col;
+    g_first_error_len = len;
     snprintf(g_first_error_msg, sizeof(g_first_error_msg), "%s", msg ? msg : "syntax error");
 }
 
 void eigs_record_first_error(int line, const char *msg) {
-    eigs_record_first_error_at(line, 0, msg);
+    eigs_record_first_error_at(line, 0, 0, msg);
 }
 
 /* Structured error payload: set by `throw` so catch can bind the thrown
@@ -116,8 +117,18 @@ void rt_error(ErrKind kind, int line, const char *fmt, ...) {
     g_has_error = 1;
     eigs_clear_error_value();   /* a new error supersedes any thrown value */
     if (g_try_depth == 0) {
-        fprintf(stderr, "%s\n", g_error_msg);
-        vm_print_stack_trace(stderr);
+        /* #407 residual: when the VM is mid-dispatch, defer the uncaught
+         * print to CHECK_ERROR, which knows the failing instruction's
+         * bytecode offset and can add a source excerpt + column caret.
+         * Same print decision, same content prefix — the excerpt is
+         * inserted between the message and the stack trace. Outside
+         * dispatch (embed API, teardown) print immediately as before. */
+        if (eigs_current && eigs_current->vm && g_vm.frame_count > 0) {
+            g_error_print_pending = 1;
+        } else {
+            fprintf(stderr, "%s\n", g_error_msg);
+            vm_print_stack_trace(stderr);
+        }
     }
 }