TRACE.md

EigenScript Trace & Replay

EigenScript can record its own execution to a tape and play it back. The tape captures every nondeterministic input a run consumed, so a replayed run produces byte-identical output: the same random sequence, the same monotonic_ns timestamps, the same HTTP responses.

Two environment variables control it:

Variable	Effect
EIGS_TRACE=<path>	Record: open <path> for writing and log line, assignment, and nondet events
EIGS_REPLAY=<path>	Replay: open a previously recorded tape and serve its nondet values to builtins in order

$ EIGS_TRACE=run.tape eigenscript sim.eigs > first.out
$ EIGS_REPLAY=run.tape eigenscript sim.eigs > second.out
$ diff first.out second.out        # identical

Both are off by default. The disabled cost at each hook site is one predicted-not-taken load + branch.

Tape Format

The tape is plain text, one record per line, five record kinds:

Record	Meaning
V <format> <runtime>	Version header — always the first record (e.g. V 2 0.29.0). Stamped once per tape-open; a journal appended across sessions carries one per session. See Format Versioning.
L <line>	Source-line event (from OP_LINE). Adjacent duplicate lines with no A/N between them are deduped — the compiler emits per-statement LINEs and bare repeats are noise.
S <fn> <depth> <serial>	Scope transition (#539 v2): the A records that follow belong to this frame instance — <fn> is the chunk name (<module>, <lambda>, or the function name), <depth> the 0-based frame depth, <serial> a per-thread monotonically increasing frame-instance id stamped at frame push. Emitted lazily with the same dedup discipline as L: only when the frame owning the next assignment differs from the last S, so the byte cost lands at call boundaries that actually assign. Two invocations of the same function carry different serials — their local streams never merge. Skipped on replay; folded by --step.
A <name>=<value>	Assignment delta: a binding changed. Fires at every scope — function locals included — and is scope-qualified by the preceding S record, so a function-local i and the top-level i are separate streams (--step resolves names innermost-first along the reconstructed call chain, with shadowing).
N <fn>=<value>	Nondeterministic builtin return — the replay-determinism substrate.

Value serialization

N records are written with full fidelity so they can be parsed back into real values on replay:

• Numbers, null, and booleans are written verbatim.
• Strings are double-quoted; \", \\, \n, \r are escaped, other control/non-printable bytes become \xNN.
• Lists and dicts are emitted recursively: [1, 2, 3], {"key": value}.
• Buffers get a leading b — b[1,2,3] — to disambiguate from lists.
• Each record has a 64 KiB byte budget. On overflow the record ends with a …<truncated:RESIDUAL> marker so partial records remain visually parseable (truncated records are not replayable; the builtin falls back to its live source).

Recorded Builtins

Every builtin whose return value is nondeterministic from the script's perspective lands on the tape as an N record:

• Random: random, random_int, random_normal, random_hex
• Time: monotonic_ns, monotonic_ms
• Environment / files: env_get, read_text, read_bytes, read_bytes_buf
• Process: args (command-line arguments — differ across invocations, so the recorded list is served on replay regardless of the live argv; #471)
• HTTP extension: http_post (success and all error paths), http_request_body, http_session_id, http_request_headers

The hook is the TRACE_NONDET_RET macro in src/trace.h, used at every nondet return site — adding a new nondet builtin means wrapping its return in the same macro. A builtin that builds its return value (a list, buffer, or dict) before returning uses the TRACE_NONDET_TAKE / TRACE_NONDET_RECORD pair instead (args does): the early TAKE short-circuits under replay before the value is built, so the live construction is neither run nor leaked.

Non-Replayable Builtins (issue #148)

Some nondet builtins are not wrapped, and fail loudly when called under EIGS_REPLAY. They sit on the wrong side of the replay boundary because the tape's recorded return value does not pin down the host-side causal structure the call depends on — re-running the underlying source under replay would re-execute real side effects that the original tape neither captured nor re-creates:

• Subprocess streaming I/O: proc_spawn, proc_write, proc_read_line, proc_read, proc_close, proc_wait. Replaying a recorded fd is meaningless — the child process from the recorded run does not exist; forking a fresh one would change the world a second time.
• Bulk-output exec: exec_capture — same reason. The tape carries the captured stdout, but the child fork would still happen, and its real side effects (writes outside the captured pipe, file changes, network calls) would re-run.
• Concurrent channel receive: recv, try_recv, recv_timeout. Channel ordering depends on the live scheduler — replay against a tape with a different interleaving would deadlock or silently diverge.

These builtins raise a catchable runtime error under EIGS_REPLAY, with the message format "<fn>: not replayable under EIGS_REPLAY (subprocess/concurrency boundary; see docs/TRACE.md)". Programs that need to be replay-safe must guard these call sites or avoid them entirely.

Replay Semantics

With EIGS_REPLAY set, each nondet builtin call takes the next N record from the tape instead of invoking its underlying source. The contract:

• Strict ordering. Records are consumed in tape order. The recorded sequence of nondet calls is the contract.
• Lenient names. If the builtin name doesn't match the record's name, a warning is logged to stderr but the recorded value is used anyway — names are for human-readable debugging. Set EIGS_REPLAY_STRICT=1 to make a mismatch fatal instead: the process reports the divergence and exits with status 3. Use it in harnesses where tape/program drift should fail loudly rather than produce a subtly wrong replay.
• Graceful exhaustion. When the tape runs out, replay switches off and remaining calls hit the real source.
• Unparseable records fall back to the builtin's live source.

All value shapes round-trip: numbers, null, booleans, strings, lists, dicts, and buffers (including nested containers).

Regression coverage: tests/test_replay.sh — each case mutates the underlying source (e.g. rewrites the file read_bytes read) between record and replay to prove the value comes from the tape.

Every Failure is a Tape (--test --trace-on-fail)

Dynamic typing surfaces errors at runtime; deterministic replay converts that weakness into a capability no incumbent ships — every failing test arrives as a byte-identical reproducer.

$ eigenscript --test --trace-on-fail tests/
  FAIL  tests/test_solver.eigs  (exit 1)
        roll: 0.297357521
        replay: EIGS_REPLAY=/tmp/eigen_bNGhgd eigenscript tests/test_solver.eigs

--test --trace-on-fail records each test into its own tape (--trace <path>, the CLI twin of EIGS_TRACE). A passing test discards its tape; a failing one keeps it and prints the exact EIGS_REPLAY=<tape> … invocation. Running that line re-drives the failure with the same recorded nondeterminism — the random draw, the clock read, the file bytes — so a flake reproduces on the first try.

The canonical loop: failure → replay → interrogate. Once you are replaying the exact run, the temporal interrogatives read its history — prev of x, state_at, and the --step tape-stepper (DEBUGGING.md) — so you inspect the trajectory that actually failed, not a fresh one that might not. The stepper reads the tape directly (no replay needed): step forward/back over its L records, reconstruct bindings from its A records, and watch each binding's observer-trajectory label at any point.

Under --json, each result carries a "tape" field for CI to archive as an artifact; the human form prints the replay line.

Same-version enforcement. A tape is a reproducer for the EigenScript version that recorded it, and since #411 the runtime enforces that mechanically: replaying an archived tape after a version bump refuses loudly instead of silently diverging (see Format Versioning). Archiving jobs need only keep the tape — it names its own version on line 1. EIGS_REPLAY_STRICT=1 additionally turns any record/replay name mismatch into a loud abort.

Format Versioning (#411)

The tape is a persisted artifact — CI archives failure tapes, bundles may carry one, embedders journal them — so it names its own provenance. The decision: version-stamped tapes, refuse-on-mismatch. There is no compatibility promise, ever — a tape is valid for exactly the format and runtime version that recorded it, matching the no-backcompat policy everywhere else in the runtime (version-and-reject, never migrate).

• Every tape's first record is V <format> <runtime>. The format integer (TRACE_FORMAT_VERSION in src/trace.h) bumps on any change to the tape encoding; the runtime string is the recording binary's version.
• On replay, a missing header, a malformed (torn) header, a different format version, a different runtime version, an empty tape, or an unopenable EIGS_REPLAY path each refuse loudly — hosted replay exits with status 3 (the replay-divergence status), and eigs_set_replay_tape returns 0 without installing the tape. Replay never falls back to a live run: the user asked for a replay, and a plausible-looking live run is exactly the silent divergence the header exists to prevent.
• Mid-stream V records are legal (a journal appended across sessions carries one per session) but each must match, or replay aborts. The embed seam validates every session header at install time, so a mixed-version journal is refused up front (return 0, the previously installed tape left untouched) rather than aborting the host mid-run; the mid-stream abort therefore only fires for EIGS_REPLAY files.

$ EIGS_REPLAY=old-v0.26.0.tape eigenscript sim.eigs
trace: tape recorded on EigenScript 0.26.0, this binary is 0.27.0 —
refusing to replay; a tape is valid only for the version that recorded
it (docs/TRACE.md)
$ echo $?
3

There is no override flag. The tape is plain text: if you are certain a tape is valid for this binary (say, two dev builds of the same tree), editing line 1 is the override — a deliberate, visible act. The residual honesty gap runs the other way: version equality is necessary, not sufficient. Two different dev builds can share the string dev (or an unreleased version), and the header cannot tell them apart — release boundaries are enforced; dev builds are on their honor.

Regression coverage: the version refuse cases in tests/test_replay.sh plant each mismatch class (format, runtime, missing header, empty file) and require the exit-3 refusal.

Temporal Interrogatives and state_at

prev of x, the at <line> qualifier, and state_at of line query a per-name assignment history (line-stamped, append-only) that is fed by the same assignment hooks. This history is independent of EIGS_TRACE — it is language surface, always on, no tape required. The tape exists for cross-run reproducibility; the history exists for in-run time travel.

• History tracks assignments at every scope, function locals included — exactly the assignments that produce A records when tracing is on. Entries are keyed by name only (no scope qualifier), so state_at merges same-named bindings from different scopes into one stream, and a query can see a local of a function that has already returned.
• Recording is compile-gated: the compiler enables it when the program contains prev of, any at <expr> qualifier, or a reference to state_at (and EIGS_TRACE enables it unconditionally). Programs with no temporal queries pay nothing per assign — profiling showed the previous always-on recording cost roughly a third of a dispatch-heavy workload's runtime. Since a program cannot observe history without containing a query, the gate is invisible — with one edge: code compiled mid-run (eval, REPL) that introduces the first temporal query starts recording at that point, so assigns executed earlier are not visible to it. Aliasing state_at through a dict or eval-built string also hides it from the compiler's scan.
• When the compiled program contains a where/why/how ... at query, each history entry also stamps an observer snapshot (entropy, dH) at assign time, so the observer-derived interrogatives answer historically with exactly what a live query at that moment would have returned. The capture is compile-gated: no such query in the program, no per-assign cost.
• state_at of line walks every tracked name's history backward and returns a dict of each binding's value at or before line.
• Backward queries (at, state_at) are pruned by a periodic line-floor index: each 64-entry segment of a name's history caches its minimum line stamp, so segments that cannot contain a hit are skipped in one compare. Loop-heavy histories — thousands of assigns stamped with the same few lines, the debugger-scrub worst case — resolve in O(history/64) instead of O(history). The index adds one int per 64 history entries and an O(1) min-update per assign.
• Per-assign cost of the history: one cache line + a pointer compare.

Language-level syntax and examples: SYNTAX.md, GRAMMAR.md.

Debugger Step-Back

The graphical debugger (examples/debugger.eigs) offers F8/F11 history navigation while paused. That layer does not read the trace tape: the tape tracks host-VM globals, and the meta-circular interpreter has its own env dict — so the debug hook captures its own (line, env-snapshot) pairs per statement, FIFO-capped at 10 000 steps.