Skip to content

PACKAGE_DESIGN.md

Latest commit

 

History

History
192 lines (160 loc) · 9.17 KB

PACKAGE_DESIGN.md

File metadata and controls

192 lines (160 loc) · 9.17 KB
Raw

EigenScript packages — design proposal

Status: partially implemented — this is the forward-looking design pass; for what actually ships today see PACKAGE_SPEC.md. The core install/lock/verify flow landed (eigenscript --pkg add/install/update/verify: namespaced deps, an eigs.json manifest, a lockfile pinning commit SHAs, and tree-hash verification). This document is the broader design it grew from; the open questions at the bottom (version ranges/solver, a registry/index format, package signing, a dependency-audit command, yank/deprecation policy) are still real decisions, not rhetorical ones. As each part ships, the shipped behavior moves into SPEC.md and becomes subject to the stability contract.

Goals

  1. Use someone else's EigenScript code with a pinned, reproducible version — today the only options are copy-paste or a git submodule you manage by hand.
  2. Reproducible installs: same project + same lockfile = same code, byte for byte, on any machine, offline once fetched.
  3. No code execution at install time. Installing a package must be inert (fetch + checkout + hash-check). No hooks, no build scripts.
  4. No new runtime dependencies. The interpreter stays a single zero-dependency C binary; everything network-ish lives in a tool.
  5. Minimal runtime surface change — the resolver grows one search step and a cache; everything else is tooling and convention.

Non-goals, deliberately: a central registry (git URLs are the namespace), native/C extensions in packages, build steps, and version constraint solving (pin exact versions; a solver can come later if real projects demand ranges).

Current state (as-built, 0.13.0)

  • import name tries lib/name.eigs (the stdlib), then name.eigs, each through the full resolution chain in resolve_eigenscript_file (builtins.c): cwd → $script_dir → $script_dir/.. → $exe_dir/.. → $exe_dir/../lib/eigenscript → ~/.local/lib/eigenscript. Public top-level names bind into a dict named name; _-prefixed names stay private (SPEC.md — Modules).
  • Every import re-executes the module. There is no module cache: two importers get two copies of the module's state, and a diamond (app → a → c, app → b → c) would run c twice with divergent state.
  • Resolution is anchored to the main script's directory (g_script_dir is global). A module imported from another directory resolves its imports relative to the app, not itself — harmless today (modules sit next to the script), wrong for packages.
  • lib/name.eigs is tried before name.eigs, so the stdlib shadows user modules of the same name — but a project-local lib/ directory shadows the installed stdlib (the chain hits cwd/lib/ first). This is how the repo runs its own tests; it's also an existing footgun the package design must not widen.

Design

Vendoring-first, git as transport

Dependencies live in eigs_modules/ at the project root, one directory per package, each a plain checked-out tree of a git repo at a pinned commit. There is no registry: a package is a git URL plus a tag, and the URL is the namespace. Committing eigs_modules/ is supported (Go-vendor-style, true offline builds); the lockfile makes it optional.

Manifest and lockfile

eigs.json at the project root, read only by the tool — the runtime never parses it (resolution works by directory convention, so a missing manifest never breaks import):

{
  "name": "myapp",
  "version": "0.1.0",
  "deps": {
    "vecmath": { "git": "https://github.com/alice/eigs-vecmath", "tag": "v1.2.0" }
  }
}

eigs.lock.json records, per package: the git URL, the resolved commit SHA, and a sha256 over the package's .eigs tree (sorted paths + contents). The commit SHA gives git's integrity; the content hash catches a force-pushed tag or a tampered mirror — same trust posture as the release CHECKSUMS file.

JSON over TOML because lib/json.eigs already exists (the tool stays dependency-free too) — and over an evaluated .eigs manifest because an executable manifest violates goal 3.

A package is just a repo with eigs.json (name, version, its own deps) and <name>.eigs at its root as the entry point. Transitive dependencies are resolved by the tool into the app's flat eigs_modules/ — one version of a name per project; two pins that disagree are an error naming both requirers, not a silent pick.

Runtime change 1: one resolver step

import name gains one step. Proposed order:

  1. lib/name.eigs — stdlib first, unchanged
  2. eigs_modules/name/name.eigs — searched from the importing file's directory upward to the project root (so packages find their dependencies in the app's flat eigs_modules/)
  3. name.eigs script-relative — unchanged

Stdlib-first means a future stdlib module can collide with an existing package name; the tool errors at add time when a dep name matches a stdlib module, and package naming guidance is "prefix it" (alice_vec, not vec). The alternative (packages shadow stdlib) trades that papercut for a supply-chain hole — a dep silently becoming your math — and loses.

Runtime change 2: import becomes cached and module-relative

Two prerequisites that are worth doing even if nothing else ships:

  • Module cache: first import of a resolved real path executes the module; subsequent imports bind the same dict. Diamond deps share one instance of module state. The cache holds counted refs (Value dict + module Env) released at teardown — the closure-cycle collector's ownership rules apply (every edge counted, walker + gc_clear_node updated in lockstep).
  • Per-file resolution base: an import executing inside a module resolves relative paths against that module's directory, not the main script's. g_script_dir becomes a stack (or a parameter threaded through the import path), with load_file keeping its current main-script-relative behavior for back-compat.

Both are observable behavior changes (re-import today re-executes; side-effecting modules can tell) — minor-version territory with CHANGELOG + SPEC.md updates per the stability contract.

The tool: eigenscript --pkg

Written in EigenScript (lib/pkg.eigs + a --pkg dispatcher in main.c next to --fmt/--lint) — dogfooding pressure on the subprocess/string/JSON APIs is a feature. It shells out to git (the one external requirement, tool-only) via the streaming proc_* API.

eigenscript --pkg add <owner>/<name> <git-url> [tag]   # manifest + fetch + lock
eigenscript --pkg install                              # reproduce eigs_modules/ from lockfile
eigenscript --pkg update [<owner>/<name>]              # re-resolve tag → new commit, re-lock
eigenscript --pkg verify                               # re-hash trees against lockfile
eigenscript --pkg list                                 # what's installed, from where

Package identifiers are namespaced <owner>/<name> from day one — bare names like tensor are reserved at the manifest and CLI layers so an early popularity spike can't fragment the namespace. The on-disk layout (eigs_modules/<name>/) and import <name> form stay flat for now: two packages sharing the leaf can't coexist in the same project yet, but disk-level nesting + scoped imports can land later without breaking any existing manifest.

Install is git clone --depth 1 + checkout + hash — nothing from the package is ever executed (goal 3). One caveat to respect: proc_* is an unwrapped replay hole (issue #148), so --pkg runs outside the trace/replay machinery entirely.

Phasing

  • Phase 0 — runtime prerequisites (small, independently valuable): module cache; per-file resolution base; the eigs_modules/ resolver step. Each lands with SPEC.md examples and suite sections.
  • Phase 1 — the tool: --pkg with add/install/verify/list/update, manifest + lockfile, docs page, and a eigs-package-template repo showing layout + semver tagging.
  • Phase 2 — ecosystem: naming/versioning guidance in CONTRIBUTING.md, the awesome-eigenscript index (a list, not a registry), and — once real packages exist — revisit version ranges, and attestation.

Open questions

  1. Manifest format: JSON (proposed) vs TOML (friendlier to hand- editing, needs a parser the project doesn't have).
  2. Stdlib-first precedence (proposed) vs packages-first: accept "new stdlib module may collide with a package name" vs accept "a dep can shadow math". Proposal picks the first.
  3. Flat eigs_modules/ (proposed, one version per name) vs nested per-package trees (npm-style, allows version skew, complicates the resolver and the mental model).
  4. Entry point: <name>.eigs at package root (proposed) vs an explicit "main" field in the package's eigs.json (more flexible, but then the runtime has to read manifests).
  5. Should import ever accept a string path (import "vendor/x.eigs")? Proposal says no — identifiers only; paths stay load_file's job.