This repository was archived by the owner on Feb 25, 2026. It is now read-only.
feat: enable full-scale Nix evaluation in sandboxed environment#40
Open
feat: enable full-scale Nix evaluation in sandboxed environment#40
Conversation
- Refactor nixec crate from monolithic main function to more modular code - Implement complete sandboxed nix-instantiate execution using birdcage - Add public run_nixec() function for testing and external usage - Update birdcage to git version with fixing nix call `unshare(CLONE_FS)` - Add anyhow dependency and improve error handling - Extract hardcoded paths to constants for maintainability - Update build system and development environment dependencies
Inside our sandbox env this works very similarly to pure evaluation without the "copy the world" requirement. Basically, any path outside of the current directory is invisible to the evaluation and eval will abort with an error if you attempt to access them. The only remaining "holes" are impure builtins of nix expressions, such as `builtins.getEnv` which are plugged by atom-nix creating a totally pure evaluation context for much less cost. Currently this change hardcodes allowed uris, since they are required to specify when setting restricted eval. In the future we should allow setting these externally, either via a flag or perhaps an environmental variable.
Adds a new "import.nix" expression used to import this specific version of the larger lock expressions. The import.nix will be read at eka compile time and stored in the binary itself, making it convenient to call into atoms without any implicit nix code, and without making the baked in expression too rigid, since it is versions and well specified in the lock. The fact that we bake this particular expression into the lock can be considered an "implementation detail" to make calling it more straight forward.
Eventually we should fetch this expression at runtime, but this will serve as a good stopgap for boilerplate free execution into atoms for the time being while the API is still unstable.
Some transports cannot persist multiple connections. We want to be able to maximally reuse transports whenever we can while still degrading gracefully when we have a transport implement which cannot. So during fetching, we call the network as usual, but if it fails, we check if the original transport we were handed is not able to persist via the `connection_persists_across_multiple_requests` method, and if not, we try again with a fresh transport. This avoids pointless retries if the connection failed for some other reason while allowing us to handle those transports in gix which do not currently support more than one request. Since we cannot know if our passed transport has already been used beforehand, we attempt the network call to find out first, then check.
Even though git only fetches the objects specified in the target, the server can actually return more refs than requested as an optimization. For this reason, the call to `next()` would return the wrong ref. Instead we use `find` to ensure the ref we return matches the one we actually requested from the server.
Implement bare monorepo cache for git remote atoms, using base58-encoded genesis hashes as remote names. Store atoms under refs/<genesis>/<label>/<version> to avoid conflicts. Add cache population and extraction functions that create temp dirs with proper permissions, symlinks, and executable bits. New 'eka plan' command evaluates atoms via nixec sandbox, supporting local and remote atoms with automatic cache management. Handles git, file, and no storage backends. Technical benefits: - Fast retrieval via git object deduplication (reuses seen objects) - Structured by genesis hash for clear disambiguation - Metadata access (e.g., origin commits in headers) - Never stale thanks to git's CA model and remote ref queries - Enables future verification and multi-URL remote support
- Add RemoteAtomCache trait to storage.rs for unified caching interface - Implement trait for git repository in cache.rs - Remove standalone retieve_atom function, use trait method instead - Update plan command to use new trait interface
update to latest version in the test
In order that we can setup atoms as fast as possible, make the remote calls in a JoinSet, and since gix transports are still blocking, use a blocking thread pool to allow new asyc tasks to continue before completion. Also, remove writing the remote to the cache repositories configuration. It doesn't really gain us anything right now, and causes a race condition.
eka will bootstrap evaluations by reexecing itself as nixec and running the sandboxed evaluations.
- Teach atom materialization to extract and embed the lock expression directly from the lock file's locker entry into the atom, enabling predictable CLI invocation points that are flexible and updatable via the lock file specification - Refactor RemoteAtomCache trait to resolve atoms to cache IDs (including optional locker resolution) and materialize from cache with custom base directories - Implement locker atom resolution in Git cache backend, automatically fetching and injecting atom.nix from locker when absent - Enhance Nix lock expression for CLI compatibility by parameterizing root path and restructuring import scope to eliminate remaining impurities - Update 'eka plan' command to leverage new cache-and-materialize workflow with lock resolution enabled This establishes a robust foundation for the full 'eka plan' implementation, ensuring atoms have well-defined, configurable entry points without rigid assumptions.
Finish initial `eka plan` API and enforce atom calling contracts and a 1-atom-to-1-derivation evaluation model. Introduce global configuration boundary and default platform selection, wiring config into plan evaluation and nix entrypoints. Prepare for json-digest-based cache keys combining config and git revision, and add shared-context evaluation support. Local atom invocation remains TODO, but this establishes the critical user-facing configuration contract.
Add support for evaluating local atoms in the plan command by manually resolving the special "locker" dependency used to evaluate the lockfile itself. This solves the bootstrap problem outside of nix by pulling the locker atom and placing it in the evaluation directory, avoiding implicit nix code execution. This is a temporary hack that symlinks local atoms instead of properly loading them into the git cache (which would require building tree objects). As a downside, it mutates the local directory by leaving the resolved locker expression after execution. A proper cache-based solution will be implemented later. Key changes: - Make AtomDep public and add accessor methods for lockfile evaluation - Refactor RemoteAtomCache trait to remove resolve_lock parameter - Implement WriteLocker trait for lockfile materialization - Update plan command to resolve locker atoms and symlink local packages - Bump nix-lock in eka-root-macro version to 0.3.1 - Update dev environment configuration for new evaluation flow
gix does not expose its internal function for properly matching one partial ref name to the closest matching target, in the same order and logic that git expects, so we have to copy and slighlty modify its code for our purposes for now.
Refactors the nix-lock atom to significantly improve performance and memory efficiency during dependency resolution. Key changes: - Modularizes the monolithic `atom.nix` into specialized components (`closure.nix`, `fetch.nix`, `scope.nix`, etc.) for better maintainability. - Adopts `builtins.genericClosure` to calculate the full dependency closure. This leverages native performance for graph traversal, replacing the previous recursive implementation. - Implements a fixed-point evaluation strategy that ensures every reference to a unique atom shares the same memory address. This guarantees that shared dependencies are evaluated exactly once, preventing redundant re-evaluation of user code across the dependency tree. - Bumps version to 0.4.0.
removes the locker entry entirely in place of a more abstract API. From now on, eka will use the version of nix-lock which corresponds to the format version as a semver major version. In other words, format version 1 should work with any version 1.* of the nix-lock atom. Since we are still unstable, the format version has, for now, been changed to 0 to indicate that we may change at any time. This change also makes the composer act more like a regular dependency (it's placed in the deps list), and only a symbolic reference to that underlying dependency is placed in the lock via the atom id and version to signify to the locking expression which atom to treat specially as our composer.
Implement path_to_cache() in RemoteAtomCache trait to import local directories directly into the atom store. This enables a unified codepath for local development changes without special-casing. Key changes: - Add `ignore` crate for gitignore-aware directory traversal - Build git tree objects recursively from filesystem entries - Generate dev-prefixed semver with tree hash for local atoms - Refactor write_atom_commit for reuse across contexts
Remove Option wrapper from AtomDep.rev field by ensuring all atoms have a revision. Local atoms now import into the store via path_to_cache(), which returns both version and commit. Changes: - AtomDep.rev is now GitDigest instead of Option<GitDigest> - path_to_cache() returns (Version, Atom) tuple - Remove duplicate From impl for ResolvedAtom - Update resolver to use local store import for local deps This eliminates special-casing for local-only dependencies in the lock format, simplifying downstream consumption.
minor bugs were preventing the import from completing sucessfully.
In order to calculate the root properly of local atoms, their path must be canonicalized first, so that gix::discover can locate their parent git repository.
The implementation is almost assuredly not functional just yet, but we have enough of the pieces in place that the resolver at least compiles. This is a good time to commit what we have of it so far before further iteration.
We will expect nix to resolve atoms from the atom store instead of directly from source. Since we will have the full transitive closure in the lockfile after we integrate SAT resolution, we no longer need this field, and it was always meant as a stop gap. In preparation for integrating the full scale resolver, we remove it now.
* add a constructor for `AtomResolver` * store an `AtomSolvableRecord` instead of just version for later use * Transport initialization * log warnings for unknown set tags
- Extend AtomDep with requires/direct fields for dependency graph reconstruction - Add new constructors and accessor methods to AtomDep - Implement ResolutionResult, ResolvedDep, and ResolutionError types - Add resolve() method as main SAT-based resolution entrypoint - Update existing AtomDep::new calls to use new_direct for backwards compatibility
- Modify sanitize() to clear all atom deps from lockfile - SAT resolver rebuilds correct set - Replace synchronize_atoms() with SAT-based resolution using AtomResolver::resolve() - Add From<ResolvedDep> impl for AtomDep conversion - Update convert_deps_to_resolvo to intern transitive deps for solver registration - Enable full transitive resolution: initial tests confirm atoms resolve transitively
- Add owner field to direct dep structs to track which atom owns each dep - Collect nix deps from transitive atom manifests during SAT resolution - Add synchronize_transitive_nix_deps() to resolve and lock directs - Update ResolutionResult to include collected deps for lock integration
- Create test/harness module with MockAtom trait and repo initialization utilities - Add mock_with_deps method for creating atoms with dependencies in tests - Refactor publish/git tests to use shared harness infrastructure - Make AtomWriter and WriteDeps pub(crate) for test access - Move init_tracing and init_repo_and_remote to harness for reuse
- Create sat/test.rs with integration test for A → B → C dependency chain - Test verifies transitive resolution captures all atoms with correct requires chains - Fix Tree entry ordering in harness to ensure consistent git objects - Improve error message in publish test for better debugging
- Introduce RawEkalaSet/RawEkalaManifest for raw deserialization - Move AtomMap validation to RawEkalaSet::resolve_to_atom_map() with filesystem/git tree support - Add EkalaManifest::open_filesystem() for validated manifest construction - Update GitPublisher to validate labels from git tree instead of filesystem - Modify git commands to accept explicit git_dir to avoid directory changes - Preserve label uniqueness invariant while enabling test harness decoupling
Add comprehensive integration tests for the SAT resolver to verify transitive dependency resolution works correctly for linear chains and diamond dependency patterns. Tests added: - test_linear_chain: A→B→C with transitive dep verification - test_diamond_dependency: A→B,C→D with dedup verification Test infrastructure: - Refactor harness for proper git history sync between local/remote - Add commit_workdir helper using recursive tree building - Export cache module helpers (collect_entries, build_tree_recursive) - Update mock_with_deps to use add_uri API with remote_url param Bug fixes discovered during testing: - Fix file:// URL handling in Uri::resolve (was incorrectly excluded) - Fix resolve_from_local path resolution (join relative with ekala root) Also extends add_uri with additional_mirrors parameter for specifying extra mirrors with genesis validation (warn and skip mismatched). Known issue: add_uri doesn't trigger transitive SAT resolution after adding direct deps. Tests work around this with second open_and_resolve.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Gets the nixec crate fully working and evaluating using nix-instantiate in a cross-platform sandbox
unshare(CLONE_FS)