diff --git a/images/inception.gif b/images/inception.gif new file mode 100644 index 0000000..e0ed32c Binary files /dev/null and b/images/inception.gif differ diff --git a/images/new-horizons-lorri.png b/images/new-horizons-lorri.png new file mode 100644 index 0000000..0fcff64 Binary files /dev/null and b/images/new-horizons-lorri.png differ diff --git a/posts/2020-09-29-lorri-direnv.md b/posts/2020-09-29-lorri-direnv.md new file mode 100644 index 0000000..890671d --- /dev/null +++ b/posts/2020-09-29-lorri-direnv.md @@ -0,0 +1,341 @@ +--- +title: "How direnv and lorri work together" +--- + + +[direnv](https://direnv.net/) is a tool that continuously updates a shell +session's environment variables (update happens on every prompt). +[lorri](https://github.com/target/) is a tool that continuously evaluates a nix +expression and exports a list of environment variables. Using them together +means that a shell session continuously gets environment variables exported +from nix, like a nix-shell. + + + +Here's what we'll talk about: + +* [What does lorri bring to the table?](#what-does-lorri-bring-to-the-table) +* [How lorri works](#how-lorri-works) +* [How direnv works](#how-direnv-works) +* [How lorri and direnv integrate](#how-lorri-and-direnv-integrate) +* [A Standalone .envrc](#a-standalone-.envrc) +* [Final Thoughts](#final-thoughts) + +Let's go! + +## What does lorri bring to the table? + +* **it knows which files to watch**: it triggers a Nix evaluation + [iff](https://en.wikipedia.org/wiki/If_and_only_if) a file + involved in the evaluation was modified. Otherwise, it offers a cached + evaluation; nix-shell, on the other hand, _always_ performs an evaluation. +* **it prebuilds dependencies as soon as possible**: it starts a nix-build in the + background as soon as the files have changed; for instance as soon as the + user switches branches. + +There is a price however: it is very heavyweight (read: bloated) and requires a +daemon running (I go a bit more into details [at the end of this +article](#issues-with-lorri)). First, we'll have a look at the internals of +lorri and direnv. + +New Horizons Satellite + +**DISCLAIMER:** for the sake of brevity I took some liberties when describing +the inner workings of both lorri and direnv. The goal is for you to get a good +enough mental model, _not_ to leave feeling like your read a 500 page +specification! + +### How lorri works + +It simply `nix-build`s a bash script that exports all the environment variables of the +shell, called the "evaluation root". + +
How the ~~sausage~~ evaluation root build is made +

+The `nix-build` is performed on a adaptation of the `shell.nix` found in the +directory. It override the derivation's builder and replaces it with `export > +$out`: + +``` bash +$ export FOO=BAR +$ export baz=( "hello" "world" ) +$ export +declare -x FOO="BAR" # `declare -x` is same as `export` +declare -ax baz=([0]="hello" [1]="world") +``` + +that file -- the evaluation root -- can then be sourced where the env variables +are needed. +

+
+ +The file -- the evaluation root -- is symlinked from `~/.cache/lorri/` +(where there's a 1:1 correspondance between `` and the absolute path of +the project) which serves two purposes. This means that -- after the first +lorri build -- there is always a shell that is ready to go; just `. +~/.cache/lorri/` and all variables will be exported. Added benefit: this +creates a gc root (to avoid nix-shell disappearing on garbage collect) + +
How does it evaluate in the background? +

+ +The local lorri executable, in our case ran with `lorri direnv` (see +[below](#how-lorri-and-direnv-integrate)), starts by adding the $PWD to a +daemon running. The daemon uses `inotify` et al. to watch for file changes and +potentially start an evaluation. Next time `lorri` is run by the user, it +simply queries the daemon for results. + +

+
+ +
How does lorri know which files to track? +

+ +It tracks three kinds of files involved in the nix evaluation: +1. nix files: nix-build is ran ultra verbose (`-vv`) and stderr is parsed (nix + files will show up as `evaluating default.nix...`). +1. string context: nix-build is ran ultra verbose (`-vv`) and stderr is parsed (the + files will show up as `copying my.tar.gz...`). This applies to files copying + to the store with for instance `src = ./.`. +1. imported files: When evaluating the (tweaked) shell, the builtins are + overriden with `scopedImports` to add tracing to `readFile` and `readDir`. + Stderr is then parsed. + +

+
+ +### How direnv works + +Whenever the user hits "Enter" for a new prompt, `direnv` updates the user's +environment variables based the content of a local file `.envrc`. + +
How can direnv run between prompts and how can it update environment variables? +

+ + +It makes use of the `PROMPT_COMMAND` environment variable, which is similar to +`PS1` and run for every prompt. +``` bash +$ direnv hook bash # to be added to your ~/.bashrc + +_direnv_hook() { + eval "$(direnv export bash)"; +}; +if ! [[ "${PROMPT_COMMAND:-}" =~ _direnv_hook ]]; then + PROMPT_COMMAND="_direnv_hook${PROMPT_COMMAND:+;$PROMPT_COMMAND}" +fi +``` +The `_direnv_hook` is the bit that actually updates the environment variables by sourcing `direnv`'s output. + +

+
+ +Here is an example `.envrc` that sets the environment variable `HELLO` and +sources a file `foo` that sets the environment variable `FOO`: + +``` bash +# .envrc +export HELLO=WORLD +. foo +``` + +``` bash +# foo +export FOO=HI +``` + +Running direnv directly will show what variables should be updated: + +``` bash +$ direnv export bash +export HELLO=WORLD;export FOO=HI; +``` + +_NOTE: this typically happens implicitly in between prompts, you wouldn't +actually run `direnv export` by hand._ + +Here's something important: direnv does not actually run `.envrc` on _every_ +prompt. Instead it checks if the file has been modified. + +
Tell me more? +

+ +It's important to note direnv does not keep state files; everything +is stored in environment variables (as a gzipped JSON): + + +``` bash +$ type direnv_watches +direnv_watches is aliased to "direnv show_dump $DIRENV_WATCHES" +$ direnv_watches +[ + { + "Exists": true, + "Modtime": 1598443984, + "Path": "/tmp/tmp.dEfuOLyc5L/.envrc" + } +] +``` + +

+
+ +this means that modifying the file `foo`, imported in `.envrc`, won't trigger a +bash re-run of `.envrc`. Instead, we need to tell direnv to watch the file: + +``` bash +$ eval "$(direnv watch bash ./foo)" +``` + +
But how... +

+ +A `direnv watch` call will actually read the current `DIRENV_WATCHES` variable, and add the new file to be watched + +``` bash +$ direnv watch bash ./foo + +$ eval "$(direnv watch bash ./foo)" +/tmp/tmp.dEfuOLyc5L$ direnv_watches +[ + { + "Exists": true, + "Modtime": 1598443984, + "Path": "/tmp/tmp.dEfuOLyc5L/.envrc" + }, + { + "Exists": true, + "Modtime": 1598444810, + "Path": "/tmp/tmp.dEfuOLyc5L/foo" + } +] +``` + +

+
+ +The smart bit about asking direnv to watch files is that this can be done from +`.envrc`, by stating `watch_file ./foo`. Now whenever `.envrc` or `foo` changes +(the files watched by direnv) then `direnv export bash` will re-run `.envrc` +inside a bash subshell and print the `export FOO=...` statements. + +
watch_file implementation +

+ +Note that `watch_file foo` is simply a shortcut for evaluating `direnv watch bash foo`. +Since this simply updates an environment variable -- `DIRENV_WATCHES` -- the +`.envrc` is the perfect place to do this. + +inception + +

+
+ +### How lorri and direnv integrate + +lorri asks users to add `eval "$(lorri direnv)"` to their `.envrc`: + +``` bash +$ lorri direnv + +EVALUATION_ROOT="~/.cache/lorri/` +watch_file "$EVALUATION_ROOT" # (1) +. $EVALUATION_ROOT # (2) +``` + +There are two important consequences: + +1. This makes sure that whenever the daemon updates the root (`` == path to + shell.nix) , direnv re-evaluates shizzle +2. It imports all the variables from the shell + + +
+ +
How does it know not to override i.e. `$PATH` and `$HOME`? +

+ +The output of `lorri direnv` is actually a tad more complicated. In particular, +it overrides the bash `declare` built-in between (1) and (2) to something like +this: + +``` bash + +function declare() { + if [ "$1" == "-x" ]; then shift; fi + + case "$1" in + "HOME="*) :;; + "USER="*) :;; + "PATH="*) PATH="$1:$PATH":;; + esac +``` + +(Remember that the evaluation root is a bash script containing `declare -x +this; declare -x that`, and that this file is sourced in the `.envrc`) + +

+
+ +### A Standalone .envrc + +Ok, the most annoying part in using lorri is definitely starting the daemon and +keeping it alive. If you want a fully standalone `.envrc` that you can drop in +any project, you'll also need to provide `lorri`. + +Based on the above, we need to provide the following: + +* the `lorri` exeuctable +* a way for `lorri` to start its daemon + +Assuming you have `direnv` installed globally, use the following as an `.envrc`: + +``` bash + +# first get tmux and lorri, and make sure the nix-builds are cached + +MY_TMUX=~/.local/share/lorri-direnv/tmux +if ! [ -d $MY_TMUX ]; then + mkdir -p "$(dirname $MY_TMUX)" + # either make sure that default.nix exports `tmux` or build it from nixpkgs + # instead + nix-build -A tmux --out-link "$MY_TMUX" >/dev/null +fi + +MY_LORRI=~/.local/share/lorri-direnv/lorri +if ! [ -d $MY_LORRI ]; then + mkdir -p "$(dirname $MY_LORRI)" + # either make sure that default.nix exports `lorri` or build it from nixpkgs + # instead + nix-build -A lorri --out-link "$MY_LORRI" >/dev/null +fi + +PATH_add "$MY_LORRI/bin" + +# -d : start as detached +# -A : re-attach if session with name "lorri-background" exists +$MY_TMUX/bin/tmux new-session -d -A -s lorri-background 'lorri daemon' + +eval "$(lorri direnv)" +``` + +This is a self-contained script that will either start a lorri daemon, or +recover a running one, as well as adding the `lorri` executable to the path. +The last bit, `eval "$(lorri direnv)"`, sets up the project for use with the +daemon. + +### Final Thoughts + +TODO address: + +* standalone: we should first check if lorri is running before checking whether the tmux session is running; some users will be running the daemon globally +* lorri: ask the daemon what it's doing (eval, build, idle) and + give user feedback on current state of shell + (https://github.com/target/lorri/issues/365) +* standalone: the user shouldn't forget to close the tmux session (I guess) +* lorri: what about `getEnv`? +* standalone: the lorri and tmux roots will live forever + +* lorri: fails with local files: `keep-env-hack` runs shellHook inside a derivation where files are not accessible +* lorri: all files seem to be watched: https://github.com/target/lorri/issues/364 +* lorri: good idea; crap implementation diff --git a/styles/default.css b/styles/default.css index 6373f0f..1a4a734 100644 --- a/styles/default.css +++ b/styles/default.css @@ -597,3 +597,24 @@ footer { text-align: center; margin: 30px 0; } .portfolio-card-container { padding: 2px 16px; } + +details { + border: 1px solid #aaa; + border-radius: 4px; + padding: .5em .5em 0; +} + +summary { + font-weight: bold; + margin: -.5em -.5em 0; + padding: .5em; +} + +details[open] { + padding: .5em; +} + +details[open] summary { + border-bottom: 1px solid #aaa; + margin-bottom: .5em; +}