Support alternative version pinning mechanism

[dcramer]

NOTE: this issue predates this project's rename to Volta.

We love the idea of Notion (and nvm) as it makes it easier to persist a shared version of Node.js throughout many environments (local, Travis, production [Docker]). However, there's a limitation in Notion that we didn't have with nvm, and that comes from locking node/yarn within package.json itself.

This limitation realistically only comes up when you're doing layer caching, which if you're not familiar with I'm probably not a good person to learn from, but here we go:

We try to install Node and capture it outside of package.json because it means we dont have to rebuild the Node.js layer every time packages change. Dockerfile looks something like:

COPY .nvmrc /usr/src/zeus/
ENV YARN_VERSION 1.7.0
RUN set -x \
  && export NODE_VERSION=$(cat /usr/src/zeus/.nvmrc) \
  [...]

Now this worked actually extremely well. We had a single file we could pull/parse to get the node version everywhere.

When moving to Notion our only real alternative is passing a Docker build arg:

--build-arg NODE_VERSION=$(shell bin/get-node-version | tr -d '\n') \

This is fine and all, but tools like Google's Cloud Build (cloudbuild.yaml) doesn't let you do dynamic arguments like this. That means we're stuck with either rebuilding the Node.js layer every time package.json changes, or hardcoding NODE_VERSION elsewhere.

I'm not proposing a solution here, but wanted to make sure everyones aware of this problem as its a shortcoming with a lot of package management tooling.

[dcramer]

/cc @mitsuhiko who also has been playing with Notion on some of our other projects and might have additional feedback

[charlespierce]

Hi @dcramer, thanks for bringing this issue up! In our design discussions, we have always wanted to avoid adding a notion configuration file to a project, as we didn't want to add yet another required file that everyone who uses the tool would have to check in to source control.

It seems like, for this use case, you would prefer a separate Notion configuration file, so that you can use that as a single source of truth that is distinct from package.json? If that's the case, it seems like a reasonable compromise would be to allow Notion to handle both cases: Configuration inside of package.json as well as configuration in a separate config file. We'll need to hash out the details of precedence and how to handle the notion pin command, but that could provide a possible solution.

Another possible option, if I'm interpreting your concern correctly, would be to have a Docker image that has Notion installed, and then let that Notion install handle fetching node the first time it is activated. Would that work for your situation as an alternative?

[dcramer]

@charlespierce the docker image wont help us (we need base images and then to layer things on top). The main concern is that we want to be able to define a version via a filesystem object that is not in package.json. The reason is that we can hash(that file) to know if Node.js needs updated or not, whereas doing that package.json will suggest Node.js needs updated almost every day based on unrelated dependency changes.

In general I think this is going to be an ongoing industry-wide concern, but I've seen it's often not been approached in a way that works well with existing systems.

The alternative is that whenever we change our Node.js or Yarn versions we have to update multiple locations. This issue is on the goal of trying to accomplish a singular canonical version reference, without compromising build caches (and thus build time/resources).

If Notion decides "package.json is the standard and we dont want to have multiple locations" I think that's totally fair. Again, to me this is a general industry-wide problem that very few package management solutions have resolved. We got someone lucky that nvm was doable in a manner that worked for us, but Notion is easier for developers to work with out of the box.

[chriskrycho]

@dcramer you're not alone in that desire; I know of at least one other team for which that would make a big difference (my old team, which is also making good use of Docker in a not-dissimilar way)!

[chriskrycho]

Related: volta-cli/rfcs#33.

[charlespierce]

FYI With the change to support extends in package.json (#755), this will be sort of supported. While intended for workspaces / monorepos, Volta's logic for extends is flexible, so you could use it like this:

package.json:

{
  ...
  "volta": {
    "extends": "./volta.json"
  }
}

volta.json

{
  "volta": {
    "node": "14.0.0",
    "yarn": "1.22.0"
  }
}

So that changing your Node / npm / Yarn version wouldn't require making a change to package.json, it would keep the same contents. If you also happened to be in a monorepo, you could then set extends in volta.json to point at the workspace root, and that would work as expected.

The main caveat with this is that with how volta pin works, you wouldn't be able to use that as a convenience method and would need to manually update the version in your settings file (volta.json in this example). Running volta pin would result in the version being written into package.json along side the extends directive.

[frangio]

I'm personally interested in this feature so that I can use Volta in my personal local environment without polluting the package.json of my open source projects. So the extends option doesn't work for me.

[charlespierce]

@frangio Understandable! While the approach I outlined is a solution to this issue, I don't think it's the solution since there are a number of problems with the usability.

Can you expand on your use case a little? Are you looking to have a completely separate file that you can add to .gitignore so that it isn't committed to the open source projects? Or are you looking to have that file completely outside of the project folder?

The former is potentially doable, though we'd need to make sure we're on top of performance and not trying to look in too many different places for files. The latter is difficult and I'm not sure really fits into the Volta data model.

Out of curiosity, what is the hesitation regarding adding volta to the package.json for the open source projects?

[frangio]

Are you looking to have a completely separate file that you can add to .gitignore so that it isn't committed to the open source projects?

Yes exactly. The performance concern is very interesting though. Maybe there could be a limitation that this separate file has to be next to a package.json file. Sounds kind of arbitrary but it would limit the effect on performance.

The hesitation is because of a sort of "separation of concerns". I view Volta as a concern of my local setup, and as much as I love it and would recommend it to everyone, other developers may choose nvm (or anything else) and it wouldn't make sense to have duplicate configuration for all of these tools. I would've thought they'd all use the engines field, but I saw there are reasons for Volta not to use that (which I haven't read).

[charlespierce]

Related proposal from Twitter: Support the .nvmrc file directly so that users of Volta and NVM can both work on the same code without changes. There's a few issues we'll need to tackle / design in order to make that work:

• Where to look for the .nvmrc? My preference here would be as a sibling of package.json iff that file exists and there isn't a volta key already. We want to minimize the file I/O operations on the hot path, to make sure that our startup time is quick.
• How to map the values supported by .nvmrc into Volta's VersionSpec type? It appears that nvm supports a number of different keywords that don't necessarily match up with Volta's.
• How to handle range versions? .nvmrc supports a range / semver specifier (e.g. 4.*), how should Volta treat that? This is the biggest hurdle that I can see, because Volta has a fundamentally different model from nvm for how to manage the local cache.
  • We don't want to fetch the Node index and re-resolve the version every time a Node command is run, since that will add unnecessary overhead to every command.
  • At the same time, if we only check the local cache, we go against Volta's philosophy of auto-downloading the necessary versions on-demand and require the user to manually manage their "installed" versions.
  • Even a hybrid model requires the user to be aware of the internal cache, which they shouldn't need to manage directly.
  • Perhaps we only support pure versions in that file, and show a warning if a file would be used but is being skipped because we don't have a specific version.

If we can work out the specifics of how to handle the .nvmrc file, the existing extends logic already supports multiple sources of truth, so it should be possible to slot .nvmrc into that logic. Similarly, the existing extends work also should support a new, separate file (perhaps .voltarc.json?) that can be loaded.