Building fails on node v20 when using yarn

[jeffesquivels]

Packages that use node-gyp-build fail to install on node v20 when using yarn in the case where they don't bundle prebuilds for the platform already (i.e. when building is needed).

Using leveldown as an example (but the same happens with sodium-native, rocksdb, node-lmdb, etc):

$ docker run -it --rm -w /root node:20 /bin/bash
root@78779b01eaaf:~# npm_config_build_from_source=true yarn add leveldown
yarn add v1.22.19
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
error /root/node_modules/leveldown: Command failed.
Exit code: 1
Command: node-gyp-build
Arguments: 
Directory: /root/node_modules/leveldown
Output:
node:events:492
      throw er; // Unhandled 'error' event
      ^

Error: spawn node-gyp ENOENT
    at ChildProcess._handle.onexit (node:internal/child_process:286:19)
    at onErrorNT (node:internal/child_process:484:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
Emitted 'error' event on ChildProcess instance at:
    at ChildProcess._handle.onexit (node:internal/child_process:292:12)
    at onErrorNT (node:internal/child_process:484:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'spawn node-gyp',
  path: 'node-gyp',
  spawnargs: [ 'rebuild' ]
}

Node.js v20.5.1
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.
root@78779b01eaaf:~#

Here the npm_config_build_from_source env variable is set to true to emulate the behaviour of --build-from-source on NPM.

When no building is needed, everything works as expected:

$ docker run -it --rm -w /root node:20 /bin/bash
root@c7c921d3c316:~# yarn add leveldown
yarn add v1.22.19
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
success Saved 12 new dependencies.
info Direct dependencies
└─ leveldown@6.1.1
info All dependencies
├─ abstract-leveldown@7.2.0
├─ base64-js@1.5.1
├─ buffer@6.0.3
├─ catering@2.1.1
├─ ieee754@1.2.1
├─ is-buffer@2.0.5
├─ level-concat-iterator@3.1.0
├─ level-supports@2.1.0
├─ leveldown@6.1.1
├─ napi-macros@2.0.0
├─ node-gyp-build@4.6.0
└─ queue-microtask@1.2.3
Done in 3.72s.
root@c7c921d3c316:~#

[jeffesquivels]

Hi again,

I did some more digging on this issue and it seems the problem started with npm/cli@3a7378d that removed node-gyp from npm's bin/ directory (where yarn looks for it when it is not present locally) which eventually made it into Node v20.0.40 via nodejs/node@350fecb.

Now, 94db31e fixed this issue for packages that depends on node-gyp directly (as that local version will be used) but it will still fail for the case where the global node-gyp needs to be used.

A quick and easy solution/workaround would be to use npm exec to execute the global node-gyp installation (which would make sure that the right path is used by yarn, which doesn't happen when invoking it directly) as can be seen here jeffesquivels@f8977d3 (I tested this with both npm and yarn, both on Linux and Windows for Node v20.5.1), but I'm not sure if the maintainers would prefer a different approach to solve this (or if this is considered not an issue).

Thanks!

[mafintosh]

Sounds good to me. That works on all later npm versions?

[vweevers]

where yarn looks for it when it is not present locally

So isn't the real bug that yarn can't find node-gyp anymore? It's a bit strange to use npm to fix a bug in yarn, which is supposed to be a (complete) alternative to npm.

[jeffesquivels]

Sounds good to me. That works on all later npm versions?

I just tested it with the following combinations of node and npm:

• For Node v16.20.2: npm versions 9.6.7 and 9.8.1.
• For Node v18.17.1: npm versions 9.67 and 9.8.1.
• For Node v20.5.1: npm versions 9.8.0 and 9.8.1.

Which is basically the npm version that came bundled with the official docker image of each major version of Node that is still supported, plus the latest npm version (upgrading via npm i npm -g).

I don't know if there's any other combinations of versions that I'm missing and that are also relevant.

[jeffesquivels]

So isn't the real bug that yarn can't find node-gyp anymore?

Yes, that is the real bug indeed and the ideal solution would be to fix it there, but yarn v1 is no longer maintained even though, unfortunately, it's still bundled with most common distribution methods for Node[0][1].

There is one argument in favor of fixing it here, though: when a package is not using node-gyp-build, yarn will detect if the package requires node-gyp and will install if it is not explicitly listed as a dependency which allows the installation to finish successfully. One could argue that node-gyp-build is kind of breaking this logic.

But I guess one could also argue that the proper fix then would be to somehow prevent yarn's logic from breaking (although I have the feeling that that would require a much bigger code change here).

It's a bit strange to use npm to fix a bug in yarn, which is supposed to be a (complete) alternative to npm.

Yes, it's also a bit ironic too 😄.

I'm also happy to implement a different solution if we can come up with one that makes sense to everybody.

[0] For example, this is the latest official docker container for Node v20:

$ docker run -it --rm node:20 /bin/bash
root@f3b700aa176b:/# yarn --version
1.22.19

[1] And this is an installation of yarn using the latest Node LTS:

root@ubuntu-s-2vcpu-4gb-amd-sfo3-01:~# nvm i --lts
Installing latest LTS version.
Downloading and installing node v18.17.1...
Downloading https://nodejs.org/dist/v18.17.1/node-v18.17.1-linux-x64.tar.xz...
####################################################################################################################################################################################### 100.0%
Computing checksum with sha256sum
Checksums matched!
Now using node v18.17.1 (npm v9.6.7)
Creating default alias: default -> lts/* (-> v18.17.1)
root@ubuntu-s-2vcpu-4gb-amd-sfo3-01:~# npm i yarn -g

added 1 package in 951ms
npm notice 
npm notice New minor version of npm available! 9.6.7 -> 9.8.1
npm notice Changelog: https://github.com/npm/cli/releases/tag/v9.8.1
npm notice Run npm install -g npm@9.8.1 to update!
npm notice 
root@ubuntu-s-2vcpu-4gb-amd-sfo3-01:~# yarn --version
1.22.19