Skip to content

Conversation

@leodido
Copy link
Contributor

@leodido leodido commented Dec 4, 2025

Description

This PR adds support for yarn packages that use link: dependencies in monorepo setups when building with --frozen-lockfile.

Fixes https://linear.app/ona-team/issue/PDE-180/build-failures-with-frozen-lockfile

Problem: link: dependencies break with --frozen-lockfile

Yarn's link: protocol references local packages via relative paths (e.g., "shared-lib": "link:../shared"). These paths are relative to the source directory, but leeway builds run in an isolated build directory where those paths don't exist.

Before --frozen-lockfile was introduced, yarn install would resolve these broken paths and update the lockfile. After introducing --frozen-lockfile (commit 3bf13c6), yarn refuses to install because the lockfile doesn't match the actual dependency resolution.

Solution

When building a yarn package that has link: dependencies on other leeway yarn packages:

  1. Extract the dependency to _link_deps/<pkg>/ in the build directory
  2. Patch package.json to replace link:../path with file:./_link_deps/<pkg>
  3. Patch yarn.lock to match the patched package.json (required for --frozen-lockfile)

Tarball structures

Leeway produces two types of yarn tarballs depending on the packaging config:

  • YarnLibrary (packaging: library, uses yarn pack):

    package/
    package/package.json
    package/index.js
    
  • YarnApp (packaging: archive, default):

    ./node_modules/<pkg-name>/
    ./node_modules/<pkg-name>/package.json
    ./node_modules/<pkg-name>/...
    

The extraction uses --strip-components with an explicit path filter to handle each case:

  • YarnLibrary: tar -xzf $tarball -C _link_deps/$pkg --strip-components=1 package/
  • YarnApp (non-scoped): tar -xzf $tarball -C _link_deps/$pkg --strip-components=3 ./node_modules/$pkg/
  • YarnApp (scoped, e.g., @scope/pkg): tar -xzf $tarball -C _link_deps/$pkg --strip-components=4 ./node_modules/@scope/$pkg/

Implementation details

  • Added extractNpmPackageNames() to discover npm package names from built tarballs (needed because leeway package names like shared-lib:lib don't match npm names like gitpod-shared)
  • Handle path normalization differences between package.json and yarn.lock (e.g., link:./../shared vs link:../shared)
  • Handle scoped npm packages (e.g., @gitpod/utils) which have an extra path component in the tarball structure

Changes

  • pkg/leeway/build.go: Add link: dependency handling for yarn packages
  • pkg/leeway/build_internal_test.go: Add unit tests for scoped package extraction
  • pkg/leeway/build_integration_test.go: Add integration test

Testing

Added integration test TestYarnPackage_LinkDependencies_Integration that:

  1. Creates a monorepo with shared-lib and app packages
  2. app depends on shared-lib via link:../shared-lib
  3. Verifies the build succeeds with --frozen-lockfile

Added unit test TestYarnAppExtraction_ScopedPackage that verifies:

  1. Non-scoped packages extract correctly with --strip-components=3
  2. Scoped packages (e.g., @test/utils) extract correctly with --strip-components=4

Also tested manually with gitpod-next vscode packages:

leeway build frontend/vscode/remote:package --dont-test -Dversion=0.0.1
leeway build frontend/vscode/desktop:package --dont-test -Dversion=0.0.1

Both builds succeed with this PR.

@leodido leodido self-assigned this Dec 4, 2025
@leodido leodido force-pushed the ld/fix-yarn-link-deps branch 2 times, most recently from a140c5f to 8cb1ccf Compare December 4, 2025 19:50
@leodido leodido requested a review from csweichel December 4, 2025 20:50
@leodido leodido force-pushed the ld/fix-yarn-link-deps branch 2 times, most recently from e7f4f8e to ff1eb1a Compare December 5, 2025 10:17
leodido and others added 5 commits December 5, 2025 10:24
…path

Problem 1: Leeway patches package.json for link: dependencies but wasn't
consistently patching yarn.lock. This caused yarn install to fail with
--frozen-lockfile because the lockfile didn't match package.json.

Problem 2: When extracting YarnApp dependencies to _link_deps/<pkg>/,
the --strip-components=2 was incorrect. It stripped './node_modules/'
but left '<pkg>/' resulting in _link_deps/<pkg>/<pkg>/. Changed to
--strip-components=3 to strip './node_modules/<pkg>/' completely.

Also handle path normalization: package.json may have 'link:./../shared'
while yarn.lock normalizes it to 'link:../shared'.

Co-authored-by: Ona <no-reply@ona.com>
Add integration test TestYarnPackage_LinkDependencies_Integration that
creates a monorepo with link: dependencies and verifies the build
succeeds with --frozen-lockfile.

Add unit tests for extractNpmPackageNames() covering YarnLibrary and
YarnApp tarball formats, scoped packages, and edge cases.

Co-authored-by: Ona <no-reply@ona.com>
Lower the speedup threshold from 1.0 to 0.75 for small package counts
(< 50). With small package counts, batch overhead can occasionally make
it slower than sequential, causing flaky test failures.

Co-authored-by: Ona <no-reply@ona.com>
Add unit test TestYarnAppExtraction_ScopedPackage demonstrating that:
- Non-scoped packages work with --strip-components=3
- Scoped packages fail with --strip-components=3 (the bug)
- Scoped packages work with --strip-components=4 (the fix)

Add integration test TestYarnPackage_ScopedLinkDependencies_Integration
for end-to-end testing of scoped package link: dependencies.

Co-authored-by: Ona <no-reply@ona.com>
Scoped npm packages (e.g., @scope/pkg) have an extra path component
in the tarball structure. The extraction command now uses
--strip-components=4 for scoped packages instead of 3.

Path components:
- Non-scoped: ./node_modules/pkg/ (3 components)
- Scoped: ./node_modules/@scope/pkg/ (4 components)

Co-authored-by: Ona <no-reply@ona.com>
@leodido leodido force-pushed the ld/fix-yarn-link-deps branch from ff1eb1a to 7eed8b0 Compare December 5, 2025 10:26
Copy link
Member

@geropl geropl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✔️ to unblock

@leodido leodido merged commit cb9e4f0 into main Dec 5, 2025
7 checks passed
@leodido leodido deleted the ld/fix-yarn-link-deps branch December 5, 2025 12:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants