Add Backpack support to Stack

[philippedev101]

Addresses #2540.

This PR adds support for GHC's Backpack module system, the feature request that has been open since August 2016. After this change, Stack can build projects that use signatures, mixins, and cross-package mixin linking, which previously only worked under cabal-install.

Background

Backpack lets you write a library against an abstract interface (a signature) and have the consumer decide which concrete implementation to plug in. The compiler recompiles the library for each implementation, so there's no runtime overhead. When the signature and the implementation live in the same package (using sub-libraries), this is "private Backpack" and has always worked in Stack. The hard part is cross-package Backpack, where the signature lives in one package and the implementation in another. This requires the build tool to create extra instantiation build steps, which Stack didn't do before.

What changed

The work breaks down into three layers:

Per-component build plan. The build plan is now keyed by ComponentKey (package name + component) instead of just package name. This was a prerequisite: Backpack instantiation tasks need their own entries in the plan alongside the regular library task for the same package. This also lets Stack detect intra-package sub-library dependencies (the Backpack pattern) and avoid splitting those packages into independent component tasks, which would break the build order that Cabal expects.

Cross-package instantiation. When a consumer package uses mixins to depend on an indefinite package, Stack now scans the consumer's dependencies, resolves which modules fill which signatures, and creates CInst (component instantiation) tasks. Each CInst task runs Setup configure --instantiate-with=Sig=impl-pkg:Module followed by Setup build in its own inst-<hash> build directory. The hash is derived from the signature-to-implementation mapping, so different instantiations of the same package get different build artifacts.

This handles the full range of Backpack patterns: default renaming (name identity), explicit ModuleRenaming, HidingRenaming (propagating unfilled holes), multiple instantiations of the same indefinite package with different implementations, transitive chains where an indefinite package depends on another indefinite package, sub-library signatures and implementations, and indefinite packages from Hackage or snapshots (not just local packages). CInst tasks also get their own config cache entries, precompiled cache support, and haddock generation.

Documentation and changelog. A new topic page at doc/topics/backpack.md introduces what Backpack is, walks through its features (signatures, mixin linking, renaming, multiple instantiations, sub-libraries, reexported modules, Template Haskell restrictions), and then explains how Stack supports it, including what the build output looks like and what the limitations are. The page is linked from mkdocs.yml under Topics, and cross-referenced from the package description tutorial (mentioning the signatures field) and the multi-package projects tutorial (mentioning Backpack as a use case for multi-package setups). A major changes entry has been added to ChangeLog.md under the unreleased section.

Testing

103 unit tests in ConstructPlanSpec cover the instantiation logic: signature resolution, renaming, hiding, deduplication, transitive chains, multiple instantiations, sub-library mixins, ADRFound (installed) packages, config cache round-trips, and various warning/error paths.

8 integration tests build real multi-package Backpack projects end-to-end: private Backpack, sub-library dependencies, cross-package with default renaming, explicit renaming, sub-library mixins, transitive chains, and multiple instantiations.

The full existing test suite (814 tests) continues to pass. The two pre-existing Stack.Config/loadConfig failures are environmental (TLS certificates) and unrelated.

What's not covered

requires hiding that actually hides signatures does not create a partial instantiation. Cabal requires a closing substitution, so the hidden signatures propagate as holes to the consumer. Template Haskell in indefinite packages doesn't work, but that's a GHC limitation that affects cabal-install equally.

[mpilgrem]

@philippedev101, thanks! There is a lot to read but, in advance of that, in implementing component-based builds, did you encounter, and overcome, the performance issue that was blocking @theobat making progress at:

• Component-based builds #6356
• Component-based builds #6356 (comment)

[philippedev101]

@mpilgrem Thanks for the pointer to #6356 and @theobat's work. After going through the discussion there and the branches on their fork in some detail: short answer, this PR takes a different architectural approach that sidesteps the performance problem, at the cost of not solving the broader set of issues that full per-component builds would address.

How the two approaches differ

@theobat's approach was to make Stack build each component of a package as a separate Setup configure / Setup build cycle. So for a package with a library and an executable, Stack would call the Setup process twice instead of once. This is what cabal-install does and it's the "correct" long-term direction, but as @theobat found, it introduces a 30-40% overhead on integration tests because each subprocess invocation has a fixed cost that adds up.

The approach in this PR is narrower. The build plan is keyed per-component using a ComponentKey type (package name + component), but the actual Setup invocations for regular packages are unchanged: one configure, one build, same as before. The only additional subprocess calls happen for Backpack instantiation tasks (CInst), and those only exist when a project actually uses cross-package Backpack. Projects that don't use Backpack see zero additional subprocess invocations and no performance difference.

What this does not solve

Full per-component execution would address several long-standing issues that this approach leaves untouched:

Unnecessary recompilation when switching between stack build and stack test (#2800). Right now, if you build just the library and then run tests, Stack reconfigures the entire package with test dependencies included. This causes the library to be unregistered and recompiled from scratch even though nothing in it changed. Per-component execution would configure and build the test suite separately, leaving the library alone.

Rebuilding components that aren't dependencies of the target (#6569). If you ask for stack build foo:exe:my-exe but a sub-library that the exe doesn't depend on has changed, Stack still rebuilds because it configures and builds the whole package. Per-component execution would only touch the components in the actual dependency chain.

Parallel builds within a package (#4391). A package with five independent executables currently builds them sequentially (Cabal handles the ordering internally). Per-component execution would let Stack schedule them in parallel, the same way it parallelizes across packages.

Component-level cycle resolution (#2583). When package A's library depends on package B's library, and B's test suite depends on A's library, that looks like a cycle at the package level. At the component level it's not: build A:lib, then B:lib, then both test suites. Per-component execution could handle this.

Where this PR sits

There are roughly three points on the spectrum:

1. No Backpack support at all (current Stack).
2. Backpack support that works correctly with no performance penalty for non-Backpack projects, but doesn't tackle the broader per-component issues (this PR).
3. Full per-component builds that solve all of the above plus Backpack, but with the subprocess overhead problem that needs to be addressed first.

This PR lands at point 2. It gets cross-package Backpack working today without regressing anything for existing users.

Transitioning to full per-component builds later

The ComponentKey plan infrastructure that this PR introduces is a step toward point 3. The plan is already split per-component, dependency edges between components within the same package are already tracked (intraPackageDeps), and the action scheduler already generates one action per component key. The remaining work to reach full per-component execution would be:

• Per-component Setup configure: thread the component name from ComponentKey into the Setup configure call so it targets a single component. Cabal has supported this since 2.0 and the types are already in place.
• Per-component config cache: key the config cache by ComponentKey instead of PackageIdentifier, so that configuring the test suite doesn't invalidate the library's cache. This PR already does this for CInst tasks (ConfigCacheTypeInstantiation), the same pattern extends to regular components.
• Per-component dependency flags: scope the --dependency flags passed to Setup configure to just the dependencies of the component being built, rather than the union of all component dependencies.
• Per-component Task creation: currently multiple ComponentKey entries in the plan share the same Task object. For true per-component execution, each component would have its own Task with its own dependency sets.

None of these changes require rethinking the plan-level architecture. They're about threading component information through the execution layer. And they're independent of the subprocess performance question, which could be tackled separately (the in-process Distribution.Simple.defaultMainArgs approach discussed in #6356 looks promising for build-type: Simple packages).

[mpilgrem]

@philippedev101, the failing STAN CI can wait, but the integration tests are failing on all operating systems for a reason that I can reproduce locally (on Windows 11), namely:

stack test does not work with this Stack executable in Stack's own project directory. It does not build the test suite (only the library) and so fails with:

TestSuiteExeMissing False "stack-unit-test.exe" "stack" "stack-unit-test"
Completed 2 action(s).

Error: [S-7282]
       Stack failed to execute the build plan.

       While executing the build plan, Stack encountered the error:

       Error: [S-1995]
       Test suite failure for package stack-3.10.0
           stack-unit-test:  executable not found

(Note to self: the poor output of the TestSuiteExeMissing in the log is an existing bug that needs to be fixed.)

However, if stack test is then commanded a second time, the library and the test suite are built.

(If stack test in then commanded a third time, the library (only) is built again - when it should be a no-op.)

I've changed the PR status to 'draft', in the interim.

[mpilgrem]

A minimal reproducer of the problem is a simple one-package project that has a custom-setup:. For example:

> stack new test6865
> cd test6865
> # Edit package.yaml to introduce a custom-setup (see below)
> stack test
...
TestSuiteExeMissing False "test6865-test.exe" "test6865" "test6865-test"
Completed 2 action(s).

Error: [S-7282]
       Stack failed to execute the build plan.

       While executing the build plan, Stack encountered the error:

       Error: [S-1995]
       Test suite failure for package test6865-0.1.0.0
           test6865-test:  executable not found

The package.yaml has:

custom-setup:
  dependencies:
  - base >= 4.7 && < 5
  - Cabal

[philippedev101]

@mpilgrem Thanks for the clean repro and the draft-status pause. Root cause identified, fix pushed as a follow-up commit on this branch. Short answer: for build-type: Custom the primary and final tasks end up keyed by the same ComponentKey CLib, and toActions was dropping the final's build action on the assumption the primary would cover it. It does not: the primary runs with isFinalBuild=False which skips --enable-tests/--enable-benchmarks at configure time and narrows the Setup build targets to lib and exe via primaryComponentOptions.

Root cause

After Phase 2, the build plan is split into plan.tasks (primary) and plan.finals (tests/benches), both keyed by ComponentKey. For Simple build-type packages shouldSplitComponents returns True, addFinal keys each final under a distinct CTest/CBench, and everything works.

For Custom / Configure / Hooks / Make, shouldSplitComponents returns False and addFinal falls back to ComponentKey name CLib. Both maps now key the same package under the same ComponentKey. toActions had:

finalBuild = case mbuild of
  Just _  -> []
  Nothing -> [ ... singleBuild ... isFinalBuild=True ... ]

When both mbuild and mfinal existed on the shared key, finalBuild became [] and the primary ran alone with isFinalBuild=False. Cabal configured without --enable-tests, built only lib:foo exe:bar, then ATRunTests fired and reported TestSuiteExeMissing because the test executable was never produced.

Stack's own project reproduces this because stack.cabal has build-type: Custom plus a custom-setup block, same shape as your test6865 minimal repro.

Fix

singleBuild gains an isMergedBuild :: Bool parameter threaded through realConfigAndBuild, realBuild, and buildAndCopyOpts. Three non-split modes, three target lists:

isFinalBuild	isMergedBuild	build targets	copy targets
False	_	primary	primary
True	True	primary ++ finals	primary
True	False	finals	[]

In toActions, the primary action now calls singleBuild ac ee task installedMap hasFinal hasFinal ck where hasFinal = isJust mfinal. So when a primary fires and a final exists on the same ComponentKey, the primary runs in merged mode: configure gets --enable-tests/--enable-benchmarks, setup build gets lib:foo exe:bar test:baz bench:qux in a single invocation, setup copy installs only the primary components. The pre-existing finalBuild action (which fires when no primary task runs because the library was clean) keeps its finals-only semantics via isMergedBuild=False, so it does not redundantly rebuild the library.

Two downstream guards match the new split: doHaddock and shouldCopy both key off not (isFinalBuild && not isMergedBuild). Haddock runs on merged builds, copy/register fires for plain primary, merged, and Backpack CLib/CInst builds, and both skip only for pure finals-only builds. Simple split packages are unaffected: mfinal is Nothing on their primary CLib keys because finals live on distinct CTest/CBench keys.

The decision table lives in a small pure helper dispatchBuildOpts that buildAndCopyOpts delegates to, which keeps it unit-testable without constructing a LocalPackage.

Tests

Seven new integration tests under tests/integration/tests/:

• custom-setup-test: the minimal repro. Before the fix, fails with TestSuiteExeMissing. After, the suite runs and passes.
• custom-setup-test-exe: mirrors the stack new default layout (lib + exe + test) plus custom-setup.
• custom-setup-bench: before the fix, fails with [Cabal-1453] No benchmarks enabled.
• custom-setup-test-and-bench: both final components together.
• custom-setup-test-idempotent: first run passes, second stack test is a no-op (no [N of M] Compiling, no Installing library, no Registering library).
• custom-setup-build-then-test: the stack build then stack test workflow.
• custom-setup-lib-only: Custom-setup with only a library. Guards the no-finals path against future regressions.

Thirteen new unit tests in tests/unit/Stack/Build/ExecuteSpec.hs cover all five arms of dispatchBuildOpts plus edge cases and componentTarget round-trips.

Stack's own stack test on the branch now passes (863 examples, 0 failures).

One incidental fix

While running the Backpack integration suite as a regression check I noticed backpack-cross-package-rename had been committed failing. Test-data bug: Consumer.hs imported Sig, but the mixin renames the signature requirement via sig-pkg requires (Sig as Impl), so the consumer has to import Impl. Fixed the import. The test now exercises the rename end-to-end.

What this does not change

The classic stack build then stack test flip-flop (#2800) is still present: switching between the two triggers a library reconfigure because --enable-tests changes, and Cabal unregisters and rebuilds the library. Same constraint I noted earlier about non-split packages sharing legacy single-invocation semantics. Full per-component execution would fix it; out of scope for this PR by design.

Happy to move the PR out of draft.

[mpilgrem]

@philippedev101, unfortunately, other existing integration tests fail. I had a look at one of them: 3229-exe-targets (reproducing its steps locally, on Windows).

With Stack 3.9.3, the first stack build :alpha has:

foo> configure (lib + exe)
Configuring foo-0...
foo> build (lib + exe) with ghc-9.10.3
Preprocessing library for foo-0...
Building library for foo-0...
[1 of 1] Compiling Foo
Preprocessing executable 'alpha' for foo-0...
Building executable 'alpha' for foo-0...
[1 of 1] Compiling Main
[2 of 2] Linking .stack-work\dist\1a191874\build\alpha\alpha.exe
foo> copy/register
Installing library in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3229-exe-targets\files\.stack-work\install\23d8aeba\lib\x86_64-windows-ghc-9.10.3-b42a\foo-0-2rEae6A22yFBRUicjP8Lud
Installing executable alpha in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3229-exe-targets\files\.stack-work\install\23d8aeba\bin
Registering library for foo-0...

and the second stack build :alpha (when the alpha source code is dirty) has:

foo-0: unregistering (local file changes: app\Alpha.hs)
foo> build (lib + exe) with ghc-9.10.3
Preprocessing library for foo-0...
Building library for foo-0...
Preprocessing executable 'alpha' for foo-0...
Building executable 'alpha' for foo-0...
[1 of 1] Compiling Main [Source file changed]
[2 of 2] Linking .stack-work\dist\1a191874\build\alpha\alpha.exe [Objects changed]
foo> copy/register
Installing library in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3229-exe-targets\files\.stack-work\install\23d8aeba\lib\x86_64-windows-ghc-9.10.3-b42a\foo-0-2rEae6A22yFBRUicjP8Lud
Installing executable alpha in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3229-exe-targets\files\.stack-work\install\23d8aeba\bin
Registering library for foo-0...

Unlike when the test was originally written, in neither case is the beta executable built.

The 'backpack' version of Stack has, for the first stack build :alpha:

foo> configure
Configuring foo-0...
foo> build with ghc-9.10.3
Preprocessing library for foo-0...
Building library for foo-0...
[1 of 1] Compiling Foo
foo> copy/register
Installing library in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3229-exe-targets\files\.stack-work\install\23d8aeba\lib\x86_64-windows-ghc-9.10.3-b42a\foo-0-2rEae6A22yFBRUicjP8Lud
Registering library for foo-0...
foo> build (exe:alpha) with ghc-9.10.3
Preprocessing library for foo-0...
Building library for foo-0...
Preprocessing executable 'alpha' for foo-0...
Building executable 'alpha' for foo-0...
[1 of 1] Compiling Main
[2 of 2] Linking .stack-work\dist\1a191874\build\alpha\alpha.exe
foo> copy/register
Installing library in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3229-exe-targets\files\.stack-work\install\23d8aeba\lib\x86_64-windows-ghc-9.10.3-b42a\foo-0-2rEae6A22yFBRUicjP8Lud
Installing executable alpha in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3229-exe-targets\files\.stack-work\install\23d8aeba\bin
Registering library for foo-0...
foo> build (exe:beta) with ghc-9.10.3
Preprocessing library for foo-0...
Building library for foo-0...
Preprocessing executable 'beta' for foo-0...
Building executable 'beta' for foo-0...
[1 of 1] Compiling Main
[2 of 2] Linking .stack-work\dist\1a191874\build\beta\beta.exe
foo> copy/register
Installing library in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3229-exe-targets\files\.stack-work\install\23d8aeba\lib\x86_64-windows-ghc-9.10.3-b42a\foo-0-2rEae6A22yFBRUicjP8Lud
Installing executable beta in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3229-exe-targets\files\.stack-work\install\23d8aeba\bin
Registering library for foo-0...
Completed 3 action(s).

In this case, the beta executable is built. That could be viewed as a regression, relative to Stack 3.9.3.

For the second stack build :alpha (when the alpha source code is dirty):

foo-0: unregistering (local file changes: app\Alpha.hs)
foo> build with ghc-9.10.3
Preprocessing library for foo-0...
Building library for foo-0...
foo> copy/register
Installing library in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3229-exe-targets\files\.stack-work\install\23d8aeba\lib\x86_64-windows-ghc-9.10.3-b42a\foo-0-2rEae6A22yFBRUicjP8Lud
Registering library for foo-0...
foo> build (exe:alpha) with ghc-9.10.3
Preprocessing library for foo-0...
Building library for foo-0...
Preprocessing executable 'alpha' for foo-0...
Building executable 'alpha' for foo-0...
[1 of 1] Compiling Main [Source file changed]
[2 of 2] Linking .stack-work\dist\1a191874\build\alpha\alpha.exe [Objects changed]
foo> copy/register
Installing library in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3229-exe-targets\files\.stack-work\install\23d8aeba\lib\x86_64-windows-ghc-9.10.3-b42a\foo-0-2rEae6A22yFBRUicjP8Lud
Installing executable alpha in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3229-exe-targets\files\.stack-work\install\23d8aeba\bin
Registering library for foo-0...
foo> build (exe:beta) with ghc-9.10.3
Preprocessing library for foo-0...
Building library for foo-0...
Preprocessing executable 'beta' for foo-0...
Building executable 'beta' for foo-0...
foo> copy/register
Installing library in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3229-exe-targets\files\.stack-work\install\23d8aeba\lib\x86_64-windows-ghc-9.10.3-b42a\foo-0-2rEae6A22yFBRUicjP8Lud
Installing executable beta in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3229-exe-targets\files\.stack-work\install\23d8aeba\bin
Registering library for foo-0...
Completed 3 action(s).

The Preprocessing executable 'beta' for foo-0... may well be a no-op, but the integration test does not view the presence of the message that way (from its Main.hs):

                 stackCheckStderr
                     ["build", ":alpha"]
                     (rejectMessage
                          (unlines
                               ["Preprocessing executable 'beta' for foo-0..."]))))

Perhaps the test should check for the absence of Installing executable beta in?

[mpilgrem]

@philippedev101, I rebased on the master branch after updating (after a false start) the integration test 3229-exe-targets - now named 6451-exe-targets. The regression sees 6451-exe-targets fail with:

Main.hs: Did not expect message here: 
"Installing executable beta in"

[mpilgrem]

I then looked at integration test 3959-order-of-flags. That is a test that involves a package with a test suite but no library.

With Stack 3.9.3, all is fine:

> stack --test --no-run-tests build
All test running disabled by --no-run-tests flag. To mute this message in future, set notify-if-no-run-tests: false in
Stack's configuration.
issue3959> configure (test)
Configuring issue3959-0.1.0.0...
issue3959> build (test) with ghc-9.10.3
Preprocessing test suite 'test' for issue3959-0.1.0.0...
Building test suite 'test' for issue3959-0.1.0.0...
[1 of 2] Compiling Main
[2 of 2] Compiling Paths_issue3959
[3 of 3] Linking .stack-work\dist\1a191874\build\test\test.exe

The 'backpack' version of Stack fails with (extract):

> stack --test --no-run-tests build
All test running disabled by --no-run-tests flag. To mute this message in future, set notify-if-no-run-tests: false in
Stack's configuration.
issue3959> configure
Configuring issue3959-0.1.0.0...
issue3959> build with ghc-9.10.3
Error: [Cabal-4444]
Unknown build target 'lib:issue3959'.
There is no library component 'issue3959' or component 'lib'.

Stack is trying to build a library component, and one that does not exist.

[mpilgrem]

@philippedev101, I have rebased on master again, as I have updated integration test 3996-sublib-not-depended-upon, which fails. This is a package with a main library and an internal library. The main library, however, does not depend on the internal library.

With Stack 3.9.3, all is fine:

myPackage> configure (lib + sub-lib)
Configuring myPackage-0.1.0.0...
myPackage> build (lib + sub-lib) with ghc-9.10.3
Preprocessing library for myPackage-0.1.0.0...
Building library for myPackage-0.1.0.0...
[1 of 1] Compiling MyPackage
Preprocessing library 'internal' for myPackage-0.1.0.0...
Building library 'internal' for myPackage-0.1.0.0...
[1 of 1] Compiling Internal
myPackage> copy/register
Installing library in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3996-sublib-not-depended-upon\files\.stack-work\install\6e44f233\lib\x86_64-windows-ghc-9.10.3-b42a\myPackage-0.1.0.0-HXh8re4RjQk4Ufw3IIAClw
Installing internal library internal in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3996-sublib-not-depended-upon\files\.stack-work\install\6e44f233\lib\x86_64-windows-ghc-9.10.3-b42a\myPackage-0.1.0.0-GPLsRg6s6iQKSRZ63H28HF-internal
Registering library for myPackage-0.1.0.0...
Registering library 'internal' for myPackage-0.1.0.0...

The 'backpack' version of Stack fails with (extract):

myPackage> configure
Configuring myPackage-0.1.0.0...
myPackage> build with ghc-9.10.3
Preprocessing library for myPackage-0.1.0.0...
Building library for myPackage-0.1.0.0...
[1 of 1] Compiling MyPackage
myPackage> copy/register
Installing library in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3996-sublib-not-depended-upon\files\.stack-work\install\6e44f233\lib\x86_64-windows-ghc-9.10.3-b42a\myPackage-0.1.0.0-HXh8re4RjQk4Ufw3IIAClw
Error: Cabal-simple_O_vy6YIf_3.12.1.0_ghc-9.10.3.exe:
'D:\sr\programs\x86_64-windows\ghc-9.10.3\bin\ghc-9.10.3.exe' exited with an
error:
<command line>: Could not find module ‘Internal’.
Use -v to see a list of the files searched for.

[mpilgrem]

Given these three regressions, and the other failing integration tests, I have changed the status of the pull request to draft.

[philippedev101]

@mpilgrem Thanks for checking the work and for your patience.

The three regressions you flagged are fixed. Three things from your review style shaped how I verified the fixes this time:

• You found Bug 03 by noticing that the existing 3229-exe-targets test passed even though the wrong executable was being installed. So for each fix I captured full normalised stderr from the previous PR head and from the new code, and diffed line-by-line. The diff catches behaviour drift that no assertion is watching for.
• You noted earlier that the third invocation of stack test rebuilt the library when it shouldn't. So each regression-relevant command now gets run three times back to back; runs 2 and 3 must emit no [N of M] Compiling, no Linking, no library install.
• For each fix I ran the regression test once on the previous PR head to confirm the failure mode in your report, and once on the new code to confirm it passes.

I also ran both Stack test suites. Unit suite all green. Integration suite has 7 failures; all 7 also fail on the previous PR head, so none are introduced by this update. Five are environmental on my NixOS host: 2465-init-no-packages, 4408-init-internal-libs, nice-resolver-names all run stack init --snapshot ghc-9.2.4 (or lts-20.26) and fail because the current nixpkgs channel doesn't expose ghc924/ghc928; 4181-clean-wo-dl-ghc is incompatible with the Nix integration by construction; 3850-cached-templates-network-errors doesn't get the expected behaviour from HTTPS_PROXY on this Nix shell. None of those will fire on standard CI. The other two are pre-existing Stack issues unrelated to this PR: multi-test fails configuring cyclic-0.1.0.0 with [Cabal-8010] Encountered missing or private dependencies: multi-test-suite, and no-rerun-tests shows stack test --no-rerun-tests re-running the test (exists2 should be False).

The three regressions

All three sit downstream of the same root cause. The previous PR head used per-component planning for every Simple package. The split path is meant only for Backpack; routing everything through it broke three different shapes. The fix is to narrow the predicate so non-Backpack packages stay on the legacy whole-package path. The Backpack code path itself (per-component planning, CInst instantiation tasks, --instantiate-with, per-instantiation --builddir) is unchanged.

Bug 01 (3996-sublib-not-depended-upon): multi-library packages crashed during setup register. The split path runs setup register per component, but Cabal's setup register is a package-wide operation: it always tries to register every library declared in the .cabal file. So when only the main library had been built, register went looking for the sub-library's Internal.hi and crashed with Could not find module 'Internal'. The split path was never the right place for multi-library packages.

shouldSplitComponents pkg =
     pkg.buildType == Simple
  && not (hasIntraPackageDeps pkg)
  && not (hasMultipleRegisterableLibraries pkg)
  && packageUsesBackpack pkg

packageUsesBackpack is True when the package has signatures or any component declares mixins:. Non-Backpack multi-library packages now route to the legacy whole-package path.

Bug 02 (3959-order-of-flags): test-only packages emitted a target that didn't exist. A package with a test suite but no library produced an empty component list. There was a fallback at the bottom of expandToComponentKeys that synthesised a ComponentKey CLib entry in that case. That entry rendered as lib:issue3959, and Cabal rejected it with [Cabal-4444] Unknown build target. The fallback shouldn't have been there; tests and benches schedule via the existing finals path that targets test:<name> directly.

-- before
in case allComps of
     [] -> [(ComponentKey name CLib, adr)]
     _  -> map (\comp -> (ComponentKey name comp, adr)) allComps

-- after
in map (\comp -> (ComponentKey name comp, adr)) allComps

Bug 03 (3229-exe-targets, now 6451-exe-targets after your update): stack build :one-exe built and installed every executable. The split path enumerated executables from the .cabal file directly and ignored the user's component selection entirely. The non-split path already had the right logic; the split path just wasn't propagating it.

-- before
exeComps = map CExe $ Set.toList $ buildableExes pkg

-- after
exeComps = map CExe $ Set.toList $
  if lp.wanted
    then exeComponents lp.components
    else buildableExes pkg

lp.wanted is True for packages the user named on the command line; for those, only the requested exes build. For dependencies (lp.wanted = False) all buildable exes still build, matching legacy behaviour.

Performance

12 rounds each, warm snapshots, default Stack parallelism (-j set to all cores).

Non-Backpack project (one library + two executables):

	Setup phases per build	Clean build (mean of 12)	Hot rebuild (mean of 12)
Stack 3.9.3	3 (configure + build + copy/register)	~3.0s	~1.4s
With this update	3 (configure + build + copy/register)	3.21s ± 0.11s	1.38s ± 0.06s

Same Setup invocation count as Stack 3.9.3. Clean-build wall clock is within noise. The opening claim of "no performance hit for non-Backpack projects" holds.

Backpack project (sig-pkg + impl-pkg + consumer-pkg, lib + 1 exe):

	Setup phases per build	Clean build (mean of 12)	Hot rebuild (mean of 12)
Backpack reference	14	4.93s ± 0.17s	1.16s ± 0.06s

Slowdown vs the equivalent non-Backpack project: ~1.5x wall clock, ~4.7x phase count. Per-phase cost is actually lower for the Backpack project; each phase compiles a tiny module and the wall-clock difference is the fixed cost of each setup subprocess.

Inherent Backpack cost vs unsolved engineering

The 14 phases break down as:

• sig-pkg indefinite library: 3 phases. Cabal's signature type-check and indefinite artifact emission. Required by Backpack.
• impl-pkg library: 3 phases. A normal library. Not Backpack overhead.
• sig-pkg CInst with Str = impl-pkg: 3 phases. The actual instantiation. Required by Backpack: each filling produces its own compiled artifact, and Cabal's Setup interface needs a separate invocation per instantiation. With N distinct fillings you get N CInst tasks.
• consumer-pkg library + executable via the split path: 5 phases (1 configure + 1 build for the lib + 1 copy/register + 1 build for the exe + 1 copy/register).

Of those:

• 6 phases are inherent to Backpack as Cabal models it (sig-pkg indefinite + sig-pkg CInst). No build tool can do better. This is the cost of Backpack's static specialisation.
• 3 phases are normal-package overhead (impl-pkg). It's a regular library and would exist in any project wanting this implementation, Backpack or not.
• 2 phases are Stack-side engineering overhead (the consumer's lib+exe split). In principle the consumer's lib and exe could build in a single Setup invocation, the way a non-Backpack package's lib+exe do. Doing that without losing the per-component planning Backpack itself needs is a follow-up. It is not a Backpack-fundamental cost.

So the extra cost of cross-package mixin use is "one indefinite build plus one instantiation per consumer-distinct filling". Everything else is either normal-package work that would exist anyway, or Stack engineering that can be reduced in a future PR.

PR #6883 was essential for these fixes: the original 3229-exe-targets assertion couldn't catch the regression my code introduced, and your replacement test did. The same goes for #6884 against 3996. The broader test review-and-conform series across the integration suite produces real signal that this PR depended on.

Moving back out of draft. I think this is ready, but please push it back to draft if anything else surfaces.

[mpilgrem]

@philippedev101, I have rebased on master. That is principally after:

• Re #6905 Various minor changes to assist with debugging #6906
• Fix #6905 Clear call stack in addFinal #6908

In case it affects your changes, the principal effect of #6908 (fixing #6905) was at the start of addFinal and temporarily clearing the call stack before:

  logDebugPlanS "addFinal" "Clearing the call stack."
  res <- local (\ctx' -> ctx' { callStack = [] }) $
           addPackageDeps package >>= \case

EDIT: I rebased again, because I added an integration test to master. That test (6342-say-ghc-version-in-build) needed to be tweaked in this pull request.

[mpilgrem]

@philippedev101, the three remaining regressions are:

1. 3770-no-rerun-tests

With Stack 3.9.3, stack --no-rerun-tests test does not re-run test suites that have previously been built and run successfully.

The 'backpack' Stack behaves as if it is ignoring the --no-rerun-tests flag. The initial stack test is still writing out the 'success' flag (the file stack-test-success in the package's dist directory). EDIT: Correction, the flag is 'success' when the command is over but 'unknown' at the point when it is examined.

2. 6905-cyclic-plan

This tests a project with project packages A and B. A depends on B and the test suite of B depends on A.

With the master branch of Stack (following a recent fix), stack test works with:

myPackageB> configure
myPackageB> Configuring myPackageB-0.0.0...
myPackageB> build with ghc-9.10.3
myPackageB> Preprocessing library for myPackageB-0.0.0...
myPackageB> Building library for myPackageB-0.0.0...
myPackageB> copy/register
myPackageB> Installing library in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\6905-cyclic-plan\files\.stack-work\install\02ded40e\lib\x86_64-windows-ghc-9.10.3-b42a\myPackageB-0.0.0-HywWAukSPzk1yiyUrr6g43
myPackageB> Registering library for myPackageB-0.0.0...
myPackageA> configure (lib)
myPackageA> Configuring myPackageA-0.0.0...
myPackageA> build (lib) with ghc-9.10.3
myPackageA> Preprocessing library for myPackageA-0.0.0...
myPackageA> Building library for myPackageA-0.0.0...
myPackageA> copy/register
myPackageA> Installing library in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\6905-cyclic-plan\files\.stack-work\install\02ded40e\lib\x86_64-windows-ghc-9.10.3-b42a\myPackageA-0.0.0-LhcSNmb57b9CezPG8koigO
myPackageA> Registering library for myPackageA-0.0.0...
myPackageB> configure (test)
myPackageB> Configuring myPackageB-0.0.0...
myPackageB> build (test) with ghc-9.10.3
myPackageB> Preprocessing test suite 'test' for myPackageB-0.0.0...
myPackageB> Building test suite 'test' for myPackageB-0.0.0...
myPackageB> test (suite: test)

myPackageB> Test suite test passed
Completed 4 action(s).

However, the 'backpack' Stack fails during the execution of the build plan with:

myPackageB> configure (test)
myPackageB> Configuring myPackageB-0.0.0...
myPackageB> Error: [Cabal-8010]
myPackageB> Encountered missing or private dependencies:
myPackageB>     myPackageA
myPackageB>

3. 6905-multi-test

The tests a project that is similar to 6905-cyclic-plan (for coverage) and fails in a similar way.

[mpilgrem]

@philippedev101, I've made unrelated changes in the master branch which mean that a further rebase is a little complex. See:

• Re #6920 Reformat/refactor and add in-code documentation #6921
• Fix #6896 Allow dependency package to have no main library #6928
• Minor reformatting and refactoring #6929

In particular, Stack now handles Cabal packages with sublibraries but no main library. That means, for Cabal packages that are installed as one or more installed packages with munged package identifiers, there is no single GhcPkgId that corresponds to the Cabal package identifier (before, when it was assumed that a Cabal package always had a main library, it was the 'GhcPkgId' of the installed package corresponding to the main library). The type representing all installed packages is now Map MungedPackageId GhcPkgId.

I tried to rebase myself, and got so far, but I was not confident that I knew in your added code when you were referring to a Cabal package and when you were referring to an installed package. So, I aborted.

One approach might be for me to rebase what I do understand and then hand over to you to fix up. However, what I hand over will necessarily not compile. Let me know what will help most.

[philippedev101]

@mpilgrem Apologies. I thought I had verification properly nailed down before the previous push, but I got the baseline wrong: I'd thought I was using the upstream master SHA when in fact I had taken a previous commit of this PR. So "fails on baseline too, pre-existing" let three real regressions through. They're fixed in this push. I've also rebased the PR onto current master past #6929, so the partial-handoff you offered isn't needed; the next thing only you can do is the cross-platform CI matrix.

The three regressions all sit downstream of the same thing. Phase 2's per-component refactor dropped master's allInOne flag, the bit on Task that distinguishes a test/benchmark build folded into the package's primary build from one that needs its own Setup invocation. toActions was left deriving the merged-vs-separate choice from isJust mfinal (and from whether mbuild existed), which conflates two unrelated facts, and it broke each of the three shapes you reported:

3770-no-rerun-tests. For a clean non-split package on the second invocation, mbuild=Nothing (lib up to date) but mfinal=Just (test wants to run). The old toActions still emitted a standalone build action for that case, running with isFinalBuild=True. That action's realBuild calls setTestStatus pkgDir TSUnknown, which clobbers the TSSuccess written by the first run's ATRunTests. ATRunTests on the second invocation then reads TSUnknown and reruns despite --no-rerun-tests. Matches your EDIT: success when the first command ended, unknown by the time --no-rerun-tests checks.

6905-cyclic-plan / 6905-multi-test. In the cyclic-plan fallback (addFinal's back-off when myPackageB:test -> myPackageA:lib -> myPackageB:lib would otherwise be a cycle), the lib-only primary task and the test final share one ComponentKey CLib. toActions was running the primary's build in merged mode (hasFinal = isJust mfinal), which configured with --enable-tests but using the primary task's configOpts. Those only carry library deps. Cabal then got --enable-tests with no --dependency myPackageA=... and bailed: [Cabal-8010] Encountered missing or private dependencies: myPackageA. The #6908 callStack clear was necessary but not sufficient on its own: the test's own configOpts also need to be what's threaded into the configure call, not the primary's.

Fix

Putting allInOne back, basically:

• Added allInOne :: Bool as a field on Task, and re-added ATBuildFinal to ActionType.
• installPackage computes the flag for the final task, curator-aware in the merged branch (a curator that expects test failures forces a split, so a failing test can't block the library from reaching its dependents), False in the cyclic-plan fallback. Curator handling was something Phase 2 had silently dropped; it's back too.
• addFinal forces False for the per-component split path (Backpack packages routed through shouldSplitComponents), whose finals live under distinct CTest/CBench keys and always need their own build step.
• toActions reads task.allInOne instead of guessing. An all-in-one final is built by the primary's ATBuild in merged mode (or skipped entirely if the package is up to date); a non-all-in-one final gets its own ATBuildFinal action built from the final task's own configOpts.
• buildAnnSuffix now also branches on task.allInOne to produce the (lib) / (lib + exe) / (lib + test) annotations your 6342-say-ghc-version-in-build update expects.

The per-component Backpack execution path (shouldSplitComponents, CInst tasks, --instantiate-with, per-instantiation --builddir) isn't touched. This just restores the merged-build option Phase 2 took away.

On the Cabal-package vs installed-package distinction you flagged when you tried the rebase: that was the substantive part, not the type renames. The boundary is most explicit at findGhcPkgId :: Map MungedPackageId GhcPkgId -> PackageName -> Maybe GhcPkgId, which takes installed state and returns the main-library id (filters via LMainLibName, returns Nothing for a no-main-library package), and at toCabalMungedPackageId pid Nothing in addInstantiationTasks for the reverse direction (Cabal-side pid into munged form for the CInst task's present map). A few inline patterns elsewhere still rely on knowing the installed-state shape; if specific call sites read ambiguously, point at them and I'll add a doc comment or split a helper out.

Verification

The baseline is freshly-fetched upstream/master@e6170f1b9 this time, not a previous commit of this PR. Unit suite is green (858 examples, 0 failures). The three target tests are red on the previous PR head and green here; isolated re-runs of the Backpack cross-package tests, backpack-doc-sublibs, and 6342-say-ghc-version-in-build all pass.