`cabal test` doesn't always solve for tests (unless `--enable-tests` or `tests: True` is given)

[Mikolaj]

@ulysses4ever's exec summary:

• this issue (and its variations) has been discussed on multiple occasions (new-test should imply --enable-tests #5079, 'cabal new-build test:test-suite' doesn't force tests to be enabled. #3923)

• a desirable converging point in the design space seems to be, as of now (as put by grayjay):

  By default, cabal build would solve for tests but not build them, and cabal test would solve for tests and build them

• the main counterpoint to the idea above (as put by fgaz):

  It could happen that there are no build plans with tests enabled but there are with tests disabled

• it feels like we get a better UX if we follow the idea still, and the answer to the counterpoint (as put by Mikolaj):

  [In the case of no plan including tests or benchmarks exists,] the user can manually disable tests or benchmarks

• anecdotally, many power users already put tests: True in their global config and deal with the above mentioned problem on the case by case basis; we just enshrine this practice in the default;

• our best hope for implementing something like that is, currently, [WIP] Only buildable tests #7829 but it looks like it needs a new champion...

---

Describe the bug

I'm dog-fooding current dev cabal from master branch.

Solver is not cooperating. Cabal says "solver picked a plan that does not include the test suites" despite there being such a plan, which can be seen when forcing with 'tests: True'.

To Reproduce

Clone

Mikolaj/mostly-harmless@3feb586

Run the following (it seems, a fresh package store is not required after all)

cabal test -w ghc-9.2.1

It shows

Resolving dependencies...
Error: cabal: Cannot test the package mostly-harmless-0.1.0.0 because none of
the components are available to build: the test suite 'mostly-harmless-test'
is not available because the solver picked a plan that does not include the
test suites, perhaps because no such plan exists. To see the error message
explaining the problems with such plans, force the solver to include the test
suites for all packages, by adding the line 'tests: True' to the
'cabal.project.local' file.

but cabal build -w ghc-9.2.1 --enable-tests (as well as cabal test -w ghc-9.2.1 with 'tests: True' in cabal.project.local) goes through fine and tests run. However, after removing cabal.project.local, cabal test -w ghc-9.2.1 fails again the same.

The same attempts with ghc-8.10.7 work fine.

Expected behavior

The solver should pick the right plan without forcing. Forcing would ideally be used only to show the error that prevents the solver from finding a plan. That's the assumption under which @gelisam and me operated in #7834 and related tickets. Now it's all put into question.

[gelisam]

It's not the assumption I was using :) Remember this comment, in which I said that I think the solver is allowed to decide to pick a plan which does not include tests even if one exists. For example, if suppose the solver is both allowed to decide whether to enable tests and whether to enable the benchmarks. The solver might try four combinations:

1. enable tests, enable benchmarks
2. disable tests, enable benchmarks
3. enable tests, disable benchmarks
4. disable tests, disable benchmarks

If the solver tries them in that order and picks the first one which succeeds, you'll observe the current bug if 1 fails and 2 and 3 succeed, because 2 is picked even though 3 succeeds. And we can't fix this by checking 3 before 2, because then we'd just have the same bug but for benchmarks!

The reality is much worse, because there are many more dimensions of choice, such as the version of every recursive dependency. So many that it's not feasible to check every allowed plan in which the tests are enabled. Instead, a heuristic must be used, to examine more promising branches of the possibility tree before others. And if that heuristic makes it look like disabling the tests is a promising avenue, the solver can easily find a plan in the disabled-tests branch without having completely explored the enabled-tests branch, and thus choose to disable the tests even if a successful plan exists somewhere in that unexplored enable-tests branch.

[Mikolaj]

@gelisam: you are right and I even responded to that comment, but then promptly repressed that, because it's too painful to believe. In fact, the cabal's blurb in my bug's description is correct and takes into account the possibility of solver not cooperating and that's exactly the blurb you changed in #7834. So, #7834 and related are not threatened.

So the problem in my case is that I have 1 library, 1 exe, 1 testsuite and the solver is not certain that the exe and the test are not excluding each other? And so it picks a plan where exe can be built and ignores the testsuite?

Could we tweak cabal to prioritize tests (respectively, benchmarks) over exes and libraries and everything, when cabal test is invoked (respectively, cabal benchmark)?

@GrayJay: do you think it's easy to ask the solver to do that and cabal just fails to ask, or is solver not able to prioritize things, apart of the brute force mode where it tries exhaustively and fails, if it can't build something (e.g., --enable-tests)?

[gelisam]

Could we tweak cabal to prioritize tests

There is a very clear description of the algorithm in this Well-Typed post on Qualified Goals in the Cabal Solver. There is a very clear phase where such preferences are applied: the "Heuristics and Preferences " section in which the tree branches are reordered. I think it would be instructive to create a variant of your repro case in which the search tree is small enough that we can make diagrams like the ones in the post, so we can see at which node the choice of enabling tests or not is made, and discuss at which node we would like to make that choice instead.

[sebright]

I think this issue is the same as #5079. The problem is that, by default, cabal runs the solver with only a preference to enable tests and benchmarks. The preference is only applied locally by sorting nodes in the search tree, which means that the solver may disable tests in order to satisfy a different preference, such as the preference to avoid deprecated versions. We could apply preferences globally by implementing a feature like #2860 and give the testing preference higher priority, but it would only reduce the frequency of the current issue, not prevent it. There would still be cases where the solver silently disables tests without --enable-tests, such as the conflict between tests and benchmarks in #7883 (comment) or a conflict between two test suites.

Could we tweak cabal to prioritize tests (respectively, benchmarks) over exes and libraries and everything, when cabal test is invoked (respectively, cabal benchmark)?

I think we should continue using the same default solver options for both "build" and "test" in order to avoid unstable install plans. I think that it would be very confusing for users to get completely different versions of dependencies depending on the cabal command that was run.

[Mikolaj]

Doh, thank you very much for the explanations. I apologise, I couldn't get around to generating that small test case and tried to do that before responding. But I feel this is an important usability point, so I don't want to just close as a duplicate.

I think we should continue using the same default solver options for both "build" and "test" in order to avoid unstable install plans. I think that it would be very confusing for users to get completely different versions of dependencies depending on the cabal command that was run.

@GrayJay: do you mean that if "test" can't be solved using a superset of dependencies generated for "build", it always fails (without --enable-tests)? That guarantee would indeed be worth something, but only if the user can depend on it. Or am I over-interpreting?

[gelisam]

do you mean that if "test" can't be solved using a superset of dependencies generated for "build"

The question implies that cabal build runs the solver to pick versions of the dependencies of lib and exe targets, then builds them, while cabal test runs the solver to pick versions of the test targets, then builds and runs them. This is not how those commands work. If they did (I think they once did?), then running cabal build followed by cabal test would likely pick different plans (what@grayjay refers to as an "unstable install plan") and thus end up building dependencies twice. Or if like you suggest, subsequent invocations of the solver only pick plans which are compatible with previous invocations, then which plan you get would depend on whether you start by running cabal build or cabal test, which is a project-specific version of the so-called "cabal hell" situation we had before the v2 commands.

Since both of those behaviors are undesirable, what is actually happening is that all the commands (except cabal-install) use the same plan. So it doesn't matter if you call cabal build or cabal test first, the solver will always look for a plan which involves all the libraries, tests, exes, and benchmarks.

I was careful to say "involves", not "includes", because a plan does not just pick version numbers for the dependencies, it also makes decisions about flags, whether to enable tests, etc. That's how we nevertheless manage to get plans which don't include the tests.

[Mikolaj]

OK, you convinced me we are doing the right thing. I take back my proposal to try harder to include tests when the solver notices it's run from cabal test. That's a bad idea.

I guess, that leaves me with a request to try harder (ideally guarantee) that the plan we find is a local maximum in terms of how many components are included. E.g., in my (non-minimized) real life case above, it was possible to add a testsuite (without resorting to changing flags or picking deprecated deps), but the chosen plan included none. Local maximum in the sense, that if it's possible to add one more component (without removing any that are already included) by changing only dependencies (and not to deprecated versions) then it's added. Perhaps what that entails is that the preference to include even a single new component should be stronger than to keep optimal dependencies (newest possible, or already found), even if that means downgrading absolutely all dependencies to stone age versions (barring deprecated versions).

Perhaps that's impossible, though. I'm reading the blog post and it underlines that decisions are local, while I require global guarantees, which may imply an exhaustive search/backtracking. I think the solver never backtracks over preferences, e.g., a preference to not exclude a component? It only backtracks over constraint-like things, e.g., the tests: True hard requirements.

Or how bad would it be to treat component inclusion as hard constraints? (The difference vs tests: True would be that the latter causes tests to be actually built.) Or perhaps we can let the user decide, via tests: Plan, that the tests have to be included in the plan, at the cost of as many backtracks, as needed? @gelisam: did our brainstormimg about a new value for the tests: setting touch this (it's been a long time)? That would an acceptable setting for a user that runs cabal build 1000 times a day and cabal test 10 times a day. Without it, a user for whom the default plan excludes tests, is forced to either switch plans and rebuild every time the user runs tests or force tests: True and slow down every error-checking build. In large projects, that's a significant cost.

Please tell me how I'm mixing up things again. ;)

[gelisam]

@gelisam: did our brainstormimg about a new value for the tests: setting touch this

The new value does not change the current behavior in any way, it just better documents the fact that they are three options rather than just True and False.

[gelisam]

Or how bad would it be to treat component inclusion as hard constraints?

If we limit our attention to plans which include all the components, then Haskellers will be happier because they are the ones who tend to be surprised by the current behaviour, but non-Haskellers, who need the most help, will be greated with an inscrutable "cannot find build plan" error more often, because the solver will not be allowed to work harder by looking for a build plan which can build the executable they care about but not the tests they don't know how to run anyway.

[gelisam]

Instead of adding even more values to tests, how about splitting the field into several smaller ones? One field to determine if the tests should be included in the plan, and one to specify which components are included in the default all:all target. Speaking of which, you can run cabal build all:libraries if you don't want to build the tests.

[Mikolaj]

If we limit our attention to plans which include all the components, then Haskellers will be happier because they are the ones who tend to be surprised by the current behaviour, but non-Haskellers, who need the most help, will be greated with an inscrutable "cannot find build plan" error more often, because the solver will not be allowed to work harder by looking for a build plan which can build the executable they care about but not the tests they don't know how to run anyway.

But aren't newcomers mostly running cabal install package-name or sometimes cabal install .? Do I remember right cabal install doesn't try to find a build plan with tests and benchmarks?

Instead of adding even more values to tests, how about splitting the field into several smaller ones? One field to determine if the tests should be included in the plan, and one to specify which components are included in the default all:all target.

Agreed.

Speaking of which, you can run cabal build all:libraries if you don't want to build the tests.

Good to know, thank you. But in a week all I remember will be 'cabal build' again. ;) OTOH, if I stick something that corresponds to that in my cabal.project.local, it will still work and for new projects I will remember to look it up in my old projects (I always keep a lot of cabal.project.local variants, with different suffixes). So IMHO it's worth it to try to match commandline and config in this case, even partially.

[gelisam]

Oh right, good point about cabal install using a different plan! In that case, I now agree that tests: True should be the default, and perhaps even the only option.