Concretizer duplicates strategy (#269)

Based on top of #268. It's easiest to only look at the unique commit for
this PR:
3ed904a.

This is the second change I need to be able to concretize `rccl-tests`
with `llvm-amdgpu` as the compiler. This adds a configuration option to
control the `concretizer:duplicates:strategy` configuration option. For
`rccl-tests` I need to set the option to `full`, instead of the default
`minimal`.

Unfortunately I can't really explain why this is required. The docs
explain a bit what the different options do:
https://spack.readthedocs.io/en/latest/build_settings.html.
```
  duplicates:
    # "none": allows a single node for any package in the DAG.
    # "minimal": allows the duplication of 'build-tools' nodes only
    # (e.g. py-setuptools, cmake etc.)
    # "full" (experimental): allows separation of the entire build-tool stack (e.g. the entire "cmake" subDAG)
```
I've asked for feedback on the spack slack:
https://spackpm.slack.com/archives/C059JUS9T38/p1761907568615009.

In the meantime I'm going to assume that this will be needed. I don't
quite like that this is another ad-hoc spack configuration option
exposed in the config. It'd be nice to just have a `concretizer` section
exposed in `environments.yaml`, but that's a breaking change (`unify` is
not nested under `concretizer`). Alternatively, perhaps it should be
possible to supply a completely separate spack configuration that can be
used to override anything, without having to add support for every
single configuration option explicitly.

---------

Co-authored-by: Alberto Invernizzi <9337627+albestro@users.noreply.github.com>

Author: msimberg

docs/recipes.md

@@ -103,6 +103,7 @@ prgenv-gnu:
   network:    # ... network configuration
   deprecated: # ... whether to allow usage of deprecated packages or not
   unify:      # ... configure Spack concretizer
+  duplicates: # ... configure how the concretizer should handle duplicates
   specs:      # ... list of packages to install
   variants:   # ... variants to apply to packages (e.g. +mpi)
   packages:   # ... list of external packages to use
@@ -277,6 +278,16 @@ cuda-env:
 !!! warning
     Stackinator does not support "spec matrices", and likely won't, because they use multiple compiler toolchains in a manner that is contrary to the Stackinator "keep it simple" principle.
 
+The `duplicates:strategy` field [controls how the concretiser handles duplicates](https://spack.readthedocs.io/en/latest/build_settings.html).
+The default strategy is `minimal` which is typically the right choice.
+In mixed compiler environments, `minimal` can sometimes be insufficient to allow concretization.
+`full` slightly relaxes the constraints that Spack imposes on the environment which can allow concretization.
+
+!!! warning
+    The `full` strategy is still experimental and may produce surprising concretizations.
+    In particular, the concretiser will more happily mix compilers using the `full` strategy.
+    Please check the concretization when using this strategy to make sure it looks as you'd expect.
+
 ### Packages
 
 To specify external packages that should be used instead of building them, use the `packages` field.

stackinator/recipe.py

@@ -145,6 +145,7 @@ def __init__(self, args):
                 "compiler": ["gcc"],
                 "network": {"mpi": None, "specs": None},
                 "unify": True,
+                "duplicates": {"strategy": "minimal"},
                 "deprecated": False,
                 "specs": ["squashfs"],
                 "views": {},

stackinator/schema/environments.json

@@ -46,6 +46,19 @@
                      "enum": ["when_possible", true, false],
                      "default": true
                  },
+                 "duplicates": {
+                     "type": "object",
+                     "required": ["strategy"],
+                     "additionalProperties": false,
+                     "properties": {
+                         "strategy": {
+                             "enum": ["minimal", "full", "none"]
+                         }
+                     },
+                     "default": {
+                         "strategy": "minimal"
+                     }
+                 },
                  "compiler": {
                      "type": "array",
                      "items": {

stackinator/templates/environments.spack.yaml

@@ -7,6 +7,8 @@ spack:
   concretizer:
     unify: {{ config.unify }}
     reuse: false
+    duplicates:
+      strategy: {{ config.duplicates.strategy }}
   specs:
 {% for spec in config.specs %}
   - '{{ spec }}'

unittests/test_schema.py

@@ -160,6 +160,7 @@ def test_environments_yaml(yaml_path):
 
         # test defaults were set correctly
         assert env["unify"] == True  # noqa: E712
+        assert env["duplicates"]["strategy"] == "minimal"
         assert env["packages"] == []
         assert env["variants"] == []
         assert env["network"] == {"mpi": None, "specs": None}
@@ -179,6 +180,7 @@ def test_environments_yaml(yaml_path):
 
         # test defaults were set correctly
         assert env["unify"] == "when_possible"
+        assert env["duplicates"]["strategy"] == "full"
         assert env["packages"] == ["perl", "git"]
         assert env["variants"] == ["+mpi", "+cuda"]
         assert env["views"] == {"default": None}

unittests/yaml/environments.full.yaml

@@ -1,6 +1,8 @@
 full-env:
   compiler: [gcc]
   unify: when_possible
+  duplicates:
+    strategy: full
   specs:
   - osu-micro-benchmarks@5.9
   - hdf5 +mpi