Cherry-picking virtual dependencies

[alalazo]

depends on #39508
fixes #15443

This PR makes it possible to select only a subset of virtual dependencies from a spec that may provide more. To select providers, a syntax to specify edge attributes is introduced:

hdf5 ^[virtuals=mpi] mpich

With that syntax we can concretize specs like:

$ spack spec strumpack ^[virtuals=mpi] intel-parallel-studio+mkl ^[virtuals=lapack] openblas

and get this concrete DAG:

On develop this would currently fail with:

$ spack spec strumpack ^intel-parallel-studio+mkl ^openblas
==> Error: Spec cannot include multiple providers for virtual 'blas'
    Requested 'intel-parallel-studio' and 'openblas'

In package recipes, virtual specs that are declared in the same provides directive need to be provided together. This means that e.g. openblas, which has:

provides("blas", "lapack")

needs to provide both lapack and blas when requested to provide at least one of them.

Additional notes

This capability is needed to model compilers. Assuming that languages are treated like virtual dependencies, we might want e.g. to use LLVM to compile C/C++ and Gnu GCC to compile Fortran. This can be accomplished by the following¹:

hdf5 ^[virtuals=c,cxx] llvm ^[virtuals=fortran] gcc

Modifications:

• Add syntax to interact with edge attributes from spec literals
• Add concretization logic to be able to cherry-pick virtual dependencies
• Extend semantic of the provides directive to express when virtuals need to be provided together
• Add unit-tests and documentation

Footnotes

1. We plan to add some syntactic sugar around this syntax, and reuse the % sigil to avoid having a lot of boilerplate around compilers. ↩

[spackbot-app]

Hi @alalazo! I noticed that the following package(s) don't yet have maintainers:

• openblas

Are you interested in adopting any of these package(s)? If so, simply add the following to the package class:

    maintainers = ["alalazo"]

If not, could you contact the developers of this package and see if they are interested? You can quickly see who has worked on a package with spack blame:

$ spack blame openblas

Thank you for your help! Please don't add maintainers without their consent.

You don't have to be a Spack expert or package developer in order to be a "maintainer," it just gives us a list of users willing to review PRs or debug issues relating to this package. A package can have multiple maintainers; just add a list of GitHub handles of anyone who wants to volunteer.

[alalazo]

@spackbot run pipeline

[spackbot-app]

I've started that pipeline for you!

[RMeli]

Hello. I tried this PR with the following spec:

cp2k@master +dlaf +libxc +libint +spglib build_system=cmake ^[virtuals=blas,lapack,scalapack,mkl]
    intel-mkl@2020.4.304 threads=openmp ^fftw ^cray-mpich

It returned this error, which is very cryptic:

==> Error: 
        cp2k@master+dlaf+libint+libxc+spglib build_system=cmake ^cray-mpich ^fftw ^[virtuals=blas,lapack,mkl,scalapack] intel-mkl@2020.4.304 threads=openmp is unsatisfiable, errors are:

That is, no error messages are actually printed for errors are:.

However, by removing mkl I was able to concretise the environment with both intel-mkl and fftw and everything seems to work as expected:

cp2k@master +dlaf +libxc +libint +spglib build_system=cmake ^[virtuals=blas,lapack,scalapack]
    intel-mkl@2020.4.304 threads=openmp ^fftw ^cray-mpich

This is a really neat and useful feature.

[alalazo]

@RMeli Thanks for the feedback! Error messages are a pain point when working with ASP, we'll try to improve them considerably for v0.21 (so roughly going to November 2023) 🙂

[alalazo]

Not completely done, but here is a partial review.

@tgamblin ☝️

[tgamblin]

I went over the rest of the code here and it LGTM.

addressed