directives: lazy execution for package metadata #51881
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Both `@when(...)` and `with when(...)` now use immutable Specs * Avoid the `Spec -> str -> Spec` conversion between `when` and `directives_meta` * Avoid copy of `kwargs` in the common case where there is no `default_args` set * Remove vestigial `if isinstance(result, Sequence)` check on the return value of a directive -- it's always a callable. * Avoid `constrain` call if there's only a single when condition. Signed-off-by: Harmen Stoppels <me@harmenstoppels.nl>
Signed-off-by: Harmen Stoppels <me@harmenstoppels.nl>
Signed-off-by: Harmen Stoppels <me@harmenstoppels.nl>
haampie
force-pushed
the
hs/fix/speedup-first-spack-load
branch
3 times, most recently
from
d6eb6a6 to
5ee3706
Compare
This refactors the directive system to execute directives lazily. Directives are now queued during class definition and only executed when the corresponding dictionary on the package class is accessed (e.g., `pkg.dependencies`, `pkg.versions`). This reduces overhead during package discovery and cache population. - `DirectiveDictDescriptor` is a singleton descriptor shared across package classes. It handles the lazy initialization of package attributes. - For each dictionary, the set of directives affecting it is computed automatically. For example, accessing `pkg.extendees` automatically triggers the execution of both `extends` and `depends_on` directives. - Refactored `depends_on` to apply nested `patch` directives immediately via `_execute_patch`. This prevents state corruption in the global directive queue when `depends_on` is evaluated lazily. In the lazy model, we cannot support directives executing directives, and besides, it's redundant. - `maintainers` directives remain eager to ensure backward compatibility with packages that define maintainers as a simple class-level list. - Added `_has_nested_directives` to `PackageBase` to optimize patch indexing by skipping dependency traversals for packages that don't need them. Signed-off-by: Harmen Stoppels <me@harmenstoppels.nl>
haampie
force-pushed
the
hs/fix/speedup-first-spack-load
branch
from
f5d81df to
3a7bc08
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
On top of #51869
This refactors the directive system to execute directives lazily. Directives
are now queued during class definition and only executed when the corresponding
dictionary on the package class is accessed (e.g.,
pkg.dependencies,pkg.versions). This reduces overhead during cache population, and in non-forkingbuild subprocesses where
spec.packageis used and metadata isn't needed.DirectiveDictDescriptoris a singleton descriptor shared across packageclasses. It handles the lazy initialization of package attributes.
automatically. For example, accessing
pkg.extendeesautomatically triggersthe execution of both
extendsanddepends_ondirectives.pkg._dependenciesand so on. It isinitially set to
Noneto indicate the directives have not run yet.depends_onnow applies nestedpatchdirectives immediately via_execute_patch. This prevents state corruption in the global directivequeue when
depends_onis evaluated lazily. In the lazy model, we cannotsupport directives executing directives, and besides, it's redundant.
maintainersdirectives remain eager to ensure backward compatibility withpackages that define maintainers as a simple class-level list.
_has_nested_directivestoPackageBaseto optimize patch indexing byskipping dependency traversals for packages that don't need them.
The descriptor indirection does not seem to impact the "setup" phase of the
solver.
The first use of Spack is significantly faster. Below are results from a best of 3.
Before (855e07a):
After (5ee3706):
The
list(spack.repo.PATH.all_package_classes())benchmark is now useless. Witha few other PRs combined (#51875, #51836, #51879), the first-use time reduces
further to 0m7.802s, which is very close to 2x faster.