Garbage collection for Spack Environments #1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…m node dictionaries
# Conflicts: # lib/spack/spack/cmd/uninstall.py
Author
|
[Ooops... I thought this was going on the main Spack site. @scheibelp it's here because your PR branch is here, not because I want you to merge it into that branch. I will reference this PR from the main Spack.] |
scheibelp
force-pushed
the
features/environments
branch
2 times, most recently
from
0ddecbe to
41cc369
Compare
tgamblin
force-pushed
the
features/environments
branch
2 times, most recently
from
30fc088 to
acdcc29
Compare
scheibelp
force-pushed
the
features/environments
branch
2 times, most recently
from
7439b78 to
3529325
Compare
tgamblin
force-pushed
the
features/environments
branch
3 times, most recently
from
0af3375 to
438737b
Compare
24 tasks
scheibelp
pushed a commit
that referenced
this pull request
Nov 27, 2018
Verification build
dantopa@darwin-fe2.lanl.gov:pr.ucx.spack $ spack install ucx % gcc @ 4.8.5
==> Installing libsigsegv
==> Installing m4
==> Installing pkgconf
==> Installing ncurses
==> Installing readline
==> Installing gdbm
==> Installing perl
==> Installing autoconf
==> Installing automake
==> Installing libtool
==> Installing numactl
==> Installing zlib
==> Installing openssl
==> Installing diffutils
==> Installing bzip2
==> Installing xz
==> Installing libxml2
==> Installing tar
==> Installing gettext
==> Installing help2man
==> Installing bison
==> Installing flex
==> Installing libnl
==> Installing rdma-core
==> Installing ucx
==> Successfully installed ucx
Fetch: 1.17s. Build: 40.59s. Total: 41.76s.
[+] /scratch/users/dantopa/new-spack/pr.ucx.spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/ucx-1.4.0-ilfonxj2f5my4ur65li3sr7kuvfg3kim
dantopa@darwin-fe2.lanl.gov:pr.ucx.spack $ spack spec ucx
Input spec
--------------------------------
ucx
Concretized
--------------------------------
ucx@1.4.0%gcc@4.8.5 arch=linux-centos7-x86_64
^numactl@2.0.11%gcc@4.8.5 patches=592f30f7f5f757dfc239ad0ffd39a9a048487ad803c26b419e0f96b8cda08c1a arch=linux-centos7-x86_64
^autoconf@2.69%gcc@4.8.5 arch=linux-centos7-x86_64
^m4@1.4.18%gcc@4.8.5 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-centos7-x86_64
^libsigsegv@2.11%gcc@4.8.5 arch=linux-centos7-x86_64
^perl@5.26.2%gcc@4.8.5+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac +shared+threads arch=linux-centos7-x86_64
^gdbm@1.18.1%gcc@4.8.5 arch=linux-centos7-x86_64
^readline@7.0%gcc@4.8.5 arch=linux-centos7-x86_64
^ncurses@6.1%gcc@4.8.5~symlinks~termlib arch=linux-centos7-x86_64
^pkgconf@1.5.4%gcc@4.8.5 arch=linux-centos7-x86_64
^automake@1.16.1%gcc@4.8.5 arch=linux-centos7-x86_64
^libtool@2.4.6%gcc@4.8.5 arch=linux-centos7-x86_64
^rdma-core@20%gcc@4.8.5 build_type=RelWithDebInfo arch=linux-centos7-x86_64
^cmake@3.12.4%gcc@4.8.5~doc+ncurses+openssl+ownlibs patches=dd3a40d4d92f6b2158b87d6fb354c277947c776424aa03f6dc8096cf3135f5d0 ~qt arch=linux-centos7-x86_64
^openssl@1.0.2o%gcc@4.8.5+systemcerts arch=linux-centos7-x86_64
^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64
^libnl@3.3.0%gcc@4.8.5 arch=linux-centos7-x86_64
^bison@3.0.5%gcc@4.8.5 arch=linux-centos7-x86_64
^diffutils@3.6%gcc@4.8.5 arch=linux-centos7-x86_64
^help2man@1.47.4%gcc@4.8.5 arch=linux-centos7-x86_64
^gettext@0.19.8.1%gcc@4.8.5+bzip2+curses+git~libunistring+libxml2 patches=9acdb4e73f67c241b5ef32505c9ddf7cf6884ca8ea661692f21dca28483b04b8 +tar+xz arch=linux-centos7-x86_64
^bzip2@1.0.6%gcc@4.8.5+shared arch=linux-centos7-x86_64
^libxml2@2.9.8%gcc@4.8.5~python arch=linux-centos7-x86_64
^xz@5.2.4%gcc@4.8.5 arch=linux-centos7-x86_64
^tar@1.30%gcc@4.8.5 arch=linux-centos7-x86_64
^flex@2.6.3%gcc@4.8.5+lex arch=linux-centos7-x86_64
Environment:
dantopa@darwin-fe2.lanl.gov:pr.ucx.spack $ echo $HOSTNAME
darwin-fe2.lanl.gov
dantopa@darwin-fe2.lanl.gov:pr.ucx.spack $ grep -i 'model name' /proc/cpuinfo | sort | uniq
model name : Intel(R) Xeon(R) CPU E5-2695 v3 @ 2.30GHz
dantopa@darwin-fe2.lanl.gov:pr.ucx.spack $ lsb_release -d
-bash: lsb_release: command not found
dantopa@darwin-fe2.lanl.gov:pr.ucx.spack $ uname -a
Linux darwin-fe2.lanl.gov 4.18.8-1.el7.elrepo.x86_64 #1 SMP Sat Sep 15 10:10:09 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
dantopa@darwin-fe2.lanl.gov:pr.ucx.spack $ date
Mon Nov 19 10:47:47 MST 2018
dantopa@darwin-fe2.lanl.gov:pr.ucx.spack $ pwd
/scratch/users/dantopa/new-spack/pr.ucx.spack
Signed-off-by: Daniel Topa <dantopa@lanl.gov>
scheibelp
pushed a commit
that referenced
this pull request
Jun 24, 2020
Whenever attempting to use any ncurses functionality within cscope, a
page fault would result within the ncurses library.
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7fad3cf in termattrs_sp () from .../lib/libncursesw.so.6
(gdb) bt
#0 0x00007ffff7fad3cf in termattrs_sp () from .../lib/libncursesw.so.6
#1 0x00007ffff7faa794 in _nc_setupscreen_sp () from .../lib/libncursesw.so.6
#2 0x00007ffff7fa614c in newterm_sp () from .../lib/libncursesw.so.6
#3 0x00007ffff7fa65b9 in newterm () from .../lib/libncursesw.so.6
#4 0x00007ffff7fa2970 in initscr () from .../lib/libncursesw.so.6
#5 0x0000000000403dc2 in main (argc=<optimized out>, argv=0x7fffffffcea8) at main.c:574
This is due to a conflict between libtinfo.so and libtinfow.so. Both are
linked into cscope:
$ ldd $(which cscope)
/bin/bash: .../lib/libtinfo.so.6: no version information available (required by /bin/bash)
linux-vdso.so.1 (0x00007fff5dbcb000)
libncursesw.so.6 => .../lib/libncursesw.so.6 (0x00007f435cc69000)
libtinfo.so.6 => .../lib/libtinfo.so.6 (0x00007f435cc2c000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f435ca29000)
libtinfow.so.6 => .../lib/libtinfow.so.6 (0x00007f435c9e8000)
/lib64/ld-linux-x86-64.so.2 (0x00007f435cca7000)
Specifically linking libtinfow.so instead of libtinfo.so resolves the
issue.
All instances of '...' above represent the path to the installed ncurses
for Spack.
scheibelp
pushed a commit
that referenced
this pull request
Oct 24, 2020
The r-devtools package was not installable due to a few issues. - The rstudioapi spec was for 0.11.0 but the rstudioapi version is actually 0.11. This caused an error during concretization. - Set r-usethis to depend on rlang@0.4.3: rather than r-lang@0.4.3. - Set r-usethis to depend on r-gh@1.1.0: rather than r-gh@1.1.0. - Added version r-gh-1.1.0 as it is not currently present in spack.
scheibelp
pushed a commit
that referenced
this pull request
Jun 1, 2021
Before: ``` $ hyperfine '~/spack/bin/spack -e . build-env rocfft' Benchmark #1: ~/spack/bin/spack -e . build-env rocfft Time (mean ± σ): 1.593 s ± 0.016 s [User: 1.468 s, System: 0.126 s] Range (min … max): 1.575 s … 1.628 s 10 runs ``` After: ``` $ hyperfine '~/spack/bin/spack -e . build-env rocfft' Benchmark #1: ~/spack/bin/spack -e . build-env rocfft Time (mean ± σ): 1.407 s ± 0.020 s [User: 1.280 s, System: 0.127 s] Range (min … max): 1.393 s … 1.455 s 10 runs ```
scheibelp
pushed a commit
that referenced
this pull request
Oct 15, 2021
Git 2.24 introduced a feature flag for repositories with many files, see: https://github.blog/2019-11-03-highlights-from-git-2-24/#feature-macros Since Spack's Git repository contains roughly 8,500 files, it can be worthwhile to enable this, especially on slow file systems such as NFS: ``` $ hyperfine --warmup 3 'cd spack-default; git status' 'cd spack-manyfiles; git status' Benchmark #1: cd spack-default; git status Time (mean ± σ): 3.388 s ± 0.095 s [User: 256.2 ms, System: 625.8 ms] Range (min … max): 3.168 s … 3.535 s 10 runs Benchmark #2: cd spack-manyfiles; git status Time (mean ± σ): 168.7 ms ± 10.9 ms [User: 98.6 ms, System: 126.1 ms] Range (min … max): 144.8 ms … 188.0 ms 19 runs Summary 'cd spack-manyfiles; git status' ran 20.09 ± 1.42 times faster than 'cd spack-default; git status' ```
scheibelp
pushed a commit
that referenced
this pull request
May 8, 2023
1. support version 3.1.3, which now depends on sundials@6 2. support version 3.1.2:, which broke the two patch files and therefore the two patch files have been replaced by more flexible filter_file() commands inside a patch() function. 3. rename the variant for python extension from using the package name "+pyuqtk" to the more standard "+python" 4. add maintainers @omsai and the upstream developer @bjdebus who offered to help with the spack packaging. 5. swig should only be a build-time dependency. swig is only necessary until @:3.1.0 6. confirmed python dependencies are correct by inspecting imports, subset python dependencies type to build, run, and confirmed all 31 build-time tests pass including the 9 python tests: ```console $ spack env create uqtk-dev $ spack add uqtk@3.1.3 $ spack install --test root && cat $(spack location -i uqtk)/.spack/install-time-test-log.txt ==> Testing package uqtk-3.1.3-nok6fut ==> [2023-04-19-14:56:25.005361] Running build-time tests ==> [2023-04-19-14:56:25.005536] RUN-TESTS: build-time tests [check] ==> [2023-04-19-14:56:25.009543] '/home/omsai/src/spack/opt/spack/linux-pureos10-skylake/gcc-10.2.1/gmake-4.4.1-b6g4apmfvxz3bn4eabh37dehcrg65fj7/bin/make' '-j4' '-n' 'test' ==> [2023-04-19-14:56:25.014903] '/home/omsai/src/spack/opt/spack/linux-pureos10-skylake/gcc-10.2.1/gmake-4.4.1-b6g4apmfvxz3bn4eabh37dehcrg65fj7/bin/make' '-j4' 'test' Running tests... /home/omsai/src/spack/opt/spack/linux-pureos10-skylake/gcc-10.2.1/cmake-3.26.3-zjmsfz23j5l4ytniz26uzvxonlu5qebr/bin/ctest --force-new-ctest-process Test project /tmp/omsai/spack-stage/spack-stage-uqtk-3.1.3-nok6fut47h42cnaau7wkoohgqy5f2qqa/spack-build-nok6fut Start 1: ArrayReadAndWrite Start 2: ArrayDelColumn Start 3: Array1DMiscTest Start 4: Array2DMiscTest 1/31 Test #1: ArrayReadAndWrite ................ Passed 0.01 sec Start 5: ArraySortTest 2/31 Test #2: ArrayDelColumn ................... Passed 0.01 sec Start 6: MultiIndexTest 3/31 Test #3: Array1DMiscTest .................. Passed 0.01 sec Start 7: CorrTest 4/31 Test #4: Array2DMiscTest .................. Passed 0.01 sec Start 8: QuadLUTest 5/31 Test #5: ArraySortTest .................... Passed 0.02 sec Start 9: MCMC2dTest 6/31 Test #6: MultiIndexTest ................... Passed 0.01 sec Start 10: MCMCRandomTest 7/31 Test #8: QuadLUTest ....................... Passed 0.02 sec Start 11: MCMCNestedTest 8/31 Test #10: MCMCRandomTest ................... Passed 0.02 sec Start 12: Deriv1dTest 9/31 Test #12: Deriv1dTest ...................... Passed 0.01 sec Start 13: SecondDeriv1dTest 10/31 Test #13: SecondDeriv1dTest ................ Passed 0.01 sec Start 14: GradHessianTest 11/31 Test #11: MCMCNestedTest ................... Passed 0.03 sec Start 15: GradientPCETest 12/31 Test #14: GradHessianTest .................. Passed 0.01 sec Start 16: PCE1dTest 13/31 Test #15: GradientPCETest .................. Passed 0.01 sec Start 17: PCEImplTest 14/31 Test #16: PCE1dTest ........................ Passed 0.01 sec Start 18: PCELogTest 15/31 Test #18: PCELogTest ....................... Passed 0.01 sec Start 19: Hessian2dTest 16/31 Test #19: Hessian2dTest .................... Passed 0.01 sec Start 20: BCS1dTest 17/31 Test #20: BCS1dTest ........................ Passed 0.01 sec Start 21: BCS2dTest 18/31 Test #21: BCS2dTest ........................ Passed 0.01 sec Start 22: LowRankRegrTest 19/31 Test #22: LowRankRegrTest .................. Passed 0.01 sec Start 23: PyModTest 20/31 Test #17: PCEImplTest ...................... Passed 0.07 sec Start 24: PyArrayTest 21/31 Test #23: PyModTest ........................ Passed 0.08 sec Start 25: PyArrayTest2 22/31 Test #25: PyArrayTest2 ..................... Passed 0.30 sec Start 26: PyQuadTest 23/31 Test #24: PyArrayTest ...................... Passed 1.44 sec Start 27: PyBCSTest1D 24/31 Test #26: PyQuadTest ....................... Passed 1.68 sec Start 28: PyBCSTest2D 25/31 Test #27: PyBCSTest1D ...................... Passed 1.66 sec Start 29: PyBADPTest 26/31 Test #7: CorrTest ......................... Passed 3.43 sec Start 30: PyRegressionTest 27/31 Test #28: PyBCSTest2D ...................... Passed 1.50 sec Start 31: PyGalerkinTest 28/31 Test #9: MCMC2dTest ....................... Passed 3.90 sec 29/31 Test #29: PyBADPTest ....................... Passed 1.66 sec 30/31 Test #30: PyRegressionTest ................. Passed 1.72 sec 31/31 Test #31: PyGalerkinTest ................... Passed 1.63 sec 100% tests passed, 0 tests failed out of 31 Total Test time (real) = 5.35 sec ==> [2023-04-19-14:56:30.382797] '/home/omsai/src/spack/opt/spack/linux-pureos10-skylake/gcc-10.2.1/gmake-4.4.1-b6g4apmfvxz3bn4eabh37dehcrg65fj7/bin/make' '-j4' '-n' 'check' ==> [2023-04-19-14:56:30.385983] Target 'check' not found in Makefile ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
@scheibelp
I ran out of inodes on our supercomputer, so I needed a way to uninstall obsolete Spack packages. Luckily, EVERYTHING I do is in a Spack Environment. So obsolete packages can be determined by looking for any packages NOT in one of my Spack Environments. The user interface is:
At that point, it works like any other
spack uninstallcommand. Mechanical garbage collection is one (more) advantage of using Spack Environments.I'm putting this as a separate PR because I'm really really really hoping Spack Environments will be merged Real Soon Now. I don't want to delay it any further with new features.