LAMMPS on multiple GPUs

[Felixrccs]

Everything works on one gpu, however I would like to run my LAMMPS simulation over multiple GPUs.
My lammps submission command for two GPUs:
srun -n 4 lmp -partition 1 1 1 1 -l lammps.log -sc screen -k on g 2 -sf kk -i lammps.in

If I run it on multiple GPUs, LAMMPS returns this error back:

Starting MPS on ravg1112
Exception: Specified device cuda:0 does not match device of data cuda:1
Exception raised from make_tensor at aten/src/ATen/Functions.cpp:26 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x14f6528553cb in /u/----/LAMMPS/lammps_mace_v1/libtorch-gpu/lib/libc10.so)
frame #1: at::TensorMaker::make_tensor() + 0xa1d (0x14f604c2641d in /u/----/LAMMPS/lammps_mace_v1/libtorch-gpu/lib/libtorch_cpu.so)
frame #2: torch::from_blob(void*, c10::ArrayRef<long>, c10::TensorOptions const&) + 0xfb (0x14f656649e2b in /u/----/LAMMPS/lammps_mace_v1/lammps/build-kokkos-cuda/liblammps.so.0)
frame #3: LAMMPS_NS::PairMACEKokkos<Kokkos::Cuda>::compute(int, int) + 0xce1 (0x14f6566620f1 in /u/----/LAMMPS/lammps_mace_v1/lammps/build-kokkos-cuda/liblammps.so.0)
frame #4: LAMMPS_NS::VerletKokkos::setup(int) + 0x5b2 (0x14f655d85fb2 in /u/----/LAMMPS/lammps_mace_v1/lammps/build-kokkos-cuda/liblammps.so.0)
frame #5: LAMMPS_NS::Temper::command(int, char**) + 0x6c4 (0x14f655583c84 in /u/----/LAMMPS/lammps_mace_v1/lammps/build-kokkos-cuda/liblammps.so.0)
frame #6: LAMMPS_NS::Input::execute_command() + 0xaec (0x14f6551e55ec in /u/----/LAMMPS/lammps_mace_v1/lammps/build-kokkos-cuda/liblammps.so.0)
frame #7: LAMMPS_NS::Input::file() + 0x155 (0x14f6551e5995 in /u/----/LAMMPS/lammps_mace_v1/lammps/build-kokkos-cuda/liblammps.so.0)
frame #8: /u/----/LAMMPS/lammps_mace_v1/lammps/build-kokkos-cuda/lmp() [0x4049e8]
frame #9: __libc_start_main + 0xef (0x14f653e3e24d in /lib64/libc.so.6)
frame #10: /u/----/LAMMPS/lammps_mace_v1/lammps/build-kokkos-cuda/lmp() [0x404b9a]

Exception: Specified device cuda:0 does not match device of data cuda:1
Exception raised from make_tensor at aten/src/ATen/Functions.cpp:26 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x14dd2ea553cb in /u/----/LAMMPS/lammps_mace_v1/libtorch-gpu/lib/libc10.so)
frame #1: at::TensorMaker::make_tensor() + 0xa1d (0x14dce0e2641d in /u/----/LAMMPS/lammps_mace_v1/libtorch-gpu/lib/libtorch_cpu.so)
frame #2: torch::from_blob(void*, c10::ArrayRef<long>, c10::TensorOptions const&) + 0xfb (0x14dd32849e2b in /u/----/LAMMPS/lammps_mace_v1/lammps/build-kokkos-cuda/liblammps.so.0)
frame #3: LAMMPS_NS::PairMACEKokkos<Kokkos::Cuda>::compute(int, int) + 0xce1 (0x14dd328620f1 in /u/----/LAMMPS/lammps_mace_v1/lammps/build-kokkos-cuda/liblammps.so.0)
frame #4: LAMMPS_NS::VerletKokkos::setup(int) + 0x5b2 (0x14dd31f85fb2 in /u/----/LAMMPS/lammps_mace_v1/lammps/build-kokkos-cuda/liblammps.so.0)
frame #5: LAMMPS_NS::Temper::command(int, char**) + 0x6c4 (0x14dd31783c84 in /u/----/LAMMPS/lammps_mace_v1/lammps/build-kokkos-cuda/liblammps.so.0)
frame #6: LAMMPS_NS::Input::execute_command() + 0xaec (0x14dd313e55ec in /u/----/LAMMPS/lammps_mace_v1/lammps/build-kokkos-cuda/liblammps.so.0)
frame #7: LAMMPS_NS::Input::file() + 0x155 (0x14dd313e5995 in /u/----/LAMMPS/lammps_mace_v1/lammps/build-kokkos-cuda/liblammps.so.0)
frame #8: /u/----/LAMMPS/lammps_mace_v1/lammps/build-kokkos-cuda/lmp() [0x4049e8]
frame #9: __libc_start_main + 0xef (0x14dd3003e24d in /lib64/libc.so.6)
frame #10: /u/----/LAMMPS/lammps_mace_v1/lammps/build-kokkos-cuda/lmp() [0x404b9a]

Abort(1) on node 1 (rank 1 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1
Abort(1) on node 3 (rank 3 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 3
slurmstepd: error: *** STEP 7590002.0 ON ravg1112 CANCELLED AT 2023-10-19T14:03:48 ***
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
srun: error: ravg1112: tasks 0,2: Killed
srun: launch/slurm: _step_signal: Terminating StepId=7590002.0
srun: error: ravg1112: task 3: Killed
srun: error: ravg1112: task 1: Killed

[ilyes319]

multi-GPU lammps is an untested beta feature, hopefully we will make progress on that soon.

[Felixrccs]

I was told that I probably formulated my issue not precise enough. So I want to run LAMMPS REPLICA EXCHANGE simulations. This means that LAMMPS creates multiple MD instances and runs them in parallel. So I want to stick to pair_style mace no_domain_decomposition. MACE-LAMMPS also manages to distribute the MD instances over one GPU. However when distributing the MD instances over multiple GPUs I run into this Bug. From the error I guess there is something build into the LAMMPS-MACE package that checks if the memory is identical for both GPUs, which is not necessary for my application.

I would be very happy if you take another look at this.

[Felixrccs]

@wcwitt @ilyes319 Any idea how to fix this?
At this point this is the only thing keeping me from starting large scale production runs.

[wcwitt]

I've never run replica exchange in LAMMPS, so I'll need to dig into the internals to understand why it's breaking. Would you please send a complete (but simple) example, including a model? I'll use that to figure it out. Thanks

[ilyes319]

The problem seems to be linked to how to the model is sent to different GPUs. If nothing is done, libtorch probably loads the model on the first GPU of the node. I think there is a probably a line to add to distribute the models on the right GPUs.

[wcwitt]

Yeah I'm sure it's not terribly complicated. There is already this logic which assigns the model to the GPU associated with the local MPI rank. That's how the domain decomposition works.

  if (!torch::cuda::is_available()) {
    std::cout << "CUDA unavailable, setting device type to torch::kCPU." << std::endl;
    device = c10::Device(torch::kCPU);
  } else {
    std::cout << "CUDA found, setting device type to torch::kCUDA." << std::endl;
    MPI_Comm local;
    MPI_Comm_split_type(world, MPI_COMM_TYPE_SHARED, 0, MPI_INFO_NULL, &local);
    int localrank;
    MPI_Comm_rank(local, &localrank);
    device = c10::Device(torch::kCUDA,localrank);
  }

I'd still like a self-contained example.

[Felixrccs]

Thanks a lot
Test.zip
Here is the Test case, I simplified it as far as possible.

[wcwitt]

Thanks for the example - I've looked this some now. What happens if you try without Kokkos? So, adapting your input command:

mpirun -n 2 /work/Software/lammps_mace/lammps/build-kokkos-cuda/lmp -partition 1 1 -l lammps.log -sc screen -in lammps.temper > lammps.out

[Felixrccs]

It returns no error. However it only uses a single GPU for all replicas (instances) instead of distributing it over the GPUs.
--> without kokkos it gives me the same result compared to using kokkos with one gpu (-k on g 2 -sf kk)

[wcwitt]

Okay thanks. Let’s keep Kokkos off until we figure this out - it complicates things. I confirmed yesterday that the distributed multi-GPU is working as intended, so the problem is specific to replica somehow. I couldn’t manage to run your example myself; it kept failing with MPI errors right at the beginning. I think it was a problem with my install - can you confirm which packages are required for the replica and partitioning? Also, do you know what happens if you try a CPU only install and distribute over CPUs? Does that work as expected?

…

On Wed, 22 Nov 2023 at 04:01, Felix Riccius ***@***.***> wrote: It returns no error. However it only uses a single GPU for all replicas (instances) instead of distributing it over the GPUs. --> without kokkos it gives me the same result compared to using kokkos with one gpu (-k on g 2 -sf kk) — Reply to this email directly, view it on GitHub <#195 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACXHHTQ6C4JJI6SLTAX7FFDYFW5N5AVCNFSM6AAAAAA6HDRU42VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRSGM2TQMRTG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Felixrccs]

I guess your install is missing the REPLICA package

Here is my current install

module purge
module load gcc/10 mkl/2022.2 gsl/2.4 impi/2021.4 fftw-mpi/3.3.9
module load anaconda/3/2023.03
module load cuda/11.6 cudnn/8.8.1
module load cmake/3.18

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$MKLROOT/lib/intel64
source ${MKLROOT}/env/vars.sh intel64

cmake \
    -D CMAKE_BUILD_TYPE=Release \
    -D CMAKE_INSTALL_PREFIX=$(pwd) \
    -D BUILD_MPI=yes \
    -D BUILD_OMP=yes \
    -D BUILD_SHARED_LIBS=yes \
    -D LAMMPS_EXCEPTIONS=yes \
    -D PKG_KOKKOS=yes \
    -D Kokkos_ARCH_AMPERE80=yes \
    -D Kokkos_ARCH_AMDAVX=yes \
    -D Kokkos_ENABLE_CUDA=yes \
    -D Kokkos_ENABLE_OPENMP=yes \
    -D Kokkos_ENABLE_DEBUG=no \
    -D Kokkos_ENABLE_DEBUG_BOUNDS_CHECK=no \
    -D Kokkos_ENABLE_CUDA_UVM=no \
    -D CMAKE_CXX_COMPILER=$(pwd)/../lib/kokkos/bin/nvcc_wrapper \
    -D PKG_ML-MACE=yes \
    -D PKG_MOLECULE=yes \
    -D PKG_REPLICA=yes \
    -D PKG_KSPACE=yes \
    -D PKG_RIGID=yes \
    -D CMAKE_PREFIX_PATH=$(pwd)/../../libtorch-gpu \
    ../cmake

make -j 12

I never installed the CPU only version, I will update you as soon I tried that.

[Felixrccs]

I've now tried the CPU installation and everything seems to work fine:

I tried it on one node and I manage to either run it with no domain decomposition and one task per replica and also with domain decomposition and multiple tasks per replica.

[Felixrccs]

Did you have any progress and managed to run the simulation on your installation? And do you have anything else I should try out?

[wcwitt]

Thanks - I'm at conference this week, so haven't had much time unfortunately. But there are some people here who may know the answer immediately - I'll track them down.