Minor CUDA driver compatibility issue with updated GPU sanity check

[JonNavarroGaritano]

On MN5 we are currently running with CUDA driver version 12.2. After the recent update to the GPU sanity check in the software-layer scripts (see PR EESSI/software-layer-scripts#202), modules built for CUDA 12.4.0 are now failing to load due to a stricter driver version check.

Loading modules such as:

module load LAMMPS/29Aug2024_update2-foss-2023b-kokkos-CUDA-12.4.0

results in the following error:

Your driver CUDA version is 12.2 but the module you want to load requires CUDA 12.4.0.

This looks like a minor version compatibility scenario (12.2 vs 12.4), where execution may still be valid but is now blocked by the updated check. This affects module usability on systems where the CUDA driver cannot be upgraded in sync with module updates.

Thanks for the update and for looking into this.

[boegel]

I've transferred this issue, the bot repository isn't really the right place.

A better option would've been to open a support ticket for this, see https://eessi.io/docs/support/, but this issue will do for now, no need to splinter the discussion further (since this is also being discussed in the EESSI Slack).

@ocaisa Is this something you can follow up on?

[casparvl]

Let me just paste here what I mentioned on chat:

@Alan O'Cais (CECAM) I'm wondering if we are too strict in raising an error on this. What I understand from the table at the bottom of https://docs.nvidia.com/deploy/cuda-compatibility/why-cuda-compatibility.html is that Jon is effectively in scenario 2: Driver is older than the CUDA runtime (provided by the CUDA toolkit), but it has the same major version. Then, you have what NVIDIA calls 'Minor version compatibility', which is compatibility with some caveats (see https://docs.nvidia.com/deploy/cuda-compatibility/minor-version-compatibility.html for details). The limitations are listed here https://docs.nvidia.com/deploy/cuda-compatibility/minor-version-compatibility.html#application-considerations-for-minor-version-compatibility .

1. Limited feature set: if a feature introduced in a new CUDA toolkit spans both the toolkit and the driver, and if this is actually used in a code, this could cause a cudaErrorCallRequiresNewerDriver error. We don't know this from the EESSI side, as we don't know what codes may or may not use. My gut feeling is that these cases will be rather exceptional (and the error is fairly descriptive)
2. Applications that compile device code to PTX will not work on older drivers. I'm pretty sure PyTorch does this when you run torch.compile.
3. Requires target architecture argument to nvcc (-arch=sm_xx). I think this should be fine, we'd at least like this to happen in all builds we do (whether the build systems -actually- do this is a bit hard to guarantee of course).

So yeah, it might be an issue in some cases - but there also might be many cases where it would just work fine.

Should we consider raising an LmodWarning instead when where in what Nvidia calls the 'Minor Version Compatibilty' regime? (and maybe just link to the nvidia docs to explain what that means...)

@ocaisa agreed with that approach, and remarked that making EESSI compatibility library aware (which means we could be even less restrictive) is possible but tricky. I agree. For now, I'd say:

• Minor Version Compatiblity regime = LmodWarning with link to nvidia docs
• Major Version Compatibility regime = LmodError (the current one, essentially)

As a temporary workaround, one can set EESSI_OVERRIDE_GPU_CHECK to make sure this check isn't done - but I wouldn't recommend that as a general solution, as you'd also miss this check in case of major version incompatiblity.