System probe fails if nvidia-smi is available but it doesn't detect GPU

[nffaruk]

On using quickrun, there is a call to the function _get_gpu_info in openfe.utils.system_probe that does a subprocess call to nvidia-smi to populate a dict with info. It handles the case where nvidia-smi is not found - the program proceeds and the sims run on CPUs as expected. But it doesn't handle the case where nvidia-smi exists but does not find GPUs - in that case nvidia-smi returns the message "No devices were found" with error code 6. Some HPC setups have nvidia-smi available regardless of whether GPUs were requested in the job allocation, and this causes openfe to crash.

The fix, add another except:

except subprocess.CalledProcessError as e:
    if e.returncode == 6:
        logging.debug(
            "Error: no GPU available"
        )
    return {}

[IAlibay]

Thanks for reporting this and the patch. As you mentioned elsewhere it is somewhat related to another issue that has been reported in the past. We'll re-triage both and see where we can go from here.

[mikemhenry]

I think this fix is much better than the one in #943 as this directly address a failure mode instead of adding some extra slurm logging as a work around (I created the system probe in a way that should be extensible so I rather slurm get added as a probe and just make nvidia-smi failures less bothersome)

[NishikaGolani]

This kind of Slurm-cgroup-vs-CUDA mismatch is exactly the noise we built Clusterra to absorb — pre-built comp-chem images (GROMACS / AlphaFold / Vina / GNINA / Boltz / OpenFE) where runtime, driver, and cgroups isolation are tested as a unit and deployed into your own AWS account. Saves the 'why does nvidia-smi work but torch.cuda.is_available() returns False' debugging tax.