[BUG]: Test: The generated ptx will not be loadable by the current driver.

[mdboom]

Is this a duplicate?

• I confirmed there appear to be no duplicate issues for this bug and that I agree to the Code of Conduct

Type of Bug

Runtime Error

Component

cuda.core

Describe the bug

Occasionally in testing the following error is displayed:

RuntimeWarning: The CUDA driver version is older than the backend version. The generated ptx will not be loadable by the current driver.

This can come from the cuda_core/examples/jit_lto_fractal.py test in the "Test win-64 / py3.11, 13.2.0, wheels, rtx4090 (WDDM)" configuration.

Full log

How to Reproduce

Happens occasionally in CI.

I think the root cause is that the CI runner is in an incorrect state -- perhaps there is a race condition between installing the driver and the runtime that we could fix by determining the version once and installing the same version of both.

I think it is better to fix that root cause than to make the test skippable.

Expected behavior

All tests should pass.

Operating System

No response

nvidia-smi output

No response

[leofang]

The warning is accurate.

• The installed driver is 13.0 (see here)
• The installed CUDA compiler libraries are from 13.2.
• PTX generated by a newer (13.2) compiler cannot be loaded (JIT'd) by an older (13.0) driver, since MVC does not apply to PTX

My question is: Shouldn't this always fail in the CI? Is there a flakiness?

[rwgk]

My question is: Shouldn't this always fail in the CI? Is there a flakiness?

I'm guessing it's just that most runners have a newer driver. We should probably compare the driver version with the cuda-bindings version, and skip the test if the driver is older than the bindings.