Turning on GPU gives PTX error

[Musty1]

I keep getting cuda error 222 the provided PTX was compiled with an unsupported toolchain after I enable GPU.
I have a VM with nvidia GPU installed. I tried every version of CUDA and reinstalled pytorch, however, still getting this error.

Can someone please help with this?

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.105.17 Driver Version: 525.105.17 CUDA Version: 12.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA A30-12C On | 00000000:02:00.0 Off | 0 |
| N/A N/A P0 N/A / N/A | 0MiB / 12288MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------

torch.version.cuda
'12.1'

I have tried CUDA version 12.1, 12.0, 11.8 yet still same issue.