CUDA.jl takes much much longer to load on Julia v1.12

[oschulz]

I'm seeing much longer package load times of CUDA.jl on Julia v1.12, compared to v1.11.

I've timed this on two very different systems (using fresh Julia installations on a local NVMe temp disk, an empty Pkg depot, etc):

Server, NVIDIA Grace-Hopper GH200, Ubuntu 24.04

Julia v1.11

$ time julia-1.11.7/bin/julia -e 'using CUDA'
real    0m3.108s
user    0m5.665s
sys     0m0.721s

julia> @time @time_imports import CUDA
[... fairly quick ...]
    535.6 ms  CUDA 3.41% compilation time
      0.3 ms  Atomix → AtomixCUDAExt
[delay here]
  2.626289 seconds (3.14 M allocations: 217.201 MiB, 10.50% gc time, 18.88% compilation time: 6% of which was recompilation)

Julia v1.12

$ time julia-1.12.2/bin/julia -e 'using CUDA'
real    0m14.615s
user    0m17.187s
sys     0m0.744s

julia> @time @time_imports import CUDA
[... fairly quick ...]
   1133.9 ms  CUDA 1.36% compilation time
      0.3 ms  Atomix → AtomixCUDAExt
[long delay here]
 14.621508 seconds (12.65 M allocations: 705.328 MiB, 4.97% gc time, 80.55% compilation time: 96% of which was recompilation)

Laptop, NVIDIA RTX 2000 Ada, Ubuntu 24.04

Julia v1.11

$ time julia-1.11 -e 'using CUDA'
real	0m7.348s
user	0m8.977s
sys	0m1.891s

julia> @time @time_imports import CUDA
[... fairly quick ...]
   1208.8 ms  CUDA 37.99% compilation time
      0.7 ms  Atomix → AtomixCUDAExt
[delay here]
  7.510932 seconds (3.92 M allocations: 253.427 MiB, 2.97% gc time, 43.74% compilation time: 19% of which was recompilation)

Julia v1.12

$ time julia-1.12 -e 'using CUDA'
real	0m37.247s
user	0m32.124s
sys	0m8.350s

julia> @time @time_imports import CUDA
[... fairly quick ...]
   1851.8 ms  CUDA 11.05% compilation time
      0.4 ms  Atomix → AtomixCUDAExt
[long delay here]
 29.482076 seconds (12.97 M allocations: 716.697 MiB, 1.12% gc time, 82.29% compilation time: 96% of which was recompilation)

When timing with @time @time_imports import CUDA, there is a long delay after the output of @time_imports ends, that's where most of the time is spent.

Is this a known issue?

(Not sure this is strictly a bug, but package load times near 30 seconds (sometimes almost 40, it varies a lot) can severely affect usability, especially when giving a presentation ...)

[ViralBShah]

There's a much larger number of allocations. Wonder if that points to invalidations.

[JamesWrigley]

FWIW, on my machine without a GPU in a project with JuliaData/SentinelArrays.jl#119 and JuliaArrays/StaticArrays.jl#1325 (i.e. invalidations down from ~17,000 to ~10,000) I see basically no difference in load time on Julia 1.12, both are ~5s.

On a cluster node with a GPU there is a slight improvement. With released versions:

$ hyperfine -w 3 'julia --startup-file=no -e "@time import CUDA"'          
Benchmark 1: julia --startup-file=no -e "@time import CUDA"
  Time (mean ± σ):     33.535 s ±  0.276 s    [User: 45.297 s, System: 2.320 s]
  Range (min … max):   33.184 s … 33.979 s    10 runs

With the above PRs:

$ hyperfine -w 3 'julia --project=@scratch --startup-file=no -e "@time import CUDA"'  
Benchmark 1: julia --project=@scratch --startup-file=no -e "@time import CUDA"
  Time (mean ± σ):     27.070 s ±  0.131 s    [User: 38.509 s, System: 2.092 s]
  Range (min … max):   26.920 s … 27.357 s    10 runs

So about 18% better. But it's still pretty slow, and it's weird that both @time and @time @eval are reporting wrong numbers.

[JamesWrigley]

Turns out it's this call to NVTX.activate():

CUDA.jl/src/initialization.jl

Line 156 in 5d9474a

NVTX.activate()

With that commented out we're back to no-GPU times:

$ hyperfine -w 1 'julia --project=@scratch --startup-file=no -e "import CUDA"'
Benchmark 1: julia --project=@scratch --startup-file=no -e "import CUDA"                                               
  Time (mean ± σ):      5.628 s ±  0.104 s    [User: 15.537 s, System: 1.439 s]
  Range (min … max):    5.538 s …  5.815 s    10 runs 

[JamesWrigley]

I am confuse 🤨

$ hyperfine -w 1 -r 5 "julia --project=@scratch --startup-file=no -e 'import NVTX; @time NVTX.activate()'"       
Benchmark 1: julia --project=@scratch --startup-file=no -e 'import NVTX; @time NVTX.activate()'
  Time (mean ± σ):     466.4 ms ±   3.3 ms    [User: 2404.9 ms, System: 266.0 ms]
  Range (min … max):   462.6 ms … 471.6 ms    5 runs

$ hyperfine -w 1 -r 5 "julia --project=@scratch --startup-file=no -e 'import REPL; import NVTX; NVTX.activate()'"     
Benchmark 1: julia --project=@scratch --startup-file=no -e 'import REPL; import NVTX; NVTX.activate()'
  Time (mean ± σ):     602.7 ms ±   5.4 ms    [User: 2476.6 ms, System: 358.9 ms]
  Range (min … max):   597.5 ms … 611.1 ms    5 runs

$ hyperfine -w 1 -r 5 "julia --project=@scratch --startup-file=no -e 'import REPL; import NVTX; @time NVTX.activate()'"
Benchmark 1: julia --project=@scratch --startup-file=no -e 'import REPL; import NVTX; @time NVTX.activate()'
  Time (mean ± σ):     13.665 s ±  0.031 s    [User: 16.984 s, System: 0.526 s]
  Range (min … max):   13.624 s … 13.700 s    5 runs

[JamesWrigley]

Narrowed down to this line in NVTX: https://github.com/JuliaGPU/NVTX.jl/blob/5c442700ecf9d6ed37d68c7b4a5d55ebab654550/src/julia.jl#L125
Which was added in JuliaGPU/NVTX.jl#44, but the SnoopCompileCore code it was based on has now changed: https://github.com/JuliaDebug/SnoopCompile.jl/blob/d9a77ad6ab8bbda90f7cf133e7e88ca0a344b7f7/SnoopCompileCore/src/snoop_inference.jl#L17

[oschulz]

Thanks a lot @JamesWrigley !