CUDA version

[nicolazilio]

Hello there,

I would just like to report an issue when running DNAscent on certain GPUs.

I ran the following

#! /bin/bash

#SBATCH --job-name="dnascent detect"
#SBATCH --partition=gpulong
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=8
#SBATCH --mem=32G
#SBATCH --nodes=1
#SBATCH --output=dnascentdetect-%A.log
#SBATCH --error=dnascentdetect-%A.log

CONTAINER_PATH=/home/xyz/DNAscent_do_not_use.sif
BASE_DIR=/fsimb/groups/imb-xyz/Nanopore/project
REFERENCE_PATH=$BASE_DIR/../sacCer3.fa
INDEX_PATH=$BASE_DIR/../index.dnascent

for FILE in $BASE_DIR/*plusDox_plusBrdU.fast.bam*;
do 
    NAME=`basename $FILE`
    NAME=${NAME::-4}
    singularity run --contain --nv --nvccli --writable-tmpfs --bind /home,/fsimb,/media,/mnt,/tmp $CONTAINER_PATH detect --GPU 0 -t 8 -q 8 -b $BASE_DIR/$NAME.bam -r $REFERENCE_PATH -i $INDEX_PATH -o /$BASE_DIR/$NAME.detect.bam
done

and started seeing the following warning (repeated 1000s of times)

W tensorflow/stream_executor/gpu/asm_compiler.cc:235] Your CUDA software stack is old. We fallback to the NVIDIA driver for some compilation. Update your CUDA version to get the best performance. The ptxas error was: ptxas fatal : Value 'sm_89' is not defined for option 'gpu-name'

The processing time skyrocketed to >5000 h.

Our sysadmin pinned this down to the version of CUDA running inside the container

Yeah, your container includes CUDA 11.0.3 (ubuntu20.04) while our
Nodes feature CUDA 12.7 (which is backward compatible). Our Ada Lovelace based GPUs
should be supported from CUDA 11.8. So if you want to get rid of this error/warning
message you have to update the singularity container. Running it on HPCGPU (which features older GPUs)
should work as well.

Running your original container on the older GPUs did indeed work: it did not raise any warnings and, importantly, meant that the job only took 26 h.

Our sysadmin also tried to update the CUDA version of your container and rebuild it master...chdietrich:DNAscent:master. This cleared the warnings and the job also took 26 h.

Thanks for the great work with DNAscent!

[maguscott]

I am also having a similar issue as our GPUs are being updated. @nicolazilio Have you got a container image you'd be willing to share with the CUDA updates?

[nicolazilio]

This worked for me master...chdietrich:DNAscent:master