Testing code on LUMI

[huangchieh]

Cod test is needed on LUMI

[huangchieh]

How to use LUMI

https://docs.lumi-supercomputer.eu/

module load LUMI/23.09
module load EasyBuild-user
module load LUMI PyTorch/2.1.0-rocm-5.6.1-python-3.10-singularity-20231123
ml
singularity shell $SIF
$WITH_CONDA 

File transfer

scp -i $HOME/.ssh/lumi -r Water-Au111/ huangjie@lumi.csc.fi:/scratch/project_462000398/jie

Ref. : https://docs.lumi-supercomputer.eu/firststeps/movingdata/

[huangchieh]

How to use Python and install Python packages

https://docs.lumi-supercomputer.eu/software/installing/container-wrapper/
And transferred Triton Base conda env to LUMI by using

module load LUMI
module load lumi-container-wrapper

conda-containerize new --prefix /scratch/project_4xxxxx/jie/.jieprograms/conda triton_base.yml

cd /scratch/project_462000398/jie/.jieprograms
conda-containerize update /scratch/project_462000398/jie/.jieprograms/conda --post-install add_apps.txt

add_apps.txt

pip install ipython

[huangchieh]

FATAL: container creation failed: mount /usr/lib64/libcxi.so.1->/usr/lib64/libcxi.so.1 error: while mounting /usr/lib64/libcxi.so.1: destination /usr/lib64/libcxi.so.1 doesn't exist in container

Error disappeared next day. Don't know why.

How to run job interactively

srun --account=project_462000398 --partition standard-g --gres=gpu:4 -c 8  --time=4:00:00 --mem=6000M --pty bash

http://slurmlearning.deic.dk/
Some useful slurm commands

squeue

[huangchieh]

srun --account=project_462000398 --partition standard-g --gpus-per-node=8 --mem=128G --nodes=2  --ntasks-per-node=1 -c 32  --time=4:00:00  --pty bash

[huangchieh]

srun torchrun \
    --nnodes $SLURM_NNODES \
    --nproc_per_node $num_gpus \
    --rdzv_backend c10d \
    --rdzv_id 7623 \
    --rdzv_endpoint $head_node_ip:29500 \
    --max_restarts 0 \
    fit_posnet.py \
        --train True \
        --test True \
        --predict True \
        --epochs 1000 \
        --num_workers $num_workers \
        --batch_size 1 \
        --pred_batches 30 \
        --lr 1e-4 \
        --zmin -10.0 \
        --z_lims -2.9 0.5 \
        --peak_std 0.20 \
        --data_dir /scratch/project_462000398/jie/Water-Au111 \
        --urls/train "Water-K-{1..10}_train_{0..31}.tar" \
        --urls/val "Water-K-{1..8}_val_{0..7}.tar" \
        --urls/test "Water-K-{1..10}_test_{0..7}.tar"

[huangchieh]

singularity pull docker://ubuntu:22.04
srun --partition=standard-g --account=project_462000398 singularity exec ubuntu_22.04.sif cat /etc/os-release
srun --partition=standard-g --account=project_462000398 singularity exec --bind /scratch/project_462000398 ubuntu_22.04.sif ls /scratch/project_462000398

[huangchieh]

mkdir -p /tmp/$USER
export SINGULARITY_TMPDIR=/tmp/$USER
export SINGULARITY_CACHEDIR=/tmp/$USER
singularity pull docker://rocm/pytorch:rocm5.4.1_ubuntu20.04_py3.7_pytorch_1.12.1

singularity exec -B $SCRATCH:$SCRATCH pytorch_rocm5.4.1_ubuntu20.04_py3.7_pytorch_1.12.1.sif bash
python -m venv pt_rocm5.4.1_env --system-site-packages
. pt_rocm5.4.1_env/bin/activate
pip install python-hostlist

module load LUMI/22.08 partition/G
module load EasyBuild-user
eb aws-ofi-rccl-66b3b31-cpeGNU-22.08.eb -r