How to use CUDA containers

The following CUDA Toolkit versions are available on the Grid as Apptainer containers rather than software modules. These containers include the CUDA Toolkit (including nvcc) and can be used to compile and run CUDA applications on the Grid's GPU nodes. Simply request a GPU allocation, load the Apptainer module, and use the desired container with the --nv option to access the GPU.

Available CUDA containers

  • /wsu/el7/containers/cuda/cuda-12.9.sif
  • /wsu/el7/containers/cuda/cuda-13.0.sif
  • /wsu/el7/containers/cuda/cuda-13.2.1.sif

Using appropriate arch flags with CUDA 13.2.1

When using cuda-13.2.1.sif to compile with nvcc, you should add the appropriate architecture flag (-arch) in the compile command for the GPU type of the GPU node you are using or targeting:

  • V100: -arch=sm_70
  • L40s : -arch=sm_89
  • H200: -arch=sm_90

You can also target all three GPU types as well with --gencode flag as well – the example commands are listed below.

CUDA 13.2.1 compile examples with appropriate arch flag

V100 (voh and rom1 nodes)

apptainer exec --nv /wsu/el7/containers/cuda/cuda-13.2.1.sif \ nvcc mycode.cu -o mycode -arch=sm_70​​​​​​+

L40S (alh nodes)

apptainer exec --nv /wsu/el7/containers/cuda/cuda-13.2.1.sif \ nvcc mycode.cu -o mycode -arch=sm_89

H200 (msa nodes)

apptainer exec --nv /wsu/el7/containers/cuda/cuda-13.2.1.sif \ nvcc mycode.cu -o mycode -arch=sm_90

All GPU types

apptainer exec --nv /wsu/el7/containers/cuda/cuda-13.2.1.sif \ nvcc mycode.cu -o mycode \ -gencode
arch=compute_70,code=sm_70 \ -gencode
arch=compute_89,code=sm_89 \ -gencode
arch=compute_90,code=sm_90

Using a CUDA container interactively

  1.  Request a GPU interactive job. You can add --constraint to request specific GPU. 

    srun -q gpu --gres=gpu:1 --mem=8G -t 2:00:00 –pty bash
     
  2. Load the Apptainer module.

    module load apptainer
     
  3. Check that the container sees the GPU.

    apptainer exec --nv /wsu/el7/containers/cuda/cuda-13.0.sif nvidia-smi
     
  4. Check the CUDA version.

    apptainer exec --nv /wsu/el7/containers/cuda/cuda-13.0.sif nvcc --version
     
  5. Start a shell inside the container.

    apptainer shell --nv /wsu/el7/containers/cuda/cuda-13.0.sif
     
  6. Compile the CUDA application.

    apptainer exec --nv /wsu/el7/containers/cuda/cuda-13.0.sif \
    nvcc mycode.cu -o mycode

  7. Run the CUDA application.

    apptainer exec --nv /wsu/el7/containers/cuda/cuda-13.0.sif \
    ./mycode

Using a CUDA container in a Slurm batch script

#!/bin/bash
# Example GPU job script - modify as needed
# Job name
#SBATCH --job-name=GPU
# Submit to the gpu QoS
#SBATCH -q gpu
# Request one node
#SBATCH -N 1
# Total number of CPU cores
#SBATCH -n 1
# Request memory
#SBATCH --mem=5G
# Request 1 GPU
#SBATCH --gres=gpu:1
# Request specific GPU type (optional)
#SBATCH --constraint=v100
# Mail notifications (begin, end, fail, requeue)
#SBATCH --mail-type=ALL
# Email address for notifications
#SBATCH --mail-user=xxyyyy@wayne.edu
# Standard output file
#SBATCH -o output_%j.out
# Standard error file
#SBATCH -e errors_%j.err
# Time limit (D-HH:MM:SS)
#SBATCH -t 1-00:00:00

# load apptainer module to use containers
module load apptainer

# check GPU
apptainer exec --nv /wsu/el7/containers/cuda/cuda-13.0.sif \
nvidia-smi

# check CUDA version
apptainer exec --nv /wsu/el7/containers/cuda/cuda-13.0.sif \
nvcc --version

# compile CUDA code
apptainer exec --nv /wsu/el7/containers/cuda/cuda-13.0.sif \
​​​​​​​nvcc mycode.cu -o mycode

# run CUDA program
​​​​​​​apptainer exec --nv /wsu/el7/containers/cuda/cuda-13.0.sif \
./mycode

 

 

 

 

 

 

Print Article