CUDA Environment Setup — infra.bea.ai

Getting CUDA to work is the first hurdle for anyone starting GPU development. This chapter covers the software stack, version compatibility rules, and a step-by-step verification checklist.

See also: Programming Essentials covers the GPU software stack, nvidia-smi, and environment variables at an introductory level. This chapter goes deeper into CUDA-specific setup.

1. The CUDA Software Stack

Four layers that must be compatible:

Your Code (Python, C++, CUDA)
    ↓
Framework (PyTorch, TensorFlow)
    ↓ links against
CUDA Toolkit (nvcc, runtime, cuBLAS, cuDNN, NCCL)
    ↓ requires
GPU Driver (kernel module)
    ↓ controls
GPU Hardware (H100, A100, etc.)

Version Compatibility Rule

Driver version ≥ CUDA Toolkit version ≥ Framework build version

CUDA Toolkit	Minimum driver	Typical GPUs
CUDA 11.8	520.61+	A100, V100
CUDA 12.1	530.30+	A100, H100
CUDA 12.4	550.54+	H100, H200
CUDA 12.6	560.28+	H100, H200, B200

The driver is backwards-compatible: a newer driver supports older CUDA toolkits. But a new toolkit on an old driver fails silently or crashes.

2. Installation

Option A: NVIDIA Container (Recommended)

Skip installation entirely — use NVIDIA’s pre-built containers:

# Pull PyTorch container with everything pre-installed
docker pull nvcr.io/nvidia/pytorch:24.07-py3

# Run with GPU access
docker run --gpus all -it nvcr.io/nvidia/pytorch:24.07-py3

This gives you: matching driver compatibility, CUDA toolkit, cuDNN, NCCL, PyTorch — all tested together.

Option B: System Installation

# 1. Install NVIDIA driver
sudo apt-get install nvidia-driver-560

# 2. Install CUDA toolkit
wget https://developer.download.nvidia.com/compute/cuda/12.6/local_installers/cuda_12.6.0_560.28.03_linux.run
sudo sh cuda_12.6.0_560.28.03_linux.run

# 3. Set PATH
echo 'export PATH=/usr/local/cuda/bin:$PATH' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc

# 4. Install PyTorch (must match CUDA version)
pip install torch --index-url https://download.pytorch.org/whl/cu124

3. Verification Checklist

Run these in order. If any fails, fix before proceeding:

# 1. Is the GPU visible?
nvidia-smi
# Should show GPU name, driver version, CUDA version

# 2. Is the driver loaded?
nvidia-smi --query-gpu=driver_version --format=csv
# Should show e.g., "560.28.03"

# 3. Is CUDA toolkit installed?
nvcc --version
# Should show e.g., "release 12.6"

# 4. Does PyTorch see CUDA?
python -c "import torch; print(torch.cuda.is_available())"
# Should print: True

# 5. Can PyTorch use the GPU?
python -c "
import torch
x = torch.randn(1000, 1000, device='cuda')
y = torch.mm(x, x)
print(f'GPU: {torch.cuda.get_device_name()}')
print(f'CUDA: {torch.version.cuda}')
print(f'cuDNN: {torch.backends.cudnn.version()}')
print('GPU compute works!')
"

# 6. Can GPUs communicate? (for multi-GPU)
python -c "
import torch
n = torch.cuda.device_count()
print(f'{n} GPUs available')
for i in range(n):
    print(f'  GPU {i}: {torch.cuda.get_device_name(i)}')
"

4. Common Issues

Symptom	Cause	Fix
`nvidia-smi` fails	Driver not installed or not loaded	Install driver, reboot
`torch.cuda.is_available()` = False	PyTorch built for wrong CUDA version	Reinstall PyTorch matching your CUDA
`nvcc not found`	CUDA toolkit not in PATH	Add `/usr/local/cuda/bin` to PATH
CUDA OOM on first operation	Other processes using GPU memory	`nvidia-smi` to check, kill other processes
`CUDA error: no kernel image`	GPU architecture not supported by toolkit	Update CUDA toolkit or compile for your GPU arch
Slow first kernel	JIT compilation of PTX	Normal — first run compiles; subsequent runs are cached

Environment Variables

# Select specific GPUs (hide others)
export CUDA_VISIBLE_DEVICES=0,1

# Debugging
export CUDA_LAUNCH_BLOCKING=1    # Synchronous execution (easier to debug)
export TORCH_USE_CUDA_DSA=1       # Device-side assertions

# NCCL (for distributed training)
export NCCL_DEBUG=INFO            # Debug distributed hangs
export NCCL_IB_DISABLE=0          # Enable InfiniBand

5. Development Tools

Tool	Purpose	When to use
`nvcc`	CUDA compiler	Compiling .cu files
`nvidia-smi`	GPU monitoring	Always — check utilization, memory, temp
`nvidia-smi dmon`	Continuous monitoring	During training/inference runs
`nsight-sys`	Timeline profiler	Finding bottlenecks (covered in Profiling chapter)
`nsight-compute`	Kernel profiler	Optimizing individual kernels
`compute-sanitizer`	Memory checker	Finding out-of-bounds, race conditions

# Compile a CUDA file
nvcc -o kernel kernel.cu -arch=sm_90  # sm_90 = H100

# Memory error checking
compute-sanitizer ./kernel

# Continuous GPU monitoring
nvidia-smi dmon -s u -d 1  # utilization, 1-second interval

Self-Check

State the driver-toolkit-framework version compatibility rule
Run the 6-step verification checklist successfully
Explain when to use containers vs system installation
Set CUDA_VISIBLE_DEVICES to restrict GPU access
Use compute-sanitizer to check for memory errors
Debug a “torch.cuda.is_available() = False” issue