Nsight Compute

Nsight Compute is the microscope — while Nsight Systems shows you which kernel is the bottleneck, Nsight Compute shows you why that kernel is slow and what to do about it.

---

1. When to Use Nsight Compute

After Nsight Systems identifies a slow kernel:

Nsight Systems: "The GEMM kernel at line 47 takes 2.3ms — 60% of forward pass"
  ↓
Nsight Compute: "This GEMM achieves 45% compute utilization because of L2 cache misses.
                 Increasing tile size from 64 to 128 would improve L2 hit rate by 40%."

---

2. Capturing Kernel Metrics

Command Line

# Profile all kernels
ncu -o kernel_report python train.py

# Profile a specific kernel by name
ncu --kernel-name "gemm_kernel" -o gemm_report python train.py

# Collect specific metric sections
ncu --set full -o full_report python train.py

# Quick summary without GUI
ncu --metrics \
    sm__throughput.avg.pct_of_peak_sustained_elapsed,\
    dram__throughput.avg.pct_of_peak_sustained_elapsed,\
    l1tex__throughput.avg.pct_of_peak_sustained_elapsed \
    python train.py

Key Metric Groups

Section	What it measures
GPU Speed of Light	Overall compute + memory utilization vs peak
Compute Workload Analysis	Which pipeline (FMA, Tensor Core, SFU) is busy
Memory Workload Analysis	L1, L2, HBM throughput and hit rates
Scheduler Statistics	Warp occupancy, eligible warps, stalls
Launch Statistics	Grid/block size, register/shared memory usage
Source Counters	Per-line metrics mapped to source code

---

3. The Roofline Chart

Nsight Compute generates a roofline chart — the single most useful visualization for understanding kernel performance:

TFLOPS │                    ┌──── Compute ceiling (989 TFLOPS FP16)
       │                   /
       │                  /    ● Your kernel (AI=50, 200 TFLOPS)
       │                /        ↑ gap = optimization opportunity
       │              /
       │            /
       │          /
       │        /  ← Bandwidth ceiling (3.35 TB/s)
       │      /
       │    /
       │──/──────────────────────────────────
       └─────────────────────── Arithmetic Intensity (FLOPs/Byte)
                    ↑
              Ridge point = 295

How to Read It

Position	Meaning	Optimization direction
Left of ridge, below BW ceiling	Memory-bound, not saturating bandwidth	Improve coalescing, reduce redundant reads
Left of ridge, on BW ceiling	Memory-bound, at peak bandwidth	Increase arithmetic intensity (fusion, tiling)
Right of ridge, below compute ceiling	Compute-bound, not saturating compute	Improve occupancy, use Tensor Cores
Right of ridge, on compute ceiling	Compute-bound, at peak compute	Algorithmic change or better hardware

---

4. Key Metrics and What They Mean

GPU Speed of Light

Compute (SM) Throughput:    45.2%    ← how busy the SMs are
Memory Throughput:          78.3%    ← how busy HBM is

If compute < 50% and memory > 70% → memory-bound. Optimize memory access. If compute > 70% and memory < 50% → compute-bound. Increase occupancy or use Tensor Cores. If both < 50% → latency-bound. Too few warps, stalling on dependencies.

Memory Analysis

L1/TEX Hit Rate:           62.3%    ← higher is better
L2 Hit Rate:               45.1%    ← higher is better for repeated access
DRAM Throughput:           2,680 GB/s (80% of peak)
Global Load Efficiency:    87.5%    ← 100% = perfect coalescing

Global Load Efficiency < 80% → you have uncoalesced access. Fix SoA/AoS or access pattern.

Occupancy

Achieved Occupancy:        47.2%    ← actual active warps / max warps
Theoretical Occupancy:     50.0%    ← limited by registers (64/thread)

If achieved << theoretical → warps are stalling (usually waiting for memory).

Warp Stall Reasons

Stall Long Scoreboard:     34.2%    ← waiting for global memory
Stall Short Scoreboard:    12.1%    ← waiting for shared memory / L1
Stall Not Selected:        18.5%    ← eligible but not scheduled (occupancy OK)
Stall Math Pipe Throttle:   8.3%    ← compute pipeline full (good!)

Long Scoreboard dominating → memory-bound. Improve access patterns or increase occupancy. Math Pipe Throttle dominating → compute-bound. This is the best case.

---

5. Optimization Workflow

Step 1: Identify the Bound

ncu --set full mykernel
# Check GPU Speed of Light → compute vs memory throughput

Step 2: For Memory-Bound Kernels

1. Check Global Load Efficiency → fix uncoalesced access
2. Check L1 Hit Rate → consider shared memory tiling
3. Check L2 Hit Rate → consider L2 persistence or reordering
4. Check Shared Memory Bank Conflicts → add padding

Step 3: For Compute-Bound Kernels

1. Check if Tensor Cores are used → switch to WMMA/MMA if applicable
2. Check occupancy → reduce register pressure if occupancy < 50%
3. Check Warp Stall Math Pipe Throttle → if high, you’re near peak
4. Consider algorithmic changes (fewer FLOPs)

Step 4: For Latency-Bound Kernels

1. Check occupancy → increase block size or reduce resources per thread
2. Check Warp Stall Long Scoreboard → more warps to hide memory latency
3. Consider overlapping loads (double buffering, async copy)

---

6. Comparing Kernel Versions

# Profile before optimization
ncu -o before mykernel_v1

# Profile after optimization
ncu -o after mykernel_v2

# Compare
ncu --diff before.ncu-rep after.ncu-rep

The diff shows exactly which metrics improved/regressed — essential for validating that your optimization actually helped.

---

Self-Check

• Explain the difference between Nsight Systems and Nsight Compute
• Capture a kernel profile and locate the roofline chart
• Read the GPU Speed of Light to determine compute-bound vs memory-bound
• Diagnose uncoalesced access from Global Load Efficiency
• Interpret warp stall reasons and determine the optimization direction
• Compare two kernel versions using --diff

---

References

• Nsight Compute Documentation
• Nsight Compute Kernel Profiling Guide
• Roofline Model for GPU Computing