Nils Rottstegge
← All projects

Project · ALPS supercomputer

Multi-GPU HPCG

HPCG solves a 3D Poisson problem with a preconditioned conjugate gradient method and stresses memory bandwidth and interconnect rather than peak FLOPs. I started from an existing single-GPU implementation that stores the 27-point stencil matrix in the DIA format and parallelises symmetric Gauss-Seidel (SymGS) with box colouring, and ported it to multiple GPUs and nodes. The first port was slower than the original even on a single GPU: the ghost cells needed for the halo exchange broke coalesced memory access. Step by step I reworked indexing and data layout until the multi-GPU code, run on one GPU, was faster than the single-GPU code it came from: 1.50× for SpMV and 1.78× for SymGS. Then I scaled it out to 512 nodes and 2,048 GH200 GPUs on ALPS.

An 8 × 8 × 8 grid of points in 3D, coloured from blue to red along its diagonal.
The structured 3D grid behind HPCG’s 27-point stencil (8 × 8 × 8 points).
SpMV speedup over the single-GPU code: Single-GPU code 1.00×, Naive multi-GPU port 0.32×, Locality-optimised indexing 1.07×, Column-major DIA 1.50×, Blocked storage 1.46×, Pipelined memcpy 0.75×, Colour-wise storage 0.72× SpMVsparse matrix-vector productSingle-GPU code1.00× 1.61 msNaive multi-GPU port0.32× 5.02 msLocality-optimised indexing1.07× 1.50 msColumn-major DIA1.50× 1.07 msBlocked storage1.46× 1.10 msPipelined memcpy0.75× 2.15 msColour-wise storage0.72× 2.23 ms1× = single-GPU code
SymGS speedup over the single-GPU code: Single-GPU code 1.00×, Naive multi-GPU port 0.68×, Locality-optimised indexing 1.07×, Column-major DIA 1.04×, Colour-wise storage 1.78× SymGSsymmetric Gauss-Seidel, one colour sweepSingle-GPU code1.00× 0.185 msNaive multi-GPU port0.68× 0.271 msLocality-optimised indexing1.07× 0.172 msColumn-major DIA1.04× 0.178 msColour-wise storage1.78× 0.104 ms1× = single-GPU code
From the naive port to faster-than-the-original: every step measured on one GH200 (256³ grid, multi-GPU code path) against the single-GPU code.
Time per CG iteration: 1 GPUs 8.35 ms, 8 GPUs 75.73 ms, 27 GPUs 225.66 ms, 64 GPUs 236.19 ms, 125 GPUs 272.15 ms, 216 GPUs 291.68 ms, 343 GPUs 283.11 ms, 512 GPUs 289.44 ms, 729 GPUs 290.79 ms, 1000 GPUs 299.74 ms, 1331 GPUs 298.69 ms, 1728 GPUs 297.7 ms ← GPUs gain halo neighbours(up to 26 each)010020030015001,0001,500GPUs (NVIDIA GH200)ms per CG iterationflat from 216 to 1,728 GPUs:283–300 ms per iteration1 GPUs · 33K points · 8.35 ms (10 runs: 8.33–8.37)1 GPU · 33K points8 GPUs · 262K points · 75.73 ms (10 runs: 75.69–75.88)27 GPUs · 885K points · 225.66 ms (10 runs: 225.55–225.96)64 GPUs · 2.1M points · 236.19 ms (10 runs: 235.91–236.34)64 GPUs · 2.1M points125 GPUs · 4.1M points · 272.15 ms (10 runs: 271.99–272.6)216 GPUs · 7.1M points · 291.68 ms (10 runs: 291.57–292.31)343 GPUs · 11.2M points · 283.11 ms (10 runs: 281.46–285.05)343 GPUs · 11.2M points512 GPUs · 16.8M points · 289.44 ms (10 runs: 288.19–289.7)729 GPUs · 23.9M points · 290.79 ms (10 runs: 290.1–291.17)1,000 GPUs · 32.8M points · 299.74 ms (10 runs: 299.46–299.97)1,331 GPUs · 43.6M points · 298.69 ms (10 runs: 298.63–299.2)1,728 GPUs · 56.6M points · 297.7 ms (10 runs: 297.47–298.12)1,728 GPUs · 56.6M points
Weak scaling with 32³ points per GPU (median of 10 runs). The time per iteration grows while processes gain halo neighbours, then stays flat once most have all 26. Hover a point for details.
Box colouring of a 12 by 6 grid with 9 colours; points of the same colour never neighbour each other.
Sweep 1 of 9

Hover a point: its 9-point stencil covers 9 different colours, so all points of one colour can be updated in the same sweep.

Box colouring makes symmetric Gauss-Seidel parallel. Shown in 2D with a 9-point stencil and 9 colours; HPCG’s 27-point stencil in 3D needs 27.

My contribution

  • Ported the DIA-based single-GPU HPCG to multiple GPUs and nodes: SpMV, SymGS, multigrid, CG, dot products and WAXPBY.
  • Halo exchange over a 3D domain decomposition with up to 26 neighbours, using MPI (host and CUDA-aware) and NCCL.
  • SpMV: locality-optimised indexing and column-major DIA storage, so consecutive threads read consecutive memory (84.6% memory utilisation efficiency).
  • SymGS: colour-wise storage, so all coefficients of one box colour are contiguous and no colour offsets are computed at run time (92.0% memory utilisation efficiency).

Results

  • On one GH200, the multi-GPU code beats the original single-GPU code: 1.50× faster SpMV and 1.78× faster SymGS (from 0.32× and 0.68× after the naive port).
  • Verified on up to 512 nodes, 2,048 NVIDIA GH200 GPUs, on the ALPS supercomputer.
  • Ported to the AMD cluster Beverin.
  • Lesson: a data layout that is best for one kernel can be the worst for another; colour-wise storage speeds up SymGS but slows SpMV to 0.72×.

Technologies

  • C++
  • CUDA
  • NCCL
  • MPI
  • Slurm
  • ALPS

Links