Skip to content

Tuning Memory-Bound Kernels

What memory-bound means

On a GPU, kernel speed is determined by whether compute or bandwidth becomes the bottleneck first. TileOPs measures a calibration factor with a macro benchmark: the hardware peak from the specification, multiplied by that factor, gives the effective peak a kernel can actually reach. That effective peak is the reference point for tuning. On an H200, the measured values are 57.27 TFLOP/s for fp32 FMA and 4.50 TB/s of memory bandwidth. The ridge point of the roofline is where the bandwidth slope meets the compute ceiling. Dividing compute by bandwidth places it at an arithmetic intensity of 12.72 flop/byte, the number of floating-point operations per byte moved when compute and bandwidth are saturated at the same time. The figure below shows the H200 roofline:

1 2 4 8 16 32 64 1 2 4 8 16 32 64 Attainable TFLOP/s (log) Arithmetic intensity: flop per byte (log) Ridge point 13 flop/byte silu (fp16) 5 flop / 4 bytes — ceiling 5.6 bandwidth roof 4.50 TB/s compute roof 57.3 TFLOP/s

The bent line is the roofline. Every kernel's performance point lies below it. To the left of the ridge, the ceiling is arithmetic intensity times bandwidth, so attainable compute rises linearly with intensity. To the right, the ceiling is the compute peak, and higher intensity no longer raises it. silu performs 5 operations per 4 bytes moved, for an arithmetic intensity of 1.25 flop/byte, one tenth of the ridge. Even with bandwidth fully saturated, it can reach only 10% of the compute ceiling.
  1. Optimizing Global Memory Access
  2. Optimizing Shared Memory Access