Norm, Conv, Pool & Other

本页暂无中文版。以下为英文原文。

36 ops, 200 workloads — Normalization 10 · Convolution 3 · Pooling 13 · Positional Encoding 6 · FFT 1 · MHC 2 · Top-k 1.

One table per op, one row per workload. Ratio is the fastest other implementation's device time divided by ours, so green is faster than it, plain is level with it, red is slower. Times are in ms. How these numbers are taken.

Normalization

BatchNormBwd (5 workloads)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
large-spatial-float16	1.72×	6.8730	torch-native-batch-norm torch-autograd	11.7961 12.9739	0.625
resnet50-fc-float16	1.79×	0.00704	torch-native-batch-norm torch-autograd	0.0126 0.0234	0.00233
resnet50-stage1-float16	1.28×	0.0148	torch-native-batch-norm torch-autograd	0.0189 0.0276	0.284
resnet50-stage2-float16	1.08×	0.0141	torch-native-batch-norm torch-autograd	0.0153 0.0239	0.298
resnet50-stage3-float16	1.03×	0.0171	torch-native-batch-norm torch-autograd	0.0177 0.0255	0.375

AdaLayerNormFwd (5 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
dit-xl-2-bfloat16	1.47×	0.00531	torch-compile torch-ref	0.00781 0.0121	1.11
dit-xl-2-float16	1.46×	0.00522	torch-compile torch-ref	0.00762 0.0121	1.13
llama-8b-decode-bfloat16	1.16×	0.00266	torch-compile torch-ref	0.00307 0.0103	0.00771
llama-8b-prefill-bfloat16	1.34×	0.0199	torch-compile torch-ref	0.0266 0.0418	2.11
llama-8b-prefill-float16	1.29×	0.0198	torch-compile torch-ref	0.0256 0.0415	2.12

AdaLayerNormZeroFwd (5 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
dit-xl-2-bfloat16	1.30×	0.00624	torch-compile torch-ref	0.00813 0.0148	1.13
dit-xl-2-float16	1.25×	0.00621	torch-compile torch-ref	0.00778 0.0148	1.14
llama-8b-decode-bfloat16	1.14×	0.00283	torch-compile torch-ref	0.00323 0.0117	0.00868
llama-8b-prefill-bfloat16	1.14×	0.0247	torch-compile torch-ref	0.0281 0.0537	2.04
llama-8b-prefill-float16	1.11×	0.0248	torch-compile torch-ref	0.0274 0.0534	2.03

FusedAddLayerNormFwd (6 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
llama-70b-decode-bfloat16	1.36×	0.00414	torch-compile torch-ref	0.00562 0.0260	0.0119
llama-70b-prefill-bfloat16	0.97×	0.0475	torch-compile torch-ref	0.0461 0.2294	2.12
llama-70b-prefill-float16	1.02×	0.0442	torch-compile torch-ref	0.0449 0.2276	2.28
llama-8b-decode-bfloat16	1.17×	0.00317	torch-compile torch-ref	0.00371 0.0189	0.00776
llama-8b-prefill-bfloat16	1.31×	0.0220	torch-compile torch-ref	0.0288 0.1169	2.29
llama-8b-prefill-float16	1.35×	0.0211	torch-compile torch-ref	0.0285 0.1159	2.39

LayerNormFwd (9 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
llama-405b-decode-bfloat16	1.28×	0.00429	torch-compile flashinfer flaggems torch	0.0055 0.00608 0.00611 0.0379	0.0191
llama-405b-prefill-bfloat16	0.99×	0.0509	flaggems torch-compile torch flashinfer	0.0504 0.0508 0.0748 0.0784	3.3
llama-405b-prefill-float16	0.93×	0.0502	torch-compile flaggems torch flashinfer	0.0468 0.0483 0.0737 0.0783	3.34
llama-70b-decode-bfloat16	1.07×	0.00342	torch-compile flashinfer flaggems torch	0.00368 0.00413 0.00422 0.0200	0.012
llama-70b-prefill-bfloat16	1.05×	0.0265	flaggems torch-compile torch flashinfer	0.0277 0.0334 0.0405 0.0466	3.17
llama-70b-prefill-float16	0.99×	0.0260	flaggems torch-compile torch flashinfer	0.0257 0.0308 0.0403 0.0465	3.23
llama-8b-decode-bfloat16	1.00×	0.00272	torch-compile flaggems flashinfer torch	0.00272 0.00282 0.00307 0.0112	0.00753
llama-8b-prefill-bfloat16	0.92×	0.0149	flaggems torch flashinfer torch-compile	0.0138 0.0213 0.0214 0.0256	2.81
llama-8b-prefill-float16	0.96×	0.0137	flaggems torch flashinfer torch-compile	0.0131 0.0212 0.0213 0.0231	3.06

RMSNormFwd (9 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
llama-405b-decode-bfloat16	0.97×	0.00362	flashinfer torch-compile vllm flaggems torch-ref	0.00352 0.00427 0.00435 0.00464 0.0201	0.0181
llama-405b-prefill-bfloat16	0.88×	0.0429	flashinfer torch-compile flaggems vllm torch-ref	0.0379 0.0409 0.0410 0.0488 0.5117	3.13
llama-405b-prefill-float16	0.88×	0.0419	flashinfer torch-compile flaggems vllm torch-ref	0.0371 0.0397 0.0399 0.0489 0.5100	3.2
llama-70b-decode-bfloat16	1.00×	0.00262	flashinfer vllm torch-compile flaggems torch-ref	0.00262 0.0031 0.00363 0.00413 0.0187	0.0125
llama-70b-prefill-bfloat16	0.92×	0.0219	flashinfer torch-compile flaggems vllm torch-ref	0.0201 0.0206 0.0213 0.0221 0.2709	3.07
llama-70b-prefill-float16	0.94×	0.0210	torch-compile flashinfer flaggems vllm torch-ref	0.0197 0.0201 0.0208 0.0217 0.2703	3.19
llama-8b-decode-bfloat16	1.05×	0.00205	flashinfer vllm torch-compile flaggems torch-ref	0.00215 0.00267 0.00269 0.00333 0.0179	0.008
llama-8b-prefill-bfloat16	0.86×	0.0126	flashinfer flaggems vllm torch-compile torch-ref	0.0109 0.0125 0.0127 0.0145 0.1461	2.65
llama-8b-prefill-float16	0.92×	0.0119	flashinfer vllm flaggems torch-compile torch-ref	0.0110 0.0125 0.0127 0.0136 0.1457	2.83

GroupNormFwd (8 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
image-g32-affine-bfloat16	1.10×	0.00368	flaggems torch-compile torch	0.00403 0.00502 0.0153	1.42
image-g32-affine-float16	1.08×	0.00374	flaggems torch-compile torch	0.00403 0.00518 0.0153	1.4
image-g32-bfloat16	1.01×	0.00355	flaggems torch-compile torch	0.00359 0.00426 0.0132	0.886
image-g32-float16	1.01×	0.00355	flaggems torch-compile torch	0.00358 0.00414 0.0132	0.886
tail-spatial-g16-affine-float16	0.67×	0.00608	flaggems torch-compile torch	0.00406 0.00416 0.0153	0.379
tail-spatial-g16-float16	0.67×	0.00522	torch-compile flaggems torch	0.00352 0.00362 0.0134	0.265
wider-channel-g32-affine-float16	0.67×	0.00595	flaggems torch-compile torch	0.00397 0.00432 0.0163	0.674
wider-channel-g32-float16	0.73×	0.0048	flaggems torch-compile torch	0.00349 0.00387 0.0142	0.502

InstanceNormFwd (8 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
image-affine-bfloat16	0.88×	0.00342	torch-compile flaggems torch	0.00301 0.00371 0.0207	1.53
image-affine-float16	0.89×	0.00346	torch-compile flaggems torch	0.00307 0.00371 0.0207	1.52
image-bfloat16	0.86×	0.00339	torch-compile flaggems torch	0.00291 0.00349 0.0171	0.927
image-float16	0.87×	0.00339	torch-compile flaggems torch	0.00294 0.00346 0.0170	0.928
tail-spatial-affine-float16	0.89×	0.00269	torch-compile flaggems torch	0.0024 0.00282 0.0111	0.429
tail-spatial-float16	0.91×	0.00253	torch-compile flaggems torch	0.0023 0.00262 0.00826	0.273
wider-channel-affine-float16	0.82×	0.00346	torch-compile flaggems torch	0.00285 0.00355 0.0206	1.16
wider-channel-float16	0.84×	0.00333	torch-compile flaggems torch	0.00278 0.0033 0.0162	0.723

FusedAddRMSNormFwd (9 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
llama-405b-decode-bfloat16	0.69×	0.00621	flashinfer vllm torch-compile torch-ref	0.00426 0.00499 0.00581 0.0317	0.0132
llama-405b-prefill-bfloat16	0.84×	0.0855	flashinfer torch-compile vllm torch-ref	0.0717 0.0722 0.0782 0.9911	1.96
llama-405b-prefill-float16	0.93×	0.0768	flashinfer torch-compile vllm torch-ref	0.0714 0.0721 0.0778 0.9831	2.18
llama-70b-decode-bfloat16	0.83×	0.00349	flashinfer torch-compile vllm torch-ref	0.00288 0.00301 0.00352 0.0300	0.0117
llama-70b-prefill-bfloat16	0.95×	0.0380	flashinfer torch-compile vllm torch-ref	0.0361 0.0364 0.0364 0.5180	2.21
llama-70b-prefill-float16	0.95×	0.0378	vllm flashinfer torch-compile torch-ref	0.0359 0.0362 0.0364 0.5142	2.22
llama-8b-decode-bfloat16	0.88×	0.00272	flashinfer vllm torch-compile torch-ref	0.00238 0.00298 0.00326 0.0288	0.00753
llama-8b-prefill-bfloat16	0.90×	0.0213	vllm flashinfer torch-compile torch-ref	0.0191 0.0193 0.0197 0.2691	1.97
llama-8b-prefill-float16	0.90×	0.0208	vllm flashinfer torch-compile torch-ref	0.0187 0.0193 0.0195 0.2672	2.02

BatchNormFwd (5 workloads)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
large-spatial-float16	0.23×	4.3442	torch-compile flaggems torch-cudnn	1.0184 3.9011 4.5332	1.24
resnet50-fc-float16	0.30×	0.00611	torch-compile flaggems torch-cudnn	0.00182 0.00554 0.0113	0.00335
resnet50-stage1-float16	0.41×	0.0109	torch-compile flaggems torch-cudnn	0.00448 0.0103 0.0114	0.479
resnet50-stage2-float16	0.31×	0.0107	torch-compile flaggems torch-cudnn	0.0033 0.00893 0.0104	0.491
resnet50-stage3-float16	0.35×	0.0129	torch-compile flaggems torch-cudnn	0.00458 0.0109 0.0111	0.622

Convolution

Conv1dFwd (16 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
encodec-deep-bfloat16	1.41×	0.0120	torch torch-compile flaggems	0.0170 0.0170 0.0715	32.3
encodec-deep-bias-bfloat16	1.50×	0.0124	torch-compile torch flaggems	0.0186 0.0203 0.0704	31.3
encodec-deep-bias-float16	1.50×	0.0124	torch-compile torch flaggems	0.0186 0.0204 0.0705	31.2
encodec-deep-float16	1.41×	0.0120	torch-compile torch flaggems	0.0170 0.0170 0.0715	32.3
encodec-init-bfloat16	1.87×	0.00355	torch-compile torch flaggems	0.00664 0.00666 0.0244	3.03
encodec-init-bias-bfloat16	2.39×	0.00358	torch-compile torch flaggems	0.00858 0.0107 0.0235	3.21
encodec-init-bias-float16	2.37×	0.00362	torch-compile torch flaggems	0.00858 0.0105 0.0236	3.19
encodec-init-float16	1.87×	0.00355	torch-compile torch flaggems	0.00662 0.00666 0.0245	3.03
wav2vec2-layer1-bfloat16	2.83×	0.00666	torch-compile torch flaggems	0.0188 0.0188 0.0401	4.92
wav2vec2-layer1-bias-bfloat16	3.12×	0.00694	torch-compile torch flaggems	0.0217 0.0255 0.0392	4.95
wav2vec2-layer1-bias-float16	3.25×	0.00691	torch-compile torch flaggems	0.0225 0.0252 0.0391	4.98
wav2vec2-layer1-float16	2.80×	0.00666	torch torch-compile flaggems	0.0186 0.0186 0.0400	4.92
whisper-large-conv1-bfloat16	1.16×	0.0485	torch torch-compile flaggems	0.0564 0.0564 0.1120	38
whisper-large-conv1-bias-bfloat16	1.27×	0.0477	torch-compile torch flaggems	0.0606 0.0693 0.1117	38.7
whisper-large-conv1-bias-float16	1.28×	0.0477	torch-compile torch flaggems	0.0611 0.0693 0.1115	38.7
whisper-large-conv1-float16	1.18×	0.0481	torch torch-compile flaggems	0.0569 0.0569 0.1120	38.3

Conv2dFwd (31 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
bottleneck-expand-1x1-bias-float16	1.91×	0.0041	torch-compile torch flaggems	0.00784 0.0088 0.0276	50.4
bottleneck-expand-1x1-float16	1.05×	0.00381	torch torch-compile flaggems	0.004 0.00739 0.0284	54
bottleneck-reduce-1x1-bias-float16	1.46×	0.00496	torch torch-compile flaggems	0.00726 0.00846 0.0256	41.5
bottleneck-reduce-1x1-float16	0.92×	0.00467	torch torch-compile flaggems	0.00432 0.00794 0.0264	44
classifier-1x1-bias-float16	0.82×	0.0095	torch-compile torch flaggems	0.00781 0.0120 0.0199	10.8
classifier-1x1-float16	0.79×	0.00915	torch-compile torch flaggems	0.0072 0.00906 0.0206	11.2
deeplabv3-aspp-3x3-rate12-float16	1.02×	0.0889	torch-compile torch flaggems	0.0909 0.1190 0.7158	109
highres-3x3-s1-bias-float16	0.76×	0.1050	torch-compile torch flaggems	0.0793 0.1148 0.7335	282
highres-3x3-s1-float16	0.75×	0.1047	torch-compile torch flaggems	0.0790 0.0947 0.7348	283
late-stage-1x1-bias-float16	1.38×	0.00528	torch-compile torch flaggems	0.0073 0.00931 0.0148	19.5
late-stage-1x1-float16	1.28×	0.00499	torch torch-compile flaggems	0.00637 0.00666 0.0154	20.6
midres-5x5-s1-bias-float16	1.01×	0.0165	torch-compile torch flaggems	0.0167 0.0228 0.2017	77.8
midres-5x5-s1-float16	0.99×	0.0162	torch-compile torch flaggems	0.0160 0.0195 0.2021	79.5
mobilenetv2-depthwise-float16	1.08×	0.00281	torch torch-compile flaggems	0.00304 0.00554 0.0542	0.642
resnet-1x1-bfloat16	0.92×	0.00435	torch torch-compile flaggems	0.004 0.00832 0.0492	47.2
resnet-1x1-bias-bfloat16	2.01×	0.00461	torch-compile torch flaggems	0.00928 0.0115 0.0484	44.9
resnet-1x1-bias-float16	1.92×	0.00458	torch-compile torch flaggems	0.00878 0.0116 0.0483	45.3
resnet-1x1-float16	0.96×	0.00435	torch torch-compile flaggems	0.00419 0.00839 0.0491	47.2
resnet-3x3-bfloat16	0.90×	0.0130	torch-compile torch flaggems	0.0117 0.0149 0.0831	35.6
resnet-3x3-bias-bfloat16	0.90×	0.0137	torch-compile torch flaggems	0.0124 0.0184 0.0823	33.7
resnet-3x3-bias-float16	0.89×	0.0132	torch-compile torch flaggems	0.0117 0.0183 0.0823	34.9
resnet-3x3-float16	0.89×	0.0130	torch-compile torch flaggems	0.0116 0.0147 0.0832	35.6
resnext-grouped-3x3-float16	4.60×	0.00413	torch torch-compile flaggems	0.0190 0.0190 0.0193	3.5
stage-transition-3x3-s2-bias-float16	0.97×	0.0141	torch-compile torch flaggems	0.0137 0.0200 0.1177	32.7
stage-transition-3x3-s2-float16	0.97×	0.0138	torch-compile torch flaggems	0.0134 0.0170 0.1184	33.5
stage-transition-5x5-s2-bias-float16	1.00×	0.0225	torch-compile torch flaggems	0.0225 0.0287 0.3096	57.1
stage-transition-5x5-s2-float16	0.99×	0.0225	torch-compile torch flaggems	0.0222 0.0254 0.3103	57.2
stem-3x3-s2-bias-float16	2.72×	0.00349	torch torch-compile flaggems	0.00947 0.00954 0.0122	3.16
stem-3x3-s2-float16	1.81×	0.00358	torch torch-compile flaggems	0.0065 0.00931 0.0130	3.02
stride2-bfloat16	1.09×	0.0111	torch-compile torch flaggems	0.0121 0.0148 0.0648	5.21
stride2-bias-bfloat16	1.08×	0.0116	torch-compile torch flaggems	0.0125 0.0177 0.0640	4.99

Conv3dFwd (8 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
3d-resnext-grouped-k3-float16	16.17×	0.0157	flaggems torch-compile torch	0.2545 0.2654 0.2809	5.51
3d-unet-aspp-3x3x3-rate6-float16	0.30×	0.1269	torch-compile torch flaggems	0.0376 0.0376 0.3002	57.1
r3d-stem-k3-s1-bias-float16	3.69×	0.0230	flaggems torch-compile torch	0.0849 0.1261 0.1548	90.9
r3d-stem-k3-s1-float16	3.74×	0.0229	flaggems torch-compile torch	0.0858 0.1148 0.1148	90.7
unet-encoder-k3-s1-bfloat16	0.33×	0.3542	torch torch-compile flaggems	0.1153 0.1153 0.3168	40.9
unet-encoder-k3-s1-bias-bfloat16	0.35×	0.3536	torch-compile torch flaggems	0.1224 0.1411 0.3158	41
video-stage-downsample-k3-s2-bias-float16	0.79×	0.0353	torch-compile torch flaggems	0.0281 0.0300 0.2158	39.3
video-stage-downsample-k3-s2-float16	0.75×	0.0349	torch-compile torch flaggems	0.0263 0.0263 0.2173	39.7

Pooling

AdaptiveMaxPool2dFwd (3 workloads)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
global-1x1-float16	1.30×	0.00291	torch-compile torch-ref	0.00378 0.0450	0.276
nondiv-7x7-bfloat16	1.76×	0.0065	torch-ref torch-compile	0.0115 0.0115	0.0757
spp-6x6-float16	2.37×	0.00602	torch-ref torch-compile	0.0143 0.0143	0.153

AdaptiveAvgPool2dFwd (3 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
nondiv-7x7-bfloat16	1.39×	0.00659	torch-compile torch-ref	0.00915 0.00915	0.0746
resnet-global-float16	1.25×	0.00298	torch-compile torch-ref	0.00371 0.00733	0.27
spp-6x6-float16	1.96×	0.00541	torch-compile torch-ref	0.0106 0.0106	0.17

MaxPool3dIndicesFwd (3 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
c3d-pool1-float16	0.43×	0.3044	torch-compile torch-ref	0.1294 0.5176	0.338
c3d-pool2-float16	0.56×	0.0588	torch-compile torch-ref	0.0327 0.0941	0.437
medicalnet-stem-bfloat16	6.14×	0.3315	torch-ref torch-compile	0.3348 2.0362	0.576

AvgPool2dFwd (3 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
ceil-divisor-bfloat16	1.24×	0.00314	torch-compile flaggems torch-ref	0.00387 0.00579 0.00765	1.12
vision-3x3-s2-float16	1.03×	0.00397	torch-compile flaggems torch-ref	0.0041 0.00662 0.00909	0.91
vision-5x5-s2-float16	1.79×	0.00403	flaggems torch-ref torch-compile	0.00723 0.00986 0.0206	1.24

MaxPool3dFwd (3 workloads)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
c3d-pool1-float16	1.01×	0.0763	torch-compile cudnn torch-ref	0.0771 0.3009 0.5176	1.35
c3d-pool2-float16	1.05×	0.0235	torch-compile cudnn torch-ref	0.0247 0.0608 0.0940	1.09
medicalnet-stem-bfloat16	2.37×	0.1113	cudnn torch-ref torch-compile	0.2639 0.3348 0.9266	1.71

AvgPool3dFwd (3 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
ceil-video-float16	0.92×	0.00438	torch-compile cudnn torch-ref	0.00403 0.00562 0.0114	0.591
divisor-bfloat16	0.83×	0.00229	torch-compile torch-ref	0.00189 0.00506	0.152
video-2x2x2-float16	1.30×	0.00371	torch-compile cudnn torch-ref	0.00483 0.00589 0.00992	0.433

MaxPool2dFwd (9 workloads)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
alexnet-ceil-bfloat16	1.25×	0.00877	torch-compile flaggems torch-ref	0.0110 0.0228 0.0348	1.53
alexnet-ceil-float16	1.25×	0.00877	torch-compile flaggems torch-ref	0.0110 0.0225 0.0348	1.53
alexnet-ceil-float32	1.22×	0.0126	torch-compile flaggems torch-ref	0.0154 0.0228 0.0341	1.06
resnet-stem-bfloat16	0.72×	0.0473	torch-compile flaggems torch-ref	0.0340 0.0784 0.1393	1.22
resnet-stem-float16	0.72×	0.0471	torch-compile flaggems torch-ref	0.0340 0.0784 0.1390	1.23
resnet-stem-float32	0.94×	0.0528	torch-compile flaggems torch-ref	0.0496 0.0813 0.1348	1.09
vgg-block-bfloat16	1.01×	0.0072	torch-compile flaggems torch-ref	0.00726 0.0148 0.0279	0.892
vgg-block-float16	1.01×	0.0072	torch-compile flaggems torch-ref	0.00726 0.0148 0.0277	0.892
vgg-block-float32	0.94×	0.0111	torch-compile flaggems torch-ref	0.0104 0.0167 0.0277	0.58

AdaptiveMaxPool2dIndicesFwd (3 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
global-1x1-float16	0.61×	0.0133	torch-compile torch-ref	0.00816 0.0450	0.0605
nondiv-7x7-bfloat16	0.73×	0.0158	torch-ref torch-compile	0.0115 0.0115	0.0312
spp-6x6-float16	0.93×	0.0154	torch-ref torch-compile	0.0143 0.0143	0.06

AvgPool1dFwd (3 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
audio-downsample-float16	0.68×	0.00614	torch-compile torch-ref	0.00419 0.0153	0.512
ceil-bfloat16	0.67×	0.00346	torch-compile torch-ref	0.0023 0.00531	0.303
long-temporal-float16	0.80×	0.0213	torch-compile torch-ref	0.0171 0.0594	0.962

MaxPool2dIndicesFwd (9 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
alexnet-ceil-bfloat16	0.73×	0.0237	torch-compile flaggems torch-ref	0.0174 0.0227 0.0348	0.567
alexnet-ceil-float16	0.75×	0.0237	torch-compile flaggems torch-ref	0.0177 0.0225 0.0348	0.567
alexnet-ceil-float32	0.82×	0.0236	torch-compile flaggems torch-ref	0.0194 0.0228 0.0341	0.569
resnet-stem-bfloat16	0.62×	0.1124	torch-compile flaggems torch-ref	0.0700 0.0784 0.1395	0.514
resnet-stem-float16	0.61×	0.1126	torch-compile flaggems torch-ref	0.0691 0.0785 0.1390	0.513
resnet-stem-float32	0.67×	0.1073	torch-compile flaggems torch-ref	0.0716 0.0810 0.1347	0.539
vgg-block-bfloat16	0.55×	0.0195	torch-compile flaggems torch-ref	0.0106 0.0148 0.0279	0.33
vgg-block-float16	0.54×	0.0196	torch-compile flaggems torch-ref	0.0107 0.0148 0.0278	0.327
vgg-block-float32	0.65×	0.0195	torch-compile flaggems torch-ref	0.0127 0.0167 0.0277	0.329

MaxPool1dFwd (3 workloads)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
ecg-cnn-dilated-bfloat16	0.82×	0.00954	torch-compile torch-ref	0.00784 0.0354	1.1
sincnet-speaker-local-float16	1.00×	0.0114	torch-compile torch-ref	0.0114 0.0506	0.92
textcnn-global-float16	0.28×	0.0134	torch-compile torch-ref	0.00371 0.0264	0.156

MaxPool1dIndicesFwd (3 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
ecg-cnn-dilated-bfloat16	0.60×	0.0223	torch-compile torch-ref	0.0134 0.0354	0.469
sincnet-speaker-local-float16	0.74×	0.0218	torch-compile torch-ref	0.0160 0.0505	0.481
textcnn-global-float16	0.30×	0.0193	torch-compile torch-ref	0.0057 0.0264	0.109

MeanPoolingForward (4 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
dense-batched	0.41×	0.0702	torch-view-mean torch-compile torch-ref	0.0286 0.1465 0.2614	0.478
dense-mainstream	0.35×	0.1352	torch-view-mean torch-compile torch-ref	0.0470 0.4243 0.6148	0.496
varlen-long	4.44×	0.1386	torch-compile torch-ref	0.6152 0.6200	0.484
varlen-tail	9.63×	0.0218	torch-compile torch-ref	0.2095 0.2139	0.41

Positional Encoding

RopeNeoxFwd (3 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
neox-1d-2k-d64-float16	1.09×	0.00182	torch-compile torch-ref	0.00198 0.00944	0.287
neox-1d-4k-d128-bfloat16	1.17×	0.00259	torch-compile torch-ref	0.00304 0.0123	0.809
neox-2d-b2-s2k-h32-d128-float16	0.60×	0.0308	torch-compile torch-ref	0.0184 0.2718	2.18

RopeNonNeoxFwd (2 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
non-neox-1d-2k-d64-float16	0.93×	0.00218	torch-compile torch-ref	0.00202 0.00947	0.241
non-neox-2d-b2-s2k-h32-d128-bfloat16	0.75×	0.0252	torch-compile torch-ref	0.0190 0.2742	2.66

RopeLongRopeFwd (2 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
longrope-1d-8k-d128-bfloat16	1.16×	0.00358	torch-compile torch-ref	0.00416 0.0159	1.17
longrope-2d-b1-s8k-h32-d128-float16	0.59×	0.0595	torch-compile torch-ref	0.0349 0.4931	2.25

RopeYarnFwd (2 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
yarn-1d-8k-d128-bfloat16	1.15×	0.00362	torch-compile torch-ref	0.00416 0.0159	1.16
yarn-2d-b1-s8k-h32-d128-float16	0.59×	0.0595	torch-compile torch-ref	0.0349 0.4923	2.25

RopeLlama31Fwd (2 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
llama31-1d-8k-d128-bfloat16	1.14×	0.00362	torch-compile torch-ref	0.00413 0.0159	1.16
llama31-2d-b1-s8k-h32-d128-float16	0.59×	0.0594	torch-compile torch-ref	0.0349 0.4926	2.26

RopeNeoxPositionIdsFwd (2 workloads)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
position-ids-s2k-h32-d128-float16	0.43×	0.0278	torch-compile vllm torch-ref	0.0119 0.0243 0.1296	1.21
position-ids-s4k-h32-d128-bfloat16	0.49×	0.0457	torch-compile vllm torch-ref	0.0222 0.0445 0.2490	1.47

FFT

FFTC2CFwd (3 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
fft-4k-c128-b64-complex128	0.28×	0.0305	torch-cufft torch-compile	0.00848 0.00848	0.516
fft-4k-c64-b64-complex64	0.37×	0.0152	torch-cufft torch-compile	0.00563 0.00563	1.03
fft-4k-c64-unbatched-complex64	0.67×	0.00813	torch-cufft torch-compile	0.00544 0.00544	0.0302

MHC

MHCPostFwd (3 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
post-large-bfloat16	1.01×	0.0016	torch-compile torch-ref	0.00162 0.0114	0.0512
post-medium-bfloat16	0.98×	0.00141	torch-compile torch-ref	0.00138 0.0110	0.0218
post-small-bfloat16	0.95×	0.00131	torch-compile torch-ref	0.00125 0.0105	0.0078

MHCPreFwd (3 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
pre-large-bfloat16	0.79×	0.0715	torch-compile torch-ref	0.0567 0.1170	0.0332
pre-medium-bfloat16	0.58×	0.0544	torch-compile torch-ref	0.0314 0.0779	0.0164
pre-small-bfloat16	0.50×	0.0395	torch-compile torch-ref	0.0197 0.0592	0.00753

Top-k

TopkSelectorFwd (2 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
topk1024-s32k-kv64k-float32	0.59×	15.6297	flashinfer torch torch-compile	9.2719 31.8215 31.8223	0.137
topk2048-s32k-kv64k-float32	0.66×	16.2375	flashinfer torch torch-compile	10.6697 33.2921 33.3000	0.132