Normalization

本页暂无中文版。以下为英文原文。

10 ops, 112 workloads.

One table per op, one row per workload. Ratio is the fastest other implementation's device time divided by ours, so green is faster than it, plain is level with it, red is slower. Times are in ms. How these numbers are taken.

LayerNormFwd

x: [*B, *normalized_shape]weight, bias: [*normalized_shape]

• llama-8b-prefillx: [2048, 4096]weight, bias: [4096]dtype=f16, bf16
• llama-8b-decodex: [1, 4096]weight, bias: [4096]dtype=bf16
• llama-70b-prefillx: [2048, 8192]weight, bias: [8192]dtype=f16, bf16
• llama-70b-decodex: [1, 8192]weight, bias: [8192]dtype=bf16
• llama-405b-prefillx: [2048, 16384]weight, bias: [16384]dtype=f16, bf16
• llama-405b-decodex: [1, 16384]weight, bias: [16384]dtype=bf16

x: [*B, *normalized_shape]weight: [*normalized_shape]

• llama-13b-prefillx: [2048, 5120]weight: [5120]dtype=f32

x: [*B, *normalized_shape]

• dit-xl-2x: [1024, 1152]dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
llama-8b-prefill	f16	1.05×	0.0125	flaggems flashinfer torch-compile quack torch	0.0132 0.0213 0.0232 0.0245 0.0812	4.68	59%	mem
	bf16	1.11×	0.0126	flaggems flashinfer torch-compile quack torch	0.0139 0.0213 0.0246 0.0247 0.0820	4.68	59%	mem
llama-8b-decode	bf16	1.14×	0.00235	torch-compile flaggems flashinfer quack torch	0.00269 0.00282 0.00298 0.0100 0.0199	0.0122	0%	lat
llama-70b-prefill	f16	1.10×	0.0235	flaggems torch-compile quack flashinfer torch	0.0259 0.0309 0.0387 0.0465 0.1592	5	63%	mem
	bf16	1.11×	0.0251	flaggems torch-compile quack flashinfer torch	0.0278 0.0336 0.0399 0.0466 0.1603	4.69	60%	mem
llama-70b-decode	bf16	1.21×	0.00301	torch-compile flashinfer flaggems quack torch	0.00365 0.00411 0.00426 0.0106 0.0260	0.0191	0%	lat
llama-405b-prefill	f16	1.04×	0.0452	torch-compile flaggems flashinfer torch quack	0.0471 0.0486 0.0786 0.3036 0.6696	5.2	66%	mem
	bf16	1.11×	0.0460	torch-compile flaggems flashinfer torch quack	0.0508 0.0512 0.0787 0.3059 0.6728	5.11	65%	mem
llama-405b-decode	bf16	1.41×	0.00394	torch-compile flashinfer flaggems torch quack	0.00554 0.00605 0.00614 0.0374 0.0657	0.0291	1%	lat
llama-13b-prefill	f32	1.10×	0.0240	torch-compile quack torch	0.0264 0.0271 0.0386	2.62	78%	mem
dit-xl-2	bf16	1.06×	0.0031	quack torch-compile torch	0.0033 0.00653 0.0164	1.9	34%	lat

FusedAddLayerNormFwd

x, residual: [*B, N]weight, bias: [N]

• llama-8b-prefillx, residual: [2048, 4096]weight, bias: [4096]dtype=f16, bf16
• llama-8b-decodex, residual: [1, 4096]weight, bias: [4096]dtype=bf16
• llama-70b-prefillx, residual: [2048, 8192]weight, bias: [8192]dtype=f16, bf16
• llama-70b-decodex, residual: [1, 8192]weight, bias: [8192]dtype=bf16
• llama-13b-prefillx, residual: [2048, 5120]weight, bias: [5120]dtype=f32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
llama-8b-prefill	f16	1.26×	0.0202	flash-attn torch-compile quack torch-ref	0.0254 0.0285 0.0360 0.1745	3.32	74%	mem
	bf16	1.19×	0.0210	flash-attn torch-compile quack torch-ref	0.0251 0.0297 0.0361 0.1758	3.19	71%	mem
llama-8b-decode	bf16	1.42×	0.00263	torch-compile flash-attn quack torch-ref	0.00373 0.00422 0.0113 0.0282	0.0125	0%	lat
llama-70b-prefill	f16	1.15×	0.0390	torch-compile flash-attn quack torch-ref	0.0449 0.0481 0.0646 0.3482	3.44	76%	mem
	bf16	1.13×	0.0415	torch-compile flash-attn quack torch-ref	0.0468 0.0486 0.0658 0.3508	3.23	72%	mem
llama-70b-decode	bf16	1.33×	0.00362	flash-attn torch-compile quack torch-ref	0.0048 0.00563 0.0117 0.0347	0.0181	1%	lat
llama-13b-prefill	f32	1.07×	0.0442	torch-compile flash-attn quack torch-ref	0.0472 0.0597 0.0599 0.0734	1.9	84%	mem

RMSNormFwd

x: [*B, *normalized_shape]weight: [*normalized_shape]

• llama-8b-prefillx: [2048, 4096]weight: [4096]eps=1e-06dtype=f16, bf16
• llama-8b-decodex: [1, 4096]weight: [4096]eps=1e-06dtype=bf16
• llama-70b-prefillx: [2048, 8192]weight: [8192]eps=1e-06dtype=f16, bf16
• llama-70b-decodex: [1, 8192]weight: [8192]eps=1e-06dtype=bf16
• llama-405b-prefillx: [2048, 16384]weight: [16384]eps=1e-06dtype=f16, bf16
• llama-405b-decodex: [1, 16384]weight: [16384]eps=1e-06dtype=bf16
• llama-13b-prefillx: [2048, 5120]weight: [5120]dtype=bf16
• quack-throughput-256x: [32768, 256]weight: [256]eps=1e-06dtype=f16, bf16
• quack-throughput-512x: [32768, 512]weight: [512]eps=1e-06dtype=f16, bf16
• quack-throughput-1024x: [32768, 1024]weight: [1024]eps=1e-06dtype=f16, bf16
• quack-throughput-2048x: [32768, 2048]weight: [2048]eps=1e-06dtype=f16, bf16
• quack-throughput-4096x: [32768, 4096]weight: [4096]eps=1e-06dtype=f16, bf16
• quack-throughput-8192x: [32768, 8192]weight: [8192]eps=1e-06dtype=f16, bf16
• quack-throughput-16384x: [32768, 16384]weight: [16384]eps=1e-06dtype=f16, bf16
• quack-throughput-32768x: [32768, 32768]weight: [32768]eps=1e-06dtype=f16, bf16
• quack-throughput-65536x: [32768, 65536]weight: [65536]eps=1e-06dtype=f16, bf16
• quack-throughput-131072x: [16384, 131072]weight: [131072]eps=1e-06dtype=f16, bf16
• quack-throughput-262144x: [8192, 262144]weight: [262144]eps=1e-06dtype=f16, bf16
• qwen3-8b-qk-norm-decodex: [1, 32, 128]weight: [128]eps=1e-06dtype=bf16

x: [*B, *normalized_shape]

• qk-norm-headx: [4, 2048, 128]dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
llama-8b-prefill	f16	0.99×	0.0108	flashinfer vllm flaggems quack torch-compile torch-ref	0.0108 0.0125 0.0126 0.0126 0.0131 0.0659	3.09	69%	mem
	bf16	0.99×	0.0109	flashinfer flaggems quack vllm torch-compile torch-ref	0.0108 0.0124 0.0126 0.0127 0.0134 0.0665	3.08	68%	mem
llama-8b-decode	bf16	1.08×	0.00192	quack flashinfer torch-compile vllm flaggems torch-ref	0.00208 0.00214 0.00262 0.00266 0.00339 0.0124	0.00853	0%	lat
llama-70b-prefill	f16	1.03×	0.0190	torch-compile flashinfer quack flaggems vllm torch-ref	0.0195 0.0200 0.0206 0.0214 0.0219 0.1348	3.54	79%	mem
	bf16	1.03×	0.0190	torch-compile flashinfer quack flaggems vllm torch-ref	0.0196 0.0200 0.0208 0.0217 0.0223 0.1359	3.53	78%	mem
llama-70b-decode	bf16	1.14×	0.00221	flashinfer quack vllm torch-compile flaggems torch-ref	0.00253 0.00266 0.0031 0.00362 0.0049 0.0153	0.0148	0%	lat
llama-405b-prefill	f16	1.01×	0.0365	quack torch-compile flashinfer flaggems vllm torch-ref	0.0368 0.0368 0.0370 0.0398 0.0488 0.2679	3.68	82%	mem
	bf16	0.98×	0.0377	torch-compile flashinfer quack flaggems vllm torch-ref	0.0369 0.0379 0.0382 0.0409 0.0489 0.2698	3.56	79%	mem
llama-405b-decode	bf16	1.20×	0.00294	flashinfer quack torch-compile vllm flaggems torch-ref	0.00352 0.00355 0.00439 0.00442 0.0048 0.0208	0.0223	1%	lat
llama-13b-prefill	bf16	1.02×	0.0132	flashinfer flaggems quack vllm torch-compile torch-ref	0.0134 0.0143 0.0153 0.0154 0.0160 0.0828	3.18	71%	mem
quack-throughput-256	f16	1.04×	0.00982	torch-compile quack flashinfer vllm flaggems torch-ref	0.0102 0.0106 0.0124 0.0228 0.0252 0.0877	3.42	76%	mem
	bf16	1.04×	0.00982	torch-compile quack flashinfer vllm flaggems torch-ref	0.0102 0.0106 0.0124 0.0228 0.0259 0.0880	3.42	76%	mem
quack-throughput-512	f16	1.03×	0.0178	flashinfer torch-compile quack vllm flaggems torch-ref	0.0182 0.0185 0.0198 0.0276 0.0286 0.1435	3.78	84%	mem
	bf16	1.03×	0.0178	flashinfer torch-compile quack vllm flaggems torch-ref	0.0183 0.0185 0.0195 0.0288 0.0292 0.1446	3.77	84%	mem
quack-throughput-1024	f16	1.04×	0.0339	flashinfer torch-compile quack flaggems vllm torch-ref	0.0352 0.0353 0.0358 0.0379 0.0438 0.2445	3.96	88%	mem
	bf16	1.01×	0.0340	flashinfer torch-compile quack flaggems vllm torch-ref	0.0345 0.0353 0.0357 0.0384 0.0455 0.2463	3.95	88%	mem
quack-throughput-2048	f16	1.02×	0.0655	flaggems quack flashinfer vllm torch-compile torch-ref	0.0667 0.0678 0.0684 0.0859 0.0869 0.4645	4.1	91%	mem
	bf16	1.02×	0.0652	flaggems quack flashinfer vllm torch-compile torch-ref	0.0668 0.0681 0.0685 0.0882 0.0892 0.4680	4.11	91%	mem
quack-throughput-4096	f16	1.00×	0.1320	quack flashinfer flaggems vllm torch-compile torch-ref	0.1324 0.1326 0.1341 0.1463 0.1601 0.9838	4.07	90%	mem
	bf16	0.99×	0.1332	quack flashinfer flaggems vllm torch-compile torch-ref	0.1321 0.1330 0.1339 0.1485 0.1654 0.9906	4.03	90%	mem
quack-throughput-8192	f16	1.00×	0.2504	torch-compile quack flashinfer flaggems vllm torch-ref	0.2510 0.2579 0.2636 0.2673 0.2806 2.0088	4.29	95%	mem
	bf16	1.02×	0.2496	torch-compile quack flashinfer flaggems vllm torch-ref	0.2546 0.2588 0.2637 0.2783 0.2826 2.0231	4.3	96%	mem
quack-throughput-16384	f16	0.98×	0.5252	torch-compile flashinfer quack flaggems vllm torch-ref	0.5141 0.5196 0.5229 0.5371 0.7252 4.0289	4.09	91%	mem
	bf16	0.95×	0.5398	torch-compile flashinfer quack flaggems vllm torch-ref	0.5129 0.5215 0.5218 0.5527 0.7258 4.0586	3.98	88%	mem
quack-throughput-32768	f16	1.00×	0.9968	torch-compile quack flaggems flashinfer vllm torch-ref	0.9948 1.0325 1.1470 1.3994 1.5144 8.0751	4.31	96%	mem
	bf16	1.02×	0.9982	torch-compile quack flaggems flashinfer vllm torch-ref	1.0152 1.0389 1.1778 1.4172 1.5152 8.1327	4.3	96%	mem
quack-throughput-65536	f16	1.02×	2.1005	quack flaggems torch-compile vllm torch-ref flashinfer	2.1341 2.6410 2.9323 3.0779 16.1635 29.1806	4.09	91%	mem
	bf16	1.02×	2.1713	quack flaggems torch-compile vllm torch-ref flashinfer	2.2091 2.6343 2.9420 3.0687 16.2620 39.4506	3.96	88%	mem
quack-throughput-131072	f16	1.09×	2.0375	quack flaggems torch-compile vllm flashinfer torch-ref	2.2155 2.9134 3.1046 3.1225 3.7997 16.2577	4.22	94%	mem
	bf16	1.12×	2.0499	quack flaggems torch-compile vllm flashinfer torch-ref	2.2947 2.9143 3.1040 3.1216 4.0071 16.3698	4.19	93%	mem
quack-throughput-262144	f16	1.10×	2.1663	quack flaggems vllm torch-compile torch-ref flashinfer	2.3920 3.1032 3.1341 3.1404 16.2714 27.1069	3.97	88%	mem
	bf16	1.14×	2.1948	quack flaggems vllm torch-compile torch-ref flashinfer	2.5057 3.0968 3.1276 3.1406 16.3816 44.2550	3.91	87%	mem
qwen3-8b-qk-norm-decode	bf16	1.02×	0.00144	quack torch-compile torch-ref	0.00147 0.0025 0.00864	0.0114	0%	lat
qk-norm-head	bf16	1.01×	0.00285	torch-compile quack torch-ref	0.00288 0.00317 0.0180	1.1	33%	lat

FusedAddRMSNormFwd

x, residual: [*B, N]weight: [N]

• llama-8b-prefillx, residual: [2048, 4096]weight: [4096]dtype=f16, bf16
• llama-8b-decodex, residual: [1, 4096]weight: [4096]dtype=bf16
• llama-70b-prefillx, residual: [2048, 8192]weight: [8192]dtype=f16, bf16
• llama-70b-decodex, residual: [1, 8192]weight: [8192]dtype=bf16
• llama-405b-prefillx, residual: [2048, 16384]weight: [16384]dtype=f16, bf16
• llama-405b-decodex, residual: [1, 16384]weight: [16384]dtype=bf16
• llama-13b-prefillx, residual: [2048, 5120]weight: [5120]dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
llama-8b-prefill	f16	1.02×	0.0185	vllm flashinfer torch-compile torch-ref	0.0188 0.0194 0.0195 0.1589	2.27	81%	mem
	bf16	1.03×	0.0185	vllm flashinfer torch-compile torch-ref	0.0191 0.0193 0.0196 0.1599	2.26	80%	mem
llama-8b-decode	bf16	1.23×	0.00192	flashinfer torch-compile vllm torch-ref	0.00237 0.00237 0.00298 0.0208	0.0107	0%	lat
llama-70b-prefill	f16	1.03×	0.0349	vllm flashinfer torch-compile torch-ref	0.0358 0.0362 0.0365 0.3246	2.4	85%	mem
	bf16	1.03×	0.0353	flashinfer vllm torch-compile torch-ref	0.0363 0.0364 0.0366 0.3269	2.37	84%	mem
llama-70b-decode	bf16	1.33×	0.00211	flashinfer vllm torch-compile torch-ref	0.00282 0.00349 0.00506 0.0242	0.0194	1%	lat
llama-405b-prefill	f16	1.05×	0.0683	torch-compile flashinfer vllm torch-ref	0.0715 0.0724 0.0780 0.6348	2.46	87%	mem
	bf16	1.05×	0.0683	torch-compile flashinfer vllm torch-ref	0.0716 0.0725 0.0782 0.6395	2.46	87%	mem
llama-405b-decode	bf16	1.52×	0.00285	flashinfer vllm torch-compile torch-ref	0.00432 0.00496 0.00605 0.0299	0.0288	1%	lat
llama-13b-prefill	bf16	1.06×	0.0224	vllm flashinfer torch-compile torch-ref	0.0236 0.0241 0.0257 0.2036	2.34	83%	mem

AdaLayerNormFwd

x, scale, shift: [*B, N]

• dit-xl-2x, scale, shift: [1024, 1152]dtype=f16, bf16, f32
• llama-8b-prefillx, scale, shift: [2048, 4096]dtype=f16, bf16
• llama-8b-decodex, scale, shift: [1, 4096]dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
dit-xl-2	f16	1.38×	0.00551	torch-compile torch-ref	0.00758 0.0373	1.5	38%	lat
	bf16	1.39×	0.0056	torch-compile torch-ref	0.00778 0.0375	1.47	37%	lat
	f32	1.12×	0.00733	torch-compile torch-ref	0.00819 0.0157	1.13	57%	mem
llama-8b-prefill	f16	1.31×	0.0197	torch-compile torch-ref	0.0259 0.1772	2.98	76%	mem
	bf16	1.36×	0.0197	torch-compile torch-ref	0.0268 0.1778	2.98	76%	mem
llama-8b-decode	bf16	0.99×	0.00269	torch-compile torch-ref	0.00267 0.0204	0.0107	0%	lat

AdaLayerNormZeroFwd

x, scale, shift, gate: [*B, N]

• dit-xl-2x, scale, shift, gate: [1024, 1152]dtype=f16, bf16, f32
• llama-8b-prefillx, scale, shift, gate: [2048, 4096]dtype=f16, bf16
• llama-8b-decodex, scale, shift, gate: [1, 4096]dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
dit-xl-2	f16	1.28×	0.00618	torch-compile torch-ref	0.0079 0.0481	1.53	42%	lat
	bf16	1.31×	0.00627	torch-compile torch-ref	0.00822 0.0484	1.5	42%	lat
	f32	1.15×	0.00925	torch-compile torch-ref	0.0106 0.0199	1.02	57%	mem
llama-8b-prefill	f16	1.12×	0.0246	torch-compile torch-ref	0.0276 0.2315	2.73	76%	mem
	bf16	1.16×	0.0244	torch-compile torch-ref	0.0284 0.2332	2.75	76%	mem
llama-8b-decode	bf16	1.13×	0.00285	torch-compile torch-ref	0.00323 0.0251	0.0115	0%	lat

BatchNormFwd

x: [B, C, *L]running_mean, running_var, weight, bias: [C], f32

• resnet-fcx: [32, 64]running_mean, running_var, weight, bias: [64], f32training=truedtype=f16
• resnet-stage1x: [8, 64, 32, 32]running_mean, running_var, weight, bias: [64], f32training=truedtype=f16, bf16
• resnet-stage2x: [4, 128, 32, 32]running_mean, running_var, weight, bias: [128], f32training=truedtype=f16
• resnet-stage3x: [4, 256, 28, 28]running_mean, running_var, weight, bias: [256], f32training=truedtype=f16, f32
• large-spatialx: [4, 128, 1024, 1024]running_mean, running_var, weight, bias: [128], f32training=truedtype=f16
• resnet-stage3-evalx: [4, 256, 28, 28]running_mean, running_var, weight, bias: [256], f32dtype=f16

x: [B, C, *L]

• resnet-stemx: [8, 64, 56, 56]training=truedtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
resnet-fc	f16	1.44×	0.00183	torch-compile flaggems torch-cudnn	0.00262 0.00717 0.0128	0.00786	0%	lat
resnet-stage1	f16	1.58×	0.00282	torch-compile flaggems torch-cudnn	0.00445 0.0123 0.0134	1.3	17%	lat
	bf16	1.63×	0.00275	torch-compile flaggems torch-cudnn	0.00448 0.0124 0.0134	1.33	17%	lat
resnet-stage2	f16	1.43×	0.00259	torch-compile flaggems torch-cudnn	0.00371 0.0108 0.0124	1.42	18%	lat
resnet-stage3	f16	1.42×	0.00349	torch-compile flaggems torch-cudnn	0.00496 0.0132 0.0134	1.61	20%	lat
	f32	1.26×	0.00445	torch-compile flaggems torch-cudnn	0.00558 0.0068 0.00726	1.26	32%	lat
large-spatial	f16	1.01×	0.7504	torch-compile flaggems torch-cudnn	0.7547 4.6856 5.3287	5.01	64%	mem
resnet-stage3-eval	f16	2.28×	0.00266	torch-compile flaggems torch-cudnn	0.00605 0.0102 0.0108	1.21	27%	lat
resnet-stem	bf16	1.38×	0.00646	torch-compile torch-cudnn	0.00893 0.0360	1.24	22%	lat

BatchNormBwd

grad_out, x: [B, C, *L]weight, mean, rstd: [C], f32

• resnet-fcgrad_out, x: [32, 64]weight, mean, rstd: [64], f32dtype=f16
• resnet-stage1grad_out, x: [8, 64, 32, 32]weight, mean, rstd: [64], f32dtype=f16, bf16
• resnet-stage2grad_out, x: [4, 128, 32, 32]weight, mean, rstd: [128], f32dtype=f16
• resnet-stage3grad_out, x: [4, 256, 28, 28]weight, mean, rstd: [256], f32dtype=f16, f32
• large-spatialgrad_out, x: [4, 128, 1024, 1024]weight, mean, rstd: [128], f32dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
resnet-fc	f16	7.16×	0.00195	torch-native-batch-norm torch-autograd	0.0140 0.0248	0.00944	0%	lat
resnet-stage1	f16	6.21×	0.00339	torch-native-batch-norm torch-autograd	0.0211 0.0296	1.39	21%	lat
	bf16	6.34×	0.00333	torch-native-batch-norm torch-autograd	0.0211 0.0296	1.42	21%	lat
resnet-stage2	f16	5.89×	0.00294	torch-native-batch-norm torch-autograd	0.0173 0.0260	1.6	24%	lat
resnet-stage3	f16	4.65×	0.00419	torch-native-batch-norm torch-autograd	0.0195 0.0279	1.72	26%	lat
	f32	1.86×	0.00547	torch-native-batch-norm torch-autograd	0.0102 0.0185	1.32	39%	lat
large-spatial	f16	10.37×	1.2059	torch-native-batch-norm torch-autograd	12.4994 14.1448	4.01	59%	mem

GroupNormFwd

x: [B, C, *L]weight, bias: [C]

• image-affinex: [8, 128, 32, 32]weight, bias: [128]num_groups=32dtype=f16, bf16
• wider-channel-affinex: [4, 256, 28, 28]weight, bias: [256]num_groups=32dtype=f16, f32
• tail-spatial-affinex: [4, 128, 30, 30]weight, bias: [128]num_groups=16dtype=f16

x: [B, C, *L]

• imagex: [8, 128, 32, 32]num_groups=32dtype=f16, bf16
• wider-channelx: [4, 256, 28, 28]num_groups=32dtype=f16, f32
• tail-spatialx: [4, 128, 30, 30]num_groups=16dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
image-affine	f16	1.21×	0.00327	flaggems torch-compile torch	0.00394 0.00518 0.0155	2.25	29%	lat
	bf16	1.20×	0.0033	flaggems torch-compile torch	0.00397 0.00531 0.0155	2.23	28%	lat
wider-channel-affine	f16	1.02×	0.00394	flaggems torch-compile torch	0.004 0.00435 0.0165	1.43	18%	lat
	f32	1.01×	0.0048	flaggems torch-compile torch	0.00483 0.00509 0.0165	1.17	30%	lat
tail-spatial-affine	f16	1.07×	0.00391	flaggems torch-compile torch	0.00416 0.00451 0.0156	0.826	10%	lat
image	f16	1.17×	0.00317	flaggems torch-compile torch	0.00371 0.00438 0.0134	1.65	29%	lat
	bf16	1.17×	0.00317	flaggems torch-compile torch	0.00371 0.00461 0.0134	1.65	29%	lat
wider-channel	f16	1.03×	0.00349	flaggems torch-compile torch	0.00358 0.00387 0.0143	1.15	20%	lat
	f32	1.01×	0.00442	flaggems torch-compile torch	0.00445 0.00486 0.0143	0.909	32%	lat
tail-spatial	f16	1.12×	0.00317	torch-compile flaggems torch	0.00355 0.00368 0.0138	0.727	13%	lat

InstanceNormFwd

x: [B, C, *L]weight, bias: [C]

• image-affinex: [8, 128, 32, 32]weight, bias: [128]dtype=f16, bf16
• wider-channel-affinex: [4, 256, 28, 28]weight, bias: [256]dtype=f16, f32
• tail-spatial-affinex: [4, 64, 30, 30]weight, bias: [64]dtype=f16

x: [B, C, *L]

• imagex: [8, 128, 32, 32]dtype=f16, bf16
• wider-channelx: [4, 256, 28, 28]dtype=f16, f32
• tail-spatialx: [4, 64, 30, 30]dtype=f16

x: [B, C, *L]running_mean, running_var: [C], f32

• image-track-statsx: [8, 128, 32, 32]running_mean, running_var: [128], f32dtype=f16

x: [B, C, *L]running_mean, running_var: [C], f32weight, bias: [C]

• image-evalx: [8, 128, 32, 32]running_mean, running_var: [128], f32weight, bias: [128]use_input_stats=falsedtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
image-affine	f16	1.05×	0.00291	torch-compile flaggems torch	0.00307 0.00378 0.0208	2.52	32%	lat
	bf16	1.03×	0.00294	torch-compile flaggems torch	0.00304 0.00378 0.0208	2.49	32%	lat
wider-channel-affine	f16	1.07×	0.00272	torch-compile flaggems torch	0.00291 0.00355 0.0207	2.07	26%	lat
	f32	1.10×	0.00342	torch-compile flaggems torch	0.00378 0.00442 0.0195	1.64	42%	lat
tail-spatial-affine	f16	1.01×	0.00237	torch-compile flaggems torch	0.0024 0.00282 0.0114	0.681	9%	lat
image	f16	1.03×	0.00285	torch-compile flaggems torch	0.00294 0.0035 0.0171	1.84	33%	lat
	bf16	1.02×	0.00288	torch-compile flaggems torch	0.00294 0.00352 0.0171	1.82	32%	lat
wider-channel	f16	1.05×	0.00266	torch-compile flaggems torch	0.00278 0.0033 0.0162	1.51	27%	lat
	f32	1.05×	0.00342	torch-compile flaggems torch	0.00359 0.00403 0.0162	1.17	42%	lat
tail-spatial	f16	1.04×	0.00224	torch-compile flaggems torch	0.00233 0.00266 0.00829	0.514	9%	lat
image-track-stats	f16	1.00×	0.00294	torch-compile torch	0.00294 0.0338	1.78	32%	lat
image-eval	f16	1.53×	0.00282	torch-compile torch	0.00432 0.0180	1.49	33%	lat