Reduction

本页暂无中文版。以下为英文原文。

19 ops, 176 workloads.

One table per op, one row per workload. Ratio is the fastest other implementation's device time divided by ours, so green is faster than it, plain is level with it, red is slower. Times are in ms. How these numbers are taken.

SumFwd

x: [*S]

• hidden-statex: [2048, 4096]dim=-1dtype=f16, bf16, bf16/f32
• long-seqx: [64, 32768]dim=-1dtype=bf16
• hidden-state-dim0x: [2048, 4096]dim=0dtype=bf16
• hidden-state-keepdimx: [2048, 4096]dim=-1keepdim=truedtype=bf16
• 3d-multidimx: [4, 128, 4096]dim=[0,2]dtype=f16
• mlp-intermediatex: [2048, 11008]dim=-1dtype=f32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state	f16	1.20×	0.00685	torch-compile flaggems torch	0.00819 0.00861 0.0504	1.22	54%	mem
	bf16	1.24×	0.00688	torch-compile flaggems torch	0.00854 0.0089 0.0506	1.22	54%	mem
	bf16/f32	1.25×	0.00685	torch-compile torch	0.00854 0.0495	1.22	54%	mem
long-seq	bf16	1.06×	0.00387	flaggems torch-compile torch	0.0041 0.00474 0.0172	0.542	24%	lat
hidden-state-dim0	bf16	1.15×	0.00954	torch-compile flaggems torch	0.0109 0.0140 0.0451	0.88	39%	mem
hidden-state-keepdim	bf16	1.24×	0.00688	torch-compile flaggems torch	0.00854 0.0089 0.0507	1.22	54%	mem
3d-multidim	f16	0.98×	0.00477	torch-compile flaggems torch	0.00467 0.0136 0.0196	0.44	20%	lat
mlp-intermediate	f32	0.98×	0.0251	torch-compile flaggems torch	0.0245 0.0288 0.0383	0.899	80%	mem

MeanFwd

x: [*S]

• hidden-statex: [2048, 4096]dim=-1dtype=f16, bf16, bf16/f32
• long-seqx: [64, 32768]dim=-1dtype=bf16
• 3d-multidimx: [4, 128, 4096]dim=[0,2]dtype=f16
• mlp-intermediatex: [2048, 11008]dim=-1dtype=f32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state	f16	1.24×	0.00685	torch-compile flaggems torch	0.00848 0.00896 0.0504	1.23	54%	mem
	bf16	1.25×	0.00688	torch-compile flaggems torch	0.00858 0.00899 0.0507	1.22	54%	mem
	bf16/f32	1.24×	0.00688	torch-compile torch	0.00854 0.0496	1.22	54%	mem
long-seq	bf16	1.06×	0.00387	flaggems torch-compile torch	0.0041 0.0047 0.0172	0.542	24%	lat
3d-multidim	f16	0.99×	0.0048	torch-compile flaggems torch	0.00475 0.0136 0.0195	0.437	19%	lat
mlp-intermediate	f32	0.98×	0.0251	torch-compile flaggems torch	0.0245 0.0288 0.0381	0.898	80%	mem

ProdFwd

x: [*S]

• hidden-statex: [2048, 4096]dim=-1dtype=f16, bf16, bf16/f32
• long-seqx: [64, 32768]dim=-1dtype=bf16
• 3d-non-last-axisx: [4, 128, 4096]dim=0dtype=f16
• mlp-intermediatex: [2048, 11008]dim=-1dtype=f32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state	f16	1.13×	0.00688	flaggems torch-compile torch	0.00778 0.00829 0.0501	1.22	54%	mem
	bf16	1.13×	0.00688	flaggems torch-compile torch	0.00774 0.00835 0.0503	1.22	54%	mem
	bf16/f32	1.21×	0.00688	torch-compile torch	0.00835 0.0494	1.22	54%	mem
long-seq	bf16	1.20×	0.0039	torch-compile flaggems torch	0.0047 0.0141 0.0171	0.537	24%	lat
3d-non-last-axis	f16	1.01×	0.00349	torch-compile torch flaggems	0.00352 0.0153 0.2858	0.601	33%	lat
mlp-intermediate	f32	0.98×	0.0252	torch-compile flaggems torch	0.0247 0.0307 0.0380	0.896	80%	mem

AmaxFwd

x: [*S]

• hidden-statex: [2048, 4096]dim=-1dtype=f16, bf16
• long-seqx: [64, 32768]dim=-1dtype=bf16
• 3d-multidimx: [4, 128, 4096]dim=[0,2]dtype=f16
• mlp-intermediatex: [2048, 11008]dim=-1dtype=f32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state	f16	1.11×	0.00688	flaggems torch-compile torch	0.00762 0.0103 0.0505	1.22	54%	mem
	bf16	1.14×	0.00688	flaggems torch-compile torch	0.00787 0.0102 0.0508	1.22	54%	mem
long-seq	bf16	1.29×	0.00387	torch-compile flaggems torch	0.00499 0.0142 0.0173	0.542	24%	lat
3d-multidim	f16	1.11×	0.00477	torch-compile flaggems torch	0.00528 0.0125 0.0194	0.44	20%	lat
mlp-intermediate	f32	0.99×	0.0251	torch-compile flaggems torch	0.0250 0.0307 0.0396	0.899	80%	mem

AminFwd

x: [*S]

• hidden-statex: [2048, 4096]dim=-1dtype=f16, bf16
• long-seqx: [64, 32768]dim=-1dtype=bf16
• 3d-multidimx: [4, 128, 4096]dim=[0,2]dtype=f16
• mlp-intermediatex: [2048, 11008]dim=-1dtype=f32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state	f16	1.49×	0.00688	torch-compile torch	0.0103 0.0505	1.22	54%	mem
	bf16	1.49×	0.00688	torch-compile torch	0.0103 0.0509	1.22	54%	mem
long-seq	bf16	1.28×	0.00384	torch-compile torch	0.00493 0.0173	0.546	24%	lat
3d-multidim	f16	1.11×	0.00477	torch-compile torch	0.00528 0.0194	0.44	20%	lat
mlp-intermediate	f32	1.00×	0.0251	torch-compile torch	0.0250 0.0396	0.9	80%	mem

VarFwd

x: [*S]

• hidden-statex: [2048, 4096]dim=-1dtype=f16, bf16
• long-seqx: [64, 32768]dim=-1dtype=bf16
• 3d-multidimx: [4, 128, 4096]dim=[0,2]dtype=f16
• mlp-intermediatex: [2048, 11008]dim=-1dtype=f32
• hidden-state-biasedx: [2048, 4096]dim=-1correction=0dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state	f16	1.79×	0.00842	flaggems torch-compile torch	0.0150 0.0183 0.0687	3.99	44%	mem
	bf16	1.82×	0.00848	flaggems torch-compile torch	0.0155 0.0188 0.0691	3.96	44%	mem
long-seq	bf16	1.19×	0.00525	torch-compile flaggems torch	0.00624 0.0116 0.0252	1.6	18%	lat
3d-multidim	f16	1.22×	0.00534	torch-compile flaggems torch	0.0065 0.0142 0.0271	1.57	17%	lat
mlp-intermediate	f32	1.16×	0.0298	torch-compile flaggems torch	0.0344 0.0366 0.0565	3.03	67%	mem
hidden-state-biased	bf16	1.86×	0.00832	flaggems torch-compile torch	0.0155 0.0188 0.0691	4.03	45%	mem

VarMeanFwd

x: [*S]

• hidden-statex: [2048, 4096]dim=-1dtype=f16, bf16
• long-seqx: [64, 32768]dim=-1dtype=bf16
• 3d-multidimx: [4, 128, 4096]dim=[0,2]dtype=f16
• mlp-intermediatex: [2048, 11008]dim=-1dtype=f32
• hidden-state-biasedx: [2048, 4096]dim=-1correction=0dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state	f16	1.79×	0.00848	flaggems torch-compile torch	0.0152 0.0189 0.0698	3.96	44%	mem
	bf16	1.82×	0.00854	flaggems torch-compile torch	0.0156 0.0194 0.0702	3.93	44%	mem
long-seq	bf16	1.21×	0.00525	torch-compile flaggems torch	0.00637 0.0116 0.0264	1.6	18%	lat
3d-multidim	f16	1.16×	0.00534	torch-compile flaggems torch	0.00618 0.0143 0.0284	1.57	17%	lat
mlp-intermediate	f32	1.16×	0.0298	torch-compile flaggems torch	0.0347 0.0367 0.0566	3.02	67%	mem
hidden-state-biased	bf16	1.86×	0.00838	flaggems torch-compile torch	0.0156 0.0194 0.0702	4	44%	mem

StdFwd

x: [*S]

• hidden-statex: [2048, 4096]dim=-1dtype=f16, bf16
• long-seqx: [64, 32768]dim=-1dtype=bf16
• 3d-multidimx: [4, 128, 4096]dim=[0,2]dtype=f16
• mlp-intermediatex: [2048, 11008]dim=-1dtype=f32
• hidden-state-biasedx: [2048, 4096]dim=-1correction=0dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state	f16	1.25×	0.00854	flaggems torch-compile torch	0.0107 0.0188 0.0688	3.93	44%	mem
	bf16	1.29×	0.00861	flaggems torch-compile torch	0.0111 0.0193 0.0692	3.9	43%	mem
long-seq	bf16	1.20×	0.00528	torch-compile flaggems torch	0.00634 0.0138 0.0252	1.59	18%	lat
3d-multidim	f16	1.17×	0.00536	torch-compile flaggems torch	0.00627 0.0144 0.0271	1.57	17%	lat
mlp-intermediate	f32	1.17×	0.0300	torch-compile flaggems torch	0.0350 0.0499 0.0566	3.01	67%	mem
hidden-state-biased	bf16	1.31×	0.00848	flaggems torch-compile torch	0.0111 0.0194 0.0691	3.96	44%	mem

ArgmaxFwd

x: [*S]

• lm-headx: [4, 102400]dim=-1dtype=f16, bf16
• hidden-statex: [2048, 4096]dim=-1dtype=f16, bf16
• 3d-non-last-axisx: [4, 128, 4096]dim=0dtype=f16
• llama-8b-lm-headx: [4, 128256]dim=-1dtype=f32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
lm-head	f16	7.25×	0.004	torch-compile flaggems torch	0.0290 0.0305 0.0358	0.102	5%	lat
	bf16	6.94×	0.004	flaggems torch-compile torch	0.0277 0.0295 0.0364	0.102	5%	lat
hidden-state	f16	1.34×	0.00925	flaggems torch-compile torch	0.0124 0.0191 0.0251	0.907	40%	mem
	bf16	1.21×	0.0097	flaggems torch-compile torch	0.0117 0.0194 0.0253	0.865	38%	mem
3d-non-last-axis	f16	1.00×	0.00381	flaggems torch-compile torch	0.00381 0.00464 0.0113	0.551	49%	lat
llama-8b-lm-head	f32	4.78×	0.00771	flaggems torch-compile torch	0.0368 0.0379 0.0444	0.0665	6%	lat

ArgminFwd

x: [*S]

• lm-headx: [4, 102400]dim=-1dtype=f16, bf16
• hidden-statex: [2048, 4096]dim=-1dtype=f16, bf16
• 3d-non-last-axisx: [4, 128, 4096]dim=0dtype=f16
• llama-8b-lm-headx: [4, 128256]dim=-1dtype=f32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
lm-head	f16	7.21×	0.00403	torch-compile torch flaggems	0.0291 0.0358 0.1157	0.102	5%	lat
	bf16	4.36×	0.00675	torch-compile torch flaggems	0.0295 0.0363 0.1013	0.0607	3%	lat
hidden-state	f16	1.03×	0.0097	flaggems torch-compile torch	0.0100 0.0191 0.0252	0.865	38%	mem
	bf16	1.00×	0.00989	flaggems torch-compile torch	0.00992 0.0195 0.0253	0.848	38%	mem
3d-non-last-axis	f16	1.00×	0.00384	torch-compile torch	0.00384 0.0112	0.546	49%	lat
llama-8b-lm-head	f32	4.89×	0.00774	torch-compile torch flaggems	0.0379 0.0445 0.2003	0.0662	6%	lat

SoftmaxFwd

x: [*S]

dim=-1

• attn-weights-4kx: [32, 32, 4096]dtype=f16, bf16, f32
• attn-weights-32kx: [32, 32, 32768]dtype=bf16
• lm-headx: [4, 102400]dtype=f16, bf16, f32
• llama-8b-lm-headx: [4, 128256]dtype=bf16/f32
• quack-throughput-256x: [32768, 256]dtype=f16, bf16, f32
• quack-throughput-512x: [32768, 512]dtype=f16, bf16, f32
• quack-throughput-1024x: [32768, 1024]dtype=f16, bf16, f32
• quack-throughput-2048x: [32768, 2048]dtype=f16, bf16, f32
• quack-throughput-4096x: [32768, 4096]dtype=f16, bf16, f32
• quack-throughput-8192x: [32768, 8192]dtype=f16, bf16, f32
• quack-throughput-16384x: [32768, 16384]dtype=f16, bf16, f32
• quack-throughput-32768x: [32768, 32768]dtype=f16, bf16, f32
• quack-throughput-65536x: [32768, 65536]dtype=f16, bf16, f32
• quack-throughput-131072x: [16384, 131072]dtype=f16, bf16, f32
• quack-throughput-262144x: [8192, 262144]dtype=f16, bf16, f32
• qwen3-8b-lm-head-b1x: [1, 151936]dtype=f32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
attn-weights-4k	f16	1.13×	0.00634	quack flaggems torch-compile torch	0.00717 0.00867 0.0161 0.0396	3.31	59%	mem
	bf16	1.14×	0.0063	quack flaggems torch-compile torch	0.0072 0.00883 0.0166 0.0396	3.33	59%	mem
	f32	1.17×	0.00931	quack flaggems torch-compile torch	0.0109 0.0111 0.0186 0.0200	2.25	80%	mem
attn-weights-32k	bf16	1.08×	0.0351	quack flaggems torch-compile torch	0.0379 0.0645 0.0945 0.2794	4.77	85%	mem
lm-head	f16	0.96×	0.00448	quack torch-compile flaggems torch	0.00429 0.00925 0.0284 0.0402	0.457	8%	lat
	bf16	0.99×	0.00451	quack torch-compile flaggems torch	0.00445 0.00973 0.0300 0.0402	0.454	8%	lat
	f32	1.63×	0.00499	quack torch-compile flaggems torch	0.00813 0.0095 0.0316 0.0398	0.41	15%	lat
llama-8b-lm-head	bf16/f32	1.76×	0.00531	torch-compile quack torch	0.00936 0.0131 0.0471	0.483	13%	lat
quack-throughput-256	f16	1.10×	0.00992	quack torch-compile flaggems torch	0.0109 0.0127 0.0275 0.0616	4.23	75%	mem
	bf16	1.10×	0.0100	quack torch-compile flaggems torch	0.0110 0.0134 0.0277 0.0621	4.17	74%	mem
	f32	1.06×	0.0171	torch-compile quack torch flaggems	0.0182 0.0184 0.0204 0.0295	2.45	87%	mem
quack-throughput-512	f16	1.04×	0.0177	quack torch-compile flaggems torch	0.0185 0.0225 0.0320 0.1226	4.73	84%	mem
	bf16	1.00×	0.0185	quack torch-compile flaggems torch	0.0186 0.0236 0.0324 0.1234	4.53	80%	mem
	f32	1.04×	0.0323	quack torch-compile flaggems torch	0.0335 0.0340 0.0386 0.0388	2.6	92%	mem
quack-throughput-1024	f16	1.04×	0.0331	quack torch-compile flaggems torch	0.0345 0.0372 0.0422 0.2319	5.07	90%	mem
	bf16	1.03×	0.0334	quack torch-compile flaggems torch	0.0345 0.0402 0.0439 0.2342	5.02	89%	mem
	f32	1.02×	0.0641	flaggems quack torch-compile torch	0.0656 0.0658 0.0670 0.0705	2.62	93%	mem
quack-throughput-2048	f16	1.04×	0.0643	quack flaggems torch-compile torch	0.0670 0.0879 0.2042 0.4724	5.22	93%	mem
	bf16	1.01×	0.0658	quack flaggems torch-compile torch	0.0668 0.0895 0.2117 0.4758	5.1	91%	mem
	f32	1.05×	0.1256	flaggems quack torch torch-compile	0.1321 0.1400 0.1572 0.2356	2.67	95%	mem
quack-throughput-4096	f16	1.01×	0.1290	quack flaggems torch-compile torch	0.1308 0.1904 0.4047 1.1616	5.2	92%	mem
	bf16	1.03×	0.1255	quack flaggems torch-compile torch	0.1293 0.1919 0.4164 1.1701	5.35	95%	mem
	f32	1.04×	0.2510	flaggems quack torch-compile torch	0.2604 0.2821 0.4136 0.5376	2.67	95%	mem
quack-throughput-8192	f16	1.04×	0.2501	quack flaggems torch-compile torch	0.2612 0.2789 0.7872 2.6301	5.37	95%	mem
	bf16	1.05×	0.2500	quack flaggems torch-compile torch	0.2625 0.2866 0.7956 2.6438	5.37	95%	mem
	f32	1.04×	0.4980	flaggems quack torch-compile torch	0.5191 0.5609 0.7229 1.3855	2.7	96%	mem
quack-throughput-16384	f16	1.04×	0.4991	quack flaggems torch-compile torch	0.5173 0.6724 1.3988 3.9435	5.38	96%	mem
	bf16	1.05×	0.5015	quack flaggems torch-compile torch	0.5276 0.6727 1.4761 3.9686	5.35	95%	mem
	f32	1.05×	0.9959	quack flaggems torch-compile torch	1.0482 1.0787 1.3605 1.4597	2.7	96%	mem
quack-throughput-32768	f16	1.01×	1.0130	quack flaggems torch-compile torch	1.0271 1.8375 2.6446 8.2614	5.3	94%	mem
	bf16	1.07×	0.9969	quack flaggems torch-compile torch	1.0667 1.9683 2.7911 8.3184	5.39	96%	mem
	f32	1.08×	1.9855	quack flaggems torch-compile torch	2.1450 2.2304 3.0248 3.2975	2.7	96%	mem
quack-throughput-65536	f16	1.10×	1.9873	quack flaggems torch-compile torch	2.1782 3.3289 5.2408 17.9628	5.4	96%	mem
	bf16	1.13×	1.9885	quack flaggems torch-compile torch	2.2475 3.4418 5.4473 18.0840	5.4	96%	mem
	f32	1.06×	4.1362	quack flaggems torch-compile torch	4.3984 5.7127 6.1339 8.0287	2.6	92%	mem
quack-throughput-131072	f16	1.10×	2.0571	quack flaggems torch-compile torch	2.2554 3.3157 5.1956 18.0238	5.22	93%	mem
	bf16	1.07×	2.1626	quack flaggems torch-compile torch	2.3186 3.4269 5.3917 18.1373	4.97	88%	mem
	f32	1.19×	4.0337	quack flaggems torch-compile torch	4.8113 5.9786 6.1635 8.0797	2.66	95%	mem
quack-throughput-262144	f16	1.12×	2.1757	quack flaggems torch-compile torch	2.4282 3.3956 5.2081 18.0531	4.94	88%	mem
	bf16	1.11×	2.2773	quack flaggems torch-compile torch	2.5258 3.4779 5.4074 18.1721	4.71	84%	mem
	f32	1.09×	4.5364	quack flaggems torch-compile torch	4.9370 6.1701 6.2321 8.1077	2.37	84%	mem
qwen3-8b-lm-head-b1	f32	1.30×	0.00454	quack torch-compile flaggems torch	0.00592 0.00951 0.0461 0.0568	0.167	6%	lat

LogSoftmaxFwd

x: [*S]

dim=-1

• attn-weights-4kx: [32, 32, 4096]dtype=f16, bf16, f32
• attn-weights-32kx: [32, 32, 32768]dtype=bf16
• lm-headx: [4, 102400]dtype=f16, bf16, f32
• llama-8b-lm-headx: [4, 128256]dtype=bf16/f32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
attn-weights-4k	f16	2.01×	0.00734	torch-compile flaggems torch	0.0148 0.0202 0.0368	2.86	51%	mem
	bf16	2.01×	0.00771	torch-compile flaggems torch	0.0155 0.0207 0.0371	2.72	48%	mem
	f32	1.53×	0.0104	torch-compile torch flaggems	0.0158 0.0182 0.0207	2.02	72%	mem
attn-weights-32k	bf16	2.11×	0.0341	torch-compile flaggems torch	0.0721 0.2570 0.2725	4.92	87%	mem
lm-head	f16	2.18×	0.00448	torch-compile torch flaggems	0.00976 0.0358 0.4264	0.457	8%	lat
	bf16	2.15×	0.00454	torch-compile torch flaggems	0.00979 0.0358 0.4275	0.451	8%	lat
	f32	2.03×	0.00499	torch-compile torch flaggems	0.0101 0.0355 0.4101	0.41	15%	lat
llama-8b-lm-head	bf16/f32	1.88×	0.00536	torch-compile torch	0.0101 0.0419	0.479	13%	lat

LogSumExpFwd

x: [*S]

• attn-weights-4kx: [32, 32, 4096]dim=-1dtype=f16, bf16
• attn-weights-32kx: [32, 32, 32768]dim=-1dtype=bf16
• lm-headx: [4, 102400]dim=-1dtype=f16, bf16
• 3d-multidimx: [4, 128, 4096]dim=[0,2]dtype=f16
• llama-8b-lm-headx: [4, 128256]dim=-1dtype=f32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
attn-weights-4k	f16	1.55×	0.00627	torch-compile torch	0.0097 0.0657	2.67	30%	lat
	bf16	1.52×	0.00656	torch-compile torch	0.00995 0.0660	2.56	28%	lat
attn-weights-32k	bf16	1.52×	0.0272	torch-compile torch	0.0413 0.3889	4.93	55%	mem
lm-head	f16	2.65×	0.00416	torch-compile torch	0.0110 0.0457	0.394	4%	lat
	bf16	2.58×	0.00413	torch-compile torch	0.0107 0.0459	0.397	4%	lat
3d-multidim	f16	1.62×	0.00624	torch-compile torch	0.0101 0.0489	1.34	15%	lat
llama-8b-lm-head	f32	1.63×	0.0047	torch-compile torch	0.00765 0.0606	0.436	10%	lat

VectorNormFwd

x: [*S]

• l1-hidden-statex: [2048, 4096]ord=1dim=-1dtype=f16, bf16
• l1-long-seqx: [64, 32768]ord=1dim=-1dtype=bf16
• l1-3d-multidimx: [4, 128, 4096]ord=1dim=[0,2]dtype=f16
• l1-mlp-intermediatex: [2048, 11008]ord=1dim=-1dtype=f32
• l1-hidden-state-fp32x: [2048, 4096]ord=1dim=-1dtype=bf16/f32
• l2-hidden-statex: [2048, 4096]dim=-1dtype=f16, bf16
• l2-long-seqx: [64, 32768]dim=-1dtype=bf16
• l2-3d-multidimx: [4, 128, 4096]dim=[0,2]dtype=f16
• l2-mlp-intermediatex: [2048, 11008]dim=-1dtype=f32
• l2-hidden-state-fp32x: [2048, 4096]dim=-1dtype=bf16/f32
• inf-hidden-statex: [2048, 4096]ord=infdim=-1dtype=f16, bf16
• inf-long-seqx: [64, 32768]ord=infdim=-1dtype=bf16
• inf-3d-multidimx: [4, 128, 4096]ord=infdim=[0,2]dtype=f16
• inf-mlp-intermediatex: [2048, 11008]ord=infdim=-1dtype=f32
• inf-hidden-state-fp32x: [2048, 4096]ord=infdim=-1dtype=bf16/f32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
l1-hidden-state	f16	1.20×	0.00685	torch-compile flaggems torch	0.00822 0.0150 0.0504	2.45	54%	mem
	bf16	1.24×	0.00688	torch-compile flaggems torch	0.00854 0.0153 0.0508	2.44	54%	mem
l1-long-seq	bf16	1.23×	0.00387	torch-compile torch flaggems	0.00477 0.0172 0.0376	1.08	24%	lat
l1-3d-multidim	f16	1.03×	0.0047	torch-compile torch flaggems	0.00483 0.0195 0.0246	0.892	20%	lat
l1-mlp-intermediate	f32	0.99×	0.0252	torch-compile flaggems torch	0.0249 0.0381 0.0384	1.79	80%	mem
l1-hidden-state-fp32	bf16/f32	1.24×	0.00688	torch-compile torch	0.00854 0.0495	2.44	54%	mem
l2-hidden-state	f16	1.14×	0.00694	flaggems torch-compile torch	0.0079 0.0088 0.0505	2.42	54%	mem
	bf16	1.13×	0.00694	flaggems torch-compile torch	0.00787 0.0089 0.0510	2.42	54%	mem
l2-long-seq	bf16	1.20×	0.00397	torch-compile flaggems torch	0.00477 0.0143 0.0172	1.06	23%	lat
l2-3d-multidim	f16	0.99×	0.0048	torch-compile flaggems torch	0.00474 0.0136 0.0195	0.874	19%	lat
l2-mlp-intermediate	f32	0.98×	0.0252	torch-compile flaggems torch	0.0246 0.0307 0.0385	1.79	80%	mem
l2-hidden-state-fp32	bf16/f32	1.28×	0.00698	torch-compile torch	0.00893 0.0497	2.41	53%	mem
inf-hidden-state	f16	1.11×	0.00688	flaggems torch-compile torch	0.00765 0.0107 0.0507	2.44	54%	mem
	bf16	1.14×	0.00691	flaggems torch-compile torch	0.0079 0.0111 0.0511	2.43	54%	mem
inf-long-seq	bf16	1.28×	0.00413	torch-compile flaggems torch	0.00528 0.0144 0.0176	1.02	23%	lat
inf-3d-multidim	f16	1.13×	0.0047	torch-compile flaggems torch	0.0053 0.0128 0.0196	0.892	20%	lat
inf-mlp-intermediate	f32	1.05×	0.0252	torch-compile flaggems torch	0.0265 0.0305 0.0397	1.79	80%	mem
inf-hidden-state-fp32	bf16/f32	1.61×	0.00691	torch-compile torch	0.0111 0.0499	2.43	54%	mem

CumsumFwd

x: [*S]

• hidden-statex: [2048, 4096]dtype=f16, bf16
• long-seqx: [64, 32768]dtype=bf16
• llama-13b-hiddenx: [4096, 5120]dtype=f32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state	f16	1.04×	0.0100	flaggems torch-compile torch	0.0104 0.0245 0.1476	0.838	74%	mem
	bf16	1.03×	0.0100	flaggems torch-compile torch	0.0104 0.0246 0.1480	0.835	74%	mem
long-seq	bf16	1.27×	0.0063	flaggems torch-compile torch	0.008 0.0122 0.0679	0.333	30%	lat
llama-13b-hidden	f32	1.05×	0.0427	flaggems torch-compile torch	0.0446 0.0657 0.1608	0.491	87%	mem

CumprodFwd

x: [*S]

• hidden-statex: [2048, 4096]dtype=f16, bf16
• long-seqx: [64, 32768]dtype=bf16
• llama-13b-hiddenx: [4096, 5120]dtype=f32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state	f16	2.45×	0.00999	torch-compile torch	0.0244 0.1476	0.84	75%	mem
	bf16	2.45×	0.0100	torch-compile torch	0.0246 0.1480	0.835	74%	mem
long-seq	bf16	1.94×	0.0063	torch-compile torch	0.0122 0.0679	0.333	30%	lat
llama-13b-hidden	f32	1.54×	0.0427	torch-compile torch	0.0658 0.1607	0.491	87%	mem

AllFwd

x: [*S]

• mask-validation-4kx: [32, 4096]dim=-1dtype=bool, f16, bf16, f32
• mask-validation-32kx: [32, 32768]dim=-1dtype=bool, i32, i64
• 3d-multidimx: [4, 128, 4096]dim=[0,2]dtype=bool, c64, c128
• mlp-intermediatex: [2048, 11008]dim=-1dtype=bool

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
mask-validation-4k	bool	1.08×	0.0017	torch-compile flaggems torch	0.00182 0.00189 0.0180	0.0773	2%	lat
	f16	1.04×	0.00179	torch-compile flaggems torch	0.00186 0.00563 0.0159	0.146	3%	lat
	bf16	1.04×	0.00179	torch-compile flaggems torch	0.00186 0.00562 0.0160	0.146	3%	lat
	f32	1.02×	0.00202	torch-compile flaggems torch	0.00205 0.00579 0.0159	0.13	6%	lat
mask-validation-32k	bool	1.40×	0.0024	torch-compile flaggems torch	0.00336 0.00669 0.0105	0.437	10%	lat
	i32	1.29×	0.00362	torch-compile flaggems torch	0.00467 0.00982 0.0127	0.58	26%	lat
	i64	1.28×	0.00515	torch-compile flaggems torch	0.00659 0.0106 0.0135	0.407	36%	lat
3d-multidim	bool	1.55×	0.00275	torch-compile torch flaggems	0.00426 0.0105 0.0118	0.762	17%	lat
	c64	2.02×	0.00704	torch-compile torch flaggems	0.0142 0.0194 0.0214	0.596	53%	mem
	c128	1.62×	0.0111	torch-compile torch flaggems	0.0180 0.0233 0.0251	0.377	67%	mem
mlp-intermediate	bool	1.30×	0.00822	flaggems torch-compile torch	0.0107 0.0151 0.0281	2.74	61%	mem

AnyFwd

x: [*S]

• mask-validation-4kx: [32, 4096]dim=-1dtype=bool, f16, bf16, f32
• mask-validation-32kx: [32, 32768]dim=-1dtype=bool, i32, i64
• 3d-multidimx: [4, 128, 4096]dim=[0,2]dtype=bool, c64, c128
• mlp-intermediatex: [2048, 11008]dim=-1dtype=bool

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
mask-validation-4k	bool	1.04×	0.0017	torch-compile flaggems torch	0.00176 0.00192 0.0181	0.0773	2%	lat
	f16	1.02×	0.00179	torch-compile flaggems torch	0.00182 0.00563 0.0164	0.146	3%	lat
	bf16	1.03×	0.00178	torch-compile flaggems torch	0.00182 0.00566 0.0165	0.148	3%	lat
	f32	1.02×	0.00198	torch-compile flaggems torch	0.00202 0.00579 0.0164	0.132	6%	lat
mask-validation-32k	bool	1.41×	0.00237	torch-compile flaggems torch	0.00333 0.00672 0.0108	0.443	10%	lat
	i32	1.26×	0.00358	torch-compile flaggems torch	0.00453 0.00986 0.0133	0.585	26%	lat
	i64	1.18×	0.00502	torch-compile flaggems torch	0.00595 0.0106 0.0140	0.417	37%	lat
3d-multidim	bool	1.42×	0.00272	torch-compile flaggems torch	0.00387 0.0119 0.0192	0.771	17%	lat
	c64	1.91×	0.00704	torch-compile flaggems torch	0.0134 0.0214 0.0241	0.596	53%	mem
	c128	1.55×	0.0111	torch-compile flaggems torch	0.0173 0.0251 0.0279	0.377	67%	mem
mlp-intermediate	bool	1.32×	0.00826	flaggems torch-compile torch	0.0109 0.0118 0.0300	2.73	61%	mem

CountNonzeroFwd

x: [*S]

• sparsity-hiddenx: [2048, 4096]dim=-1dtype=f16, bf16, f32
• sparsity-seqx: [32, 32768]dim=-1dtype=f16, bool, i32, i64
• 3d-multidimx: [4, 128, 4096]dim=[0,2]dtype=f16, c64, c128
• sparsity-mlpx: [2048, 11008]dim=-1dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
sparsity-hidden	f16	1.33×	0.00688	torch-compile torch	0.00918 0.0651	2.44	54%	mem
	bf16	1.33×	0.00688	torch-compile torch	0.00918 0.0651	2.44	54%	mem
	f32	1.06×	0.0108	torch-compile torch	0.0114 0.0697	1.56	69%	mem
sparsity-seq	f16	1.24×	0.00304	torch-compile torch	0.00378 0.0152	0.69	15%	lat
	bool	1.42×	0.00243	torch-compile torch	0.00346 0.0134	0.862	10%	lat
	i32	1.27×	0.00368	torch-compile torch	0.00467 0.0161	0.57	25%	lat
	i64	1.25×	0.00522	torch-compile torch	0.0065 0.0175	0.402	36%	lat
3d-multidim	f16	1.29×	0.00371	torch-compile torch	0.00478 0.0219	1.13	25%	lat
	c64	1.58×	0.00701	torch-compile torch	0.0111 0.0256	0.599	53%	mem
	c128	1.91×	0.0112	torch-compile torch	0.0213 0.0358	0.376	67%	mem
sparsity-mlp	f16	1.21×	0.0141	torch-compile torch	0.0172 0.1584	3.19	71%	mem