Elementwise

70 ops, 527 workloads.

One table per op, one row per workload. Ratio is the fastest other implementation's device time divided by ours, so green is faster than it, plain is level with it, red is slower. Times are in ms. How these numbers are taken.

AddFwd

input: [*A]other: [*B]

dtype=f16, bf16, f32

• hidden-state-prefillinput, other: [2048, 4096]
• cnn-feat-broadcastinput: [16, 256, 56, 56]other: [256, 1, 1]
• llama-7b-ffninput, other: [1024, 11008]
• linear-biasinput: [1024, 4096]other: [1, 4096]

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state-prefill	f16	1.00×	0.0147	torch-compile torch	0.0148 0.0149	0.57	76%	mem
	bf16	1.00×	0.0147	torch-compile torch	0.0148 0.0149	0.57	76%	mem
	f32	1.00×	0.0263	torch-compile torch	0.0264 0.0265	0.319	85%	mem
cnn-feat-broadcast	f16	1.00×	0.0146	torch-compile torch	0.0146 0.0462	0.88	78%	mem
	bf16	1.00×	0.0146	torch-compile torch	0.0146 0.0467	0.882	78%	mem
	f32	1.00×	0.0265	torch-compile torch	0.0265 0.0500	0.485	86%	mem
llama-7b-ffn	f16	1.00×	0.0191	torch-compile torch	0.0191 0.0192	0.591	79%	mem
	bf16	1.00×	0.0191	torch-compile torch	0.0191 0.0193	0.59	79%	mem
	f32	1.00×	0.0341	torch-compile torch	0.0340 0.0341	0.33	88%	mem
linear-bias	f16	1.01×	0.00592	torch-compile torch	0.00598 0.0151	0.709	63%	mem
	bf16	1.02×	0.00589	torch-compile torch	0.00598 0.0153	0.712	63%	mem
	f32	1.11×	0.0102	torch-compile torch	0.0113 0.0166	0.411	73%	mem

SubFwd

input: [*A]other: [*B]

• hidden-state-prefillinput, other: [2048, 4096]dtype=f16, bf16, f32
• cnn-feat-broadcastinput: [16, 256, 56, 56]other: [256, 1, 1]dtype=f16, bf16, f32
• llama-7b-ffninput, other: [1024, 11008]dtype=f16, bf16, f32
• linear-biasinput: [1024, 4096]other: [1, 4096]dtype=f16, bf16, f32
• llama-7b-hiddeninput, other: [1024, 4096]dtype=f16
• phi-2-ffninput, other: [1024, 10240]dtype=f16
• phi-2-ffn-biasinput: [1024, 10240]other: [1, 10240]dtype=f16
• llama-7b-ffn-biasinput: [1024, 11008]other: [1, 11008]dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state-prefill	f16	1.00×	0.0147	torch-compile torch	0.0148 0.0149	0.57	76%	mem
	bf16	1.00×	0.0147	torch-compile torch	0.0147 0.0149	0.57	76%	mem
	f32	1.00×	0.0263	torch-compile torch	0.0263 0.0264	0.319	85%	mem
cnn-feat-broadcast	f16	1.00×	0.0146	torch-compile torch	0.0146 0.0463	0.881	78%	mem
	bf16	1.00×	0.0146	torch-compile torch	0.0146 0.0468	0.882	78%	mem
	f32	1.00×	0.0264	torch-compile torch	0.0265 0.0499	0.486	86%	mem
llama-7b-ffn	f16	1.00×	0.0191	torch-compile torch	0.0191 0.0192	0.591	79%	mem
	bf16	1.00×	0.0191	torch-compile torch	0.0191 0.0192	0.591	79%	mem
	f32	1.00×	0.0341	torch-compile torch	0.0341 0.0341	0.331	88%	mem
linear-bias	f16	1.02×	0.00589	torch-compile torch	0.00598 0.0151	0.712	63%	mem
	bf16	1.02×	0.00589	torch-compile torch	0.00602 0.0153	0.712	63%	mem
	f32	1.01×	0.0102	torch-compile torch	0.0103 0.0166	0.411	73%	mem
llama-7b-hidden	f16	1.00×	0.00853	torch torch-compile	0.00854 0.00858	0.492	66%	mem
phi-2-ffn	f16	1.00×	0.0179	torch-compile torch	0.0179 0.0181	0.587	78%	mem
phi-2-ffn-bias	f16	1.01×	0.0123	torch-compile torch	0.0124 0.0336	0.853	76%	mem
llama-7b-ffn-bias	f16	1.00×	0.0132	torch-compile torch	0.0132 0.0359	0.857	76%	mem

MulFwd

input: [*A]other: [*B]

• hidden-state-prefillinput, other: [2048, 4096]dtype=f16, bf16, f32
• cnn-feat-broadcastinput: [16, 256, 56, 56]other: [256, 1, 1]dtype=f16, bf16, f32
• llama-7b-ffninput, other: [1024, 11008]dtype=f16, bf16, f32
• linear-biasinput: [1024, 4096]other: [1, 4096]dtype=f16, bf16, f32
• llama-7b-hiddeninput, other: [1024, 4096]dtype=f16
• phi-2-ffninput, other: [1024, 10240]dtype=f16
• phi-2-ffn-biasinput: [1024, 10240]other: [1, 10240]dtype=f16
• llama-7b-ffn-biasinput: [1024, 11008]other: [1, 11008]dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state-prefill	f16	1.00×	0.0148	torch-compile torch	0.0148 0.0149	0.569	76%	mem
	bf16	1.00×	0.0147	torch-compile torch	0.0148 0.0149	0.57	76%	mem
	f32	1.00×	0.0264	torch torch-compile	0.0264 0.0264	0.318	85%	mem
cnn-feat-broadcast	f16	1.00×	0.0146	torch-compile torch	0.0146 0.0459	0.882	78%	mem
	bf16	1.00×	0.0146	torch-compile torch	0.0146 0.0463	0.882	78%	mem
	f32	1.00×	0.0264	torch-compile torch	0.0264 0.0502	0.487	86%	mem
llama-7b-ffn	f16	1.00×	0.0191	torch-compile torch	0.0191 0.0193	0.59	79%	mem
	bf16	1.00×	0.0191	torch-compile torch	0.0191 0.0192	0.591	79%	mem
	f32	1.00×	0.0340	torch torch-compile	0.0341 0.0341	0.331	88%	mem
linear-bias	f16	1.02×	0.00589	torch-compile torch	0.00598 0.0151	0.712	63%	mem
	bf16	1.02×	0.00589	torch-compile torch	0.00598 0.0152	0.712	63%	mem
	f32	1.01×	0.0102	torch-compile torch	0.0103 0.0166	0.41	73%	mem
llama-7b-hidden	f16	1.00×	0.00851	torch torch-compile	0.00854 0.00858	0.493	66%	mem
phi-2-ffn	f16	1.00×	0.0179	torch-compile torch	0.0178 0.0181	0.587	78%	mem
phi-2-ffn-bias	f16	1.01×	0.0123	torch-compile torch	0.0124 0.0336	0.853	76%	mem
llama-7b-ffn-bias	f16	1.00×	0.0132	torch-compile torch	0.0132 0.0357	0.857	76%	mem

DivFwd

input: [*A]other: [*B]

• hidden-state-prefillinput, other: [2048, 4096]dtype=f16, bf16, f32
• cnn-feat-broadcastinput: [16, 256, 56, 56]other: [256, 1, 1]dtype=f16, bf16, f32
• llama-7b-ffninput, other: [1024, 11008]dtype=f16, bf16, f32
• linear-biasinput: [1024, 4096]other: [1, 4096]dtype=f16, bf16, f32
• llama-7b-hiddeninput, other: [1024, 4096]dtype=f16
• phi-2-ffninput, other: [1024, 10240]dtype=f16
• phi-2-ffn-biasinput: [1024, 10240]other: [1, 10240]dtype=f16
• llama-7b-ffn-biasinput: [1024, 11008]other: [1, 11008]dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state-prefill	f16	1.00×	0.0150	torch-compile torch	0.0150 0.0156	0.56	75%	mem
	bf16	1.00×	0.0150	torch-compile torch	0.0150 0.0156	0.558	74%	mem
	f32	1.00×	0.0265	torch-compile torch	0.0264 0.0266	0.317	84%	mem
cnn-feat-broadcast	f16	1.00×	0.0147	torch-compile torch	0.0147 0.0512	0.876	78%	mem
	bf16	1.00×	0.0148	torch-compile torch	0.0147 0.0517	0.871	77%	mem
	f32	1.00×	0.0266	torch-compile torch	0.0266 0.0534	0.483	86%	mem
llama-7b-ffn	f16	1.00×	0.0187	torch-compile torch	0.0187 0.0192	0.603	80%	mem
	bf16	1.00×	0.0187	torch-compile torch	0.0188 0.0193	0.602	80%	mem
	f32	1.00×	0.0346	torch-compile torch	0.0345 0.0347	0.326	87%	mem
linear-bias	f16	1.00×	0.00621	torch-compile torch	0.00621 0.0165	0.676	60%	mem
	bf16	0.98×	0.00627	torch-compile torch	0.00618 0.0166	0.669	59%	mem
	f32	1.00×	0.0103	torch-compile torch	0.0102 0.0175	0.408	73%	mem
llama-7b-hidden	f16	1.01×	0.00874	torch-compile torch	0.0088 0.00902	0.48	64%	mem
phi-2-ffn	f16	1.00×	0.0177	torch-compile torch	0.0178 0.0183	0.591	79%	mem
phi-2-ffn-bias	f16	1.00×	0.0125	torch-compile torch	0.0125 0.0366	0.84	75%	mem
llama-7b-ffn-bias	f16	1.01×	0.0132	torch-compile torch	0.0133 0.0391	0.852	76%	mem

FloorDivideFwd

input: [*A]other: [*B]

• hidden-state-prefillinput, other: [2048, 4096]dtype=f16, bf16, f32
• cnn-feat-broadcastinput: [16, 256, 56, 56]other: [256, 1, 1]dtype=f16, bf16, f32
• llama-7b-ffninput, other: [1024, 11008]dtype=f16, bf16, f32
• linear-biasinput: [1024, 4096]other: [1, 4096]dtype=f16, bf16, f32
• llama-7b-hiddeninput, other: [1024, 4096]dtype=f16
• phi-2-ffninput, other: [1024, 10240]dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state-prefill	f16	1.01×	0.0150	torch-compile torch	0.0152 0.0485	1.12	75%	mem
	bf16	1.01×	0.0150	torch-compile torch	0.0152 0.0504	1.12	75%	mem
	f32	1.00×	0.0267	torch-compile torch	0.0266 0.0475	0.629	84%	mem
cnn-feat-broadcast	f16	1.06×	0.0149	torch-compile torch	0.0158 0.1115	1.72	77%	mem
	bf16	1.06×	0.0150	torch-compile torch	0.0160 0.1140	1.71	76%	mem
	f32	0.98×	0.0271	torch-compile torch	0.0266 0.1016	0.948	84%	mem
llama-7b-ffn	f16	1.01×	0.0187	torch-compile torch	0.0188 0.0637	1.21	80%	mem
	bf16	1.00×	0.0187	torch-compile torch	0.0188 0.0663	1.21	80%	mem
	f32	1.00×	0.0348	torch-compile torch	0.0346 0.0620	0.648	86%	mem
linear-bias	f16	1.07×	0.00613	torch-compile torch	0.00653 0.0364	1.37	61%	mem
	bf16	1.07×	0.00618	torch-compile torch	0.00659 0.0372	1.36	60%	mem
	f32	0.99×	0.0104	torch-compile torch	0.0103 0.0329	0.809	72%	mem
llama-7b-hidden	f16	1.01×	0.0087	torch-compile torch	0.00883 0.0264	0.964	64%	mem
phi-2-ffn	f16	1.01×	0.0177	torch-compile torch	0.0179 0.0595	1.19	79%	mem

RemainderFwd

input: [*A]other: [*B]

• hidden-state-prefillinput, other: [2048, 4096]dtype=f16, bf16, f32
• cnn-feat-broadcastinput: [16, 256, 56, 56]other: [256, 1, 1]dtype=f16, bf16, f32
• llama-7b-ffninput, other: [1024, 11008]dtype=f16, bf16, f32
• linear-biasinput: [1024, 4096]other: [1, 4096]dtype=f16, bf16, f32
• llama-7b-hiddeninput, other: [1024, 4096]dtype=f16
• phi-2-ffninput, other: [1024, 10240]dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state-prefill	f16	1.01×	0.0150	torch-compile torch	0.0153 0.0180	2.23	74%	mem
	bf16	1.02×	0.0151	torch-compile torch	0.0154 0.0185	2.23	74%	mem
	f32	1.00×	0.0268	torch-compile torch	0.0269 0.0272	1.25	83%	mem
cnn-feat-broadcast	f16	1.15×	0.0156	torch-compile torch	0.0180 0.0623	3.3	73%	mem
	bf16	1.18×	0.0158	torch-compile torch	0.0187 0.0645	3.24	72%	mem
	f32	0.97×	0.0275	torch-compile torch	0.0267 0.0644	1.87	83%	mem
llama-7b-ffn	f16	1.00×	0.0188	torch-compile torch	0.0188 0.0231	2.4	80%	mem
	bf16	1.01×	0.0188	torch-compile torch	0.0189 0.0238	2.4	80%	mem
	f32	1.00×	0.0349	torch-compile torch	0.0349 0.0355	1.29	86%	mem
linear-bias	f16	1.18×	0.0063	torch-compile torch	0.00746 0.0202	2.66	59%	mem
	bf16	1.20×	0.0064	torch-compile torch	0.00768 0.0208	2.62	58%	mem
	f32	1.18×	0.0106	torch-compile torch	0.0125 0.0206	1.59	71%	mem
llama-7b-hidden	f16	1.03×	0.0088	torch-compile torch	0.00902 0.0106	1.91	64%	mem
phi-2-ffn	f16	1.01×	0.0178	torch-compile torch	0.0180 0.0217	2.36	79%	mem

PowFwd

input: [*A]exponent: [*B]

• hidden-state-prefillinput, exponent: [2048, 4096]dtype=f16, bf16, f32
• cnn-feat-broadcastinput: [16, 256, 56, 56]exponent: [256, 1, 1]dtype=f16, bf16, f32
• llama-7b-ffninput, exponent: [1024, 11008]dtype=f16, bf16, f32
• linear-biasinput: [1024, 4096]exponent: [1, 4096]dtype=f16, bf16, f32
• llama-7b-hiddeninput, exponent: [1024, 4096]dtype=f16
• phi-2-ffninput, exponent: [1024, 10240]dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state-prefill	f16	1.59×	0.0232	torch torch-compile	0.0370 0.0437	0.362	48%	mem
	bf16	1.59×	0.0237	torch torch-compile	0.0376 0.0449	0.354	47%	mem
	f32	1.33×	0.0279	torch torch-compile	0.0372 0.0420	0.3	80%	mem
cnn-feat-broadcast	f16	1.30×	0.0470	torch-compile torch	0.0613 0.0943	0.273	24%	mem
	bf16	1.30×	0.0474	torch-compile torch	0.0614 0.0954	0.271	24%	mem
	f32	1.19×	0.0493	torch-compile torch	0.0588 0.0939	0.26	46%	mem
llama-7b-ffn	f16	1.63×	0.0298	torch torch-compile	0.0485 0.0579	0.378	50%	mem
	bf16	1.62×	0.0307	torch torch-compile	0.0496 0.0594	0.368	49%	mem
	f32	1.34×	0.0365	torch torch-compile	0.0487 0.0550	0.309	82%	mem
linear-bias	f16	1.33×	0.0177	torch-compile torch	0.0234 0.0304	0.237	21%	mem
	bf16	1.34×	0.0178	torch-compile torch	0.0238 0.0308	0.236	21%	mem
	f32	1.27×	0.0179	torch-compile torch	0.0227 0.0297	0.234	42%	mem
llama-7b-hidden	f16	1.54×	0.0131	torch torch-compile	0.0202 0.0253	0.32	43%	mem
phi-2-ffn	f16	1.61×	0.0283	torch torch-compile	0.0456 0.0541	0.371	49%	mem

MaximumFwd

input: [*A]other: [*B]

• hidden-state-prefillinput, other: [2048, 4096]dtype=f16, bf16, f32
• cnn-feat-broadcastinput: [16, 256, 56, 56]other: [256, 1, 1]dtype=f16, bf16, f32
• llama-7b-ffninput, other: [1024, 11008]dtype=f16, bf16, f32
• linear-biasinput: [1024, 4096]other: [1, 4096]dtype=f16, bf16, f32
• llama-7b-hiddeninput, other: [1024, 4096]dtype=f16
• phi-2-ffninput, other: [1024, 10240]dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state-prefill	f16	1.00×	0.0146	torch-compile torch	0.0146 0.0150	0.575	77%	mem
	bf16	1.00×	0.0146	torch-compile torch	0.0146 0.0150	0.575	77%	mem
	f32	1.00×	0.0262	torch-compile torch	0.0262 0.0265	0.32	85%	mem
cnn-feat-broadcast	f16	1.01×	0.0144	torch-compile torch	0.0145 0.0508	0.892	79%	mem
	bf16	1.00×	0.0144	torch-compile torch	0.0145 0.0517	0.89	79%	mem
	f32	1.00×	0.0263	torch-compile torch	0.0262 0.0537	0.489	87%	mem
llama-7b-ffn	f16	0.99×	0.0190	torch-compile torch	0.0189 0.0194	0.595	79%	mem
	bf16	1.00×	0.0189	torch-compile torch	0.0189 0.0194	0.595	79%	mem
	f32	1.00×	0.0340	torch-compile torch	0.0340 0.0342	0.332	88%	mem
linear-bias	f16	1.03×	0.00563	torch-compile torch	0.00579 0.0176	0.745	66%	mem
	bf16	1.02×	0.00563	torch-compile torch	0.00576 0.0178	0.745	66%	mem
	f32	1.00×	0.0101	torch-compile torch	0.0101 0.0183	0.417	74%	mem
llama-7b-hidden	f16	0.98×	0.00835	torch-compile torch	0.00822 0.00886	0.502	67%	mem
phi-2-ffn	f16	1.00×	0.0177	torch-compile torch	0.0177 0.0181	0.594	79%	mem

MinimumFwd

input: [*A]other: [*B]

• hidden-state-prefillinput, other: [2048, 4096]dtype=f16, bf16, f32
• cnn-feat-broadcastinput: [16, 256, 56, 56]other: [256, 1, 1]dtype=f16, bf16, f32
• llama-7b-ffninput, other: [1024, 11008]dtype=f16, bf16, f32
• linear-biasinput: [1024, 4096]other: [1, 4096]dtype=f16, bf16, f32
• llama-7b-hiddeninput, other: [1024, 4096]dtype=f16
• phi-2-ffninput, other: [1024, 10240]dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state-prefill	f16	1.00×	0.0145	torch-compile torch	0.0145 0.0150	0.577	77%	mem
	bf16	1.00×	0.0145	torch-compile torch	0.0145 0.0150	0.577	77%	mem
	f32	1.00×	0.0262	torch-compile torch	0.0262 0.0265	0.32	85%	mem
cnn-feat-broadcast	f16	1.00×	0.0144	torch-compile torch	0.0144 0.0511	0.89	79%	mem
	bf16	1.01×	0.0144	torch-compile torch	0.0145 0.0516	0.892	79%	mem
	f32	1.00×	0.0263	torch-compile torch	0.0263 0.0538	0.488	87%	mem
llama-7b-ffn	f16	1.00×	0.0189	torch-compile torch	0.0190 0.0194	0.596	79%	mem
	bf16	1.00×	0.0189	torch-compile torch	0.0190 0.0194	0.596	79%	mem
	f32	1.00×	0.0340	torch-compile torch	0.0340 0.0342	0.331	88%	mem
linear-bias	f16	1.01×	0.0057	torch-compile torch	0.00576 0.0176	0.736	65%	mem
	bf16	1.01×	0.00574	torch-compile torch	0.00579 0.0178	0.73	65%	mem
	f32	1.00×	0.0102	torch-compile torch	0.0101 0.0183	0.412	73%	mem
llama-7b-hidden	f16	0.96×	0.00842	torch-compile torch	0.00811 0.00883	0.498	66%	mem
phi-2-ffn	f16	1.00×	0.0177	torch-compile torch	0.0177 0.0182	0.593	79%	mem

LerpScalarFwd

input: [*A]end: [*B]

• hidden-state-prefillinput, end: [2048, 4096]dtype=f16, bf16, f32
• cnn-feat-broadcastinput: [16, 256, 56, 56]end: [256, 1, 1]dtype=f16, bf16, f32
• llama-7b-ffninput, end: [1024, 11008]dtype=f16, bf16, f32
• linear-biasinput: [1024, 4096]end: [1, 4096]dtype=f16, bf16, f32
• llama-7b-hiddeninput, end: [1024, 4096]dtype=f16
• phi-2-ffninput, end: [1024, 10240]dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state-prefill	f16	1.00×	0.0147	torch-compile torch	0.0146 0.0148	1.72	76%	mem
	bf16	1.00×	0.0146	torch-compile torch	0.0146 0.0147	1.72	77%	mem
	f32	1.00×	0.0263	torch-compile torch	0.0263 0.0264	0.957	85%	mem
cnn-feat-broadcast	f16	1.00×	0.0145	torch-compile torch	0.0145 0.0480	2.66	79%	mem
	bf16	1.00×	0.0145	torch-compile torch	0.0145 0.0487	2.66	79%	mem
	f32	1.00×	0.0263	torch-compile torch	0.0263 0.0515	1.46	87%	mem
llama-7b-ffn	f16	1.00×	0.0190	torch-compile torch	0.0190 0.0192	1.78	79%	mem
	bf16	1.00×	0.0190	torch-compile torch	0.0189 0.0192	1.78	79%	mem
	f32	1.00×	0.0340	torch-compile torch	0.0340 0.0340	0.993	88%	mem
linear-bias	f16	1.00×	0.00587	torch-compile torch	0.00586 0.0156	2.14	63%	mem
	bf16	1.01×	0.00582	torch-compile torch	0.00589 0.0157	2.16	64%	mem
	f32	1.01×	0.0100	torch-compile torch	0.0101 0.0169	1.26	74%	mem
llama-7b-hidden	f16	1.01×	0.00829	torch-compile torch	0.00837 0.00854	1.52	67%	mem
phi-2-ffn	f16	1.00×	0.0177	torch-compile torch	0.0177 0.0179	1.78	79%	mem

LerpTensorFwd

input: [*A]end: [*E]weight: [*W]

• 16Minput, end, weight: [4096, 4096]dtype=f16, bf16, f32
• 256Minput, end, weight: [16384, 16384]dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	f16	1.00×	0.0344	torch-compile torch	0.0343 0.0345	1.46	87%	mem
	bf16	1.00×	0.0345	torch-compile torch	0.0344 0.0345	1.46	86%	mem
	f32	1.00×	0.0650	torch torch-compile	0.0648 0.0652	0.775	92%	mem
256M	f16	1.00×	0.4830	torch-compile torch	0.4829 0.4829	1.67	99%	mem
	bf16	1.00×	0.4829	torch-compile torch	0.4829 0.4830	1.67	99%	mem

ExpFwd

input: [*S]

• 16Minput: [4096, 4096]dtype=f16, bf16, f32
• 256Minput: [16384, 16384]dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	f16	1.00×	0.0179	torch-compile torch	0.0180 0.0181	0.938	83%	mem
	bf16	1.01×	0.0180	torch torch-compile	0.0181 0.0182	0.935	83%	mem
	f32	1.01×	0.0332	torch torch-compile	0.0336 0.0336	0.505	90%	mem
256M	f16	1.01×	0.2567	torch-compile torch	0.2586 0.2599	1.05	93%	mem
	bf16	1.01×	0.2591	torch torch-compile	0.2622 0.2658	1.04	92%	mem

Expm1Fwd

input: [*S]

• 16Minput: [4096, 4096]dtype=f16, bf16, f32
• 256Minput: [16384, 16384]dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	f16	1.42×	0.0179	torch torch-compile	0.0255 0.0270	0.938	83%	mem
	bf16	1.56×	0.0180	torch-compile torch	0.0280 0.0282	0.935	83%	mem
	f32	1.02×	0.0332	torch torch-compile	0.0338 0.0339	0.505	90%	mem
256M	f16	1.45×	0.2568	torch torch-compile	0.3717 0.3955	1.05	93%	mem
	bf16	1.59×	0.2595	torch-compile torch	0.4120 0.4151	1.03	92%	mem

LogFwd

input: [*S]

• 16Minput: [4096, 4096]dtype=f16, bf16, f32
• 256Minput: [16384, 16384]dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	f16	1.54×	0.0178	torch torch-compile	0.0273 0.0274	0.945	84%	mem
	bf16	1.57×	0.0178	torch-compile torch	0.0279 0.0282	0.943	84%	mem
	f32	1.00×	0.0339	torch-compile torch	0.0339 0.0340	0.496	88%	mem
256M	f16	1.57×	0.2549	torch torch-compile	0.4011 0.4029	1.05	94%	mem
	bf16	1.62×	0.2549	torch-compile torch	0.4119 0.4158	1.05	94%	mem

Log1pFwd

input: [*S]

• 16Minput: [4096, 4096]dtype=f16, bf16, f32
• 256Minput: [16384, 16384]dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	f16	1.43×	0.0178	torch-compile torch	0.0255 0.0263	0.943	84%	mem
	bf16	1.47×	0.0178	torch-compile torch	0.0262 0.0269	0.943	84%	mem
	f32	1.00×	0.0337	torch-compile torch	0.0337 0.0339	0.497	88%	mem
256M	f16	1.46×	0.2549	torch-compile torch	0.3711 0.3840	1.05	94%	mem
	bf16	1.51×	0.2550	torch-compile torch	0.3838 0.3911	1.05	94%	mem

SqrtFwd

input: [*S]

• 16Minput: [4096, 4096]dtype=f16, bf16, f32
• 256Minput: [16384, 16384]dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	f16	1.01×	0.0181	torch-compile torch	0.0184 0.0187	0.925	82%	mem
	bf16	1.02×	0.0181	torch-compile torch	0.0185 0.0188	0.926	82%	mem
	f32	1.00×	0.0336	torch-compile torch	0.0337 0.0338	0.499	89%	mem
256M	f16	1.04×	0.2550	torch-compile torch	0.2652 0.2691	1.05	94%	mem
	bf16	1.04×	0.2549	torch-compile torch	0.2664 0.2702	1.05	94%	mem

RsqrtFwd

input: [*S]

• 16Minput: [4096, 4096]dtype=f16, bf16, f32
• 256Minput: [16384, 16384]dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	f16	1.00×	0.0178	torch-compile torch	0.0178 0.0178	0.945	84%	mem
	bf16	1.00×	0.0177	torch-compile torch	0.0177 0.0178	0.948	84%	mem
	f32	1.01×	0.0332	torch-compile torch	0.0336 0.0336	0.505	90%	mem
256M	f16	1.00×	0.2544	torch-compile torch	0.2542 0.2548	1.06	94%	mem
	bf16	1.00×	0.2543	torch-compile torch	0.2538 0.2551	1.06	94%	mem

ReciprocalFwd

input: [*S]

• 16Minput: [4096, 4096]dtype=f16, bf16, f32
• 256Minput: [16384, 16384]dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	f16	1.00×	0.0178	torch-compile torch	0.0179 0.0188	0.941	84%	mem
	bf16	0.99×	0.0182	torch-compile torch	0.0180 0.0188	0.92	82%	mem
	f32	1.00×	0.0337	torch-compile torch	0.0336 0.0339	0.498	88%	mem
256M	f16	1.01×	0.2549	torch-compile torch	0.2572 0.2702	1.05	94%	mem
	bf16	1.00×	0.2584	torch-compile torch	0.2588 0.2706	1.04	92%	mem

AbsFwd

input: [*S]

• 16Minput: [4096, 4096]dtype=f16, bf16, f32
• 256Minput: [16384, 16384]dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	f16	1.00×	0.0178	torch-compile torch	0.0178 0.0179	0.945	84%	mem
	bf16	1.00×	0.0178	torch-compile torch	0.0178 0.0179	0.945	84%	mem
	f32	1.01×	0.0332	torch-compile torch	0.0335 0.0335	0.506	90%	mem
256M	f16	1.00×	0.2525	torch-compile torch	0.2523 0.2526	1.06	94%	mem
	bf16	1.00×	0.2523	torch-compile torch	0.2521 0.2529	1.06	95%	mem

NegFwd

input: [*S]

• 16Minput: [4096, 4096]dtype=f16, bf16, f32
• 256Minput: [16384, 16384]dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	f16	1.01×	0.0178	torch-compile torch	0.0180 0.0188	0.943	84%	mem
	bf16	1.00×	0.0178	torch-compile torch	0.0178 0.0179	0.941	84%	mem
	f32	1.01×	0.0332	torch torch-compile	0.0336 0.0336	0.506	90%	mem
256M	f16	1.00×	0.2521	torch-compile torch	0.2521 0.2723	1.06	95%	mem
	bf16	1.00×	0.2525	torch torch-compile	0.2521 0.2524	1.06	94%	mem

SignFwd

input: [*S]

• 16Minput: [4096, 4096]dtype=f16, bf16, f32
• 256Minput: [16384, 16384]dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	f16	1.00×	0.0178	torch-compile torch	0.0179 0.0180	1.88	84%	mem
	bf16	1.00×	0.0180	torch-compile torch	0.0180 0.0181	1.87	83%	mem
	f32	1.01×	0.0332	torch-compile torch	0.0336 0.0337	1.01	90%	mem
256M	f16	1.00×	0.2526	torch-compile torch	0.2534 0.2583	2.13	94%	mem
	bf16	0.99×	0.2580	torch-compile torch	0.2562 0.2614	2.08	92%	mem

SinFwd

input: [*S]

• 16Minput: [4096, 4096]dtype=f16, bf16, f32
• 256Minput: [16384, 16384]dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	f16	1.14×	0.0231	torch torch-compile	0.0264 0.0266	0.727	65%	mem
	bf16	1.16×	0.0232	torch torch-compile	0.0269 0.0271	0.724	64%	mem
	f32	1.01×	0.0338	torch-compile torch	0.0340 0.0340	0.497	88%	mem
256M	f16	1.18×	0.3228	torch torch-compile	0.3819 0.3866	0.832	74%	mem
	bf16	1.20×	0.3265	torch torch-compile	0.3903 0.3929	0.822	73%	mem

CosFwd

input: [*S]

• 16Minput: [4096, 4096]dtype=f16, bf16, f32
• 256Minput: [16384, 16384]dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	f16	1.16×	0.0236	torch torch-compile	0.0274 0.0281	0.712	63%	mem
	bf16	1.15×	0.0239	torch torch-compile	0.0274 0.0285	0.702	62%	mem
	f32	1.00×	0.0338	torch-compile torch	0.0339 0.0340	0.496	88%	mem
256M	f16	1.19×	0.3323	torch torch-compile	0.3969 0.4089	0.808	72%	mem
	bf16	1.17×	0.3389	torch torch-compile	0.3971 0.4141	0.792	70%	mem

ErfFwd

input: [*S]

• 16Minput: [4096, 4096]dtype=f16, bf16, f32
• 256Minput: [16384, 16384]dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	f16	1.19×	0.0224	torch torch-compile	0.0267 0.0294	0.748	66%	mem
	bf16	1.21×	0.0232	torch torch-compile	0.0280 0.0297	0.723	64%	mem
	f32	1.01×	0.0336	torch torch-compile	0.0338 0.0340	0.5	89%	mem
256M	f16	1.28×	0.3059	torch torch-compile	0.3902 0.4336	0.878	78%	mem
	bf16	1.31×	0.3138	torch torch-compile	0.4104 0.4387	0.855	76%	mem

FloorFwd

input: [*S]

• 16Minput: [4096, 4096]dtype=f16, bf16, f32
• 256Minput: [16384, 16384]dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	f16	1.00×	0.0178	torch-compile torch	0.0179 0.0180	0.941	84%	mem
	bf16	1.00×	0.0179	torch-compile torch	0.0179 0.0180	0.94	83%	mem
	f32	1.01×	0.0332	torch-compile torch	0.0335 0.0336	0.506	90%	mem
256M	f16	1.00×	0.2521	torch-compile torch	0.2521 0.2530	1.06	95%	mem
	bf16	1.00×	0.2530	torch-compile torch	0.2525 0.2531	1.06	94%	mem

CeilFwd

input: [*S]

• 16Minput: [4096, 4096]dtype=f16, bf16, f32
• 256Minput: [16384, 16384]dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	f16	1.00×	0.0179	torch-compile torch	0.0179 0.0180	0.94	83%	mem
	bf16	1.00×	0.0179	torch-compile torch	0.0179 0.0180	0.94	83%	mem
	f32	1.01×	0.0332	torch torch-compile	0.0336 0.0336	0.505	90%	mem
256M	f16	1.00×	0.2522	torch-compile torch	0.2525 0.2527	1.06	95%	mem
	bf16	1.00×	0.2525	torch-compile torch	0.2525 0.2527	1.06	94%	mem

RoundFwd

input: [*S]

• 16Minput: [4096, 4096]dtype=f16, bf16, f32
• 256Minput: [16384, 16384]dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	f16	1.00×	0.0178	torch-compile torch	0.0179 0.0180	0.941	84%	mem
	bf16	1.00×	0.0178	torch-compile torch	0.0179 0.0179	0.944	84%	mem
	f32	1.01×	0.0332	torch-compile torch	0.0334 0.0335	0.506	90%	mem
256M	f16	1.00×	0.2527	torch torch-compile	0.2525 0.2528	1.06	94%	mem
	bf16	1.00×	0.2523	torch torch-compile	0.2526 0.2529	1.06	95%	mem

TruncFwd

input: [*S]

• 16Minput: [4096, 4096]dtype=f16, bf16, f32
• 256Minput: [16384, 16384]dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	f16	1.00×	0.0179	torch-compile torch	0.0179 0.0180	0.94	83%	mem
	bf16	1.00×	0.0178	torch-compile torch	0.0179 0.0179	0.941	84%	mem
	f32	1.01×	0.0332	torch-compile torch	0.0334 0.0335	0.506	90%	mem
256M	f16	1.00×	0.2528	torch-compile torch	0.2524 0.2526	1.06	94%	mem
	bf16	1.00×	0.2530	torch-compile torch	0.2526 0.2527	1.06	94%	mem

ReluFwd

input: [*S]

• hidden-state-prefillinput: [2048, 4096]dtype=f16, bf16
• hidden-state-decodeinput: [1, 4096]dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state-prefill	f16	1.00×	0.00976	torch-compile torch	0.00976 0.0105	0.859	76%	mem
	bf16	1.01×	0.00973	torch-compile torch	0.00979 0.00995	0.862	77%	mem
hidden-state-decode	bf16	1.00×	0.00125	torch-compile torch	0.00125 0.00144	0.00328	0%	lat

LeakyReluFwd

input: [*S]

dtype=f16, bf16

• gan-featinput: [16, 256, 64, 64]
• gan-feat-deepinput: [16, 512, 32, 32]

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
gan-feat	f16	1.00×	0.0179	torch-compile torch	0.0179 0.0180	1.88	83%	mem
	bf16	1.00×	0.0179	torch-compile torch	0.0179 0.0180	1.87	83%	mem
gan-feat-deep	f16	1.00×	0.00982	torch-compile torch	0.00979 0.00986	1.71	76%	mem
	bf16	1.00×	0.00981	torch-compile torch	0.00981 0.00986	1.71	76%	mem

EluFwd

input: [*S]

dtype=f16, bf16

• mlp-hiddeninput: [2048, 4096]
• mlp-hidden-wideinput: [2048, 8192]

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
mlp-hidden	f16	1.33×	0.0120	torch-compile torch	0.0159 0.0179	2.8	62%	mem
	bf16	1.42×	0.0118	torch-compile torch	0.0168 0.0181	2.83	63%	mem
mlp-hidden-wide	f16	1.36×	0.0219	torch-compile torch	0.0299 0.0328	3.07	68%	mem
	bf16	1.46×	0.0216	torch-compile torch	0.0314 0.0333	3.11	69%	mem

SeluFwd

input: [*S]

dtype=f16, bf16

• snn-fcinput: [2048, 4096]
• snn-fc-wideinput: [2048, 8192]

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
snn-fc	f16	1.39×	0.0116	torch-compile torch	0.0161 0.0179	3.62	64%	mem
	bf16	1.31×	0.0119	torch-compile torch	0.0157 0.0181	3.51	62%	mem
snn-fc-wide	f16	1.43×	0.0211	torch-compile torch	0.0301 0.0329	3.98	71%	mem
	bf16	1.35×	0.0217	torch-compile torch	0.0293 0.0332	3.87	69%	mem

GeluFwd

input: [*S]

• llama-8b-ffn-prefillinput: [2048, 14336]dtype=f16, bf16
• llama-8b-ffn-decodeinput: [1, 14336]dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
llama-8b-ffn-prefill	f16	1.37×	0.0353	torch torch-compile	0.0482 0.0541	4.16	74%	mem
	bf16	1.38×	0.0357	torch torch-compile	0.0492 0.0561	4.11	73%	mem
llama-8b-ffn-decode	bf16	1.02×	0.00138	torch-compile torch	0.00141 0.00166	0.0521	1%	lat

SiluFwd

input: [*S]

• llama-8b-ffn-prefillinput: [2048, 14336]dtype=f16, bf16
• llama-8b-ffn-decodeinput: [1, 14336]dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
llama-8b-ffn-prefill	f16	1.03×	0.0347	torch-compile torch	0.0356 0.0379	4.24	75%	mem
	bf16	1.01×	0.0354	torch-compile torch	0.0358 0.0381	4.14	74%	mem
llama-8b-ffn-decode	bf16	1.02×	0.00144	torch-compile torch	0.00147 0.00198	0.0498	1%	lat

MishFwd

input: [*S]

dtype=f16, bf16

• yolo-p3input: [16, 256, 80, 80]
• yolo-p4input: [16, 512, 40, 40]

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
yolo-p3	f16	1.79×	0.0357	torch torch-compile	0.0639 0.0734	2.94	65%	mem
	bf16	1.78×	0.0364	torch torch-compile	0.0645 0.0741	2.88	64%	mem
yolo-p4	f16	1.76×	0.0190	torch torch-compile	0.0334 0.0383	2.76	61%	mem
	bf16	1.75×	0.0192	torch torch-compile	0.0337 0.0386	2.73	61%	mem

SigmoidFwd

input: [*S]

• 16Minput: [4096, 4096]dtype=f16, bf16, f32
• 256Minput: [16384, 16384]dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	f16	1.07×	0.0170	torch-compile torch	0.0182 0.0230	3.94	88%	mem
	bf16	1.03×	0.0178	torch-compile torch	0.0183 0.0235	3.78	84%	mem
	f32	1.02×	0.0329	torch-compile torch	0.0335 0.0339	2.04	91%	mem
256M	f16	1.05×	0.2492	torch-compile torch	0.2627 0.3247	4.31	96%	mem
	bf16	1.04×	0.2546	torch-compile torch	0.2658 0.3334	4.22	94%	mem

TanhFwd

input: [*S]

• 16Minput: [4096, 4096]dtype=f16, bf16, f32
• 256Minput: [16384, 16384]dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	f16	1.06×	0.0196	torch torch-compile	0.0206 0.0240	0.858	76%	mem
	bf16	1.01×	0.0216	torch torch-compile	0.0218 0.0245	0.778	69%	mem
	f32	1.00×	0.0335	torch torch-compile	0.0336 0.0340	0.501	89%	mem
256M	f16	1.07×	0.2767	torch torch-compile	0.2961 0.3442	0.97	86%	mem
	bf16	1.05×	0.2996	torch torch-compile	0.3137 0.3513	0.896	80%	mem

HardsigmoidFwd

input: [*S]

dtype=f16, bf16

• mbv3-se-gateinput: [32, 240, 1, 1]
• mbv3-se-gate-deepinput: [32, 960, 1, 1]

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
mbv3-se-gate	f16	1.07×	0.00128	torch-compile torch	0.00138 0.00173	0.018	1%	lat
	bf16	1.05×	0.00131	torch-compile torch	0.00138 0.00173	0.0176	1%	lat
mbv3-se-gate-deep	f16	1.05×	0.00141	torch torch-compile	0.00147 0.0015	0.0655	2%	lat
	bf16	1.05×	0.00141	torch torch-compile	0.00147 0.00147	0.0655	2%	lat

HardswishFwd

input: [*S]

dtype=f16, bf16

• mbv3-stage2input: [32, 96, 56, 56]
• mbv3-stage3input: [32, 240, 28, 28]

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
mbv3-stage2	f16	1.01×	0.0111	torch-compile torch	0.0112 0.0113	3.48	77%	mem
	bf16	1.01×	0.0111	torch-compile torch	0.0112 0.0113	3.47	77%	mem
mbv3-stage3	f16	1.02×	0.00739	torch-compile torch	0.00752 0.00758	3.26	72%	mem
	bf16	1.01×	0.00739	torch-compile torch	0.00749 0.00758	3.26	72%	mem

HardtanhFwd

input: [*S]

dtype=f16, bf16

• bounded-hiddeninput: [2048, 4096]
• bounded-conv-featinput: [16, 256, 56, 56]

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
bounded-hidden	f16	1.01×	0.00982	torch-compile torch	0.00989 0.0110	0.854	76%	mem
	bf16	1.01×	0.00986	torch-compile torch	0.00998 0.0104	0.851	76%	mem
bounded-conv-feat	f16	1.00×	0.0142	torch-compile torch	0.0142 0.0160	0.902	80%	mem
	bf16	1.01×	0.0142	torch-compile torch	0.0144 0.0150	0.902	80%	mem

SoftplusFwd

input: [*S]

dtype=f16, bf16

• mlp-hiddeninput: [2048, 4096]
• mlp-hidden-wideinput: [2048, 8192]

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
mlp-hidden	f16	1.45×	0.0123	torch-compile torch	0.0178 0.0237	3.42	61%	mem
	bf16	1.44×	0.0126	torch-compile torch	0.0182 0.0240	3.33	59%	mem
mlp-hidden-wide	f16	1.49×	0.0222	torch-compile torch	0.0331 0.0448	3.78	67%	mem
	bf16	1.48×	0.0229	torch-compile torch	0.0340 0.0453	3.66	65%	mem

PreluFwd

input: [*S]weight: [*W]

dtype=f16, bf16

• cnn-featinput: [16, 256, 56, 56]weight: [256]
• cnn-feat-deepinput: [16, 512, 28, 28]weight: [512]

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
cnn-feat	f16	1.00×	0.0146	torch-compile torch	0.0146 0.0476	1.76	78%	mem
	bf16	1.00×	0.0146	torch-compile torch	0.0146 0.0489	1.76	78%	mem
cnn-feat-deep	f16	1.00×	0.00819	torch-compile torch	0.00819 0.0252	1.57	70%	mem
	bf16	1.00×	0.00819	torch-compile torch	0.00822 0.0259	1.57	70%	mem

DropoutFwd

input: [*S]

• hidden-4kinput: [1024, 4096]dtype=f16, f32
• hidden-10kinput: [1024, 10240]dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-4k	f16	1.97×	0.00576	torch-compile torch	0.0113 0.0117	1.46	65%	mem
	f32	1.18×	0.0100	torch-compile torch	0.0119 0.0148	0.835	74%	mem
hidden-10k	bf16	1.94×	0.0122	torch-compile torch	0.0236 0.0239	1.72	76%	mem

SiluAndMulFwd

x: [M, 2 * N]

• llama-8b-ffn-prefillx: [2048, 28672]dtype=f16, bf16
• llama-8b-ffn-decodex: [1, 28672]dtype=bf16
• qwen3-8b-swiglu-b32x: [32, 24576]dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
llama-8b-ffn-prefill	f16	1.06×	0.0416	torch-compile flashinfer torch-ref	0.0442 0.0540 0.4232	4.23	94%	mem
	bf16	0.98×	0.0469	torch-compile flashinfer torch-ref	0.0460 0.0543 0.4256	3.76	83%	mem
llama-8b-ffn-decode	bf16	1.00×	0.0015	torch-compile flashinfer torch-ref	0.0015 0.00426 0.0116	0.0572	1%	lat
qwen3-8b-swiglu-b32	bf16	1.01×	0.00237	torch-compile flashinfer torch-ref	0.0024 0.00438 0.0156	0.996	22%	lat

GeluAndMulFwd

x: [M, 2 * N]

• llama-8b-ffn-prefillx: [2048, 28672]dtype=f16, bf16
• llama-8b-ffn-decodex: [1, 28672]dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
llama-8b-ffn-prefill	f16	1.26×	0.0484	torch-compile flashinfer torch-ref	0.0610 0.1056 0.4231	3.64	81%	mem
	bf16	1.27×	0.0484	torch-compile flashinfer torch-ref	0.0617 0.1068 0.4256	3.64	81%	mem
llama-8b-ffn-decode	bf16	0.97×	0.00155	torch-compile flashinfer torch-ref	0.00151 0.00666 0.0115	0.0554	1%	lat

GeluTanhAndMulFwd

x: [M, 2 * N]

• llama-8b-ffn-prefillx: [2048, 28672]dtype=f16, bf16
• llama-8b-ffn-decodex: [1, 28672]dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
llama-8b-ffn-prefill	f16	1.09×	0.0480	torch-compile flashinfer torch-ref	0.0523 0.0569 0.4231	6.11	81%	mem
	bf16	1.07×	0.0496	torch-compile flashinfer torch-ref	0.0529 0.0580 0.4255	5.92	79%	mem
llama-8b-ffn-decode	bf16	1.02×	0.0015	torch-compile flashinfer torch-ref	0.00154 0.00438 0.0114	0.0953	1%	lat

EqFwd

input: [*A]other: [*B]

• hidden-state-prefillinput, other: [2048, 4096]dtype=f16, bf16, f32
• cnn-feat-broadcastinput: [16, 256, 56, 56]other: [256, 1, 1]dtype=f16, bf16, f32
• llama-7b-ffninput, other: [1024, 11008]dtype=f16, bf16, f32
• linear-biasinput: [1024, 4096]other: [1, 4096]dtype=f16, bf16, f32
• llama-7b-hiddeninput, other: [1024, 4096]dtype=f16
• phi-2-ffninput, other: [1024, 10240]dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state-prefill	f16	1.00×	0.0129	torch-compile torch	0.0130 0.0133	0.649	72%	mem
	bf16	1.02×	0.0130	torch-compile torch	0.0132 0.0132	0.647	72%	mem
	f32	1.00×	0.0223	torch-compile torch	0.0223 0.0224	0.377	75%	mem
cnn-feat-broadcast	f16	0.99×	0.0119	torch-compile torch	0.0118 0.0482	1.08	72%	mem
	bf16	1.02×	0.0118	torch-compile torch	0.0120 0.0490	1.09	73%	mem
	f32	1.00×	0.0185	torch-compile torch	0.0185 0.0483	0.694	77%	mem
llama-7b-ffn	f16	1.01×	0.0167	torch-compile torch	0.0168 0.0169	0.676	75%	mem
	bf16	1.00×	0.0167	torch-compile torch	0.0167 0.0169	0.676	75%	mem
	f32	1.00×	0.0286	torch-compile torch	0.0286 0.0287	0.394	79%	mem
linear-bias	f16	0.98×	0.00483	torch-compile torch	0.00472 0.0154	0.868	58%	lat
	bf16	1.00×	0.00461	torch-compile torch	0.00462 0.0157	0.91	61%	lat
	f32	0.94×	0.00768	torch-compile torch	0.00722 0.0158	0.546	61%	mem
llama-7b-hidden	f16	1.03×	0.00733	torch-compile torch	0.00758 0.00765	0.572	64%	mem
phi-2-ffn	f16	1.01×	0.0156	torch torch-compile	0.0158 0.0158	0.673	75%	mem

NeFwd

input: [*A]other: [*B]

• hidden-state-prefillinput, other: [2048, 4096]dtype=f16, bf16, f32
• cnn-feat-broadcastinput: [16, 256, 56, 56]other: [256, 1, 1]dtype=f16, bf16, f32
• llama-7b-ffninput, other: [1024, 11008]dtype=f16, bf16, f32
• linear-biasinput: [1024, 4096]other: [1, 4096]dtype=f16, bf16, f32
• llama-7b-hiddeninput, other: [1024, 4096]dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state-prefill	f16	1.02×	0.0130	torch-compile torch	0.0132 0.0132	0.647	72%	mem
	bf16	1.00×	0.0129	torch-compile torch	0.0129 0.0132	0.649	72%	mem
	f32	1.00×	0.0223	torch-compile torch	0.0223 0.0224	0.376	75%	mem
cnn-feat-broadcast	f16	1.01×	0.0118	torch-compile torch	0.0120 0.0478	1.09	72%	mem
	bf16	1.02×	0.0118	torch-compile torch	0.0120 0.0493	1.09	72%	mem
	f32	0.99×	0.0185	torch-compile torch	0.0184 0.0481	0.693	77%	mem
llama-7b-ffn	f16	1.01×	0.0167	torch-compile torch	0.0168 0.0168	0.676	75%	mem
	bf16	1.00×	0.0167	torch-compile torch	0.0167 0.0168	0.676	75%	mem
	f32	1.00×	0.0286	torch-compile torch	0.0286 0.0287	0.394	79%	mem
linear-bias	f16	1.03×	0.00483	torch-compile torch	0.00496 0.0154	0.868	58%	lat
	bf16	0.95×	0.00483	torch-compile torch	0.00458 0.0157	0.868	58%	lat
	f32	0.95×	0.00765	torch-compile torch	0.00726 0.0158	0.548	61%	mem
llama-7b-hidden	f16	0.95×	0.00752	torch torch-compile	0.00717 0.00722	0.558	62%	mem

GtFwd

input: [*A]other: [*B]

• hidden-state-prefillinput, other: [2048, 4096]dtype=f16, bf16, f32
• cnn-feat-broadcastinput: [16, 256, 56, 56]other: [256, 1, 1]dtype=f16, bf16, f32
• llama-7b-ffninput, other: [1024, 11008]dtype=f16, bf16, f32
• linear-biasinput: [1024, 4096]other: [1, 4096]dtype=f16, bf16, f32
• llama-7b-hiddeninput, other: [1024, 4096]dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state-prefill	f16	1.02×	0.0129	torch-compile torch	0.0132 0.0132	0.65	72%	mem
	bf16	1.01×	0.0130	torch-compile torch	0.0132 0.0132	0.647	72%	mem
	f32	0.99×	0.0223	torch-compile torch	0.0222 0.0222	0.376	75%	mem
cnn-feat-broadcast	f16	1.02×	0.0118	torch-compile torch	0.0120 0.0486	1.09	72%	mem
	bf16	1.00×	0.0118	torch-compile torch	0.0119 0.0493	1.08	72%	mem
	f32	1.00×	0.0185	torch-compile torch	0.0186 0.0483	0.693	77%	mem
llama-7b-ffn	f16	1.00×	0.0166	torch-compile torch	0.0167 0.0168	0.677	75%	mem
	bf16	1.00×	0.0166	torch torch-compile	0.0167 0.0167	0.677	75%	mem
	f32	1.00×	0.0286	torch-compile torch	0.0286 0.0287	0.394	79%	mem
linear-bias	f16	0.99×	0.00464	torch-compile torch	0.00461 0.0157	0.904	60%	lat
	bf16	0.95×	0.0048	torch-compile torch	0.00456 0.0159	0.874	58%	lat
	f32	0.98×	0.00762	torch-compile torch	0.00746 0.0158	0.551	61%	mem
llama-7b-hidden	f16	0.96×	0.00752	torch-compile torch	0.00725 0.00733	0.558	62%	mem

GeFwd

input: [*A]other: [*B]

• hidden-state-prefillinput, other: [2048, 4096]dtype=f16, bf16, f32
• cnn-feat-broadcastinput: [16, 256, 56, 56]other: [256, 1, 1]dtype=f16, bf16, f32
• llama-7b-ffninput, other: [1024, 11008]dtype=f16, bf16, f32
• linear-biasinput: [1024, 4096]other: [1, 4096]dtype=f16, bf16, f32
• llama-7b-hiddeninput, other: [1024, 4096]dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state-prefill	f16	0.99×	0.0130	torch-compile torch	0.0129 0.0130	0.646	72%	mem
	bf16	1.00×	0.0129	torch-compile torch	0.0130 0.0131	0.648	72%	mem
	f32	1.00×	0.0223	torch-compile torch	0.0222 0.0224	0.377	75%	mem
cnn-feat-broadcast	f16	1.02×	0.0118	torch-compile torch	0.0120 0.0472	1.08	72%	mem
	bf16	1.02×	0.0118	torch-compile torch	0.0121 0.0479	1.08	72%	mem
	f32	1.00×	0.0186	torch-compile torch	0.0185 0.0473	0.692	77%	mem
llama-7b-ffn	f16	1.01×	0.0166	torch-compile torch	0.0168 0.0168	0.68	76%	mem
	bf16	1.01×	0.0166	torch-compile torch	0.0168 0.0168	0.677	75%	mem
	f32	1.00×	0.0286	torch-compile torch	0.0286 0.0287	0.394	79%	mem
linear-bias	f16	1.02×	0.00483	torch-compile torch	0.00494 0.0153	0.868	58%	lat
	bf16	1.89×	0.00477	torch-compile torch	0.00899 0.0155	0.88	59%	lat
	f32	0.97×	0.00768	torch-compile torch	0.00749 0.0156	0.546	61%	mem
llama-7b-hidden	f16	0.95×	0.00749	torch torch-compile	0.00714 0.00726	0.56	62%	mem

LtFwd

input: [*A]other: [*B]

• hidden-state-prefillinput, other: [2048, 4096]dtype=f16, bf16, f32
• cnn-feat-broadcastinput: [16, 256, 56, 56]other: [256, 1, 1]dtype=f16, bf16, f32
• llama-7b-ffninput, other: [1024, 11008]dtype=f16, bf16, f32
• linear-biasinput: [1024, 4096]other: [1, 4096]dtype=f16, bf16, f32
• llama-7b-hiddeninput, other: [1024, 4096]dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state-prefill	f16	1.01×	0.0130	torch-compile torch	0.0131 0.0132	0.647	72%	mem
	bf16	1.01×	0.0130	torch-compile torch	0.0131 0.0132	0.646	72%	mem
	f32	1.00×	0.0223	torch-compile torch	0.0223 0.0224	0.377	75%	mem
cnn-feat-broadcast	f16	1.01×	0.0119	torch-compile torch	0.0120 0.0501	1.08	72%	mem
	bf16	1.02×	0.0118	torch-compile torch	0.0120 0.0507	1.08	72%	mem
	f32	1.00×	0.0185	torch-compile torch	0.0185 0.0496	0.693	77%	mem
llama-7b-ffn	f16	1.01×	0.0166	torch-compile torch	0.0168 0.0168	0.678	75%	mem
	bf16	1.00×	0.0167	torch-compile torch	0.0167 0.0168	0.676	75%	mem
	f32	1.00×	0.0286	torch-compile torch	0.0287 0.0287	0.394	79%	mem
linear-bias	f16	1.00×	0.00483	torch-compile torch	0.00483 0.0161	0.868	58%	lat
	bf16	1.00×	0.00483	torch-compile torch	0.00482 0.0163	0.868	58%	lat
	f32	0.95×	0.00765	torch-compile torch	0.00726 0.0161	0.548	61%	mem
llama-7b-hidden	f16	0.98×	0.00752	torch torch-compile	0.00736 0.00758	0.558	62%	mem

LeFwd

input: [*A]other: [*B]

• hidden-state-prefillinput, other: [2048, 4096]dtype=f16, bf16, f32
• cnn-feat-broadcastinput: [16, 256, 56, 56]other: [256, 1, 1]dtype=f16, bf16, f32
• llama-7b-ffninput, other: [1024, 11008]dtype=f16, bf16, f32
• linear-biasinput: [1024, 4096]other: [1, 4096]dtype=f16, bf16, f32
• llama-7b-hiddeninput, other: [1024, 4096]dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state-prefill	f16	1.00×	0.0130	torch torch-compile	0.0130 0.0132	0.647	72%	mem
	bf16	1.00×	0.0129	torch torch-compile	0.0129 0.0132	0.649	72%	mem
	f32	1.00×	0.0223	torch-compile torch	0.0223 0.0223	0.376	75%	mem
cnn-feat-broadcast	f16	1.00×	0.0119	torch-compile torch	0.0119 0.0501	1.08	72%	mem
	bf16	1.02×	0.0119	torch-compile torch	0.0121 0.0507	1.08	72%	mem
	f32	1.00×	0.0185	torch-compile torch	0.0185 0.0497	0.694	77%	mem
llama-7b-ffn	f16	0.99×	0.0167	torch torch-compile	0.0165 0.0168	0.675	75%	mem
	bf16	0.99×	0.0167	torch-compile torch	0.0165 0.0167	0.676	75%	mem
	f32	1.00×	0.0286	torch-compile torch	0.0286 0.0286	0.394	79%	mem
linear-bias	f16	0.97×	0.00483	torch-compile torch	0.0047 0.0161	0.868	58%	lat
	bf16	0.96×	0.00483	torch-compile torch	0.00464 0.0163	0.868	58%	lat
	f32	0.99×	0.00768	torch-compile torch	0.00762 0.0161	0.546	61%	mem
llama-7b-hidden	f16	0.95×	0.00752	torch torch-compile	0.00717 0.00725	0.558	62%	mem

IsnanFwd

input: [*S]

• 16Minput: [4096, 4096]dtype=f16, bf16, f32
• 256Minput: [16384, 16384]dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	f16	1.05×	0.0140	torch-compile torch	0.0147 0.0151	1.2	80%	mem
	bf16	1.06×	0.0140	torch-compile torch	0.0148 0.0152	1.2	80%	mem
	f32	1.01×	0.0228	torch-compile torch	0.0229 0.0229	0.737	82%	mem
256M	f16	1.07×	0.1854	torch-compile torch	0.1986 0.2037	1.45	96%	mem
	bf16	1.08×	0.1855	torch-compile torch	0.2000 0.2057	1.45	96%	mem

IsinfFwd

input: [*S]

• 16Minput: [4096, 4096]dtype=f16, bf16, f32
• 256Minput: [16384, 16384]dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	f16	1.06×	0.0140	torch-compile torch	0.0148 0.0314	1.2	80%	mem
	bf16	1.06×	0.0141	torch-compile torch	0.0149 0.0315	1.19	79%	mem
	f32	1.00×	0.0228	torch-compile torch	0.0228 0.0563	0.737	82%	mem
256M	f16	1.08×	0.1855	torch-compile torch	0.2004 0.4550	1.45	96%	mem
	bf16	1.09×	0.1851	torch-compile torch	0.2018 0.4559	1.45	97%	mem

IsfiniteFwd

input: [*S]

• 16Minput: [4096, 4096]dtype=f16, bf16, f32
• 256Minput: [16384, 16384]dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	f16	1.05×	0.0140	torch-compile torch	0.0147 0.0634	1.2	80%	mem
	bf16	1.05×	0.0140	torch-compile torch	0.0147 0.0636	1.2	80%	mem
	f32	1.01×	0.0228	torch-compile torch	0.0229 0.0952	0.737	82%	mem
256M	f16	1.07×	0.1854	torch-compile torch	0.1982 0.9197	1.45	96%	mem
	bf16	1.07×	0.1856	torch-compile torch	0.1982 0.9231	1.45	96%	mem

LogicalAndFwd

input: [*A]other: [*B]

• hidden-state-prefillinput, other: [2048, 4096]dtype=bool, f16, bf16, f32
• cnn-feat-broadcastinput: [16, 256, 56, 56]other: [256, 1, 1]dtype=bool, f16, bf16, f32
• llama-7b-ffninput, other: [1024, 11008]dtype=bool, f16, bf16, f32
• linear-biasinput: [1024, 4096]other: [1, 4096]dtype=bool, f16, bf16, f32
• llama-7b-hiddeninput, other: [1024, 4096]dtype=f16
• phi-2-ffninput, other: [1024, 10240]dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state-prefill	bool	1.07×	0.00819	torch-compile torch	0.00875 0.00998	1.02	68%	mem
	f16	0.99×	0.0130	torch torch-compile	0.0129 0.0132	1.93	72%	mem
	bf16	1.00×	0.0130	torch torch-compile	0.0130 0.0132	1.94	72%	mem
	f32	0.99×	0.0221	torch torch-compile	0.0220 0.0221	1.14	76%	mem
cnn-feat-broadcast	bool	1.21×	0.00829	torch-compile torch	0.0100 0.0453	1.55	69%	mem
	f16	1.05×	0.0116	torch-compile torch	0.0122 0.0472	3.31	73%	mem
	bf16	1.04×	0.0117	torch-compile torch	0.0122 0.0482	3.3	73%	mem
	f32	1.00×	0.0186	torch-compile torch	0.0185 0.0469	2.08	77%	mem
llama-7b-ffn	bool	1.10×	0.0103	torch-compile torch	0.0114 0.0131	1.09	73%	mem
	f16	0.98×	0.0166	torch torch-compile	0.0163 0.0167	2.04	75%	mem
	bf16	0.99×	0.0166	torch torch-compile	0.0165 0.0166	2.03	75%	mem
	f32	1.00×	0.0287	torch torch-compile	0.0286 0.0288	1.18	78%	mem
linear-bias	bool	1.36×	0.0036	torch-compile torch	0.0049 0.0146	1.17	52%	lat
	f16	0.97×	0.00483	torch-compile torch	0.00467 0.0153	2.6	58%	lat
	bf16	0.97×	0.0048	torch-compile torch	0.00466 0.0155	2.62	58%	lat
	f32	0.94×	0.00746	torch-compile torch	0.00702 0.0154	1.69	63%	mem
llama-7b-hidden	f16	0.95×	0.00738	torch-compile torch	0.00698 0.0071	1.71	63%	mem
phi-2-ffn	f16	1.01×	0.0156	torch-compile torch	0.0158 0.0158	2.02	75%	mem

LogicalOrFwd

input: [*A]other: [*B]

• hidden-state-prefillinput, other: [2048, 4096]dtype=bool, f16, bf16, f32
• cnn-feat-broadcastinput: [16, 256, 56, 56]other: [256, 1, 1]dtype=bool, f16, bf16, f32
• llama-7b-ffninput, other: [1024, 11008]dtype=bool, f16, bf16, f32
• linear-biasinput: [1024, 4096]other: [1, 4096]dtype=bool, f16, bf16, f32
• llama-7b-hiddeninput, other: [1024, 4096]dtype=f16
• phi-2-ffninput, other: [1024, 10240]dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state-prefill	bool	1.04×	0.00837	torch-compile torch	0.00874 0.00901	1	67%	mem
	f16	1.01×	0.0129	torch-compile torch	0.0130 0.0131	1.95	72%	mem
	bf16	1.01×	0.0129	torch torch-compile	0.0130 0.0130	1.95	72%	mem
	f32	1.00×	0.0222	torch-compile torch	0.0222 0.0223	1.13	75%	mem
cnn-feat-broadcast	bool	1.26×	0.0081	torch-compile torch	0.0102 0.0445	1.59	70%	mem
	f16	1.04×	0.0116	torch-compile torch	0.0120 0.0479	3.34	74%	mem
	bf16	1.04×	0.0116	torch-compile torch	0.0120 0.0491	3.34	74%	mem
	f32	1.00×	0.0183	torch-compile torch	0.0183 0.0471	2.11	78%	mem
llama-7b-ffn	bool	1.04×	0.0104	torch-compile torch	0.0108 0.0113	1.08	72%	mem
	f16	1.00×	0.0164	torch torch-compile	0.0164 0.0164	2.06	76%	mem
	bf16	0.99×	0.0165	torch-compile torch	0.0164 0.0165	2.05	76%	mem
	f32	1.00×	0.0289	torch-compile torch	0.0289 0.0290	1.17	78%	mem
linear-bias	bool	1.28×	0.00354	torch-compile torch	0.00451 0.0144	1.19	53%	lat
	f16	1.03×	0.00467	torch-compile torch	0.0048 0.0154	2.69	60%	lat
	bf16	1.01×	0.00474	torch-compile torch	0.00477 0.0157	2.66	59%	lat
	f32	0.98×	0.00736	torch-compile torch	0.00718 0.0154	1.71	63%	mem
llama-7b-hidden	f16	1.02×	0.00723	torch-compile torch	0.00736 0.00746	1.74	64%	mem
phi-2-ffn	f16	1.00×	0.0156	torch-compile torch	0.0155 0.0157	2.02	75%	mem

LogicalNotFwd

input: [*S]

• 16Minput: [4096, 4096]dtype=bool, f16, f32
• 256Minput: [16384, 16384]dtype=bool

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	bool	1.26×	0.00954	torch-compile torch	0.0120 0.0131	1.76	78%	mem
	f16	1.03×	0.0142	torch-compile torch	0.0146 0.0149	1.18	79%	mem
	f32	1.00×	0.0226	torch-compile torch	0.0226 0.0228	0.743	82%	mem
256M	bool	1.30×	0.1277	torch-compile torch	0.1658 0.1832	2.1	93%	mem

BitwiseAndFwd

input: [*A]other: [*B]

• hidden-state-prefillinput, other: [2048, 4096]dtype=bool, i32, i64
• cnn-feat-broadcastinput: [16, 256, 56, 56]other: [256, 1, 1]dtype=bool, i32, i64
• llama-7b-ffninput, other: [1024, 11008]dtype=bool, i32, i64
• linear-biasinput: [1024, 4096]other: [1, 4096]dtype=bool, i32, i64
• llama-7b-hiddeninput, other: [1024, 4096]dtype=i32
• phi-2-ffninput, other: [1024, 10240]dtype=i32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state-prefill	bool	1.08×	0.0081	torch-compile torch	0.00877 0.0100	1.04	69%	mem
	i32	1.00×	0.0261	torch torch-compile	0.0260 0.0261	0.322	86%	mem
	i64	1.01×	0.0489	torch torch-compile	0.0496 0.0497	0.172	92%	mem
cnn-feat-broadcast	bool	1.27×	0.00797	torch-compile torch	0.0101 0.0455	1.61	72%	mem
	i32	0.99×	0.0268	torch-compile torch	0.0266 0.0499	0.48	85%	mem
	i64	1.00×	0.0505	torch-compile torch	0.0504 0.0588	0.254	90%	mem
llama-7b-ffn	bool	1.11×	0.0103	torch-compile torch	0.0114 0.0132	1.09	73%	mem
	i32	1.00×	0.0342	torch torch-compile	0.0342 0.0342	0.329	88%	mem
	i64	1.00×	0.0649	torch-compile torch	0.0649 0.0652	0.174	93%	mem
linear-bias	bool	1.35×	0.00349	torch-compile torch	0.0047 0.0146	1.2	53%	lat
	i32	1.15×	0.00976	torch-compile torch	0.0113 0.0165	0.43	76%	mem
	i64	1.03×	0.0179	torch-compile torch	0.0185 0.0209	0.234	83%	mem
llama-7b-hidden	i32	1.00×	0.0143	torch-compile torch	0.0143 0.0143	0.293	78%	mem
phi-2-ffn	i32	1.01×	0.0316	torch-compile torch	0.0318 0.0319	0.331	88%	mem

BitwiseOrFwd

input: [*A]other: [*B]

• hidden-state-prefillinput, other: [2048, 4096]dtype=bool, i32, i64
• cnn-feat-broadcastinput: [16, 256, 56, 56]other: [256, 1, 1]dtype=bool, i32, i64
• llama-7b-ffninput, other: [1024, 11008]dtype=bool, i32, i64
• linear-biasinput: [1024, 4096]other: [1, 4096]dtype=bool, i32, i64
• llama-7b-hiddeninput, other: [1024, 4096]dtype=i32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state-prefill	bool	1.04×	0.00794	torch-compile torch	0.00829 0.00864	1.06	70%	mem
	i32	1.00×	0.0260	torch torch-compile	0.0261 0.0261	0.322	86%	mem
	i64	1.01×	0.0490	torch torch-compile	0.0494 0.0496	0.171	91%	mem
cnn-feat-broadcast	bool	1.30×	0.0079	torch-compile torch	0.0102 0.0445	1.63	72%	mem
	i32	0.99×	0.0267	torch-compile torch	0.0266 0.0498	0.481	85%	mem
	i64	1.00×	0.0509	torch-compile torch	0.0507 0.0593	0.252	90%	mem
llama-7b-ffn	bool	1.06×	0.0103	torch-compile torch	0.0109 0.0113	1.1	73%	mem
	i32	1.00×	0.0341	torch-compile torch	0.0342 0.0342	0.33	88%	mem
	i64	1.00×	0.0650	torch-compile torch	0.0650 0.0654	0.174	92%	mem
linear-bias	bool	1.27×	0.00344	torch-compile torch	0.00438 0.0143	1.22	54%	lat
	i32	1.15×	0.00976	torch-compile torch	0.0112 0.0165	0.43	76%	mem
	i64	1.03×	0.0179	torch-compile torch	0.0184 0.0208	0.235	84%	mem
llama-7b-hidden	i32	1.00×	0.0142	torch-compile torch	0.0142 0.0143	0.295	79%	mem

BitwiseXorFwd

input: [*A]other: [*B]

• hidden-state-prefillinput, other: [2048, 4096]dtype=bool, i32, i64
• cnn-feat-broadcastinput: [16, 256, 56, 56]other: [256, 1, 1]dtype=bool, i32, i64
• llama-7b-ffninput, other: [1024, 11008]dtype=bool, i32, i64
• linear-biasinput: [1024, 4096]other: [1, 4096]dtype=bool, i32, i64
• llama-7b-hiddeninput, other: [1024, 4096]dtype=i32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
hidden-state-prefill	bool	1.10×	0.00784	torch-compile torch	0.00864 0.0100	1.07	71%	mem
	i32	1.00×	0.0259	torch-compile torch	0.0260 0.0260	0.324	86%	mem
	i64	1.01×	0.0489	torch-compile torch	0.0493 0.0495	0.172	91%	mem
cnn-feat-broadcast	bool	1.29×	0.00784	torch-compile torch	0.0101 0.0454	1.64	73%	mem
	i32	0.99×	0.0267	torch-compile torch	0.0266 0.0500	0.481	85%	mem
	i64	1.00×	0.0507	torch-compile torch	0.0508 0.0593	0.253	90%	mem
llama-7b-ffn	bool	1.11×	0.0102	torch-compile torch	0.0114 0.0132	1.1	74%	mem
	i32	0.99×	0.0341	torch-compile torch	0.0339 0.0342	0.33	88%	mem
	i64	1.00×	0.0648	torch-compile torch	0.0650 0.0653	0.174	93%	mem
linear-bias	bool	2.59×	0.00342	torch-compile torch	0.00886 0.0145	1.22	54%	lat
	i32	1.15×	0.00974	torch-compile torch	0.0112 0.0165	0.43	77%	mem
	i64	1.03×	0.0179	torch-compile torch	0.0183 0.0208	0.235	84%	mem
llama-7b-hidden	i32	1.00×	0.0141	torch torch-compile	0.0142 0.0142	0.297	79%	mem

BitwiseNotFwd

input: [*S]

• 16Minput: [4096, 4096]dtype=i32, i64
• 256Minput: [16384, 16384]dtype=i32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	i32	1.01×	0.0331	torch-compile torch	0.0334 0.0335	0.507	90%	mem
	i64	0.99×	0.0654	torch-compile torch	0.0650 0.0680	0.256	91%	mem
256M	i32	1.01×	0.5018	torch-compile torch	0.5070 0.5071	0.535	95%	mem

WhereFwd

condition: [*C], boolinput: [*A]other: [*B]

• 16Mcondition: [4096, 4096], boolinput, other: [4096, 4096]dtype=f16, bf16, f32
• 256Mcondition: [16384, 16384], boolinput, other: [16384, 16384]dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	f16	1.00×	0.0301	torch torch-compile	0.0302 0.0304	0.557	87%	mem
	bf16	0.99×	0.0303	torch-compile torch	0.0301 0.0301	0.554	86%	mem
	f32	0.99×	0.0532	torch-compile torch	0.0528 0.0534	0.316	91%	mem
256M	f16	1.00×	0.4249	torch-compile torch	0.4248 0.4251	0.632	98%	mem
	bf16	1.00×	0.4248	torch torch-compile	0.4249 0.4249	0.632	98%	mem

ClampTensorFwd

input: [*A]min: [*L]max: [*U]

• 16Minput, min, max: [4096, 4096]dtype=f16, bf16, f32
• 256Minput, min, max: [16384, 16384]dtype=f16, bf16

input: [*A]min: [*L]

• 16M-min-onlyinput, min: [4096, 4096]dtype=f16, bf16, f32
• 256M-min-onlyinput, min: [16384, 16384]dtype=f16, bf16

input: [*A]max: [*U]

• 16M-max-onlyinput, max: [4096, 4096]dtype=f16, bf16, f32
• 256M-max-onlyinput, max: [16384, 16384]dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	f16	1.00×	0.0342	torch torch-compile	0.0344 0.0345	0.49	87%	mem
	bf16	1.00×	0.0343	torch torch-compile	0.0344 0.0348	0.489	87%	mem
	f32	1.00×	0.0647	torch torch-compile	0.0648 0.0648	0.259	92%	mem
256M	f16	1.00×	0.4829	torch-compile torch	0.4822 0.4823	0.556	99%	mem
	bf16	1.00×	0.4829	torch-compile torch	0.4821 0.4822	0.556	99%	mem
16M-min-only	f16	1.00×	0.0260	torch-compile torch	0.0260 0.0264	0.645	86%	mem
	bf16	1.00×	0.0260	torch-compile torch	0.0260 0.0264	0.644	86%	mem
	f32	0.99×	0.0493	torch-compile torch	0.0488 0.0489	0.34	91%	mem
256M-min-only	f16	1.00×	0.3673	torch torch-compile	0.3666 0.3673	0.731	97%	mem
	bf16	1.00×	0.3670	torch torch-compile	0.3660 0.3672	0.731	97%	mem
16M-max-only	f16	1.00×	0.0259	torch-compile torch	0.0260 0.0263	0.647	86%	mem
	bf16	1.00×	0.0260	torch-compile torch	0.0260 0.0264	0.646	86%	mem
	f32	0.99×	0.0492	torch-compile torch	0.0488 0.0490	0.341	91%	mem
256M-max-only	f16	1.00×	0.3671	torch torch-compile	0.3668 0.3672	0.731	97%	mem
	bf16	1.00×	0.3673	torch torch-compile	0.3661 0.3672	0.731	97%	mem

ClampScalarFwd

input: [*S]

min=-0.5max=0.5

• 16Minput: [4096, 4096]dtype=f16, bf16, f32
• 256Minput: [16384, 16384]dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	f16	1.00×	0.0180	torch-compile torch	0.0179 0.0202	0.935	83%	mem
	bf16	1.02×	0.0180	torch-compile torch	0.0183 0.0189	0.933	83%	mem
	f32	1.01×	0.0334	torch-compile torch	0.0336 0.0336	0.503	89%	mem
256M	f16	0.99×	0.2576	torch-compile torch	0.2560 0.2953	1.04	93%	mem
	bf16	1.03×	0.2597	torch-compile torch	0.2674 0.2788	1.03	92%	mem

MaskedFillTensorFwd

input: [*A]mask: [*M], boolvalue: []

• 16Minput: [4096, 4096]mask: [4096, 4096], boolvalue: []dtype=f16, bf16, f32
• 256Minput: [16384, 16384]mask: [16384, 16384], boolvalue: []dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	f16	1.00×	0.0225	torch-compile torch	0.0225 0.0409	0.746	83%	mem
	bf16	1.00×	0.0225	torch-compile torch	0.0225 0.0407	0.746	83%	mem
	f32	1.00×	0.0378	torch-compile torch	0.0376 0.0741	0.444	89%	mem
256M	f16	1.00×	0.3099	torch-compile torch	0.3100 0.5731	0.866	96%	mem
	bf16	1.00×	0.3100	torch-compile torch	0.3100 0.5730	0.866	96%	mem

MaskedFillScalarFwd

input: [*A]mask: [*M], bool

• 16Minput: [4096, 4096]mask: [4096, 4096], booldtype=f16, bf16, f32
• 256Minput: [16384, 16384]mask: [16384, 16384], booldtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	f16	1.00×	0.0224	torch-compile torch	0.0225 0.0380	0.748	83%	mem
	bf16	1.00×	0.0225	torch-compile torch	0.0225 0.0381	0.746	83%	mem
	f32	0.99×	0.0377	torch-compile torch	0.0375 0.0715	0.445	89%	mem
256M	f16	1.00×	0.3099	torch-compile torch	0.3101 0.5706	0.866	96%	mem
	bf16	1.00×	0.3099	torch-compile torch	0.3100 0.5707	0.866	96%	mem

NanToNumFwd

input: [*S]

• 16Minput: [4096, 4096]dtype=f16, bf16, f32
• 256Minput: [16384, 16384]dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
16M	f16	1.00×	0.0180	torch-compile torch	0.0180 0.0191	1.86	83%	mem
	bf16	1.00×	0.0181	torch-compile torch	0.0180 0.0191	1.85	82%	mem
	f32	1.01×	0.0333	torch-compile torch	0.0335 0.0336	1.01	90%	mem
256M	f16	1.00×	0.2613	torch-compile torch	0.2608 0.2781	2.05	91%	mem
	bf16	0.99×	0.2635	torch-compile torch	0.2604 0.2772	2.04	91%	mem

AlibiFwd

num_heads=32dtype=f16, bf16

• mpt-7b-2kseq_len=2048
• mpt-7b-4kseq_len=4096

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
mpt-7b-2k	f16	1.33×	0.0649	torch-compile torch-ref	0.0864 0.5930	6.2	92%	mem
	bf16	1.33×	0.0648	torch-compile torch-ref	0.0862 0.5931	6.21	92%	mem
mpt-7b-4k	f16	1.19×	0.2861	torch-compile torch-ref	0.3405 2.6323	5.63	83%	mem
	bf16	1.20×	0.2849	torch-compile torch-ref	0.3404 2.6320	5.65	84%	mem

SinusoidalFwd

d_model=4096dtype=f16, bf16

• transformer-2k-4kseq_len=2048
• transformer-4k-4kseq_len=4096

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
transformer-2k-4k	f16	4.16×	0.0136	torch-compile torch-ref	0.0568 0.1053	0.923	27%	mem
	bf16	4.16×	0.0136	torch-compile torch-ref	0.0567 0.1043	0.923	27%	mem
transformer-4k-4k	f16	4.45×	0.0251	torch-compile torch-ref	0.1118 0.1977	1	30%	mem
	bf16	4.45×	0.0251	torch-compile torch-ref	0.1118 0.1973	1	30%	mem