Elementwise & Reduction¶

本页暂无中文版。以下为英文原文。

97 ops, 508 workloads — Elementwise 76 · Reduction 21.

One table per op, one row per workload. Ratio is the fastest other implementation's device time divided by ours, so green is faster than it, plain is level with it, red is slower. Times are in ms. How these numbers are taken.

Elementwise¶

DropoutFwd (3 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`tokens-1k-hidden-10k-bfloat16`	1.91×	0.0123	`torch-compile` `torch`	0.0236 0.0236	0.851
`tokens-1k-hidden-4k-float16`	1.83×	0.00618	`torch-compile` `torch`	0.0113 0.0117	0.679
`tokens-1k-hidden-4k-float32`	1.16×	0.0103	`torch-compile` `torch`	0.0120 0.0149	0.407

MishFwd (4 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`yolo-p3-bfloat16`	1.58×	0.0405	`torch` `torch-compile`	0.0642 0.0741	2.59
`yolo-p3-float16`	1.58×	0.0403	`torch` `torch-compile`	0.0636 0.0733	2.6
`yolo-p4-bfloat16`	1.55×	0.0216	`torch` `torch-compile`	0.0336 0.0493	2.42
`yolo-p4-float16`	1.55×	0.0214	`torch` `torch-compile`	0.0333 0.0382	2.45

SoftplusFwd (4 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`mlp-hidden-bfloat16`	1.43×	0.0128	`torch-compile` `torch`	0.0183 0.0241	3.28
`mlp-hidden-float16`	1.42×	0.0126	`torch-compile` `torch`	0.0179 0.0238	3.33
`mlp-hidden-wide-bfloat16`	1.46×	0.0233	`torch-compile` `torch`	0.0339 0.0453	3.61
`mlp-hidden-wide-float16`	1.45×	0.0228	`torch-compile` `torch`	0.0330 0.0448	3.67

EluFwd (4 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`mlp-hidden-bfloat16`	1.39×	0.0120	`torch-compile` `torch`	0.0167 0.0182	2.79
`mlp-hidden-float16`	1.31×	0.0122	`torch-compile` `torch`	0.0159 0.0180	2.76
`mlp-hidden-wide-bfloat16`	1.45×	0.0215	`torch-compile` `torch`	0.0312 0.0332	3.12
`mlp-hidden-wide-float16`	1.36×	0.0219	`torch-compile` `torch`	0.0298 0.0328	3.06

SeluFwd (4 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`snn-fc-bfloat16`	1.29×	0.0121	`torch-compile` `torch`	0.0156 0.0182	3.47
`snn-fc-float16`	1.37×	0.0117	`torch-compile` `torch`	0.0161 0.0180	3.57
`snn-fc-wide-bfloat16`	1.34×	0.0218	`torch-compile` `torch`	0.0292 0.0333	3.85
`snn-fc-wide-float16`	1.42×	0.0210	`torch-compile` `torch`	0.0299 0.0328	3.99

SinusoidalFwd (4 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`transformer-2k-4k-bfloat16`	1.34×	0.0424	`torch-compile` `torch-ref`	0.0567 0.1066	1.19
`transformer-2k-4k-float16`	1.34×	0.0424	`torch-compile` `torch-ref`	0.0567 0.1064	1.19
`transformer-4k-4k-bfloat16`	1.36×	0.0819	`torch-compile` `torch-ref`	0.1117 0.1994	1.23
`transformer-4k-4k-float16`	1.37×	0.0818	`torch-compile` `torch-ref`	0.1117 0.1993	1.23

Log1pFwd (5 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bfloat16`	1.45×	0.0181	`torch-compile` `torch`	0.0263 0.0268	1.85
`elementwise-16M-float16`	1.41×	0.0181	`torch-compile` `torch`	0.0256 0.0262	1.85
`elementwise-16M-float32`	0.97×	0.0352	`torch-compile` `torch`	0.0340 0.0340	0.954
`elementwise-256M-bfloat16`	1.50×	0.2549	`torch-compile` `torch`	0.3818 0.3878	2.11
`elementwise-256M-float16`	1.45×	0.2542	`torch-compile` `torch`	0.3695 0.3799	2.11

LogFwd (5 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bfloat16`	1.54×	0.0181	`torch-compile` `torch`	0.0278 0.0281	0.926
`elementwise-16M-float16`	1.50×	0.0181	`torch` `torch-compile`	0.0271 0.0272	0.925
`elementwise-16M-float32`	0.96×	0.0357	`torch-compile` `torch`	0.0341 0.0343	0.47
`elementwise-256M-bfloat16`	1.61×	0.2544	`torch-compile` `torch`	0.4103 0.4127	1.06
`elementwise-256M-float16`	1.57×	0.2534	`torch` `torch-compile`	0.3974 0.4020	1.06

Expm1Fwd (5 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bfloat16`	1.54×	0.0181	`torch-compile` `torch`	0.0279 0.0279	1.86
`elementwise-16M-float16`	1.41×	0.0180	`torch` `torch-compile`	0.0252 0.0268	1.87
`elementwise-16M-float32`	1.01×	0.0340	`torch` `torch-compile`	0.0342 0.0344	0.986
`elementwise-256M-bfloat16`	1.60×	0.2572	`torch-compile` `torch`	0.4103 0.4111	2.09
`elementwise-256M-float16`	1.45×	0.2542	`torch` `torch-compile`	0.3678 0.3937	2.11

AlibiFwd (4 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`llama-prefill-2k-bfloat16`	1.34×	0.0645	`torch-compile` `torch-ref`	0.0864 0.5918	6.24
`llama-prefill-2k-float16`	1.32×	0.0651	`torch-compile` `torch-ref`	0.0862 0.5918	6.18
`llama-prefill-4k-bfloat16`	1.20×	0.2847	`torch-compile` `torch-ref`	0.3406 2.6085	5.66
`llama-prefill-4k-float16`	1.20×	0.2852	`torch-compile` `torch-ref`	0.3415 2.6083	5.65

sub_bcast (3 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`sub-1024x10240-1x10240-float16-SubFwdOp-sub-normal`	1.01×	0.0122	`torch-compile` `torch`	0.0123 0.0332	0.86
`sub-1024x11008-1x11008-float16-SubFwdOp-sub-normal`	1.00×	0.0130	`torch-compile` `torch`	0.0130 0.0354	0.87
`sub-1024x4096-1x4096-float16-SubFwdOp-sub-normal`	1.59×	0.00582	`torch-compile` `torch`	0.00928 0.0149	0.72

LogicalNotFwd (4 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bool`	1.19×	0.0101	`torch-compile` `torch`	0.0121 0.0130	1.65
`elementwise-16M-float16`	1.02×	0.0148	`torch-compile` `torch`	0.0150 0.0152	1.14
`elementwise-16M-float32`	0.99×	0.0235	`torch-compile` `torch`	0.0234 0.0234	0.713
`elementwise-256M-bool`	1.30×	0.1263	`torch-compile` `torch`	0.1648 0.1819	2.12

BitwiseXorFwd (7 workloads · ✅)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`bitwise_xor-1024x4096-BitwiseXorFwdOp-bitwise_xor`	1.01×	0.0147	`torch-compile` `torch`	0.0148 0.0148	0.286
`cnn-feat-broadcast-bool`	1.25×	0.008	`torch-compile` `torch`	0.0100 0.0452	1.61
`cnn-feat-broadcast-int32`	1.16×	0.0263	`torch-compile` `torch`	0.0306 0.0494	0.488
`cnn-feat-broadcast-int64`	0.99×	0.0501	`torch-compile` `torch`	0.0498 0.0582	0.256
`hidden-state-prefill-bool`	1.08×	0.00814	`torch-compile` `torch`	0.0088 0.00995	1.03
`hidden-state-prefill-int32`	0.99×	0.0265	`torch` `torch-compile`	0.0263 0.0277	0.317
`hidden-state-prefill-int64`	1.00×	0.0492	`torch-compile` `torch`	0.0493 0.0495	0.171

GeluTanhAndMulFwd (3 workloads · ✅)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`ffn-gelu-tanh-decode-bfloat16`	1.01×	0.00147	`torch-compile` `torch-ref` `flashinfer`	0.00149 0.00304 0.00438	0.0974
`ffn-gelu-tanh-prefill-bfloat16`	1.07×	0.0493	`torch-compile` `flashinfer` `torch-ref`	0.0526 0.0577 0.1933	5.95
`ffn-gelu-tanh-prefill-float16`	1.08×	0.0477	`torch-compile` `flashinfer` `torch-ref`	0.0516 0.0567 0.1921	6.15

IsinfFwd (5 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bfloat16`	1.03×	0.0148	`torch-compile` `torch`	0.0152 0.0314	1.14
`elementwise-16M-float16`	1.03×	0.0148	`torch-compile` `torch`	0.0151 0.0313	1.14
`elementwise-16M-float32`	1.00×	0.0234	`torch-compile` `torch`	0.0233 0.0571	0.716
`elementwise-256M-bfloat16`	1.08×	0.1857	`torch-compile` `torch`	0.2001 0.4510	1.45
`elementwise-256M-float16`	1.07×	0.1862	`torch-compile` `torch`	0.1989 0.4500	1.44

CosFwd (5 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bfloat16`	1.03×	0.0264	`torch` `torch-compile`	0.0272 0.0284	0.635
`elementwise-16M-float16`	1.05×	0.0260	`torch` `torch-compile`	0.0272 0.0280	0.646
`elementwise-16M-float32`	0.98×	0.0352	`torch-compile` `torch`	0.0344 0.0344	0.477
`elementwise-256M-bfloat16`	1.03×	0.3828	`torch` `torch-compile`	0.3933 0.4129	0.701
`elementwise-256M-float16`	1.04×	0.3772	`torch` `torch-compile`	0.3929 0.4086	0.712

BitwiseOrFwd (7 workloads · ✅)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`bitwise_or-1024x4096-BitwiseOrFwdOp-bitwise_or`	0.98×	0.0148	`torch-compile` `torch`	0.0146 0.0146	0.284
`cnn-feat-broadcast-bool`	1.25×	0.00811	`torch-compile` `torch`	0.0102 0.0443	1.58
`cnn-feat-broadcast-int32`	0.99×	0.0266	`torch-compile` `torch`	0.0265 0.0494	0.482
`cnn-feat-broadcast-int64`	0.99×	0.0502	`torch-compile` `torch`	0.0499 0.0582	0.256
`hidden-state-prefill-bool`	1.05×	0.00813	`torch-compile` `torch`	0.00854 0.0088	1.03
`hidden-state-prefill-int32`	1.00×	0.0265	`torch-compile` `torch`	0.0265 0.0265	0.316
`hidden-state-prefill-int64`	1.00×	0.0493	`torch-compile` `torch`	0.0493 0.0496	0.17

RemainderFwd (8 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`cnn-feat-broadcast-bfloat16`	1.16×	0.0161	`torch-compile` `torch`	0.0186 0.0642	3.19
`cnn-feat-broadcast-float16`	1.11×	0.0160	`torch-compile` `torch`	0.0178 0.0623	3.2
`cnn-feat-broadcast-float32`	1.00×	0.0269	`torch-compile` `torch`	0.0268 0.0643	1.91
`hidden-state-prefill-bfloat16`	1.01×	0.0149	`torch-compile` `torch`	0.0151 0.0186	2.25
`hidden-state-prefill-float16`	1.01×	0.0154	`torch-compile` `torch`	0.0156 0.0181	2.18
`hidden-state-prefill-float32`	1.01×	0.0264	`torch-compile` `torch`	0.0267 0.0272	1.27
`remainder-1024x10240-float16-float16-RemainderFwdOp-remainder-positive`	1.01×	0.0181	`torch-compile` `torch`	0.0182 0.0217	0.58
`remainder-1024x4096-float16-float16-RemainderFwdOp-remainder-positive`	1.00×	0.00858	`torch-compile` `torch`	0.00861 0.0107	0.489

BitwiseAndFwd (8 workloads · ✅)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`bitwise_and-1024x10240-BitwiseAndFwdOp-bitwise_and`	0.99×	0.0321	`torch-compile` `torch`	0.0320 0.0321	0.326
`bitwise_and-1024x4096-BitwiseAndFwdOp-bitwise_and`	1.00×	0.0148	`torch-compile` `torch`	0.0147 0.0148	0.284
`cnn-feat-broadcast-bool`	1.24×	0.0081	`torch-compile` `torch`	0.0100 0.0452	1.59
`cnn-feat-broadcast-int32`	1.00×	0.0264	`torch-compile` `torch`	0.0265 0.0494	0.486
`cnn-feat-broadcast-int64`	0.99×	0.0500	`torch-compile` `torch`	0.0498 0.0583	0.257
`hidden-state-prefill-bool`	1.07×	0.00835	`torch-compile` `torch`	0.00896 0.0101	1
`hidden-state-prefill-int32`	1.00×	0.0262	`torch-compile` `torch`	0.0261 0.0263	0.32
`hidden-state-prefill-int64`	1.00×	0.0491	`torch-compile` `torch`	0.0491 0.0496	0.171

IsnanFwd (5 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bfloat16`	1.03×	0.0147	`torch-compile` `torch`	0.0151 0.0155	1.14
`elementwise-16M-float16`	1.02×	0.0147	`torch-compile` `torch`	0.0151 0.0154	1.14
`elementwise-16M-float32`	1.00×	0.0235	`torch-compile` `torch`	0.0234 0.0235	0.715
`elementwise-256M-bfloat16`	1.07×	0.1863	`torch-compile` `torch`	0.1985 0.2040	1.44
`elementwise-256M-float16`	1.06×	0.1863	`torch-compile` `torch`	0.1972 0.2016	1.44

SinFwd (5 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bfloat16`	1.04×	0.0258	`torch` `torch-compile`	0.0268 0.0270	0.65
`elementwise-16M-float16`	1.03×	0.0254	`torch` `torch-compile`	0.0263 0.0266	0.659
`elementwise-16M-float32`	0.98×	0.0349	`torch-compile` `torch`	0.0343 0.0343	0.481
`elementwise-256M-bfloat16`	1.03×	0.3740	`torch` `torch-compile`	0.3861 0.3905	0.718
`elementwise-256M-float16`	1.03×	0.3669	`torch` `torch-compile`	0.3776 0.3850	0.732

IsfiniteFwd (5 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bfloat16`	1.02×	0.0147	`torch-compile` `torch`	0.0150 0.0634	1.14
`elementwise-16M-float16`	1.02×	0.0147	`torch-compile` `torch`	0.0150 0.0631	1.14
`elementwise-16M-float32`	1.00×	0.0234	`torch-compile` `torch`	0.0234 0.0961	0.716
`elementwise-256M-bfloat16`	1.06×	0.1862	`torch-compile` `torch`	0.1967 0.9154	1.44
`elementwise-256M-float16`	1.06×	0.1862	`torch-compile` `torch`	0.1968 0.9124	1.44

GeluAndMulFwd (3 workloads · ✅)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`ffn-gelu-decode-bfloat16`	0.94×	0.00154	`torch-compile` `torch-ref` `flashinfer`	0.00144 0.00317 0.00666	0.056
`ffn-gelu-prefill-bfloat16`	1.03×	0.0595	`torch-compile` `flashinfer` `torch-ref`	0.0610 0.1060 0.2047	2.96
`ffn-gelu-prefill-float16`	1.10×	0.0550	`torch-compile` `flashinfer` `torch-ref`	0.0603 0.1050 0.2028	3.2

ClampScalarFwd (5 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bfloat16`	1.01×	0.0184	`torch-compile` `torch`	0.0187 0.0191	0.91
`elementwise-16M-float16`	1.00×	0.0184	`torch-compile` `torch`	0.0185 0.0203	0.91
`elementwise-16M-float32`	1.01×	0.0338	`torch-compile` `torch`	0.0340 0.0341	0.496
`elementwise-256M-bfloat16`	1.05×	0.2521	`torch-compile` `torch`	0.2656 0.2756	1.06
`elementwise-256M-float16`	1.01×	0.2524	`torch-compile` `torch`	0.2539 0.2919	1.06

ExpFwd (5 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bfloat16`	1.01×	0.0182	`torch` `torch-compile`	0.0183 0.0184	0.923
`elementwise-16M-float16`	1.01×	0.0181	`torch-compile` `torch`	0.0182 0.0182	0.928
`elementwise-16M-float32`	1.00×	0.0340	`torch-compile` `torch`	0.0341 0.0341	0.494
`elementwise-256M-bfloat16`	1.01×	0.2573	`torch` `torch-compile`	0.2590 0.2632	1.04
`elementwise-256M-float16`	1.01×	0.2544	`torch` `torch-compile`	0.2564 0.2565	1.06

TanhFwd (5 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bfloat16`	1.03×	0.0213	`torch` `torch-compile`	0.0219 0.0246	0.786
`elementwise-16M-float16`	1.00×	0.0208	`torch` `torch-compile`	0.0207 0.0242	0.808
`elementwise-16M-float32`	1.01×	0.0339	`torch` `torch-compile`	0.0341 0.0344	0.495
`elementwise-256M-bfloat16`	1.02×	0.3031	`torch` `torch-compile`	0.3097 0.3500	0.886
`elementwise-256M-float16`	0.99×	0.2956	`torch` `torch-compile`	0.2917 0.3444	0.908

HardtanhFwd (4 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`bounded-conv-feat-bfloat16`	1.01×	0.0146	`torch-compile` `torch`	0.0148 0.0152	0.878
`bounded-conv-feat-float16`	1.00×	0.0146	`torch-compile` `torch`	0.0147 0.0162	0.88
`bounded-hidden-bfloat16`	1.01×	0.0104	`torch-compile` `torch`	0.0105 0.0107	0.807
`bounded-hidden-float16`	1.00×	0.0104	`torch-compile` `torch`	0.0104 0.0113	0.809

mul_bcast (3 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`mul-1024x10240-1x10240-float16-MulFwdOp-mul-normal`	1.00×	0.0123	`torch-compile` `torch`	0.0124 0.0331	0.851
`mul-1024x11008-1x11008-float16-MulFwdOp-mul-normal`	1.00×	0.0129	`torch-compile` `torch`	0.0129 0.0353	0.874
`mul-1024x4096-1x4096-float16-MulFwdOp-mul-normal`	1.01×	0.00598	`torch-compile` `torch`	0.00605 0.0149	0.701

SqrtFwd (5 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bfloat16`	1.00×	0.0187	`torch-compile` `torch`	0.0187 0.0189	0.898
`elementwise-16M-float16`	1.00×	0.0186	`torch-compile` `torch`	0.0186 0.0189	0.901
`elementwise-16M-float32`	1.02×	0.0334	`torch-compile` `torch`	0.0340 0.0340	0.503
`elementwise-256M-bfloat16`	1.00×	0.2636	`torch-compile` `torch`	0.2642 0.2671	1.02
`elementwise-256M-float16`	1.00×	0.2628	`torch-compile` `torch`	0.2629 0.2659	1.02

RsqrtFwd (5 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bfloat16`	1.00×	0.0181	`torch-compile` `torch`	0.0181 0.0182	0.926
`elementwise-16M-float16`	1.00×	0.0181	`torch-compile` `torch`	0.0182 0.0182	0.926
`elementwise-16M-float32`	1.02×	0.0331	`torch-compile` `torch`	0.0337 0.0338	0.507
`elementwise-256M-bfloat16`	1.00×	0.2530	`torch-compile` `torch`	0.2533 0.2540	1.06
`elementwise-256M-float16`	1.00×	0.2537	`torch-compile` `torch`	0.2533 0.2538	1.06

LeakyReluFwd (4 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`gan-feat-bfloat16`	1.00×	0.0185	`torch-compile` `torch`	0.0184 0.0185	1.82
`gan-feat-deep-bfloat16`	1.00×	0.0103	`torch-compile` `torch`	0.0104 0.0104	1.62
`gan-feat-deep-float16`	1.00×	0.0104	`torch` `torch-compile`	0.0104 0.0104	1.62
`gan-feat-float16`	1.00×	0.0184	`torch-compile` `torch`	0.0184 0.0185	1.82

ReluFwd (3 workloads · ✅)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`hidden-state-decode-bfloat16`	1.00×	0.00122	`torch-compile` `torch`	0.00122 0.00141	0.00337
`hidden-state-prefill-bfloat16`	1.00×	0.0103	`torch-compile` `torch`	0.0104 0.0105	0.812
`hidden-state-prefill-float16`	1.00×	0.0103	`torch-compile` `torch`	0.0103 0.0108	0.812

FloorDivideFwd (8 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`cnn-feat-broadcast-bfloat16`	1.00×	0.0159	`torch-compile` `torch`	0.0159 0.1192	1.62
`cnn-feat-broadcast-float16`	0.99×	0.0159	`torch-compile` `torch`	0.0158 0.1093	1.61
`cnn-feat-broadcast-float32`	1.00×	0.0266	`torch-compile` `torch`	0.0267 0.1014	0.965
`floor_divide-1024x10240-float16-float16-FloorDivideFwdOp-floor_divide-positive`	1.00×	0.0180	`torch-compile` `torch`	0.0180 0.0595	0.584
`floor_divide-1024x4096-float16-float16-FloorDivideFwdOp-floor_divide-positive`	1.00×	0.0087	`torch-compile` `torch`	0.00874 0.0264	0.482
`hidden-state-prefill-bfloat16`	1.00×	0.0149	`torch-compile` `torch`	0.0149 0.0504	1.13
`hidden-state-prefill-float16`	1.00×	0.0151	`torch-compile` `torch`	0.0151 0.0485	1.11
`hidden-state-prefill-float32`	1.01×	0.0264	`torch-compile` `torch`	0.0265 0.0475	0.636

CeilFwd (5 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bfloat16`	1.00×	0.0179	`torch-compile` `torch`	0.0180 0.0180	0.938
`elementwise-16M-float16`	1.00×	0.0179	`torch-compile` `torch`	0.0180 0.0180	0.938
`elementwise-16M-float32`	1.00×	0.0340	`torch` `torch-compile`	0.0339 0.0340	0.494
`elementwise-256M-bfloat16`	1.00×	0.2497	`torch-compile` `torch`	0.2498 0.2504	1.07
`elementwise-256M-float16`	1.00×	0.2506	`torch-compile` `torch`	0.2498 0.2503	1.07

PreluFwd (4 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`cnn-feat-per-channel-bfloat16`	1.00×	0.0144	`torch-compile` `torch`	0.0144 0.0486	1.78
`cnn-feat-per-channel-deep-bfloat16`	1.00×	0.00819	`torch-compile` `torch`	0.00819 0.0257	1.57
`cnn-feat-per-channel-deep-float16`	1.00×	0.00835	`torch-compile` `torch`	0.00838 0.0250	1.54
`cnn-feat-per-channel-float16`	1.00×	0.0146	`torch-compile` `torch`	0.0146 0.0471	1.76

FloorFwd (5 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bfloat16`	1.00×	0.0179	`torch-compile` `torch`	0.0179 0.0180	0.938
`elementwise-16M-float16`	1.00×	0.0179	`torch` `torch-compile`	0.0180 0.0180	0.936
`elementwise-16M-float32`	1.00×	0.0340	`torch-compile` `torch`	0.0340 0.0340	0.493
`elementwise-256M-bfloat16`	1.00×	0.2496	`torch-compile` `torch`	0.2500 0.2504	1.08
`elementwise-256M-float16`	1.00×	0.2496	`torch-compile` `torch`	0.2498 0.2503	1.08

SubFwd (9 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`cnn-feat-broadcast-bfloat16`	1.00×	0.0145	`torch-compile` `torch`	0.0145 0.0464	1.77
`cnn-feat-broadcast-float16`	1.00×	0.0144	`torch-compile` `torch`	0.0144 0.0458	1.78
`cnn-feat-broadcast-float32`	1.00×	0.0265	`torch-compile` `torch`	0.0265 0.0497	0.968
`hidden-state-prefill-bfloat16`	1.00×	0.0148	`torch-compile` `torch`	0.0148 0.0149	1.13
`hidden-state-prefill-float16`	1.00×	0.0148	`torch` `torch-compile`	0.0148 0.0148	1.13
`hidden-state-prefill-float32`	1.00×	0.0265	`torch-compile` `torch`	0.0264 0.0265	0.634
`sub-1024x10240-float16-float16-SubFwdOp-sub-normal`	1.00×	0.0176	`torch-compile` `torch`	0.0176 0.0177	0.595
`sub-1024x11008-float16-float16-SubFwdOp-sub-normal`	1.00×	0.0185	`torch-compile` `torch`	0.0186 0.0186	0.608
`sub-1024x4096-float16-float16-SubFwdOp-sub-normal`	1.01×	0.00842	`torch-compile` `torch`	0.00846 0.00851	0.498

BitwiseNotFwd (3 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-int32`	1.00×	0.0340	`torch-compile` `torch`	0.0339 0.0340	0.493
`elementwise-16M-int64`	0.99×	0.0652	`torch-compile` `torch`	0.0647 0.0680	0.257
`elementwise-256M-int32`	1.01×	0.4986	`torch-compile` `torch`	0.5044 0.5045	0.538

AddFwd (6 workloads · ✅)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`cnn-feat-broadcast-bfloat16`	1.00×	0.0144	`torch-compile` `torch`	0.0144 0.0468	1.79
`cnn-feat-broadcast-float16`	1.00×	0.0144	`torch-compile` `torch`	0.0144 0.0458	1.78
`cnn-feat-broadcast-float32`	1.00×	0.0266	`torch-compile` `torch`	0.0266 0.0495	0.966
`hidden-state-prefill-bfloat16`	1.00×	0.0147	`torch-compile` `torch`	0.0147 0.0149	1.14
`hidden-state-prefill-float16`	1.00×	0.0147	`torch-compile` `torch`	0.0147 0.0148	1.14
`hidden-state-prefill-float32`	1.00×	0.0265	`torch-compile` `torch`	0.0264 0.0265	0.634

MulFwd (9 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`cnn-feat-broadcast-bfloat16`	1.00×	0.0145	`torch-compile` `torch`	0.0145 0.0462	0.888
`cnn-feat-broadcast-float16`	1.00×	0.0142	`torch-compile` `torch`	0.0142 0.0455	0.902
`cnn-feat-broadcast-float32`	1.00×	0.0265	`torch-compile` `torch`	0.0266 0.0493	0.485
`hidden-state-prefill-bfloat16`	1.00×	0.0147	`torch` `torch-compile`	0.0147 0.0147	0.572
`hidden-state-prefill-float16`	1.00×	0.0148	`torch-compile` `torch`	0.0149 0.0149	0.565
`hidden-state-prefill-float32`	0.99×	0.0265	`torch-compile` `torch`	0.0263 0.0263	0.317
`mul-1024x10240-float16-float16-MulFwdOp-mul-normal`	1.00×	0.0176	`torch-compile` `torch`	0.0176 0.0178	0.596
`mul-1024x11008-float16-float16-MulFwdOp-mul-normal`	1.00×	0.0185	`torch` `torch-compile`	0.0186 0.0186	0.608
`mul-1024x4096-float16-float16-MulFwdOp-mul-normal`	1.00×	0.00842	`torch-compile` `torch`	0.00842 0.00858	0.498

TruncFwd (5 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bfloat16`	1.00×	0.0179	`torch-compile` `torch`	0.0179 0.0180	0.938
`elementwise-16M-float16`	1.00×	0.0179	`torch-compile` `torch`	0.0179 0.0180	0.938
`elementwise-16M-float32`	1.00×	0.0340	`torch-compile` `torch`	0.0339 0.0340	0.493
`elementwise-256M-bfloat16`	1.00×	0.2505	`torch` `torch-compile`	0.2503 0.2504	1.07
`elementwise-256M-float16`	1.00×	0.2498	`torch-compile` `torch`	0.2498 0.2502	1.07

NegFwd (5 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bfloat16`	1.00×	0.0179	`torch-compile` `torch`	0.0179 0.0180	0.94
`elementwise-16M-float16`	1.00×	0.0179	`torch-compile` `torch`	0.0179 0.0188	0.938
`elementwise-16M-float32`	1.00×	0.0340	`torch-compile` `torch`	0.0340 0.0340	0.493
`elementwise-256M-bfloat16`	1.00×	0.2501	`torch-compile` `torch`	0.2495 0.2497	1.07
`elementwise-256M-float16`	1.00×	0.2498	`torch-compile` `torch`	0.2497 0.2685	1.07

RoundFwd (5 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bfloat16`	1.00×	0.0179	`torch-compile` `torch`	0.0179 0.0179	0.939
`elementwise-16M-float16`	1.00×	0.0179	`torch-compile` `torch`	0.0180 0.0180	0.936
`elementwise-16M-float32`	1.00×	0.0340	`torch` `torch-compile`	0.0340 0.0340	0.494
`elementwise-256M-bfloat16`	1.00×	0.2498	`torch-compile` `torch`	0.2499 0.2503	1.07
`elementwise-256M-float16`	1.00×	0.2506	`torch-compile` `torch`	0.2499 0.2503	1.07

AbsFwd (5 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bfloat16`	1.00×	0.0179	`torch-compile` `torch`	0.0179 0.0179	0.938
`elementwise-16M-float16`	1.00×	0.0179	`torch-compile` `torch`	0.0179 0.0180	0.938
`elementwise-16M-float32`	1.00×	0.0340	`torch-compile` `torch`	0.0339 0.0339	0.493
`elementwise-256M-bfloat16`	1.00×	0.2499	`torch-compile` `torch`	0.2495 0.2498	1.07
`elementwise-256M-float16`	1.00×	0.2498	`torch` `torch-compile`	0.2499 0.2500	1.07

LerpFwd (8 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`cnn-feat-broadcast-bfloat16`	1.00×	0.0143	`torch-compile` `torch`	0.0143 0.0483	2.69
`cnn-feat-broadcast-float16`	1.00×	0.0144	`torch-compile` `torch`	0.0144 0.0477	2.67
`cnn-feat-broadcast-float32`	1.00×	0.0266	`torch-compile` `torch`	0.0265 0.0511	1.45
`hidden-state-prefill-bfloat16`	1.00×	0.0146	`torch-compile` `torch`	0.0146 0.0147	1.72
`hidden-state-prefill-float16`	1.00×	0.0148	`torch-compile` `torch`	0.0148 0.0148	1.71
`hidden-state-prefill-float32`	0.99×	0.0263	`torch-compile` `torch`	0.0262 0.0262	0.956
`lerp-1024x10240-float16-float16-LerpFwdOp-<lambda>-normal`	1.00×	0.0177	`torch-compile` `torch`	0.0177 0.0177	0.593
`lerp-1024x4096-float16-float16-LerpFwdOp-<lambda>-normal`	1.00×	0.00816	`torch-compile` `torch`	0.00816 0.00824	0.514

PowFwd (8 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`cnn-feat-broadcast-bfloat16`	1.14×	0.0542	`torch-compile` `torch`	0.0616 0.0955	0.711
`cnn-feat-broadcast-float16`	1.13×	0.0542	`torch-compile` `torch`	0.0612 0.0941	0.711
`cnn-feat-broadcast-float32`	1.02×	0.0573	`torch-compile` `torch`	0.0587 0.0939	0.672
`hidden-state-prefill-bfloat16`	1.01×	0.0377	`torch` `torch-compile`	0.0380 0.0451	0.667
`hidden-state-prefill-float16`	1.00×	0.0369	`torch` `torch-compile`	0.0370 0.0437	0.681
`hidden-state-prefill-float32`	0.96×	0.0388	`torch` `torch-compile`	0.0375 0.0421	0.648
`pow-1024x10240-float16-float16-PowFwdOp-pow-positive`	1.00×	0.0452	`torch` `torch-compile`	0.0453 0.0541	0.232
`pow-1024x4096-float16-float16-PowFwdOp-pow-positive`	1.00×	0.0201	`torch` `torch-compile`	0.0202 0.0252	0.209

MaskedFillFwd (5 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bfloat16`	1.00×	0.0226	`torch-compile` `torch`	0.0225 0.0401	0.742
`elementwise-16M-float16`	0.99×	0.0227	`torch-compile` `torch`	0.0226 0.0402	0.738
`elementwise-16M-float32`	0.99×	0.0380	`torch-compile` `torch`	0.0375 0.0732	0.441
`elementwise-256M-bfloat16`	1.00×	0.3103	`torch-compile` `torch`	0.3100 0.5709	0.865
`elementwise-256M-float16`	1.00×	0.3093	`torch-compile` `torch`	0.3098 0.5701	0.868

LerpTensorFwd (5 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bfloat16`	0.99×	0.0350	`torch-compile` `torch`	0.0348 0.0348	1.44
`elementwise-16M-float16`	0.99×	0.0350	`torch-compile` `torch`	0.0347 0.0348	1.44
`elementwise-16M-float32`	0.99×	0.0655	`torch` `torch-compile`	0.0651 0.0653	0.769
`elementwise-256M-bfloat16`	1.00×	0.4860	`torch` `torch-compile`	0.4860 0.4860	1.66
`elementwise-256M-float16`	1.00×	0.4858	`torch-compile` `torch`	0.4858 0.4860	1.66

MaskedFillScalarFwd (5 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bfloat16`	1.00×	0.0225	`torch-compile` `torch`	0.0225 0.0377	0.746
`elementwise-16M-float16`	0.99×	0.0228	`torch-compile` `torch`	0.0226 0.0375	0.737
`elementwise-16M-float32`	0.99×	0.0379	`torch-compile` `torch`	0.0373 0.0710	0.443
`elementwise-256M-bfloat16`	1.00×	0.3104	`torch-compile` `torch`	0.3100 0.5679	0.865
`elementwise-256M-float16`	1.00×	0.3101	`torch-compile` `torch`	0.3095 0.5673	0.866

WhereFwd (5 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bfloat16`	0.99×	0.0311	`torch-compile` `torch`	0.0307 0.0308	0.539
`elementwise-16M-float16`	0.99×	0.0309	`torch-compile` `torch`	0.0307 0.0307	0.542
`elementwise-16M-float32`	1.00×	0.0533	`torch-compile` `torch`	0.0531 0.0535	0.315
`elementwise-256M-bfloat16`	1.00×	0.4286	`torch` `torch-compile`	0.4282 0.4283	0.626
`elementwise-256M-float16`	1.00×	0.4291	`torch-compile` `torch`	0.4282 0.4283	0.626

DivFwd (9 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`cnn-feat-broadcast-bfloat16`	1.00×	0.0144	`torch-compile` `torch`	0.0144 0.0513	0.89
`cnn-feat-broadcast-float16`	0.92×	0.0160	`torch-compile` `torch`	0.0148 0.0508	0.803
`cnn-feat-broadcast-float32`	1.00×	0.0265	`torch-compile` `torch`	0.0265 0.0531	0.485
`div-1024x10240-float16-float16-DivFwdOp-div-positive`	1.00×	0.0181	`torch-compile` `torch`	0.0181 0.0185	0.578
`div-1024x11008-float16-float16-DivFwdOp-div-positive`	0.99×	0.0189	`torch-compile` `torch`	0.0188 0.0193	0.596
`div-1024x4096-float16-float16-DivFwdOp-div-positive`	1.00×	0.00848	`torch-compile` `torch`	0.00845 0.00877	0.495
`hidden-state-prefill-bfloat16`	1.00×	0.0147	`torch-compile` `torch`	0.0147 0.0151	0.572
`hidden-state-prefill-float16`	0.99×	0.0148	`torch-compile` `torch`	0.0147 0.0151	0.567
`hidden-state-prefill-float32`	1.00×	0.0263	`torch-compile` `torch`	0.0262 0.0264	0.319

ClampFwd (15 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bfloat16`	0.98×	0.0355	`torch` `torch-compile`	0.0349 0.0351	0.473
`elementwise-16M-float16`	0.98×	0.0355	`torch` `torch-compile`	0.0349 0.0351	0.473
`elementwise-16M-float32`	0.99×	0.0658	`torch-compile` `torch`	0.0654 0.0656	0.255
`elementwise-16M-max-only-bfloat16`	0.98×	0.0271	`torch-compile` `torch`	0.0266 0.0269	0.62
`elementwise-16M-max-only-float16`	0.99×	0.0266	`torch-compile` `torch`	0.0263 0.0265	0.632
`elementwise-16M-max-only-float32`	0.98×	0.0500	`torch-compile` `torch`	0.0491 0.0494	0.336
`elementwise-16M-min-only-bfloat16`	0.98×	0.0269	`torch-compile` `torch`	0.0264 0.0268	0.623
`elementwise-16M-min-only-float16`	0.99×	0.0267	`torch-compile` `torch`	0.0264 0.0266	0.628
`elementwise-16M-min-only-float32`	0.98×	0.0500	`torch-compile` `torch`	0.0493 0.0495	0.335
`elementwise-256M-bfloat16`	1.00×	0.4855	`torch` `torch-compile`	0.4851 0.4851	0.553
`elementwise-256M-float16`	1.00×	0.4859	`torch-compile` `torch`	0.4849 0.4853	0.552
`elementwise-256M-max-only-bfloat16`	1.00×	0.3677	`torch` `torch-compile`	0.3676 0.3688	0.73
`elementwise-256M-max-only-float16`	1.00×	0.3688	`torch` `torch-compile`	0.3682 0.3688	0.728
`elementwise-256M-min-only-bfloat16`	1.00×	0.3684	`torch` `torch-compile`	0.3677 0.3687	0.729
`elementwise-256M-min-only-float16`	1.00×	0.3693	`torch` `torch-compile`	0.3684 0.3690	0.727

NanToNumFwd (5 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bfloat16`	0.98×	0.0189	`torch-compile` `torch`	0.0186 0.0192	5.32
`elementwise-16M-float16`	0.98×	0.0189	`torch-compile` `torch`	0.0186 0.0192	5.31
`elementwise-16M-float32`	1.00×	0.0340	`torch-compile` `torch`	0.0340 0.0340	2.96
`elementwise-256M-bfloat16`	0.98×	0.2639	`torch-compile` `torch`	0.2584 0.2736	6.1
`elementwise-256M-float16`	0.98×	0.2650	`torch-compile` `torch`	0.2587 0.2743	6.08

SiluAndMulFwd (3 workloads · ✅)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`llama-8b-swiglu-decode-bfloat16`	0.87×	0.00173	`torch-compile` `torch-ref` `flashinfer`	0.0015 0.00355 0.00429	0.0498
`llama-8b-swiglu-prefill-bfloat16`	1.06×	0.0433	`torch-compile` `flashinfer` `torch-ref`	0.0457 0.0540 0.1902	4.07
`llama-8b-swiglu-prefill-float16`	1.02×	0.0434	`torch-compile` `flashinfer` `torch-ref`	0.0442 0.0536 0.1893	4.06

ReciprocalFwd (5 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bfloat16`	0.96×	0.0189	`torch-compile` `torch`	0.0183 0.0190	0.886
`elementwise-16M-float16`	0.96×	0.0189	`torch-compile` `torch`	0.0182 0.0189	0.887
`elementwise-16M-float32`	1.01×	0.0335	`torch-compile` `torch`	0.0338 0.0339	0.501
`elementwise-256M-bfloat16`	0.97×	0.2673	`torch-compile` `torch`	0.2580 0.2674	1
`elementwise-256M-float16`	0.96×	0.2671	`torch-compile` `torch`	0.2561 0.2672	1.01

SignFwd (5 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bfloat16`	0.97×	0.0186	`torch-compile` `torch`	0.0180 0.0182	1.8
`elementwise-16M-float16`	0.97×	0.0186	`torch-compile` `torch`	0.0180 0.0181	1.8
`elementwise-16M-float32`	1.00×	0.0341	`torch-compile` `torch`	0.0340 0.0340	0.985
`elementwise-256M-bfloat16`	0.96×	0.2635	`torch-compile` `torch`	0.2541 0.2586	2.04
`elementwise-256M-float16`	0.96×	0.2636	`torch-compile` `torch`	0.2521 0.2560	2.04

SiluFwd (3 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`llama-8b-ffn-decode-bfloat16`	0.96×	0.0015	`torch-compile` `torch`	0.00144 0.00195	0.0477
`llama-8b-ffn-prefill-bfloat16`	0.98×	0.0365	`torch-compile` `torch`	0.0358 0.0381	4.02
`llama-8b-ffn-prefill-float16`	0.97×	0.0366	`torch-compile` `torch`	0.0355 0.0379	4.01

LogicalAndFwd (10 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`cnn-feat-broadcast-bfloat16`	0.75×	0.0159	`torch-compile` `torch`	0.0120 0.0480	2.42
`cnn-feat-broadcast-bool`	1.23×	0.00806	`torch-compile` `torch`	0.00995 0.0452	4.78
`cnn-feat-broadcast-float16`	0.76×	0.0159	`torch-compile` `torch`	0.0121 0.0469	2.42
`cnn-feat-broadcast-float32`	0.88×	0.0208	`torch-compile` `torch`	0.0183 0.0467	1.86
`hidden-state-prefill-bfloat16`	1.00×	0.0130	`torch-compile` `torch`	0.0130 0.0131	1.94
`hidden-state-prefill-bool`	1.07×	0.00826	`torch-compile` `torch`	0.00883 0.0101	3.05
`hidden-state-prefill-float16`	1.01×	0.0132	`torch-compile` `torch`	0.0133 0.0134	1.91
`hidden-state-prefill-float32`	1.00×	0.0225	`torch` `torch-compile`	0.0225 0.0251	1.12
`logical_and-1024x10240-float16-LogicalAndFwdOp-logical_and`	1.02×	0.0157	`torch` `torch-compile`	0.0160 0.0160	0.666
`logical_and-1024x4096-float16-LogicalAndFwdOp-logical_and`	1.03×	0.00749	`torch-compile` `torch`	0.00768 0.00771	0.56

LogicalOrFwd (10 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`cnn-feat-broadcast-bfloat16`	0.74×	0.0164	`torch-compile` `torch`	0.0122 0.0488	2.36
`cnn-feat-broadcast-bool`	1.27×	0.00806	`torch-compile` `torch`	0.0102 0.0442	4.78
`cnn-feat-broadcast-float16`	0.73×	0.0164	`torch-compile` `torch`	0.0120 0.0477	2.36
`cnn-feat-broadcast-float32`	0.89×	0.0209	`torch-compile` `torch`	0.0185 0.0465	1.84
`hidden-state-prefill-bfloat16`	1.01×	0.0129	`torch-compile` `torch`	0.0130 0.0130	1.96
`hidden-state-prefill-bool`	1.08×	0.00819	`torch-compile` `torch`	0.00886 0.00906	3.07
`hidden-state-prefill-float16`	1.01×	0.0132	`torch-compile` `torch`	0.0133 0.0133	1.91
`hidden-state-prefill-float32`	1.00×	0.0225	`torch-compile` `torch`	0.0224 0.0224	1.12
`logical_or-1024x10240-float16-LogicalOrFwdOp-logical_or`	1.01×	0.0158	`torch` `torch-compile`	0.0161 0.0161	0.662
`logical_or-1024x4096-float16-LogicalOrFwdOp-logical_or`	1.02×	0.00749	`torch` `torch-compile`	0.00765 0.0102	0.56

ErfFwd (5 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bfloat16`	0.97×	0.0285	`torch` `torch-compile`	0.0277 0.0296	0.588
`elementwise-16M-float16`	0.93×	0.0284	`torch` `torch-compile`	0.0265 0.0292	0.59
`elementwise-16M-float32`	0.98×	0.0350	`torch` `torch-compile`	0.0343 0.0346	0.479
`elementwise-256M-bfloat16`	0.96×	0.4236	`torch` `torch-compile`	0.4065 0.4372	0.634
`elementwise-256M-float16`	0.92×	0.4215	`torch` `torch-compile`	0.3862 0.4315	0.637

div_bcast (3 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`div-1024x10240-1x10240-float16-DivFwdOp-div-positive`	0.92×	0.0133	`torch-compile` `torch`	0.0123 0.0363	0.786
`div-1024x11008-1x11008-float16-DivFwdOp-div-positive`	0.93×	0.0141	`torch-compile` `torch`	0.0131 0.0388	0.797
`div-1024x4096-1x4096-float16-DivFwdOp-div-positive`	0.96×	0.00646	`torch-compile` `torch`	0.00621 0.0163	0.649

LeFwd (7 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`cnn-feat-broadcast-bfloat16`	0.74×	0.0159	`torch-compile` `torch`	0.0119 0.0505	0.806
`cnn-feat-broadcast-float16`	0.75×	0.0159	`torch-compile` `torch`	0.0119 0.0496	0.806
`cnn-feat-broadcast-float32`	0.89×	0.0209	`torch-compile` `torch`	0.0185 0.0492	0.615
`hidden-state-prefill-bfloat16`	1.00×	0.0131	`torch-compile` `torch`	0.0131 0.0132	0.641
`hidden-state-prefill-float16`	0.99×	0.0130	`torch-compile` `torch`	0.0129 0.0129	0.644
`hidden-state-prefill-float32`	1.00×	0.0225	`torch-compile` `torch`	0.0224 0.0225	0.373
`le-1024x4096-float16-le`	1.01×	0.00762	`torch-compile` `torch`	0.00771 0.00771	0.551

GeluFwd (3 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`llama-8b-ffn-decode-bfloat16`	1.02×	0.00138	`torch-compile` `torch`	0.00141 0.00166	0.0521
`llama-8b-ffn-prefill-bfloat16`	0.89×	0.0553	`torch` `torch-compile`	0.0490 0.0557	2.66
`llama-8b-ffn-prefill-float16`	0.90×	0.0527	`torch` `torch-compile`	0.0477 0.0540	2.78

HardswishFwd (4 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`mbv3-stage2-bfloat16`	0.87×	0.0132	`torch-compile` `torch`	0.0115 0.0116	2.92
`mbv3-stage2-float16`	0.89×	0.0129	`torch-compile` `torch`	0.0115 0.0116	2.99
`mbv3-stage3-bfloat16`	0.89×	0.00902	`torch-compile` `torch`	0.00806 0.00816	2.67
`mbv3-stage3-float16`	0.91×	0.00886	`torch-compile` `torch`	0.00806 0.00813	2.72

SigmoidFwd (5 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`elementwise-16M-bfloat16`	0.86×	0.0218	`torch-compile` `torch`	0.0187 0.0235	3.08
`elementwise-16M-float16`	0.87×	0.0214	`torch-compile` `torch`	0.0186 0.0229	3.13
`elementwise-16M-float32`	0.99×	0.0344	`torch-compile` `torch`	0.0340 0.0344	1.95
`elementwise-256M-bfloat16`	0.85×	0.3084	`torch-compile` `torch`	0.2634 0.3310	3.48
`elementwise-256M-float16`	0.86×	0.3018	`torch-compile` `torch`	0.2604 0.3216	3.56

NeFwd (7 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`cnn-feat-broadcast-bfloat16`	0.74×	0.0159	`torch-compile` `torch`	0.0118 0.0488	0.809
`cnn-feat-broadcast-float16`	0.75×	0.0159	`torch-compile` `torch`	0.0119 0.0476	0.809
`cnn-feat-broadcast-float32`	0.88×	0.0208	`torch-compile` `torch`	0.0184 0.0477	0.617
`hidden-state-prefill-bfloat16`	1.01×	0.0132	`torch-compile` `torch`	0.0134 0.0134	0.635
`hidden-state-prefill-float16`	1.01×	0.0130	`torch-compile` `torch`	0.0131 0.0132	0.645
`hidden-state-prefill-float32`	1.00×	0.0223	`torch` `torch-compile`	0.0223 0.0249	0.376
`ne-1024x4096-float16-ne`	1.02×	0.00765	`torch` `torch-compile`	0.00781 0.00781	0.548

EqFwd (8 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`cnn-feat-broadcast-bfloat16`	0.75×	0.0159	`torch-compile` `torch`	0.0119 0.0488	0.806
`cnn-feat-broadcast-float16`	0.75×	0.0159	`torch-compile` `torch`	0.0119 0.0476	0.809
`cnn-feat-broadcast-float32`	0.88×	0.0210	`torch-compile` `torch`	0.0184 0.0480	0.613
`eq-1024x10240-float16-eq`	1.01×	0.0158	`torch` `torch-compile`	0.0160 0.0160	0.663
`eq-1024x4096-float16-eq`	1.03×	0.00758	`torch` `torch-compile`	0.00784 0.00784	0.553
`hidden-state-prefill-bfloat16`	1.00×	0.0131	`torch` `torch-compile`	0.0131 0.0131	0.639
`hidden-state-prefill-float16`	1.02×	0.0133	`torch-compile` `torch`	0.0135 0.0136	0.632
`hidden-state-prefill-float32`	1.00×	0.0224	`torch-compile` `torch`	0.0223 0.0223	0.375

GtFwd (7 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`cnn-feat-broadcast-bfloat16`	0.75×	0.0159	`torch-compile` `torch`	0.0120 0.0490	0.808
`cnn-feat-broadcast-float16`	0.75×	0.0160	`torch-compile` `torch`	0.0119 0.0483	0.804
`cnn-feat-broadcast-float32`	0.88×	0.0209	`torch-compile` `torch`	0.0183 0.0482	0.615
`gt-1024x4096-float16-gt`	1.02×	0.00762	`torch-compile` `torch`	0.00778 0.00781	0.551
`hidden-state-prefill-bfloat16`	1.01×	0.0130	`torch` `torch-compile`	0.0131 0.0132	0.644
`hidden-state-prefill-float16`	1.01×	0.0130	`torch` `torch-compile`	0.0132 0.0132	0.646
`hidden-state-prefill-float32`	1.00×	0.0223	`torch` `torch-compile`	0.0223 0.0250	0.376

LtFwd (7 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`cnn-feat-broadcast-bfloat16`	0.77×	0.0159	`torch-compile` `torch`	0.0122 0.0505	0.808
`cnn-feat-broadcast-float16`	0.74×	0.0159	`torch-compile` `torch`	0.0118 0.0496	0.806
`cnn-feat-broadcast-float32`	0.88×	0.0209	`torch-compile` `torch`	0.0184 0.0491	0.615
`hidden-state-prefill-bfloat16`	1.01×	0.0132	`torch-compile` `torch`	0.0133 0.0133	0.636
`hidden-state-prefill-float16`	1.01×	0.0130	`torch` `torch-compile`	0.0131 0.0132	0.646
`hidden-state-prefill-float32`	1.00×	0.0225	`torch` `torch-compile`	0.0225 0.0247	0.372
`lt-1024x4096-float16-lt`	1.02×	0.00755	`torch` `torch-compile`	0.00771 0.00775	0.555

GeFwd (7 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`cnn-feat-broadcast-bfloat16`	0.76×	0.0159	`torch-compile` `torch`	0.0120 0.0476	0.809
`cnn-feat-broadcast-float16`	0.74×	0.0159	`torch-compile` `torch`	0.0118 0.0469	0.808
`cnn-feat-broadcast-float32`	0.89×	0.0207	`torch-compile` `torch`	0.0184 0.0469	0.619
`ge-1024x4096-float16-ge`	1.03×	0.00758	`torch` `torch-compile`	0.00778 0.00778	0.553
`hidden-state-prefill-bfloat16`	1.00×	0.0130	`torch-compile` `torch`	0.0131 0.0131	0.644
`hidden-state-prefill-float16`	1.00×	0.0132	`torch` `torch-compile`	0.0132 0.0132	0.637
`hidden-state-prefill-float32`	1.00×	0.0224	`torch` `torch-compile`	0.0224 0.0249	0.374

HardsigmoidFwd (4 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`mbv3-se-gate-bfloat16`	0.82×	0.00163	`torch-compile` `torch`	0.00134 0.00173	0.0141
`mbv3-se-gate-deep-bfloat16`	0.88×	0.00167	`torch` `torch-compile`	0.00147 0.0015	0.0554
`mbv3-se-gate-deep-float16`	0.88×	0.00167	`torch` `torch-compile`	0.00147 0.00147	0.0554
`mbv3-se-gate-float16`	0.82×	0.00163	`torch-compile` `torch`	0.00135 0.00173	0.0141

MinimumFwd (9 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`cnn-feat-broadcast-bfloat16`	0.37×	0.0392	`torch-compile` `torch`	0.0144 0.0514	0.327
`cnn-feat-broadcast-float16`	0.38×	0.0374	`torch-compile` `torch`	0.0144 0.0504	0.344
`cnn-feat-broadcast-float32`	0.90×	0.0294	`torch-compile` `torch`	0.0264 0.0532	0.436
`hidden-state-prefill-bfloat16`	0.99×	0.0150	`torch-compile` `torch`	0.0148 0.0151	0.559
`hidden-state-prefill-float16`	0.99×	0.0149	`torch-compile` `torch`	0.0148 0.0150	0.562
`hidden-state-prefill-float32`	0.99×	0.0265	`torch-compile` `torch`	0.0263 0.0265	0.317
`minimum-1024x10240-float16-float16-MinimumFwdOp-minimum-normal`	0.99×	0.0181	`torch-compile` `torch`	0.0179 0.0182	0.58
`minimum-1024x11008-float16-float16-MinimumFwdOp-minimum-normal`	0.99×	0.0189	`torch-compile` `torch`	0.0188 0.0190	0.595
`minimum-1024x4096-float16-float16-MinimumFwdOp-minimum-normal`	0.97×	0.00861	`torch-compile` `torch`	0.00835 0.00874	0.487

MaximumFwd (9 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`cnn-feat-broadcast-bfloat16`	0.36×	0.0392	`torch-compile` `torch`	0.0142 0.0514	0.327
`cnn-feat-broadcast-float16`	0.38×	0.0374	`torch-compile` `torch`	0.0143 0.0505	0.343
`cnn-feat-broadcast-float32`	0.90×	0.0293	`torch-compile` `torch`	0.0264 0.0531	0.438
`hidden-state-prefill-bfloat16`	0.98×	0.0149	`torch-compile` `torch`	0.0147 0.0150	0.561
`hidden-state-prefill-float16`	0.98×	0.0147	`torch-compile` `torch`	0.0145 0.0148	0.571
`hidden-state-prefill-float32`	1.00×	0.0263	`torch-compile` `torch`	0.0263 0.0265	0.319
`maximum-1024x10240-float16-float16-MaximumFwdOp-maximum-normal`	0.99×	0.0180	`torch-compile` `torch`	0.0179 0.0182	0.581
`maximum-1024x11008-float16-float16-MaximumFwdOp-maximum-normal`	0.99×	0.0189	`torch-compile` `torch`	0.0187 0.0190	0.597
`maximum-1024x4096-float16-float16-MaximumFwdOp-maximum-normal`	0.98×	0.00858	`torch-compile` `torch`	0.00838 0.00867	0.489

gelu_and_mul_strategy (2 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`gelu_and_mul-1024-4096-float16-GeluAndMulFwdKernel-direct`	0.49×	0.0216	`torch-compile` `torch`	0.0107 0.0334	0.388
`gelu_and_mul-1024-4096-float16-GeluAndMulFwdKernel-explicit_parallel`	1.06×	0.0100	`torch-compile` `torch`	0.0107 0.0334	0.837

gelu_tanh_and_mul_strategy (2 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`gelu_tanh_and_mul-1024-4096-float16-GeluTanhAndMulFwdKernel-direct`	0.45×	0.0216	`torch-compile` `torch`	0.00976 0.0319	0.388
`gelu_tanh_and_mul-1024-4096-float16-GeluTanhAndMulFwdKernel-explicit_parallel`	1.11×	0.00883	`torch-compile` `torch`	0.00979 0.0319	0.95

silu_and_mul_strategy (10 workloads)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`silu_and_mul-1024-11008-float16-SiluAndMulFwdKernel-direct`	0.34×	0.0555	`torch-compile` `torch`	0.0190 0.0769	0.406
`silu_and_mul-1024-11008-float16-SiluAndMulFwdKernel-explicit_parallel`	1.01×	0.0189	`torch-compile` `torch`	0.0190 0.0769	1.19
`silu_and_mul-1024-4096-bfloat16-SiluAndMulFwdKernel-direct`	0.39×	0.0216	`torch-compile` `torch`	0.00842 0.0317	0.388
`silu_and_mul-1024-4096-bfloat16-SiluAndMulFwdKernel-explicit_parallel`	1.00×	0.00845	`torch-compile` `torch`	0.00842 0.0317	0.993
`silu_and_mul-1024-4096-float16-SiluAndMulFwdKernel-direct`	0.39×	0.0216	`torch-compile` `torch`	0.00835 0.0316	0.388
`silu_and_mul-1024-4096-float16-SiluAndMulFwdKernel-explicit_parallel`	0.99×	0.00842	`torch-compile` `torch`	0.00832 0.0316	0.997
`silu_and_mul-1024-4096-float32-SiluAndMulFwdKernel-direct`	0.67×	0.0218	`torch-compile` `torch`	0.0146 0.0347	0.384
`silu_and_mul-1024-4096-float32-SiluAndMulFwdKernel-explicit_parallel`	0.98×	0.0148	`torch-compile` `torch`	0.0146 0.0348	0.566
`silu_and_mul-4096-4096-float16-SiluAndMulFwdKernel-direct`	0.33×	0.0822	`torch-compile` `torch`	0.0269 0.1112	0.408
`silu_and_mul-4096-4096-float16-SiluAndMulFwdKernel-explicit_parallel`	1.01×	0.0266	`torch-compile` `torch`	0.0269 0.1113	1.26

Reduction¶

ArgminFwd (4 workloads · ✅)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`hidden-state-argmin-bfloat16`	1.00×	0.00986	`flaggems` `torch-compile` `torch`	0.00986 0.0194 0.0248	0.851
`hidden-state-argmin-float16`	1.03×	0.00963	`flaggems` `torch-compile` `torch`	0.00992 0.0190 0.0246	0.871
`lm-head-argmin-bfloat16`	4.38×	0.00664	`torch-compile` `torch` `flaggems`	0.0291 0.0359 0.1027	0.0617
`lm-head-argmin-float16`	7.15×	0.004	`torch-compile` `torch` `flaggems`	0.0286 0.0354 0.1156	0.102

CumprodFwd (3 workloads · ✅)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`hidden-state-scan-bfloat16`	2.23×	0.0109	`torch-compile` `torch`	0.0244 0.1462	0.769
`hidden-state-scan-float16`	2.11×	0.0115	`torch-compile` `torch`	0.0242 0.1458	0.73
`long-seq-scan-bfloat16`	1.73×	0.00704	`torch-compile` `torch`	0.0122 0.0677	0.298

ArgmaxFwd (5 workloads · ✅)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`3d-non-last-axis-argmax-float16`	0.98×	0.0039	`flaggems` `torch-compile` `torch`	0.00381 0.00458 0.0112	0.537
`hidden-state-argmax-bfloat16`	1.21×	0.00963	`flaggems` `torch-compile` `torch`	0.0116 0.0194 0.0249	0.871
`hidden-state-argmax-float16`	1.33×	0.00915	`flaggems` `torch-compile` `torch`	0.0122 0.0190 0.0246	0.916
`lm-head-argmax-bfloat16`	7.16×	0.00397	`flaggems` `torch-compile` `torch`	0.0284 0.0291 0.0359	0.103
`lm-head-argmax-float16`	7.41×	0.00387	`torch-compile` `flaggems` `torch`	0.0287 0.0300 0.0352	0.106

VarMeanFwd (4 workloads · ✅)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`3d-multidim-reduce-float16`	0.63×	0.0121	`torch-compile` `flaggems` `torch`	0.00758 0.0142 0.0452	0.865
`hidden-state-var-mean-bfloat16`	1.84×	0.00842	`flaggems` `torch-compile` `torch`	0.0155 0.0219 0.1171	4.98
`hidden-state-var-mean-float16`	1.80×	0.00838	`flaggems` `torch-compile` `torch`	0.0151 0.0210 0.1166	5
`long-seq-var-mean-bfloat16`	1.47×	0.00515	`torch-compile` `flaggems` `torch`	0.00758 0.0111 0.0406	2.04

VarFwd (4 workloads · ✅)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`3d-multidim-reduce-float16`	0.50×	0.0120	`torch-compile` `flaggems` `torch`	0.00605 0.0141 0.0268	0.874
`hidden-state-var-bfloat16`	1.84×	0.00837	`flaggems` `torch-compile` `torch`	0.0154 0.0188 0.0680	5.01
`hidden-state-var-float16`	1.80×	0.00835	`flaggems` `torch-compile` `torch`	0.0150 0.0183 0.0677	5.02
`long-seq-var-bfloat16`	1.19×	0.00515	`torch-compile` `flaggems` `torch`	0.00611 0.0111 0.0249	2.04

StdFwd (4 workloads · ✅)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`3d-multidim-reduce-float16`	0.52×	0.0121	`torch-compile` `flaggems` `torch`	0.0063 0.0143 0.0269	0.869
`hidden-state-std-bfloat16`	1.30×	0.00851	`flaggems` `torch-compile` `torch`	0.0111 0.0193 0.0680	4.93
`hidden-state-std-float16`	1.25×	0.00845	`flaggems` `torch-compile` `torch`	0.0106 0.0188 0.0678	4.97
`long-seq-std-bfloat16`	1.19×	0.00518	`torch-compile` `flaggems` `torch`	0.00618 0.0132 0.0249	2.02

L2NormFwd (4 workloads · ✅)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`3d-multidim-reduce-float16`	0.41×	0.0113	`torch-compile` `flaggems` `torch`	0.00464 0.0135 0.0193	0.37
`hidden-state-l2-bfloat16`	1.06×	0.00742	`flaggems` `torch-compile` `torch`	0.00787 0.0088 0.0498	2.26
`hidden-state-l2-float16`	1.06×	0.00746	`flaggems` `torch-compile` `torch`	0.00787 0.00867 0.0496	2.25
`long-seq-l2-bfloat16`	1.12×	0.004	`torch-compile` `flaggems` `torch`	0.00448 0.0135 0.0169	1.05

InfNormFwd (4 workloads · ✅)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`3d-multidim-reduce-float16`	0.44×	0.0112	`torch-compile` `flaggems` `torch`	0.00493 0.0127 0.0192	0.373
`hidden-state-inf-bfloat16`	1.07×	0.00736	`flaggems` `torch-compile` `torch`	0.00784 0.0111 0.0502	2.28
`hidden-state-inf-float16`	1.04×	0.00733	`flaggems` `torch-compile` `torch`	0.00765 0.0106 0.0499	2.29
`long-seq-inf-bfloat16`	1.24×	0.00403	`torch-compile` `flaggems` `torch`	0.00499 0.0137 0.0173	1.04

AmaxFwd (4 workloads · ✅)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`3d-multidim-reduce-float16`	0.44×	0.0113	`torch-compile` `flaggems` `torch`	0.00496 0.0124 0.0129	0.186
`hidden-state-reduce-bfloat16`	1.05×	0.00742	`flaggems` `torch-compile` `torch`	0.00782 0.0101 0.0193	1.13
`hidden-state-reduce-float16`	1.03×	0.00742	`flaggems` `torch-compile` `torch`	0.00768 0.0101 0.0192	1.13
`long-seq-reduce-bfloat16`	1.20×	0.0041	`torch-compile` `torch` `flaggems`	0.00493 0.0113 0.0135	0.512

SoftmaxFwd (7 workloads · ✅)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`attn-weights-32k-bfloat16`	1.05×	0.0615	`flaggems` `torch` `torch-compile`	0.0644 0.0834 0.0943	2.73
`attn-weights-4k-bfloat16`	1.03×	0.00838	`flaggems` `torch-compile` `torch`	0.00867 0.0167 0.0196	2.5
`attn-weights-4k-float16`	1.02×	0.00842	`flaggems` `torch-compile` `torch`	0.00861 0.0161 0.0198	2.49
`attn-weights-4k-float32`	1.00×	0.0110	`flaggems` `torch-compile` `torch`	0.0111 0.0184 0.0202	1.9
`lm-head-logits-bfloat16`	0.31×	0.0308	`torch-compile` `flaggems` `torch`	0.00954 0.0297 0.0340	0.0665
`lm-head-logits-float16`	0.33×	0.0284	`torch-compile` `flaggems` `torch`	0.0095 0.0281 0.0332	0.0721
`lm-head-logits-float32`	0.28×	0.0348	`torch-compile` `flaggems` `torch`	0.00979 0.0313 0.0394	0.0589

ProdFwd (3 workloads · ✅)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`hidden-state-reduce-bfloat16`	1.04×	0.00747	`flaggems` `torch-compile` `torch`	0.00781 0.00835 0.0496	1.12
`hidden-state-reduce-float16`	1.00×	0.00787	`flaggems` `torch-compile` `torch`	0.00784 0.00829 0.0492	1.07
`long-seq-reduce-bfloat16`	1.05×	0.00429	`torch-compile` `flaggems` `torch`	0.00451 0.0136 0.0168	0.489

CumsumFwd (3 workloads · ✅)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`hidden-state-scan-bfloat16`	0.97×	0.0109	`flaggems` `torch-compile` `torch`	0.0106 0.0244 0.1462	0.769
`hidden-state-scan-float16`	0.92×	0.0115	`flaggems` `torch-compile` `torch`	0.0106 0.0242 0.1459	0.73
`long-seq-scan-bfloat16`	1.14×	0.00704	`flaggems` `torch-compile` `torch`	0.008 0.0122 0.0677	0.298

AminFwd (4 workloads · ✅)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`3d-multidim-reduce-float16`	0.44×	0.0113	`torch-compile` `torch`	0.00496 0.0129	0.186
`hidden-state-reduce-bfloat16`	1.36×	0.00742	`torch-compile` `torch`	0.0101 0.0193	1.13
`hidden-state-reduce-float16`	1.37×	0.00742	`torch-compile` `torch`	0.0101 0.0192	1.13
`long-seq-reduce-bfloat16`	1.14×	0.0041	`torch-compile` `torch`	0.00467 0.0113	0.512

LogSumExpFwd (6 workloads · ✅)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`3d-multidim-reduce-float16`	0.81×	0.0125	`torch-compile` `torch`	0.0102 0.0412	0.669
`attn-weights-32k-bfloat16`	1.01×	0.0411	`torch-compile` `torch`	0.0415 0.1979	3.27
`attn-weights-4k-bfloat16`	1.34×	0.00746	`torch-compile` `torch`	0.0100 0.0494	2.25
`attn-weights-4k-float16`	1.36×	0.00742	`torch-compile` `torch`	0.0101 0.0488	2.26
`lm-head-logits-bfloat16`	0.62×	0.0165	`torch-compile` `torch`	0.0103 0.0463	0.0994
`lm-head-logits-float16`	0.75×	0.0141	`torch-compile` `torch`	0.0106 0.0463	0.116

L1NormFwd (4 workloads · ✅)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`3d-multidim-reduce-float16`	0.41×	0.0113	`torch-compile` `torch` `flaggems`	0.00464 0.0193 0.0246	0.372
`hidden-state-l1-bfloat16`	1.15×	0.00736	`torch-compile` `flaggems` `torch`	0.00845 0.0154 0.0497	2.28
`hidden-state-l1-float16`	1.13×	0.00736	`torch-compile` `flaggems` `torch`	0.00835 0.0150 0.0494	2.28
`long-seq-l1-bfloat16`	1.16×	0.0039	`torch-compile` `torch` `flaggems`	0.00451 0.0169 0.0368	1.07

SumFwd (6 workloads · ✅)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`3d-multidim-reduce-float16`	0.41×	0.0113	`torch-compile` `flaggems` `torch`	0.00458 0.0135 0.0193	0.186
`hidden-state-reduce-bfloat16`	1.13×	0.00742	`torch-compile` `flaggems` `torch`	0.00842 0.0088 0.0498	1.13
`hidden-state-reduce-dim0-bfloat16`	0.92×	0.0121	`torch-compile` `flaggems` `torch`	0.0111 0.0137 0.0447	0.692
`hidden-state-reduce-float16`	1.12×	0.00742	`torch-compile` `flaggems` `torch`	0.00835 0.00877 0.0494	1.13
`hidden-state-reduce-keepdim-bfloat16`	1.13×	0.00745	`torch-compile` `flaggems` `torch`	0.00842 0.0088 0.0499	1.13
`long-seq-reduce-bfloat16`	0.94×	0.00406	`flaggems` `torch-compile` `torch`	0.00381 0.00442 0.0169	0.516

CountNonzeroFwd (4 workloads · ✅)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`3d-multidim-reduce-float16`	0.41×	0.0116	`torch-compile` `torch`	0.00475 0.0218	0.363
`sparsity-hidden-bfloat16`	1.12×	0.0081	`torch-compile` `torch`	0.0091 0.0638	2.07
`sparsity-hidden-float16`	1.12×	0.00813	`torch-compile` `torch`	0.00909 0.0638	2.06
`sparsity-seq-float16`	1.03×	0.00371	`torch-compile` `torch`	0.00384 0.0152	0.565

MeanFwd (4 workloads · ✅)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`3d-multidim-reduce-float16`	0.41×	0.0113	`torch-compile` `flaggems` `torch`	0.00467 0.0135 0.0191	0.186
`hidden-state-reduce-bfloat16`	1.14×	0.00742	`torch-compile` `flaggems` `torch`	0.00845 0.00893 0.0499	1.13
`hidden-state-reduce-float16`	1.12×	0.00742	`torch-compile` `flaggems` `torch`	0.00835 0.0089 0.0495	1.13
`long-seq-reduce-bfloat16`	0.94×	0.00405	`flaggems` `torch-compile` `torch`	0.00381 0.00442 0.0169	0.518

LogSoftmaxFwd (7 workloads · ✅)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`attn-weights-32k-bfloat16`	1.09×	0.0567	`torch` `torch-compile` `flaggems`	0.0616 0.0717 0.2500	2.96
`attn-weights-4k-bfloat16`	1.75×	0.00883	`torch-compile` `torch` `flaggems`	0.0154 0.0172 0.0204	2.37
`attn-weights-4k-float16`	1.65×	0.00902	`torch-compile` `torch` `flaggems`	0.0149 0.0172 0.0199	2.32
`attn-weights-4k-float32`	1.37×	0.0115	`torch-compile` `torch` `flaggems`	0.0157 0.0185 0.0206	1.83
`lm-head-logits-bfloat16`	0.34×	0.0262	`torch-compile` `torch` `flaggems`	0.00901 0.0231 0.4227	0.0781
`lm-head-logits-float16`	0.36×	0.0249	`torch-compile` `torch` `flaggems`	0.00906 0.0224 0.4222	0.0824
`lm-head-logits-float32`	0.31×	0.0319	`torch-compile` `torch` `flaggems`	0.00982 0.0352 0.4035	0.0641

AllFwd (3 workloads · ✅)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`3d-multidim-reduce-bool`	0.38×	0.0111	`torch-compile` `torch` `flaggems`	0.00426 0.0103 0.0117	0.189
`mask-validation-32k-bool`	0.91×	0.00375	`torch-compile` `flaggems` `torch`	0.00339 0.00643 0.0102	0.28
`mask-validation-4k-bool`	0.92×	0.00195	`torch-compile` `flaggems` `torch`	0.00179 0.00189 0.0173	0.0671

AnyFwd (3 workloads · ✅)¶

Workload	Ratio	Device time	Alternatives		Throughput
Workload	alt / ours	ms	name	ms	TFLOP/s
`3d-multidim-reduce-bool`	0.34×	0.0111	`torch-compile` `flaggems` `torch`	0.00371 0.0117 0.0188	0.189
`mask-validation-32k-bool`	0.87×	0.00378	`torch-compile` `flaggems` `torch`	0.0033 0.00643 0.0105	0.278
`mask-validation-4k-bool`	0.89×	0.00195	`torch-compile` `flaggems` `torch`	0.00173 0.00189 0.0178	0.0671