Attention

15 ops, 119 workloads.

One table per op, one row per workload. Ratio is the fastest other implementation's device time divided by ours, so green is faster than it, plain is level with it, red is slower. Times are in ms. How these numbers are taken.

Attention

DeepSeekSparseAttentionDecodeWithKVCacheFwd (2 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
longer-kv-lower-topk-float16	3.62×	0.5005	torch-gather torch-sdpa torch-compile torch-ref	1.8136 5.2718 16.2783 19.3187	292
single-batch-mainstream-float16	2.83×	1.8607	torch-sdpa torch-compile torch-ref	5.2580 16.6112 18.9702	314

MultiHeadLatentAttentionDecodeWithKVCacheFwd (6 workloads)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
deepseek-v2-32k-bfloat16	2.15×	0.1188	torch-compile torch-ref	0.2558 0.2779	181
deepseek-v2-32k-float16	2.12×	0.1188	torch-compile torch-ref	0.2520 0.2742	181
deepseek-v2-4k-bfloat16	3.50×	0.0375	torch-compile torch-ref	0.1313 0.1639	287
deepseek-v2-4k-float16	3.42×	0.0374	torch-compile torch-ref	0.1280 0.1646	287
deepseek-v3-32k-bfloat16	1.39×	0.1179	torch-compile torch-ref	0.1635 0.1711	91.1
deepseek-v3-4k-bfloat16	3.24×	0.0216	torch-compile torch-ref	0.0700 0.0849	248

MultiHeadAttentionDecodePagedWithKVCacheFwd (4 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
batch2-page256-float16	1.72×	0.00568	flashinfer fa3	0.00976 0.0185	0.738
longer-cache-float16	1.81×	0.00533	flashinfer fa3	0.00966 0.0182	0.394
shorter-cache-float16	2.01×	0.00464	flashinfer fa3	0.00931 0.0179	0.226
single-token-page128-float16	1.54×	0.006	flashinfer	0.00925	0.699

GroupedQueryAttentionDecodeWithKVCacheFwd (17 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
llama-70b-32k-bfloat16	1.08×	0.1376	fa3 flashinfer	0.1493 0.4417	31.2
llama-70b-32k-float16	1.09×	0.1382	fa3 flashinfer	0.1504 0.3868	31.1
llama-70b-4k-bfloat16	1.08×	0.0791	fa3 flashinfer	0.0852 0.2274	27.2
llama-70b-4k-float16	1.08×	0.0793	fa3 flashinfer	0.0855 0.1995	27.1
llama-8b-32k-bfloat16	1.04×	0.2567	fa3 flashinfer	0.2678 0.4963	16.7
llama-8b-32k-float16	1.04×	0.2579	fa3 flashinfer	0.2689 0.4288	16.7
llama-8b-4k-bfloat16	1.02×	0.1503	fa3 flashinfer	0.1530 0.2566	14.3
llama-8b-4k-float16	1.02×	0.1513	fa3 flashinfer	0.1540 0.2239	14.2
llama-8b-4k-softcap50-float16	82.46×	0.1616	torch-sdpa	13.3261	13.3
qwen3-30b-a3b-bs1-128k-float16	1.09×	0.0764	flashinfer fa3	0.0832 0.0929	28.1
qwen3-30b-a3b-bs1-16k-float16	1.21×	0.0182	flashinfer fa3	0.0219 0.0281	14.8
qwen3-30b-a3b-bs1-1k-float16	1.39×	0.00698	flashinfer fa3	0.0097 0.0174	2.4
qwen3-30b-a3b-bs1-256k-float16	1.04×	0.1362	flashinfer fa3	0.1420 0.1618	31.5
qwen3-30b-a3b-bs1-32k-float16	1.22×	0.0283	flashinfer fa3	0.0347 0.0376	18.9
qwen3-30b-a3b-bs1-4k-float16	1.21×	0.0096	flashinfer fa3	0.0116 0.0212	6.99
qwen3-30b-a3b-bs1-64k-float16	1.16×	0.0456	flashinfer fa3	0.0531 0.0576	23.6
qwen3-30b-a3b-bs1-8k-float16	1.07×	0.0132	flashinfer fa3	0.0141 0.0232	10.2

GroupedQueryAttentionPrefillFwd (16 workloads)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
llama-70b-prefill-dense-q-lt-kv-bfloat16	1.00×	0.1238	flashinfer torch-ref	0.1235 3.7621	521
llama-70b-prefill-dense-q-lt-kv-float16	1.00×	0.1252	flashinfer torch-ref	0.1251 3.7608	515
llama-70b-prefill-fp8tc-bn224-s3584-bfloat16	0.71×	0.7495	fa3 torch-sdpa-dequant	0.5289 1.0509	562
llama-70b-prefill-fp8tc-bn224-s3584-float16	0.71×	0.7495	fa3 torch-sdpa-dequant	0.5302 1.0531	562
llama-70b-prefill-fp8tc-bn224-s7168-bfloat16	0.71×	2.8453	fa3 torch-sdpa-dequant	2.0226 3.4318	592
llama-70b-prefill-fp8tc-bn224-s7168-float16	0.71×	2.8449	fa3 torch-sdpa-dequant	2.0283 3.4318	592
llama-8b-prefill-dense-bfloat16	1.06×	0.0369	flashinfer torch-ref	0.0393 1.1011	233
llama-8b-prefill-dense-float16	1.05×	0.0373	flashinfer torch-ref	0.0393 1.1007	231
llama-8b-prefill-dense-q-lt-kv-bfloat16	1.01×	0.1243	flashinfer torch-ref	0.1252 4.1005	518
llama-8b-prefill-dense-q-lt-kv-float16	1.00×	0.1263	flashinfer torch-ref	0.1265 4.0955	510
llama-8b-prefill-dense-sm-scale-0.125-float16	1.06×	0.0371	flashinfer torch-ref	0.0393 1.1009	232
llama-8b-prefill-dense-softcap50-float16	1.09×	0.0421	flashinfer torch-ref	0.0457 1.2966	205
llama-8b-prefill-fp8tc-bn224-s1792-bfloat16	0.67×	0.1290	fa3 torch-sdpa-dequant	0.0863 0.2255	408
llama-8b-prefill-fp8tc-bn224-s1792-float16	0.67×	0.1288	fa3 torch-sdpa-dequant	0.0860 0.2270	408
llama-8b-prefill-fp8tc-bn224-s896-bfloat16	0.62×	0.0454	fa3 torch-sdpa-dequant	0.0283 0.0926	290
llama-8b-prefill-fp8tc-bn224-s896-float16	0.63×	0.0453	fa3 torch-sdpa-dequant	0.0286 0.0918	290

MultiHeadAttentionDecodeWithKVCacheFwd (8 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
llama-70b-32k-bfloat16	1.01×	0.9798	fa3 flashinfer	0.9864 0.9966	4.38
llama-70b-32k-float16	1.01×	0.9811	fa3 flashinfer	0.9864 0.9962	4.38
llama-70b-4k-bfloat16	1.00×	0.5139	fa3 flashinfer	0.5143 0.5298	4.18
llama-70b-4k-float16	1.00×	0.5139	fa3 flashinfer	0.5150 0.5308	4.18
llama-8b-32k-bfloat16	1.00×	0.9820	fa3 flashinfer	0.9862 0.9989	4.37
llama-8b-32k-float16	1.01×	0.9809	fa3 flashinfer	0.9866 0.9989	4.38
llama-8b-4k-bfloat16	1.00×	0.5104	fa3 flashinfer	0.5116 0.5303	4.21
llama-8b-4k-float16	1.00×	0.5111	fa3 flashinfer	0.5127 0.5294	4.2

GroupedQueryAttentionPrefillPagedWithKVCacheFwd (7 workloads)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
gqa-prefill-paged-fp8-cache-softcap50-b4-prefix4k-chunk512-p64-float16	—	0.2071	·	·	88.1
gqa-prefill-paged-softcap50-b4-prefix4k-chunk512-p64-float16	0.91×	0.1499	fa3	0.1367	122
llama-8b-prefill-paged-b8-prefix4k-chunk512-p64-full-rope-float16	—	1.9500	·	·	150
llama-8b-prefill-paged-fp8-cache-b8-prefix4k-chunk512-p64-float16	—	2.0061	·	·	146
qwen35-9b-prefill-paged-fp8-cache-b8-prefix32k-chunk1k-p64-float16	—	55.9919	·	·	160
qwen35-9b-prefill-paged-fullattn-b8-prefix32k-chunk1k-p64-partial-rope64-float16	—	60.6283	·	·	147
qwen35-9b-prefill-paged-fullattn-mixed-b8-p64-partial-rope64-float16	—	30.7355	·	·	108

GroupedQueryAttentionFwd (8 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
llama-70b-long-bfloat16	0.82×	0.1617	fa3 flashinfer	0.1322 0.1598	425
llama-70b-long-float16	0.82×	0.1627	fa3 flashinfer	0.1339 0.1619	422
llama-70b-short-bfloat16	0.84×	0.0380	fa3 flashinfer	0.0317 0.0392	226
llama-70b-short-float16	0.84×	0.0382	fa3 flashinfer	0.0319 0.0391	225
llama-8b-long-bfloat16	0.83×	0.1613	fa3 flashinfer	0.1331 0.1612	426
llama-8b-long-float16	0.83×	0.1627	fa3 flashinfer	0.1348 0.1615	422
llama-8b-short-bfloat16	0.86×	0.0369	fa3 flashinfer	0.0319 0.0393	233
llama-8b-short-float16	0.86×	0.0371	fa3 flashinfer	0.0319 0.0395	232

GroupedQueryAttentionDecodePagedWithKVCacheFwd (8 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
serving-405b-p256-float16	0.47×	0.0563	fa3	0.0266	19.1
serving-70b-p256-float16	0.54×	0.0685	fa3 flashinfer	0.0368 0.0572	15.7
serving-70b-p64-float16	0.90×	0.0496	flashinfer	0.0445	21.6
serving-8b-long-p64-float16	1.36×	0.2206	flashinfer	0.2996	19.5
serving-8b-p256-float16	0.48×	0.1683	fa3 flashinfer	0.0812 0.1253	12.8
serving-8b-p64-float16	0.75×	0.1669	flashinfer	0.1252	12.9
serving-8b-p64-softcap50-float16	0.71×	0.1765	flashinfer	0.1252	12.2
throughput-8b-p64-float16	0.60×	0.2518	flashinfer	0.1507	8.53

MultiHeadAttentionFwd (8 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
llama-70b-long-bfloat16	0.82×	0.1672	fa3 flashinfer	0.1368 0.1628	411
llama-70b-long-float16	0.83×	0.1679	fa3 flashinfer	0.1395 0.1644	409
llama-70b-short-bfloat16	0.83×	0.0425	fa3 flashinfer	0.0353 0.0411	202
llama-70b-short-float16	0.83×	0.0428	fa3 flashinfer	0.0354 0.0412	201
llama-8b-long-bfloat16	0.82×	0.1669	fa3 flashinfer	0.1365 0.1620	412
llama-8b-long-float16	0.82×	0.1682	fa3 flashinfer	0.1383 0.1652	408
llama-8b-short-bfloat16	0.83×	0.0425	fa3 flashinfer	0.0355 0.0409	202
llama-8b-short-float16	0.82×	0.0425	fa3 flashinfer	0.0348 0.0412	202

GroupedQueryAttentionSlidingWindowFwd (8 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
llama-70b-long-w1024-bfloat16	0.78×	0.1516	fa3 flashinfer	0.1184 0.1514	340
llama-70b-long-w1024-float16	0.78×	0.1528	fa3 flashinfer	0.1199 0.1531	338
llama-70b-short-w256-bfloat16	0.86×	0.0395	fa3 flashinfer	0.0340 0.0408	164
llama-70b-short-w256-float16	0.86×	0.0396	fa3 flashinfer	0.0342 0.0411	164
llama-8b-long-w1024-bfloat16	0.78×	0.1517	fa3 flashinfer	0.1183 0.1529	340
llama-8b-long-w1024-float16	0.79×	0.1529	fa3 flashinfer	0.1209 0.1553	337
llama-8b-short-w256-bfloat16	0.85×	0.0397	fa3 flashinfer	0.0340 0.0411	163
llama-8b-short-w256-float16	0.86×	0.0398	fa3 flashinfer	0.0342 0.0412	162

GroupedQueryAttentionSlidingWindowVarlenFwd (8 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
llama-70b-long-w1024-bfloat16	0.78×	0.6661	flashinfer fa3	0.5171 0.5271	310
llama-70b-long-w1024-float16	0.78×	0.6677	flashinfer fa3	0.5185 0.5281	309
llama-70b-short-w256-bfloat16	0.75×	0.0930	flashinfer fa3	0.0693 0.0834	139
llama-70b-short-w256-float16	0.74×	0.0931	flashinfer fa3	0.0691 0.0834	139
llama-8b-long-w1024-bfloat16	0.77×	0.3492	fa3 flashinfer	0.2700 0.2734	295
llama-8b-long-w1024-float16	0.77×	0.3518	fa3 flashinfer	0.2714 0.2767	293
llama-8b-short-w256-bfloat16	0.73×	0.0567	flashinfer fa3	0.0411 0.0471	114
llama-8b-short-w256-float16	0.73×	0.0569	flashinfer fa3	0.0414 0.0472	114

GroupedQueryAttentionBwd (8 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
llama-70b-long-bfloat16	0.57×	1.0192	fa3	0.5770	169
llama-70b-long-float16	0.72×	0.8080	fa3	0.5802	213
llama-70b-short-bfloat16	0.39×	0.4092	fa3	0.1588	52.5
llama-70b-short-float16	0.81×	0.1961	fa3	0.1592	109
llama-8b-long-bfloat16	0.47×	1.2427	fa3	0.5891	138
llama-8b-long-float16	0.71×	0.8322	fa3	0.5925	206
llama-8b-short-bfloat16	0.40×	0.4154	fa3	0.1654	51.7
llama-8b-short-float16	0.82×	0.2029	fa3	0.1661	106

GroupedQueryAttentionPrefillVarlenFwd (3 workloads)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
llama-70b-prefill-varlen-q-lt-kv-bf16	0.50×	0.1963	fa3 torch-ref	0.0985 2.7647	219
llama-8b-prefill-varlen-mixed-fp16	0.44×	0.1405	fa3 torch-ref	0.0614 1.6792	143
llama-8b-prefill-varlen-uniform-fp16	0.57×	0.1251	fa3 torch-ref	0.0715 2.0383	206

MultiHeadAttentionBwd (8 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
llama-70b-long-bfloat16	0.50×	1.1015	fa3	0.5495	156
llama-70b-long-float16	0.62×	0.8926	fa3	0.5515	192
llama-70b-short-bfloat16	0.31×	0.4565	fa3	0.1431	47
llama-70b-short-float16	0.59×	0.2444	fa3	0.1433	87.9
llama-8b-long-bfloat16	0.42×	1.3118	fa3	0.5471	131
llama-8b-long-float16	0.61×	0.9026	fa3	0.5513	190
llama-8b-short-bfloat16	0.31×	0.4562	fa3	0.1433	47.1
llama-8b-short-float16	0.59×	0.2435	fa3	0.1435	88.2