Attention

本页暂无中文版。以下为英文原文。

15 ops, 223 workloads.

One table per op, one row per workload. Ratio is the fastest other implementation's device time divided by ours, so green is faster than it, plain is level with it, red is slower. Times are in ms. How these numbers are taken.

MHADecodePagedWithKVCacheFwd

q: [B, S_q, H, D]k, v: [N_kv, H, D]real_seqlen_kv: [B], i32block_table: [B, W], i32

dtype=f16

• serving-p128q: [1, 1, 16, 128]k, v: [512, 16, 128]real_seqlen_kv: [1], i32block_table: [1, 4], i32kv_lens=[512]page_size=128
• serving-b2q: [2, 1, 8, 64]k, v: [1024, 8, 64]real_seqlen_kv: [2], i32block_table: [2, 4], i32kv_lens=repeat(1024, 2)page_size=256
• serving-kv1kq: [1, 1, 8, 64]k, v: [1024, 8, 64]real_seqlen_kv: [1], i32block_table: [1, 4], i32kv_lens=[1024]page_size=256
• serving-kv512q: [1, 1, 8, 64]k, v: [512, 8, 64]real_seqlen_kv: [1], i32block_table: [1, 2], i32kv_lens=[512]page_size=256
• speculative-decodeq: [2, 4, 8, 64]k, v: [1024, 8, 64]real_seqlen_kv: [2], i32block_table: [2, 4], i32kv_lens=repeat(1000, 2)page_size=256is_causal=true
• opt-1.3b-b1q: [1, 1, 32, 64]k, v: [2048, 32, 64]real_seqlen_kv: [1], i32block_table: [1, 16], i32kv_lens=[2048]page_size=128
• opt-1.3b-b16q: [16, 1, 32, 64]k, v: [32768, 32, 64]real_seqlen_kv: [16], i32block_table: [16, 16], i32kv_lens=repeat(2048, 16)page_size=128

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
serving-p128	f16	1.44×	0.00643	flashinfer	0.00925	0.659	15%	lat
serving-b2	f16	1.71×	0.00576	flashinfer fa3	0.00982 0.0184	0.743	8%	lat
serving-kv1k	f16	1.80×	0.00538	flashinfer fa3	0.00966 0.0183	0.398	9%	lat
serving-kv512	f16	1.95×	0.00477	flashinfer fa3	0.00928 0.0178	0.224	5%	lat
speculative-decode	f16	2.51×	0.00739	fa3	0.0186	2.26	6%	lat
opt-1.3b-b1	f16	1.51×	0.0102	flashinfer	0.0154	1.68	37%	mem
opt-1.3b-b16	f16	0.98×	0.0793	flashinfer	0.0774	3.45	75%	mem

GQABwd

q, o, do: [B, S, H, D]k, v: [B, S, H_kv, D]lse: [B, H, S], f32

dtype=f16, bf16

• llama-8b-shortq, o, do: [4, 512, 32, 128]k, v: [4, 512, 8, 128]lse: [4, 32, 512], f32
• llama-8b-longq, o, do: [2, 2048, 32, 128]k, v: [2, 2048, 8, 128]lse: [2, 32, 2048], f32
• llama-70b-shortq, o, do: [2, 512, 64, 128]k, v: [2, 512, 8, 128]lse: [2, 64, 512], f32
• llama-70b-longq, o, do: [1, 2048, 64, 128]k, v: [1, 2048, 8, 128]lse: [1, 64, 2048], f32
• llama-8b-short-mhaq, k, v, o, do: [4, 512, 32, 128]lse: [4, 32, 512], f32
• llama-8b-long-mhaq, k, v, o, do: [2, 2048, 32, 128]lse: [2, 32, 2048], f32
• llama-70b-short-mhaq, k, v, o, do: [2, 512, 64, 128]lse: [2, 64, 512], f32
• llama-70b-long-mhaq, k, v, o, do: [1, 2048, 64, 128]lse: [1, 64, 2048], f32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
llama-8b-short	f16	0.91×	0.1503	fa3	0.1373	143	20%	comp
	bf16	0.91×	0.1492	fa3	0.1364	144	19%	comp
llama-8b-long	f16	0.81×	0.5705	fa3	0.4599	301	41%	comp
	bf16	0.81×	0.5686	fa3	0.4584	302	40%	comp
llama-70b-short	f16	0.92×	0.1418	fa3	0.1309	151	21%	comp
	bf16	0.93×	0.1415	fa3	0.1309	152	20%	comp
llama-70b-long	f16	0.79×	0.5656	fa3	0.4479	304	42%	comp
	bf16	0.79×	0.5647	fa3	0.4459	304	40%	comp
llama-8b-short-mha	f16	1.00×	0.1123	fa3	0.1123	191	27%	mem
	bf16	1.00×	0.1120	fa3	0.1116	192	27%	mem
llama-8b-long-mha	f16	0.94×	0.4425	fa3	0.4147	388	53%	comp
	bf16	0.95×	0.4391	fa3	0.4153	391	51%	comp
llama-70b-short-mha	f16	1.00×	0.1130	fa3	0.1125	190	26%	mem
	bf16	1.00×	0.1126	fa3	0.1122	191	27%	mem
llama-70b-long-mha	f16	0.94×	0.4417	fa3	0.4155	389	53%	comp
	bf16	0.94×	0.4410	fa3	0.4137	390	51%	comp

GQADenseFwd

q: [B, S_q, H, D]k, v: [B, S_kv, H_kv, D]

• llama-8b-4kq: [32, 1, 32, 128]k, v: [32, 4096, 8, 128]dtype=f16, bf16
• llama-8b-32kq: [8, 1, 32, 128]k, v: [8, 32768, 8, 128]dtype=f16, bf16
• llama-70b-4kq: [16, 1, 64, 128]k, v: [16, 4096, 8, 128]dtype=f16, bf16
• llama-70b-32kq: [4, 1, 64, 128]k, v: [4, 32768, 8, 128]dtype=f16, bf16
• llama-8b-softcapq: [32, 1, 32, 128]k, v: [32, 4096, 8, 128]softcap=50dtype=f16
• qwen3-30b-1kq: [1, 1, 32, 128]k, v: [1, 1024, 4, 128]dtype=f16
• qwen3-30b-4kq: [1, 1, 32, 128]k, v: [1, 4096, 4, 128]dtype=f16
• qwen3-30b-8kq: [1, 1, 32, 128]k, v: [1, 8192, 4, 128]dtype=f16
• qwen3-30b-16kq: [1, 1, 32, 128]k, v: [1, 16384, 4, 128]dtype=f16
• qwen3-30b-32kq: [1, 1, 32, 128]k, v: [1, 32768, 4, 128]dtype=f16
• qwen3-30b-64kq: [1, 1, 32, 128]k, v: [1, 65536, 4, 128]dtype=f16
• qwen3-30b-128kq: [1, 1, 32, 128]k, v: [1, 131072, 4, 128]dtype=f16
• qwen3-30b-256kq: [1, 1, 32, 128]k, v: [1, 262144, 4, 128]dtype=f16
• falcon-7b-2kq: [32, 1, 71, 64]k, v: [32, 2048, 1, 64]dtype=bf16
• starcoder-8kq: [16, 1, 48, 128]k, v: [16, 8192, 1, 128]dtype=f16
• starcoderbase-3b-8kq: [16, 1, 22, 128]k, v: [16, 8192, 1, 128]dtype=bf16
• gemma-2b-8k-bs1q: [1, 1, 8, 256]k, v: [1, 8192, 1, 256]dtype=f16, bf16
• gemma-2b-8k-b16q: [16, 1, 8, 256]k, v: [16, 8192, 1, 256]dtype=f16, bf16
• qwen3-8b-chunked-prefillq: [1, 128, 32, 128]k, v: [1, 2048, 8, 128]dtype=bf16
• qwen3-8b-chunked-b8q: [8, 128, 32, 128]k, v: [8, 2048, 8, 128]dtype=bf16

q: [B, S_q, H, D]k, v: [B, S_kv, H_kv, D]rope_cos, rope_sin: [S_kv + P_pad, R // 2]

• prefill-ropeq: [1, 256, 8, 128]k, v: [1, 1792, 2, 128]rope_cos, rope_sin: [1792, 32]pos_encoding_mode=roperotary_dim=64dtype=f16, bf16

q: [B, S_q, H, D]k, v: [B, S_kv, H_kv, D]q_scale, k_scale, v_scale: [B, H_kv], f32

• prefill-scaledq: [1, 256, 8, 128]k, v: [1, 1792, 2, 128]q_scale, k_scale, v_scale: [1, 2], f32dtype=fp8e4m3/f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
llama-8b-4k	f16	0.95×	0.1422	flashinfer fa3	0.1354 0.1516	15.3	84%	mem
	bf16	0.95×	0.1416	flashinfer fa3	0.1353 0.1504	15.3	84%	mem
llama-8b-32k	f16	0.95×	0.2667	flashinfer fa3	0.2541 0.2633	16.3	89%	mem
	bf16	0.95×	0.2664	flashinfer fa3	0.2542 0.2626	16.3	90%	mem
llama-70b-4k	f16	1.06×	0.0780	flashinfer fa3	0.0823 0.0841	27.8	77%	mem
	bf16	1.05×	0.0779	flashinfer fa3	0.0822 0.0838	27.8	77%	mem
llama-70b-32k	f16	0.97×	0.1432	flashinfer fa3	0.1388 0.1478	30.3	83%	mem
	bf16	0.97×	0.1422	flashinfer fa3	0.1386 0.1468	30.5	84%	mem
llama-8b-softcap	f16	0.87×	0.1564	flashinfer fa3	0.1355 0.1710	13.9	76%	mem
qwen3-30b-1k	f16	1.66×	0.00582	flashinfer fa3	0.0097 0.0173	2.91	8%	lat
qwen3-30b-4k	f16	1.37×	0.00854	flashinfer fa3	0.0117 0.0212	7.93	22%	lat
qwen3-30b-8k	f16	1.27×	0.0112	flashinfer fa3	0.0142 0.0234	12.1	33%	mem
qwen3-30b-16k	f16	1.33×	0.0163	flashinfer fa3	0.0217 0.0283	16.6	46%	mem
qwen3-30b-32k	f16	1.30×	0.0265	flashinfer fa3	0.0346 0.0375	20.4	56%	mem
qwen3-30b-64k	f16	1.21×	0.0436	flashinfer fa3	0.0525 0.0569	24.9	68%	mem
qwen3-30b-128k	f16	1.09×	0.0756	flashinfer fa3	0.0822 0.0907	28.7	79%	mem
qwen3-30b-256k	f16	1.03×	0.1354	flashinfer fa3	0.1393 0.1587	32	88%	mem
falcon-7b-2k	bf16	0.75×	0.0372	fa3 flashinfer	0.0278 0.0321	32.7	10%	mem
starcoder-8k	f16	1.09×	0.0317	flashinfer fa3	0.0345 0.0368	103	47%	mem
starcoderbase-3b-8k	bf16	1.05×	0.0318	flashinfer fa3	0.0333 0.0391	46.9	47%	mem
gemma-2b-8k-bs1	f16	0.94×	0.0172	flashinfer fa3	0.0162 0.0238	3.92	11%	lat
	bf16	0.95×	0.0172	flashinfer fa3	0.0163 0.0237	3.92	11%	lat
gemma-2b-8k-b16	f16	1.16×	0.0461	flashinfer fa3	0.0537 0.0541	23.4	65%	mem
	bf16	1.17×	0.0461	flashinfer fa3	0.0537 0.0540	23.4	65%	mem
qwen3-8b-chunked-prefill	bf16	1.27×	0.00915	flashinfer fa3	0.0116 0.0207	3.7	20%	lat
qwen3-8b-chunked-b8	bf16	1.11×	0.0300	flashinfer fa3	0.0333 0.0347	9.03	50%	mem
prefill-rope	f16	1.25×	0.0319	fa3 torch-compile torch-ref	0.0399 0.1272 0.2717	55.2	8%	comp
	bf16	1.25×	0.0319	fa3 torch-compile torch-ref	0.0399 0.1271 0.2728	55.3	7%	comp
prefill-scaled	fp8e4m3/f16	2.13×	0.0270	fa3 torch-compile torch-ref	0.0573 0.1230 0.2385	65.3	5%	comp

GQAPagedFwd

q: [T_q, H, D]k_pages, v_pages: [NP, PS, H_kv, D]page_table: [B, W], i32cache_seqlens: [B], i32cu_seqlens_q: [B + 1], i32

• llama-8bq: [4096, 32, 128]k_pages, v_pages: [576, 64, 8, 128]page_table: [8, 72], i32cache_seqlens: [8], i32cu_seqlens_q: [9], i32q_lens=repeat(512, 8)cache_lens=repeat(4096, 8)dtype=f16/f16
• llama-8b-p64q: [32, 32, 128]k_pages, v_pages: [2048, 64, 8, 128]page_table: [32, 64], i32cache_seqlens: [32], i32cu_seqlens_q: [33], i32q_lens=repeat(1, 32)cache_lens=repeat(4096, 32)dtype=f16/f16
• llama-8b-32k-p64q: [8, 32, 128]k_pages, v_pages: [4096, 64, 8, 128]page_table: [8, 512], i32cache_seqlens: [8], i32cu_seqlens_q: [9], i32q_lens=repeat(1, 8)cache_lens=repeat(32768, 8)dtype=f16/f16
• llama-8b-b64-p64q: [64, 32, 128]k_pages, v_pages: [2048, 64, 8, 128]page_table: [64, 32], i32cache_seqlens: [64], i32cu_seqlens_q: [65], i32q_lens=repeat(1, 64)cache_lens=repeat(2048, 64)dtype=f16/f16
• llama-70b-p64q: [8, 64, 128]k_pages, v_pages: [512, 64, 8, 128]page_table: [8, 64], i32cache_seqlens: [8], i32cu_seqlens_q: [9], i32q_lens=repeat(1, 8)cache_lens=repeat(4096, 8)dtype=f16/f16
• llama-8b-p256q: [32, 32, 128]k_pages, v_pages: [512, 256, 8, 128]page_table: [32, 16], i32cache_seqlens: [32], i32cu_seqlens_q: [33], i32q_lens=repeat(1, 32)cache_lens=repeat(4096, 32)dtype=f16/f16
• llama-70b-p256q: [8, 64, 128]k_pages, v_pages: [128, 256, 8, 128]page_table: [8, 16], i32cache_seqlens: [8], i32cu_seqlens_q: [9], i32q_lens=repeat(1, 8)cache_lens=repeat(4096, 8)dtype=f16/f16
• llama-405b-p256q: [4, 128, 128]k_pages, v_pages: [64, 256, 8, 128]page_table: [4, 16], i32cache_seqlens: [4], i32cu_seqlens_q: [5], i32q_lens=repeat(1, 4)cache_lens=repeat(4096, 4)dtype=f16/f16
• llama-8b-p64-softcapq: [32, 32, 128]k_pages, v_pages: [2048, 64, 8, 128]page_table: [32, 64], i32cache_seqlens: [32], i32cu_seqlens_q: [33], i32q_lens=repeat(1, 32)cache_lens=repeat(4096, 32)softcap=50dtype=f16/f16
• llama-8b-p16q: [32, 32, 128]k_pages, v_pages: [8192, 16, 8, 128]page_table: [32, 256], i32cache_seqlens: [32], i32cu_seqlens_q: [33], i32q_lens=repeat(1, 32)cache_lens=repeat(4096, 32)dtype=f16/f16
• llama-8b-b64-p16q: [64, 32, 128]k_pages, v_pages: [8192, 16, 8, 128]page_table: [64, 128], i32cache_seqlens: [64], i32cu_seqlens_q: [65], i32q_lens=repeat(1, 64)cache_lens=repeat(2048, 64)dtype=f16/f16
• llama-70b-p16q: [8, 64, 128]k_pages, v_pages: [2048, 16, 8, 128]page_table: [8, 256], i32cache_seqlens: [8], i32cu_seqlens_q: [9], i32q_lens=repeat(1, 8)cache_lens=repeat(4096, 8)dtype=f16/f16
• llama-8b-p64-cutq: [32, 32, 128]k_pages, v_pages: [2048, 64, 8, 128]page_table: [32, 64], i32cache_seqlens: [32], i32cu_seqlens_q: [33], i32q_lens=repeat(1, 32)cache_lens=repeat(4001, 32)dtype=f16/f16
• llama-8b-p256-cutq: [32, 32, 128]k_pages, v_pages: [512, 256, 8, 128]page_table: [32, 16], i32cache_seqlens: [32], i32cu_seqlens_q: [33], i32q_lens=repeat(1, 32)cache_lens=repeat(4001, 32)dtype=f16/f16
• llama-8b-serving-stepq: [1054, 32, 128]k_pages, v_pages: [2048, 64, 8, 128]page_table: [32, 64], i32cache_seqlens: [32], i32cu_seqlens_q: [33], i32q_lens=[1024,1,1,1…]cache_lens=repeat(4096, 32)dtype=f16/f16
• llama-8b-sliding-windowq: [32, 32, 128]k_pages, v_pages: [8192, 64, 8, 128]page_table: [32, 256], i32cache_seqlens: [32], i32cu_seqlens_q: [33], i32q_lens=repeat(1, 32)cache_lens=repeat(16384, 32)window_size_left=4096dtype=f16/f16
• llama-8b-noncausalq: [263, 32, 128]k_pages, v_pages: [64, 64, 8, 128]page_table: [8, 8], i32cache_seqlens: [8], i32cu_seqlens_q: [9], i32q_lens=[256,1,1,1…]cache_lens=repeat(512, 8)is_causal=falsedtype=f16/f16
• gpt-oss-120b-windowq: [526, 64, 64]k_pages, v_pages: [512, 64, 8, 64]page_table: [16, 32], i32cache_seqlens: [16], i32cu_seqlens_q: [17], i32q_lens=[512,1,1,1…]cache_lens=repeat(2048, 16)window_size_left=128dtype=bf16/bf16
• qwen3-ragged-oddq: [1007, 32, 128]k_pages, v_pages: [774, 48, 8, 128]page_table: [9, 86], i32cache_seqlens: [9], i32cu_seqlens_q: [10], i32q_lens=[1000,1,1,1…]cache_lens=repeat(4099, 9)dtype=bf16/bf16

q: [T_q, H, D]k_pages, v_pages: [NP, PS, H_kv, D]page_table: [B, W], i32cache_seqlens: [B], i32cu_seqlens_q: [B + 1], i32rope_cos, rope_sin: [P, R // 2]

pos_encoding_mode=ropedtype=f16/f16, bf16/bf16

• llama-8b-rope-decodeq: [8, 32, 128]k_pages, v_pages: [512, 64, 8, 128]page_table: [8, 64], i32cache_seqlens: [8], i32cu_seqlens_q: [9], i32rope_cos, rope_sin: [4096, 64]q_lens=1(×8)cache_lens=4096(×8)
• llama-8b-rope-prefillq: [2048, 32, 128]k_pages, v_pages: [256, 64, 8, 128]page_table: [4, 64], i32cache_seqlens: [4], i32cu_seqlens_q: [5], i32rope_cos, rope_sin: [4096, 64]q_lens=512(×4)cache_lens=4096(×4)
• qwen-ragged-ropeq: [289, 32, 128]k_pages, v_pages: [88, 48, 8, 128]page_table: [4, 22], i32cache_seqlens: [4], i32cu_seqlens_q: [5], i32rope_cos, rope_sin: [1025, 32]q_lens=[257,1,0,31]cache_lens=[1025,513,0,127]rotary_dim=64rope_layout=interleavedwindow_size_left=256

q: [T_q, H, D]k_pages, v_pages: [NP, PS, H_kv, D]page_table: [B, W], i32cache_seqlens: [B], i32cu_seqlens_q: [B + 1], i32sinks: [H], f32

dtype=bf16/bf16

• gpt-oss-sink-mixedq: [288, 64, 64]k_pages, v_pages: [68, 64, 8, 64]page_table: [4, 17], i32cache_seqlens: [4], i32cu_seqlens_q: [5], i32sinks: [64], f32q_lens=[256,1,0,31]cache_lens=[1025,513,0,127]window_size_left=128
• gpt-oss-sink-decodeq: [8, 64, 64]k_pages, v_pages: [256, 64, 8, 64]page_table: [8, 32], i32cache_seqlens: [8], i32cu_seqlens_q: [9], i32sinks: [64], f32q_lens=1(×8)cache_lens=2048(×8)

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
llama-8b	f16/f16	1.05×	0.4825	fa3 flashinfer torch-ref	0.5060 0.5534 15.9288	539	74%	comp
llama-8b-p64	f16/f16	1.18×	0.1354	fa3 flashinfer torch-ref	0.1594 0.2246 6.9859	16	88%	mem
llama-8b-32k-p64	f16/f16	1.00×	0.2815	fa3 flashinfer torch-ref	0.2827 0.4359 13.1393	15.4	85%	mem
llama-8b-b64-p64	f16/f16	1.18×	0.1405	fa3 flashinfer torch-ref	0.1663 0.2279 8.9220	15.4	85%	mem
llama-70b-p64	f16/f16	1.07×	0.0512	fa3 flashinfer torch-ref	0.0547 0.1044 2.8987	21.2	58%	mem
llama-8b-p256	f16/f16	1.13×	0.1402	fa3 flashinfer torch-ref	0.1581 0.2245 6.9774	15.5	85%	mem
llama-70b-p256	f16/f16	1.02×	0.0521	fa3 flashinfer torch-ref	0.0530 0.1041 2.8941	20.8	57%	mem
llama-405b-p256	f16/f16	0.92×	0.0395	fa3 flashinfer torch-ref	0.0364 0.2812 2.5816	27.5	38%	mem
llama-8b-p64-softcap	f16/f16	1.29×	0.1395	fa3 flashinfer torch-ref	0.1796 0.2298 7.1265	15.6	86%	mem
llama-8b-p16	f16/f16	1.16×	0.1385	fa3 flashinfer torch-ref	0.1607 0.2254 7.0022	15.7	86%	mem
llama-8b-b64-p16	f16/f16	1.16×	0.1439	fa3 flashinfer torch-ref	0.1676 0.2288 8.9227	15.1	83%	mem
llama-70b-p16	f16/f16	0.99×	0.0558	fa3 flashinfer torch-ref	0.0552 0.1043 2.9011	19.4	54%	mem
llama-8b-p64-cut	f16/f16	1.18×	0.1343	fa3 flashinfer torch-ref	0.1584 0.2244 6.9156	15.8	87%	mem
llama-8b-p256-cut	f16/f16	1.11×	0.1421	fa3 flashinfer torch-ref	0.1574 0.2242 6.9121	14.9	82%	mem
llama-8b-serving-step	f16/f16	1.03×	0.2586	fa3 flashinfer torch-ref	0.2664 0.6729 10.4514	243	46%	mem
llama-8b-sliding-window	f16/f16	1.18×	0.1381	fa3 flashinfer torch-ref	0.1631 0.8667 24.3383	15.7	86%	mem
llama-8b-noncausal	f16/f16	1.23×	0.0164	fa3 flashinfer torch-ref	0.0203 0.0424 0.5562	136	28%	mem
gpt-oss-120b-window	bf16/bf16	1.48×	0.0245	fa3 flashinfer torch-ref	0.0362 0.1011 3.5841	46.4	12%	mem
qwen3-ragged-odd	bf16/bf16	0.96×	0.1975	fa3 flashinfer torch-ref	0.1894 0.2814 5.4834	304	40%	comp
llama-8b-rope-decode	f16/f16	4.20×	0.1438	fa3 torch-ref	0.6040 2.5531	3.77	21%	mem
	bf16/bf16	4.29×	0.1406	fa3 torch-ref	0.6039 2.5555	3.86	21%	mem
llama-8b-rope-prefill	f16/f16	1.63×	0.3193	fa3 torch-ref	0.5218 8.4965	407	56%	comp
	bf16/bf16	1.63×	0.3185	fa3 torch-ref	0.5189 8.4952	409	54%	comp
qwen-ragged-rope	f16/f16	2.02×	0.0315	fa3 torch-ref	0.0636 0.7036	36.7	6%	mem
	bf16/bf16	2.07×	0.0310	fa3 torch-ref	0.0643 0.7042	37.2	6%	mem
gpt-oss-sink-mixed	bf16/bf16	2.37×	0.0140	flashinfer torch-ref	0.0332 0.7200	43.8	10%	lat
gpt-oss-sink-decode	bf16/bf16	5.26×	0.0281	flashinfer torch-ref	0.1479 1.0388	9.73	27%	mem

GQAPrefillPagedWithKVCacheFwd

q: [T_q, H, D]k_new, v_new: [T_q, H_kv, D]k_pages, v_pages: [PT, H_kv, D]k_scale, v_scale: [1], f32cu_seqlens_q: [B + 1], i32cache_seqlens: [B], i32block_table: [B, W], i32

page_size=64

• qwen35-9b-ropeq: [8192, 32, 256]k_new, v_new: [8192, 8, 256]k_pages, v_pages: [270336, 8, 256]k_scale, v_scale: [1], f32cu_seqlens_q: [9], i32cache_seqlens: [8], i32block_table: [8, 528], i32q_lens=repeat(1024, 8)cache_lens=repeat(32768, 8)max_seqlen_q=1024fuse_rope=truemax_position=33792rotary_dim=64dtype=f16
• qwen35-9b-mixedq: [4608, 32, 256]k_new, v_new: [4608, 8, 256]k_pages, v_pages: [270336, 8, 256]k_scale, v_scale: [1], f32cu_seqlens_q: [9], i32cache_seqlens: [8], i32block_table: [8, 528], i32q_lens=[256,512,768,1024…]cache_lens=[4096,8192,16384,32768…]max_seqlen_q=1024fuse_rope=truemax_position=33792rotary_dim=64dtype=f16
• llama-8b-ropeq: [4096, 32, 128]k_new, v_new: [4096, 8, 128]k_pages, v_pages: [36864, 8, 128]k_scale, v_scale: [1], f32cu_seqlens_q: [9], i32cache_seqlens: [8], i32block_table: [8, 72], i32q_lens=repeat(512, 8)cache_lens=repeat(4096, 8)max_seqlen_q=512fuse_rope=truemax_position=4608dtype=f16
• synthetic-softcapq: [2048, 8, 64]k_new, v_new: [2048, 2, 64]k_pages, v_pages: [18432, 2, 64]k_scale, v_scale: [1], f32cu_seqlens_q: [5], i32cache_seqlens: [4], i32block_table: [4, 72], i32q_lens=repeat(512, 4)cache_lens=repeat(4096, 4)max_seqlen_q=512softcap=50dtype=f16
• falcon-7bq: [2048, 71, 64]k_new, v_new: [2048, 1, 64]k_pages, v_pages: [18432, 1, 64]k_scale, v_scale: [1], f32cu_seqlens_q: [5], i32cache_seqlens: [4], i32block_table: [4, 72], i32q_lens=repeat(512, 4)cache_lens=repeat(1536, 4)max_seqlen_q=512dtype=bf16
• starcoderq: [2048, 48, 128]k_new, v_new: [2048, 1, 128]k_pages, v_pages: [18432, 1, 128]k_scale, v_scale: [1], f32cu_seqlens_q: [5], i32cache_seqlens: [4], i32block_table: [4, 72], i32q_lens=repeat(512, 4)cache_lens=repeat(4096, 4)max_seqlen_q=512dtype=f16
• gemma-2bq: [2048, 8, 256]k_new, v_new: [2048, 1, 256]k_pages, v_pages: [18432, 1, 256]k_scale, v_scale: [1], f32cu_seqlens_q: [5], i32cache_seqlens: [4], i32block_table: [4, 72], i32q_lens=repeat(512, 4)cache_lens=repeat(4096, 4)max_seqlen_q=512dtype=bf16

q: [T_q, H, D]k_new, v_new: [T_q, H_kv, D]k_pages, v_pages: [PT, H_kv, D], fp8e4m3k_scale, v_scale: [1], f32cu_seqlens_q: [B + 1], i32cache_seqlens: [B], i32block_table: [B, W], i32

page_size=64dtype=f16/fp8e4m3

• synthetic-softcapq: [2048, 8, 64]k_new, v_new: [2048, 2, 64]k_pages, v_pages: [18432, 2, 64], fp8e4m3k_scale, v_scale: [1], f32cu_seqlens_q: [5], i32cache_seqlens: [4], i32block_table: [4, 72], i32q_lens=repeat(512, 4)cache_lens=repeat(4096, 4)max_seqlen_q=512softcap=50
• qwen35-9bq: [8192, 32, 256]k_new, v_new: [8192, 8, 256]k_pages, v_pages: [270336, 8, 256], fp8e4m3k_scale, v_scale: [1], f32cu_seqlens_q: [9], i32cache_seqlens: [8], i32block_table: [8, 528], i32q_lens=repeat(1024, 8)cache_lens=repeat(32768, 8)max_seqlen_q=1024
• llama-8bq: [4096, 32, 128]k_new, v_new: [4096, 8, 128]k_pages, v_pages: [36864, 8, 128], fp8e4m3k_scale, v_scale: [1], f32cu_seqlens_q: [9], i32cache_seqlens: [8], i32block_table: [8, 72], i32q_lens=repeat(512, 8)cache_lens=repeat(4096, 8)max_seqlen_q=512

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
qwen35-9b-rope	f16	0.33×	42.4098	fa3 torch-ref	14.1401 374.8520	212	29%	comp
qwen35-9b-mixed	f16	0.23×	22.7306	fa3 torch-ref	5.1875 148.3550	147	20%	comp
llama-8b-rope	f16	0.41×	1.6757	fa3 torch-ref	0.6898 20.1232	176	24%	comp
synthetic-softcap	f16	1.04×	0.1318	fa3 torch-ref	0.1374 3.0103	143	20%	comp
falcon-7b	bf16	0.81×	0.3136	fa3 torch-ref	0.2533 6.6964	217	29%	comp
starcoder	f16	0.64×	0.7879	fa3 torch-ref	0.5060 14.3921	281	38%	comp
gemma-2b	bf16	0.56×	0.3033	fa3 torch-ref	0.1686 4.3211	242	32%	comp
synthetic-softcap	f16/fp8e4m3	1.21×	0.1750	fa3 torch-ref	0.2114 3.1301	108	15%	comp
qwen35-9b	f16/fp8e4m3	0.41×	56.6261	fa3 torch-ref	23.4157 381.6930	159	22%	comp
llama-8b	f16/fp8e4m3	0.71×	1.9787	fa3 torch-ref	1.4050 19.7368	149	20%	comp

GQAVarlenFwd

q: [T_q, H, D]k, v: [T_kv, H_kv, D]cu_seqlens_q, cu_seqlens_kv: [B + 1], i32

dtype=f16, bf16

• llama-8b-uniformq: [2048, 32, 128]k, v: [2048, 8, 128]cu_seqlens_q, cu_seqlens_kv: [5], i32q_lens=repeat(512, 4)kv_lens=repeat(512, 4)
• llama-8b-q-lt-kvq: [1024, 32, 128]k, v: [3072, 8, 128]cu_seqlens_q, cu_seqlens_kv: [3], i32q_lens=repeat(512, 2)kv_lens=[1024,2048]
• llama-70b-q-lt-kvq: [1024, 64, 128]k, v: [3072, 8, 128]cu_seqlens_q, cu_seqlens_kv: [3], i32q_lens=repeat(512, 2)kv_lens=[1024,2048]
• gpt-oss-uniformq: [2048, 64, 64]k, v: [2048, 8, 64]cu_seqlens_q, cu_seqlens_kv: [5], i32q_lens=repeat(512, 4)kv_lens=repeat(512, 4)
• gpt-oss-q-lt-kvq: [1024, 64, 64]k, v: [3072, 8, 64]cu_seqlens_q, cu_seqlens_kv: [3], i32q_lens=repeat(512, 2)kv_lens=[1024,2048]
• llama-8b-short-w256q: [2048, 32, 128]k, v: [2048, 8, 128]cu_seqlens_q, cu_seqlens_kv: [5], i32q_lens=repeat(512, 4)kv_lens=repeat(512, 4)window_size_left=256
• llama-8b-long-w1024q: [8192, 32, 128]k, v: [8192, 8, 128]cu_seqlens_q, cu_seqlens_kv: [5], i32q_lens=repeat(2048, 4)kv_lens=repeat(2048, 4)window_size_left=1024
• llama-70b-short-w256q: [2048, 64, 128]k, v: [2048, 8, 128]cu_seqlens_q, cu_seqlens_kv: [5], i32q_lens=repeat(512, 4)kv_lens=repeat(512, 4)window_size_left=256
• llama-70b-long-w1024q: [8192, 64, 128]k, v: [8192, 8, 128]cu_seqlens_q, cu_seqlens_kv: [5], i32q_lens=repeat(2048, 4)kv_lens=repeat(2048, 4)window_size_left=1024
• gemma4-31b-uniformq: [4096, 32, 512]k, v: [4096, 4, 512]cu_seqlens_q, cu_seqlens_kv: [3], i32q_lens=repeat(2048, 2)kv_lens=repeat(2048, 2)
• gemma4-31b-q-lt-kvq: [1024, 32, 512]k, v: [3072, 4, 512]cu_seqlens_q, cu_seqlens_kv: [3], i32q_lens=repeat(512, 2)kv_lens=[1024,2048]
• gemma4-26b-a4b-raggedq: [6048, 16, 512]k, v: [6048, 2, 512]cu_seqlens_q, cu_seqlens_kv: [4], i32q_lens=[1000,2048,3000]kv_lens=[1000,2048,3000]

q: [T_q, H, D]k, v: [T_kv, H_kv, D]cu_seqlens_q, cu_seqlens_kv: [B + 1], i32rope_cos, rope_sin: [P, R // 2]

pos_encoding_mode=ropedtype=f16, bf16

• llama-8b-uniform-ropeq: [2048, 32, 128]k, v: [2048, 8, 128]cu_seqlens_q, cu_seqlens_kv: [5], i32rope_cos, rope_sin: [512, 64]q_lens=repeat(512, 4)kv_lens=repeat(512, 4)
• gpt-oss-rope-interleavedq: [2048, 64, 64]k, v: [2048, 8, 64]cu_seqlens_q, cu_seqlens_kv: [5], i32rope_cos, rope_sin: [512, 32]q_lens=repeat(512, 4)kv_lens=repeat(512, 4)rope_layout=interleaved
• llama-8b-rope-partialq: [1018, 32, 128]k, v: [2041, 8, 128]cu_seqlens_q, cu_seqlens_kv: [5], i32rope_cos, rope_sin: [1024, 32]q_lens=[1,0,1000,17]kv_lens=[1024,0,1000,17]rotary_dim=64
• qwen-ragged-ropeq: [3018, 16, 128]k, v: [4017, 2, 128]cu_seqlens_q, cu_seqlens_kv: [4], i32rope_cos, rope_sin: [3000, 64]q_lens=[3000,1,17]kv_lens=[3000,1000,17]rope_layout=interleaved

q: [T_q, H, D]k, v: [T_kv, H_kv, D]cu_seqlens_q, cu_seqlens_kv: [B + 1], i32q_scale, k_scale, v_scale: [B, H_kv], f32

dtype=fp8e4m3/bf16

• llama-8b-fp8q: [2048, 32, 128]k, v: [2048, 8, 128]cu_seqlens_q, cu_seqlens_kv: [5], i32q_scale, k_scale, v_scale: [4, 8], f32q_lens=repeat(512, 4)kv_lens=repeat(512, 4)
• llama-70b-fp8-q-lt-kvq: [1024, 64, 128]k, v: [3072, 8, 128]cu_seqlens_q, cu_seqlens_kv: [3], i32q_scale, k_scale, v_scale: [2, 8], f32q_lens=repeat(512, 2)kv_lens=[1024,2048]
• qwen3-fp8-raggedq: [3017, 32, 128]k, v: [6146, 4, 128]cu_seqlens_q, cu_seqlens_kv: [6], i32q_scale, k_scale, v_scale: [5, 4], f32q_lens=[1,17,1000,0…]kv_lens=[1,17,1000,3000…]
• llama-8b-fp8-w256q: [2048, 32, 128]k, v: [2048, 8, 128]cu_seqlens_q, cu_seqlens_kv: [5], i32q_scale, k_scale, v_scale: [4, 8], f32q_lens=repeat(512, 4)kv_lens=repeat(512, 4)window_size_left=256

q: [T_q, H, D]k, v: [T_kv, H_kv, D]cu_seqlens_q, cu_seqlens_kv: [B + 1], i32sinks: [H], f32

• gpt-oss-sink-mixedq: [1042, 64, 64]k, v: [2066, 8, 64]cu_seqlens_q, cu_seqlens_kv: [5], i32sinks: [64], f32q_lens=[1024,1,0,17]kv_lens=[1024,1025,0,17]window_size_left=128dtype=f16, bf16

q: [T_q, H, D]k, v: [T_kv, H_kv, D]cu_seqlens_q, cu_seqlens_kv: [B + 1], i32q_scale, k_scale, v_scale: [B, H_kv], f32sinks: [H], f32

• llama-8b-fp8-sinkq: [2048, 32, 128]k, v: [2048, 8, 128]cu_seqlens_q, cu_seqlens_kv: [5], i32q_scale, k_scale, v_scale: [4, 8], f32sinks: [32], f32q_lens=repeat(512, 4)kv_lens=repeat(512, 4)dtype=fp8e4m3/bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
llama-8b-uniform	f16	1.10×	0.0360	flashinfer fa3 torch-ref	0.0395 0.0449 1.3825	241	33%	comp
	bf16	1.10×	0.0357	flashinfer fa3 torch-ref	0.0392 0.0449 1.3804	243	32%	comp
llama-8b-q-lt-kv	f16	1.13×	0.0474	fa3 flashinfer torch-ref	0.0536 0.0633 1.7887	458	63%	comp
	bf16	1.12×	0.0473	fa3 flashinfer torch-ref	0.0533 0.0632 1.7910	458	60%	comp
llama-70b-q-lt-kv	f16	1.13×	0.0871	fa3 flashinfer torch-ref	0.0987 0.1065 3.4193	498	68%	comp
	bf16	1.14×	0.0861	fa3 flashinfer torch-ref	0.0981 0.1055 3.4240	504	66%	comp
gpt-oss-uniform	f16	1.19×	0.0495	flashinfer fa3 torch-ref	0.0587 0.0608 1.9537	177	24%	comp
	bf16	1.19×	0.0492	flashinfer fa3 torch-ref	0.0585 0.0608 1.9527	179	23%	comp
gpt-oss-q-lt-kv	f16	1.05×	0.0712	fa3 flashinfer torch-ref	0.0745 0.0935 2.5629	308	42%	comp
	bf16	1.05×	0.0711	fa3 flashinfer torch-ref	0.0744 0.0929 2.5634	308	41%	comp
llama-8b-short-w256	f16	1.05×	0.0394	flashinfer fa3 torch-ref	0.0412 0.0469 1.4067	166	24%	mem
	bf16	1.06×	0.0389	flashinfer fa3 torch-ref	0.0412 0.0469 1.4065	168	24%	mem
llama-8b-long-w1024	f16	1.13×	0.2423	fa3 flashinfer torch-ref	0.2728 0.2800 17.6706	430	59%	comp
	bf16	1.12×	0.2414	fa3 flashinfer torch-ref	0.2715 0.2780 17.6730	432	57%	comp
llama-70b-short-w256	f16	1.04×	0.0661	flashinfer fa3 torch-ref	0.0688 0.0833 2.5719	198	27%	comp
	bf16	1.04×	0.0660	flashinfer fa3 torch-ref	0.0688 0.0832 2.5729	198	26%	comp
llama-70b-long-w1024	f16	1.12×	0.4638	flashinfer fa3 torch-ref	0.5212 0.5307 33.0182	449	62%	comp
	bf16	1.13×	0.4624	flashinfer fa3 torch-ref	0.5209 0.5277 33.0353	451	59%	comp
gemma4-31b-uniform	f16	14.78×	1.3547	torch-ref	20.0168	204	28%	comp
	bf16	14.83×	1.3517	torch-ref	20.0417	204	27%	comp
gemma4-31b-q-lt-kv	f16	8.60×	0.5032	torch-ref	4.3271	171	23%	comp
	bf16	8.61×	0.5029	torch-ref	4.3281	171	23%	comp
gemma4-26b-a4b-ragged	f16	14.42×	1.2174	torch-ref	17.5576	192	26%	comp
	bf16	14.42×	1.2175	torch-ref	17.5608	192	25%	comp
llama-8b-uniform-rope	f16	1.12×	0.0486	flashinfer fa3 torch-ref	0.0543 0.0586 1.7464	179	25%	comp
	bf16	1.09×	0.0500	flashinfer fa3 torch-ref	0.0544 0.0587 1.7477	174	23%	comp
gpt-oss-rope-interleaved	f16	1.07×	0.0657	flashinfer fa3 torch-ref	0.0704 0.0714 2.3336	134	18%	comp
	bf16	1.06×	0.0663	flashinfer fa3 torch-ref	0.0703 0.0711 2.3366	132	17%	comp
llama-8b-rope-partial	f16	0.91×	0.0514	fa3 flashinfer torch-ref	0.0468 0.0514 1.4933	162	22%	comp
	bf16	0.90×	0.0520	fa3 flashinfer torch-ref	0.0469 0.0514 1.4956	160	21%	comp
qwen-ragged-rope	f16	0.96×	0.1012	fa3 flashinfer torch-ref	0.0968 0.1162 5.4322	368	50%	comp
	bf16	0.95×	0.1012	fa3 flashinfer torch-ref	0.0965 0.1156 5.4350	368	48%	comp
llama-8b-fp8	fp8e4m3/bf16	1.03×	0.0425	fa3 torch-ref	0.0438 1.5430	205	15%	mem
llama-70b-fp8-q-lt-kv	fp8e4m3/bf16	0.85×	0.1011	fa3 torch-ref	0.0857 3.6087	429	31%	comp
qwen3-fp8-ragged	fp8e4m3/bf16	0.91×	0.1098	fa3 torch-ref	0.0996 6.4426	416	30%	comp
llama-8b-fp8-w256	fp8e4m3/bf16	1.08×	0.0467	fa3 torch-ref	0.0506 1.5693	140	14%	mem
gpt-oss-sink-mixed	f16	1.48×	0.0373	flashinfer-paged-sink torch-ref	0.0554 2.6547	55.7	13%	mem
	bf16	1.48×	0.0374	flashinfer-paged-sink torch-ref	0.0553 2.6552	55.5	13%	mem
llama-8b-fp8-sink	fp8e4m3/bf16	44.98×	0.0433	torch-ref	1.9461	201	15%	mem

MLADecodeWithKVCacheFwd

q: [B, H, D]q_pe: [B, H, PE]k: [B, N_kv, H_kv, D]k_pe: [B, N_kv, H_kv, PE]

• ds-v2-4kq: [32, 128, 512]q_pe: [32, 128, 64]k: [32, 4096, 1, 512]k_pe: [32, 4096, 1, 64]dtype=f16, bf16
• ds-v2-32kq: [8, 128, 512]q_pe: [8, 128, 64]k: [8, 32768, 1, 512]k_pe: [8, 32768, 1, 64]dtype=f16, bf16
• ds-v3-4kq: [16, 128, 512]q_pe: [16, 128, 64]k: [16, 4096, 1, 512]k_pe: [16, 4096, 1, 64]dtype=bf16
• ds-v3-32kq: [4, 128, 512]q_pe: [4, 128, 64]k: [4, 32768, 1, 512]k_pe: [4, 32768, 1, 64]dtype=bf16
• ds-v3-single-requestq: [1, 128, 512]q_pe: [1, 128, 64]k: [1, 4096, 1, 512]k_pe: [1, 4096, 1, 64]dtype=f16, bf16
• ds-v3-serving-batchq: [12, 128, 512]q_pe: [12, 128, 64]k: [12, 6144, 1, 512]k_pe: [12, 6144, 1, 64]dtype=f16, bf16
• mla-b128-4096q: [128, 128, 512]q_pe: [128, 128, 64]k: [128, 4096, 1, 512]k_pe: [128, 4096, 1, 64]dtype=f16, bf16
• mla-b128-8192q: [128, 128, 512]q_pe: [128, 128, 64]k: [128, 8192, 1, 512]k_pe: [128, 8192, 1, 64]dtype=f16, bf16
• mla-b128-16384q: [128, 128, 512]q_pe: [128, 128, 64]k: [128, 16384, 1, 512]k_pe: [128, 16384, 1, 64]dtype=f16, bf16
• mla-b128-32768q: [128, 128, 512]q_pe: [128, 128, 64]k: [128, 32768, 1, 512]k_pe: [128, 32768, 1, 64]dtype=f16, bf16
• mla-tp-b1q: [1, 64, 512]q_pe: [1, 64, 64]k: [1, 4096, 1, 512]k_pe: [1, 4096, 1, 64]dtype=f16, bf16
• mla-tp-b2q: [2, 64, 512]q_pe: [2, 64, 64]k: [2, 4096, 1, 512]k_pe: [2, 4096, 1, 64]dtype=f16, bf16
• mla-tp-b6q: [6, 64, 512]q_pe: [6, 64, 64]k: [6, 4096, 1, 512]k_pe: [6, 4096, 1, 64]dtype=f16, bf16
• mla-tp-b64q: [64, 64, 512]q_pe: [64, 64, 64]k: [64, 4096, 1, 512]k_pe: [64, 4096, 1, 64]dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
ds-v2-4k	f16	0.77×	0.1167	flashinfer flashmla	0.0898 0.1802	314	43%	comp
	bf16	0.77×	0.1158	flashinfer flashmla	0.0894 0.1810	316	42%	comp
ds-v2-32k	f16	0.41×	0.3915	flashinfer flashmla	0.1592 0.3270	187	26%	comp
	bf16	0.41×	0.3916	flashinfer flashmla	0.1594 0.3288	187	25%	comp
ds-v3-4k	bf16	0.82×	0.0656	flashinfer flashmla	0.0535 0.1066	279	37%	comp
ds-v3-32k	bf16	0.22×	0.3926	flashinfer flashmla	0.0870 0.1921	93.2	12%	comp
ds-v3-single-request	f16	0.41×	0.0553	flashinfer flashmla	0.0228 0.0419	20.7	3%	comp
	bf16	0.41×	0.0553	flashinfer flashmla	0.0228 0.0424	20.7	3%	comp
ds-v3-serving-batch	f16	0.70×	0.0889	flashinfer flashmla	0.0625 0.1221	231	32%	comp
	bf16	0.70×	0.0886	flashinfer flashmla	0.0621 0.1224	232	31%	comp
mla-b128-4096	f16	0.72×	0.4141	flashinfer flashmla	0.2974 0.6202	354	48%	comp
	bf16	0.71×	0.4144	flashinfer flashmla	0.2953 0.6222	353	46%	comp
mla-b128-8192	f16	0.70×	0.8004	flashinfer flashmla	0.5636 1.2032	366	50%	comp
	bf16	0.70×	0.7977	flashinfer flashmla	0.5589 1.2038	367	48%	comp
mla-b128-16384	f16	0.70×	1.5651	flashinfer flashmla	1.0929 2.3395	374	51%	comp
	bf16	0.70×	1.5624	flashinfer flashmla	1.0937 2.3391	375	49%	comp
mla-b128-32768	f16	0.68×	3.1716	flashinfer flashmla	2.1604 8.4949	369	51%	comp
	bf16	0.68×	3.1715	flashinfer flashmla	2.1564 8.4964	369	49%	comp
mla-tp-b1	f16	0.42×	0.0551	flashinfer flashmla	0.0233 0.0459	10.4	2%	mem
	bf16	0.42×	0.0551	flashinfer flashmla	0.0233 0.0477	10.4	2%	mem
mla-tp-b2	f16	0.43×	0.0557	flashinfer flashmla	0.0238 0.0448	20.5	4%	mem
	bf16	0.43×	0.0557	flashinfer flashmla	0.0239 0.0467	20.5	4%	mem
mla-tp-b6	f16	0.44×	0.0610	flashinfer flashmla	0.0268 0.0636	56.2	11%	mem
	bf16	0.44×	0.0610	flashinfer flashmla	0.0269 0.0649	56.2	11%	mem
mla-tp-b64	f16	0.79×	0.1252	flashinfer flashmla	0.0992 0.2768	292	55%	mem
	bf16	0.79×	0.1247	flashinfer flashmla	0.0987 0.2775	293	55%	mem

NSACompressedVarlenFwd

q: [T_q, H, DK]k_cmp: [C, H_kv, DK]v_cmp: [C, H_kv, DV]offsets, chunk_offsets: [N + 1], i32token_indices: [T_q, 2], i32

scale=0.0883883bs=32

• prefill-8x1kq: [8192, 32, 128]k_cmp, v_cmp: [256, 2, 128]offsets, chunk_offsets: [9], i32token_indices: [8192, 2], i32seq_lens=repeat(1024, 8)dtype=f16
• prefill-4x1kq: [4096, 32, 128]k_cmp, v_cmp: [128, 2, 128]offsets, chunk_offsets: [5], i32token_indices: [4096, 2], i32seq_lens=repeat(1024, 4)dtype=f16
• prefill-raggedq: [1024, 32, 128]k_cmp, v_cmp: [33, 2, 128]offsets, chunk_offsets: [3], i32token_indices: [1024, 2], i32seq_lens=[257,767]dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
prefill-8x1k	f16	2.25×	0.0450	fla	0.1014	47.5	67%	mem
prefill-4x1k	f16	2.12×	0.0255	fla	0.0541	41.9	59%	mem
prefill-ragged	bf16	1.91×	0.00947	fla	0.0181	17.4	40%	mem

NSATopKVarlenFwd

q: [T_q, H, D]k_cmp: [C, H_kv, D]offsets, chunk_offsets: [N + 1], i32token_indices: [T_q, 2], i32

dtype=f16, bf16

• prefill-8x1kq: [8192, 32, 128]k_cmp: [256, 2, 128]offsets, chunk_offsets: [9], i32token_indices: [8192, 2], i32seq_lens=repeat(1024, 8)scale=1selected_block_num=16bs=32
• prefill-4x1kq: [4096, 32, 128]k_cmp: [128, 2, 128]offsets, chunk_offsets: [5], i32token_indices: [4096, 2], i32seq_lens=repeat(1024, 4)scale=1selected_block_num=16bs=32
• prefill-raggedq: [5153, 32, 128]k_cmp: [163, 2, 128]offsets, chunk_offsets: [4], i32token_indices: [5153, 2], i32seq_lens=[31,1025,4097]scale=0.0883883selected_block_num=16bs=32
• prefill-long-contextq: [16384, 32, 128]k_cmp: [256, 2, 128]offsets, chunk_offsets: [2], i32token_indices: [16384, 2], i32seq_lens=[16384]scale=0.0883883selected_block_num=32bs=64
• prefill-wide-gqaq: [8192, 64, 64]k_cmp: [256, 2, 64]offsets, chunk_offsets: [5], i32token_indices: [8192, 2], i32seq_lens=repeat(2048, 4)scale=0.125selected_block_num=8bs=32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
prefill-8x1k	f16	1.70×	0.0795	fla	0.1348	27.1	19%	mem
	bf16	1.68×	0.0805	fla	0.1351	26.7	19%	mem
prefill-4x1k	f16	1.65×	0.0431	fla	0.0709	25	18%	mem
	bf16	1.64×	0.0435	fla	0.0711	24.7	17%	mem
prefill-ragged	f16	1.90×	0.0832	fla	0.1583	54.9	11%	mem
	bf16	1.92×	0.0826	fla	0.1585	55.3	12%	mem
prefill-long-context	f16	3.59×	0.4029	fla	1.4462	85.3	12%	comp
	bf16	3.57×	0.4061	fla	1.4495	84.6	11%	comp
prefill-wide-gqa	f16	1.34×	0.1207	fla	0.1620	35.6	12%	mem
	bf16	1.34×	0.1220	fla	0.1630	35.2	12%	mem

NSAVarlenFwd

q: [T_q, H, D]k, v: [T_q, H_kv, D]block_indices: [T_q, H_kv, SEL], i32block_counts: [T_q, H_kv], i32offsets: [N + 1], i32token_indices: [T_q, 2], i32

block_size=32

• prefill-4x2kq: [8192, 16, 64]k, v: [8192, 1, 64]block_indices: [8192, 1, 1], i32block_counts: [8192, 1], i32offsets: [5], i32token_indices: [8192, 2], i32seq_lens=repeat(2048, 4)is_causal=truescale=0.1dtype=f16
• prefill-2x4kq: [8192, 16, 64]k, v: [8192, 1, 64]block_indices: [8192, 1, 4], i32block_counts: [8192, 1], i32offsets: [3], i32token_indices: [8192, 2], i32seq_lens=repeat(4096, 2)is_causal=truescale=0.1dtype=f16
• prefill-noncausal-2x4kq: [8190, 16, 64]k, v: [8190, 1, 64]block_indices: [8190, 1, 4], i32block_counts: [8190, 1], i32offsets: [3], i32token_indices: [8190, 2], i32seq_lens=[4000,4190]is_causal=falsescale=0.1dtype=f16
• prefill-raggedq: [1024, 32, 128]k, v: [1024, 2, 128]block_indices: [1024, 2, 8], i32block_counts, token_indices: [1024, 2], i32offsets: [3], i32seq_lens=[257,767]is_causal=truescale=0.0883883dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
prefill-4x2k	f16	1.53×	0.0178	fla	0.0272	59.9	45%	mem
prefill-2x4k	f16	1.60×	0.0282	fla	0.0451	95.8	28%	mem
prefill-noncausal-2x4k	f16	939×	0.0310	torch-ref	29.1381	87.7	26%	mem
prefill-ragged	bf16	1.71×	0.0250	fla	0.0428	80.3	16%	mem

DSADecodeWithKVCacheFwd

q: [B, S, H, D + dim_tail]kv: [B, S_kv, H_kv, D + dim_tail]indices: [B, S, H_kv, K], i32

stride_kv=1

• ds-v32-kv2kq: [1, 1024, 128, 576]kv: [1, 2048, 1, 576]indices: [1, 1024, 1, 2048], i32q_start_index_s=1024dtype=f16, bf16
• ds-v32-kv4kq: [1, 512, 128, 576]kv: [1, 4096, 1, 576]indices: [1, 512, 1, 1024], i32q_start_index_s=512dtype=f16, bf16
• mla-d576-h128-kv8kq: [1, 4096, 128, 576]kv: [1, 8192, 1, 576]indices: [1, 4096, 1, 2048], i32q_start_index_s=4096dtype=bf16
• mla-d576-h128-kv32kq: [1, 4096, 128, 576]kv: [1, 32768, 1, 576]indices: [1, 4096, 1, 2048], i32q_start_index_s=28672dtype=bf16
• mla-d576-h128-kv64kq: [1, 4096, 128, 576]kv: [1, 65536, 1, 576]indices: [1, 4096, 1, 2048], i32q_start_index_s=61440dtype=bf16
• mla-d576-h128-kv96kq: [1, 4096, 128, 576]kv: [1, 98304, 1, 576]indices: [1, 4096, 1, 2048], i32q_start_index_s=94208dtype=bf16
• mla-d576-h128-kv128kq: [1, 4096, 128, 576]kv: [1, 131072, 1, 576]indices: [1, 4096, 1, 2048], i32q_start_index_s=126976dtype=bf16
• mla-d512-h64-kv8kq: [1, 4096, 64, 512]kv: [1, 8192, 1, 512]indices: [1, 4096, 1, 512], i32q_start_index_s=4096dtype=bf16
• mla-d512-h64-kv32kq: [1, 4096, 64, 512]kv: [1, 32768, 1, 512]indices: [1, 4096, 1, 512], i32q_start_index_s=28672dtype=bf16
• mla-d512-h64-kv48kq: [1, 4096, 64, 512]kv: [1, 49152, 1, 512]indices: [1, 4096, 1, 512], i32q_start_index_s=45056dtype=bf16
• mla-d512-h64-kv64kq: [1, 4096, 64, 512]kv: [1, 65536, 1, 512]indices: [1, 4096, 1, 512], i32q_start_index_s=61440dtype=bf16
• mla-d512-h128-kv8kq: [1, 4096, 128, 512]kv: [1, 8192, 1, 512]indices: [1, 4096, 1, 1024], i32q_start_index_s=4096dtype=bf16
• mla-d512-h128-kv32kq: [1, 4096, 128, 512]kv: [1, 32768, 1, 512]indices: [1, 4096, 1, 1024], i32q_start_index_s=28672dtype=bf16
• mla-d512-h128-kv48kq: [1, 4096, 128, 512]kv: [1, 49152, 1, 512]indices: [1, 4096, 1, 1024], i32q_start_index_s=45056dtype=bf16
• mla-d512-h128-kv64kq: [1, 4096, 128, 512]kv: [1, 65536, 1, 512]indices: [1, 4096, 1, 1024], i32q_start_index_s=61440dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
ds-v32-kv2k	f16	4.85×	1.1002	torch-sdpa torch-compile torch-ref	5.3383 16.8663 19.3138	399	55%	comp
	bf16	1.12×	1.0970	flashmla	1.2282	400	53%	comp
ds-v32-kv4k	f16	5.82×	0.3128	torch-gather torch-sdpa torch-compile torch-ref	1.8203 5.2870 16.5532 19.5675	351	48%	comp
	bf16	1.12×	0.3107	flashmla	0.3471	353	46%	comp
mla-d576-h128-kv8k	bf16	1.14×	4.0602	flashmla	4.6332	577	76%	comp
mla-d576-h128-kv32k	bf16	1.12×	4.1493	flashmla	4.6540	564	74%	comp
mla-d576-h128-kv64k	bf16	1.11×	4.2377	flashmla	4.7131	553	73%	comp
mla-d576-h128-kv96k	bf16	1.11×	4.2677	flashmla	4.7450	549	72%	comp
mla-d576-h128-kv128k	bf16	1.11×	4.2840	flashmla	4.7629	547	72%	comp
mla-d512-h64-kv8k	bf16	1.13×	0.6210	flashmla	0.6995	444	58%	comp
mla-d512-h64-kv32k	bf16	1.10×	0.6460	flashmla	0.7105	427	56%	comp
mla-d512-h64-kv48k	bf16	1.09×	0.6617	flashmla	0.7188	417	55%	comp
mla-d512-h64-kv64k	bf16	1.08×	0.6725	flashmla	0.7285	410	54%	comp
mla-d512-h128-kv8k	bf16	1.13×	2.0808	flashmla	2.3473	530	70%	comp
mla-d512-h128-kv32k	bf16	1.12×	2.1359	flashmla	2.3837	516	68%	comp
mla-d512-h128-kv48k	bf16	1.10×	2.1701	flashmla	2.3976	508	67%	comp
mla-d512-h128-kv64k	bf16	1.10×	2.1874	flashmla	2.4044	504	66%	comp

FP8LightningIndexerFwd

index_q: [B, S, H, D]index_k: [B, S_kv, G, D]weights: [S, H], f32cu_seqlen_ks, cu_seqlen_ke: [S], i32

• chunk-prefillindex_q: [1, 8192, 32, 64]index_k: [1, 32768, 1, 64]weights: [8192, 32], f32cu_seqlen_ks, cu_seqlen_ke: [8192], i32seq_lens=repeat(512, 64)first=24576dtype=bf16, fp8e4m3
• short-prefillindex_q: [1, 2048, 32, 64]index_k: [1, 2048, 1, 64]weights: [2048, 32], f32cu_seqlen_ks, cu_seqlen_ke: [2048], i32seq_lens=[2048]first=0dtype=bf16
• context-chunkindex_q: [1, 3072, 32, 64]index_k: [1, 8192, 1, 64]weights: [3072, 32], f32cu_seqlen_ks, cu_seqlen_ke: [3072], i32seq_lens=[8192]first=5120dtype=bf16

index_q: [B, S, H, D]index_k: [B, S_kv, G, D]weights: [S, H], f32cu_seqlen_ks, cu_seqlen_ke: [S], i32index_k_scale: [B, S_kv, G], f32

dtype=fp8e4m3

• chunk-prefill-kscaleindex_q: [1, 8192, 32, 64]index_k: [1, 32768, 1, 64]weights: [8192, 32], f32cu_seqlen_ks, cu_seqlen_ke: [8192], i32index_k_scale: [1, 32768, 1], f32seq_lens=repeat(512, 64)first=24576
• index-q2048-ctx4096index_q: [1, 2048, 64, 128]index_k: [1, 4096, 1, 128]weights: [2048, 64], f32cu_seqlen_ks, cu_seqlen_ke: [2048], i32index_k_scale: [1, 4096, 1], f32seq_lens=[4096]first=2048
• index-q2048-ctx8192index_q: [1, 2048, 64, 128]index_k: [1, 8192, 1, 128]weights: [2048, 64], f32cu_seqlen_ks, cu_seqlen_ke: [2048], i32index_k_scale: [1, 8192, 1], f32seq_lens=[8192]first=6144
• index-q2048-ctx16384index_q: [1, 2048, 64, 128]index_k: [1, 16384, 1, 128]weights: [2048, 64], f32cu_seqlen_ks, cu_seqlen_ke: [2048], i32index_k_scale: [1, 16384, 1], f32seq_lens=[16384]first=14336
• index-q2048-ctx32768index_q: [1, 2048, 64, 128]index_k: [1, 32768, 1, 128]weights: [2048, 64], f32cu_seqlen_ks, cu_seqlen_ke: [2048], i32index_k_scale: [1, 32768, 1], f32seq_lens=[32768]first=30720
• index-q2048-ctx65536index_q: [1, 2048, 64, 128]index_k: [1, 65536, 1, 128]weights: [2048, 64], f32cu_seqlen_ks, cu_seqlen_ke: [2048], i32index_k_scale: [1, 65536, 1], f32seq_lens=[65536]first=63488
• index-q2048-ctx131072index_q: [1, 2048, 64, 128]index_k: [1, 131072, 1, 128]weights: [2048, 64], f32cu_seqlen_ks, cu_seqlen_ke: [2048], i32index_k_scale: [1, 131072, 1], f32seq_lens=[131072]first=129024
• index-q4096-ctx4096index_q: [1, 4096, 64, 128]index_k: [1, 4096, 1, 128]weights: [4096, 64], f32cu_seqlen_ks, cu_seqlen_ke: [4096], i32index_k_scale: [1, 4096, 1], f32seq_lens=[4096]first=0
• index-q4096-ctx8192index_q: [1, 4096, 64, 128]index_k: [1, 8192, 1, 128]weights: [4096, 64], f32cu_seqlen_ks, cu_seqlen_ke: [4096], i32index_k_scale: [1, 8192, 1], f32seq_lens=[8192]first=4096
• index-q4096-ctx16384index_q: [1, 4096, 64, 128]index_k: [1, 16384, 1, 128]weights: [4096, 64], f32cu_seqlen_ks, cu_seqlen_ke: [4096], i32index_k_scale: [1, 16384, 1], f32seq_lens=[16384]first=12288
• index-q4096-ctx32768index_q: [1, 4096, 64, 128]index_k: [1, 32768, 1, 128]weights: [4096, 64], f32cu_seqlen_ks, cu_seqlen_ke: [4096], i32index_k_scale: [1, 32768, 1], f32seq_lens=[32768]first=28672
• index-q4096-ctx65536index_q: [1, 4096, 64, 128]index_k: [1, 65536, 1, 128]weights: [4096, 64], f32cu_seqlen_ks, cu_seqlen_ke: [4096], i32index_k_scale: [1, 65536, 1], f32seq_lens=[65536]first=61440
• index-q4096-ctx131072index_q: [1, 4096, 64, 128]index_k: [1, 131072, 1, 128]weights: [4096, 64], f32cu_seqlen_ks, cu_seqlen_ke: [4096], i32index_k_scale: [1, 131072, 1], f32seq_lens=[131072]first=126976
• ds-v32-decode-b1index_q: [1, 1, 64, 128]index_k: [1, 16384, 1, 128]weights: [1, 64], f32cu_seqlen_ks, cu_seqlen_ke: [1], i32index_k_scale: [1, 16384, 1], f32seq_lens=[16384]first=16383

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
chunk-prefill	bf16	0.73×	0.5294	deepgemm torch-ref	0.3847 109.3380	16.6	47%	mem
	fp8e4m3	0.72×	0.5153	deepgemm torch-ref	0.3720 109.3340	17.1	47%	mem
short-prefill	bf16	0.81×	0.0701	deepgemm torch-ref	0.0565 1.7409	125	9%	comp
context-chunk	bf16	0.66×	0.3203	deepgemm torch-ref	0.2116 10.2699	268	19%	comp
chunk-prefill-kscale	fp8e4m3	0.68×	0.4719	deepgemm torch-ref	0.3211 109.4180	18.7	51%	mem
index-q2048-ctx4096	fp8e4m3	0.57×	0.2336	deepgemm torch-ref	0.1324 8.9373	446	32%	comp
index-q2048-ctx8192	fp8e4m3	0.56×	0.5040	deepgemm torch-ref	0.2803 17.7869	483	35%	comp
index-q2048-ctx16384	fp8e4m3	0.55×	1.0429	deepgemm torch-ref	0.5729 35.4839	500	36%	comp
index-q2048-ctx32768	fp8e4m3	0.54×	2.1352	deepgemm torch-ref	1.1484 70.8616	505	37%	comp
index-q2048-ctx65536	fp8e4m3	0.53×	4.3488	deepgemm torch-ref	2.2911 141.3730	504	36%	comp
index-q2048-ctx131072	fp8e4m3	0.52×	8.7765	deepgemm torch-ref	4.5616 282.2890	503	36%	comp
index-q4096-ctx4096	fp8e4m3	0.58×	0.3269	deepgemm torch-ref	0.1900 17.8794	425	31%	comp
index-q4096-ctx8192	fp8e4m3	0.55×	0.8641	deepgemm torch-ref	0.4787 35.5649	483	35%	comp
index-q4096-ctx16384	fp8e4m3	0.54×	1.9426	deepgemm torch-ref	1.0512 71.0273	501	36%	comp
index-q4096-ctx32768	fp8e4m3	0.53×	4.1456	deepgemm torch-ref	2.1981 141.8950	503	36%	comp
index-q4096-ctx65536	fp8e4m3	0.52×	8.5583	deepgemm torch-ref	4.4803 283.6260	504	36%	comp
index-q4096-ctx131072	fp8e4m3	0.52×	17.4120	deepgemm torch-ref	8.9723 565.1990	503	36%	comp
ds-v32-decode-b1	fp8e4m3	0.25×	0.2888	torch-ref deepgemm	0.0659 0.0724	0.94	0%	mem

PagedKVCacheWriteFwd

k, v: [N, H_kv, D]k_pages, v_pages: [NP, PS, H_kv, D]slot_mapping: [N], i64

• llama-8b-decodek, v: [64, 8, 128]k_pages, v_pages: [4096, 16, 8, 128]slot_mapping: [64], i64dtype=bf16/bf16, f16/f16
• llama-8b-prefillk, v: [8192, 8, 128]k_pages, v_pages: [4096, 16, 8, 128]slot_mapping: [8192], i64dtype=bf16/bf16
• gpt-oss-120bk, v: [256, 8, 64]k_pages, v_pages: [4096, 16, 8, 64]slot_mapping: [256], i64dtype=bf16/bf16
• odd-tokens-and-pagek, v: [7, 2, 64]k_pages, v_pages: [5, 7, 2, 64]slot_mapping: [7], i64dtype=f16/f16

k, v: [N, H_kv, D]k_pages, v_pages: [NP, PS, H_kv, D], fp8e4m3slot_mapping: [N], i64k_scale, v_scale: [1], f32

• qwen3-235b-fp8k, v: [4096, 4, 128]k_pages, v_pages: [4096, 16, 4, 128], fp8e4m3slot_mapping: [4096], i64k_scale, v_scale: [1], f32dtype=bf16/fp8e4m3
• qwen3-235b-fp8-fp16k, v: [1024, 4, 128]k_pages, v_pages: [512, 32, 4, 128], fp8e4m3slot_mapping: [1024], i64k_scale, v_scale: [1], f32dtype=f16/fp8e4m3

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
llama-8b-decode	bf16/bf16	1.18×	0.00278	vllm flashinfer torch-ref	0.0033 0.00352 0.0199	·	4%	lat
	f16/f16	1.18×	0.00285	vllm flashinfer torch-ref	0.00336 0.00365 0.0203	·	4%	lat
llama-8b-prefill	bf16/bf16	1.05×	0.0202	flashinfer vllm torch-ref	0.0213 0.0587 0.1270	·	74%	mem
gpt-oss-120b	bf16/bf16	1.28×	0.00301	vllm flashinfer torch-ref	0.00384 0.0039 0.0213	·	8%	lat
odd-tokens-and-page	f16/f16	1.63×	0.00163	vllm flashinfer torch-ref	0.00266 0.00307 0.0186	·	0%	lat
qwen3-235b-fp8	bf16/fp8e4m3	5.11×	0.00915	vllm torch-ref	0.0467 0.0905	0.917	31%	lat
qwen3-235b-fp8-fp16	f16/fp8e4m3	1.68×	0.00784	vllm torch-ref	0.0132 0.0468	0.267	9%	lat

TopKSelectFwd

index_score: [B, S, S_kv, G], f32starts, ends: [B, S]

dtype=i32

• prefill-k1024index_score: [1, 32768, 65536, 1], f32starts, ends: [1, 32768]topk=1024
• prefill-k2048index_score: [1, 32768, 65536, 1], f32starts, ends: [1, 32768]topk=2048
• ds-v32-topk-decode-b1index_score: [1, 1, 16384, 1], f32starts, ends: [1, 1]topk=2048

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
prefill-k1024	i32	0.85×	10.9833	flashinfer torch-compile torch	9.3230 36.4245 63.2675	0.196	18%	mem
prefill-k2048	i32	0.81×	13.1517	flashinfer torch-compile torch	10.7128 38.0025 65.3694	0.163	15%	mem
ds-v32-topk-decode-b1	i32	0.34×	0.0644	flashinfer torch-compile torch	0.0218 0.1213 0.1464	0.000255	0%	mem

PagedKVCacheGatherFwd

dst: [T_q, *E], bf16cache: [NP, PS, *E]block_table: [B, W], i32cu_seq_lens: [B + 1], i32scale: [1], f32

• ds-v3-chunk-fp8dst: [131072, 576], bf16cache: [2048, 64, 576]block_table: [8, 256], i32cu_seq_lens: [9], i32scale: [1], f32seq_lens=repeat(16384, 8)dtype=fp8e4m3/bf16

dst: [T_q, *E]cache: [NP, PS, *E]block_table: [B, W], i32cu_seq_lens: [B + 1], i32

• ds-v3-mixeddst: [129024, 576]cache: [6144, 64, 576]block_table: [6, 1024], i32cu_seq_lens: [7], i32seq_lens=[2048,4096,8192,16384…]dtype=bf16
• mla-smalldst: [31, 576]cache: [8, 64, 576]block_table: [2, 4], i32cu_seq_lens: [3], i32seq_lens=[17,14]dtype=f16
• mla-b256dst: [131072, 576]cache: [2048, 64, 576]block_table: [256, 8], i32cu_seq_lens: [257], i32seq_lens=repeat(512, 256)dtype=bf16

dst: [T_q, *E]cache: [NP, PS, *E]block_table: [B, W], i32cu_seq_lens: [B + 1], i32seq_starts: [B], i32

dtype=bf16

• llama-70b-cpdst: [32768, 8, 128]cache: [4096, 16, 8, 128]block_table: [8, 512], i32cu_seq_lens: [9], i32seq_starts: [8], i32seq_lens=repeat(4096, 8)starts=repeat(2048, 8)
• llama-cp-unaligneddst: [1024, 8, 128]cache: [128, 16, 8, 128]block_table: [2, 64], i32cu_seq_lens: [3], i32seq_starts: [2], i32seq_lens=[17,1007]starts=[15,1]

dst: [T_q, *E], f16cache: [NP, PS, *E]block_table: [B, W], i32cu_seq_lens: [B + 1], i32seq_starts: [B], i32scale: [1], f32

• mla-chunk-fp8-fp16dst: [8192, 576], f16cache: [256, 64, 576]block_table: [2, 128], i32cu_seq_lens: [3], i32seq_starts: [2], i32scale: [1], f32seq_lens=[2048,6144]starts=[3,127]dtype=fp8e4m3/f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
ds-v3-chunk-fp8	fp8e4m3/bf16	1.15×	0.0899	vllm torch-ref	0.1035 0.8339	0.84	56%	mem
ds-v3-mixed	bf16	1.48×	0.0910	vllm torch-ref vllm-cp-gather	0.1343 0.1905 2.3382	·	73%	mem
mla-small	f16	1.06×	0.00307	vllm vllm-cp-gather torch-ref	0.00325 0.00333 0.0274	·	1%	lat
mla-b256	bf16	1.41×	0.0975	vllm vllm-cp-gather torch-ref	0.1376 0.3734 3.7555	·	69%	mem
llama-70b-cp	bf16	4.41×	0.0379	vllm-cp-gather torch-ref	0.1672 0.1684	·	79%	mem
llama-cp-unaligned	bf16	8.70×	0.00483	torch-ref vllm-cp-gather	0.0389 0.0420	·	19%	lat
mla-chunk-fp8-fp16	fp8e4m3/f16	1.19×	0.00842	vllm torch-ref	0.00998 0.0825	0.561	37%	mem