Linear Attention

本页暂无中文版。以下为英文原文。

10 ops, 385 workloads.

One table per op, one row per workload. Ratio is the fastest other implementation's device time divided by ours, so green is faster than it, plain is level with it, red is slower. Times are in ms. How these numbers are taken.

DeltaNetChunkFwd

q, k: [B, H, L, DK]v: [B, H, L, DV]beta: [B, H, L]

• train-2kq, k, v: [2, 4, 2048, 64]beta: [2, 4, 2048]dtype=f16, bf16
• train-4kq, k, v: [2, 4, 4096, 64]beta: [2, 4, 4096]dtype=f16, bf16
• train-8kq, k, v: [2, 4, 8192, 64]beta: [2, 4, 8192]dtype=f16, bf16
• train-16kq, k, v: [2, 4, 16384, 64]beta: [2, 4, 16384]dtype=f16, bf16
• deltanet-1.3b-trainq, k, v: [1, 16, 256, 128]beta: [1, 16, 256]dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
train-2k	f16	0.98×	0.0625	fla	0.0612	12.3	8%	mem
	bf16	0.98×	0.0630	fla	0.0616	12.2	7%	mem
train-4k	f16	1.02×	0.1089	fla	0.1112	14.1	9%	mem
	bf16	1.02×	0.1094	fla	0.1119	14	9%	mem
train-8k	f16	0.98×	0.2157	fla	0.2119	14.3	9%	mem
	bf16	0.99×	0.2165	fla	0.2136	14.2	9%	mem
train-16k	f16	1.00×	0.4160	fla	0.4151	14.8	9%	mem
	bf16	1.00×	0.4201	fla	0.4186	14.6	9%	mem
deltanet-1.3b-train	bf16	1.33×	0.0303	fla	0.0404	19.1	9%	mem

DeltaNetChunkBwd

do, v, u: [B, H, L, DV]q, k, w: [B, H, L, DK]beta: [B, H, L]S: [B, H, NC + 1, DK, DV], f32Aw, Au: [B, H, L, chunk_size]

• train-2kdo, q, k, v, Aw, Au, w, u: [2, 4, 2048, 64]beta: [2, 4, 2048]S: [2, 4, 33, 64, 64], f32dtype=f16, bf16
• train-4kdo, q, k, v, Aw, Au, w, u: [2, 4, 4096, 64]beta: [2, 4, 4096]S: [2, 4, 65, 64, 64], f32dtype=f16, bf16
• train-8kdo, q, k, v, Aw, Au, w, u: [2, 4, 8192, 64]beta: [2, 4, 8192]S: [2, 4, 129, 64, 64], f32dtype=f16, bf16
• train-16kdo, q, k, v, Aw, Au, w, u: [2, 4, 16384, 64]beta: [2, 4, 16384]S: [2, 4, 257, 64, 64], f32dtype=f16, bf16
• deltanet-1.3b-traindo, q, k, v, w, u: [1, 16, 256, 128]beta: [1, 16, 256]S: [1, 16, 5, 128, 128], f32Aw, Au: [1, 16, 256, 64]dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
train-2k	f16	0.86×	0.1321	fla	0.1133	13.7	4%	mem
	bf16	0.83×	0.1373	fla	0.1145	13.2	4%	mem
train-4k	f16	0.83×	0.2578	fla	0.2151	14.1	4%	mem
	bf16	0.82×	0.2660	fla	0.2170	13.6	4%	mem
train-8k	f16	0.85×	0.5092	fla	0.4317	14.3	4%	mem
	bf16	0.83×	0.5252	fla	0.4352	13.8	4%	mem
train-16k	f16	0.86×	1.0038	fla	0.8662	14.5	4%	mem
	bf16	0.84×	1.0372	fla	0.8720	14	4%	mem
deltanet-1.3b-train	bf16	0.76×	0.0724	fla	0.0548	18.9	5%	mem

DeltaNetRecurrentFwd

q, k: [B, H, DK]v: [B, H, DV]beta: [B, H]state: [B, H, DK, DV]

dtype=bf16

• serving-b1-h8q, k, v: [1, 8, 128]beta: [1, 8]state: [1, 8, 128, 128]
• serving-b1-h16q, k, v: [1, 16, 128]beta: [1, 16]state: [1, 16, 128, 128]
• serving-b1-h32q, k, v: [1, 32, 128]beta: [1, 32]state: [1, 32, 128, 128]
• serving-b1-h48q, k, v: [1, 48, 128]beta: [1, 48]state: [1, 48, 128, 128]
• serving-b1-h64q, k, v: [1, 64, 128]beta: [1, 64]state: [1, 64, 128, 128]
• serving-b8-h32q, k, v: [8, 32, 128]beta: [8, 32]state: [8, 32, 128, 128]
• serving-b8-h48q, k, v: [8, 48, 128]beta: [8, 48]state: [8, 48, 128, 128]
• serving-b8-h64q, k, v: [8, 64, 128]beta: [8, 64]state: [8, 64, 128, 128]

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
serving-b1-h8	bf16	3.01×	0.00288	fla torch-compile torch	0.00867 0.0131 0.0334	0.274	4%	lat
serving-b1-h16	bf16	2.94×	0.00314	fla torch-compile torch	0.00922 0.0142 0.0356	0.503	8%	lat
serving-b1-h32	bf16	3.10×	0.00336	fla torch-compile torch	0.0104 0.0163 0.0394	0.939	14%	lat
serving-b1-h48	bf16	3.14×	0.00365	fla torch-compile torch	0.0115 0.0194 0.0436	1.3	19%	lat
serving-b1-h64	bf16	3.18×	0.00397	fla torch-compile torch	0.0126 0.0178 0.0444	1.59	24%	lat
serving-b8-h32	bf16	3.92×	0.0087	fla torch-compile torch	0.0341 0.0383 0.0900	2.9	43%	mem
serving-b8-h48	bf16	3.23×	0.0123	torch-compile fla torch	0.0398 0.0490 0.1123	3.07	46%	mem
serving-b8-h64	bf16	3.22×	0.0162	torch-compile fla torch	0.0520 0.0680 0.1438	3.12	47%	mem

DeltaNetFwd

q, k: [B, S, H, K]v: [B, S, H, V]beta: [B, S, H]

dtype=bf16

• prefill-2kq, k, v: [2, 2048, 4, 64]beta: [2, 2048, 4]
• prefill-8kq, k, v: [2, 8192, 4, 64]beta: [2, 8192, 4]
• prefill-16kq, k, v: [2, 16384, 4, 64]beta: [2, 16384, 4]
• prefill-4kq, k, v: [1, 4096, 16, 128]beta: [1, 4096, 16]
• prefill-3000q, k, v: [2, 3000, 4, 64]beta: [2, 3000, 4]
• prefill-l2norm-4kq, k, v: [1, 4096, 16, 128]beta: [1, 4096, 16]use_qk_l2norm_in_kernel=true
• decode-freshq, k, v: [8, 1, 32, 128]beta: [8, 1, 32]

q, k: [B, S, H, K]v: [B, S, H, V]beta: [B, S, H]initial_state: [B, H, K, V], f32

• prefill-12k-continuedq, k, v: [2, 12288, 4, 64]beta: [2, 12288, 4]initial_state: [2, 4, 64, 64], f32dtype=bf16
• decode-continuedq, k, v: [8, 1, 32, 128]beta: [8, 1, 32]initial_state: [8, 32, 128, 128], f32dtype=f16, bf16
• decode-l2normq, k, v: [4, 1, 16, 128]beta: [4, 1, 16]initial_state: [4, 16, 128, 128], f32use_qk_l2norm_in_kernel=truedtype=bf16
• decode-narrow-b17q, k, v: [17, 1, 6, 64]beta: [17, 1, 6]initial_state: [17, 6, 64, 64], f32dtype=bf16
• deltanet-1.3b-continueq, k, v: [1, 257, 16, 128]beta: [1, 257, 16]initial_state: [1, 16, 128, 128], f32use_qk_l2norm_in_kernel=truedtype=bf16
• deltanet-1.3b-decode-b32q, k, v: [32, 1, 16, 128]beta: [32, 1, 16]initial_state: [32, 16, 128, 128], f32use_qk_l2norm_in_kernel=truedtype=bf16
• deltanet-1.3b-cont-b4q, k, v: [4, 257, 16, 128]beta: [4, 257, 16]initial_state: [4, 16, 128, 128], f32use_qk_l2norm_in_kernel=truedtype=bf16

q, k: [B, S, H, K]v: [B, S, H, V]beta: [B, S, H]cu_seqlens, cu_seqlens_cpu: [N + 1], i64

dtype=bf16

• varlen-4x1kq, k, v: [1, 4096, 4, 64]beta: [1, 4096, 4]cu_seqlens, cu_seqlens_cpu: [5], i64seq_lens=repeat(1024, 4)
• prefill-l2norm-raggedq, k, v: [1, 3000, 6, 64]beta: [1, 3000, 6]cu_seqlens, cu_seqlens_cpu: [5], i64seq_lens=[1,59,937,2003]use_qk_l2norm_in_kernel=true

q, k: [B, S, H, K]v: [B, S, H, V]beta: [B, S, H]initial_state: [N, H, K, V], f32cu_seqlens: [N + 1], i64

• varlen-raggedq, k, v: [1, 4096, 16, 128]beta: [1, 4096, 16]initial_state: [4, 16, 128, 128], f32cu_seqlens: [5], i64seq_lens=[1,63,1000,3032]dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
prefill-2k	bf16	1.90×	0.0329	fla	0.0626	12.3	6%	mem
prefill-8k	bf16	3.74×	0.0572	fla	0.2137	28.3	13%	mem
prefill-16k	bf16	4.45×	0.0937	fla	0.4169	34.6	16%	mem
prefill-4k	bf16	1.85×	0.1084	fla	0.2003	59.6	14%	mem
prefill-3000	bf16	2.67×	0.0359	fla	0.0957	16.5	8%	mem
prefill-l2norm-4k	bf16	1.68×	0.1300	fla	0.2185	50.1	12%	mem
decode-fresh	bf16	1.87×	0.0055	fla	0.0103	4.58	69%	mem
prefill-12k-continued	bf16	3.03×	0.1051	fla	0.3180	23.1	11%	mem
decode-continued	f16	1.36×	0.0109	fla	0.0149	2.31	69%	mem
	bf16	1.37×	0.0109	fla	0.0149	2.31	69%	mem
decode-l2norm	bf16	1.79×	0.00499	fla	0.00896	1.27	38%	lat
decode-narrow-b17	bf16	1.43×	0.00294	fla	0.00422	0.856	26%	lat
deltanet-1.3b-continue	bf16	1.44×	0.0343	fla	0.0495	11.9	4%	mem
deltanet-1.3b-decode-b32	bf16	1.45×	0.0214	fla	0.0309	2.38	70%	mem
deltanet-1.3b-cont-b4	bf16	2.14×	0.0373	fla	0.0799	43.8	15%	mem
varlen-4x1k	bf16	2.66×	0.0309	fla	0.0821	13.1	6%	mem
prefill-l2norm-ragged	bf16	2.79×	0.0379	fla	0.1055	11.9	6%	mem
varlen-ragged	bf16	1.62×	0.1376	fla	0.2229	46.9	12%	mem

GDNFwd

q, k: [B, S, H, K]v: [B, S, HV, V]g, beta: [B, S, HV]

• prefill-4kq, k, v: [1, 4096, 16, 128]g, beta: [1, 4096, 16]dtype=f16, bf16
• prefill-12kq, k, v: [1, 12288, 16, 128]g, beta: [1, 12288, 16]dtype=f16, bf16
• prefill-32kq, k, v: [1, 32768, 16, 128]g, beta: [1, 32768, 16]dtype=f16, bf16
• prefill-narrow-stateq, k, v: [1, 4096, 16, 64]g, beta: [1, 4096, 16]dtype=bf16
• prefill-3000q, k, v: [1, 3000, 16, 128]g, beta: [1, 3000, 16]dtype=bf16
• decode-freshq, k, v: [8, 1, 32, 128]g, beta: [8, 1, 32]dtype=bf16
• prefill-v-first-narrowq, k, v: [1, 3000, 16, 64]g, beta: [1, 3000, 16]state_v_first=truedtype=bf16
• q35-397b-tp8-65536q, k: [1, 65536, 2, 128]v: [1, 65536, 8, 128]g, beta: [1, 65536, 8]dtype=f16, bf16
• q35-397b-tp8-32768q, k: [1, 32768, 2, 128]v: [1, 32768, 8, 128]g, beta: [1, 32768, 8]dtype=f16, bf16
• q35-397b-tp8-16384q, k: [1, 16384, 2, 128]v: [1, 16384, 8, 128]g, beta: [1, 16384, 8]dtype=f16, bf16
• q35-397b-tp8-8192q, k: [1, 8192, 2, 128]v: [1, 8192, 8, 128]g, beta: [1, 8192, 8]dtype=f16, bf16
• q35-397b-tp8-4096q, k: [1, 4096, 2, 128]v: [1, 4096, 8, 128]g, beta: [1, 4096, 8]dtype=f16, bf16
• q35-397b-tp8-2048q, k: [1, 2048, 2, 128]v: [1, 2048, 8, 128]g, beta: [1, 2048, 8]dtype=f16, bf16
• q35-397b-tp4-65536q, k: [1, 65536, 4, 128]v: [1, 65536, 16, 128]g, beta: [1, 65536, 16]dtype=f16, bf16
• q35-397b-tp4-32768q, k: [1, 32768, 4, 128]v: [1, 32768, 16, 128]g, beta: [1, 32768, 16]dtype=f16, bf16
• q35-397b-tp4-16384q, k: [1, 16384, 4, 128]v: [1, 16384, 16, 128]g, beta: [1, 16384, 16]dtype=f16, bf16
• q35-397b-tp4-8192q, k: [1, 8192, 4, 128]v: [1, 8192, 16, 128]g, beta: [1, 8192, 16]dtype=f16, bf16
• q35-397b-tp4-4096q, k: [1, 4096, 4, 128]v: [1, 4096, 16, 128]g, beta: [1, 4096, 16]dtype=f16, bf16
• q35-397b-tp4-2048q, k: [1, 2048, 4, 128]v: [1, 2048, 16, 128]g, beta: [1, 2048, 16]dtype=f16, bf16
• q35-397b-tp2-65536q, k: [1, 65536, 8, 128]v: [1, 65536, 32, 128]g, beta: [1, 65536, 32]dtype=f16, bf16
• q35-397b-tp2-32768q, k: [1, 32768, 8, 128]v: [1, 32768, 32, 128]g, beta: [1, 32768, 32]dtype=f16, bf16
• q35-397b-tp2-16384q, k: [1, 16384, 8, 128]v: [1, 16384, 32, 128]g, beta: [1, 16384, 32]dtype=f16, bf16
• q35-397b-tp2-8192q, k: [1, 8192, 8, 128]v: [1, 8192, 32, 128]g, beta: [1, 8192, 32]dtype=f16, bf16
• q35-397b-tp2-4096q, k: [1, 4096, 8, 128]v: [1, 4096, 32, 128]g, beta: [1, 4096, 32]dtype=f16, bf16
• q35-397b-tp2-2048q, k: [1, 2048, 8, 128]v: [1, 2048, 32, 128]g, beta: [1, 2048, 32]dtype=f16, bf16
• q35-397b-tp1-65536q, k: [1, 65536, 16, 128]v: [1, 65536, 64, 128]g, beta: [1, 65536, 64]dtype=f16, bf16
• q35-397b-tp1-32768q, k: [1, 32768, 16, 128]v: [1, 32768, 64, 128]g, beta: [1, 32768, 64]dtype=f16, bf16
• q35-397b-tp1-16384q, k: [1, 16384, 16, 128]v: [1, 16384, 64, 128]g, beta: [1, 16384, 64]dtype=f16, bf16
• q35-397b-tp1-8192q, k: [1, 8192, 16, 128]v: [1, 8192, 64, 128]g, beta: [1, 8192, 64]dtype=f16, bf16
• q35-397b-tp1-4096q, k: [1, 4096, 16, 128]v: [1, 4096, 64, 128]g, beta: [1, 4096, 64]dtype=f16, bf16
• q35-397b-tp1-2048q, k: [1, 2048, 16, 128]v: [1, 2048, 64, 128]g, beta: [1, 2048, 64]dtype=f16, bf16
• q35-35b-65536q, k: [1, 65536, 16, 128]v: [1, 65536, 32, 128]g, beta: [1, 65536, 32]dtype=f16, bf16
• q35-35b-32768q, k: [1, 32768, 16, 128]v: [1, 32768, 32, 128]g, beta: [1, 32768, 32]dtype=f16, bf16
• q35-35b-16384q, k: [1, 16384, 16, 128]v: [1, 16384, 32, 128]g, beta: [1, 16384, 32]dtype=f16, bf16
• q35-35b-8192q, k: [1, 8192, 16, 128]v: [1, 8192, 32, 128]g, beta: [1, 8192, 32]dtype=f16, bf16
• q35-35b-4096q, k: [1, 4096, 16, 128]v: [1, 4096, 32, 128]g, beta: [1, 4096, 32]dtype=f16, bf16
• q35-35b-2048q, k: [1, 2048, 16, 128]v: [1, 2048, 32, 128]g, beta: [1, 2048, 32]dtype=f16, bf16
• q35-27b-65536q, k: [1, 65536, 16, 128]v: [1, 65536, 48, 128]g, beta: [1, 65536, 48]dtype=f16, bf16
• q35-27b-32768q, k: [1, 32768, 16, 128]v: [1, 32768, 48, 128]g, beta: [1, 32768, 48]dtype=f16, bf16
• q35-27b-16384q, k: [1, 16384, 16, 128]v: [1, 16384, 48, 128]g, beta: [1, 16384, 48]dtype=f16, bf16
• q35-27b-8192q, k: [1, 8192, 16, 128]v: [1, 8192, 48, 128]g, beta: [1, 8192, 48]dtype=f16, bf16
• q35-27b-4096q, k: [1, 4096, 16, 128]v: [1, 4096, 48, 128]g, beta: [1, 4096, 48]dtype=f16, bf16
• q35-27b-2048q, k: [1, 2048, 16, 128]v: [1, 2048, 48, 128]g, beta: [1, 2048, 48]dtype=f16, bf16
• q35-2b-65536q, k, v: [1, 65536, 16, 128]g, beta: [1, 65536, 16]dtype=f16, bf16
• q35-2b-16384q, k, v: [1, 16384, 16, 128]g, beta: [1, 16384, 16]dtype=f16, bf16
• q35-2b-8192q, k, v: [1, 8192, 16, 128]g, beta: [1, 8192, 16]dtype=f16, bf16
• q35-2b-2048q, k, v: [1, 2048, 16, 128]g, beta: [1, 2048, 16]dtype=f16, bf16
• q35-symmetric-65536q, k, v: [1, 65536, 32, 128]g, beta: [1, 65536, 32]dtype=f16, bf16
• q35-symmetric-32768q, k, v: [1, 32768, 32, 128]g, beta: [1, 32768, 32]dtype=f16, bf16
• q35-symmetric-16384q, k, v: [1, 16384, 32, 128]g, beta: [1, 16384, 32]dtype=f16, bf16
• q35-symmetric-8192q, k, v: [1, 8192, 32, 128]g, beta: [1, 8192, 32]dtype=f16, bf16
• q35-symmetric-4096q, k, v: [1, 4096, 32, 128]g, beta: [1, 4096, 32]dtype=f16, bf16
• q35-symmetric-2048q, k, v: [1, 2048, 32, 128]g, beta: [1, 2048, 32]dtype=f16, bf16

q, k: [B, S, H, K]v: [B, S, HV, V]g, beta: [B, S, HV]initial_state: [B, HV, K, V], f32

• decode-b1q, k, v: [1, 1, 16, 128]g, beta: [1, 1, 16]initial_state: [1, 16, 128, 128], f32dtype=f16, bf16
• decode-b8q, k, v: [8, 1, 32, 128]g, beta: [8, 1, 32]initial_state: [8, 32, 128, 128], f32dtype=f16, bf16
• prefill-continuedq, k, v: [1, 4096, 16, 128]g, beta: [1, 4096, 16]initial_state: [1, 16, 128, 128], f32dtype=bf16
• decode-beta-l2normq, k, v: [2, 1, 4, 128]g, beta: [2, 1, 4]initial_state: [2, 4, 128, 128], f32use_qk_l2norm_in_kernel=trueuse_beta_sigmoid_in_kernel=trueallow_neg_eigval=truedtype=bf16
• q35-gva-decode-b1q, k: [1, 1, 16, 128]v: [1, 1, 32, 128]g, beta: [1, 1, 32]initial_state: [1, 32, 128, 128], f32dtype=f16, bf16
• q35-gva-decode-b4q, k: [4, 1, 16, 128]v: [4, 1, 32, 128]g, beta: [4, 1, 32]initial_state: [4, 32, 128, 128], f32dtype=f16, bf16
• q35-gva-decode-b8q, k: [8, 1, 16, 128]v: [8, 1, 32, 128]g, beta: [8, 1, 32]initial_state: [8, 32, 128, 128], f32dtype=f16, bf16
• q35-gva-decode-b16q, k: [16, 1, 16, 128]v: [16, 1, 32, 128]g, beta: [16, 1, 32]initial_state: [16, 32, 128, 128], f32dtype=f16, bf16
• q35-gva-decode-b32q, k: [32, 1, 16, 128]v: [32, 1, 32, 128]g, beta: [32, 1, 32]initial_state: [32, 32, 128, 128], f32dtype=f16, bf16
• q35-gva-decode-b64q, k: [64, 1, 16, 128]v: [64, 1, 32, 128]g, beta: [64, 1, 32]initial_state: [64, 32, 128, 128], f32dtype=f16, bf16
• q35-gva-decode-b128q, k: [128, 1, 16, 128]v: [128, 1, 32, 128]g, beta: [128, 1, 32]initial_state: [128, 32, 128, 128], f32dtype=f16, bf16
• q35-gva-decode-b256q, k: [256, 1, 16, 128]v: [256, 1, 32, 128]g, beta: [256, 1, 32]initial_state: [256, 32, 128, 128], f32dtype=f16, bf16
• q35-gva-decode-b512q, k: [512, 1, 16, 128]v: [512, 1, 32, 128]g, beta: [512, 1, 32]initial_state: [512, 32, 128, 128], f32dtype=f16, bf16

q, k: [B, S, H, K]v: [B, S, HV, V]g, beta: [B, S, HV]cu_seqlens, cu_seqlens_cpu: [N + 1], i64

• varlen-4x1kq, k, v: [1, 4096, 16, 128]g, beta: [1, 4096, 16]cu_seqlens, cu_seqlens_cpu: [5], i64seq_lens=repeat(1024, 4)dtype=bf16
• q35-397b-tp8-6144-2048q, k: [1, 8192, 2, 128]v: [1, 8192, 8, 128]g, beta: [1, 8192, 8]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=[6144,2048]dtype=f16, bf16
• q35-397b-tp8-4096-4096q, k: [1, 8192, 2, 128]v: [1, 8192, 8, 128]g, beta: [1, 8192, 8]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=repeat(4096, 2)dtype=f16, bf16
• q35-397b-tp8-2048-6144q, k: [1, 8192, 2, 128]v: [1, 8192, 8, 128]g, beta: [1, 8192, 8]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=[2048,6144]dtype=f16, bf16
• q35-397b-tp8-1024-7168q, k: [1, 8192, 2, 128]v: [1, 8192, 8, 128]g, beta: [1, 8192, 8]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=[1024,7168]dtype=f16, bf16
• q35-397b-tp8-4x2048q, k: [1, 8192, 2, 128]v: [1, 8192, 8, 128]g, beta: [1, 8192, 8]cu_seqlens, cu_seqlens_cpu: [5], i64seq_lens=repeat(2048, 4)dtype=f16, bf16
• q35-397b-tp8-8x1024q, k: [1, 8192, 2, 128]v: [1, 8192, 8, 128]g, beta: [1, 8192, 8]cu_seqlens, cu_seqlens_cpu: [9], i64seq_lens=repeat(1024, 8)dtype=f16, bf16
• q35-397b-tp8-8x8192q, k: [1, 65536, 2, 128]v: [1, 65536, 8, 128]g, beta: [1, 65536, 8]cu_seqlens, cu_seqlens_cpu: [9], i64seq_lens=repeat(8192, 8)dtype=f16, bf16
• q35-397b-tp8-16x8192q, k: [1, 131072, 2, 128]v: [1, 131072, 8, 128]g, beta: [1, 131072, 8]cu_seqlens, cu_seqlens_cpu: [17], i64seq_lens=repeat(8192, 16)dtype=f16, bf16
• q35-397b-tp8-32x8192q, k: [1, 262144, 2, 128]v: [1, 262144, 8, 128]g, beta: [1, 262144, 8]cu_seqlens, cu_seqlens_cpu: [33], i64seq_lens=repeat(8192, 32)dtype=f16, bf16
• q35-397b-tp4-6144-2048q, k: [1, 8192, 4, 128]v: [1, 8192, 16, 128]g, beta: [1, 8192, 16]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=[6144,2048]dtype=f16, bf16
• q35-397b-tp4-4096-4096q, k: [1, 8192, 4, 128]v: [1, 8192, 16, 128]g, beta: [1, 8192, 16]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=repeat(4096, 2)dtype=f16, bf16
• q35-397b-tp4-2048-6144q, k: [1, 8192, 4, 128]v: [1, 8192, 16, 128]g, beta: [1, 8192, 16]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=[2048,6144]dtype=f16, bf16
• q35-397b-tp4-1024-7168q, k: [1, 8192, 4, 128]v: [1, 8192, 16, 128]g, beta: [1, 8192, 16]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=[1024,7168]dtype=f16, bf16
• q35-397b-tp4-4x2048q, k: [1, 8192, 4, 128]v: [1, 8192, 16, 128]g, beta: [1, 8192, 16]cu_seqlens, cu_seqlens_cpu: [5], i64seq_lens=repeat(2048, 4)dtype=f16, bf16
• q35-397b-tp4-8x1024q, k: [1, 8192, 4, 128]v: [1, 8192, 16, 128]g, beta: [1, 8192, 16]cu_seqlens, cu_seqlens_cpu: [9], i64seq_lens=repeat(1024, 8)dtype=f16, bf16
• q35-397b-tp4-8x8192q, k: [1, 65536, 4, 128]v: [1, 65536, 16, 128]g, beta: [1, 65536, 16]cu_seqlens, cu_seqlens_cpu: [9], i64seq_lens=repeat(8192, 8)dtype=f16, bf16
• q35-397b-tp4-16x8192q, k: [1, 131072, 4, 128]v: [1, 131072, 16, 128]g, beta: [1, 131072, 16]cu_seqlens, cu_seqlens_cpu: [17], i64seq_lens=repeat(8192, 16)dtype=f16, bf16
• q35-397b-tp4-32x8192q, k: [1, 262144, 4, 128]v: [1, 262144, 16, 128]g, beta: [1, 262144, 16]cu_seqlens, cu_seqlens_cpu: [33], i64seq_lens=repeat(8192, 32)dtype=f16, bf16
• q35-397b-tp2-6144-2048q, k: [1, 8192, 8, 128]v: [1, 8192, 32, 128]g, beta: [1, 8192, 32]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=[6144,2048]dtype=f16, bf16
• q35-397b-tp2-4096-4096q, k: [1, 8192, 8, 128]v: [1, 8192, 32, 128]g, beta: [1, 8192, 32]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=repeat(4096, 2)dtype=f16, bf16
• q35-397b-tp2-2048-6144q, k: [1, 8192, 8, 128]v: [1, 8192, 32, 128]g, beta: [1, 8192, 32]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=[2048,6144]dtype=f16, bf16
• q35-397b-tp2-1024-7168q, k: [1, 8192, 8, 128]v: [1, 8192, 32, 128]g, beta: [1, 8192, 32]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=[1024,7168]dtype=f16, bf16
• q35-397b-tp2-4x2048q, k: [1, 8192, 8, 128]v: [1, 8192, 32, 128]g, beta: [1, 8192, 32]cu_seqlens, cu_seqlens_cpu: [5], i64seq_lens=repeat(2048, 4)dtype=f16, bf16
• q35-397b-tp2-8x1024q, k: [1, 8192, 8, 128]v: [1, 8192, 32, 128]g, beta: [1, 8192, 32]cu_seqlens, cu_seqlens_cpu: [9], i64seq_lens=repeat(1024, 8)dtype=f16, bf16
• q35-397b-tp2-8x8192q, k: [1, 65536, 8, 128]v: [1, 65536, 32, 128]g, beta: [1, 65536, 32]cu_seqlens, cu_seqlens_cpu: [9], i64seq_lens=repeat(8192, 8)dtype=f16, bf16
• q35-397b-tp2-16x8192q, k: [1, 131072, 8, 128]v: [1, 131072, 32, 128]g, beta: [1, 131072, 32]cu_seqlens, cu_seqlens_cpu: [17], i64seq_lens=repeat(8192, 16)dtype=f16, bf16
• q35-397b-tp2-32x8192q, k: [1, 262144, 8, 128]v: [1, 262144, 32, 128]g, beta: [1, 262144, 32]cu_seqlens, cu_seqlens_cpu: [33], i64seq_lens=repeat(8192, 32)dtype=f16, bf16
• q35-397b-tp1-6144-2048q, k: [1, 8192, 16, 128]v: [1, 8192, 64, 128]g, beta: [1, 8192, 64]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=[6144,2048]dtype=f16, bf16
• q35-397b-tp1-4096-4096q, k: [1, 8192, 16, 128]v: [1, 8192, 64, 128]g, beta: [1, 8192, 64]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=repeat(4096, 2)dtype=f16, bf16
• q35-397b-tp1-2048-6144q, k: [1, 8192, 16, 128]v: [1, 8192, 64, 128]g, beta: [1, 8192, 64]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=[2048,6144]dtype=f16, bf16
• q35-397b-tp1-1024-7168q, k: [1, 8192, 16, 128]v: [1, 8192, 64, 128]g, beta: [1, 8192, 64]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=[1024,7168]dtype=f16, bf16
• q35-397b-tp1-4x2048q, k: [1, 8192, 16, 128]v: [1, 8192, 64, 128]g, beta: [1, 8192, 64]cu_seqlens, cu_seqlens_cpu: [5], i64seq_lens=repeat(2048, 4)dtype=f16, bf16
• q35-397b-tp1-8x1024q, k: [1, 8192, 16, 128]v: [1, 8192, 64, 128]g, beta: [1, 8192, 64]cu_seqlens, cu_seqlens_cpu: [9], i64seq_lens=repeat(1024, 8)dtype=f16, bf16
• q35-397b-tp1-8x8192q, k: [1, 65536, 16, 128]v: [1, 65536, 64, 128]g, beta: [1, 65536, 64]cu_seqlens, cu_seqlens_cpu: [9], i64seq_lens=repeat(8192, 8)dtype=f16, bf16
• q35-397b-tp1-16x8192q, k: [1, 131072, 16, 128]v: [1, 131072, 64, 128]g, beta: [1, 131072, 64]cu_seqlens, cu_seqlens_cpu: [17], i64seq_lens=repeat(8192, 16)dtype=f16, bf16
• q35-397b-tp1-32x8192q, k: [1, 262144, 16, 128]v: [1, 262144, 64, 128]g, beta: [1, 262144, 64]cu_seqlens, cu_seqlens_cpu: [33], i64seq_lens=repeat(8192, 32)dtype=f16, bf16
• q35-35b-6144-2048q, k: [1, 8192, 16, 128]v: [1, 8192, 32, 128]g, beta: [1, 8192, 32]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=[6144,2048]dtype=f16, bf16
• q35-35b-4096-4096q, k: [1, 8192, 16, 128]v: [1, 8192, 32, 128]g, beta: [1, 8192, 32]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=repeat(4096, 2)dtype=f16, bf16
• q35-35b-2048-6144q, k: [1, 8192, 16, 128]v: [1, 8192, 32, 128]g, beta: [1, 8192, 32]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=[2048,6144]dtype=f16, bf16
• q35-35b-1024-7168q, k: [1, 8192, 16, 128]v: [1, 8192, 32, 128]g, beta: [1, 8192, 32]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=[1024,7168]dtype=f16, bf16
• q35-35b-4x2048q, k: [1, 8192, 16, 128]v: [1, 8192, 32, 128]g, beta: [1, 8192, 32]cu_seqlens, cu_seqlens_cpu: [5], i64seq_lens=repeat(2048, 4)dtype=f16, bf16
• q35-35b-8x1024q, k: [1, 8192, 16, 128]v: [1, 8192, 32, 128]g, beta: [1, 8192, 32]cu_seqlens, cu_seqlens_cpu: [9], i64seq_lens=repeat(1024, 8)dtype=f16, bf16
• q35-35b-8x8192q, k: [1, 65536, 16, 128]v: [1, 65536, 32, 128]g, beta: [1, 65536, 32]cu_seqlens, cu_seqlens_cpu: [9], i64seq_lens=repeat(8192, 8)dtype=f16, bf16
• q35-35b-16x8192q, k: [1, 131072, 16, 128]v: [1, 131072, 32, 128]g, beta: [1, 131072, 32]cu_seqlens, cu_seqlens_cpu: [17], i64seq_lens=repeat(8192, 16)dtype=f16, bf16
• q35-35b-32x8192q, k: [1, 262144, 16, 128]v: [1, 262144, 32, 128]g, beta: [1, 262144, 32]cu_seqlens, cu_seqlens_cpu: [33], i64seq_lens=repeat(8192, 32)dtype=f16, bf16
• q35-27b-6144-2048q, k: [1, 8192, 16, 128]v: [1, 8192, 48, 128]g, beta: [1, 8192, 48]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=[6144,2048]dtype=f16, bf16
• q35-27b-4096-4096q, k: [1, 8192, 16, 128]v: [1, 8192, 48, 128]g, beta: [1, 8192, 48]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=repeat(4096, 2)dtype=f16, bf16
• q35-27b-2048-6144q, k: [1, 8192, 16, 128]v: [1, 8192, 48, 128]g, beta: [1, 8192, 48]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=[2048,6144]dtype=f16, bf16
• q35-27b-1024-7168q, k: [1, 8192, 16, 128]v: [1, 8192, 48, 128]g, beta: [1, 8192, 48]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=[1024,7168]dtype=f16, bf16
• q35-27b-4x2048q, k: [1, 8192, 16, 128]v: [1, 8192, 48, 128]g, beta: [1, 8192, 48]cu_seqlens, cu_seqlens_cpu: [5], i64seq_lens=repeat(2048, 4)dtype=f16, bf16
• q35-27b-8x1024q, k: [1, 8192, 16, 128]v: [1, 8192, 48, 128]g, beta: [1, 8192, 48]cu_seqlens, cu_seqlens_cpu: [9], i64seq_lens=repeat(1024, 8)dtype=f16, bf16
• q35-27b-8x8192q, k: [1, 65536, 16, 128]v: [1, 65536, 48, 128]g, beta: [1, 65536, 48]cu_seqlens, cu_seqlens_cpu: [9], i64seq_lens=repeat(8192, 8)dtype=f16, bf16
• q35-27b-16x8192q, k: [1, 131072, 16, 128]v: [1, 131072, 48, 128]g, beta: [1, 131072, 48]cu_seqlens, cu_seqlens_cpu: [17], i64seq_lens=repeat(8192, 16)dtype=f16, bf16
• q35-27b-32x8192q, k: [1, 262144, 16, 128]v: [1, 262144, 48, 128]g, beta: [1, 262144, 48]cu_seqlens, cu_seqlens_cpu: [33], i64seq_lens=repeat(8192, 32)dtype=f16, bf16
• q35-2b-6144-2048q, k, v: [1, 8192, 16, 128]g, beta: [1, 8192, 16]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=[6144,2048]dtype=f16, bf16
• q35-2b-4096-4096q, k, v: [1, 8192, 16, 128]g, beta: [1, 8192, 16]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=repeat(4096, 2)dtype=f16, bf16
• q35-2b-2048-6144q, k, v: [1, 8192, 16, 128]g, beta: [1, 8192, 16]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=[2048,6144]dtype=f16, bf16
• q35-2b-1024-7168q, k, v: [1, 8192, 16, 128]g, beta: [1, 8192, 16]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=[1024,7168]dtype=f16, bf16
• q35-2b-4x2048q, k, v: [1, 8192, 16, 128]g, beta: [1, 8192, 16]cu_seqlens, cu_seqlens_cpu: [5], i64seq_lens=repeat(2048, 4)dtype=f16, bf16
• q35-2b-8x1024q, k, v: [1, 8192, 16, 128]g, beta: [1, 8192, 16]cu_seqlens, cu_seqlens_cpu: [9], i64seq_lens=repeat(1024, 8)dtype=f16, bf16
• q35-2b-8x8192q, k, v: [1, 65536, 16, 128]g, beta: [1, 65536, 16]cu_seqlens, cu_seqlens_cpu: [9], i64seq_lens=repeat(8192, 8)dtype=f16, bf16
• q35-2b-16x8192q, k, v: [1, 131072, 16, 128]g, beta: [1, 131072, 16]cu_seqlens, cu_seqlens_cpu: [17], i64seq_lens=repeat(8192, 16)dtype=f16, bf16
• q35-2b-32x8192q, k, v: [1, 262144, 16, 128]g, beta: [1, 262144, 16]cu_seqlens, cu_seqlens_cpu: [33], i64seq_lens=repeat(8192, 32)dtype=f16, bf16
• q35-symmetric-6144-2048q, k, v: [1, 8192, 32, 128]g, beta: [1, 8192, 32]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=[6144,2048]dtype=f16, bf16
• q35-symmetric-4096-4096q, k, v: [1, 8192, 32, 128]g, beta: [1, 8192, 32]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=repeat(4096, 2)dtype=f16, bf16
• q35-symmetric-2048-6144q, k, v: [1, 8192, 32, 128]g, beta: [1, 8192, 32]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=[2048,6144]dtype=f16, bf16
• q35-symmetric-1024-7168q, k, v: [1, 8192, 32, 128]g, beta: [1, 8192, 32]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=[1024,7168]dtype=f16, bf16
• q35-symmetric-4x2048q, k, v: [1, 8192, 32, 128]g, beta: [1, 8192, 32]cu_seqlens, cu_seqlens_cpu: [5], i64seq_lens=repeat(2048, 4)dtype=f16, bf16
• q35-symmetric-8x1024q, k, v: [1, 8192, 32, 128]g, beta: [1, 8192, 32]cu_seqlens, cu_seqlens_cpu: [9], i64seq_lens=repeat(1024, 8)dtype=f16, bf16
• q35-symmetric-8x8192q, k, v: [1, 65536, 32, 128]g, beta: [1, 65536, 32]cu_seqlens, cu_seqlens_cpu: [9], i64seq_lens=repeat(8192, 8)dtype=f16, bf16
• q35-symmetric-16x8192q, k, v: [1, 131072, 32, 128]g, beta: [1, 131072, 32]cu_seqlens, cu_seqlens_cpu: [17], i64seq_lens=repeat(8192, 16)dtype=f16, bf16
• q35-symmetric-32x8192q, k, v: [1, 262144, 32, 128]g, beta: [1, 262144, 32]cu_seqlens, cu_seqlens_cpu: [33], i64seq_lens=repeat(8192, 32)dtype=f16, bf16

q, k: [B, S, H, K]v: [B, S, HV, V]g, beta: [B, S, HV]initial_state: [N, HV, K, V], f32cu_seqlens: [N + 1], i64

• varlen-ragged-groupedq, k: [1, 4096, 4, 128]v: [1, 4096, 16, 128]g, beta: [1, 4096, 16]initial_state: [4, 16, 128, 128], f32cu_seqlens: [5], i64seq_lens=[1,63,1000,3032]dtype=f16

q, k: [B, S, H, K]v: [B, S, HV, V]g, beta: [B, S, HV]A_log, dt_bias: [HV], f32

• prefill-raw-4kq, k, v: [1, 4096, 16, 128]g, beta: [1, 4096, 16]A_log, dt_bias: [16], f32use_qk_l2norm_in_kernel=trueuse_beta_sigmoid_in_kernel=trueuse_gate_in_kernel=truedtype=bf16

q, k: [B, S, H, K]v: [B, S, HV, V]g, beta: [B, S, HV]cu_seqlens, cu_seqlens_cpu: [N + 1], i64A_log, dt_bias: [HV], f32

• prefill-raw-raggedq, k: [1, 3000, 6, 64]v: [1, 3000, 24, 64]g, beta: [1, 3000, 24]cu_seqlens, cu_seqlens_cpu: [5], i64A_log, dt_bias: [24], f32seq_lens=[1,59,937,2003]use_qk_l2norm_in_kernel=trueuse_beta_sigmoid_in_kernel=trueuse_gate_in_kernel=truedtype=bf16

q, k: [B, S, H, K]v: [B, S, HV, V]g, beta: [B, S, HV]initial_state: [B, HV, K, V], f32A_log, dt_bias: [HV], f32

• decode-gate-fusedq, k: [32, 1, 8, 64]v: [32, 1, 16, 64]g, beta: [32, 1, 16]initial_state: [32, 16, 64, 64], f32A_log, dt_bias: [16], f32use_gate_in_kernel=truedtype=bf16

q, k: [B, S, H, K]v: [B, S, HV, V]g, beta: [B, S, HV]initial_state: [B, HV, V, K], f32

state_v_first=truedtype=bf16

• decode-vfirst-b17q, k: [17, 1, 6, 128]v: [17, 1, 12, 128]g, beta: [17, 1, 12]initial_state: [17, 12, 128, 128], f32
• prefill-v-first-4kq, k, v: [1, 4096, 16, 128]g, beta: [1, 4096, 16]initial_state: [1, 16, 128, 128], f32

q, k: [B, S, H, K]v: [B, S, HV, V]g, beta: [B, S, HV]initial_state: [N, HV, V, K], f32cu_seqlens, cu_seqlens_cpu: [N + 1], i64

• varlen-v-first-raggedq, k: [1, 4099, 4, 128]v: [1, 4099, 16, 128]g, beta: [1, 4099, 16]initial_state: [4, 16, 128, 128], f32cu_seqlens, cu_seqlens_cpu: [5], i64seq_lens=[1,67,1031,3000]state_v_first=truedtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
prefill-4k	f16	1.76×	0.0785	flashinfer fla	0.1380 0.1966	95.7	19%	mem
	bf16	1.72×	0.0807	flashinfer fla	0.1387 0.1989	93.1	19%	mem
prefill-12k	f16	1.18×	0.2032	flashinfer fla	0.2397 0.5589	111	22%	mem
	bf16	1.15×	0.2083	flashinfer fla	0.2401 0.5613	108	22%	mem
prefill-32k	f16	1.34×	0.3694	flashinfer fla	0.4950 1.4625	163	32%	mem
	bf16	1.30×	0.3816	flashinfer fla	0.4965 1.4683	158	31%	mem
prefill-narrow-state	bf16	2.32×	0.0553	fla flashinfer	0.1281 0.1994	34	14%	mem
prefill-3000	bf16	1.67×	0.0730	flashinfer fla	0.1222 0.1578	75.4	15%	mem
decode-fresh	bf16	1.88×	0.00547	fla flashinfer flashinfer-decode	0.0103 0.0552 0.0573	5.36	69%	mem
prefill-v-first-narrow	bf16	1.96×	0.0510	fla flashinfer	0.1001 0.1648	27	11%	mem
q35-397b-tp8-65536	f16	1.11×	0.4340	flashinfer fla	0.4827 2.1027	139	19%	comp
	bf16	1.09×	0.4495	flashinfer fla	0.4878 2.1034	134	18%	comp
q35-397b-tp8-32768	f16	0.99×	0.2898	flashinfer fla	0.2876 1.0593	104	14%	comp
	bf16	0.97×	0.2976	flashinfer fla	0.2900 1.0597	101	13%	comp
q35-397b-tp8-16384	f16	1.25×	0.1507	flashinfer fla	0.1888 0.5359	99.8	14%	comp
	bf16	1.22×	0.1557	flashinfer fla	0.1903 0.5355	96.6	13%	comp
q35-397b-tp8-8192	f16	1.27×	0.1104	flashinfer fla	0.1403 0.2765	68.1	9%	comp
	bf16	1.25×	0.1128	flashinfer fla	0.1410 0.2765	66.6	9%	comp
q35-397b-tp8-4096	f16	1.66×	0.0642	flashinfer fla	0.1068 0.1550	58.5	8%	comp
	bf16	1.66×	0.0651	flashinfer fla	0.1078 0.1550	57.7	8%	comp
q35-397b-tp8-2048	f16	1.83×	0.0479	flashinfer fla	0.0878 0.0956	39.2	5%	comp
	bf16	1.83×	0.0484	flashinfer fla	0.0887 0.0957	38.8	5%	comp
q35-397b-tp4-65536	f16	1.09×	0.7975	flashinfer fla	0.8683 2.8185	151	21%	comp
	bf16	1.06×	0.8229	flashinfer fla	0.8750 2.8252	146	19%	comp
q35-397b-tp4-32768	f16	1.18×	0.4086	flashinfer fla	0.4817 1.4122	147	20%	comp
	bf16	1.15×	0.4228	flashinfer fla	0.4854 1.4138	142	19%	comp
q35-397b-tp4-16384	f16	1.07×	0.2635	flashinfer fla	0.2828 0.7120	114	16%	comp
	bf16	1.05×	0.2715	flashinfer fla	0.2841 0.7124	111	15%	comp
q35-397b-tp4-8192	f16	1.32×	0.1380	flashinfer fla	0.1818 0.3638	109	15%	comp
	bf16	1.28×	0.1424	flashinfer fla	0.1827 0.3641	106	14%	comp
q35-397b-tp4-4096	f16	1.36×	0.0976	flashinfer fla	0.1328 0.1901	77	11%	comp
	bf16	1.34×	0.0999	flashinfer fla	0.1336 0.1906	75.2	10%	comp
q35-397b-tp4-2048	f16	1.82×	0.0575	flashinfer fla	0.1047 0.1110	65.4	9%	comp
	bf16	1.82×	0.0580	flashinfer fla	0.1056 0.1113	64.8	9%	comp
q35-397b-tp2-65536	f16	1.22×	1.3459	flashinfer fla	1.6460 4.2889	179	25%	comp
	bf16	1.19×	1.3947	flashinfer fla	1.6582 4.2905	172	23%	comp
q35-397b-tp2-32768	f16	1.12×	0.7717	flashinfer fla	0.8661 2.1326	156	21%	comp
	bf16	1.10×	0.7965	flashinfer fla	0.8732 2.1330	151	20%	comp
q35-397b-tp2-16384	f16	1.21×	0.3953	flashinfer fla	0.4766 1.0655	152	21%	comp
	bf16	1.17×	0.4097	flashinfer fla	0.4809 1.0661	147	19%	comp
q35-397b-tp2-8192	f16	1.11×	0.2506	flashinfer fla	0.2771 0.5456	120	16%	comp
	bf16	1.08×	0.2581	flashinfer fla	0.2782 0.5455	117	15%	comp
q35-397b-tp2-4096	f16	1.33×	0.1320	flashinfer fla	0.1753 0.2807	114	16%	comp
	bf16	1.29×	0.1362	flashinfer fla	0.1763 0.2809	110	15%	comp
q35-397b-tp2-2048	f16	1.39×	0.0914	flashinfer fla	0.1270 0.1484	82.3	11%	comp
	bf16	1.36×	0.0940	flashinfer fla	0.1276 0.1487	79.9	11%	comp
q35-397b-tp1-65536	f16	1.20×	2.6499	flashinfer fla	3.1828 7.5602	182	25%	comp
	bf16	1.16×	2.7513	flashinfer fla	3.2018 7.5921	175	23%	comp
q35-397b-tp1-32768	f16	1.22×	1.3340	flashinfer fla	1.6328 3.7270	180	25%	comp
	bf16	1.18×	1.3906	flashinfer fla	1.6433 3.7498	173	23%	comp
q35-397b-tp1-16384	f16	1.13×	0.7620	flashinfer fla	0.8588 1.8364	158	22%	comp
	bf16	1.10×	0.7859	flashinfer fla	0.8633 1.8473	153	20%	comp
q35-397b-tp1-8192	f16	1.20×	0.3899	flashinfer fla	0.4697 0.9362	154	21%	comp
	bf16	1.16×	0.4058	flashinfer fla	0.4713 0.9407	148	19%	comp
q35-397b-tp1-4096	f16	1.27×	0.2126	flashinfer fla	0.2692 0.4830	141	19%	comp
	bf16	1.24×	0.2180	flashinfer fla	0.2704 0.4857	138	18%	comp
q35-397b-tp1-2048	f16	1.47×	0.1140	flashinfer fla	0.1679 0.2493	132	18%	comp
	bf16	1.45×	0.1168	flashinfer fla	0.1689 0.2505	129	17%	comp
q35-35b-65536	f16	1.23×	1.3507	flashinfer fla	1.6555 4.2921	178	27%	mem
	bf16	1.19×	1.3954	flashinfer fla	1.6656 4.3003	172	26%	mem
q35-35b-32768	f16	1.27×	0.6863	flashinfer fla	0.8743 2.1512	175	26%	mem
	bf16	1.24×	0.7114	flashinfer fla	0.8790 2.1540	169	25%	mem
q35-35b-16384	f16	1.21×	0.3979	flashinfer fla	0.4813 1.0816	151	23%	mem
	bf16	1.18×	0.4119	flashinfer fla	0.4843 1.0828	146	22%	mem
q35-35b-8192	f16	1.34×	0.2079	flashinfer fla	0.2795 0.5509	145	22%	mem
	bf16	1.31×	0.2150	flashinfer fla	0.2808 0.5515	140	21%	mem
q35-35b-4096	f16	1.33×	0.1336	flashinfer fla	0.1776 0.2837	113	17%	mem
	bf16	1.29×	0.1382	flashinfer fla	0.1786 0.2841	109	17%	mem
q35-35b-2048	f16	1.78×	0.0723	flashinfer fla	0.1284 0.1501	104	16%	mem
	bf16	1.72×	0.0750	flashinfer fla	0.1292 0.1503	100	16%	mem
q35-27b-65536	f16	1.45×	2.0538	flashinfer fla	2.9686 6.2266	176	24%	comp
	bf16	1.41×	2.1260	flashinfer fla	3.0019 6.2671	170	23%	mem
q35-27b-32768	f16	1.45×	1.0472	flashinfer fla	1.5225 3.0934	172	24%	comp
	bf16	1.42×	1.0843	flashinfer fla	1.5367 3.1138	166	22%	mem
q35-27b-16384	f16	1.32×	0.6051	flashinfer fla	0.7970 1.5342	149	20%	comp
	bf16	1.28×	0.6257	flashinfer fla	0.8035 1.5436	144	19%	mem
q35-27b-8192	f16	1.39×	0.3130	flashinfer fla	0.4341 0.7811	144	20%	comp
	bf16	1.35×	0.3254	flashinfer fla	0.4384 0.7860	139	19%	mem
q35-27b-4096	f16	1.26×	0.1992	flashinfer fla	0.2501 0.4030	113	16%	comp
	bf16	1.22×	0.2057	flashinfer fla	0.2512 0.4054	110	15%	mem
q35-27b-2048	f16	1.49×	0.1057	flashinfer fla	0.1573 0.2208	107	15%	mem
	bf16	1.45×	0.1095	flashinfer fla	0.1585 0.2224	103	14%	mem
q35-2b-65536	f16	1.25×	0.7106	flashinfer fla	0.8880 2.9054	169	34%	mem
	bf16	1.21×	0.7370	flashinfer fla	0.8937 2.9162	163	33%	mem
q35-2b-16384	f16	1.32×	0.2215	flashinfer fla	0.2916 0.7426	136	27%	mem
	bf16	1.28×	0.2286	flashinfer fla	0.2920 0.7461	132	26%	mem
q35-2b-8192	f16	1.55×	0.1207	flashinfer fla	0.1867 0.3779	125	25%	mem
	bf16	1.49×	0.1257	flashinfer fla	0.1875 0.3806	120	24%	mem
q35-2b-2048	f16	1.88×	0.0562	flashinfer fla	0.1057 0.1154	66.9	14%	mem
	bf16	1.86×	0.0573	flashinfer fla	0.1064 0.1169	65.6	13%	mem
q35-symmetric-65536	f16	1.32×	1.2791	flashinfer fla	1.6845 4.3861	188	37%	mem
	bf16	1.28×	1.3254	flashinfer fla	1.6950 4.3913	181	36%	mem
q35-symmetric-32768	f16	1.28×	0.7000	flashinfer fla	0.8944 2.2123	172	34%	mem
	bf16	1.24×	0.7234	flashinfer fla	0.8956 2.2161	166	33%	mem
q35-symmetric-16384	f16	1.36×	0.3626	flashinfer fla	0.4934 1.1124	166	33%	mem
	bf16	1.32×	0.3754	flashinfer fla	0.4952 1.1155	160	32%	mem
q35-symmetric-8192	f16	1.34×	0.2152	flashinfer fla	0.2875 0.5676	140	28%	mem
	bf16	1.30×	0.2225	flashinfer fla	0.2889 0.5688	135	27%	mem
q35-symmetric-4096	f16	1.55×	0.1178	flashinfer fla	0.1824 0.2925	128	26%	mem
	bf16	1.49×	0.1228	flashinfer fla	0.1830 0.2934	122	25%	mem
q35-symmetric-2048	f16	1.77×	0.0747	flashinfer fla	0.1325 0.1545	101	21%	mem
	bf16	1.72×	0.0772	flashinfer fla	0.1331 0.1548	97.3	20%	mem
decode-b1	f16	1.29×	0.00323	fla flashinfer-decode flashinfer	0.00416 0.0374 0.0532	0.568	15%	lat
	bf16	1.29×	0.00323	fla flashinfer-decode flashinfer	0.00416 0.0239 0.0538	0.568	15%	lat
decode-b8	f16	1.42×	0.0109	fla flashinfer flashinfer-decode	0.0154 0.0877 0.0995	2.7	69%	mem
	bf16	1.41×	0.0109	fla flashinfer-decode flashinfer	0.0154 0.0826 0.0882	2.69	69%	mem
prefill-continued	bf16	1.78×	0.0822	flashinfer fla	0.1465 0.2004	91.5	19%	mem
decode-beta-l2norm	bf16	2.62×	0.00301	fla flashinfer	0.00787 0.0810	0.307	8%	lat
q35-gva-decode-b1	f16	1.29×	0.00333	fla flashinfer-decode flashinfer	0.00429 0.0419 0.0622	1.1	28%	lat
	bf16	1.28×	0.00333	fla flashinfer-decode flashinfer	0.00427 0.0275 0.0628	1.1	28%	lat
q35-gva-decode-b4	f16	1.30×	0.00614	fla flashinfer flashinfer-decode	0.008 0.0500 0.0660	2.39	61%	mem
	bf16	1.31×	0.00611	fla flashinfer flashinfer-decode	0.008 0.0500 0.0501	2.4	61%	mem
q35-gva-decode-b8	f16	1.44×	0.0103	fla flashinfer flashinfer-decode	0.0148 0.0880 0.0995	2.86	73%	mem
	bf16	1.42×	0.0104	fla flashinfer-decode flashinfer	0.0147 0.0825 0.0881	2.83	72%	mem
q35-gva-decode-b16	f16	1.34×	0.0199	fla flashinfer flashinfer-decode	0.0268 0.1648 0.1651	2.95	75%	mem
	bf16	1.34×	0.0200	fla flashinfer-decode flashinfer	0.0267 0.1479 0.1646	2.94	75%	mem
q35-gva-decode-b32	f16	1.33×	0.0377	fla flashinfer-decode flashinfer	0.0501 0.2879 0.3080	3.12	80%	mem
	bf16	1.33×	0.0378	fla flashinfer-decode flashinfer	0.0501 0.2705 0.3087	3.11	79%	mem
q35-gva-decode-b64	f16	1.33×	0.0721	fla flashinfer-decode flashinfer	0.0958 0.5322 0.5944	3.26	83%	mem
	bf16	1.33×	0.0722	fla flashinfer-decode flashinfer	0.0958 0.5140 0.5946	3.25	83%	mem
q35-gva-decode-b128	f16	1.35×	0.1396	fla flashinfer-decode flashinfer	0.1879 1.0191 1.1594	3.37	86%	mem
	bf16	1.35×	0.1393	fla flashinfer-decode flashinfer	0.1879 0.9999 1.1606	3.37	86%	mem
q35-gva-decode-b256	f16	1.35×	0.2752	fla flashinfer-decode flashinfer	0.3717 1.9922 2.2841	3.41	87%	mem
	bf16	1.35×	0.2750	fla flashinfer-decode flashinfer	0.3718 1.9724 2.2846	3.42	87%	mem
q35-gva-decode-b512	f16	1.35×	0.5465	fla flashinfer-decode flashinfer	0.7400 3.9443 4.5313	3.44	88%	mem
	bf16	1.35×	0.5464	fla flashinfer-decode flashinfer	0.7397 3.9144 4.5335	3.44	88%	mem
varlen-4x1k	bf16	1.82×	0.0828	flashinfer fla	0.1503 0.1521	90.8	19%	mem
q35-397b-tp8-6144-2048	f16	1.30×	0.1055	flashinfer fla	0.1373 0.2477	71.2	10%	comp
	bf16	1.28×	0.1081	flashinfer fla	0.1380 0.2485	69.5	9%	comp
q35-397b-tp8-4096-4096	f16	1.41×	0.0986	flashinfer fla	0.1386 0.2026	76.2	10%	comp
	bf16	1.37×	0.1013	flashinfer fla	0.1390 0.2032	74.2	10%	comp
q35-397b-tp8-2048-6144	f16	1.33×	0.1048	flashinfer fla	0.1393 0.2477	71.8	10%	comp
	bf16	1.31×	0.1073	flashinfer fla	0.1403 0.2485	70.1	9%	comp
q35-397b-tp8-1024-7168	f16	1.29×	0.1082	flashinfer fla	0.1401 0.2710	69.5	10%	comp
	bf16	1.27×	0.1107	flashinfer fla	0.1408 0.2718	67.9	9%	comp
q35-397b-tp8-4x2048	f16	1.69×	0.0922	flashinfer fla	0.1557 0.1579	81.6	11%	comp
	bf16	1.65×	0.0946	flashinfer fla	0.1563 0.1581	79.5	10%	comp
q35-397b-tp8-8x1024	f16	1.67×	0.0969	fla flashinfer	0.1618 0.1848	77.5	11%	comp
	bf16	1.62×	0.0998	fla flashinfer	0.1620 0.1861	75.3	10%	mem
q35-397b-tp8-8x8192	f16	1.38×	0.4061	flashinfer fla	0.5621 1.1029	148	20%	comp
	bf16	1.35×	0.4205	flashinfer fla	0.5693 1.1063	143	19%	comp
q35-397b-tp8-16x8192	f16	0.59×	0.7969	flashinfer fla	0.4664 2.1939	151	21%	comp
	bf16	0.57×	0.8190	flashinfer fla	0.4684 2.1970	147	19%	comp
q35-397b-tp8-32x8192	f16	0.66×	1.3950	flashinfer fla	0.9195 4.3303	172	24%	comp
	bf16	0.64×	1.4508	flashinfer fla	0.9224 4.3350	166	22%	comp
q35-397b-tp4-6144-2048	f16	1.32×	0.1369	flashinfer fla	0.1811 0.3381	110	15%	comp
	bf16	1.29×	0.1413	flashinfer fla	0.1824 0.3386	106	14%	comp
q35-397b-tp4-4096-4096	f16	1.35×	0.1346	flashinfer fla	0.1816 0.2925	112	15%	comp
	bf16	1.32×	0.1387	flashinfer fla	0.1832 0.2933	108	14%	comp
q35-397b-tp4-2048-6144	f16	1.33×	0.1372	flashinfer fla	0.1828 0.3383	110	15%	comp
	bf16	1.30×	0.1413	flashinfer fla	0.1839 0.3392	106	14%	comp
q35-397b-tp4-1024-7168	f16	1.31×	0.1392	flashinfer fla	0.1830 0.3609	108	15%	comp
	bf16	1.28×	0.1434	flashinfer fla	0.1840 0.3619	105	14%	comp
q35-397b-tp4-4x2048	f16	1.51×	0.1317	flashinfer fla	0.1982 0.2964	114	16%	comp
	bf16	1.47×	0.1358	flashinfer fla	0.1994 0.2972	111	15%	comp
q35-397b-tp4-8x1024	f16	0.63×	0.1376	flashinfer fla	0.0867 0.3028	109	15%	comp
	bf16	0.61×	0.1419	flashinfer fla	0.0869 0.3035	106	15%	mem
q35-397b-tp4-8x8192	f16	0.60×	0.7827	flashinfer fla	0.4675 2.1942	154	21%	comp
	bf16	0.58×	0.8058	flashinfer fla	0.4688 2.1970	149	20%	comp
q35-397b-tp4-16x8192	f16	0.65×	1.4066	flashinfer fla	0.9199 4.3882	171	23%	comp
	bf16	0.63×	1.4599	flashinfer fla	0.9243 4.3890	165	22%	comp
q35-397b-tp4-32x8192	f16	0.66×	2.7555	flashinfer fla	1.8187 8.7174	175	24%	comp
	bf16	0.64×	2.8563	flashinfer fla	1.8349 8.7277	168	22%	comp
q35-397b-tp2-6144-2048	f16	1.07×	0.2559	flashinfer fla	0.2731 0.5634	117	16%	comp
	bf16	1.04×	0.2635	flashinfer fla	0.2747 0.5653	114	15%	comp
q35-397b-tp2-4096-4096	f16	1.07×	0.2519	flashinfer fla	0.2696 0.5057	119	16%	comp
	bf16	1.05×	0.2592	flashinfer fla	0.2720 0.5070	116	15%	comp
q35-397b-tp2-2048-6144	f16	1.08×	0.2558	flashinfer fla	0.2765 0.5640	118	16%	comp
	bf16	1.06×	0.2636	flashinfer fla	0.2783 0.5653	114	15%	comp
q35-397b-tp2-1024-7168	f16	1.15×	0.2599	flashinfer fla	0.2986 0.5932	116	16%	comp
	bf16	1.12×	0.2673	flashinfer fla	0.2998 0.5941	112	15%	comp
q35-397b-tp2-4x2048	f16	0.58×	0.2474	flashinfer fla	0.1434 0.5108	122	17%	comp
	bf16	0.56×	0.2553	flashinfer fla	0.1436 0.5116	118	15%	comp
q35-397b-tp2-8x1024	f16	0.65×	0.2524	flashinfer fla	0.1630 0.5197	119	16%	comp
	bf16	0.63×	0.2606	flashinfer fla	0.1634 0.5207	115	16%	mem
q35-397b-tp2-8x8192	f16	0.67×	1.3859	flashinfer fla	0.9247 3.9198	174	24%	comp
	bf16	0.65×	1.4393	flashinfer fla	0.9312 3.9285	167	22%	comp
q35-397b-tp2-16x8192	f16	0.66×	2.7860	flashinfer fla	1.8370 7.8662	173	24%	comp
	bf16	0.64×	2.8843	flashinfer fla	1.8432 7.8708	167	22%	comp
q35-397b-tp2-32x8192	f16	0.69×	5.2570	flashinfer fla	3.6422 15.7322	183	25%	comp
	bf16	0.67×	5.4601	flashinfer fla	3.6678 15.7273	176	23%	comp
q35-397b-tp1-6144-2048	f16	0.89×	0.4023	flashinfer fla	0.3583 0.9784	149	20%	comp
	bf16	0.87×	0.4154	flashinfer fla	0.3602 0.9795	145	19%	comp
q35-397b-tp1-4096-4096	f16	0.63×	0.4038	flashinfer fla	0.2530 0.9787	149	20%	comp
	bf16	0.61×	0.4167	flashinfer fla	0.2541 0.9790	144	19%	comp
q35-397b-tp1-2048-6144	f16	0.89×	0.4047	flashinfer fla	0.3587 0.9787	149	20%	comp
	bf16	0.86×	0.4176	flashinfer fla	0.3610 0.9804	144	19%	comp
q35-397b-tp1-1024-7168	f16	1.02×	0.4038	flashinfer fla	0.4109 0.9793	149	20%	comp
	bf16	0.99×	0.4167	flashinfer fla	0.4139 0.9811	144	19%	comp
q35-397b-tp1-4x2048	f16	0.68×	0.4020	flashinfer fla	0.2722 0.9864	150	21%	comp
	bf16	0.66×	0.4165	flashinfer fla	0.2730 0.9878	144	19%	comp
q35-397b-tp1-8x1024	f16	0.78×	0.3950	flashinfer fla	0.3098 1.0033	152	21%	comp
	bf16	0.76×	0.4096	flashinfer fla	0.3107 1.0053	147	20%	mem
q35-397b-tp1-8x8192	f16	0.66×	2.7763	flashinfer fla	1.8361 7.8779	173	24%	comp
	bf16	0.65×	2.8641	flashinfer fla	1.8483 7.8871	168	22%	comp
q35-397b-tp1-16x8192	f16	0.69×	5.3117	flashinfer fla	3.6511 15.8174	181	25%	comp
	bf16	0.67×	5.4971	flashinfer fla	3.6768 15.8403	175	23%	comp
q35-397b-tp1-32x8192	f16	0.70×	10.4274	flashinfer fla	7.2814 32.1304	185	25%	comp
	bf16	0.67×	10.8544	flashinfer fla	7.3089 32.1754	177	23%	comp
q35-35b-6144-2048	f16	1.29×	0.2139	flashinfer fla	0.2766 0.5681	141	21%	mem
	bf16	1.25×	0.2225	flashinfer fla	0.2782 0.5702	135	21%	mem
q35-35b-4096-4096	f16	1.27×	0.2153	flashinfer fla	0.2727 0.5099	140	21%	mem
	bf16	1.23×	0.2237	flashinfer fla	0.2742 0.5118	134	21%	mem
q35-35b-2048-6144	f16	1.29×	0.2177	flashinfer fla	0.2798 0.5689	138	21%	mem
	bf16	1.25×	0.2258	flashinfer fla	0.2815 0.5706	133	20%	mem
q35-35b-1024-7168	f16	1.37×	0.2196	flashinfer fla	0.3007 0.5970	137	21%	mem
	bf16	1.32×	0.2276	flashinfer fla	0.3010 0.5987	132	20%	mem
q35-35b-4x2048	f16	0.67×	0.2151	flashinfer fla	0.1439 0.5158	140	22%	mem
	bf16	0.65×	0.2221	flashinfer fla	0.1442 0.5170	135	21%	mem
q35-35b-8x1024	f16	0.79×	0.2078	flashinfer fla	0.1639 0.5244	145	23%	mem
	bf16	0.76×	0.2158	flashinfer fla	0.1640 0.5265	139	23%	mem
q35-35b-8x8192	f16	0.66×	1.3968	flashinfer fla	0.9261 3.9314	172	26%	mem
	bf16	0.65×	1.4315	flashinfer fla	0.9323 3.9398	168	25%	mem
q35-35b-16x8192	f16	0.69×	2.6518	flashinfer fla	1.8335 7.8602	181	27%	mem
	bf16	0.67×	2.7503	flashinfer fla	1.8459 7.8803	175	26%	mem
q35-35b-32x8192	f16	0.70×	5.2276	flashinfer fla	3.6478 15.7222	184	28%	mem
	bf16	0.68×	5.4279	flashinfer fla	3.6706 15.7482	177	27%	mem
q35-27b-6144-2048	f16	1.08×	0.3241	flashinfer fla	0.3489 0.7701	139	19%	comp
	bf16	1.05×	0.3355	flashinfer fla	0.3507 0.7695	134	18%	mem
q35-27b-4096-4096	f16	0.75×	0.3266	flashinfer fla	0.2435 0.8380	138	19%	comp
	bf16	0.72×	0.3375	flashinfer fla	0.2445 0.8369	134	18%	mem
q35-27b-2048-6144	f16	1.07×	0.3267	flashinfer fla	0.3486 0.8304	138	19%	comp
	bf16	1.04×	0.3373	flashinfer fla	0.3503 0.8288	134	18%	mem
q35-27b-1024-7168	f16	1.23×	0.3265	flashinfer fla	0.4013 0.8290	138	19%	comp
	bf16	1.20×	0.3368	flashinfer fla	0.4038 0.8267	134	18%	mem
q35-27b-4x2048	f16	0.82×	0.3196	flashinfer fla	0.2621 0.7731	141	20%	mem
	bf16	0.79×	0.3315	flashinfer fla	0.2629 0.7716	136	19%	mem
q35-27b-8x1024	f16	0.77×	0.3071	flashinfer fla	0.2371 0.7865	147	21%	mem
	bf16	0.74×	0.3193	flashinfer fla	0.2376 0.7850	141	21%	mem
q35-27b-8x8192	f16	0.65×	2.1190	flashinfer fla	1.3735 6.0310	170	23%	comp
	bf16	0.63×	2.1832	flashinfer fla	1.3827 6.0268	165	22%	mem
q35-27b-16x8192	f16	0.68×	4.0057	flashinfer fla	2.7296 12.0840	180	25%	comp
	bf16	0.66×	4.1508	flashinfer fla	2.7463 12.0715	174	23%	mem
q35-27b-32x8192	f16	0.69×	7.9033	flashinfer fla	5.4325 24.3901	183	25%	comp
	bf16	0.67×	8.1813	flashinfer fla	5.4534 24.4015	176	24%	mem
q35-2b-6144-2048	f16	1.57×	0.1187	flashinfer fla	0.1867 0.3475	127	26%	mem
	bf16	1.52×	0.1233	flashinfer fla	0.1872 0.3876	122	25%	mem
q35-2b-4096-4096	f16	1.61×	0.1173	flashinfer fla	0.1887 0.3031	128	26%	mem
	bf16	1.56×	0.1213	flashinfer fla	0.1893 0.3292	124	25%	mem
q35-2b-2048-6144	f16	1.57×	0.1191	flashinfer fla	0.1867 0.3476	126	26%	mem
	bf16	1.52×	0.1237	flashinfer fla	0.1876 0.3873	122	25%	mem
q35-2b-1024-7168	f16	1.56×	0.1199	flashinfer fla	0.1868 0.3698	125	25%	mem
	bf16	1.50×	0.1247	flashinfer fla	0.1874 0.4165	121	24%	mem
q35-2b-4x2048	f16	1.75×	0.1154	flashinfer fla	0.2023 0.3063	130	27%	mem
	bf16	1.71×	0.1194	flashinfer fla	0.2036 0.2726	126	26%	mem
q35-2b-8x1024	f16	0.78×	0.1129	flashinfer fla	0.0876 0.3118	133	28%	mem
	bf16	0.75×	0.1167	flashinfer fla	0.0880 0.2802	129	27%	mem
q35-2b-8x8192	f16	0.65×	0.7216	flashinfer fla	0.4713 2.2448	167	33%	mem
	bf16	0.64×	0.7438	flashinfer fla	0.4732 1.9857	162	32%	mem
q35-2b-16x8192	f16	0.69×	1.3402	flashinfer fla	0.9261 4.4295	179	36%	mem
	bf16	0.67×	1.3827	flashinfer fla	0.9298 3.9106	174	35%	mem
q35-2b-32x8192	f16	0.70×	2.6215	flashinfer fla	1.8308 8.7885	183	37%	mem
	bf16	0.68×	2.7092	flashinfer fla	1.8355 7.7688	178	36%	mem
q35-symmetric-6144-2048	f16	1.30×	0.2199	flashinfer fla	0.2849 0.5835	137	28%	mem
	bf16	1.26×	0.2273	flashinfer fla	0.2859 0.5853	132	27%	mem
q35-symmetric-4096-4096	f16	1.27×	0.2209	flashinfer fla	0.2811 0.5262	136	28%	mem
	bf16	1.24×	0.2282	flashinfer fla	0.2827 0.5281	132	27%	mem
q35-symmetric-2048-6144	f16	1.30×	0.2227	flashinfer fla	0.2888 0.5837	135	27%	mem
	bf16	1.26×	0.2302	flashinfer fla	0.2897 0.5863	131	26%	mem
q35-symmetric-1024-7168	f16	1.37×	0.2255	flashinfer fla	0.3081 0.6118	133	27%	mem
	bf16	1.33×	0.2323	flashinfer fla	0.3093 0.6140	129	26%	mem
q35-symmetric-4x2048	f16	0.66×	0.2201	flashinfer fla	0.1457 0.5332	137	28%	mem
	bf16	0.64×	0.2275	flashinfer fla	0.1464 0.5344	132	27%	mem
q35-symmetric-8x1024	f16	0.78×	0.2123	flashinfer fla	0.1664 0.5420	142	30%	mem
	bf16	0.76×	0.2197	flashinfer fla	0.1669 0.5429	137	29%	mem
q35-symmetric-8x8192	f16	0.70×	1.3384	flashinfer fla	0.9320 3.9793	180	36%	mem
	bf16	0.68×	1.3766	flashinfer fla	0.9356 3.9901	175	35%	mem
q35-symmetric-16x8192	f16	0.69×	2.6499	flashinfer fla	1.8375 7.9190	182	36%	mem
	bf16	0.68×	2.7321	flashinfer fla	1.8510 7.9450	176	35%	mem
q35-symmetric-32x8192	f16	0.72×	5.0507	flashinfer fla	3.6513 15.8504	190	38%	mem
	bf16	0.70×	5.2099	flashinfer fla	3.6720 15.8576	185	37%	mem
varlen-ragged-grouped	f16	1.27×	0.1458	flashinfer fla	0.1845 0.2141	51.6	8%	mem
prefill-raw-4k	bf16	2.30×	0.1041	fla flashinfer	0.2398 0.4426	72.7	15%	mem
prefill-raw-ragged	bf16	1.63×	0.0997	fla flashinfer	0.1626 0.2541	20.8	6%	mem
decode-gate-fused	bf16	2.91×	0.00704	fla flashinfer-decode flashinfer	0.0205 0.1321 0.1722	2.09	54%	mem
decode-vfirst-b17	bf16	1.00×	0.00957	fla flashinfer-decode flashinfer	0.00955 0.0237 0.0304	2.45	62%	mem
prefill-v-first-4k	bf16	1.67×	0.0835	flashinfer fla	0.1392 0.2034	90.1	18%	mem
varlen-v-first-ragged	f16	1.44×	0.1179	flashinfer fla	0.1699 0.2150	63.8	10%	mem

KDAFwd

q, k: [B, S, H, K]v: [B, S, HV, V]g: [B, S, HV, K]beta: [B, S, HV]cu_seqlens, cu_seqlens_cpu: [N + 1], i64

use_qk_l2norm_in_kernel=true

• kda-tp1-4x1024q, k, v, g: [1, 4096, 32, 128]beta: [1, 4096, 32]cu_seqlens, cu_seqlens_cpu: [5], i64seq_lens=repeat(1024, 4)dtype=bf16, f16
• kda-tp1-1024-7168q, k, v, g: [1, 8192, 32, 128]beta: [1, 8192, 32]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=[1024,7168]dtype=bf16
• kda-tp1-16x8192q, k, v, g: [1, 131072, 32, 128]beta: [1, 131072, 32]cu_seqlens, cu_seqlens_cpu: [17], i64seq_lens=repeat(8192, 16)dtype=bf16
• kda-tp8-1024-7168q, k, v, g: [1, 8192, 4, 128]beta: [1, 8192, 4]cu_seqlens, cu_seqlens_cpu: [3], i64seq_lens=[1024,7168]dtype=bf16
• kda-tp8-16x8192q, k, v, g: [1, 131072, 4, 128]beta: [1, 131072, 4]cu_seqlens, cu_seqlens_cpu: [17], i64seq_lens=repeat(8192, 16)dtype=bf16

q, k: [B, S, H, K]v: [B, S, HV, V]g: [B, S, HV, K]beta: [B, S, HV]cu_seqlens: [N + 1], i64

• kda-tp1-4x8192q, k, v, g: [1, 32768, 32, 128]beta: [1, 32768, 32]cu_seqlens: [5], i64seq_lens=repeat(8192, 4)use_qk_l2norm_in_kernel=truedtype=bf16

q, k: [B, S, H, K]v: [B, S, HV, V]g: [B, S, HV, K]beta: [B, S, HV]initial_state: [B, HV, K, V], f32

use_qk_l2norm_in_kernel=truedtype=bf16

• kda-tp1-decode-b1q, k, v, g: [1, 1, 32, 128]beta: [1, 1, 32]initial_state: [1, 32, 128, 128], f32
• kda-tp1-decode-b64q, k, v, g: [64, 1, 32, 128]beta: [64, 1, 32]initial_state: [64, 32, 128, 128], f32
• kda-tp1-decode-b8q, k, v, g: [8, 1, 32, 128]beta: [8, 1, 32]initial_state: [8, 32, 128, 128], f32
• kda-tp1-decode-b32q, k, v, g: [32, 1, 32, 128]beta: [32, 1, 32]initial_state: [32, 32, 128, 128], f32
• kda-tp1-decode-b128q, k, v, g: [128, 1, 32, 128]beta: [128, 1, 32]initial_state: [128, 32, 128, 128], f32
• kda-tp1-decode-b256q, k, v, g: [256, 1, 32, 128]beta: [256, 1, 32]initial_state: [256, 32, 128, 128], f32
• kda-tp8-decode-b32q, k, v, g: [32, 1, 4, 128]beta: [32, 1, 4]initial_state: [32, 4, 128, 128], f32
• kda-tp8-decode-b256q, k, v, g: [256, 1, 4, 128]beta: [256, 1, 4]initial_state: [256, 4, 128, 128], f32

q, k: [B, S, H, K]v: [B, S, HV, V]g: [B, S, HV, K]beta: [B, S, HV]

use_qk_l2norm_in_kernel=truedtype=bf16

• kda-tp1-4096q, k, v, g: [1, 4096, 32, 128]beta: [1, 4096, 32]
• kda-tp1-16384q, k, v, g: [1, 16384, 32, 128]beta: [1, 16384, 32]
• kda-tp1-65536q, k, v, g: [1, 65536, 32, 128]beta: [1, 65536, 32]
• kda-tp2-4096q, k, v, g: [1, 4096, 16, 128]beta: [1, 4096, 16]
• kda-tp2-16384q, k, v, g: [1, 16384, 16, 128]beta: [1, 16384, 16]
• kda-tp2-65536q, k, v, g: [1, 65536, 16, 128]beta: [1, 65536, 16]
• kda-tp4-4096q, k, v, g: [1, 4096, 8, 128]beta: [1, 4096, 8]
• kda-tp4-16384q, k, v, g: [1, 16384, 8, 128]beta: [1, 16384, 8]
• kda-tp4-65536q, k, v, g: [1, 65536, 8, 128]beta: [1, 65536, 8]
• kda-tp8-4096q, k, v, g: [1, 4096, 4, 128]beta: [1, 4096, 4]
• kda-tp8-16384q, k, v, g: [1, 16384, 4, 128]beta: [1, 16384, 4]
• kda-tp8-65536q, k, v, g: [1, 65536, 4, 128]beta: [1, 65536, 4]

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
kda-tp1-4x1024	bf16	1.32×	0.4467	fla	0.5894	34	9%	mem
	f16	1.34×	0.4380	fla	0.5855	34.7	9%	mem
kda-tp1-1024-7168	bf16	0.90×	1.3473	fla	1.2096	22.6	6%	mem
kda-tp1-16x8192	bf16	1.27×	13.1874	fla	16.8034	36.9	9%	mem
kda-tp8-1024-7168	bf16	0.51×	0.6895	fla	0.3544	5.51	1%	mem
kda-tp8-16x8192	bf16	1.11×	2.1177	fla	2.3439	28.7	7%	mem
kda-tp1-4x8192	bf16	1.27×	3.3764	fla	4.2837	36	9%	mem
kda-tp1-decode-b1	bf16	1.43×	0.00429	fla	0.00611	0.865	22%	lat
kda-tp1-decode-b64	bf16	1.11×	0.0777	fla	0.0861	3.06	77%	mem
kda-tp1-decode-b8	bf16	1.34×	0.0122	fla	0.0163	2.44	62%	mem
kda-tp1-decode-b32	bf16	1.11×	0.0423	fla	0.0467	2.81	71%	mem
kda-tp1-decode-b128	bf16	1.13×	0.1464	fla	0.1658	3.24	82%	mem
kda-tp1-decode-b256	bf16	1.15×	0.2827	fla	0.3240	3.36	85%	mem
kda-tp8-decode-b32	bf16	1.20×	0.00803	fla	0.0096	1.85	47%	mem
kda-tp8-decode-b256	bf16	1.11×	0.0423	fla	0.0468	2.81	71%	mem
kda-tp1-4096	bf16	0.83×	0.7319	fla	0.6094	20.8	5%	mem
kda-tp1-16384	bf16	0.82×	2.7977	fla	2.3022	21.7	5%	mem
kda-tp1-65536	bf16	0.83×	10.9486	fla	9.0410	22.2	5%	mem
kda-tp2-4096	bf16	0.65×	0.5696	fla	0.3691	13.3	3%	mem
kda-tp2-16384	bf16	0.64×	2.1537	fla	1.3792	14.1	3%	mem
kda-tp2-65536	bf16	0.64×	8.4025	fla	5.3881	14.5	4%	mem
kda-tp4-4096	bf16	0.59×	0.4445	fla	0.2638	8.55	2%	mem
kda-tp4-16384	bf16	0.55×	1.6948	fla	0.9313	8.97	2%	mem
kda-tp4-65536	bf16	0.55×	6.6078	fla	3.6132	9.2	2%	mem
kda-tp8-4096	bf16	0.52×	0.3910	fla	0.2020	4.86	1%	mem
kda-tp8-16384	bf16	0.45×	1.5401	fla	0.6985	4.93	1%	mem
kda-tp8-65536	bf16	0.45×	6.0266	fla	2.6897	5.04	1%	mem

GLAChunkFwd

q, k, g: [B, S, H, DK]v: [B, S, H, DV]

scale=0.125dtype=f16, bf16

• train-2kq, k, v, g: [2, 2048, 4, 64]
• train-8kq, k, v, g: [2, 8192, 4, 64]

q, k, g: [B, S, H, DK]v: [B, S, H, DV]initial_state: [B, H, DK, DV], f32

scale=0.125dtype=f16, bf16

• train-4kq, k, v, g: [2, 4096, 4, 64]initial_state: [2, 4, 64, 64], f32
• train-16kq, k, v, g: [2, 16384, 4, 64]initial_state: [2, 4, 64, 64], f32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
train-2k	f16	0.79×	0.0946	fla	0.0751	4.4	2%	mem
	bf16	0.81×	0.0938	fla	0.0760	4.44	3%	mem
train-8k	f16	0.89×	0.3061	fla	0.2721	5.44	3%	mem
	bf16	0.90×	0.3059	fla	0.2746	5.44	3%	mem
train-4k	f16	0.81×	0.1567	fla	0.1265	5.31	3%	mem
	bf16	0.84×	0.1558	fla	0.1312	5.34	3%	mem
train-16k	f16	0.79×	0.6176	fla	0.4854	5.39	3%	mem
	bf16	0.82×	0.6155	fla	0.5039	5.41	3%	mem

GLAChunkBwd

q, k, g: [B, S, H, DK]v, do: [B, S, H, DV]h: [B, NC + 1, H, DK, DV], f32dht: [B, H, DK, DV], f32

scale=0.125dtype=f16, bf16

• train-2kq, k, v, g, do: [2, 2048, 4, 64]h: [2, 33, 4, 64, 64], f32dht: [2, 4, 64, 64], f32
• train-4kq, k, v, g, do: [2, 4096, 4, 64]h: [2, 65, 4, 64, 64], f32dht: [2, 4, 64, 64], f32
• train-8kq, k, v, g, do: [2, 8192, 4, 64]h: [2, 129, 4, 64, 64], f32dht: [2, 4, 64, 64], f32
• train-16kq, k, v, g, do: [2, 16384, 4, 64]h: [2, 257, 4, 64, 64], f32dht: [2, 4, 64, 64], f32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
train-2k	f16	1.04×	0.1726	fla	0.1789	5.24	4%	mem
	bf16	1.03×	0.1734	fla	0.1785	5.22	4%	mem
train-4k	f16	0.98×	0.3540	fla	0.3474	5.11	4%	mem
	bf16	1.01×	0.3455	fla	0.3477	5.24	4%	mem
train-8k	f16	0.96×	0.7065	fla	0.6763	5.12	4%	mem
	bf16	0.99×	0.6808	fla	0.6774	5.32	4%	mem
train-16k	f16	0.92×	1.4448	fla	1.3256	5.01	4%	mem
	bf16	0.96×	1.3758	fla	1.3243	5.26	4%	mem

GLARecurrentFwd

q, k, gk: [B, H, DK]v: [B, H, DV]state: [B, H, DK, DV]

scale=0.0883883dtype=bf16

• serving-b1-h8q, k, v, gk: [1, 8, 128]state: [1, 8, 128, 128]
• serving-b1-h16q, k, v, gk: [1, 16, 128]state: [1, 16, 128, 128]
• serving-b1-h32q, k, v, gk: [1, 32, 128]state: [1, 32, 128, 128]
• serving-b1-h48q, k, v, gk: [1, 48, 128]state: [1, 48, 128, 128]
• serving-b1-h64q, k, v, gk: [1, 64, 128]state: [1, 64, 128, 128]
• serving-b8-h32q, k, v, gk: [8, 32, 128]state: [8, 32, 128, 128]
• serving-b8-h48q, k, v, gk: [8, 48, 128]state: [8, 48, 128, 128]
• serving-b8-h64q, k, v, gk: [8, 64, 128]state: [8, 64, 128, 128]

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
serving-b1-h8	bf16	0.86×	0.00742	torch-compile fla torch	0.0064 0.00816 0.0307	0.0886	2%	lat
serving-b1-h16	bf16	0.89×	0.00768	torch-compile fla torch	0.00681 0.00848 0.0321	0.171	3%	lat
serving-b1-h32	bf16	1.06×	0.00787	torch-compile fla torch	0.00838 0.00912 0.0363	0.334	6%	lat
serving-b1-h48	bf16	1.29×	0.00809	torch-compile fla torch	0.0104 0.0112 0.0407	0.487	9%	lat
serving-b1-h64	bf16	1.07×	0.00829	torch-compile fla torch	0.00886 0.0111 0.0423	0.635	11%	lat
serving-b8-h32	bf16	1.33×	0.0162	torch-compile fla torch	0.0214 0.0225 0.0906	1.3	24%	mem
serving-b8-h48	bf16	1.05×	0.0235	torch-compile fla torch	0.0247 0.0296 0.1219	1.34	24%	mem
serving-b8-h64	bf16	1.05×	0.0309	torch-compile fla torch	0.0325 0.0382 0.1596	1.36	25%	mem

GLAFwd

q, k, g: [B, S, H, K]v: [B, S, H, V]

• prefill-2kq, k, v, g: [2, 2048, 4, 64]dtype=bf16
• prefill-8kq, k, v, g: [2, 8192, 4, 64]dtype=bf16
• prefill-3000q, k, v, g: [2, 3000, 4, 64]scale=0.125dtype=bf16
• decode-first-tokenq, k, v, g: [3, 1, 24, 128]dtype=f16

q, k, g: [B, S, H, K]v: [B, S, H, V]initial_state: [B, H, K, V], f32

dtype=bf16

• prefill-4kq, k, v, g: [2, 4096, 4, 64]initial_state: [2, 4, 64, 64], f32
• prefill-16kq, k, v, g: [2, 16384, 4, 64]initial_state: [2, 4, 64, 64], f32
• decode-b1-d64q, k, v, g: [1, 1, 32, 64]initial_state: [1, 32, 64, 64], f32
• decode-b8-d128q, k, v, g: [8, 1, 32, 128]initial_state: [8, 32, 128, 128], f32
• gla-short-continuationq, k, v, g: [1, 257, 16, 128]initial_state: [1, 16, 128, 128], f32
• gla-continue-b4q, k, v, g: [4, 257, 16, 128]initial_state: [4, 16, 128, 128], f32

q, k, g: [B, S, H, K]v: [B, S, H, V]cu_seqlens, cu_seqlens_cpu: [N + 1], i64

• varlen-4x1kq, k, v, g: [1, 4096, 4, 64]cu_seqlens, cu_seqlens_cpu: [5], i64seq_lens=repeat(1024, 4)dtype=bf16

q, k, g: [B, S, H, K]v: [B, S, H, V]initial_state: [N, H, K, V], f32cu_seqlens: [N + 1], i64

• varlen-raggedq, k, v, g: [1, 4096, 16, 128]initial_state: [4, 16, 128, 128], f32cu_seqlens: [5], i64seq_lens=[1,63,1000,3032]dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
prefill-2k	bf16	1.28×	0.0596	fla	0.0761	5.67	4%	mem
prefill-8k	bf16	1.41×	0.1957	fla	0.2750	6.9	5%	mem
prefill-3000	bf16	2.01×	0.0543	fla	0.1090	9.11	6%	mem
decode-first-token	f16	3.23×	0.00272	fla	0.0088	2.18	39%	lat
prefill-4k	bf16	1.30×	0.1008	fla	0.1306	6.7	5%	mem
prefill-16k	bf16	1.93×	0.2617	fla	0.5046	10.3	7%	mem
decode-b1-d64	bf16	3.68×	0.00218	fla	0.008	0.303	11%	lat
decode-b8-d128	bf16	1.55×	0.0120	fla	0.0186	1.75	63%	mem
gla-short-continuation	bf16	1.23×	0.0352	fla	0.0432	9.61	5%	mem
gla-continue-b4	bf16	1.22×	0.0870	fla	0.1058	15.5	8%	mem
varlen-4x1k	bf16	2.08×	0.0421	fla	0.0877	8.03	6%	mem
varlen-ragged	f16	1.45×	0.2495	fla	0.3614	21.6	8%	mem