GEMM

本页暂无中文版。以下为英文原文。

6 ops, 108 workloads.

One table per op, one row per workload. Ratio is the fastest other implementation's device time divided by ours, so green is faster than it, plain is level with it, red is slower. Times are in ms. How these numbers are taken.

GemmFwd

a: [M, K]b: [K, N]

• square-1ka, b: [1024, 1024]trans_b=falsedtype=f16, bf16

a: [M, K]b: [N, K]

• ds-v3-decode-qkv-aa: [128, 7168]b: [2112, 7168]dtype=bf16
• ds-v3-decode-downa: [128, 2048]b: [7168, 2048]dtype=bf16
• ds-v3-prefill-qkv-aa: [4096, 7168]b: [2112, 7168]dtype=bf16
• ds-v3-prefill-downa: [4096, 2048]b: [7168, 2048]dtype=bf16
• ds-v3-prefill-mlp-upa, b: [4096, 7168]dtype=f16, bf16
• ds-v3-prefill-o-proja: [4096, 16384]b: [7168, 16384]dtype=bf16
• ds-v3-prefill-q-ba: [4096, 1536]b: [24576, 1536]dtype=bf16
• ds-v3-m16-mlp-upa: [16, 7168]b: [4096, 7168]dtype=bf16
• ds-v3-m32-mlp-upa: [32, 7168]b: [4096, 7168]dtype=bf16
• ds-v3-m64-downa: [64, 2048]b: [7168, 2048]dtype=bf16
• ds-v3-m96-qkv-aa: [96, 7168]b: [2112, 7168]dtype=bf16
• ds-v3-m1-qkv-aa: [1, 7168]b: [2112, 7168]dtype=bf16
• ds-v3-m2-qkv-aa: [2, 7168]b: [2112, 7168]dtype=bf16
• qwen3-8b-gate-up-decodea: [1, 4096]b: [24576, 4096]dtype=bf16
• qwen3-8b-gate-up-b32a: [32, 4096]b: [24576, 4096]dtype=bf16
• qwen3-8b-down-prefilla: [128, 12288]b: [4096, 12288]dtype=bf16
• vit-l16-in21k-heada: [256, 1024]b: [21843, 1024]dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
square-1k	f16	1.08×	0.00669	cublaslt-best torch-cublas quack flaggems	0.00723 0.00723 0.0104 0.0115	322	44%	lat
	bf16	1.08×	0.00662	cublaslt-best torch-cublas quack flaggems	0.00717 0.00718 0.0104 0.0115	325	43%	lat
ds-v3-decode-qkv-a	bf16	1.12×	0.0156	torch-cublas cublaslt-best deepgemm quack	0.0174 0.0174 0.0214 0.0431	249	46%	mem
ds-v3-decode-down	bf16	0.99×	0.0134	cublaslt-best torch-cublas deepgemm quack	0.0133 0.0134 0.0138 0.0172	281	52%	mem
ds-v3-prefill-qkv-a	bf16	0.97×	0.1831	torch-cublas cublaslt-best deepgemm quack	0.1767 0.1768 0.1796 0.1934	678	89%	comp
ds-v3-prefill-down	bf16	0.98×	0.1735	cublaslt-best deepgemm torch-cublas quack	0.1696 0.1784 0.1798 0.1827	694	91%	comp
ds-v3-prefill-mlp-up	f16	0.99×	0.3378	cublaslt-best torch-cublas quack	0.3337 0.3342 0.3384	712	98%	comp
	bf16	0.99×	0.3330	cublaslt-best deepgemm torch-cublas quack	0.3287 0.3298 0.3313 0.3338	723	95%	comp
ds-v3-prefill-o-proj	bf16	0.99×	1.2701	deepgemm torch-cublas quack cublaslt-best	1.2540 1.2540 1.2569 1.2690	758	100%	comp
ds-v3-prefill-q-b	bf16	0.98×	0.4122	cublaslt-best deepgemm quack torch-cublas	0.4058 0.4398 0.4458 0.4492	751	99%	comp
ds-v3-m16-mlp-up	bf16	1.15×	0.0208	cublaslt-best torch-cublas deepgemm quack	0.0239 0.0243 0.0344 0.0434	46.6	63%	mem
ds-v3-m32-mlp-up	bf16	1.15×	0.0210	torch-cublas cublaslt-best deepgemm quack	0.0242 0.0242 0.0304 0.0435	90.8	63%	mem
ds-v3-m64-down	bf16	1.08×	0.0120	cublaslt-best torch-cublas deepgemm quack	0.0129 0.0134 0.0135 0.0169	158	57%	mem
ds-v3-m96-qkv-a	bf16	1.08×	0.0156	cublaslt-best torch-cublas deepgemm quack	0.0169 0.0171 0.0219 0.0429	187	46%	mem
ds-v3-m1-qkv-a	bf16	1.26×	0.0118	cublaslt-best torch-cublas deepgemm quack	0.0148 0.0158 0.0317 0.0427	3.86	57%	mem
ds-v3-m2-qkv-a	bf16	1.09×	0.0137	cublaslt-best torch-cublas deepgemm quack	0.0149 0.0158 0.0312 0.0427	5.53	49%	mem
qwen3-8b-gate-up-decode	bf16	1.05×	0.0541	cublaslt-best torch-cublas deepgemm quack	0.0567 0.0568 0.0585 0.0722	5.58	83%	mem
qwen3-8b-gate-up-b32	bf16	1.01×	0.0565	cublaslt-best torch-cublas deepgemm quack	0.0571 0.0571 0.0586 0.0638	116	80%	mem
qwen3-8b-down-prefill	bf16	1.04×	0.0358	cublaslt-best torch-cublas deepgemm quack	0.0373 0.0392 0.0402 0.0718	362	65%	mem
vit-l16-in21k-head	bf16	8.00×	0.0268	cublaslt-best torch-cublas	0.2147 0.2226	428	56%	comp

GemmFP8Fwd

a: [M, K]b: [N, K]scale_a: [*SA], f32scale_b: [*SB], f32

• ds-v3-decode-qkv-aa: [128, 7168]b: [2112, 7168]scale_a, scale_b: [1, 1], f32dtype=fp8e4m3/bf16, fp8e4m3/f16
• ds-v3-decode-downa: [128, 2048]b: [7168, 2048]scale_a, scale_b: [1, 1], f32dtype=fp8e4m3/bf16
• ds-v3-prefill-qkv-aa: [4096, 7168]b: [2112, 7168]scale_a, scale_b: [1, 1], f32dtype=fp8e4m3/bf16
• ds-v3-prefill-downa: [4096, 2048]b: [7168, 2048]scale_a, scale_b: [1, 1], f32dtype=fp8e4m3/bf16
• ds-v3-decode-qkv-a-blka: [128, 7168]b: [2112, 7168]scale_a: [128, 56], f32scale_b: [2112, 56], f32dtype=fp8e4m3/bf16, fp8e4m3/f16
• ds-v3-decode-down-blka: [128, 2048]b: [7168, 2048]scale_a: [128, 16], f32scale_b: [7168, 16], f32dtype=fp8e4m3/bf16
• ds-v3-prefill-qkv-a-blka: [4096, 7168]b: [2112, 7168]scale_a: [4096, 56], f32scale_b: [2112, 56], f32dtype=fp8e4m3/bf16
• ds-v3-prefill-down-blka: [4096, 2048]b: [7168, 2048]scale_a: [4096, 16], f32scale_b: [7168, 16], f32dtype=fp8e4m3/bf16
• ds-v3-prefill-mlp-upa, b: [4096, 7168]scale_a, scale_b: [4096, 56], f32dtype=fp8e4m3/bf16
• ds-v3-prefill-o-proja: [4096, 16384]b: [7168, 16384]scale_a: [4096, 128], f32scale_b: [7168, 128], f32dtype=fp8e4m3/bf16
• ds-v3-prefill-q-ba: [4096, 1536]b: [24576, 1536]scale_a: [4096, 12], f32scale_b: [24576, 12], f32dtype=fp8e4m3/bf16
• ds-v3-m8-downa: [8, 2048]b: [7168, 2048]scale_a, scale_b: [1, 1], f32dtype=fp8e4m3/bf16
• ds-v3-m1-downa: [1, 2048]b: [7168, 2048]scale_a, scale_b: [1, 1], f32dtype=fp8e4m3/bf16
• ds-v3-m1-down-blka: [1, 2048]b: [7168, 2048]scale_a: [1, 16], f32scale_b: [7168, 16], f32dtype=fp8e4m3/bf16
• ds-v3-decode-qkv-a-2da: [128, 7168]b: [2112, 7168]scale_a: [128, 56], f32scale_b: [17, 56], f32dtype=fp8e4m3/bf16
• ds-v3-decode-down-2da: [128, 2048]b: [7168, 2048]scale_a: [128, 16], f32scale_b: [56, 16], f32dtype=fp8e4m3/bf16
• ds-v3-prefill-qkv-a-2da: [4096, 7168]b: [2112, 7168]scale_a: [4096, 56], f32scale_b: [17, 56], f32dtype=fp8e4m3/bf16
• ds-v3-prefill-down-2da: [4096, 2048]b: [7168, 2048]scale_a: [4096, 16], f32scale_b: [56, 16], f32dtype=fp8e4m3/bf16
• ds-v3-prefill-mlp-up-2da, b: [4096, 7168]scale_a: [4096, 56], f32scale_b: [32, 56], f32dtype=fp8e4m3/bf16
• ds-v3-prefill-o-proj-2da: [4096, 16384]b: [7168, 16384]scale_a: [4096, 128], f32scale_b: [56, 128], f32dtype=fp8e4m3/bf16
• ds-v3-prefill-q-b-2da: [4096, 1536]b: [24576, 1536]scale_a: [4096, 12], f32scale_b: [192, 12], f32dtype=fp8e4m3/bf16
• ds-v3-m1-down-2da: [1, 2048]b: [7168, 2048]scale_a: [1, 16], f32scale_b: [56, 16], f32dtype=fp8e4m3/bf16
• ds-v3-kv-a-m128-s1a: [128, 7168]b: [576, 7168]scale_a: [128, 56], f32scale_b: [576, 56], f32dtype=fp8e4m3/bf16
• ds-v3-kv-a-m128-s128a: [128, 7168]b: [576, 7168]scale_a: [128, 56], f32scale_b: [5, 56], f32dtype=fp8e4m3/bf16
• ds-v3-q-b-m128-s1a: [128, 1536]b: [24576, 1536]scale_a: [128, 12], f32scale_b: [24576, 12], f32dtype=fp8e4m3/bf16
• ds-v3-q-b-m128-s128a: [128, 1536]b: [24576, 1536]scale_a: [128, 12], f32scale_b: [192, 12], f32dtype=fp8e4m3/bf16
• ds-v3-kv-b-m128-s1a: [128, 512]b: [32768, 512]scale_a: [128, 4], f32scale_b: [32768, 4], f32dtype=fp8e4m3/bf16
• ds-v3-kv-b-m128-s128a: [128, 512]b: [32768, 512]scale_a: [128, 4], f32scale_b: [256, 4], f32dtype=fp8e4m3/bf16
• ds-v3-o-proj-m128-s1a: [128, 16384]b: [7168, 16384]scale_a: [128, 128], f32scale_b: [7168, 128], f32dtype=fp8e4m3/bf16
• ds-v3-o-proj-m128-s128a: [128, 16384]b: [7168, 16384]scale_a: [128, 128], f32scale_b: [56, 128], f32dtype=fp8e4m3/bf16
• ds-v3-mlp-up-m128-s1a: [128, 7168]b: [4096, 7168]scale_a: [128, 56], f32scale_b: [4096, 56], f32dtype=fp8e4m3/bf16
• ds-v3-mlp-up-m128-s128a: [128, 7168]b: [4096, 7168]scale_a: [128, 56], f32scale_b: [32, 56], f32dtype=fp8e4m3/bf16
• ds-v3-kv-a-m4096-s1a: [4096, 7168]b: [576, 7168]scale_a: [4096, 56], f32scale_b: [576, 56], f32dtype=fp8e4m3/bf16
• ds-v3-kv-a-m4096-s128a: [4096, 7168]b: [576, 7168]scale_a: [4096, 56], f32scale_b: [5, 56], f32dtype=fp8e4m3/bf16
• ds-v3-kv-b-m4096-s1a: [4096, 512]b: [32768, 512]scale_a: [4096, 4], f32scale_b: [32768, 4], f32dtype=fp8e4m3/bf16
• ds-v3-kv-b-m4096-s128a: [4096, 512]b: [32768, 512]scale_a: [4096, 4], f32scale_b: [256, 4], f32dtype=fp8e4m3/bf16

a: [M, K]b: [N, K]scale_a: [*SA], f32scale_b: [*SB], f32bias: [N], bf16

• ds-v3-decode-qkv-a-biasa: [128, 7168]b: [2112, 7168]scale_a, scale_b: [1, 1], f32bias: [2112], bf16dtype=fp8e4m3/bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
ds-v3-decode-qkv-a	fp8e4m3/bf16	1.02×	0.0116	torch-scaled-mm deepgemm torch-fp32-ref	0.0118 0.0177 0.2454	337	32%	mem
	fp8e4m3/f16	1.01×	0.0115	torch-scaled-mm deepgemm torch-fp32-ref	0.0116 0.0268 0.2453	338	32%	mem
ds-v3-decode-down	fp8e4m3/bf16	1.10×	0.00819	torch-scaled-mm deepgemm torch-fp32-ref	0.00902 0.0145 0.2471	461	45%	mem
ds-v3-prefill-qkv-a	fp8e4m3/bf16	1.00×	0.1083	torch-scaled-mm deepgemm torch-fp32-ref	0.1084 0.1142 3.4374	1,145	83%	comp
ds-v3-prefill-down	fp8e4m3/bf16	1.04×	0.1056	torch-scaled-mm deepgemm torch-fp32-ref	0.1094 0.1110 3.3412	1,140	83%	comp
ds-v3-decode-qkv-a-blk	fp8e4m3/bf16	1.40×	0.0173	deepgemm torch-fp32-ref	0.0242 0.2818	225	22%	mem
	fp8e4m3/f16	1.39×	0.0173	deepgemm torch-fp32-ref	0.0241 0.2815	225	22%	mem
ds-v3-decode-down-blk	fp8e4m3/bf16	1.62×	0.0118	deepgemm torch-fp32-ref	0.0192 0.2811	319	32%	mem
ds-v3-prefill-qkv-a-blk	fp8e4m3/bf16	1.12×	0.1663	deepgemm torch-fp32-ref	0.1858 3.5357	746	54%	comp
ds-v3-prefill-down-blk	fp8e4m3/bf16	1.45×	0.1760	deepgemm torch-fp32-ref	0.2547 3.3930	684	50%	comp
ds-v3-prefill-mlp-up	fp8e4m3/bf16	1.03×	0.3205	deepgemm torch-fp32-ref	0.3308 6.6838	751	54%	comp
ds-v3-prefill-o-proj	fp8e4m3/bf16	1.01×	1.0794	deepgemm torch-fp32-ref	1.0877 26.3639	891	65%	comp
ds-v3-prefill-q-b	fp8e4m3/bf16	1.72×	0.4245	deepgemm torch-fp32-ref	0.7312 8.5299	729	53%	comp
ds-v3-m8-down	fp8e4m3/bf16	0.94×	0.00803	torch-scaled-mm deepgemm torch-fp32-ref	0.00758 0.0118 0.1691	31.1	41%	mem
ds-v3-m1-down	fp8e4m3/bf16	0.95×	0.00806	torch-scaled-mm deepgemm torch-fp32-ref	0.00768 0.0121 0.1318	5.46	40%	mem
ds-v3-m1-down-blk	fp8e4m3/bf16	1.80×	0.00995	deepgemm torch-fp32-ref	0.0180 0.1658	4.43	34%	mem
ds-v3-decode-qkv-a-2d	fp8e4m3/bf16	1.10×	0.0138	deepgemm flashinfer-fp8-blockscale-sm90 torch-fp32-ref	0.0152 0.0155 0.2843	282	27%	mem
ds-v3-decode-down-2d	fp8e4m3/bf16	1.13×	0.0102	flashinfer-fp8-blockscale-sm90 deepgemm torch-fp32-ref	0.0116 0.0122 0.2823	369	36%	mem
ds-v3-prefill-qkv-a-2d	fp8e4m3/bf16	0.97×	0.1145	deepgemm flashinfer-fp8-blockscale-sm90 torch-fp32-ref	0.1114 0.1448 3.5377	1,083	78%	comp
ds-v3-prefill-down-2d	fp8e4m3/bf16	1.00×	0.1083	deepgemm flashinfer-fp8-blockscale-sm90 torch-fp32-ref	0.1087 0.1469 3.3944	1,111	80%	comp
ds-v3-prefill-mlp-up-2d	fp8e4m3/bf16	0.99×	0.1948	deepgemm flashinfer-fp8-blockscale-sm90 torch-fp32-ref	0.1927 0.2204 6.6768	1,235	89%	comp
ds-v3-prefill-o-proj-2d	fp8e4m3/bf16	0.98×	0.7046	deepgemm flashinfer-fp8-blockscale-sm90 torch-fp32-ref	0.6895 0.7676 26.3643	1,366	99%	comp
ds-v3-prefill-q-b-2d	fp8e4m3/bf16	1.00×	0.2650	deepgemm flashinfer-fp8-blockscale-sm90 torch-fp32-ref	0.2658 0.3850 8.5248	1,168	85%	comp
ds-v3-m1-down-2d	fp8e4m3/bf16	1.02×	0.00922	flashinfer-fp8-blockscale-sm90 deepgemm torch-fp32-ref	0.00944 0.0119 0.1665	4.78	35%	mem
ds-v3-kv-a-m128-s1	fp8e4m3/bf16	1.51×	0.0151	deepgemm torch-fp32-ref	0.0227 0.1015	70.4	8%	lat
ds-v3-kv-a-m128-s128	fp8e4m3/bf16	1.13×	0.0117	deepgemm flashinfer-fp8-blockscale-sm90 torch-fp32-ref	0.0132 0.0142 0.1043	90.6	10%	lat
ds-v3-q-b-m128-s1	fp8e4m3/bf16	1.52×	0.0236	deepgemm torch-fp32-ref	0.0359 0.6857	411	43%	mem
ds-v3-q-b-m128-s128	fp8e4m3/bf16	1.06×	0.0182	deepgemm flashinfer-fp8-blockscale-sm90 torch-fp32-ref	0.0193 0.0202 0.6875	534	54%	mem
ds-v3-kv-b-m128-s1	fp8e4m3/bf16	2.33×	0.0124	deepgemm torch-fp32-ref	0.0288 0.2955	349	46%	mem
ds-v3-kv-b-m128-s128	fp8e4m3/bf16	0.99×	0.0121	deepgemm flashinfer-fp8-blockscale-sm90 torch-fp32-ref	0.0121 0.0149 0.2960	356	46%	mem
ds-v3-o-proj-m128-s1	fp8e4m3/bf16	1.06×	0.0581	deepgemm torch-fp32-ref	0.0616 1.8502	519	48%	mem
ds-v3-o-proj-m128-s128	fp8e4m3/bf16	0.96×	0.0494	flashinfer-fp8-blockscale-sm90 deepgemm torch-fp32-ref	0.0472 0.0549 1.8540	611	55%	mem
ds-v3-mlp-up-m128-s1	fp8e4m3/bf16	1.16×	0.0259	deepgemm torch-fp32-ref	0.0301 0.4906	292	28%	mem
ds-v3-mlp-up-m128-s128	fp8e4m3/bf16	1.06×	0.0164	deepgemm flashinfer-fp8-blockscale-sm90 torch-fp32-ref	0.0175 0.0190 0.4923	460	42%	mem
ds-v3-kv-a-m4096-s1	fp8e4m3/bf16	1.08×	0.0543	deepgemm torch-fp32-ref	0.0584 1.2337	623	45%	comp
ds-v3-kv-a-m4096-s128	fp8e4m3/bf16	1.18×	0.0369	deepgemm flashinfer-fp8-blockscale-sm90 torch-fp32-ref	0.0435 0.0472 1.2356	918	66%	comp
ds-v3-kv-b-m4096-s1	fp8e4m3/bf16	3.19×	0.2453	deepgemm torch-fp32-ref	0.7832 4.0980	561	41%	comp
ds-v3-kv-b-m4096-s128	fp8e4m3/bf16	0.96×	0.1583	deepgemm flashinfer-fp8-blockscale-sm90 torch-fp32-ref	0.1517 0.2879 4.0996	870	63%	comp
ds-v3-decode-qkv-a-bias	fp8e4m3/bf16	1.02×	0.0119	torch-scaled-mm deepgemm torch-fp32-ref	0.0122 0.0331 0.2515	326	31%	mem

GemmW4A16Fwd

activation: [M, K]packed_weight: [N, K // 2], u8weight_scale: [N, K // group_size]weight_zero: [N, K // group_size], u8

dtype=f16

• single-tileactivation: [64, 128]packed_weight: [64, 64], u8weight_scale: [64, 1]weight_zero: [64, 1], u8
• multi-tileactivation: [128, 256]packed_weight: [256, 128], u8weight_scale: [256, 2]weight_zero: [256, 2], u8
• decode-l2-residentactivation: [1, 8192]packed_weight: [8192, 4096], u8weight_scale: [8192, 64]weight_zero: [8192, 64], u8
• decode-hbm-streamingactivation: [1, 16384]packed_weight: [8192, 8192], u8weight_scale: [8192, 128]weight_zero: [8192, 128], u8
• decode-non-pow2activation: [1, 20480]packed_weight: [7168, 10240], u8weight_scale: [7168, 160]weight_zero: [7168, 160], u8
• decode-long-kactivation: [1, 81920]packed_weight: [8192, 40960], u8weight_scale: [8192, 640]weight_zero: [8192, 640], u8
• decode-b2activation: [2, 8192]packed_weight: [8192, 4096], u8weight_scale: [8192, 64]weight_zero: [8192, 64], u8
• decode-b8activation: [8, 8192]packed_weight: [8192, 4096], u8weight_scale: [8192, 64]weight_zero: [8192, 64], u8
• decode-b32activation: [32, 8192]packed_weight: [8192, 4096], u8weight_scale: [8192, 64]weight_zero: [8192, 64], u8
• chunked-prefill-tileactivation: [128, 8192]packed_weight: [8192, 4096], u8weight_scale: [8192, 64]weight_zero: [8192, 64], u8
• prefill-mlp-upactivation: [512, 8192]packed_weight: [8192, 4096], u8weight_scale: [8192, 64]weight_zero: [8192, 64], u8
• llama-8b-decode-downactivation: [1, 14336]packed_weight: [4096, 7168], u8weight_scale: [4096, 112]weight_zero: [4096, 112], u8
• llama-8b-prefill-downactivation: [128, 14336]packed_weight: [4096, 7168], u8weight_scale: [4096, 112]weight_zero: [4096, 112], u8
• qwen3-32b-prefill-downactivation: [128, 25600]packed_weight: [5120, 12800], u8weight_scale: [5120, 200]weight_zero: [5120, 200], u8
• decode-tp8-attn-outactivation: [1, 8192]packed_weight: [1024, 4096], u8weight_scale: [1024, 64]weight_zero: [1024, 64], u8

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
single-tile	f16	1.06×	0.0025	torch-dequantized-matmul marlin-fp32 marlin-fp16	0.00266 0.00662 0.00662	0.427	0%	lat
multi-tile	f16	1.07×	0.00288	torch-dequantized-matmul marlin-fp32 marlin-fp16	0.00307 0.0095 0.00954	5.87	1%	lat
decode-l2-resident	f16	1.16×	0.0186	marlin-fp32 marlin-fp16 torch-dequantized-matmul	0.0215 0.0218 0.0461	14.4	42%	mem
decode-hbm-streaming	f16	1.06×	0.0358	marlin-fp32 marlin-fp16 torch-dequantized-matmul	0.0379 0.0382 0.0734	15	44%	mem
decode-non-pow2	f16	1.09×	0.0373	marlin-fp16 marlin-fp32 torch-dequantized-matmul	0.0408 0.0408 0.0848	15.8	46%	mem
decode-long-k	f16	1.02×	0.1383	marlin-fp16 marlin-fp32 torch-dequantized-matmul	0.1414 0.1415 0.3122	19.4	56%	mem
decode-b2	f16	1.15×	0.0186	marlin-fp32 marlin-fp16 torch-dequantized-matmul	0.0215 0.0219 0.0461	21.7	42%	mem
decode-b8	f16	1.16×	0.0187	marlin-fp32 marlin-fp16 torch-dequantized-matmul	0.0216 0.0220 0.0457	64.5	42%	mem
decode-b32	f16	1.49×	0.0224	marlin-fp32 marlin-fp16 torch-dequantized-matmul	0.0333 0.0336 0.0412	198	36%	mem
chunked-prefill-tile	f16	1.01×	0.0422	torch-dequantized-matmul marlin-fp16 marlin-fp32	0.0428 0.0649 0.0654	410	56%	comp
prefill-mlp-up	f16	1.00×	0.1043	torch-dequantized-matmul marlin-fp16 marlin-fp32	0.1041 0.2306 0.2312	660	90%	comp
llama-8b-decode-down	f16	1.51×	0.0169	marlin-fp32 marlin-fp16 torch-dequantized-matmul	0.0255 0.0256 0.0389	13.9	40%	mem
llama-8b-prefill-down	f16	1.09×	0.0396	torch-dequantized-matmul marlin-fp16 marlin-fp32	0.0434 0.0790 0.0798	382	52%	comp
qwen3-32b-prefill-down	f16	0.93×	0.0887	torch-dequantized-matmul marlin-fp16 marlin-fp32	0.0822 0.1247 0.1271	381	52%	comp
decode-tp8-attn-out	f16	1.45×	0.00726	torch-dequantized-matmul marlin-fp32 marlin-fp16	0.0105 0.0157 0.0161	4.62	13%	lat

BmmFwd

a: [B, M, K]b: [B, K, N]

• square-128a, b: [8, 128, 128]dtype=f16, bf16
• square-1ka, b: [8, 1024, 1024]dtype=f16, bf16
• square-512a, b: [16, 512, 512]dtype=f16, bf16
• square-256a, b: [32, 256, 256]dtype=f16, bf16
• square-4ka, b: [4, 4096, 4096]dtype=bf16
• square-2ka, b: [8, 2048, 2048]dtype=f16, bf16
• mha-decode-qka: [64, 128, 128]b: [64, 128, 2048]dtype=f16, bf16
• mha-decode-pva: [64, 128, 2048]b: [64, 2048, 128]dtype=f16, bf16
• moe-prefilla: [128, 512, 2048]b: [128, 2048, 512]dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
square-128	f16	1.18×	0.00269	flaggems torch-cublas quack	0.00317 0.00326 0.00454	12.5	6%	lat
	bf16	1.17×	0.00272	flaggems torch-cublas quack	0.00318 0.00323 0.00455	12.3	6%	lat
square-1k	f16	1.00×	0.0309	torch-cublas quack flaggems	0.0308 0.0314 0.0456	556	76%	comp
	bf16	0.96×	0.0308	quack torch-cublas flaggems	0.0296 0.0310 0.0453	558	73%	comp
square-512	f16	1.01×	0.0119	torch-cublas quack flaggems	0.0120 0.0129 0.0151	361	49%	comp
	bf16	1.01×	0.0118	torch-cublas quack flaggems	0.0120 0.0129 0.0151	364	48%	comp
square-256	f16	1.10×	0.0064	torch-cublas quack flaggems	0.00704 0.00739 0.0079	168	44%	lat
	bf16	1.10×	0.00637	torch-cublas quack flaggems	0.00701 0.00739 0.0079	169	44%	lat
square-4k	bf16	1.00×	0.7442	quack torch-cublas flaggems	0.7456 0.7599 0.9631	739	97%	comp
square-2k	f16	1.00×	0.2074	torch-cublas quack flaggems	0.2067 0.2122 0.2766	663	91%	comp
	bf16	1.00×	0.2046	torch-cublas quack flaggems	0.2041 0.2079 0.2718	672	88%	comp
mha-decode-qk	f16	1.07×	0.0197	torch-cublas quack flaggems	0.0212 0.0218 0.0267	218	78%	mem
	bf16	1.08×	0.0196	torch-cublas quack flaggems	0.0211 0.0219 0.0265	219	78%	mem
mha-decode-pv	f16	1.02×	0.0236	torch-cublas quack flaggems	0.0241 0.0277 0.0404	182	65%	mem
	bf16	1.02×	0.0236	torch-cublas quack flaggems	0.0241 0.0278 0.0406	182	65%	mem
moe-prefill	bf16	1.01×	0.2095	torch-cublas quack flaggems	0.2118 0.2186 0.2958	656	86%	comp

BmmFP8Fwd

a: [B, M, K]b: [B, K, N]scale_a, scale_b: [], f32

• square-1ka, b: [4, 1024, 1024]scale_a, scale_b: [], f32dtype=fp8e4m3/bf16, fp8e4m3/f16
• square-2ka, b: [8, 2048, 2048]scale_a, scale_b: [], f32dtype=fp8e4m3/bf16
• mha-decode-pva: [32, 128, 2048]b: [32, 2048, 128]scale_a, scale_b: [], f32dtype=fp8e4m3/bf16
• mha-decode-qka: [64, 128, 128]b: [64, 128, 2048]scale_a, scale_b: [], f32dtype=fp8e4m3/bf16
• moe-prefilla: [128, 512, 2048]b: [128, 2048, 512]scale_a, scale_b: [], f32dtype=fp8e4m3/bf16

a: [B, M, K]b: [B, N, K]scale_a, scale_b: [], f32

trans_b=truedtype=fp8e4m3/bf16

• square-1k-nka, b: [4, 1024, 1024]scale_a, scale_b: [], f32
• square-2k-nka, b: [8, 2048, 2048]scale_a, scale_b: [], f32
• mha-decode-pv-nka, b: [32, 128, 2048]scale_a, scale_b: [], f32
• mha-decode-qk-nka: [64, 128, 128]b: [64, 2048, 128]scale_a, scale_b: [], f32
• moe-prefill-nka, b: [128, 512, 2048]scale_a, scale_b: [], f32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
square-1k	fp8e4m3/bf16	2.22×	0.0160	flashinfer-bmm-fp8 torch-fp32-ref	0.0355 0.2939	538	39%	comp
	fp8e4m3/f16	2.22×	0.0160	flashinfer-bmm-fp8 torch-fp32-ref	0.0355 0.2937	538	39%	comp
square-2k	fp8e4m3/bf16	4.43×	0.1406	flashinfer-bmm-fp8 torch-fp32-ref	0.6234 4.0674	977	71%	comp
mha-decode-pv	fp8e4m3/bf16	1.35×	0.0185	flashinfer-bmm-fp8 torch-fp32-ref	0.0250 0.2400	116	21%	mem
mha-decode-qk	fp8e4m3/bf16	2.04×	0.0247	flashinfer-bmm-fp8 torch-fp32-ref	0.0503 0.2897	174	46%	mem
moe-prefill	fp8e4m3/bf16	3.13×	0.1994	flashinfer-bmm-fp8 torch-fp32-ref	0.6242 5.4322	689	50%	comp
square-1k-nk	fp8e4m3/bf16	1.12×	0.0118	flashinfer-bmm-fp8 torch-fp32-ref	0.0132 0.2971	727	53%	comp
square-2k-nk	fp8e4m3/bf16	1.04×	0.1200	flashinfer-bmm-fp8 torch-fp32-ref	0.1254 4.0965	1,145	83%	comp
mha-decode-pv-nk	fp8e4m3/bf16	1.07×	0.00902	flashinfer-bmm-fp8 torch-fp32-ref	0.00966 0.2359	238	44%	mem
mha-decode-qk-nk	fp8e4m3/bf16	1.34×	0.0158	flashinfer-bmm-fp8 torch-fp32-ref	0.0212 0.3175	271	72%	mem
moe-prefill-nk	fp8e4m3/bf16	1.03×	0.1314	flashinfer-bmm-fp8 torch-fp32-ref	0.1355 5.4610	1,046	76%	comp

GroupedGemmFwd

a: [M, K]b: [G, N, K]batch_sizes, batch_offsets: [G], i32

• moe-fwda: [4096, 4096]b: [16, 4096, 4096]batch_sizes, batch_offsets: [16], i32dtype=f16, bf16
• qwen3-30b-ep8-emptya: [8, 2048]b: [16, 1536, 2048]batch_sizes, batch_offsets: [16], i32dtype=bf16
• qwen3-30b-ep8-routed1ka: [1024, 2048]b: [16, 1536, 2048]batch_sizes, batch_offsets: [16], i32dtype=bf16
• qwen3-30b-ep8-raggeda: [257, 2048]b: [16, 1536, 2048]batch_sizes, batch_offsets: [16], i32dtype=bf16

a: [M, K]b: [G, K, N]batch_sizes, batch_offsets: [G], i32

• moe-dgrada: [4096, 4096]b: [16, 4096, 4096]batch_sizes, batch_offsets: [16], i32transpose_b=falsedtype=f16

a: [M, N]b: [M, K]batch_sizes, batch_offsets: [G], i32

• moe-wgrada, b: [4096, 4096]batch_sizes, batch_offsets: [16], i32transpose_a=truetranspose_b=falsedtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
moe-fwd	f16	1.01×	0.2201	quack torch torch-compile torch-ref	0.2216 0.2701 0.5394 2.3285	625	86%	comp
	bf16	1.01×	0.2139	quack torch torch-compile torch-ref	0.2158 0.2225 0.5381 2.2868	643	85%	comp
qwen3-30b-ep8-empty	bf16	1.15×	0.0249	quack torch torch-compile torch-ref	0.0287 0.0319 0.1071 0.4458	2.15	90%	mem
qwen3-30b-ep8-routed1k	bf16	1.01×	0.0380	quack torch torch-compile torch-ref	0.0383 0.0462 0.1772 0.5037	170	63%	mem
qwen3-30b-ep8-ragged	bf16	1.05×	0.0360	quack torch torch-compile torch-ref	0.0377 0.0481 0.1730 0.4998	45	63%	mem
moe-dgrad	f16	1.01×	0.2176	quack torch torch-compile torch-ref	0.2201 0.2688 0.2706 0.3057	632	87%	comp
moe-wgrad	f16	1.05×	0.2488	quack torch torch-compile torch-ref	0.2606 0.3541 0.3555 0.5226	553	76%	comp