GEMM, MoE & Quantization

本页暂无中文版。以下为英文原文。

21 ops, 152 workloads — Linear Algebra (GEMM) 10 · Mixture of Experts 9 · Quantization 2.

One table per op, one row per workload. Ratio is the fastest other implementation's device time divided by ours, so green is faster than it, plain is level with it, red is slower. Times are in ms. How these numbers are taken.

Linear Algebra (GEMM)

BmmFp8NKFwd (5 workloads)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
mha-decode-b32-pv-per-tensor-float8_e4m3fn	1.05×	0.00902	flashinfer-bmm-fp8 torch-fp32-ref	0.0095 0.2365	238
mha-decode-b64-qk-per-tensor-float8_e4m3fn	1.37×	0.0157	flashinfer-bmm-fp8 torch-fp32-ref	0.0216 0.2880	273
moe-prefill-b128-per-tensor-float8_e4m3fn	1.05×	0.1319	flashinfer-bmm-fp8 torch-fp32-ref	0.1384 5.4095	1,042
square-b4-1k-per-tensor-float8_e4m3fn	1.10×	0.0119	flashinfer-bmm-fp8 torch-fp32-ref	0.0131 0.2935	722
square-b8-2k-per-tensor-float8_e4m3fn	1.06×	0.1196	flashinfer-bmm-fp8 torch-fp32-ref	0.1262 4.0650	1,150

grouped_gemm_nt (2 workloads)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
nt-batch16-m4096-n4096-k4096-bfloat16	0.99×	0.2266	torch torch-compile torch-ref	0.2250 2.2494 2.2824	606
nt-batch16-m4096-n4096-k4096-float16	1.15×	0.2327	torch torch-compile torch-ref	0.2682 2.2945 2.3274	591

grouped_gemm_3wg_baselines (12 workloads)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
GLM-5-744B-down-T=131072	1.15×	38.4690	triton-tma deepgemm torch triton	44.1777 44.2571 45.2484 57.7138	686
GLM-5-744B-down-T=262144	1.00×	77.8555	deepgemm torch triton-tma triton	77.9715 82.1943 84.3373 115.0890	678
GLM-5-744B-down-T=32768	1.00×	9.6454	deepgemm torch triton-tma triton	9.6108 9.9411 11.4355 14.6883	684
GLM-5-744B-down-T=65536	0.98×	19.1609	deepgemm triton-tma torch triton	18.7690 20.9368 24.6188 28.9118	689
GLM-5-744B-up-T=131072	1.00×	74.8445	deepgemm torch triton-tma triton	75.0057 80.4625 84.9312 106.2630	705
GLM-5-744B-up-T=262144	1.00×	151.3490	deepgemm triton-tma torch triton	151.8770 163.4910 170.0550 216.2940	697
GLM-5-744B-up-T=32768	1.02×	18.1814	deepgemm torch triton-tma triton	18.4979 23.0794 23.1377 27.6289	726
GLM-5-744B-up-T=65536	1.06×	37.5122	torch deepgemm triton-tma triton	39.6492 41.1302 41.2708 54.6895	703
Llama4-128E-down-T=131072	1.01×	15.2954	deepgemm torch triton-tma triton	15.5135 15.8305 18.8759 23.0366	719
Llama4-128E-up-T=131072	0.99×	31.0676	deepgemm torch triton-tma triton	30.8841 32.1667 40.8244 52.2075	708
qwen3.5-397B-down-T52429	0.95×	6.9651	deepgemm torch triton-tma triton	6.6147 7.3609 9.0427 10.2177	631
qwen3.5-397B-up-T52429	1.00×	12.6930	deepgemm torch triton-tma triton	12.6909 13.5139 18.4964 19.6010	693

BmmFwd (16 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
mha-decode-b64-pv-bfloat16	1.02×	0.0239	torch-cublas flaggems	0.0244 0.0407	179
mha-decode-b64-pv-float16	1.02×	0.0239	torch-cublas flaggems	0.0244 0.0407	179
mha-decode-b64-qk-bfloat16	0.94×	0.0225	torch-cublas flaggems	0.0212 0.0260	191
mha-decode-b64-qk-float16	0.94×	0.0225	torch-cublas flaggems	0.0212 0.0260	191
moe-prefill-b128-bfloat16	0.74×	0.2911	torch-cublas flaggems	0.2152 0.2952	472
small-b8-128-bfloat16	1.16×	0.00272	flaggems torch-cublas	0.00317 0.00323	12.3
small-b8-128-float16	1.16×	0.00272	flaggems torch-cublas	0.00317 0.00323	12.3
square-b16-512-bfloat16	0.90×	0.0133	torch-cublas flaggems	0.0120 0.0151	323
square-b16-512-float16	0.91×	0.0132	torch-cublas flaggems	0.0121 0.0151	324
square-b32-256-bfloat16	1.07×	0.00656	torch-cublas flaggems	0.00704 0.0079	164
square-b32-256-float16	1.07×	0.00659	torch-cublas flaggems	0.00707 0.0079	163
square-b4-4k-bfloat16	0.74×	1.0435	torch-cublas flaggems	0.7705 0.9716	527
square-b8-1k-bfloat16	0.76×	0.0407	torch-cublas flaggems	0.0310 0.0448	422
square-b8-1k-float16	0.77×	0.0406	torch-cublas flaggems	0.0311 0.0449	423
square-b8-2k-bfloat16	0.73×	0.2802	torch-cublas flaggems	0.2045 0.2735	491
square-b8-2k-float16	0.74×	0.2832	torch-cublas flaggems	0.2089 0.2760	485

grouped_gemm_nn (1 workloads)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
nn-batch16-m4096-n4096-k4096-float16	0.79×	0.3407	torch torch-compile torch-ref	0.2696 0.2750 0.3074	403

BmmFp8KNFwd (5 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
mha-decode-b32-pv-per-tensor-float8_e4m3fn	0.38×	0.0647	flashinfer-bmm-fp8 torch-fp32-ref	0.0249 0.2360	33.2
mha-decode-b64-qk-per-tensor-float8_e4m3fn	0.43×	0.1154	flashinfer-bmm-fp8 torch-fp32-ref	0.0497 0.2888	37.2
moe-prefill-b128-per-tensor-float8_e4m3fn	0.69×	0.9006	flashinfer-bmm-fp8 torch-fp32-ref	0.6245 5.3998	153
square-b4-1k-per-tensor-float8_e4m3fn	0.91×	0.0390	flashinfer-bmm-fp8 torch-fp32-ref	0.0355 0.2940	220
square-b8-2k-per-tensor-float8_e4m3fn	2.03×	0.3060	flashinfer-bmm-fp8 torch-fp32-ref	0.6227 4.0584	449

GemmW4A16Fwd (6 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
compile-smoke-rect-128x256x256-float16	0.52×	0.00586	torch-dequantized-matmul	0.00307	2.87
compile-smoke-square-64x64x128-float16	0.63×	0.00429	torch-dequantized-matmul	0.00269	0.245
decode-hbm-streaming-threshold-float16	0.62×	0.0606	marlin-fp32 marlin-fp16 torch-dequantized-matmul	0.0379 0.0380 0.0748	4.43
decode-l2-resident-ish-float16	0.66×	0.0330	marlin-fp16 marlin-fp32 torch-dequantized-matmul	0.0217 0.0219 0.0468	4.06
decode-long-k-pressure-float16	0.50×	0.2834	marlin-fp16 marlin-fp32 torch-dequantized-matmul	0.1408 0.1414 0.3227	4.74
decode-non-power2-low-cta-float16	0.55×	0.0745	marlin-fp16 marlin-fp32 torch-dequantized-matmul	0.0406 0.0407 0.0876	3.94

grouped_gemm_tn (1 workloads)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
tn-batch16-m4096-n4096-k4096-float16	0.45×	0.7820	torch torch-compile torch-ref	0.3525 0.5222 0.5251	176

GemmFwd (14 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
ds-v3-decode-down-bfloat16	0.54×	0.0246	torch-cublas deepgemm	0.0132 0.0138	153
ds-v3-decode-gate-up-bfloat16	0.25×	0.0678	torch-cublas deepgemm	0.0172 0.0213	57.2
ds-v3-prefill-attn-proj-bfloat16	0.61×	0.5398	deepgemm torch-cublas	0.3301 0.3314	446
ds-v3-prefill-attn-proj-float16	0.62×	0.5448	torch-cublas	0.3368	441
ds-v3-prefill-down-bfloat16	0.56×	0.3214	deepgemm torch-cublas	0.1798 0.1804	374
ds-v3-prefill-gate-up-bfloat16	0.52×	0.3370	torch-cublas deepgemm	0.1766 0.1810	368
k-dominant-7168x16384-bfloat16	0.61×	2.0598	deepgemm torch-cublas	1.2622 1.2683	467
mid-m16-attn-bfloat16	0.37×	0.0658	torch-cublas deepgemm	0.0245 0.0339	14.3
mid-m32-attn-bfloat16	0.37×	0.0662	torch-cublas deepgemm	0.0243 0.0305	28.4
mid-m64-down-bfloat16	0.64×	0.0207	torch-cublas deepgemm	0.0132 0.0135	90.9
mid-m96-gate-up-bfloat16	0.25×	0.0687	torch-cublas deepgemm	0.0169 0.0220	42.3
square-1k-nn-bfloat16	0.50×	0.0145	torch-cublas flaggems	0.0072 0.0115	148
square-1k-nn-float16	0.50×	0.0145	torch-cublas flaggems	0.00726 0.0118	148
wide-n-24576-bfloat16	0.49×	0.9017	deepgemm torch-cublas	0.4419 0.4524	343

GemmFp8Fwd (15 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
ds-v3-decode-down-block128-float8_e4m3fn	0.25×	0.0377	flashinfer-fp8-blockscale-sm90 torch-scaled-mm	0.00925 0.2793	99.6
ds-v3-decode-down-per-tensor-float8_e4m3fn	0.41×	0.0253	deepgemm torch-scaled-mm	0.0104 0.2448	148
ds-v3-decode-gate-up-block128-float8_e4m3fn	0.09×	0.1481	flashinfer-fp8-blockscale-sm90 torch-scaled-mm	0.0129 0.2794	26.2
ds-v3-decode-gate-up-per-tensor-bias-float8_e4m3fn	2.13×	0.1166	torch-scaled-mm	0.2481	33.2
ds-v3-decode-gate-up-per-tensor-float8_e4m3fn	2.09×	0.1162	torch-scaled-mm	0.2423	33.4
ds-v3-prefill-attn-proj-block128-float8_e4m3fn	0.28×	0.7704	flashinfer-fp8-blockscale-sm90 torch-scaled-mm	0.2147 6.6789	312
ds-v3-prefill-down-block128-float8_e4m3fn	0.32×	0.4457	flashinfer-fp8-blockscale-sm90 torch-scaled-mm	0.1431 3.3979	270
ds-v3-prefill-down-per-tensor-float8_e4m3fn	0.50×	0.2109	deepgemm torch-scaled-mm	0.1056 3.3435	570
ds-v3-prefill-gate-up-block128-float8_e4m3fn	0.36×	0.3870	flashinfer-fp8-blockscale-sm90 torch-scaled-mm	0.1408 3.5322	320
ds-v3-prefill-gate-up-per-tensor-float8_e4m3fn	6.72×	0.5108	torch-scaled-mm	3.4349	243
gemv-down-m1-block128-float8_e4m3fn	0.17×	0.0446	flashinfer-fp8-blockscale-sm90 torch-scaled-mm	0.00778 0.1644	0.658
gemv-down-m1-per-tensor-float8_e4m3fn	0.39×	0.0258	deepgemm torch-scaled-mm	0.0101 0.1304	1.14
k-dominant-7168x16384-block128-float8_e4m3fn	0.22×	3.5879	flashinfer-fp8-blockscale-sm90 torch-scaled-mm	0.7726 26.4188	268
small-batch-down-m8-per-tensor-float8_e4m3fn	0.31×	0.0267	deepgemm torch-scaled-mm	0.00829 0.1666	8.81
wide-n-24576-block128-float8_e4m3fn	0.37×	1.0266	flashinfer-fp8-blockscale-sm90 torch-scaled-mm	0.3822 8.5216	301

Mixture of Experts

MoeGateUpFwd (2 workloads)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
deepseek-v3-decode-gate-up-bfloat16	2.27×	3.4594	torch-ref torch-compile	6.6366 7.8601	69.5
deepseek-v3-prefill-gate-up-bfloat16	6.14×	4.4089	torch-ref torch-compile	6.9703 27.0804	436

MoeGroupedGemmNopadFwd (4 workloads)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
deepseek-v3-decode-down-bfloat16	2.92×	1.9105	torch-ref torch-compile	2.6916 5.5871	62.9
deepseek-v3-decode-gate-up-bfloat16	1.56×	3.7451	torch-ref torch-compile	5.1691 5.8544	64.2
deepseek-v3-prefill-down-bfloat16	12.00×	2.1523	torch-ref torch-compile	2.8422 25.8232	447
deepseek-v3-prefill-gate-up-bfloat16	2.51×	4.2967	torch-ref torch-compile	5.4109 10.8057	448

MoePermuteAlignFwd (12 workloads)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
deepseek-v3-decode-int32	2.29×	0.0148	triton	0.0337	·
deepseek-v3-medium-int32	2.36×	0.0177	triton	0.0419	·
deepseek-v3-prefill-int32	1.97×	0.0378	triton	0.0743	·
deepseek-v3-small-int32	2.20×	0.0153	triton	0.0337	·
kimi-k2-decode-int32	2.87×	0.0169	triton	0.0484	·
kimi-k2-medium-int32	2.58×	0.0217	triton	0.0559	·
kimi-k2-prefill-int32	2.09×	0.0410	triton	0.0856	·
kimi-k2-small-int32	2.47×	0.0195	triton	0.0481	·
qwen3-decode-int32	1.57×	0.0108	triton	0.0169	·
qwen3-medium-int32	2.12×	0.0141	triton	0.0298	·
qwen3-prefill-int32	2.51×	0.0318	triton	0.0800	·
qwen3-small-int32	1.50×	0.0121	triton	0.0181	·

MoeUnpermuteFwd (8 workloads)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
large-hidden-decode-bfloat16	2.39×	0.00698	vllm	0.0166	0.0164
large-hidden-medium-bfloat16	1.37×	0.0214	vllm	0.0293	2.75
large-hidden-prefill-bfloat16	1.05×	0.1329	vllm	0.1391	3.53
large-hidden-small-bfloat16	2.27×	0.00787	vllm	0.0179	0.466
small-hidden-decode-bfloat16	1.57×	0.0057	vllm	0.00893	0.00863
small-hidden-medium-bfloat16	1.28×	0.0116	vllm	0.0148	2.18
small-hidden-prefill-bfloat16	1.10×	0.0615	vllm	0.0675	3.27
small-hidden-small-bfloat16	1.52×	0.00653	vllm	0.00994	0.241

MoePermuteNopadFwd (20 workloads)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
deepseek-v3-decode-bfloat16	1.25×	0.00925	vllm	0.0116	·
deepseek-v3-ep2-decode-bfloat16	—	0.0087	·	·	·
deepseek-v3-ep2-medium-bfloat16	—	0.0280	·	·	·
deepseek-v3-ep2-prefill-bfloat16	—	0.2101	·	·	·
deepseek-v3-medium-bfloat16	1.37×	0.0337	vllm	0.0460	·
deepseek-v3-prefill-bfloat16	0.97×	0.2789	vllm	0.2700	·
deepseek-v3-small-bfloat16	1.32×	0.0104	vllm	0.0138	·
kimi-k2-decode-bfloat16	1.10×	0.0106	vllm	0.0117	·
kimi-k2-medium-bfloat16	1.29×	0.0356	vllm	0.0460	·
kimi-k2-prefill-bfloat16	0.95×	0.2855	vllm	0.2709	·
kimi-k2-small-bfloat16	1.17×	0.0118	vllm	0.0139	·
qwen3-235b-decode-bfloat16	1.43×	0.00803	vllm	0.0115	·
qwen3-235b-ep2-medium-bfloat16	—	0.0264	·	·	·
qwen3-235b-medium-bfloat16	1.47×	0.0313	vllm	0.0461	·
qwen3-235b-prefill-bfloat16	0.97×	0.2686	vllm	0.2612	·
qwen3-235b-small-bfloat16	1.53×	0.00902	vllm	0.0138	·
qwen3-30b-decode-bfloat16	1.67×	0.0063	vllm	0.0105	·
qwen3-30b-medium-bfloat16	1.40×	0.0207	vllm	0.0290	·
qwen3-30b-prefill-bfloat16	0.91×	0.1419	vllm	0.1295	·
qwen3-30b-small-bfloat16	1.74×	0.0072	vllm	0.0125	·

FusedTopK (8 workloads)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
1-128-8-softmax-norenormalize	1.42×	0.00426	vllm	0.00605	0.000541
1-384-8-sigmoid-renormalize	1.00×	0.00829	vllm	0.00829	0.000834
32-128-8-softmax-norenormalize	1.12×	0.00739	vllm	0.00829	0.00997
32-384-8-sigmoid-renormalize	0.81×	0.0119	vllm	0.0097	0.0186
4096-128-8-softmax-norenormalize	1.47×	0.0110	vllm	0.0161	0.86
4096-384-8-sigmoid-renormalize	1.17×	0.0203	vllm	0.0238	1.4
512-128-8-softmax-norenormalize	1.16×	0.00781	vllm	0.00902	0.151
512-384-8-sigmoid-renormalize	0.83×	0.0126	vllm	0.0105	0.281

FusedMoeFwd (6 workloads)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
deepseek-v3-decode-bfloat16	1.02×	5.4253	vllm	5.5191	66.5
deepseek-v3-prefill-bfloat16	1.06×	8.3550	vllm	8.8652	345
kimi-k2-decode-bfloat16	14.56×	3.8889	torch-ref	56.6357	92.8
kimi-k2-prefill-bfloat16	17.88×	7.8753	torch-ref	140.8010	366
qwen3-235b-decode-bfloat16	1.03×	2.7727	vllm	2.8575	130
qwen3-235b-prefill-bfloat16	1.20×	6.0667	vllm	7.2789	476

FusedMoEExpertsNopadPersistent3WGFwd (6 workloads)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
deepseek-v3-decode-bfloat16	1.02×	5.4159	vllm-triton	5.5120	66.6
deepseek-v3-ep2-decode-bfloat16	—	2.7228	·	·	133
deepseek-v3-ep2-prefill-bfloat16	—	4.1410	·	·	697
deepseek-v3-prefill-bfloat16	1.04×	8.4890	vllm-triton	8.8313	340
qwen3-235b-decode-bfloat16	1.03×	2.7726	vllm-triton	2.8545	130
qwen3-235b-prefill-bfloat16	1.20×	6.0374	vllm-triton	7.2463	478

SharedFusedMoE (5 workloads)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
1-384-8-7168-2048-18432-sigmoid-renormalize-correctionbias-2.827-bfloat16	0.17×	2.5280	vllm	0.4262	0.592
2048-384-8-7168-2048-18432-sigmoid-renormalize-correctionbias-2.827-bfloat16	0.59×	19.5743	vllm	11.6008	157
32-384-8-7168-2048-18432-sigmoid-renormalize-correctionbias-2.827-bfloat16	0.84×	4.7455	vllm	3.9647	10.1
4096-384-8-7168-2048-18432-sigmoid-renormalize-correctionbias-2.827-bfloat16	0.45×	32.5463	vllm	14.6270	188
512-384-8-7168-2048-18432-sigmoid-renormalize-correctionbias-2.827-bfloat16	1.09×	8.0566	vllm	8.7697	95.2

Quantization

FP8LightningIndexerFwd (1 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
lightning-indexer-s8k-h32-d64-bfloat16	80.40×	0.6174	torch-compile torch-ref	49.6420 111.9970	55.6

FP8QuantFwd (3 workloads · ✅)

Workload	Ratio	Device time	Alternatives		Throughput
	alt / ours	ms	name	ms	TFLOP/s
kv-index-4k-d128-float32	1.06×	0.00397	torch-compile torch-ref	0.00422 0.0155	0.794
kv-index-8k-d64-bfloat16	0.99×	0.00275	torch-compile torch-ref	0.00272 0.0169	1.15
kv-index-8k-d64-float16	2.43×	0.00275	torch-compile torch-ref	0.00669 0.0167	1.15