MoE

本页暂无中文版。以下为英文原文。

12 ops, 132 workloads.

One table per op, one row per workload. Ratio is the fastest other implementation's device time divided by ours, so green is faster than it, plain is level with it, red is slower. Times are in ms. How these numbers are taken.

FusedMoEFwd

hidden_states: [T, H]gating_output: [T, E], f32w_gate_up: [E, 2 * F, H]w_down: [E, H, F]

top_k=8renormalize=true

• qwen3-235b-decodehidden_states: [512, 4096]gating_output: [512, 128], f32w_gate_up: [128, 3072, 4096]w_down: [128, 4096, 1536]dtype=bf16, f16
• qwen3-235b-prefillhidden_states: [4096, 4096]gating_output: [4096, 128], f32w_gate_up: [128, 3072, 4096]w_down: [128, 4096, 1536]dtype=bf16
• qwen3-30b-decode-b1hidden_states: [1, 2048]gating_output: [1, 128], f32w_gate_up: [128, 1536, 2048]w_down: [128, 2048, 768]dtype=bf16
• qwen3-30b-decode-b32hidden_states: [32, 2048]gating_output: [32, 128], f32w_gate_up: [128, 1536, 2048]w_down: [128, 2048, 768]dtype=bf16

hidden_states: [T, H]gating_output: [T, E], f32w_gate_up: [E, 2 * F, H]w_down: [E, H, F]correction_bias: [E], f32

top_k=8scoring_func=sigmoidrenormalize=truedtype=bf16

• ds-v3-grouped-decodehidden_states: [512, 7168]gating_output: [512, 256], f32w_gate_up: [256, 4096, 7168]w_down: [256, 7168, 2048]correction_bias: [256], f32num_expert_group=8topk_group=4routed_scaling_factor=2.5
• ds-v3-grouped-prefillhidden_states: [4096, 7168]gating_output: [4096, 256], f32w_gate_up: [256, 4096, 7168]w_down: [256, 7168, 2048]correction_bias: [256], f32num_expert_group=8topk_group=4routed_scaling_factor=2.5
• kimi-k2-decodehidden_states: [512, 7168]gating_output: [512, 256], f32w_gate_up: [256, 4096, 7168]w_down: [256, 7168, 2048]correction_bias: [256], f32routed_scaling_factor=2.827
• kimi-k2-prefillhidden_states: [4096, 7168]gating_output: [4096, 256], f32w_gate_up: [256, 4096, 7168]w_down: [256, 7168, 2048]correction_bias: [256], f32routed_scaling_factor=2.827

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
qwen3-235b-decode	bf16	1.01×	1.2042	vllm flashinfer-cutlass	1.2195 12.9682	128	89%	mem
	f16	1.01×	1.2004	vllm flashinfer-cutlass	1.2179 13.0539	129	90%	mem
qwen3-235b-prefill	bf16	1.22×	2.5018	vllm flashinfer-cutlass	3.0587 20.3113	495	65%	comp
qwen3-30b-decode-b1	bf16	1.04×	0.0396	vllm flashinfer-cutlass	0.0413 1.7471	1.91	42%	mem
qwen3-30b-decode-b32	bf16	1.03×	0.2656	vllm flashinfer-cutlass	0.2733 3.3502	9.11	88%	mem
ds-v3-grouped-decode	bf16	1.01×	3.7285	vllm flashinfer-cutlass	3.7556 41.2632	96.8	91%	mem
ds-v3-grouped-prefill	bf16	1.22×	7.1078	vllm flashinfer-cutlass	8.6549 61.3657	406	56%	mem
kimi-k2-decode	bf16	1.02×	3.6029	vllm flashinfer-cutlass	3.6846 41.0826	100	91%	mem
kimi-k2-prefill	bf16	1.21×	6.9423	vllm flashinfer-cutlass	8.3939 60.6928	416	55%	mem

FusedMoESharedExpertFwd

hidden_states: [T, H]gating_output: [T, E], f32w_gate_up: [E, 2 * F, H]w_down: [E, H, F]correction_bias: [E], f32shared_w_gate_up: [2 * S, H]shared_w_down: [H, S]

top_k=8scoring_func=sigmoidrenormalize=true

• kimi-k2-t1hidden_states: [1, 7168]gating_output: [1, 384], f32w_gate_up: [384, 4096, 7168]w_down: [384, 7168, 2048]correction_bias: [384], f32shared_w_gate_up: [36864, 7168]shared_w_down: [7168, 18432]routed_scaling_factor=2.827dtype=bf16
• kimi-k2-t32hidden_states: [32, 7168]gating_output: [32, 384], f32w_gate_up: [384, 4096, 7168]w_down: [384, 7168, 2048]correction_bias: [384], f32shared_w_gate_up: [36864, 7168]shared_w_down: [7168, 18432]routed_scaling_factor=2.827dtype=bf16
• kimi-k2-t64hidden_states: [64, 7168]gating_output: [64, 384], f32w_gate_up: [384, 4096, 7168]w_down: [384, 7168, 2048]correction_bias: [384], f32shared_w_gate_up: [36864, 7168]shared_w_down: [7168, 18432]routed_scaling_factor=2.827dtype=bf16
• kimi-k2-t128hidden_states: [128, 7168]gating_output: [128, 384], f32w_gate_up: [384, 4096, 7168]w_down: [384, 7168, 2048]correction_bias: [384], f32shared_w_gate_up: [36864, 7168]shared_w_down: [7168, 18432]routed_scaling_factor=2.827dtype=bf16
• kimi-k2-t512hidden_states: [512, 7168]gating_output: [512, 384], f32w_gate_up: [384, 4096, 7168]w_down: [384, 7168, 2048]correction_bias: [384], f32shared_w_gate_up: [36864, 7168]shared_w_down: [7168, 18432]routed_scaling_factor=2.827dtype=bf16
• kimi-k2-t2048hidden_states: [2048, 7168]gating_output: [2048, 384], f32w_gate_up: [384, 4096, 7168]w_down: [384, 7168, 2048]correction_bias: [384], f32shared_w_gate_up: [36864, 7168]shared_w_down: [7168, 18432]routed_scaling_factor=2.827dtype=bf16
• kimi-k2-t4096hidden_states: [4096, 7168]gating_output: [4096, 384], f32w_gate_up: [384, 4096, 7168]w_down: [384, 7168, 2048]correction_bias: [384], f32shared_w_gate_up: [36864, 7168]shared_w_down: [7168, 18432]routed_scaling_factor=2.827dtype=bf16
• ds-v3-grouped-t1hidden_states: [1, 7168]gating_output: [1, 256], f32w_gate_up: [256, 4096, 7168]w_down: [256, 7168, 2048]correction_bias: [256], f32shared_w_gate_up: [4096, 7168]shared_w_down: [7168, 2048]num_expert_group=8topk_group=4routed_scaling_factor=2.5dtype=bf16
• ds-v3-grouped-t32hidden_states: [32, 7168]gating_output: [32, 256], f32w_gate_up: [256, 4096, 7168]w_down: [256, 7168, 2048]correction_bias: [256], f32shared_w_gate_up: [4096, 7168]shared_w_down: [7168, 2048]num_expert_group=8topk_group=4routed_scaling_factor=2.5dtype=bf16, f16
• ds-v3-grouped-t64hidden_states: [64, 7168]gating_output: [64, 256], f32w_gate_up: [256, 4096, 7168]w_down: [256, 7168, 2048]correction_bias: [256], f32shared_w_gate_up: [4096, 7168]shared_w_down: [7168, 2048]num_expert_group=8topk_group=4routed_scaling_factor=2.5dtype=bf16
• ds-v3-grouped-t128hidden_states: [128, 7168]gating_output: [128, 256], f32w_gate_up: [256, 4096, 7168]w_down: [256, 7168, 2048]correction_bias: [256], f32shared_w_gate_up: [4096, 7168]shared_w_down: [7168, 2048]num_expert_group=8topk_group=4routed_scaling_factor=2.5dtype=bf16
• ds-v3-grouped-t512hidden_states: [512, 7168]gating_output: [512, 256], f32w_gate_up: [256, 4096, 7168]w_down: [256, 7168, 2048]correction_bias: [256], f32shared_w_gate_up: [4096, 7168]shared_w_down: [7168, 2048]num_expert_group=8topk_group=4routed_scaling_factor=2.5dtype=bf16
• ds-v3-grouped-t2048hidden_states: [2048, 7168]gating_output: [2048, 256], f32w_gate_up: [256, 4096, 7168]w_down: [256, 7168, 2048]correction_bias: [256], f32shared_w_gate_up: [4096, 7168]shared_w_down: [7168, 2048]num_expert_group=8topk_group=4routed_scaling_factor=2.5dtype=bf16
• ds-v3-grouped-t4096hidden_states, shared_w_gate_up: [4096, 7168]gating_output: [4096, 256], f32w_gate_up: [256, 4096, 7168]w_down: [256, 7168, 2048]correction_bias: [256], f32shared_w_down: [7168, 2048]num_expert_group=8topk_group=4routed_scaling_factor=2.5dtype=bf16
• glm45-t1hidden_states: [1, 5120]gating_output: [1, 160], f32w_gate_up: [160, 3072, 5120]w_down: [160, 5120, 1536]correction_bias: [160], f32shared_w_gate_up: [3072, 5120]shared_w_down: [5120, 1536]routed_scaling_factor=2.5dtype=bf16
• glm45-t32hidden_states: [32, 5120]gating_output: [32, 160], f32w_gate_up: [160, 3072, 5120]w_down: [160, 5120, 1536]correction_bias: [160], f32shared_w_gate_up: [3072, 5120]shared_w_down: [5120, 1536]routed_scaling_factor=2.5dtype=bf16
• glm45-t64hidden_states: [64, 5120]gating_output: [64, 160], f32w_gate_up: [160, 3072, 5120]w_down: [160, 5120, 1536]correction_bias: [160], f32shared_w_gate_up: [3072, 5120]shared_w_down: [5120, 1536]routed_scaling_factor=2.5dtype=bf16
• glm45-t128hidden_states: [128, 5120]gating_output: [128, 160], f32w_gate_up: [160, 3072, 5120]w_down: [160, 5120, 1536]correction_bias: [160], f32shared_w_gate_up: [3072, 5120]shared_w_down: [5120, 1536]routed_scaling_factor=2.5dtype=bf16
• glm45-t512hidden_states: [512, 5120]gating_output: [512, 160], f32w_gate_up: [160, 3072, 5120]w_down: [160, 5120, 1536]correction_bias: [160], f32shared_w_gate_up: [3072, 5120]shared_w_down: [5120, 1536]routed_scaling_factor=2.5dtype=bf16
• glm45-t2048hidden_states: [2048, 5120]gating_output: [2048, 160], f32w_gate_up: [160, 3072, 5120]w_down: [160, 5120, 1536]correction_bias: [160], f32shared_w_gate_up: [3072, 5120]shared_w_down: [5120, 1536]routed_scaling_factor=2.5dtype=bf16
• glm45-t4096hidden_states: [4096, 5120]gating_output: [4096, 160], f32w_gate_up: [160, 3072, 5120]w_down: [160, 5120, 1536]correction_bias: [160], f32shared_w_gate_up: [3072, 5120]shared_w_down: [5120, 1536]routed_scaling_factor=2.5dtype=bf16

hidden_states: [T, H]gating_output: [T, E], f32w_gate_up: [E, 2 * F, H]w_down: [E, H, F]

• qwen3-235b-t512hidden_states: [512, 4096]gating_output: [512, 128], f32w_gate_up: [128, 3072, 4096]w_down: [128, 4096, 1536]top_k=8renormalize=truedtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
kimi-k2-t1	bf16	1.05×	0.3858	vllm flashinfer-cutlass	0.4069 31.6349	3.88	86%	mem
kimi-k2-t32	bf16	1.00×	2.3044	vllm flashinfer-cutlass	2.3114 43.9815	20.8	98%	mem
kimi-k2-t64	bf16	1.00×	2.9120	vllm flashinfer-cutlass	2.9263 47.9848	32.9	97%	mem
kimi-k2-t128	bf16	1.00×	3.5305	vllm flashinfer-cutlass	3.5384 50.3655	54.3	95%	mem
kimi-k2-t512	bf16	1.01×	5.0723	vllm flashinfer-cutlass	5.1333 56.6982	151	86%	mem
kimi-k2-t2048	bf16	1.07×	8.5828	vllm flashinfer-cutlass	9.1615 66.3708	357	57%	mem
kimi-k2-t4096	bf16	1.12×	12.8296	vllm flashinfer-cutlass	14.4330 83.3147	478	63%	comp
ds-v3-grouped-t1	bf16	1.13×	0.2106	vllm flashinfer-cutlass	0.2380 21.3874	3.76	84%	mem
ds-v3-grouped-t32	bf16	1.00×	2.1154	vllm flashinfer-cutlass	2.1242 33.7453	12	99%	mem
	f16	1.00×	2.1155	vllm flashinfer-cutlass	2.1258 33.7535	12	99%	mem
ds-v3-grouped-t64	bf16	1.00×	2.4982	vllm flashinfer-cutlass	2.4986 35.8025	20.3	99%	mem
ds-v3-grouped-t128	bf16	1.01×	3.0030	vllm flashinfer-cutlass	3.0242 38.4954	33.8	96%	mem
ds-v3-grouped-t512	bf16	1.02×	3.9543	vllm flashinfer-cutlass	4.0385 42.8967	103	91%	mem
ds-v3-grouped-t2048	bf16	1.10×	5.7451	vllm flashinfer-cutlass	6.3275 51.0606	283	70%	mem
ds-v3-grouped-t4096	bf16	1.20×	7.8860	vllm flashinfer-cutlass	9.4558 62.5880	412	54%	comp
glm45-t1	bf16	1.11×	0.1317	vllm flashinfer-cutlass	0.1459 7.4850	3.23	72%	mem
glm45-t32	bf16	1.02×	0.8389	vllm flashinfer-cutlass	0.8570 12.3194	16.2	95%	mem
glm45-t64	bf16	1.00×	0.9949	vllm flashinfer-cutlass	0.9995 13.0568	27.3	93%	mem
glm45-t128	bf16	1.04×	1.1947	vllm flashinfer-cutlass	1.2484 14.0787	45.5	92%	mem
glm45-t512	bf16	1.01×	1.5990	vllm flashinfer-cutlass	1.6214 15.4943	136	85%	mem
glm45-t2048	bf16	1.17×	2.4754	vllm flashinfer-cutlass	2.9013 20.1323	351	60%	mem
glm45-t4096	bf16	1.22×	3.7208	vllm flashinfer-cutlass	4.5399 26.6657	468	62%	comp
qwen3-235b-t512	bf16	1.02×	1.1995	vllm flashinfer-cutlass	1.2177 12.9531	129	90%	mem

FusedTopKFwd

gating_output: [T, E]

top_k=8renormalize=true

• kimi-k2-t32gating_output: [32, 384]scoring_func=sigmoiddtype=bf16
• kimi-k2-t512gating_output: [512, 384]scoring_func=sigmoiddtype=bf16
• kimi-k2-t4096gating_output: [4096, 384]scoring_func=sigmoiddtype=bf16
• qwen3-235b-t32gating_output: [32, 128]dtype=bf16
• qwen3-235b-t512gating_output: [512, 128]dtype=bf16, f16, f32
• qwen3-235b-t4096gating_output: [4096, 128]dtype=bf16

gating_output: [T, E]correction_bias: [E], f32

top_k=8scoring_func=sigmoidrenormalize=true

• kimi-k2-t512-biasgating_output: [512, 384]correction_bias: [384], f32dtype=bf16, f16, f32
• ds-v3-grouped-t1gating_output: [1, 256]correction_bias: [256], f32num_expert_group=8topk_group=4routed_scaling_factor=2.5dtype=bf16
• ds-v3-grouped-t32gating_output: [32, 256]correction_bias: [256], f32num_expert_group=8topk_group=4routed_scaling_factor=2.5dtype=bf16
• ds-v3-grouped-t64gating_output: [64, 256]correction_bias: [256], f32num_expert_group=8topk_group=4routed_scaling_factor=2.5dtype=bf16, f16, f32
• ds-v3-grouped-t4096gating_output: [4096, 256]correction_bias: [256], f32num_expert_group=8topk_group=4routed_scaling_factor=2.5dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
kimi-k2-t32	bf16	3.02×	0.00326	vllm	0.00986	0.0453	0%	lat
kimi-k2-t512	bf16	2.92×	0.00362	vllm	0.0106	0.655	3%	lat
kimi-k2-t4096	bf16	2.27×	0.0106	vllm	0.0241	1.78	7%	lat
qwen3-235b-t32	bf16	2.98×	0.00282	vllm	0.00838	0.0162	0%	lat
qwen3-235b-t512	bf16	3.00×	0.00304	vllm	0.00912	0.24	1%	lat
	f16	2.98×	0.00304	vllm	0.00906	0.24	1%	lat
	f32	1.82×	0.00307	vllm	0.0056	0.237	2%	lat
qwen3-235b-t4096	bf16	2.47×	0.00659	vllm	0.0163	0.885	4%	lat
kimi-k2-t512-bias	bf16	2.63×	0.00419	vllm	0.0110	0.612	2%	lat
	f16	2.62×	0.00419	vllm	0.0110	0.612	2%	lat
	f32	1.67×	0.00432	vllm	0.0072	0.594	4%	lat
ds-v3-grouped-t1	bf16	1.15×	0.00355	vllm	0.0041	0.00117	0%	lat
ds-v3-grouped-t32	bf16	1.21×	0.00355	vllm	0.00429	0.0375	0%	lat
ds-v3-grouped-t64	bf16	1.19×	0.00365	vllm	0.00435	0.073	0%	lat
	f16	1.20×	0.00362	vllm	0.00435	0.0736	0%	lat
	f32	1.20×	0.00362	vllm	0.00435	0.0736	0%	lat
ds-v3-grouped-t4096	bf16	1.49×	0.0120	vllm	0.0180	1.42	4%	lat

MoEPrePermuteFwd

hidden_states: [T, H]local_expert_ids: [T, K], i32

layout=contiguous(packing=tight,metadata_kind=physical_psum,alignment=1)

• decode-t32hidden_states: [32, 128]local_expert_ids: [32, 2], i32num_local_experts=4dtype=f16, bf16
• prefill-t512hidden_states: [512, 128]local_expert_ids: [512, 2], i32num_local_experts=4dtype=f16, bf16
• ds-v3-ep8-t100hidden_states: [100, 7168]local_expert_ids: [100, 8], i32num_local_experts=32dtype=f16, bf16
• qwen3-30b-ep8-b1hidden_states: [1, 2048]local_expert_ids: [1, 8], i32num_local_experts=16dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
decode-t32	f16	3.61×	0.00278	vllm torch-ref	0.0100 0.0251	·	0%	lat
	bf16	3.68×	0.00282	vllm torch-ref	0.0104 0.0260	·	0%	lat
prefill-t512	f16	3.44×	0.00358	vllm torch-ref	0.0123 0.0673	·	2%	lat
	bf16	3.44×	0.00358	vllm torch-ref	0.0123 0.0672	·	2%	lat
ds-v3-ep8-t100	f16	2.41×	0.00762	vllm torch-ref	0.0183 0.0660	·	38%	lat
	bf16	2.41×	0.00762	vllm torch-ref	0.0183 0.0660	·	38%	lat
qwen3-30b-ep8-b1	bf16	3.46×	0.00278	vllm torch-ref	0.00963 0.0234	·	0%	lat

MoEPermuteAlignFwd

topk_ids: [T, K]

dtype=i32

• kimi-k2-t1topk_ids: [1, 8]num_experts=384
• kimi-k2-t32topk_ids: [32, 8]num_experts=384
• kimi-k2-t512topk_ids: [512, 8]num_experts=384
• kimi-k2-t4096topk_ids: [4096, 8]num_experts=384
• ds-v3-t1topk_ids: [1, 8]num_experts=256
• ds-v3-t32topk_ids: [32, 8]num_experts=256
• ds-v3-t512topk_ids: [512, 8]num_experts=256
• ds-v3-t4096topk_ids: [4096, 8]num_experts=256
• qwen3-235b-t1topk_ids: [1, 8]num_experts=128
• qwen3-235b-t32topk_ids: [32, 8]num_experts=128
• qwen3-235b-t512topk_ids: [512, 8]num_experts=128
• qwen3-235b-t4096topk_ids: [4096, 8]num_experts=128

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
kimi-k2-t1	i32	1.22×	0.00422	vllm triton	0.00515 0.0460	·	1%	lat
kimi-k2-t32	i32	1.18×	0.00493	vllm triton	0.00579 0.0477	·	0%	lat
kimi-k2-t512	i32	1.43×	0.00547	vllm triton	0.00781 0.0553	·	1%	lat
kimi-k2-t4096	i32	2.82×	0.00925	vllm triton	0.0260 0.0842	·	1%	lat
ds-v3-t1	i32	1.16×	0.00451	vllm triton	0.00525 0.0327	·	0%	lat
ds-v3-t32	i32	1.15×	0.00486	vllm triton	0.0056 0.0319	·	0%	lat
ds-v3-t512	i32	1.48×	0.00541	vllm triton	0.00798 0.0415	·	0%	lat
ds-v3-t4096	i32	3.14×	0.00899	vllm triton	0.0282 0.0752	·	1%	lat
qwen3-235b-t1	i32	1.18×	0.00438	vllm triton	0.00518 0.0165	·	0%	lat
qwen3-235b-t32	i32	1.18×	0.0047	vllm triton	0.00557 0.0172	·	0%	lat
qwen3-235b-t512	i32	1.44×	0.00518	vllm triton	0.00749 0.0293	·	0%	lat
qwen3-235b-t4096	i32	3.13×	0.00826	vllm triton	0.0259 0.0788	·	1%	lat

MoEPostPermuteFwd

expert_output: [P, H]topk_weights: [T, K], f32inverse_indices: [T * K], i32

layout=contiguous(packing=tight,metadata_kind=physical_psum,alignment=1)

• decode-t32expert_output: [64, 128]topk_weights: [32, 2], f32inverse_indices: [64], i32dtype=f16, bf16, f16/f16, bf16/bf16
• prefill-t512expert_output: [1024, 128]topk_weights: [512, 2], f32inverse_indices: [1024], i32dtype=f16, bf16
• ds-v3-ep8-t100expert_output: [800, 7168]topk_weights: [100, 8], f32inverse_indices: [800], i32dtype=f16, bf16
• qwen3-30b-ep8-b1expert_output: [8, 2048]topk_weights: [1, 8], f32inverse_indices: [8], i32dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
decode-t32	f16	1.42×	0.00176	vllm torch-ref	0.0025 0.0137	0.00931	0%	lat
	bf16	1.38×	0.00179	vllm torch-ref	0.00248 0.0136	0.00915	0%	lat
	f16/f16	1.39×	0.00179	vllm torch-ref	0.0025 0.0137	0.00914	0%	lat
	bf16/bf16	1.39×	0.00179	vllm torch-ref	0.0025 0.0136	0.00914	0%	lat
prefill-t512	f16	1.29×	0.00224	vllm torch-ref	0.00288 0.0177	0.117	4%	lat
	bf16	1.29×	0.00224	vllm torch-ref	0.00288 0.0177	0.117	4%	lat
ds-v3-ep8-t100	f16	3.22×	0.00608	vllm torch-ref	0.0196 0.0605	1.89	47%	lat
	bf16	3.19×	0.00614	vllm torch-ref	0.0196 0.0610	1.87	47%	lat
qwen3-30b-ep8-b1	bf16	2.52×	0.00211	vllm torch-ref	0.00531 0.0139	0.0155	0%	lat

MoEGroupedGemmFwd

a: [P, K]b: [E, N, K]layout_metadata: [E], i32

layout=contiguous(packing=tight,metadata_kind=physical_psum,alignment=1)

• ds-v3-decode-mlp-upa: [4096, 7168]b: [256, 4096, 7168]layout_metadata: [256], i32dtype=bf16, f16
• ds-v3-prefill-mlp-upa: [32768, 7168]b: [256, 4096, 7168]layout_metadata: [256], i32dtype=bf16
• ds-v3-decode-downa: [4096, 2048]b: [256, 7168, 2048]layout_metadata: [256], i32dtype=bf16, bf16/f32
• ds-v3-prefill-downa: [32768, 2048]b: [256, 7168, 2048]layout_metadata: [256], i32dtype=bf16
• qwen3-30b-ep8-decodea: [8, 2048]b: [16, 1536, 2048]layout_metadata: [16], i32dtype=bf16
• qwen3-30b-ep8-routed256a: [256, 2048]b: [16, 1536, 2048]layout_metadata: [16], i32dtype=bf16

a: [P, K]b: [E, 2 * N, K]layout_metadata: [E], i32

activation=silu_and_mullayout=contiguous(packing=tight,metadata_kind=physical_psum,alignment=1)dtype=bf16

• ds-v3-decode-swiglua: [4096, 7168]b: [256, 4096, 7168]layout_metadata: [256], i32
• ds-v3-prefill-swiglua: [32768, 7168]b: [256, 4096, 7168]layout_metadata: [256], i32

a: [E, layout.max_m, K]b: [E, N, K]layout_metadata: [E], i32

• ds-v3-ep8-maskeda: [32, 128, 7168]b: [32, 4096, 7168]layout_metadata: [32], i32dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
ds-v3-decode-mlp-up	bf16	1.36×	3.4318	torch-grouped-mm	4.6780	70.1	98%	mem
	f16	1.48×	3.4407	torch-grouped-mm	5.0776	69.9	98%	mem
ds-v3-prefill-mlp-up	bf16	1.09×	3.9855	torch-grouped-mm	4.3552	483	88%	mem
ds-v3-decode-down	bf16	1.29×	1.7814	torch-grouped-mm	2.2965	67.5	95%	mem
	bf16/f32	—	1.7978	·	·	66.9	95%	mem
ds-v3-prefill-down	bf16	1.18×	2.0244	torch-grouped-mm	2.3801	475	89%	mem
qwen3-30b-ep8-decode	bf16	1.35×	0.0238	torch-grouped-mm	0.0321	2.11	47%	mem
qwen3-30b-ep8-routed256	bf16	1.30×	0.0348	torch-grouped-mm	0.0453	46.3	65%	mem
ds-v3-decode-swiglu	bf16	1.38×	3.4283	torch-grouped-mm	4.7245	70.2	98%	mem
ds-v3-prefill-swiglu	bf16	1.19×	3.9920	torch-grouped-mm	4.7685	482	87%	mem
ds-v3-ep8-masked	bf16	—	0.4851	·	·	372	89%	mem

MoEGroupedGemmFP8Fwd

a: [E, layout.max_m, K]a_scale: [E, layout.max_m, ceil_div(K, 128)], f32b: [E, N, K]b_scale: [E, ceil_div(N, 128), ceil_div(K, 128)], f32layout_metadata: [E], i32

dtype=fp8e4m3

• ds-v3-e8-up-maskeda: [8, 128, 7168]a_scale: [8, 128, 56], f32b: [8, 4096, 7168]b_scale: [8, 32, 56], f32layout_metadata: [8], i32T=768
• ds-v3-e8-down-maskeda: [8, 128, 2048]a_scale: [8, 128, 16], f32b: [8, 7168, 2048]b_scale: [8, 56, 16], f32layout_metadata: [8], i32T=768
• ds-v3-dec-b64-upa: [32, 64, 7168]a_scale: [32, 64, 56], f32b: [32, 4096, 7168]b_scale: [32, 32, 56], f32layout_metadata: [32], i32T=64
• ds-v3-dec-b64-downa: [32, 64, 2048]a_scale: [32, 64, 16], f32b: [32, 7168, 2048]b_scale: [32, 56, 16], f32layout_metadata: [32], i32T=64
• ds-v3-dec-b256-upa: [32, 256, 7168]a_scale: [32, 256, 56], f32b: [32, 4096, 7168]b_scale: [32, 32, 56], f32layout_metadata: [32], i32T=256
• ds-v3-dec-b256-downa: [32, 256, 2048]a_scale: [32, 256, 16], f32b: [32, 7168, 2048]b_scale: [32, 56, 16], f32layout_metadata: [32], i32T=256
• qwen3-235b-dec-b64-upa: [16, 64, 4096]a_scale: [16, 64, 32], f32b: [16, 3072, 4096]b_scale: [16, 24, 32], f32layout_metadata: [16], i32T=64
• qwen3-235b-dec-b64-downa: [16, 64, 1536]a_scale: [16, 64, 12], f32b: [16, 4096, 1536]b_scale: [16, 32, 12], f32layout_metadata: [16], i32T=64
• qwen3-235b-dec-b256-upa: [16, 256, 4096]a_scale: [16, 256, 32], f32b: [16, 3072, 4096]b_scale: [16, 24, 32], f32layout_metadata: [16], i32T=256
• qwen3-235b-dec-b256-downa: [16, 256, 1536]a_scale: [16, 256, 12], f32b: [16, 4096, 1536]b_scale: [16, 32, 12], f32layout_metadata: [16], i32T=256
• kimi-k2-dec-b64-upa: [48, 64, 7168]a_scale: [48, 64, 56], f32b: [48, 4096, 7168]b_scale: [48, 32, 56], f32layout_metadata: [48], i32T=64
• kimi-k2-dec-b64-downa: [48, 64, 2048]a_scale: [48, 64, 16], f32b: [48, 7168, 2048]b_scale: [48, 56, 16], f32layout_metadata: [48], i32T=64
• kimi-k2-dec-b256-upa: [48, 256, 7168]a_scale: [48, 256, 56], f32b: [48, 4096, 7168]b_scale: [48, 32, 56], f32layout_metadata: [48], i32T=256
• kimi-k2-dec-b256-downa: [48, 256, 2048]a_scale: [48, 256, 16], f32b: [48, 7168, 2048]b_scale: [48, 56, 16], f32layout_metadata: [48], i32T=256

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
ds-v3-e8-up-masked	fp8e4m3	0.98×	0.0743	deepgemm	0.0728	607	74%	mem
ds-v3-e8-down-masked	fp8e4m3	1.00×	0.0424	deepgemm	0.0424	532	68%	mem
ds-v3-dec-b64-up	fp8e4m3	1.02×	0.2266	deepgemm	0.2318	16.6	81%	mem
ds-v3-dec-b64-down	fp8e4m3	1.04×	0.1157	deepgemm	0.1209	16.2	79%	mem
ds-v3-dec-b256-up	fp8e4m3	1.01×	0.2420	deepgemm	0.2444	62.1	87%	mem
ds-v3-dec-b256-down	fp8e4m3	1.02×	0.1309	deepgemm	0.1338	57.4	80%	mem
qwen3-235b-dec-b64-up	fp8e4m3	1.02×	0.0598	deepgemm	0.0609	26.9	75%	mem
qwen3-235b-dec-b64-down	fp8e4m3	1.05×	0.0351	deepgemm	0.0368	22.9	64%	mem
qwen3-235b-dec-b256-up	fp8e4m3	1.01×	0.0605	deepgemm	0.0612	106	75%	mem
qwen3-235b-dec-b256-down	fp8e4m3	1.03×	0.0358	deepgemm	0.0370	90	64%	mem
kimi-k2-dec-b64-up	fp8e4m3	1.02×	0.2638	deepgemm	0.2683	14.2	87%	mem
kimi-k2-dec-b64-down	fp8e4m3	1.04×	0.1430	deepgemm	0.1493	13.1	80%	mem
kimi-k2-dec-b256-up	fp8e4m3	1.01×	0.3545	deepgemm	0.3568	42.4	89%	mem
kimi-k2-dec-b256-down	fp8e4m3	1.01×	0.1892	deepgemm	0.1906	39.7	83%	mem

MoEExpertMLPFwd

expert_input: [P, H]w_gate_up: [E, 2 * F, H]w_down: [E, H, F]layout_metadata: [E], i32

layout=contiguous(packing=tight,metadata_kind=physical_psum,alignment=1)dtype=f16, bf16

• qwen3-235b-decodeexpert_input: [4096, 4096]w_gate_up: [128, 3072, 4096]w_down: [128, 4096, 1536]layout_metadata: [128], i32
• qwen3-235b-prefillexpert_input: [32768, 4096]w_gate_up: [128, 3072, 4096]w_down: [128, 4096, 1536]layout_metadata: [128], i32
• ds-v3-decodeexpert_input: [4096, 7168]w_gate_up: [256, 4096, 7168]w_down: [256, 7168, 2048]layout_metadata: [256], i32
• ds-v3-prefillexpert_input: [32768, 7168]w_gate_up: [256, 4096, 7168]w_down: [256, 7168, 2048]layout_metadata: [256], i32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
qwen3-235b-decode	f16	2.11×	1.1702	torch-grouped-mm flashinfer-segment-mlp	2.4680 2.8404	132	93%	mem
	bf16	1.43×	1.1708	torch-grouped-mm flashinfer-segment-mlp	1.6755 2.8402	132	93%	mem
qwen3-235b-prefill	f16	1.53×	2.0191	flashinfer-segment-mlp torch-grouped-mm	3.0815 3.2543	613	84%	comp
	bf16	1.10×	2.0104	torch-grouped-mm flashinfer-segment-mlp	2.2076 3.0765	615	81%	comp
ds-v3-decode	f16	1.50×	5.2062	torch-grouped-mm flashinfer-segment-mlp	7.8231 11.8226	69.3	97%	mem
	bf16	1.35×	5.2066	torch-grouped-mm flashinfer-segment-mlp	7.0159 11.8359	69.3	97%	mem
ds-v3-prefill	f16	1.53×	6.0469	torch-grouped-mm flashinfer-segment-mlp	9.2235 12.9855	477	86%	mem
	bf16	1.19×	6.0220	torch-grouped-mm flashinfer-segment-mlp	7.1685 13.0060	479	87%	mem

FusedMoEExpertsFwd

output, hidden_states: [T, H]w_gate_up: [E, 2 * F, H]w_down: [E, H, F]topk_weights: [T, K], f32topk_ids: [T, K], i32

dtype=f16, bf16

• qwen3-235b-decodeoutput, hidden_states: [512, 4096]w_gate_up: [128, 3072, 4096]w_down: [128, 4096, 1536]topk_weights: [512, 8], f32topk_ids: [512, 8], i32
• qwen3-235b-prefilloutput, hidden_states: [4096, 4096]w_gate_up: [128, 3072, 4096]w_down: [128, 4096, 1536]topk_weights: [4096, 8], f32topk_ids: [4096, 8], i32
• ds-v3-decodeoutput, hidden_states: [512, 7168]w_gate_up: [256, 4096, 7168]w_down: [256, 7168, 2048]topk_weights: [512, 8], f32topk_ids: [512, 8], i32
• ds-v3-prefilloutput, hidden_states: [4096, 7168]w_gate_up: [256, 4096, 7168]w_down: [256, 7168, 2048]topk_weights: [4096, 8], f32topk_ids: [4096, 8], i32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
qwen3-235b-decode	f16	1.01×	1.1975	vllm-triton flashinfer-cutlass	1.2120 13.0702	129	90%	mem
	bf16	1.01×	1.1971	vllm-triton flashinfer-cutlass	1.2131 13.1021	129	90%	mem
qwen3-235b-prefill	f16	1.24×	2.5387	vllm-triton flashinfer-cutlass	3.1515 20.5776	487	67%	comp
	bf16	1.24×	2.5035	vllm-triton flashinfer-cutlass	3.1130 20.5923	494	65%	comp
ds-v3-decode	f16	1.01×	5.2572	vllm-triton flashinfer-cutlass	5.3102 50.9557	68.6	95%	mem
	bf16	1.01×	5.2509	vllm-triton flashinfer-cutlass	5.3117 51.0572	68.7	95%	mem
ds-v3-prefill	f16	1.16×	7.8773	vllm-triton flashinfer-cutlass	9.1509 62.2428	367	64%	mem
	bf16	1.16×	7.6333	vllm-triton flashinfer-cutlass	8.8278 62.3009	378	66%	mem

IndexedExpertMLPFwd

output, hidden_states: [T, H]w_gate_up: [E, 2 * F, H]w_down: [E, H, F]topk_weights: [T, K], f32topk_ids: [T, K], i32

dtype=f16, bf16

• ds-v3-t1output, hidden_states: [1, 7168]w_gate_up: [256, 4096, 7168]w_down: [256, 7168, 2048]topk_weights: [1, 8], f32topk_ids: [1, 8], i32
• ds-v3-t32output, hidden_states: [32, 7168]w_gate_up: [256, 4096, 7168]w_down: [256, 7168, 2048]topk_weights: [32, 8], f32topk_ids: [32, 8], i32
• ds-v3-t64output, hidden_states: [64, 7168]w_gate_up: [256, 4096, 7168]w_down: [256, 7168, 2048]topk_weights: [64, 8], f32topk_ids: [64, 8], i32
• qwen3-235b-t32output, hidden_states: [32, 4096]w_gate_up: [128, 3072, 4096]w_down: [128, 4096, 1536]topk_weights: [32, 8], f32topk_ids: [32, 8], i32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
ds-v3-t1	f16	1.13×	0.1784	vllm-triton staged flashinfer-cutlass	0.2010 0.2231 21.3177	3.95	88%	mem
	bf16	1.13×	0.1785	vllm-triton staged flashinfer-cutlass	0.2009 0.2234 21.3185	3.95	88%	mem
ds-v3-t32	f16	1.00×	3.0657	vllm-triton staged flashinfer-cutlass	3.0711 3.1773 39.8752	7.36	101%	mem
	bf16	1.00×	3.0668	vllm-triton staged flashinfer-cutlass	3.0707 3.1803 40.0186	7.35	101%	mem
ds-v3-t64	f16	1.01×	4.2356	vllm-triton staged flashinfer-cutlass	4.2648 4.3833 46.7907	10.7	102%	mem
	bf16	1.01×	4.2376	vllm-triton staged flashinfer-cutlass	4.2639 4.3793 46.7030	10.6	102%	mem
qwen3-235b-t32	f16	1.00×	0.9740	vllm-triton staged flashinfer-cutlass	0.9780 1.0137 12.5703	9.93	98%	mem
	bf16	1.00×	0.9745	vllm-triton staged flashinfer-cutlass	0.9772 1.0145 12.5612	9.92	98%	mem

SharedExpertMLPFwd

hidden_states: [T, H]w_gate_up: [2 * S, H]w_down: [H, S]

• kimi-k2-t1hidden_states: [1, 7168]w_gate_up: [36864, 7168]w_down: [7168, 18432]dtype=bf16
• kimi-k2-t512hidden_states: [512, 7168]w_gate_up: [36864, 7168]w_down: [7168, 18432]dtype=bf16
• ds-v3-t32hidden_states: [32, 7168]w_gate_up: [4096, 7168]w_down: [7168, 2048]dtype=bf16, f16
• ds-v3-t4096hidden_states, w_gate_up: [4096, 7168]w_down: [7168, 2048]dtype=bf16
• glm45-t32hidden_states: [32, 5120]w_gate_up: [3072, 5120]w_down: [5120, 1536]dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
kimi-k2-t1	bf16	1.01×	0.2062	torch-cublas quack	0.2092 0.3028	3.84	85%	mem
kimi-k2-t512	bf16	0.93×	0.6036	quack torch-cublas	0.5608 0.6338	672	88%	comp
ds-v3-t32	bf16	1.28×	0.0333	torch-cublas quack	0.0426 0.0627	84.7	59%	mem
	f16	1.30×	0.0333	torch-cublas quack	0.0433 0.0629	84.7	59%	mem
ds-v3-t4096	bf16	1.01×	0.5127	quack torch-cublas	0.5156 0.5614	704	93%	comp
glm45-t32	bf16	1.39×	0.0221	torch-cublas quack	0.0308 0.0460	68.2	48%	mem