Sampling

本页暂无中文版。以下为英文原文。

6 ops, 48 workloads.

One table per op, one row per workload. Ratio is the fastest other implementation's device time divided by ours, so green is faster than it, plain is level with it, red is slower. Times are in ms. How these numbers are taken.

TopKMaskFwd

logits: [B, V]k: [B], i32

• llama-8b-b1logits: [1, 128256]k: [1], i32k_list=repeat(50, 1)dtype=bf16, f32
• llama-8b-b64logits: [64, 128256]k: [64], i32k_list=repeat(50, 64)dtype=bf16
• llama-8b-b256logits: [256, 128256]k: [256], i32k_list=repeat(50, 256)dtype=f16, bf16
• qwen3-235b-b128logits: [128, 151936]k: [128], i32k_list=repeat(20, 128)dtype=bf16
• gpt-oss-120b-b64logits: [64, 201088]k: [64], i32k_list=repeat(50, 64)dtype=bf16
• gpt2-b17-mixedlogits: [17, 50257]k: [17], i32k_list=[1,2,5,10…]dtype=f16
• deepseek-v3-b17-mixedlogits: [17, 129280]k: [17], i32k_list=[1,2,5,10…]dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
llama-8b-b1	bf16	2.55×	0.0147	flashinfer torch-compile vllm torch-ref	0.0374 0.0549 0.0622 0.0726	0.0175	1%	lat
	f32	2.39×	0.0190	flashinfer torch-compile torch-ref vllm	0.0453 0.0582 0.0735 0.0823	0.0135	1%	lat
llama-8b-b64	bf16	1.12×	0.0360	flashinfer vllm torch-compile torch-ref	0.0404 0.1792 0.6729 0.7456	0.456	20%	mem
llama-8b-b256	f16	1.21×	0.1164	flashinfer vllm torch-compile torch-ref	0.1413 0.5216 2.7494 3.0146	0.564	25%	mem
	bf16	1.18×	0.1242	flashinfer vllm torch-compile torch-ref	0.1463 0.5297 2.6971 2.9635	0.529	23%	mem
qwen3-235b-b128	bf16	1.06×	0.0827	flashinfer vllm torch-compile torch-ref	0.0881 0.3797 2.3282 2.4853	0.47	21%	mem
gpt-oss-120b-b64	bf16	1.10×	0.0538	flashinfer vllm torch-compile torch-ref	0.0593 0.2498 1.0039 1.1057	0.479	21%	mem
gpt2-b17-mixed	f16	2.20×	0.0232	flashinfer vllm torch-compile torch-ref	0.0511 0.1252 0.1399 0.1493	0.0649	3%	mem
deepseek-v3-b17-mixed	bf16	1.68×	0.0233	flashinfer vllm torch-compile torch-ref	0.0392 0.1432 0.2351 0.2629	0.166	8%	mem

MinPMaskFwd

logits: [B, V]min_p: [B], f32

• llama-8b-b1logits: [1, 128256]min_p: [1], f32dtype=bf16, f32
• llama-8b-b64logits: [64, 128256]min_p: [64], f32dtype=bf16
• llama-8b-b256logits: [256, 128256]min_p: [256], f32dtype=f16, bf16
• qwen3-235b-b128logits: [128, 151936]min_p: [128], f32dtype=bf16
• deepseek-v3-b17logits: [17, 129280]min_p: [17], f32dtype=bf16
• llama2-7b-b100logits: [100, 32000]min_p: [100], f32dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
llama-8b-b1	bf16	1.26×	0.0039	torch-compile torch-ref vllm	0.00493 0.0285 0.0674	0.0657	3%	lat
	f32	1.27×	0.004	torch-compile torch-ref vllm	0.00509 0.0379 0.0696	0.0641	6%	lat
llama-8b-b64	bf16	1.47×	0.0140	torch-compile torch-ref vllm	0.0206 0.1017 0.1032	1.17	52%	mem
llama-8b-b256	f16	1.75×	0.0377	torch-compile vllm torch-ref	0.0660 0.2774 0.3031	1.74	77%	mem
	bf16	1.74×	0.0386	torch-compile vllm torch-ref	0.0670 0.2811 0.3048	1.7	76%	mem
qwen3-235b-b128	bf16	2.00×	0.0236	torch-compile vllm torch-ref	0.0473 0.1775 0.1888	1.65	73%	mem
deepseek-v3-b17	bf16	1.54×	0.00768	torch-compile torch-ref vllm	0.0118 0.0443 0.0782	0.572	25%	lat
llama2-7b-b100	f16	2.18×	0.00646	torch-compile torch-ref vllm	0.0141 0.0392 0.0434	0.99	44%	lat

TopPMaskFwd

logits: [B, V]p: [B], f32

• llama-8b-b1logits: [1, 128256]p: [1], f32dtype=bf16, f32
• llama-8b-b64logits: [64, 128256]p: [64], f32dtype=bf16
• llama-8b-b256logits: [256, 128256]p: [256], f32dtype=f16, bf16
• qwen3-235b-b128logits: [128, 151936]p: [128], f32dtype=bf16
• gpt-oss-120b-b64logits: [64, 201088]p: [64], f32dtype=bf16
• deepseek-v3-b17logits: [17, 129280]p: [17], f32dtype=bf16
• llama2-7b-b100logits: [100, 32000]p: [100], f32dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
llama-8b-b1	bf16	4.37×	0.0185	torch-compile vllm torch-ref	0.0809 0.1130 0.1354	0.0485	1%	lat
	f32	2.88×	0.0287	torch-compile vllm torch-ref	0.0825 0.1164 0.1403	0.0313	1%	mem
llama-8b-b64	bf16	11.96×	0.0793	vllm torch-compile torch-ref	0.9476 1.0613 1.5673	0.725	9%	mem
llama-8b-b256	f16	8.24×	0.2897	vllm torch-compile torch-ref	2.3884 4.1716 5.4111	0.793	10%	mem
	bf16	8.80×	0.2563	vllm torch-compile torch-ref	2.2546 4.1310 5.3330	0.897	11%	mem
qwen3-235b-b128	bf16	9.15×	0.1559	vllm torch-compile torch-ref	1.4262 3.2115 4.0344	0.873	11%	mem
gpt-oss-120b-b64	bf16	13.80×	0.1179	torch-compile vllm torch-ref	1.6275 1.7347 2.4030	0.764	10%	mem
deepseek-v3-b17	bf16	9.31×	0.0388	torch-compile torch-ref vllm	0.3615 0.6890 0.8580	0.396	5%	mem
llama2-7b-b100	f16	5.16×	0.0458	vllm torch-compile torch-ref	0.2367 0.4836 0.6925	0.489	6%	mem

TopKTopPMaskFwd

logits: [B, V]k: [B], i32p: [B], f32

• llama-8b-b1logits: [1, 128256]k: [1], i32p: [1], f32k_list=repeat(50, 1)dtype=bf16, f32
• llama-8b-b64logits: [64, 128256]k: [64], i32p: [64], f32k_list=repeat(50, 64)dtype=bf16
• llama-8b-b256logits: [256, 128256]k: [256], i32p: [256], f32k_list=repeat(50, 256)dtype=f16, bf16
• qwen3-235b-b128logits: [128, 151936]k: [128], i32p: [128], f32k_list=repeat(20, 128)dtype=bf16
• gpt2-b17-mixedlogits: [17, 50257]k: [17], i32p: [17], f32k_list=[1,2,5,10…]dtype=f16
• deepseek-v3-b100logits: [100, 129280]k: [100], i32p: [100], f32k_list=repeat(40, 100)dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
llama-8b-b1	bf16	5.58×	0.0226	torch-compile vllm torch-ref	0.1258 0.1509 0.2227	0.0114	1%	mem
	f32	3.57×	0.0348	torch-compile vllm torch-ref	0.1245 0.1535 0.2192	0.00737	1%	mem
llama-8b-b64	bf16	2.80×	0.1089	vllm torch-compile torch-ref	0.3049 1.5510 2.0837	0.151	7%	mem
llama-8b-b256	f16	2.50×	0.2646	vllm torch-compile torch-ref	0.6617 6.4284 7.6985	0.248	11%	mem
	bf16	2.03×	0.3243	vllm torch-compile torch-ref	0.6594 6.3785 7.6482	0.203	9%	mem
qwen3-235b-b128	bf16	1.55×	0.2628	vllm torch-compile torch-ref	0.4073 5.2894 6.1605	0.148	7%	mem
gpt2-b17-mixed	f16	5.07×	0.0623	torch-compile torch-ref vllm	0.3160 0.4394 0.7133	0.0376	1%	mem
deepseek-v3-b100	bf16	2.12×	0.1454	vllm torch-compile torch-ref	0.3085 2.3777 3.0010	0.178	8%	mem

SamplingFromProbsFwd

probs: [B, V], f32seed, offset: [1]

dtype=i64

• llama-8b-b1probs: [1, 128256], f32seed, offset: [1]
• llama-8b-b64probs: [64, 128256], f32seed, offset: [1]
• llama-8b-b256probs: [256, 128256], f32seed, offset: [1]
• qwen3-235b-b128probs: [128, 151936], f32seed, offset: [1]
• gpt-oss-120b-b64probs: [64, 201088], f32seed, offset: [1]
• ds-v3-b17probs: [17, 129280], f32seed, offset: [1]
• llama2-7b-b100probs: [100, 32000], f32seed, offset: [1]

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
llama-8b-b1	i64	4.14×	0.00614	flashinfer vllm torch-multinomial torch-compile torch-ref	0.0254 0.0413 0.1017 0.5291 0.9100	0.0209	2%	lat
llama-8b-b64	i64	2.78×	0.0177	flashinfer vllm torch-multinomial torch-compile torch-ref	0.0493 0.0955 0.1562 0.6880 1.4345	0.463	41%	mem
llama-8b-b256	i64	1.65×	0.0522	flashinfer vllm torch-multinomial torch-compile torch-ref	0.0861 0.2359 0.3321 1.1840 2.3254	0.629	56%	mem
qwen3-235b-b128	i64	1.97×	0.0311	flashinfer vllm torch-multinomial torch-compile torch-ref	0.0613 0.1533 0.2240 0.9797 1.9070	0.626	56%	mem
gpt-oss-120b-b64	i64	3.12×	0.0241	flashinfer vllm torch-multinomial torch-compile torch-ref	0.0752 0.1078 0.1634 1.0780 1.9867	0.533	47%	mem
ds-v3-b17	i64	4.60×	0.0100	flashinfer vllm torch-multinomial torch-compile torch-ref	0.0462 0.0558 0.0944 0.5839 1.2422	0.219	19%	lat
llama2-7b-b100	i64	1.86×	0.00877	flashinfer vllm torch-multinomial torch-compile torch-ref	0.0164 0.0351 0.0650 0.2931 0.8618	0.365	32%	lat

ChainSpeculativeSamplingFwd

draft_probs: [B, N, V]draft_token_ids: [B, N], i32target_probs: [B, N + 1, V]seed, offset: [1], i64

dtype=f32

• llama-8b-b16-n4draft_probs: [16, 4, 128256]draft_token_ids: [16, 4], i32target_probs: [16, 5, 128256]seed, offset: [1], i64
• llama-8b-b64-n4draft_probs: [64, 4, 128256]draft_token_ids: [64, 4], i32target_probs: [64, 5, 128256]seed, offset: [1], i64
• ds-v3-mtp-n1draft_probs: [64, 1, 129280]draft_token_ids: [64, 1], i32target_probs: [64, 2, 129280]seed, offset: [1], i64
• ds-v3-mtp-n3draft_probs: [64, 3, 129280]draft_token_ids: [64, 3], i32target_probs: [64, 4, 129280]seed, offset: [1], i64
• qwen3-235b-eagle-n5draft_probs: [32, 5, 151936]draft_token_ids: [32, 5], i32target_probs: [32, 6, 151936]seed, offset: [1], i64
• llama-8b-b17-n8draft_probs: [17, 8, 128256]draft_token_ids: [17, 8], i32target_probs: [17, 9, 128256]seed, offset: [1], i64
• llama2-7b-b256-n1draft_probs: [256, 1, 32000]draft_token_ids: [256, 1], i32target_probs: [256, 2, 32000]seed, offset: [1], i64

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
llama-8b-b16-n4	f32	6.92×	0.0140	flashinfer vllm torch-compile torch-ref	0.0972 0.2632 0.5733 1.3182	0.438	13%	lat
llama-8b-b64-n4	f32	4.19×	0.0271	flashinfer vllm torch-compile torch-ref	0.1136 0.5796 0.7175 1.7210	0.909	27%	mem
ds-v3-mtp-n1	f32	4.19×	0.0260	flashinfer vllm torch-compile torch-ref	0.1090 0.3444 0.7826 1.6444	0.954	28%	mem
ds-v3-mtp-n3	f32	4.24×	0.0264	flashinfer vllm torch-compile torch-ref	0.1120 0.5062 0.7236 1.7009	0.94	28%	mem
qwen3-235b-eagle-n5	f32	5.79×	0.0210	flashinfer vllm torch-compile torch-ref	0.1215 0.4426 0.7971 1.6665	0.696	21%	mem
llama-8b-b17-n8	f32	6.45×	0.0153	flashinfer vllm torch-compile torch-ref	0.0989 0.3527 0.6077 1.3891	0.427	13%	lat
llama2-7b-b256-n1	f32	2.98×	0.0212	flashinfer vllm torch-compile torch-ref	0.0634 0.2536 0.4372 1.2682	1.16	34%	mem