Quantization & Dequantization

本页暂无中文版。以下为英文原文。

10 ops, 71 workloads.

One table per op, one row per workload. Ratio is the fastest other implementation's device time divided by ours, so green is faster than it, plain is level with it, red is slower. Times are in ms. How these numbers are taken.

INT8QuantPerTensorFwd

x: [M, K]

• llama-8b-prefillx: [4096, 4096]dtype=f16, bf16
• llama-8b-decodex: [32, 4096]dtype=bf16
• qwen3-32b-prefillx: [8192, 5120]dtype=bf16, f32
• gpt-oss-20b-decodex: [17, 2880]dtype=bf16
• gpt-oss-20b-prefillx: [3000, 2880]dtype=bf16
• odd-hidden-prefillx: [1000, 4099]dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
llama-8b-prefill	f16	1.92×	0.0183	torch-compile vllm torch-ref	0.0352 0.0645 0.3029	4.58	61%	mem
	bf16	1.96×	0.0184	torch-compile vllm torch-ref	0.0359 0.0653 0.3043	4.57	61%	mem
llama-8b-decode	bf16	1.54×	0.0033	torch-compile vllm torch-ref	0.00509 0.0172 0.0257	0.199	3%	lat
qwen3-32b-prefill	bf16	1.91×	0.0438	torch-compile vllm torch-ref	0.0835 0.1364 0.7153	4.79	64%	mem
	f32	1.22×	0.0818	torch-compile vllm torch-ref	0.0998 0.1886 0.5738	2.56	57%	mem
gpt-oss-20b-decode	bf16	1.44×	0.0033	torch-compile vllm torch-ref	0.00474 0.0195 0.0269	0.0743	1%	lat
gpt-oss-20b-prefill	bf16	1.90×	0.0119	torch-compile vllm torch-ref	0.0226 0.0436 0.1617	3.64	48%	mem
odd-hidden-prefill	f16	1.95×	0.008	torch-compile vllm torch-ref	0.0156 0.0307 0.0765	2.56	34%	lat

INT8QuantPerChannelFwd

w: [N, K]

• llama-8b-mlp-upw: [14336, 4096]dtype=f16, bf16
• llama-70b-mlp-downw: [8192, 28672]dtype=bf16
• qwen3-32b-o-projw: [5120, 8192]dtype=bf16, f32
• gpt-oss-20b-mlp-upw: [5760, 2880]dtype=bf16
• ragged-kw: [3000, 4099]dtype=f16
• ragged-k-few-rowsw: [17, 4099]dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
llama-8b-mlp-up	f16	1.08×	0.0455	vllm vllm-cuda torch-compile torch-ref	0.0490 0.0498 0.1108 1.0026	6.46	86%	mem
	bf16	1.09×	0.0458	vllm vllm-cuda torch-compile torch-ref	0.0497 0.0515 0.1154 1.0050	6.42	86%	mem
llama-70b-mlp-down	bf16	1.56×	0.1769	vllm-cuda vllm torch-compile torch-ref	0.2756 0.3530 0.3535 3.9229	6.64	88%	mem
qwen3-32b-o-proj	bf16	0.97×	0.0350	vllm vllm-cuda torch-compile torch-ref	0.0339 0.0363 0.0702 0.7405	5.99	80%	mem
	f32	1.00×	0.0540	vllm vllm-cuda torch-compile torch-ref	0.0539 0.0689 0.0713 0.5985	3.88	86%	mem
gpt-oss-20b-mlp-up	bf16	1.26×	0.0158	vllm-cuda vllm torch-compile torch-ref	0.0200 0.0204 0.0463 0.3034	5.24	70%	mem
ragged-k	f16	1.77×	0.0139	vllm vllm-cuda torch-compile torch-ref	0.0246 0.0265 0.0319 0.2378	4.43	59%	mem
ragged-k-few-rows	bf16	1.17×	0.00224	vllm torch-compile vllm-cuda torch-ref	0.00262 0.00291 0.00342 0.0304	0.156	2%	lat

INT8QuantPerBlockFwd

x: [M, K]

• ds-v3-prefillx: [4096, 7168]dtype=f16, bf16
• ds-v3-decodex: [64, 7168]dtype=bf16
• qwen3-235b-prefillx: [4096, 4096]dtype=bf16, f32
• ds-v3-decode-few-tokensx: [17, 7168]dtype=bf16
• gpt-oss-20b-prefillx: [3000, 2880]dtype=bf16
• ragged-kx: [1000, 4099]dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
ds-v3-prefill	f16	1.49×	0.0246	torch-compile vllm torch-ref	0.0367 0.0564 0.6195	5.99	80%	mem
	bf16	1.56×	0.0249	torch-compile vllm torch-ref	0.0389 0.0571 0.6208	5.91	79%	mem
ds-v3-decode	bf16	1.43×	0.00214	vllm torch-compile torch-ref	0.00307 0.00458 0.0284	1.07	14%	lat
qwen3-235b-prefill	bf16	1.53×	0.0154	torch-compile vllm torch-ref	0.0236 0.0340 0.3671	5.47	73%	mem
	f32	1.07×	0.0234	torch-compile vllm torch-ref	0.0251 0.0428 0.3077	3.6	80%	mem
ds-v3-decode-few-tokens	bf16	1.15×	0.00192	torch-compile vllm torch-ref	0.00221 0.00259 0.0248	0.318	4%	lat
gpt-oss-20b-prefill	bf16	4.97×	0.00944	torch-compile torch-ref	0.0469 0.2468	4.59	62%	mem
ragged-k	f16	2.68×	0.0096	torch-compile torch-ref	0.0257 0.1163	2.14	29%	lat

SmoothQuantFwd

x: [M, K]smooth: [K], f32

• llama-8b-prefillx: [4096, 4096]smooth: [4096], f32dtype=f16, bf16
• llama-8b-decodex: [32, 4096]smooth: [4096], f32dtype=bf16
• llama-70b-prefillx: [4096, 8192]smooth: [8192], f32dtype=bf16
• gpt-oss-20b-prefillx: [3000, 2880]smooth: [2880], f32dtype=bf16
• ragged-kx: [1000, 4099]smooth: [4099], f32dtype=f16
• ragged-k-decodex: [17, 4099]smooth: [4099], f32dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
llama-8b-prefill	f16	2.26×	0.0193	torch-compile vllm torch-ref	0.0435 0.1204 0.3699	5.23	58%	mem
	bf16	2.21×	0.0200	torch-compile vllm torch-ref	0.0441 0.1225 0.3704	5.04	56%	mem
llama-8b-decode	bf16	1.23×	0.00253	torch-compile vllm torch-ref	0.0031 0.00922 0.0318	0.311	4%	lat
llama-70b-prefill	bf16	1.92×	0.0383	torch-compile vllm torch-ref	0.0734 0.2382 0.7208	5.26	58%	mem
gpt-oss-20b-prefill	bf16	2.31×	0.0143	torch-compile vllm torch-ref	0.0330 0.0668 0.2009	3.63	40%	mem
ragged-k	f16	2.07×	0.0121	torch-compile vllm torch-ref	0.0251 0.0447 0.0970	2.03	23%	lat
ragged-k-decode	bf16	1.25×	0.00362	torch-compile vllm torch-ref	0.00451 0.0109 0.0325	0.116	1%	lat

FP8QuantFwd

input_tensor: [batch, seq_len_kv, kv_group, index_dim]

• kv-index-8kinput_tensor: [1, 8192, 1, 64]dtype=f16, bf16
• kv-index-4kinput_tensor: [1, 4096, 1, 128]dtype=f32
• kv-index-context-chunkinput_tensor: [1, 3072, 1, 128]dtype=f16, bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
kv-index-8k	f16	1.24×	0.00224	torch-compile vllm torch-ref	0.00278 0.00474 0.0211	0.944	16%	lat
	bf16	1.23×	0.00224	torch-compile vllm torch-ref	0.00275 0.00477 0.0211	0.944	16%	lat
kv-index-4k	f32	1.10×	0.00278	torch-compile vllm torch-ref	0.00307 0.00442 0.0156	0.756	21%	lat
kv-index-context-chunk	f16	1.66×	0.00214	vllm torch-compile torch-ref	0.00355 0.00448 0.0188	0.736	12%	lat
	bf16	1.66×	0.00214	vllm torch-compile torch-ref	0.00355 0.00448 0.0188	0.736	12%	lat

FP8QuantPerBlockFwd

w: [N, K]

• ds-v3-mlp-downw: [7168, 18432]dtype=bf16
• ds-v3-expert-gate-upw: [4096, 7168]dtype=bf16, f32
• qwen3-235b-qkvw: [9216, 4096]dtype=bf16, f16
• gpt-oss-expert-downw: [2880, 2880]dtype=bf16
• ragged-kw: [1000, 4099]dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
ds-v3-mlp-down	bf16	2.44×	0.0953	torch-compile torch-ref vllm	0.2323 2.1144 3.6444	5.55	92%	mem
ds-v3-expert-gate-up	bf16	2.24×	0.0245	torch-compile torch-ref vllm	0.0551 0.4938 0.8240	4.79	80%	mem
	f32	1.86×	0.0388	torch-compile torch-ref vllm	0.0724 0.3931 0.7638	3.02	84%	mem
qwen3-235b-qkv	bf16	2.28×	0.0303	torch-compile torch-ref vllm	0.0692 0.6264 1.0554	4.99	83%	mem
	f16	2.19×	0.0302	torch-compile torch-ref vllm	0.0661 0.6239 1.0543	5	83%	mem
gpt-oss-expert-down	bf16	3.09×	0.00954	torch-compile torch-ref vllm	0.0294 0.2074 0.2796	3.48	58%	mem
ragged-k	f16	2.24×	0.00902	torch-compile torch-ref vllm	0.0202 0.1039 0.1517	1.82	30%	lat

INT4QuantPerGroupFwd

w: [N, K]

dtype=f16

• llama-8b-mlp-upw: [14336, 4096]
• llama-8b-q-projw: [4096, 4096]group_size=64
• llama-70b-mlp-downw: [8192, 28672]
• qwen3-32b-o-projw: [5120, 8192]
• qwen2-7b-mlp-upw: [18944, 3584]
• qwen2-7b-mlp-downw: [3584, 18944]
• opt-13b-mlp-upw: [20480, 5120]group_size=64
• llama-8b-mlp-up-chanw: [14336, 4096]group_size=4096

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
llama-8b-mlp-up	f16	1.44×	0.0400	deepspeed	0.0575	10.3	84%	mem
llama-8b-q-proj	f16	1.25×	0.0144	deepspeed	0.0181	8.25	68%	mem
llama-70b-mlp-down	f16	1.44×	0.1532	deepspeed	0.2208	10.8	87%	mem
qwen3-32b-o-proj	f16	1.40×	0.0299	deepspeed	0.0418	9.89	80%	mem
qwen2-7b-mlp-up	f16	1.45×	0.0457	deepspeed	0.0661	10.5	85%	mem
qwen2-7b-mlp-down	f16	1.44×	0.0457	deepspeed	0.0659	10.5	85%	mem
opt-13b-mlp-up	f16	1.39×	0.0713	deepspeed	0.0992	10.4	86%	mem
llama-8b-mlp-up-chan	f16	1.25×	0.0403	deepspeed	0.0505	10.2	81%	mem

INT8DequantPerTensorFwd

q: [M, K], i8scale: [1], f32

• llama-8b-prefillq: [4096, 4096], i8scale: [1], f32dtype=bf16
• llama-8b-decodeq: [32, 4096], i8scale: [1], f32dtype=bf16
• qwen3-32b-prefillq: [8192, 5120], i8scale: [1], f32dtype=f16, f32
• gpt-oss-prefillq: [3000, 2880], i8scale: [1], f32dtype=bf16
• ragged-k-few-rowsq: [17, 4099], i8scale: [1], f32dtype=f16

q: [M, K], i8scale: [1]

• ragged-kq: [1000, 4099], i8scale: [1]dtype=f32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
llama-8b-prefill	bf16	1.06×	0.0135	torch-compile torch-ref	0.0143 0.1360	1.24	83%	mem
llama-8b-decode	bf16	1.10×	0.00163	torch-compile torch-ref	0.00179 0.0073	0.0803	5%	lat
qwen3-32b-prefill	f16	1.03×	0.0316	torch-compile torch-ref	0.0325 0.3287	1.33	88%	mem
	f32	1.02×	0.0529	torch-compile torch-ref	0.0538 0.2697	0.792	88%	mem
gpt-oss-prefill	bf16	1.15×	0.00739	torch-compile torch-ref	0.00851 0.0720	1.17	78%	mem
ragged-k-few-rows	f16	1.02×	0.00163	torch-compile torch-ref	0.00166 0.0071	0.0427	3%	lat
ragged-k	f32	1.05×	0.00611	torch-compile torch-ref	0.0064 0.0272	0.671	74%	mem

INT8DequantPerChannelFwd

q: [M, K], i8scale: [M], f32

• llama-8b-mlp-upq: [14336, 4096], i8scale: [14336], f32dtype=bf16, f16
• llama-70b-mlp-downq: [8192, 28672], i8scale: [8192], f32dtype=bf16
• gpt-oss-20b-mlp-upq: [5760, 2880], i8scale: [5760], f32dtype=bf16
• ragged-kq: [3000, 4099], i8scale: [3000], f32dtype=f16
• ragged-k-few-rowsq: [17, 4099], i8scale: [17], f32dtype=bf16

q: [M, K], i8scale: [M]

• llama-8b-mlp-downq: [4096, 14336], i8scale: [4096]dtype=f32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
llama-8b-mlp-up	bf16	1.07×	0.0440	torch-compile torch-ref	0.0470 0.4721	1.33	89%	mem
	f16	1.07×	0.0440	torch-compile torch-ref	0.0471 0.4721	1.33	89%	mem
llama-70b-mlp-down	bf16	1.06×	0.1726	torch-compile torch-ref	0.1828 1.8544	1.36	91%	mem
gpt-oss-20b-mlp-up	bf16	1.06×	0.0140	torch-compile torch-ref	0.0148 0.1392	1.19	79%	mem
ragged-k	f16	1.14×	0.0112	torch-compile torch-ref	0.0128 0.1047	1.1	73%	mem
ragged-k-few-rows	bf16	1.11×	0.00147	torch-compile torch-ref	0.00163 0.00717	0.0473	3%	lat
llama-8b-mlp-down	f32	1.01×	0.0742	torch-compile torch-ref	0.0747 0.3884	0.792	88%	mem

INT8DequantPerBlockFwd

q: [M, K], i8scale: [M, ceil_div(K, 128)], f32

• ds-v3-prefillq: [4096, 7168], i8scale: [4096, 56], f32dtype=bf16
• ds-v3-decodeq: [64, 7168], i8scale: [64, 56], f32dtype=bf16
• qwen3-235b-prefillq: [4096, 4096], i8scale: [4096, 32], f32dtype=f16
• gpt-oss-prefillq: [3000, 2880], i8scale: [3000, 23], f32dtype=bf16
• ragged-k-few-rowsq: [17, 4099], i8scale: [17, 33], f32dtype=f16

q: [M, K], i8scale: [M, ceil_div(K, 128)]

• ragged-kq: [1000, 4099], i8scale: [1000, 33]dtype=f32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
ds-v3-prefill	bf16	1.06×	0.0229	torch-compile torch-ref	0.0243 0.2941	1.28	86%	mem
ds-v3-decode	bf16	1.20×	0.00186	torch-compile torch-ref	0.00222 0.0107	0.247	17%	lat
qwen3-235b-prefill	f16	1.08×	0.0136	torch-compile torch-ref	0.0148 0.1729	1.23	83%	mem
gpt-oss-prefill	bf16	1.81×	0.00794	torch-compile torch-ref	0.0143 0.1025	1.09	73%	mem
ragged-k-few-rows	f16	1.04×	0.0016	torch-compile torch-ref	0.00166 0.00896	0.0435	3%	lat
ragged-k	f32	1.29×	0.00654	torch-compile torch-ref	0.00842 0.0427	0.626	70%	mem