Conv & Pool

本页暂无中文版。以下为英文原文。

16 ops, 120 workloads — Pooling 13 · Convolution 3.

One table per op, one row per workload. Ratio is the fastest other implementation's device time divided by ours, so green is faster than it, plain is level with it, red is slower. Times are in ms. How these numbers are taken.

Pooling

AvgPool1dFwd

input: [N, C, L_in]

• audio-downsampleinput: [4, 128, 4096]kernel_size=3stride=2padding=1dtype=f16, f32
• long-temporalinput: [2, 256, 32000]kernel_size=5stride=4padding=2dtype=f16
• ceilinput: [2, 128, 2048]kernel_size=4stride=2padding=1ceil_mode=truecount_include_pad=falsedtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
audio-downsample	f16	1.11×	0.00374	torch-compile torch-ref	0.00416 0.0154	1.12	37%	lat
	f32	1.01×	0.00528	torch-compile torch-ref	0.00534 0.0154	0.794	53%	lat
long-temporal	f16	1.21×	0.0142	torch-compile torch-ref	0.0172 0.0595	1.73	64%	mem
ceil	bf16	1.44×	0.00234	torch-compile torch-ref	0.00336 0.00534	0.561	15%	lat

AvgPool2dFwd

input: [N, C, H_in, W_in]

stride=[2,2]

• vision-3x3-s2input: [2, 64, 112, 112]kernel_size=[3,3]padding=[1,1]dtype=f16, f32
• vision-5x5-s2input: [2, 128, 56, 56]kernel_size=[5,5]padding=[2,2]dtype=f16
• ceil-divisorinput: [3, 96, 55, 57]kernel_size=[3,5]padding=[1,2]ceil_mode=truecount_include_pad=falsedivisor_override=7dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
vision-3x3-s2	f16	1.19×	0.00339	torch-compile flaggems torch-ref	0.00403 0.00669 0.00918	1.17	26%	lat
	f32	1.19×	0.00442	torch-compile flaggems torch-ref	0.00525 0.0071 0.00944	0.899	40%	lat
vision-5x5-s2	f16	1.78×	0.00406	flaggems torch-ref torch-compile	0.00723 0.00986 0.0218	1.23	11%	lat
ceil-divisor	bf16	1.24×	0.00314	torch-compile flaggems torch-ref	0.0039 0.00586 0.00765	1.14	16%	lat

AvgPool3dFwd

input: [N, C, D_in, H_in, W_in]

stride=[2,2,2]

• video-2x2x2input: [1, 32, 16, 56, 56]kernel_size=[2,2,2]padding=[0,0,0]dtype=f16, f32
• ceil-videoinput: [2, 64, 8, 28, 28]kernel_size=[2,3,3]padding=[1,1,1]ceil_mode=truecount_include_pad=falsedtype=f16
• divisorinput: [2, 24, 10, 20, 22]kernel_size=[2,2,3]padding=[0,1,1]divisor_override=7dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
video-2x2x2	f16	1.15×	0.00298	torch-compile cudnn torch-ref	0.00342 0.00586 0.00998	0.607	27%	lat
	f32	1.06×	0.00387	torch-compile cudnn torch-ref	0.0041 0.00653 0.0103	0.467	41%	lat
ceil-video	f16	1.21×	0.00333	torch-compile cudnn torch-ref	0.00403 0.00566 0.0114	0.586	13%	lat
divisor	bf16	1.02×	0.00192	torch-compile torch-ref	0.00195 0.00509	0.175	6%	lat

MaxPool1dFwd

input: [N, C, L_in]

• sincnet-localinput: [32, 80, 4096]kernel_size=3stride=3dtype=f16, f32
• textcnn-globalinput: [64, 256, 128]kernel_size=128stride=1dtype=f16
• ecg-cnn-dilatedinput: [16, 128, 2048]kernel_size=5stride=2padding=2dilation=2ceil_mode=truedtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
sincnet-local	f16	1.10×	0.0104	torch-compile torch-ref	0.0115 0.0506	1	60%	mem
	f32	1.00×	0.0158	torch-compile torch-ref	0.0157 0.0501	0.665	79%	mem
textcnn-global	f16	1.10×	0.00346	torch-compile torch-ref	0.00381 0.0265	0.607	27%	lat
ecg-cnn-dilated	bf16	1.33×	0.00595	torch-compile torch-ref	0.0079 0.0354	1.76	31%	lat

MaxPool1dIndicesFwd

input: [N, C, L_in]

• sincnet-localinput: [32, 80, 4096]kernel_size=3stride=3dtype=f16, f32
• textcnn-globalinput: [64, 256, 128]kernel_size=128stride=1dtype=f16
• ecg-cnn-dilatedinput: [16, 128, 2048]kernel_size=5stride=2padding=2dilation=2ceil_mode=truedtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
sincnet-local	f16	1.03×	0.0158	torch-compile torch-ref	0.0163 0.0506	0.663	79%	mem
	f32	1.00×	0.0215	torch-compile torch-ref	0.0216 0.0500	0.488	87%	mem
textcnn-global	f16	1.26×	0.00461	torch-compile torch-ref	0.00579 0.0265	0.455	21%	lat
ecg-cnn-dilated	bf16	1.29×	0.0104	torch-compile torch-ref	0.0134 0.0354	1.01	54%	mem

MaxPool2dFwd

input: [N, C, H_in, W_in]

stride=[2,2]dtype=f16, bf16, f32

• resnet-steminput: [32, 64, 112, 112]kernel_size=[3,3]padding=[1,1]
• vgg-blockinput: [16, 128, 56, 56]kernel_size=[2,2]padding=[0,0]
• alexnet-ceilinput: [32, 64, 55, 55]kernel_size=[3,3]padding=[0,0]ceil_mode=true

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
resnet-stem	f16	1.47×	0.0235	torch-compile flaggems torch-ref	0.0344 0.0788 0.1394	2.44	61%	mem
	bf16	1.39×	0.0248	torch-compile flaggems torch-ref	0.0344 0.0788 0.1396	2.31	58%	mem
	f32	1.48×	0.0334	torch-compile flaggems torch-ref	0.0496 0.0816 0.1348	1.71	85%	mem
vgg-block	f16	1.19×	0.00608	torch-compile flaggems torch-ref	0.00723 0.0148 0.0278	1.06	59%	mem
	bf16	1.19×	0.00608	torch-compile flaggems torch-ref	0.00726 0.0148 0.0279	1.06	59%	mem
	f32	1.04×	0.00995	torch-compile flaggems torch-ref	0.0103 0.0169 0.0277	0.645	72%	mem
alexnet-ceil	f16	1.27×	0.0087	torch-compile flaggems torch-ref	0.0111 0.0226 0.0348	1.54	39%	mem
	bf16	1.27×	0.00874	torch-compile flaggems torch-ref	0.0111 0.0225 0.0348	1.54	39%	mem
	f32	1.27×	0.0123	torch-compile flaggems torch-ref	0.0155 0.0229 0.0342	1.1	56%	mem

MaxPool2dIndicesFwd

input: [N, C, H_in, W_in]

stride=[2,2]dtype=f16, bf16, f32

• resnet-steminput: [32, 64, 112, 112]kernel_size=[3,3]padding=[1,1]
• vgg-blockinput: [16, 128, 56, 56]kernel_size=[2,2]padding=[0,0]
• alexnet-ceilinput: [32, 64, 55, 55]kernel_size=[3,3]padding=[0,0]ceil_mode=true

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
resnet-stem	f16	2.21×	0.0315	torch-compile flaggems torch-ref	0.0695 0.0789 0.1393	1.82	82%	mem
	bf16	2.16×	0.0324	torch-compile flaggems torch-ref	0.0701 0.0788 0.1397	1.76	79%	mem
	f32	1.52×	0.0473	torch-compile flaggems torch-ref	0.0721 0.0813 0.1349	1.21	84%	mem
vgg-block	f16	1.09×	0.00986	torch-compile flaggems torch-ref	0.0108 0.0148 0.0278	0.652	65%	mem
	bf16	1.25×	0.00982	torch-compile flaggems torch-ref	0.0123 0.0148 0.0279	0.654	65%	mem
	f32	0.99×	0.0131	torch-compile flaggems torch-ref	0.0129 0.0169 0.0277	0.491	76%	mem
alexnet-ceil	f16	1.59×	0.0112	torch-compile flaggems torch-ref	0.0178 0.0226 0.0347	1.2	54%	mem
	bf16	1.58×	0.0111	torch-compile flaggems torch-ref	0.0175 0.0226 0.0348	1.21	55%	mem
	f32	1.32×	0.0147	torch-compile flaggems torch-ref	0.0194 0.0229 0.0342	0.913	64%	mem

MaxPool3dFwd

input: [N, C, D_in, H_in, W_in]

• c3d-pool1input: [8, 64, 16, 112, 112]kernel_size=[1,2,2]stride=[1,2,2]padding=[0,0,0]dtype=f16, f32
• c3d-pool2input: [4, 128, 16, 56, 56]kernel_size=[2,2,2]stride=[2,2,2]padding=[0,0,0]dtype=f16
• medicalnet-steminput: [2, 64, 32, 112, 112]kernel_size=[3,3,3]stride=[2,2,2]padding=[1,1,1]ceil_mode=truedtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
c3d-pool1	f16	1.04×	0.0751	torch-compile cudnn torch-ref	0.0783 0.3043 0.5201	1.37	76%	mem
	f32	1.00×	0.1211	torch-compile cudnn torch-ref	0.1208 0.3466 0.5261	0.849	94%	mem
c3d-pool2	f16	1.03×	0.0240	torch-compile cudnn torch-ref	0.0248 0.0619 0.0951	1.07	53%	mem
medicalnet-stem	bf16	2.68×	0.0997	cudnn torch-ref torch-compile	0.2669 0.3359 0.9276	1.74	26%	mem

MaxPool3dIndicesFwd

input: [N, C, D_in, H_in, W_in]

• c3d-pool1input: [8, 64, 16, 112, 112]kernel_size=[1,2,2]stride=[1,2,2]padding=[0,0,0]dtype=f16, f32
• c3d-pool2input: [4, 128, 16, 56, 56]kernel_size=[2,2,2]stride=[2,2,2]padding=[0,0,0]dtype=f16
• medicalnet-steminput: [2, 64, 32, 112, 112]kernel_size=[3,3,3]stride=[2,2,2]padding=[1,1,1]ceil_mode=truedtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
c3d-pool1	f16	1.07×	0.1224	torch-compile torch-ref	0.1305 0.5201	0.84	84%	mem
	f32	1.00×	0.1667	torch-compile torch-ref	0.1668 0.5260	0.616	96%	mem
c3d-pool2	f16	1.20×	0.0274	torch-compile torch-ref	0.0329 0.0950	0.939	68%	mem
medicalnet-stem	bf16	15.63×	0.1304	torch-ref torch-compile	0.3359 2.0378	1.33	30%	mem

AdaptiveAvgPool2dFwd

input: [*B, C, H_in, W_in]

• resnet-gapinput: [8, 2048, 7, 7]output_size=[1,1]dtype=f16
• spp-6x6input: [2, 128, 56, 56]output_size=[6,6]dtype=f16
• nondiv-7x7input: [2, 64, 55, 57]output_size=[7,7]dtype=bf16
• resnet-gap-unbatchedinput: [2048, 7, 7]output_size=[1,1]dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
resnet-gap	f16	1.27×	0.00294	torch-compile torch-ref	0.00374 0.0073	0.273	12%	lat
spp-6x6	f16	1.96×	0.00546	torch-compile torch-ref	0.0107 0.0108	0.169	7%	lat
nondiv-7x7	bf16	1.39×	0.00669	torch-ref torch-compile	0.00925 0.00928	0.0736	3%	lat
resnet-gap-unbatched	f16	2.00×	0.00243	torch-ref torch-compile	0.00317 0.00486	0.0413	2%	lat

AdaptiveMaxPool2dFwd

input: [*B, C, H_in, W_in]

• global-1x1input: [8, 2048, 7, 7]output_size=[1,1]dtype=f16
• spp-6x6input: [2, 128, 56, 56]output_size=[6,6]dtype=f16
• nondiv-7x7input: [2, 64, 55, 57]output_size=[7,7]dtype=bf16
• global-1x1-unbatchedinput: [2048, 7, 7]output_size=[1,1]dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
global-1x1	f16	2.17×	0.00278	torch-compile torch-ref	0.00605 0.0445	0.288	13%	lat
spp-6x6	f16	4.06×	0.00355	torch-ref torch-compile	0.0144 0.0144	0.259	10%	lat
nondiv-7x7	bf16	3.43×	0.00338	torch-compile torch-ref	0.0116 0.0116	0.146	5%	lat
global-1x1-unbatched	f16	1.05×	0.00192	torch-compile torch-ref	0.00202 0.0128	0.0523	2%	lat

AdaptiveMaxPool2dIndicesFwd

input: [*B, C, H_in, W_in]

• global-1x1input: [8, 2048, 7, 7]output_size=[1,1]dtype=f16
• spp-6x6input: [2, 128, 56, 56]output_size=[6,6]dtype=f16
• nondiv-7x7input: [2, 64, 55, 57]output_size=[7,7]dtype=bf16
• global-1x1-unbatchedinput: [2048, 7, 7]output_size=[1,1]dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
global-1x1	f16	2.91×	0.00282	torch-compile torch-ref	0.00819 0.0446	0.285	14%	lat
spp-6x6	f16	3.99×	0.00362	torch-ref torch-compile	0.0144 0.0144	0.255	10%	lat
nondiv-7x7	bf16	3.32×	0.00349	torch-compile torch-ref	0.0116 0.0116	0.141	6%	lat
global-1x1-unbatched	f16	1.56×	0.00205	torch-compile torch-ref	0.0032 0.0128	0.049	2%	lat

MeanPoolingFwd

x: [B, S, H, D]

chunk_size=64

• uniform-8kx: [1, 8192, 64, 128]dtype=f16/f32, bf16/f32
• uniform-batchedx: [2, 2048, 64, 128]dtype=f16/f32

x: [B, S, H, D]offsets: [NS + 1], i32indices: [NC, 2], i32

• ragged-evenx: [1, 8192, 64, 128]offsets: [5], i32indices: [128, 2], i32seq_lens=repeat(2048, 4)chunk_size=64dtype=f16/f32, f32/f32
• ragged-tailx: [1, 1000, 64, 128]offsets: [5], i32indices: [34, 2], i32seq_lens=[100,200,300,400]chunk_size=32dtype=f16/f32
• ragged-batchedx: [2, 2048, 64, 128]offsets: [3], i32indices: [32, 2], i32seq_lens=repeat(1024, 2)chunk_size=64dtype=f16/f32

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
uniform-8k	f16/f32	1.21×	0.0388	torch-view-mean torch-compile torch-ref	0.0471 0.3804 0.6260	1.76	78%	mem
	bf16/f32	1.25×	0.0393	torch-view-mean torch-compile torch-ref	0.0491 0.3800 0.6311	1.73	77%	mem
uniform-batched	f16/f32	1.29×	0.0223	torch-view-mean torch-compile torch-ref	0.0287 0.1402 0.1832	1.53	68%	mem
ragged-even	f16/f32	7.50×	0.0406	torch-segment-reduce torch-compile torch-ref	0.3041 0.3811 0.6306	1.68	75%	mem
	f32/f32	1.15×	0.0699	torch-segment-reduce torch-compile torch-ref	0.0801 0.3797 0.6518	0.975	87%	mem
ragged-tail	f16/f32	4.82×	0.00938	torch-segment-reduce torch-compile torch-ref	0.0452 0.0964 0.2181	0.903	40%	mem
ragged-batched	f16/f32	5.85×	0.0240	torch-compile torch-ref torch-segment-reduce	0.1407 0.1876 0.2657	1.42	63%	mem

Convolution

Conv1dFwd

input: [N, C_in, L_in]weight: [C_out, C_in_g, kW]

dtype=f16, bf16

• whisper-conv1input: [1, 80, 3000]weight: [1280, 80, 3]
• wav2vec2-layer1input: [1, 1, 16000]weight: [512, 1, 10]stride=5
• encodec-initinput: [1, 1, 24000]weight: [32, 1, 7]
• encodec-deepinput: [1, 128, 600]weight: [256, 128, 10]

input: [N, C_in, L_in]weight: [C_out, C_in_g, kW]bias: [C_out]

dtype=f16, bf16

• whisper-conv1-biasinput: [1, 80, 3000]weight: [1280, 80, 3]bias: [1280]
• wav2vec2-layer1-biasinput: [1, 1, 16000]weight: [512, 1, 10]bias: [512]stride=5
• encodec-init-biasinput: [1, 1, 24000]weight: [32, 1, 7]bias: [32]
• encodec-deep-biasinput: [1, 128, 600]weight: [256, 128, 10]bias: [256]

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
whisper-conv1	f16	2.52×	0.0225	torch torch-compile flaggems	0.0568 0.0568 0.1131	81.8	11%	comp
	bf16	2.50×	0.0226	torch torch-compile flaggems	0.0566 0.0566 0.1116	81.4	11%	comp
wav2vec2-layer1	f16	2.96×	0.0063	torch torch-compile flaggems	0.0187 0.0187 0.0405	5.2	12%	lat
	bf16	2.98×	0.0063	torch torch-compile flaggems	0.0188 0.0188 0.0404	5.2	12%	lat
encodec-init	f16	1.99×	0.00333	torch torch-compile flaggems	0.00662 0.00666 0.0241	3.23	11%	lat
	bf16	2.00×	0.00333	torch torch-compile flaggems	0.00666 0.00666 0.0240	3.23	11%	lat
encodec-deep	f16	1.74×	0.00982	torch torch-compile flaggems	0.0171 0.0171 0.0715	39.4	5%	lat
	bf16	1.74×	0.00982	torch torch-compile flaggems	0.0171 0.0171 0.0714	39.4	5%	lat
whisper-conv1-bias	f16	2.76×	0.0231	torch-compile torch flaggems	0.0637 0.0693 0.1115	79.9	11%	comp
	bf16	2.75×	0.0232	torch-compile torch flaggems	0.0637 0.0694 0.1118	79.7	10%	comp
wav2vec2-layer1-bias	f16	3.15×	0.00685	torch-compile torch flaggems	0.0216 0.0252 0.0396	5.02	11%	lat
	bf16	3.27×	0.00662	torch-compile torch flaggems	0.0217 0.0255 0.0394	5.19	11%	lat
encodec-init-bias	f16	2.42×	0.00355	torch-compile torch flaggems	0.00861 0.0107 0.0232	3.24	10%	lat
	bf16	2.71×	0.00336	torch-compile torch flaggems	0.00912 0.0108 0.0232	3.43	10%	lat
encodec-deep-bias	f16	1.81×	0.0103	torch-compile torch flaggems	0.0187 0.0204 0.0706	37.6	5%	lat
	bf16	1.81×	0.0103	torch-compile torch flaggems	0.0186 0.0204 0.0705	37.7	5%	lat

Conv2dFwd

input: [N, C_in, H, W]weight: [C_out, C_in_g, kH, kW]

• resnet-3x3input: [2, 64, 56, 56]weight: [64, 64, 3, 3]stride=[1,1]padding=[1,1]dtype=f16, bf16
• steminput: [1, 3, 112, 112]weight: [64, 3, 3, 3]stride=[2,2]padding=[1,1]dtype=f16
• stage-down-3x3input: [1, 128, 56, 56]weight: [256, 128, 3, 3]stride=[2,2]padding=[1,1]dtype=f16
• highresinput: [1, 256, 112, 112]weight: [512, 256, 3, 3]stride=[1,1]padding=[1,1]dtype=f16
• midresinput: [1, 64, 56, 56]weight: [128, 64, 5, 5]stride=[1,1]padding=[2,2]dtype=f16
• stage-down-5x5input: [1, 128, 56, 56]weight: [256, 128, 5, 5]stride=[2,2]padding=[2,2]dtype=f16
• stride2input: [1, 128, 28, 28]weight: [128, 128, 3, 3]stride=[2,2]padding=[1,1]dtype=bf16
• resnet-1x1input: [2, 64, 56, 56]weight: [256, 64, 1, 1]stride=[1,1]padding=[0,0]dtype=f16, bf16
• bottleneck-expandinput: [2, 128, 28, 28]weight: [512, 128, 1, 1]stride=[1,1]padding=[0,0]dtype=f16
• bottleneck-reduceinput: [2, 512, 28, 28]weight: [128, 512, 1, 1]stride=[1,1]padding=[0,0]dtype=f16
• late-stage-1x1input: [1, 256, 14, 14]weight: [1024, 256, 1, 1]stride=[1,1]padding=[0,0]dtype=f16
• classifier-1x1input: [1, 512, 7, 7]weight: [2048, 512, 1, 1]stride=[1,1]padding=[0,0]dtype=f16
• deeplabv3-asppinput: [1, 2048, 32, 32]weight: [256, 2048, 3, 3]stride=[1,1]padding=[12,12]dilation=[12,12]dtype=f16
• mbv2-depthwiseinput: [1, 32, 56, 56]weight: [32, 1, 3, 3]stride=[1,1]padding=[1,1]groups=32dtype=f16
• resnext-grouped-3x3input: [1, 128, 28, 28]weight: [256, 4, 3, 3]stride=[1,1]padding=[1,1]groups=32dtype=f16

input: [N, C_in, H, W]weight: [C_out, C_in_g, kH, kW]bias: [C_out]

• resnet-3x3-biasinput: [2, 64, 56, 56]weight: [64, 64, 3, 3]bias: [64]stride=[1,1]padding=[1,1]dtype=f16, bf16
• stem-biasinput: [1, 3, 112, 112]weight: [64, 3, 3, 3]bias: [64]stride=[2,2]padding=[1,1]dtype=f16
• stage-down-3x3-biasinput: [1, 128, 56, 56]weight: [256, 128, 3, 3]bias: [256]stride=[2,2]padding=[1,1]dtype=f16
• highres-biasinput: [1, 256, 112, 112]weight: [512, 256, 3, 3]bias: [512]stride=[1,1]padding=[1,1]dtype=f16
• midres-biasinput: [1, 64, 56, 56]weight: [128, 64, 5, 5]bias: [128]stride=[1,1]padding=[2,2]dtype=f16
• stage-down-5x5-biasinput: [1, 128, 56, 56]weight: [256, 128, 5, 5]bias: [256]stride=[2,2]padding=[2,2]dtype=f16
• stride2-biasinput: [1, 128, 28, 28]weight: [128, 128, 3, 3]bias: [128]stride=[2,2]padding=[1,1]dtype=bf16
• resnet-1x1-biasinput: [2, 64, 56, 56]weight: [256, 64, 1, 1]bias: [256]stride=[1,1]padding=[0,0]dtype=f16, bf16
• bottleneck-expand-biasinput: [2, 128, 28, 28]weight: [512, 128, 1, 1]bias: [512]stride=[1,1]padding=[0,0]dtype=f16
• bottleneck-reduce-biasinput: [2, 512, 28, 28]weight: [128, 512, 1, 1]bias: [128]stride=[1,1]padding=[0,0]dtype=f16
• late-stage-1x1-biasinput: [1, 256, 14, 14]weight: [1024, 256, 1, 1]bias: [1024]stride=[1,1]padding=[0,0]dtype=f16
• classifier-1x1-biasinput: [1, 512, 7, 7]weight: [2048, 512, 1, 1]bias: [2048]stride=[1,1]padding=[0,0]dtype=f16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
resnet-3x3	f16	1.63×	0.00714	torch-compile torch flaggems	0.0116 0.0148 0.0834	63.3	9%	lat
	bf16	1.66×	0.00714	torch-compile torch flaggems	0.0118 0.0149 0.0833	63.3	8%	lat
stem	f16	2.11×	0.00312	torch torch-compile flaggems	0.00659 0.00925 0.0132	3.43	3%	lat
stage-down-3x3	f16	1.45×	0.00931	torch-compile torch flaggems	0.0135 0.0170 0.1183	48.5	7%	lat
highres	f16	1.23×	0.0641	torch-compile torch flaggems	0.0785 0.0949 0.7347	456	63%	comp
midres	f16	1.53×	0.0106	torch-compile torch flaggems	0.0163 0.0193 0.2017	116	16%	lat
stage-down-5x5	f16	1.28×	0.0174	torch-compile torch flaggems	0.0222 0.0254 0.3096	70.8	10%	lat
stride2	bf16	1.51×	0.00803	torch-compile torch flaggems	0.0121 0.0148 0.0647	6.86	2%	lat
resnet-1x1	f16	1.06×	0.004	torch torch-compile flaggems	0.00422 0.00845 0.0495	51.4	22%	lat
	bf16	1.01×	0.004	torch torch-compile flaggems	0.00403 0.00835 0.0495	51.4	22%	lat
bottleneck-expand	f16	1.18×	0.00343	torch torch-compile flaggems	0.00403 0.00742 0.0290	60	14%	lat
bottleneck-reduce	f16	1.06×	0.00413	torch torch-compile flaggems	0.00438 0.008 0.0264	49.8	12%	lat
late-stage-1x1	f16	1.55×	0.0041	torch torch-compile flaggems	0.00637 0.00669 0.0157	25.1	6%	lat
classifier-1x1	f16	1.03×	0.0071	torch-compile torch flaggems	0.00733 0.00922 0.0211	14.5	7%	lat
deeplabv3-aspp	f16	1.09×	0.0836	torch-compile torch flaggems	0.0915 0.1193 0.7178	65	9%	comp
mbv2-depthwise	f16	1.39×	0.00221	torch torch-compile flaggems	0.00307 0.00563 0.0542	0.799	4%	lat
resnext-grouped-3x3	f16	5.05×	0.00378	torch-compile torch flaggems	0.0191 0.0191 0.0193	3.65	4%	lat
resnet-3x3-bias	f16	1.64×	0.00755	torch-compile torch flaggems	0.0124 0.0183 0.0827	59.8	8%	lat
	bf16	1.60×	0.00755	torch-compile torch flaggems	0.0121 0.0185 0.0826	59.8	8%	lat
stem-bias	f16	2.70×	0.00355	torch torch-compile flaggems	0.0096 0.0097 0.0124	3.07	3%	lat
stage-down-3x3-bias	f16	1.39×	0.00992	torch-compile torch flaggems	0.0138 0.0200 0.1172	45.5	6%	lat
highres-bias	f16	1.19×	0.0665	torch-compile torch flaggems	0.0789 0.1145 0.7313	440	60%	comp
midres-bias	f16	1.42×	0.0115	torch-compile torch flaggems	0.0164 0.0225 0.2015	107	15%	lat
stage-down-5x5-bias	f16	1.26×	0.0179	torch-compile torch flaggems	0.0225 0.0287 0.3088	68.7	9%	lat
stride2-bias	bf16	1.48×	0.00855	torch-compile torch flaggems	0.0127 0.0175 0.0638	6.45	1%	lat
resnet-1x1-bias	f16	2.13×	0.00422	torch-compile torch flaggems	0.00899 0.0116 0.0486	49	21%	lat
	bf16	2.02×	0.00419	torch-compile torch flaggems	0.00845 0.0116 0.0486	49.4	21%	lat
bottleneck-expand-bias	f16	2.11×	0.00362	torch-compile torch flaggems	0.00765 0.00867 0.0280	57.1	13%	lat
bottleneck-reduce-bias	f16	1.76×	0.00416	torch torch-compile flaggems	0.00733 0.00829 0.0255	49.5	11%	lat
late-stage-1x1-bias	f16	1.65×	0.00435	torch-compile torch flaggems	0.00717 0.00923 0.0150	23.7	5%	lat
classifier-1x1-bias	f16	1.05×	0.00758	torch-compile torch flaggems	0.00797 0.0121 0.0204	13.6	7%	lat

Conv3dFwd

input: [N, C_in, D, H, W]weight: [C_out, C_in_g, kD, kH, kW]

• r3d-steminput: [1, 3, 16, 112, 112]weight: [64, 3, 3, 3, 3]stride=[1,1,1]padding=[1,1,1]dtype=f16
• video-downinput: [1, 64, 8, 56, 56]weight: [128, 64, 3, 3, 3]stride=[2,2,2]padding=[1,1,1]dtype=f16
• unet-encoderinput: [1, 32, 32, 64, 64]weight: [64, 32, 3, 3, 3]stride=[1,1,1]padding=[1,1,1]dtype=bf16
• unet-asppinput: [1, 256, 8, 16, 16]weight: [256, 256, 3, 3, 3]stride=[1,1,1]padding=[6,6,6]dilation=[6,6,6]dtype=f16
• resnext-groupedinput: [1, 64, 8, 28, 28]weight: [128, 2, 3, 3, 3]stride=[1,1,1]padding=[1,1,1]groups=32dtype=f16

input: [N, C_in, D, H, W]weight: [C_out, C_in_g, kD, kH, kW]bias: [C_out]

padding=[1,1,1]

• r3d-stem-biasinput: [1, 3, 16, 112, 112]weight: [64, 3, 3, 3, 3]bias: [64]stride=[1,1,1]dtype=f16
• video-down-biasinput: [1, 64, 8, 56, 56]weight: [128, 64, 3, 3, 3]bias: [128]stride=[2,2,2]dtype=f16
• unet-encoder-biasinput: [1, 32, 32, 64, 64]weight: [64, 32, 3, 3, 3]bias: [64]stride=[1,1,1]dtype=bf16

Workload	dtype	Ratio	Device time	Alternatives		Throughput	SOL	Bound
		alt / ours	ms	name	ms	TFLOP/s	of ceiling	by
r3d-stem	f16	3.90×	0.0220	flaggems torch-compile torch	0.0858 0.1148 0.1148	89.5	27%	mem
video-down	f16	1.47×	0.0180	torch-compile torch flaggems	0.0265 0.0265 0.2172	69	9%	lat
unet-encoder	bf16	1.76×	0.0660	torch torch-compile flaggems	0.1164 0.1168 0.3176	211	28%	comp
unet-aspp	f16	1.31×	0.0290	torch-compile torch flaggems	0.0379 0.0379 0.3009	70.4	10%	comp
resnext-grouped	f16	16.58×	0.0154	flaggems torch torch-compile	0.2546 0.3312 0.3361	4.93	4%	lat
r3d-stem-bias	f16	3.81×	0.0223	flaggems torch-compile torch	0.0850 0.1261 0.1547	88.9	27%	mem
video-down-bias	f16	1.52×	0.0184	torch-compile torch flaggems	0.0280 0.0299 0.2157	67.4	9%	lat
unet-encoder-bias	bf16	1.86×	0.0666	torch-compile torch flaggems	0.1235 0.1420 0.3167	209	27%	comp