Profiling Tutorial

Profiling Tutorial

这个 notebook 面向工程实践，展示一个专业工程师如何用 cProfile、torch.profiler、nsys、ncu 逐层缩小瓶颈。

from pathlib import Path
import subprocess, sys
ROOT = Path.cwd()
MONI_PYTHON = Path(r'C:\Software\Miniconda\envs\moni\python.exe')
NSYS = Path(r'C:\Program Files\NVIDIA Corporation\Nsight Compute 2025.4.1\host\target-windows-x64\nsys.exe')
NCU = Path(r'C:\Program Files\NVIDIA Corporation\Nsight Compute 2025.4.1\target\windows-desktop-win7-x64\ncu.exe')
print(ROOT)
print(MONI_PYTHON)
print(NSYS)
print(NCU)

D:\_code\mlsys\pj1\profiling_tutorialC:\Software\Miniconda\envs\moni\python.exeC:\Program Files\NVIDIA Corporation\Nsight Compute 2025.4.1\host\target-windows-x64\nsys.exeC:\Program Files\NVIDIA Corporation\Nsight Compute 2025.4.1\target\windows-desktop-win7-x64\ncu.exe

Case 1: CPU-only hotspot

先用纯 Python 案例训练基本动作：先看 baseline，再用 cProfile 确认热点到底在不在 Python 控制层。

import subprocess, sys
subprocess.run([sys.executable, 'examples/cpu_hotspot_case.py', '--mode', 'slow', '--lines', '6000'], cwd='profiling_tutorial')

mode=slowlines=6000elapsed_ms=67.840top10=[('launch', 6602), ('occupancy', 6594), ('stream', 6561), ('profile', 6536), ('memory', 6512), ('torch', 6503), ('kernel', 6501), ('trace', 6499), ('gpu', 6443), ('synchronize', 6440)]
import subprocess, sys
subprocess.run([sys.executable, 'examples/cpu_hotspot_case.py', '--mode', 'fast', '--lines', '6000'], cwd='profiling_tutorial')

mode=fastlines=6000elapsed_ms=26.033top10=[('launch', 6602), ('occupancy', 6594), ('stream', 6561), ('profile', 6536), ('memory', 6512), ('torch', 6503), ('kernel', 6501), ('trace', 6499), ('gpu', 6443), ('synchronize', 6440)]

How to read this step

import subprocess, sys
subprocess.run([
    sys.executable, '-m', 'cProfile', '-o', 'outputs/cpu_slow.prof',
    'examples/cpu_hotspot_case.py', '--mode', 'slow', '--lines', '6000'
], cwd='profiling_tutorial')
subprocess.run([
    sys.executable, 'examples/show_pstats.py', 'outputs/cpu_slow.prof', '--sort', 'cumtime', '--limit', '15'
], cwd='profiling_tutorial')

Sat Apr  4 23:20:15 2026    D:\_code\mlsys\pj1\profiling_tutorial\outputs\cpu_slow.prof         2851294 function calls (2851117 primitive calls) in 1.121 seconds   Ordered by: cumulative time   List reduced from 369 to 15 due to restriction <15>   ncalls  tottime  percall  cumtime  percall filename:lineno(function)     12/1    0.000    0.000    1.121    1.121 {built-in method builtins.exec}        1    0.000    0.000    1.121    1.121 cpu_hotspot_case.py:1(<module>)        1    0.000    0.000    1.113    1.113 cpu_hotspot_case.py:60(main)        1    0.037    0.037    0.789    0.789 cpu_hotspot_case.py:42(slow_count_words)     6000    0.409    0.000    0.733    0.000 cpu_hotspot_case.py:32(_normalize_line_slow)        1    0.036    0.036    0.314    0.314 cpu_hotspot_case.py:18(make_lines)   128892    0.079    0.000    0.211    0.000 random.py:345(choice)    18310    0.020    0.000    0.127    0.000 {method 'join' of 'str' objects}   140892    0.076    0.000    0.118    0.000 random.py:245(_randbelow_with_getrandbits)   629523    0.117    0.000    0.117    0.000 {method 'append' of 'list' objects}    56928    0.021    0.000    0.107    0.000 cpu_hotspot_case.py:27(<genexpr>)   622863    0.103    0.000    0.103    0.000 {method 'isalpha' of 'str' objects}   493983    0.094    0.000    0.094    0.000 {method 'lower' of 'str' objects}    12000    0.004    0.000    0.030    0.000 random.py:336(randint)    12000    0.010    0.000    0.026    0.000 random.py:295(randrange)

Engineering conclusion

这个 CPU 案例的热点在 Python 文本清洗和词频统计逻辑本身，因此正确动作是继续改 Python 代码，而不是跳去 torch.profiler 或 nsys。

Case 2: GPU matmul bottleneck

这一组案例最重要。我们会先对一个故意写得很差的 GPU matmul 程序做 baseline，然后逐层上工具。

import subprocess
subprocess.run([r'C:\Software\Miniconda\envs\moni\python.exe', 'examples/gpu_matmul_case.py', '--mode', 'slow_loop_sync', '--device', 'cuda', '--repeats', '128', '--tile-size', '128'], cwd='profiling_tutorial')

mode=slow_loop_syncdevice=cudarepeats=128tile_size=128elapsed_ms=38.528checksum=5959.013672
import subprocess
subprocess.run([r'C:\Software\Miniconda\envs\moni\python.exe', 'examples/gpu_matmul_case.py', '--mode', 'batched_bmm', '--device', 'cuda', '--repeats', '128', '--tile-size', '128'], cwd='profiling_tutorial')

mode=batched_bmmdevice=cudarepeats=128tile_size=128elapsed_ms=3.153checksum=5959.012695

How to read this step

import subprocess
subprocess.run([r'C:\Software\Miniconda\envs\moni\python.exe', 'examples/torch_profile_matmul.py', '--mode', 'slow_loop_sync', '--device', 'cuda', '--repeats', '128', '--tile-size', '128', '--output', 'outputs/torch_slow_summary.json'], cwd='profiling_tutorial')

-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------                                                     Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls  -------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------                                                 aten::mm        37.72%     124.221ms        43.65%     143.718ms       1.123ms     938.816us        81.98%     962.528us       7.520us           128                          ampere_sgemm_32x32_sliced1x4_tn         0.00%       0.000us         0.00%       0.000us       0.000us     938.816us        81.98%     938.816us       7.335us           128                                 slow_loop_sync_iteration         0.00%       0.000us         0.00%       0.000us       0.000us     938.816us        81.98%     938.816us       7.335us           128                                    Lazy Function Loading         0.13%     418.400us         0.13%     418.400us     139.467us     160.737us        14.04%     160.737us      53.579us             3                                            aten::normal_         0.78%       2.555ms         1.76%       5.788ms       2.894ms     105.568us         9.22%     157.600us      78.800us             2  _ZN2at6native54_GLOBAL__N__966ce9c1_21_DistributionN...         0.00%       0.000us         0.00%       0.000us       0.000us     105.568us         9.22%     105.568us      52.784us             2                                                aten::cat         2.83%       9.329ms         4.54%      14.955ms      14.955ms     100.801us         8.80%     201.602us     201.602us             1  _ZN2at6native40_GLOBAL__N__0b46c3ea_8_Shape_cu_85d9f...         0.00%       0.000us         0.00%       0.000us       0.000us     100.801us         8.80%     100.801us     100.801us             1                                       cudaGetDeviceCount         0.00%      10.300us         0.00%      10.300us       3.433us      52.032us         4.54%      52.032us      17.344us             3                                    cudaDeviceSynchronize         0.33%       1.072ms         0.33%       1.072ms       8.248us      22.016us         1.92%      22.016us       0.169us           130                         Runtime Triggered Module Loading         1.90%       6.268ms         1.90%       6.268ms       3.134ms      15.808us         1.38%      15.808us       7.904us             2                               cudaGetDeviceProperties_v2         0.00%       3.500us         0.00%       3.500us       1.167us       7.456us         0.65%       7.456us       2.485us             3  -------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  Self CPU time total: 329.281msSelf CUDA time total: 1.145msC:\Software\Miniconda\envs\moni\Lib\site-packages\torch\profiler\profiler.py:217: UserWarning: Warning: Profiler clears events at the end of each cycle.Only events from the current cycle will be reported.To keep events across cycles, set acc_events=True.  _warn_once(
import subprocess
subprocess.run([r'C:\Software\Miniconda\envs\moni\python.exe', 'examples/torch_profile_matmul.py', '--mode', 'batched_bmm', '--device', 'cuda', '--repeats', '128', '--tile-size', '128', '--output', 'outputs/torch_fast_summary.json'], cwd='profiling_tutorial')

-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------                                                     Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls  -------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------                                    Lazy Function Loading         0.26%     667.000us         0.26%     667.000us     111.167us       1.111ms       350.10%       1.111ms     185.169us             6                         Runtime Triggered Module Loading         2.55%       6.527ms         2.55%       6.527ms       3.263ms     423.810us       133.55%     423.810us     211.905us             2                                                aten::bmm        45.82%     117.203ms        50.18%     128.359ms     128.359ms     211.905us        66.77%       1.695ms       1.695ms             1                                  ampere_sgemm_128x128_tn         0.00%       0.000us         0.00%       0.000us       0.000us     211.905us        66.77%     211.905us     211.905us             1                                            aten::normal_         1.18%       3.007ms         2.82%       7.222ms       3.611ms     105.440us        33.23%     156.928us      78.464us             2  _ZN2at6native54_GLOBAL__N__966ce9c1_21_DistributionN...         0.00%       0.000us         0.00%       0.000us       0.000us     105.440us        33.23%     105.440us      52.720us             2                                       cudaGetDeviceCount         0.00%      11.700us         0.00%      11.700us       3.900us      51.488us        16.22%      51.488us      17.163us             3                                  Activity Buffer Request         0.61%       1.558ms         0.61%       1.558ms       1.558ms       0.000us         0.00%       0.000us       0.000us             1                               cudaGetDeviceProperties_v2         0.00%       2.600us         0.00%       2.600us       0.867us       0.000us         0.00%       0.000us       0.000us             3                                              aten::randn         0.18%     460.500us        49.12%     125.647ms      62.823ms       0.000us         0.00%     156.928us      78.464us             2                                              aten::empty         0.31%     798.700us        46.12%     117.965ms      58.982ms       0.000us         0.00%       0.000us       0.000us             2                         cudaDeviceGetStreamPriorityRange        42.19%     107.900ms        45.68%     116.830ms     116.830ms       0.000us         0.00%       0.000us       0.000us             1  -------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  Self CPU time total: 255.776msSelf CUDA time total: 317.345usC:\Software\Miniconda\envs\moni\Lib\site-packages\torch\profiler\profiler.py:217: UserWarning: Warning: Profiler clears events at the end of each cycle.Only events from the current cycle will be reported.To keep events across cycles, set acc_events=True.  _warn_once(

How to read `torch.profiler`

如果看到很多次 aten::mm，而单次 CUDA 时间很短，那下一步优先怀疑 launch/sync，而不是先怀疑单个 kernel 太慢。

import subprocess
subprocess.run([r'C:\Program Files\NVIDIA Corporation\Nsight Compute 2025.4.1\host\target-windows-x64\nsys.exe', 'profile', '--trace=cuda,nvtx', '--sample=none', '--cpuctxsw=none', '--stats=true', '--force-overwrite=true', '--output', 'outputs/tutorial_nsys_slow', r'C:\Software\Miniconda\envs\moni\python.exe', 'examples/gpu_matmul_case.py', '--mode', 'slow_loop_sync', '--device', 'cuda', '--repeats', '128', '--tile-size', '128'], cwd='profiling_tutorial')
subprocess.run([r'C:\Program Files\NVIDIA Corporation\Nsight Compute 2025.4.1\host\target-windows-x64\nsys.exe', 'stats', '--force-export=true', '--report', 'cuda_api_sum,cuda_gpu_kern_sum', '--format', 'table', '--output', '-', 'outputs/tutorial_nsys_slow.nsys-rep'], cwd='profiling_tutorial')

Generating SQLite file D:\_code\mlsys\pj1\profiling_tutorial\outputs\tutorial_nsys_slow.sqlite from D:\_code\mlsys\pj1\profiling_tutorial\outputs\tutorial_nsys_slow.nsys-repProcessing [D:\_code\mlsys\pj1\profiling_tutorial\outputs\tutorial_nsys_slow.sqlite] with [C:\Program Files\NVIDIA Corporation\Nsight Compute 2025.4.1\host\target-windows-x64\reports\cuda_api_sum.py]...  ** CUDA API Summary (cuda_api_sum):+----------+-----------------+-----------+-----------+-----------+----------+----------+-------------+-----------------------------------------+| Time (%) | Total Time (ns) | Num Calls | Avg (ns)  | Med (ns)  | Min (ns) | Max (ns) | StdDev (ns) |                  Name                   |+----------+-----------------+-----------+-----------+-----------+----------+----------+-------------+-----------------------------------------+|     59.9 |        56057036 |         7 | 8008148.0 | 5290899.0 |  1172937 | 27717663 |   9117274.4 | cuLibraryLoadData                       ||     31.6 |        29570635 |       176 |  168015.0 |   25530.5 |    14959 | 15682630 |   1258291.8 | cudaLaunchKernel                        ||      3.5 |         3257346 |       165 |   19741.5 |    5687.0 |     3735 |   686554 |     69242.3 | cudaDeviceSynchronize                   ||      2.3 |         2123464 |        10 |  212346.4 |  216812.0 |     4891 |   336824 |     89380.4 | cudaMalloc                              ||      1.6 |         1524564 |         1 | 1524564.0 | 1524564.0 |  1524564 |  1524564 |         0.0 | cudaHostAlloc                           ||      0.4 |          364222 |         1 |  364222.0 |  364222.0 |   364222 |   364222 |         0.0 | cudaStreamSynchronize                   ||      0.2 |          159500 |       467 |     341.5 |     276.0 |      149 |     3822 |       298.7 | cuGetProcAddress_v2                     ||      0.1 |          110484 |         7 |   15783.4 |   11989.0 |     7257 |    27966 |      7694.2 | cuLibraryGetKernel                      ||      0.1 |          106960 |        31 |    3450.3 |     892.0 |      301 |    47293 |      8799.6 | cudaStreamIsCapturing_v10000            ||      0.1 |           80314 |         1 |   80314.0 |   80314.0 |    80314 |    80314 |         0.0 | cudaMemcpyAsync                         ||      0.1 |           66268 |         2 |   33134.0 |   33134.0 |      643 |    65625 |     45949.2 | cuModuleGetLoadingMode                  ||      0.1 |           50413 |       176 |     286.4 |     241.5 |      133 |      884 |       136.9 | cuKernelGetName                         ||      0.0 |           41088 |         1 |   41088.0 |   41088.0 |    41088 |    41088 |         0.0 | cudaMemsetAsync                         ||      0.0 |           37686 |        18 |    2093.7 |     772.5 |      538 |    16222 |      3769.5 | cudaEventCreateWithFlags                ||      0.0 |           23596 |         1 |   23596.0 |   23596.0 |    23596 |    23596 |         0.0 | cuCtxSynchronize                        ||      0.0 |           16699 |         1 |   16699.0 |   16699.0 |    16699 |    16699 |         0.0 | cudaFree                                ||      0.0 |           10480 |         2 |    5240.0 |    5240.0 |     1001 |     9479 |      5994.9 | cudaEventCreate                         ||      0.0 |            8578 |         3 |    2859.3 |    1666.0 |     1662 |     5250 |      2070.4 | cuInit                                  ||      0.0 |            4523 |         2 |    2261.5 |    2261.5 |     1158 |     3365 |      1560.6 | cudaGetDriverEntryPointByVersion_v12050 ||      0.0 |            3207 |         2 |    1603.5 |    1603.5 |     1111 |     2096 |       696.5 | cuGetProcAddress                        ||      0.0 |            2201 |         3 |     733.7 |     414.0 |      383 |     1404 |       580.7 | cudaGetDeviceProperties_v12000          ||      0.0 |            1255 |         2 |     627.5 |     627.5 |      475 |      780 |       215.7 | cuDeviceGetLuid                         |+----------+-----------------+-----------+-----------+-----------+----------+----------+-------------+-----------------------------------------+Processing [D:\_code\mlsys\pj1\profiling_tutorial\outputs\tutorial_nsys_slow.sqlite] with [C:\Program Files\NVIDIA Corporation\Nsight Compute 2025.4.1\host\target-windows-x64\reports\cuda_gpu_kern_sum.py]...  ** CUDA GPU Kernel Summary (cuda_gpu_kern_sum):+----------+-----------------+-----------+----------+----------+----------+----------+-------------+------------------------------------------------------------------------------------------------------+| Time (%) | Total Time (ns) | Instances | Avg (ns) | Med (ns) | Min (ns) | Max (ns) | StdDev (ns) |                                                 Name                                                 |+----------+-----------------+-----------+----------+----------+----------+----------+-------------+------------------------------------------------------------------------------------------------------+|     77.1 |         1131204 |       160 |   7070.0 |   7104.0 |     6144 |     8544 |       476.5 | ampere_sgemm_32x32_sliced1x4_tn                                                                      ||     11.2 |          164737 |        10 |  16473.7 |   7632.0 |     7584 |    52352 |     18542.5 | void at::native::<unnamed>::distribution_elementwise_grid_stride_kernel<float, (int)4, void at::nat� ||      8.0 |          116736 |         5 |  23347.2 |   4000.0 |     3840 |   100032 |     42870.3 | void at::native::<unnamed>::CatArrayBatchedCopy_vectorized<at::native::<unnamed>::OpaqueType<(unsig� ||      3.8 |           55393 |         1 |  55393.0 |  55393.0 |    55393 |    55393 |         0.0 | void at::native::reduce_kernel<(int)512, (int)1, at::native::ReduceOp<float, at::native::func_wrapp� |+----------+-----------------+-----------+----------+----------+----------+----------+-------------+------------------------------------------------------------------------------------------------------+
import subprocess
subprocess.run([r'C:\Program Files\NVIDIA Corporation\Nsight Compute 2025.4.1\host\target-windows-x64\nsys.exe', 'profile', '--trace=cuda,nvtx', '--sample=none', '--cpuctxsw=none', '--stats=true', '--force-overwrite=true', '--output', 'outputs/tutorial_nsys_fast', r'C:\Software\Miniconda\envs\moni\python.exe', 'examples/gpu_matmul_case.py', '--mode', 'batched_bmm', '--device', 'cuda', '--repeats', '128', '--tile-size', '128'], cwd='profiling_tutorial')
subprocess.run([r'C:\Program Files\NVIDIA Corporation\Nsight Compute 2025.4.1\host\target-windows-x64\nsys.exe', 'stats', '--force-export=true', '--report', 'cuda_api_sum,cuda_gpu_kern_sum', '--format', 'table', '--output', '-', 'outputs/tutorial_nsys_fast.nsys-rep'], cwd='profiling_tutorial')

Generating SQLite file D:\_code\mlsys\pj1\profiling_tutorial\outputs\tutorial_nsys_fast.sqlite from D:\_code\mlsys\pj1\profiling_tutorial\outputs\tutorial_nsys_fast.nsys-repProcessing [D:\_code\mlsys\pj1\profiling_tutorial\outputs\tutorial_nsys_fast.sqlite] with [C:\Program Files\NVIDIA Corporation\Nsight Compute 2025.4.1\host\target-windows-x64\reports\cuda_api_sum.py]...  ** CUDA API Summary (cuda_api_sum):+----------+-----------------+-----------+-----------+-----------+----------+----------+-------------+-----------------------------------------+| Time (%) | Total Time (ns) | Num Calls | Avg (ns)  | Med (ns)  | Min (ns) | Max (ns) | StdDev (ns) |                  Name                   |+----------+-----------------+-----------+-----------+-----------+----------+----------+-------------+-----------------------------------------+|     60.6 |        27202311 |         6 | 4533718.5 | 3789019.5 |   698312 | 12725464 |   4354534.3 | cuLibraryLoadData                       ||     31.4 |        14110499 |        16 |  881906.2 |   24798.5 |     7503 | 11320185 |   2844561.8 | cudaLaunchKernel                        ||      2.5 |         1125637 |         5 |  225127.4 |   29047.0 |    12727 |  1017263 |    443110.4 | cudaDeviceSynchronize                   ||      2.1 |          929287 |         6 |  154881.2 |  139678.0 |     5649 |   386457 |    133083.0 | cudaMalloc                              ||      1.4 |          636628 |         1 |  636628.0 |  636628.0 |   636628 |   636628 |         0.0 | cudaHostAlloc                           ||      1.0 |          455117 |         1 |  455117.0 |  455117.0 |   455117 |   455117 |         0.0 | cudaStreamSynchronize                   ||      0.3 |          116375 |        27 |    4310.2 |     479.0 |      176 |    77258 |     14886.2 | cudaStreamIsCapturing_v10000            ||      0.2 |           74355 |       467 |     159.2 |     111.0 |       70 |     2437 |       177.8 | cuGetProcAddress_v2                     ||      0.2 |           68585 |         6 |   11430.8 |   11097.5 |     5239 |    19465 |      6015.3 | cuLibraryGetKernel                      ||      0.1 |           48342 |         1 |   48342.0 |   48342.0 |    48342 |    48342 |         0.0 | cudaMemcpyAsync                         ||      0.1 |           33514 |        18 |    1861.9 |     524.5 |      309 |    11433 |      3148.2 | cudaEventCreateWithFlags                ||      0.1 |           33404 |         1 |   33404.0 |   33404.0 |    33404 |    33404 |         0.0 | cudaMemsetAsync                         ||      0.1 |           28215 |         1 |   28215.0 |   28215.0 |    28215 |    28215 |         0.0 | cuCtxSynchronize                        ||      0.0 |           12118 |         2 |    6059.0 |    6059.0 |      249 |    11869 |      8216.6 | cuModuleGetLoadingMode                  ||      0.0 |           10410 |         1 |   10410.0 |   10410.0 |    10410 |    10410 |         0.0 | cudaFree                                ||      0.0 |            5510 |         3 |    1836.7 |    1515.0 |      607 |     3388 |      1418.1 | cuInit                                  ||      0.0 |            4995 |         2 |    2497.5 |    2497.5 |      732 |     4263 |      2496.8 | cudaEventCreate                         ||      0.0 |            4029 |        16 |     251.8 |     181.5 |      101 |      632 |       168.4 | cuKernelGetName                         ||      0.0 |            2999 |         2 |    1499.5 |    1499.5 |      890 |     2109 |       862.0 | cudaGetDriverEntryPointByVersion_v12050 ||      0.0 |            2256 |         2 |    1128.0 |    1128.0 |     1032 |     1224 |       135.8 | cuGetProcAddress                        ||      0.0 |            2054 |         3 |     684.7 |     746.0 |      226 |     1082 |       431.3 | cudaGetDeviceProperties_v12000          ||      0.0 |             803 |         2 |     401.5 |     401.5 |      369 |      434 |        46.0 | cuDeviceGetLuid                         |+----------+-----------------+-----------+-----------+-----------+----------+----------+-------------+-----------------------------------------+Processing [D:\_code\mlsys\pj1\profiling_tutorial\outputs\tutorial_nsys_fast.sqlite] with [C:\Program Files\NVIDIA Corporation\Nsight Compute 2025.4.1\host\target-windows-x64\reports\cuda_gpu_kern_sum.py]...  ** CUDA GPU Kernel Summary (cuda_gpu_kern_sum):+----------+-----------------+-----------+----------+----------+----------+----------+-------------+------------------------------------------------------------------------------------------------------+| Time (%) | Total Time (ns) | Instances | Avg (ns) | Med (ns) | Min (ns) | Max (ns) | StdDev (ns) |                                                 Name                                                 |+----------+-----------------+-----------+----------+----------+----------+----------+-------------+------------------------------------------------------------------------------------------------------+|     58.6 |          314192 |         5 |  62838.4 |  23327.0 |    23231 |   220245 |     87993.7 | ampere_sgemm_128x128_tn                                                                              ||     31.1 |          166489 |        10 |  16648.9 |   7615.5 |     7583 |    53598 |     18965.6 | void at::native::<unnamed>::distribution_elementwise_grid_stride_kernel<float, (int)4, void at::nat� ||     10.3 |           55357 |         1 |  55357.0 |  55357.0 |    55357 |    55357 |         0.0 | void at::native::reduce_kernel<(int)512, (int)1, at::native::ReduceOp<float, at::native::func_wrapp� |+----------+-----------------+-----------+----------+----------+----------+----------+-------------+------------------------------------------------------------------------------------------------------+

How to read `nsys`

如果慢版里 launch/sync 很多，而优化版里明显下降，那么说明主要收益来自程序组织方式，而不是 kernel 微观优化。

import subprocess
subprocess.run([r'C:\Program Files\NVIDIA Corporation\Nsight Compute 2025.4.1\target\windows-desktop-win7-x64\ncu.exe', '--target-processes', 'all', '--set', 'basic', '--kernel-name', 'regex:.*(gemm|mm).*', '--launch-count', '1', '--force-overwrite', '--export', 'outputs/tutorial_ncu_slow', r'C:\Software\Miniconda\envs\moni\python.exe', 'examples/gpu_matmul_case.py', '--mode', 'slow_loop_sync', '--device', 'cuda', '--repeats', '128', '--tile-size', '128'], cwd='profiling_tutorial')
subprocess.run([r'C:\Program Files\NVIDIA Corporation\Nsight Compute 2025.4.1\target\windows-desktop-win7-x64\ncu.exe', '--import', 'outputs/tutorial_ncu_slow.ncu-rep', '--page', 'details', '--print-summary', 'per-kernel'], cwd='profiling_tutorial')

[23100] python.exe@127.0.0.1  ampere_sgemm_32x32_sliced1x4_tn (4, 4, 1)x(128, 1, 1), Device 0, CC 8.6, Invocations 1    Section: GPU Speed Of Light Throughput    ----------------------- ----------- --------- --------- ---------    Metric Name             Metric Unit   Minimum   Maximum   Average    ----------------------- ----------- --------- --------- ---------    DRAM Frequency                  Ghz      5.46      5.46      5.46    SM Frequency                    Ghz      1.21      1.21      1.21    Elapsed Cycles                cycle 11,384.00 11,384.00 11,384.00    Memory Throughput                 %     21.05     21.05     21.05    DRAM Throughput                   %      9.28      9.28      9.28    Duration                         us      9.38      9.38      9.38    L1/TEX Cache Throughput           %     30.12     30.12     30.12    L2 Cache Throughput               %     11.57     11.57     11.57    SM Active Cycles              cycle  7,896.10  7,896.10  7,896.10    Compute (SM) Throughput           %     18.13     18.13     18.13    ----------------------- ----------- --------- --------- ---------    Section: GPU and Memory Workload Distribution    -------------------------- ----------- ---------- ---------- ----------    Metric Name                Metric Unit    Minimum    Maximum    Average    -------------------------- ----------- ---------- ---------- ----------    Average DRAM Active Cycles       cycle   4,752.00   4,752.00   4,752.00    Total DRAM Elapsed Cycles        cycle 204,800.00 204,800.00 204,800.00    Average L1 Active Cycles         cycle   7,896.10   7,896.10   7,896.10    Total L1 Elapsed Cycles          cycle 225,960.00 225,960.00 225,960.00    Average L2 Active Cycles         cycle   3,566.56   3,566.56   3,566.56    Total L2 Elapsed Cycles          cycle 171,600.00 171,600.00 171,600.00    Average SM Active Cycles         cycle   7,896.10   7,896.10   7,896.10    Total SM Elapsed Cycles          cycle 225,960.00 225,960.00 225,960.00    Average SMSP Active Cycles       cycle   7,753.68   7,753.68   7,753.68    Total SMSP Elapsed Cycles        cycle 903,840.00 903,840.00 903,840.00    -------------------------- ----------- ---------- ---------- ----------    Section: Launch Statistics    -------------------------------- --------------- -------- -------- --------    Metric Name                          Metric Unit  Minimum  Maximum  Average    -------------------------------- --------------- -------- -------- --------    Block Size                                         128.00   128.00   128.00    Grid Size                                           16.00    16.00    16.00    Registers Per Thread             register/thread    86.00    86.00    86.00    Shared Memory Configuration Size           Kbyte   102.40   102.40   102.40    Driver Shared Memory Per Block       Kbyte/block     1.02     1.02     1.02    Dynamic Shared Memory Per Block       byte/block     0.00     0.00     0.00    Static Shared Memory Per Block       Kbyte/block    32.77    32.77    32.77    # SMs                                         SM    20.00    20.00    20.00    Stack Size                                       1,024.00 1,024.00 1,024.00    Threads                                   thread 2,048.00 2,048.00 2,048.00    # TPCs                                              10.00    10.00    10.00    Uses Green Context                                   0.00     0.00     0.00    Waves Per SM                                         0.27     0.27     0.27    -------------------------------- --------------- -------- -------- --------    Section: Occupancy    ------------------------------- ----------- ------- ------- -------    Metric Name                     Metric Unit Minimum Maximum Average    ------------------------------- ----------- ------- ------- -------    Block Limit SM                        block   16.00   16.00   16.00    Block Limit Registers                 block    5.00    5.00    5.00    Block Limit Shared Mem                block    3.00    3.00    3.00    Block Limit Warps                     block   12.00   12.00   12.00    Theoretical Active Warps per SM        warp   12.00   12.00   12.00    Theoretical Occupancy                     %   25.00   25.00   25.00    Achieved Occupancy                        %    8.32    8.32    8.32    Achieved Active Warps Per SM           warp    4.00    4.00    4.00    ------------------------------- ----------- ------- ------- -------  Note: The shown averages are calculated as the arithmetic mean of the metric values after the evaluation of the      metrics for each individual kernel launch.                                                                           If aggregating across varying launch configurations (like shared memory, cache config settings), the arithmetic      mean can be misleading and looking at the individual results is recommended instead.
import subprocess
subprocess.run([r'C:\Program Files\NVIDIA Corporation\Nsight Compute 2025.4.1\target\windows-desktop-win7-x64\ncu.exe', '--target-processes', 'all', '--set', 'basic', '--kernel-name', 'regex:.*(gemm|mm).*', '--launch-count', '1', '--force-overwrite', '--export', 'outputs/tutorial_ncu_fast', r'C:\Software\Miniconda\envs\moni\python.exe', 'examples/gpu_matmul_case.py', '--mode', 'batched_bmm', '--device', 'cuda', '--repeats', '128', '--tile-size', '128'], cwd='profiling_tutorial')
subprocess.run([r'C:\Program Files\NVIDIA Corporation\Nsight Compute 2025.4.1\target\windows-desktop-win7-x64\ncu.exe', '--import', 'outputs/tutorial_ncu_fast.ncu-rep', '--page', 'details', '--print-summary', 'per-kernel'], cwd='profiling_tutorial')

[11064] python.exe@127.0.0.1  ampere_sgemm_128x128_tn (1, 1, 8)x(256, 1, 1), Device 0, CC 8.6, Invocations 1    Section: GPU Speed Of Light Throughput    ----------------------- ----------- --------- --------- ---------    Metric Name             Metric Unit   Minimum   Maximum   Average    ----------------------- ----------- --------- --------- ---------    DRAM Frequency                  Ghz      5.49      5.49      5.49    SM Frequency                    Ghz      1.22      1.22      1.22    Elapsed Cycles                cycle 41,080.00 41,080.00 41,080.00    Memory Throughput                 %     22.87     22.87     22.87    DRAM Throughput                   %     22.87     22.87     22.87    Duration                         us     33.66     33.66     33.66    L1/TEX Cache Throughput           %     41.66     41.66     41.66    L2 Cache Throughput               %      8.24      8.24      8.24    SM Active Cycles              cycle 15,446.80 15,446.80 15,446.80    Compute (SM) Throughput           %     20.49     20.49     20.49    ----------------------- ----------- --------- --------- ---------    Section: GPU and Memory Workload Distribution    -------------------------- ----------- ------------ ------------ ------------    Metric Name                Metric Unit      Minimum      Maximum      Average    -------------------------- ----------- ------------ ------------ ------------    Average DRAM Active Cycles       cycle    42,264.00    42,264.00    42,264.00    Total DRAM Elapsed Cycles        cycle   739,328.00   739,328.00   739,328.00    Average L1 Active Cycles         cycle    15,446.80    15,446.80    15,446.80    Total L1 Elapsed Cycles          cycle   802,320.00   802,320.00   802,320.00    Average L2 Active Cycles         cycle    17,892.25    17,892.25    17,892.25    Total L2 Elapsed Cycles          cycle   619,296.00   619,296.00   619,296.00    Average SM Active Cycles         cycle    15,446.80    15,446.80    15,446.80    Total SM Elapsed Cycles          cycle   802,320.00   802,320.00   802,320.00    Average SMSP Active Cycles       cycle    15,620.40    15,620.40    15,620.40    Total SMSP Elapsed Cycles        cycle 3,209,280.00 3,209,280.00 3,209,280.00    -------------------------- ----------- ------------ ------------ ------------    Section: Launch Statistics    -------------------------------- --------------- -------- -------- --------    Metric Name                          Metric Unit  Minimum  Maximum  Average    -------------------------------- --------------- -------- -------- --------    Block Size                                         256.00   256.00   256.00    Grid Size                                            8.00     8.00     8.00    Registers Per Thread             register/thread   118.00   118.00   118.00    Shared Memory Configuration Size           Kbyte    65.54    65.54    65.54    Driver Shared Memory Per Block       Kbyte/block     1.02     1.02     1.02    Dynamic Shared Memory Per Block       byte/block     0.00     0.00     0.00    Static Shared Memory Per Block       Kbyte/block    16.90    16.90    16.90    # SMs                                         SM    20.00    20.00    20.00    Stack Size                                       1,024.00 1,024.00 1,024.00    Threads                                   thread 2,048.00 2,048.00 2,048.00    # TPCs                                              10.00    10.00    10.00    Uses Green Context                                   0.00     0.00     0.00    Waves Per SM                                         0.20     0.20     0.20    -------------------------------- --------------- -------- -------- --------    Section: Occupancy    ------------------------------- ----------- ------- ------- -------    Metric Name                     Metric Unit Minimum Maximum Average    ------------------------------- ----------- ------- ------- -------    Block Limit SM                        block   16.00   16.00   16.00    Block Limit Registers                 block    2.00    2.00    2.00    Block Limit Shared Mem                block    3.00    3.00    3.00    Block Limit Warps                     block    6.00    6.00    6.00    Theoretical Active Warps per SM        warp   16.00   16.00   16.00    Theoretical Occupancy                     %   33.33   33.33   33.33    Achieved Occupancy                        %   16.65   16.65   16.65    Achieved Active Warps Per SM           warp    7.99    7.99    7.99    ------------------------------- ----------- ------- ------- -------  Note: The shown averages are calculated as the arithmetic mean of the metric values after the evaluation of the      metrics for each individual kernel launch.                                                                           If aggregating across varying launch configurations (like shared memory, cache config settings), the arithmetic      mean can be misleading and looking at the individual results is recommended instead.

How to read `ncu`

如果慢版 kernel 很小、occupancy 很低，而优化版明显放大了工作粒度，那么你已经回答了“为什么 GPU 没吃满”。

Final workflow

专业工程师不是“把所有工具都用一遍”，而是：

1. baseline

2. cProfile 排除 Python 控制层

3. torch.profiler 找到热点 op

4. nsys 证实 launch/sync/timeline 问题

5. ncu 深挖单个 kernel

6. 做优化

7. 重新从低成本工具开始验证优化是否真的有效