You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm currently trying to run inference profiling on a cuda kernel that's launched from pytorch. I am inside a docker container based which has cuda 12.5. I run:
KINETO_DAEMON_INIT_DELAY_S=3 KINETO_USE_DAEMON=True python profiling/profile.py : which runs the kernel I would like to profile in a loop of 100 iterations.
The kernel being profiled is wrapped in the following torch profiler command: with torch.profiler.profiler(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA], record_shapes=True, profile_memory=True, with_stack=True)
When I launch the script, I get the message: INFO:2024-07-25 20:57:39 4278:4278 init.cpp:131] Registering daemon config loader, cpuOnly = 0
I then run dyno gputrace --log-file /home/trace.json which outputs Trace output files will be written to: /home/trace_7605.json
But I never end up seeing the trace files that are written. I am using the latest version of dynolog, i.e. dynolog_0.3.2-0-amd64.deb. Any help on how I could get these traces would be much appreciated.
The text was updated successfully, but these errors were encountered:
@adirajagopal quick note, you don't need to add any code to include torch profiler in your program ("with torch.profiler.profiler(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA], record_shapes=True, profile_memory=True, with_stack=True)")
It can so happen that the torch profiler based trace collection conflicted with dyno based collection. Torch profiler will always take precedence and so you won't get an output trace.
I'm currently trying to run inference profiling on a cuda kernel that's launched from pytorch. I am inside a docker container based which has cuda 12.5. I run:
dynolog --enable_ipc_monitor --enable_gpu_monitor --dcgm_lib_path=/usr/lib/x86_64-linux-gnu/libdcgm.so
KINETO_DAEMON_INIT_DELAY_S=3 KINETO_USE_DAEMON=True python profiling/profile.py : which runs the kernel I would like to profile in a loop of 100 iterations.
The kernel being profiled is wrapped in the following torch profiler command:
with torch.profiler.profiler(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA], record_shapes=True, profile_memory=True, with_stack=True)
When I launch the script, I get the message:
INFO:2024-07-25 20:57:39 4278:4278 init.cpp:131] Registering daemon config loader, cpuOnly = 0
I then run
dyno gputrace --log-file /home/trace.json
which outputsTrace output files will be written to: /home/trace_7605.json
But I never end up seeing the trace files that are written. I am using the latest version of dynolog, i.e. dynolog_0.3.2-0-amd64.deb. Any help on how I could get these traces would be much appreciated.
The text was updated successfully, but these errors were encountered: