What is the format of the peak-memory.prof file ?

[yguel]

Hi,
We are hunting a memory leak on an embedded system. A very small leak less than 512kB of the 17MB RAM used at the start of the system. The leak occurs every one to two hours, it is hard to accelerate in simulation the processus since the leak happens when the system is connected to hardware sensors.
What we would really need is a timestamped list of memory allocations. I do not know if it would require a big work, but our team would be willing to contribute to achieve this goal.
In the mean time our strategy would be to run the code and stop it at several interval ideally every hour and compare the results to identify what are the differences between the peaks (as there is a leak we have only peaks after peaks).

What would be great is

1. to have a brief description of the fields in peak-memory.prof file in order to write some code that automatically display differences
2. to be able to automatically take peak snapshot at regular interval without the need to stop the program
3. to have snapshot made upon a criterion : increase of memory for instance.

What are your thoughts about those ideas ?

[itamarst]

File format

The format of the data is the "folded" input format to the Flamegraph tools, e.g. the original (https://github.com/brendangregg/FlameGraph) or what I'm using (https://github.com/jonhoo/inferno). In particular flamegraph.pl or inferno-flamegraph. The latter is much faster, and has more features these days.

Specifically, it's:

function-1-in-stack;function-2-in-stack;...;function-n-in-stack  <allocated_bytes>
function-1-in-stack;function-2-in-stack;...;function-m-in-stack  <allocated_bytes>
...

Or maybe kilobytes, I forget.

Diffs

One thing that inferno-flamegraph exposes is a diff mode. So given two .prof files you can do a diff: https://docs.rs/inferno/0.10.7/inferno/#differential-flame-graphs

Dumping at intervals

Fil supports dumping on demand when the SIGUSR2 signal is sent to the process. So just call that at intervals (kill -USR2 <your-process-pid>).

[yguel]

Great ! That's exactly what I was looking for.

[itamarst]

Let me know how it goes.