Extract peak memory usage

[jiange91]

This is indeed a wonderful tool. I am using it to profile memory usage in serverless computing to improve resource elasticity. While the prof file contains much useful information, where can I derive this peak memory usage which can be essential for me to allocate just-as-needed memory for each container.

[itamarst]

Can you explain a bit more what you're trying to do? Like, are you using Fil in production, or are you trying to do something like https://pythonspeed.com/articles/estimating-memory-usage/ where you're building a model of memory usage, and then will just hardcode the formula in production code?

[jiange91]

What I am trying to accomplish is to estimate the peak memory usage of a program, assuming no swap space will be available when this program is deployed so I may not measure maximum resident memory in this case. I think the number "Peak tracked memory usage" that appeared on the webpage is exactly what I need. I tried to add up all occurring sizes in the blocks below but it is slightly bigger than the PEAK size. I wonder where I can extract this piece of information.

[itamarst]

The "peak tracked memory usage" is the number Fil reports that is most relevant... but it has caveats.

1. It fully counts zero-allocated mmaps() and calloc()s (e.g. np.zeros()) even though those may not use any actual physical memory if they remain as zeroes.
2. In some cases it imports NumPy first, and it runs NumPy and a couple other libraries in single-threaded mode, which distorts the memory usage (in part depending on whether you're using BLAS). This and previous point may be improved eventually (see Sparse mmap()s are counted as fully allocated from the start, which can be very misleading #308).
3. It doesn't count memory overhead from Python interpreter itself, since it starts tracking after Python starts.

If you want a really accurate measure, I suggest using peak resident RAM as in the article linked above, make sure you run your tests on machine with enough physical RAM such that it doesn't use much swap, and then add a little more just to be safe.