Limit memory usage

[andreasnoack]

I believe that current default is to use most of the available memory on a machine because it gives better performance. However, for various reasons, you might want to use less. E.g. you are sharing the machine with other users so allocating all physical memory is not good social behavior.

[shashi]

If you're using a mostly memory-mappable dataset (e.g. very few string fields, mostly numeric or PooledArrays), then the memory usage you are seeing is virtual memory. The OS will take care of evicting pages from memory as and when other users require it.

Will be working with Tanmay to do this at the JuliaDB level soon. MemPool already has this feature, however, we need to fix JuliaData/MemPool.jl#2 and flip the switch.

[StefanKarpinski]

The OS will take care of evicting pages from memory as and when other users require it.

My impression is that in practice this does not work that well so it may well be worth having better application-level support for memory limiting instead of relying on OS virtual memory swapping.

[shashi]

instead of relying on OS virtual memory swapping.

Agree, swapping is not good. However, in the specific case I described (with mmapped data) we are not relying on the OS to swap pages as in write them to disk. We just have a lot of memory mapped data, but use it little by little. OS just needs to remove some stuff that was not used recently. In my experiments so far linux is doing this pretty well.