fhir_crack() | ncores > 1 takes unclear amount of RAM

[christiangierschner]

when testing v2 with cracking 2600 bundles and various count of cores -> R took different amounts of RAM

e.g. ncores = 12 cracking went from 7,6 GB to 44 GB

i dont know if number of cores is related to amount of RAM taken

but so far performance is great :)

good job!

Christian

[palmjulia]

Hi Christian,

Thanks for submitting this issue. I would actually expect the RAM to increase when more cores are used because I think more or less the entire R session is copied into each fork, so more cores (=forks) -> more memory. The question is if we can find a way to reduce the memory load a bit. The first obvious attempt would be to keep the working space as small as possible when cracking a bundle list, i.e. make sure you don't have any other big objects lying around that you don't need at the moment. When I find the time I'll definetly look into this a bit more. Thanks!

Julia

[palmjulia]

Update: The fork process used by R should do copy on write, so objects that are not touched during cracking should not be a problem.

I'm looking into possible solution from here.