secure_dump should predump in the same folder as the destination

[nouiz]

https://github.com/mila-udem/blocks/blob/master/blocks/serialization.py#L226

This do a dump to a temp file and then move it to the destination. This is safer in case of a crash during the dump. But it cause problem as by default save the tmp file under /tmp. If there is a crash, this will left over file there and would consume space in a shared directory. THis is bad. Also, it is possible that this path don't have enough memory.

Also, the move if they aren't on the same partition will take times and could crash too.

A much better default tmp directory is the same directory as the destination. The user should have checked there is enough space there. He will see the partial dump in case of crash or ctrl-C of the job so would do the clean up. It will prevent a copy between partition. You just need to use a different file name, and appending some (maybe random) letters to the file name of the destination file.

This caused problemes many times already. This is not just theoretical problems.

[bartvm]

I wouldn't append random letters but just do what e.g. Chrome does when downloading, and add a suffix like .dump at the end. This avoids a situation in which the user ends up with many partial dump files taking up lots of space.

[nouiz]

good idea.

On Wed, Aug 17, 2016 at 10:50 AM, Bart van Merriënboer <
[email protected]> wrote:

I wouldn't append random letters but just do what e.g. Chrome does when
downloading, and add a suffix like .dump at the end. This avoids a
situation in which the user ends up with many partial dump files taking up
lots of space.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#1137 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AALC-5GFZ5kCJPGSkStePNz2x4n_ov--ks5qgx-tgaJpZM4JmhnG
.