Big files vs small files

[Sdub76]

I'm working on trying to backup my gmail. Right now I'm using Thunderbird to backup via IMAP to an mbox file, which is one big monolithic 15TB file. The mbox file format is plain concatenated text.

An alternative is to use a program that saves in maildir format. This would result in 200k+ small files.

Which would be more efficient to back up? Presumably since the mbox is concatenated text and not encrypted, large parts of it should be identical when chunked (though I admit I don't really understand how that works). Weigh that against the overhead of scanning 200k files.

Thoughts? I'm running my borg routine against local and remote repos twice daily.

Thanks

[ThomasWaldmann]

Usually dealing with fewer bigger files has better performance than with huge amounts of small files.

In case of borg, of course the boundaries between emails get lost if all emails are concatenated in a single huge file, so it will not create chunks with usually 1 email per chunk (as for maildir format), but at boundaries determined by the chunking algorithm.

As long as you do not shuffle or delete/compact huge amount of emails frequently, I'ld guess mbox is still more efficient and dedup will still work quite ok. Compression might work better for mbox as chunks might be bigger.