remove ad-hoc mkdir calls, fixes #46

[ThomasWaldmann]

Store: require levels configuration

No levels == [0] default anymore, we need all namespaces to be configured so Store.create_levels is able to pre-create all required directories.

[ThomasWaldmann]

@ncw I've seen that the rclone backend does not do any internal mkdir calls except maybe for the base path.

That was different for posixfs and sftp, they previously always ad-hoc needed to make sure that all namespace and nesting directories are present when they created a new object in the store.

These are now all created when the store is created. For example, borg uses 2 nesting layers, so that are ~64k mkdir calls once at create time for the sake of not having to care for these again while using the store.

Shall I add a backend instance flag like backend.mkdir_needed and set that to False for rclone?

[ncw]

In rclone, store will create intermediate directories when needed (not all backends need this, eg S3).

So yes a backend instance flag sounds like a good idea.

[ThomasWaldmann]

Can rclone be told to not do ad-hoc directory creation to optimize performance?

[ThomasWaldmann]

Put this into 0.1.0 milestone, because borg2 b11 does not yet give a complete levels configuration, e.g. the "locks/" namespace is missing.

[ncw]

You can create the directories in advance and rclone will use them. The directory creations are a noop on S3 like backends.

For non S3 backends creating the directories in advance will probably make things quicker.

For s3 like backends (rclone calls these bucket based backends) it will make no difference. You'll waste a bit of time doing noops.

So maybe the best course of action is to create the directories in advance always, just don't expect them to actually be there as they won't be for s3 and friends.

Note that creating lots of directories is likely to be quite slow (on non S3).