BackendAlreadyExists: rclone storage base path already exists on emtpy dir

[bjo81]

As discussed with @ThomasWaldmann on IRC who told me to ping @ncw:

borg2 --debug repo-create -e repokey-chacha20-poly1305  --copy-crypt-key  --repo rclone://fsn:borg2store/mail  --from-borg1 --other-repo rclone://sb:mail                  
                                                                                                                                                                                              
using builtin fallback logging configuration                                                                                                                                                  
19 self tests completed in 0.05 seconds                                                                                                                                                       
Starting new HTTP connection (1): 127.0.0.1:35507                                                                                                                                             
http://127.0.0.1:35507 "POST /operations/stat HTTP/11" 200 93                                                                                                                                 
Local Exception                                                                                                                                                                               
                                                                                                                                                                                              
Error:                                                                                                                                                                                        
                                                                                                                                                                                              
BackendAlreadyExists: rclone storage base path already exists: fsn:borg2store/mail/                                                                                                           
                                                                                                                                                                                              
If reporting bugs, please include the following:                                                                                                                                              
                                                                                                                                                                                              
Traceback (most recent call last):                                                             
  File "borg/archiver/__init__.py", line 623, in main                                                                                                                                         
  File "borg/archiver/__init__.py", line 517, in run                                                                                                                                          
  File "borg/archiver/_common.py", line 151, in wrapper                                                                                                                                       
  File "borg/repository.py", line 148, in __enter__                                                                                                                                           
  File "borg/repository.py", line 162, in create                                                                                                                                              
  File "borgstore/store.py", line 65, in create                                                                                                                                               
  File "borgstore/backends/rclone.py", line 163, in create                                                                                                                                    
borgstore.backends.errors.BackendAlreadyExists: rclone storage base path already exists: fsn:borg2store/mail/                                                                                 

But it's empty:

rclone ls fsn:borg2store
root@mail /tmp # 

The bucket was freshly created and it's nothing in there.

[ncw]

The tests for the backends encourage backends to throw the BackendAlreadyExists if the directory for the backend already exists.

In this case the directory is the borg2store bucket.

I'm not sure this is absolutely correct or sensible though - what do you think @ThomasWaldmann ?

Should rclone only raise this error if the destination has something in it? For use with s3, people often create the buckets in advance and then don't give rclone enough permissions to create buckets.

Possible workarounds for you @bjo81

• delete the bucket and let rclone create it - rclone may not have permissions to do this though
• use a subdirectory of the bucket to store the backup fsn:borg2store/mybackup

[bjo81]

@ThomasWaldmann also thought it's related to the existing bucket, but as already quoted:

 rclone storage base path already exists: fsn:borg2store/mail/

and the bucket is empty, so it should be possible to create the "mail" directory in it.

[ThomasWaldmann]

Guess the point of the other backends raising if the base dir exists is because for them (fs, sftp), an existing directory means that "there is already something". A bit more precise would be to only raise if there is an existing non-empty directory I guess.

[ThomasWaldmann]

@bjo81 yeah, we seem to have 2 issues here:

• it should not complain about the existing bucket dir
• it complains also when using a non-existing dir

[ThomasWaldmann]

I am working on fixing the tests, guess it will become something like this:

def test_already_exists(tested_backends, request):
    backend = get_backend_from_fixture(tested_backends, request)
    with backend:
        backend.store("key", b"value")  # make the backend "not empty"
    # the backend must reject (re-)creation if there is already something at that place
    with pytest.raises(BackendAlreadyExists):
        backend.create()

[ncw]

I am working on fixing the tests, guess it will become something like this:

I think that works better, and it fixes the problem with s3 directories not existing unless they have files in.

[ncw]

@ThomasWaldmann give me a ping when you've updated the test and I'll send a PR for the rclone backend to fix.

[ThomasWaldmann]

Fixed by #62.