Pseudo multi-threading

[bpozdena]

Hi @abraunegg,

Regarding the feature requests #232 for multi-threading, I'm not sure if the project is in progress or not. But since you mentioned it will require a significant re-write, I wanted to ask what do you think about the idea of just running multiple OneDrive client instances for the same account, each syncing only specific directories.

I performed a quick test by running three OneDrive monitoring instances and it appears to work flawlessly.

Console 1
onedrive --monitor --single-directory 'large_files/folder1' --resync --resync-auth -v

Console 2
onedrive --monitor --single-directory 'large_files/folder2' --resync --resync-auth -v

Console 3
onedrive --monitor --single-directory 'large_files/folder3' --resync --resync-auth -v

Since I did not see any issues with the SQLite DB lockups or file sync issues, it almost seems like you have designed the client with this use case in mind :-).

My thinking was to implement this pseudo-multi-threading into OneDriveGUI as it already supports multi-threading with multi-account setups. It would theoretically not require that much work to add multi-threading for the same account.

Assuming I ensure there is no directory overlap between the instances, do you see any potential issues that would cause data-loss/sync issues/DB corruptions/etc ?

In theory, the only thing I would need from the client is to somehow obtain and dump the directory structure from the user's OneDrive storage. And/or perhaps add a new option to load OneDrive client with specific sync_list file e.i. onedrive -monitor --sync-list sync_list1, onedrive -monitor --sync-list sync_list2, where each sync list would have different directories listed.

[abraunegg]

@bpozdena
Can you please check the client version you are running - as you specifically should not be able to run 2+ instances of the client at the same time using the same configuration directory .. it causes database sync issues and all sorts of other problems.

[bpozdena]

OK, thanks. I run v2.4.20 installed from AUR on Manjaro.

[abraunegg]

@bpozdena
The issue will be compounded by your use of --resync ... stop using resync unless the application advises it.

What that option is doing, at startup, is removing the existing cache file - so for each instace, you are removing the file.

This means that for 2 of those instances - the cache file will not contain the data you think it will contain.

Because of this - it is masking the db protections.

Please redo your testing without resync.

[bpozdena]

Yes, without --resync , I get The database is currently locked by another process - cannot sync.

Anyways, if you don't recommend this use case even for initial 'burst sync', you can label this as answered. I was just looking for a way to quickly finish my initial OneDrive sync as I have over 60k of files there..

Thanks as always!

[abraunegg]

@bpozdena
I do not condone or endorse your process.

I was just looking for a way to quickly finish my initial OneDrive sync as I have over 60k of files there..

The process you are currently using - your local database file will be broken ... and not contain the data you think it contains ... thus - you may even get into a potential data loss scenario .... please be VERY careful.

What you have highlighted is also a gap in the protections - where by using --resync effectively removes the protections put in place ... this needs to be fixed.

Update:
This gap is closed by #2067