Cloud/Virtual Files (OneDrive, Dropbox, etc.)

[bionemesis]

It appears that Borg will trigger OneDrive and possibly other software that uses on-demand sync to only download files locally when opened to start downloading all the files as it attempts to access them. Is this something that can be prevented from happening beyond exclusions? Thank you.

[ThomasWaldmann]

You mean your source ("to get backed up") files are on OneDrive?

borg accesses ALL files in the source set for each backup you create.

It needs to stat() them at least (for change detection) and usually also fetches ACLs, xattrs and bsdflags from them (even if file content is unchanged - this can be prevented by not backing up this metadata, see borg create options).

[bionemesis]

No, not exactly. The OneDrive folder is included in the source files. But the issue is that OneDrive technically doesn't download the files until another application or Finder requests them (at least on macOS). So what's happening is that as soon as borg tries to access the files, they start downloading. Windows has similar functionality for OneDrive and others, but I'm not sure if Borg exhibits the same behaviour. We want to deploy borg en masse, and figuring out an exclusion is going to be a challenge. This is because while OneDrive uses a standard folder name, in most cases, OneDrive for Business doesn't. I also expect that Synology Drive will experience a similar issue as they also have an on-demand sync functionality. But that could further complicate things because whereas OneDrive doesn't let you change the root folder name, Synology Drive does. I don't believe I've come across this issue with other commercial Mac backup solutions, like Backblaze, Crashplan, Comet and MSP360, so I suspect there is a way to detect these fake placeholder files.

[bionemesis]

So it looks like this might be the ultimate answer, but I see there has been no update since February. #4972 I've compared a file in the OneDrive folder that has been downloaded locally against one that has not and xaddr shows these two attributes for files that have not been downloaded:

com.apple.metadata:com_apple_backup_excludeItem
com.microsoft.OneDrive.RecallOnOpen

So if there was a way to tell borg to skip files with the com.apple.metadata:com_apple_backup_excludeItem attribute, it would likely fix this for any and all of these virtual files.

[bionemesis]

In the meantime though, how would I go about excluding OneDrive folders? The root folder will always have OneDrive in the name, just "OneDrive" for personal and "OneDrive - Tenant Name" for business ("OneDrive - Github Inc." for example). In most cases, it's located at /Users/User/OneDrive... but it could technically be anywhere under /Users/Users/.

[ThomasWaldmann]

borg does not support including/excluding files based on xattrs.

but you can use patterns on the path/filename (see docs). especially the ! ... pattern ("exclude norecurse") is very careful not going into the excluded folder, not even looking inside. there are also regex patterns and ** wildcards that might be useful.

or put a .nobackup file into folders that should not be backed up. whether listing a folder will still trigger some expensive stuff has to be seen in that case.