Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[NOTICE] Version 3.30.1 Causing Sync Issues / Synchronising All Files #13872

Open
ZetaTom opened this issue Oct 24, 2024 · 88 comments
Open

[NOTICE] Version 3.30.1 Causing Sync Issues / Synchronising All Files #13872

ZetaTom opened this issue Oct 24, 2024 · 88 comments
Labels
feature: sync meta Issues about repository, contribution, development stable-3.30

Comments

@ZetaTom
Copy link
Collaborator

ZetaTom commented Oct 24, 2024

Nextcloud Android Client Version 3.30.1

Dear Community,

It is with great regret that we have to inform you that a recent version of the Nextcloud Android Client, specifically version 3.30.1, has introduced a bug that may cause one or more of the following issues:

  • increased battery usage and device temperature,
  • increased mobile and non-mobile data usage,
  • increased storage usage, and
  • synchronisation of files not previously selected for synchronisation.

Please note that manual intervention will be required by users who have upgraded to the affected version.

Updating - How To Disable Two-Way Synchronisation

We have carefully investigated this issue and released new versions which aim to address these issues.
With Version 3.30.4 we have introduced a new way to manage two-way sync. To access this redesigned menu open SettingsInternal two way sync.

Image Image

This menu allows users to globally enable or disable two-way synchronisation. Disabling this will ensure that no two-way synchronisation will occur in the future. As before, users have the option to selectively remove individual folders using the × icon next to the folder name. For advanced users, there is also an option to set the interval at which the Nextcloud Android Client will attempt two-way synchronisation.

We have also added a button to disable two-way synchronisation for all folders. This option will appear in the overflow menu (⋮) if two-way synchronisation is enabled for at least one folder. Furthermore, this will attempt to cancel any transfers in progress and update the database to exclude those folders from being synchronised again.

Removing Local Files

Some users have experienced increased storage usage on their device. Allocated storage space can be freed by removing local files: open the On device tab and long press on any file or folder. Then open the overflow menu (⋮) to Select all files. When all files are selected, re-open the overflow menu and select Delete. When prompted, select Local only. This is important, as selecting Delete in this dialogue will also remove all selected files from the server. The removal process will take some time depending on how many files have been downloaded.

How to Upgrade

The Nextcloud Android Client can be updated via your preferred installation method. All versions are also available on GitHub for download.

As of 05/11/2024, version 3.30.4 is available from F-Droid. This version includes all major fixes and the redesigned management interface.

Unfortunately, the Google Play release will take some more time to be available as we are currently blocked by Google. This is, however, tracked in a different issue: #13871.

We will update this issue as soon as the releases are available.

Mitigation

We strongly recommend that all users to upgrade to version 3.30.4 or later using your preferred method to resolve these issues. This version includes several fixes and changes that should resolve any issues related to two-way synchronisation. Please refer to the explanation above for information. If this version is not available for you, or if something else is blocking this update, there are several options to mitigate any unwanted behaviour.

Option 1 - Manual Removal

From version 3.30.2, users have the option to quickly check and stop two-way synchronisation for specific folders. To access this menu open SettingsInternal two way sync. On the following screen one will see all folders scheduled for two-way synchronisation and have the option to exclude folders using the × icon next to the folder name. If a blank page is displayed instead, no folders are scheduled for two-way synchronisation.

Two-Way Sync Settings

Option 2 - Reinstalling

For version 3.30.1 to 3.30.3 (outdated)

The simplest option is to reinstall the Nextcloud Android Client. Due to the technical nature of the issue reinstalling any version above 3.30.0 will resolve the issue. The process of reinstalling an app varies from Android version to Android version. However, the general process will be similar to the following:

  1. Close any open app and open the app drawer
  2. Press and hold on the Nextcloud app
  3. Select App info
  4. Select Uninstall
  5. Confirm with Ok
  6. Open App Centre of choice
    • Depending on your vendor, this will have different names, such as Google Play Store, App Gallery, F-Droid, etc.
  7. Download and install Nextcloud Android Client

Please note that any local data of the Nextcloud Android Client that is present on the mobile device will be lost during the removal process.

Option 3 - Reinitialise Local Account

For version 3.30.1 to 3.30.3 (outdated)

Another option, if you prefer to keep the current version installed, is to remove the current user account and it add again. This should have a similar effect to reinstalling the application. To re-initialise your account in the Nextcloud Android Client, follow these steps:

  1. Open Nextcloud app
  2. Open user selector (your profile picture on the top right)
  3. Select Manage accounts
  4. Open the overflow menu (⋮) for the affected account
  5. Select Remove account
  6. Confirm by choosing Remove local account and pressing the Remove account button
  7. Re-add your account by selecting Add account
  8. Finish account set-up as normal

Please note that any local data of the Nextcloud Android Client that is present on the mobile device will be lost during the removal process.

General - Enabling Data Limit and Warning

Many mobile network operators charge their subscribers an additional fee when they exceed their data allowance. To avoid these charges, most versions of Android offer some kind of safety measures. These include a Data Limit and Data Warning. The latter notifies you once the specified amount of mobile data has been used up. The former allows you to set a hard limit after which Android will disable mobile data and prevent you from using any more. Also note that you will not be able to use any apps that require an active data connection, such as Internet messengers.

Settings to enable and configure the Data Limit and Warnings can usually be found in the device settings.

  1. Open Settings app
  2. Select Network and Internet
  3. Select Internet
  4. Select ⚙ next to your mobile (data) carrier
  5. Select Data warning and Data limit

Related Issues

Technical Background

Internally, the Nextcloud Android Client uses a database table to keep track of all files. This table has been extended by two columns to accommodate the new two-way sync feature: internal_two_way_sync_timestamp and internal_two_way_sync_result. The latter of these columns is not relevant for the purposes of this investigation.

We use the timestamp to keep track of two things. Firstly, the date when the corresponding file was last synchronised. Secondly, whether the file should be synchronised at all. If the timestamp value of a file contains a valid timestamp greater than 0 the file will be scheduled for synchronisation. In this case 0 means the file has never been synchronised. A value of -1 indicates that a file should not be synchronised at all.

On a clean install of the Nextcloud client, all records in the filelist table will be initialised with a timestamp value of -1. When upgrading from an older version, the database is migrated to fit the new scheme. During this migration, the timestamp column is added to the table and initialised with a default value of null. This must be handled carefully when interpreting database values, as improperly handled null values are known to cause a variety of problems. In this case, null represents unknown; the file has never been synchronised and the user hasn't made a decision about whether or not it should be synchronised.

In a recent update, a change was made to mitigate null pointer exceptions during file synchronisation. Due to a small oversight, this also meant that all null values would be interpreted as 0. While this fixed the null pointer issues, it unfortunately introduced a new problem.

When a user updates to an affected version, the column is filled with null, which is interpreted as 0. After a while, a background worker will look up all files that need to be synchronised in the aforementioned table. Since all files have a 0 timestamp, it will try to synchronise them all.

For technical reasons, we also store the root folder, which contains all other files, in the same database. This folder is represented by a record having / as its filename. Therefore, all folders and files are scheduled for synchronisation.

As it is normally not possible to select the root folder for two-way sync, we have excluded it from the synchronisation process in the patched versions.

Due to the nature of our apps and release channels, rollbacks to previous versions are not possible. Instead, we have released newer versions that fix these issues.

Version History

  • 3.30.1 released on 11/10/2024
    • introduced two-way sync
    • upgrading to this version will cause the app to malfunction
  • 3.30.2 released on 21/10/2024
    • first patch to address two-way sync issues
    • fixes null conversion
    • includes UI to remove folders from two-way sync
    • includes an additional check to stop unwanted background synchronisation of root folder
  • 3.30.3 released on 22/10/2024
    • second patch to address two-way sync issues
    • two-way sync will now only run when connected to Wi-Fi
  • 3.30.4 released on 31/10/2024
    • third patch to address two-way sync issues
    • further improvements to two-way sync logic
    • includes improved UI to manage two-way sync
    • includes global toggle to disable two-way sync
    • includes button to stop all scheduled two-way sync operations and remove all folders from two-way sync
@ZetaTom ZetaTom pinned this issue Oct 24, 2024
@nextcloud nextcloud locked and limited conversation to collaborators Oct 24, 2024
@nextcloud nextcloud unlocked this conversation Oct 24, 2024
@noci2012
Copy link

noci2012 commented Oct 24, 2024

The github versions don't install on devices that install from F-Droid / Gplay / ?.

Older versions 3.29.1 available as APK on F-Droid does NOT install citing invalid package ....
So without uninstall no remedy.? Gplay doesn't offer a downgrade AFAICT

@ZetaTom
Copy link
Collaborator Author

ZetaTom commented Oct 24, 2024

@noci2012, F-Droid builds are signed by F-Droid. Builds published on GitHub are signed using our key. This means that one must, unfortunately, choose and stay with one source for continued updates.

In some cases, the versions differ in some aspects due to some other constraints, e.g. Google Libraries. This is the case for Google Play and F-Droid builds.

Downgrading is not officially supported, as far as I'm aware. To install an older version, one should first remove the newer one.

@bessman
Copy link

bessman commented Oct 24, 2024

Technical Background

I'm curious why the same files were repeatedly synchronized ad infinitum? I don't think the Technical Background currently addresses this.

In my case, Nextcloud downloaded 15 GB of data from a server containing 1 GB of data. Others have reported the same.

@zukunft
Copy link

zukunft commented Oct 24, 2024

I see in the F-Droid build log:

com.nextcloud.client:30300290 2024-10-22 15:20:18,028 INFO: Building version 3.30.2 (30300290) of com.nextcloud.client .
...
BUILD SUCCESSFUL in 13m 56s 56 actionable tasks: 56 executed 2024-10-22 15:36:53,598

It seems that it takes 3 to 4 days until the build is distributed. Maybe we should help F-Droid to speed up the distribution process. I some knows more, please help out.

@zukunft
Copy link

zukunft commented Oct 24, 2024

I'm curious why the same files were repeatedly synchronized ad infinitum?

I don't think so, because I have version 3.30.1 at the moment on LineageOS and it sync all files once (using almost all the battery), but at the moment it is fine. I checked it on another phone and it seems, that it sync the same file several times, but not ad infinitum. (Just my personal experience with 3 clients, so nothing to really relay on)

Possible that the repeating sync start is caused by a timeout. With a lower server space usage, the sync seems to be repeated less often.

@joshtrichards joshtrichards added bug meta Issues about repository, contribution, development stable-3.30 and removed bug labels Oct 24, 2024
@szotsaki
Copy link

Mitigation (i.e. reinstall, but explained really verbosely) is for technical users only.

Many users don't know their password or what a password is (an example from my circles where they just want to see pictures by swiping). For them re-logging into the app via a computer with a QR code through an obscure web UI path is impossible.

You littered every single user's phone with gigabytes of difficult-to-remove data. This is a major mess. Especially, that is comes just months after another major synchronisation problem.

I already received complaints that Nextcloud duplicates all files into its cache and filling up the phone eventually (details in #13400 (comment)).

Please, create a proper, automated solution for this problem ASAP. Otherwise you offload solving this chaos to every single technical user who chose and distributed your application.

@tobiasKaminsky
Copy link
Member

You littered every single user's phone with gigabytes of difficult-to-remove data.

Data can be simply removed via "on device" and then removing the folder that you do not want, choosing "local only".

Please, create a proper, automated solution for this problem ASAP.

I am all for it, but do you have an idea how?
Internally the synced folders are stored in a database and we cannot differentiate between wanted and wrongly added folders there.
This is why the new overview allows user to also cancel 2way sync, but only manually.

@szotsaki
Copy link

In the short term an "Empty local storage" button would be a good interim solution. I can communicate to press this button and this problem goes away. This would also conveniently fix #9128, too.

As a next step I could imagine a Trashbin-like system which, after a configurable amount of used local storage (e.g. 2% of device storage), it'd warn the user to press this button.

One more step further, the application could be re-architected in a way that it outsources the internal storage handling into a regular Linux directory which the user can manage themselves. This would fix #13400. Even if you agree with this direction, I see this would have many edge cases and is not a quick solution.

@ltguillaume
Copy link

ltguillaume commented Oct 25, 2024

Internally the synced folders are stored in a database and we cannot differentiate between wanted and wrongly added folders there. This is why the new overview allows user to also cancel 2way sync, but only manually.

Back when this issue arose, it could be safely assumed that no one started using this feature just yet.

Even if they did, it would have been a far more decent solution to have said "sorry, we made an error, and to solve it for everyone automatically, we have reset the two-way sync feature and removed the local copies. For those 2 people who were using it already, please set up the two-way sync again".

Notifications like that should also be possible within the app. You could then also use this to inform people of new features.

@Alifoss
Copy link

Alifoss commented Oct 25, 2024

#13738 (comment)

@mase76
Copy link

mase76 commented Oct 27, 2024

Even when not in Wifi, 3.30.2 is syncing the same files over and over again, eating my monthly mobile data completely within a few hours. Removing the files from the sync did not stop it. I had to delete the file database to stop the sync. The app seems to be completely broken now.

@timkrief
Copy link
Member

timkrief commented Oct 27, 2024

Hi, I just updated to 3.30.2 from f-droid and I saw "check listed folders in settings -> internal two sync", so I went and checked, the app took a really long time and then showed me a list of a LOT of folders, I could swipe for minutes (EDIT: it's so long I did not reach the bottom yet, I think this is just a list of all folders...). It says "Pas encore, bientôt synchronisé" (my app is in French, it means, "not yet, soon synchronized") for each folder. I don't know what that means, I don't know when it will do that and how to avoid it (EDIT: It started syncing after an hour or so after the update). I tried to press on the cross but it just make the app not responding (EDIT: it takes roughly 1 min 30 to get the app responding again) and the time it would take to press that cross for each folder even if it was working would be insane.

I'm scared by the idea of a two way sync being triggered on data I never wanted to two way sync. Apart from the issue of my phone storage being filled, my data is important and I don't want data from my phone to corrupt data on my server. I never selected those folders and I hope they won't sync, but I don't know what "not yet, soon synchronized" means.

What should we do? Is there a way to prevent the app from attempting any sync for the time being, so that the issue can be fixed? I like being able to sync my photos to nextcloud and being able to sync some files manually, but I didn't know auto two way sync was a thing on android, is there a way to disable automatic two way sync?

EDIT: For context, in my case none of the folders were wrongly synced (yet?), my only issue is that all my folders are listed under "settings -> internal two sync" and marked as "Pas encore, bientôt synchronisé"/"not yet, soon synchronized" and I don't want them to sync if they have no previous version locally (which would mean that they are not supposed to be synced)

EDIT 2: I waited after tapping on a cross on one of the folder, I did it twice. It took 1 min 30 sec to register. I don't know how many folders and files there are, but if I had to press them one by one and wait for 1 min 30 sec each time, it would take days (Edit: 10 full days).

EDIT 3: That's it, random files started to be downloaded. I put my phone into airplane mode from now on... I might have to reinstall and re-setup everything, which would mean a lot of work since tens of Go of files were synced and would have to be re-downloaded (mainly auto synced photos and videos) and other apps use identification from the main nextcloud app. I have multiple users hit with the bug. I'm going to wait a bit in the hope that we'll have a button to just remove all files that don't have any local version from being synced.

EDIT 4: Thanks to a notification, I now know that 15000 items are queued for sync, and it takes a full minute for each item to be removed (I'm using a samsung galaxy s10 with the latest update available) which would take 10 full days with an autoclicker. Since there is no button to juste remove them all in one go, I have no other option but to reinstall the app, reconfigure everything and download all the files I had again. I was able to remove all the items with an autoclicker on a galaxy s21, the removal on that device of each item in "two way sync" was almost instantaneous. Most user won't setup an autoclicker, and I expect that removing all items by hand would have taken half an hour at least. Even in that case a single button to remove everything would be necessary.

TL;DR: Please provide a single button to remove all files that don't have any local version yet from the "internal two sync", if you have just been hit by this bug, a button like this would stop any further damage without removing properly synced files. If that's too complicated, just a button to remove all items from "internal two sync" in one tap.

@ltguillaume
Copy link

@timkrief There's only one proper solution: note down any automatic upload configurations you have, then uninstall and reinstall.

@timkrief
Copy link
Member

@ltguillaume I finally did it and I'm in the process of re-syncing all the files that were (intentionally) synced before. It reveals a lot of flaws with the sync system altogether, but it's not related to this issue. I'm still settings things up, but it seems to work for the time being, the issue of having all the folders added to two way sync is not happening anymore, but if I do add things to the two way sync system, they will sync again and again even with no modification, so I do have to make sure there's nothing in the two way sync page.

I would like to say that when going to the "two way sync" settings to see if there is no issue, it's quite troubling to have a completely blank screen. When I had the issue, I had a blank screen for a minute before all 15000 folders appeared in the list, so I couldn't tell easily if the issue was still there or if everything was fine. Many user go to that screen and think this is a bug. I think it should be a priority to add at least a single line "no two way sync folder yet" or something to make sure the experience is smooth. see for instance:
#13738

and:

This is especially confusing because the fix notes for recent critical bugs explicitly say to check the two way sync configuration, so many users like me are being directed to this screen for the first time.

Originally posted by @danepowell in #13693

@ltguillaume
Copy link

if I do add things to the two way sync system, they will sync again and again even with no modification, so I do have to make sure there's nothing in the two way sync page.

This still happens with the new (cleanly installed) version? Wow, I didn't expect that and I don't know if the devs are aware of this. This should be addressed in a new issue!

I would like to say that when going to the "two way sync" settings to see if there is no issue, it's quite troubling to have a completely blank screen.

Agreed, this is a UX 101 violation 😛

@JeroenVreeken
Copy link

IMHO only half of the bug is fixed at best. The 2-way sync seems to completely ignore my wifi-only and charging-only settings and ate up my data-plan. This does not seem to be addressed in the 3.30.2 version at all!
Syncing forever is not the only problem, the fact that it syncs over a connection it is not supposed to use is an even bigger problem.

@ltguillaume
Copy link

ltguillaume commented Oct 28, 2024

The 2-way sync seems to completely ignore my wifi-only and charging-only settings

I may be missing something, but are there even settings for this? To my knowledge, those settings are only implemented for auto upload, not for two way sync. Apparently 3.30.3 does force two way sync to only be done via Wi-Fi, but if you're running Nextcloud via F-Droid, then you haven't received that update yet.

@mase76
Copy link

mase76 commented Oct 28, 2024

The sync of the same files over and over again completely ate my monthly mobile data quota. Yes, we have a quota in Germany.
Image

@AHeinlein
Copy link

I have been following this issue for some time now. Fortunately I read about it before the automatic upgrade from F-Droid triggered, so I am now still at 3.29.1 and waiting until it is safe to upgrade. But I am confused whether this is already the case (or will be once 3.30.3 hits the F-Droid store). Has anybody tested if the upgrade now works as intended, or are we assuming everyone has already been hit?

@Zocker1999NET
Copy link

To the Nextcloud developers, with all the information I have, this bug is/was essentially a Denial of Service "attack" across most users of Nextcloud Android. A bug can happen, as a dev myself I have full understanding how that can happen. But IMO, the correct response to a bug with this severity is to disable this feature fully, temporarily, for all users by immediately rolling back to 3.29.1.

Why do I consider this bug a servere Denial of Service:

  • after an probably automatic app update, the app syncs all files, by itself & against the will of most users
  • filling up the storage of the users on their phone, making them potentially unusable
  • using a huge amount resources on Android
    • in my case comparable to running a 3D game, but in the background
    • using up valuable battery
    • creating a large amount of lag & heat visible to the user
  • using up valuable data from cellular data plans
    • effectively taking the ability to use cellular data for other purposes
    • sometimes directly incurring costs to because of "features" like "Datenautomatik" (after using up your data plan, you will automatically get 100 MiB extra for just 5 €, up to three times, before the data usage is limited to 350 kbit/s)
  • given that, so far as I understand this bug, seems able to affect all users of the app, without further restrictions, a huge amount of users might be affected
    • in the context of an organization using Nextcloud, this will overload the IT support

So this necessarily includes a large amount of "normal" users which (like others stated before):

  • will probably not no what happens to their phone
  • will probably just remove Nextcloud from their phones (if they are able to detect it as the culprit)
  • will most probably not read any of the mitigations which were, at least now & to my knowledge, only communicated through GitHub ("normal" users should not required to use GitHub)
  • even so, they might not be able (or willing) to execute those mitigation steps you posted

And I know, these upcoming points are not something you could change, but are still true:

  • most Nextcloud Android installations are from an app store (Google Play, F-Droid, …) with automatic updates enabled
  • administrators of Nextcloud instances have, at least in many cases, no way of controlling the Nextcloud Android app version their users are using
    • (ignoring cases where users use their private phones instead of provided MDM-phones)

And new reports seem to suggest that this new internal feature can still cause issues even through it is expected to be fixed. This brings me to the conclusion that IMO, for now, there is one thing to do now, before trying to fix this bug: ASAP release a new version based on 3.29.1 (before 2way sync) with a notification explaining to "normal" users what happened & (if so) what they need to do now (can of course link to this GH issue for the occasional nerd).

Also, but this can be deferred, it would be nice if a next version could delete the files the app created while syncing. However, I can understand that this might be a problematic step to take if users might have modified those files in the mean time.

(For clarification: I suggest reactions with 👍/👎 are referring to the idea of the immediate roll back to 3.29.1 & not for other points I made, so this can be used as a poll.)

@ltguillaume
Copy link

Could you please please please just release a new stable version with everything two way sync disabled completely, then start building up this feature properly in the dev builds?

@joe-average-user
Copy link

Update: It did NOT work in fact!

Files are starting to sync again even though I went through and canceled them! I'm watching them (in one folder) get re-added to the queue automatically. And I can't find any way to stop it.

Well, simply believe me. Whatever you read about "solutions", the only thing that does work is removing the app and re-installing, then disable 2.-way, and then add your syncs again.
The ''2-way"' sync options is not the real cause of the problem. They seem to have broken the database-handling, which means they broke the update-path. Re-installing from scratch does not show the problem. Since the version is out now for some time there is probably no way back.
This is not the only place where nextcloud (and accompanying tools) has problems with the database handling. It is obvious there is no QA.

@rugk
Copy link

rugk commented Nov 19, 2024

I use v3.30.4 and I've just encountered this issue (AFAIK) for the first time. So it seems not really fixed at all.
STR: IIRC, I just used KeePassDX to synchronize my (kbdx) file/DB, which I first downloaded (or so, when unlocking) and then pressed "Syncronize" in KeePassDX, which AFAIK uses the Android file provider to download the file, merge it, and upload it again. (aka the way before two-way-sync was implemented)

I was connected to wifi though, so no damage done, but anyway…

@rugk
Copy link

rugk commented Nov 19, 2024

includes button to stop all scheduled two-way sync operations and remove all folders from two-way sync

Where is that?

I've now disabled two-way sync via the button, but it continues to download files…

Edit: Ah I need to enable 2-way-sync, then click on all Xes there to remove them from the queue? I did that, and disabled 2-way-sync again. Aaaand it still sync again.

Hint: At least for CalyxOS you can pause an app entirely, so it won't re-start.

@joe-average-user
Copy link

Edit: Ah I need to enable 2-way-sync, then click on all Xes there to remove them from the queue? I did that, and disabled 2-way-sync again. Aaaand it still sync again.

Hint: At least for CalyxOS you can pause an app entirely, so it won't re-start.

You will have a lot less troubles if you simply believe me it is broken. You can only restore original bahaviour if you follow my above advice.

@tobiasKaminsky
Copy link
Member

Edit: Ah I need to enable 2-way-sync, then click on all Xes there to remove them from the queue? I did that, and disabled 2-way-sync again. Aaaand it still sync again.

Is this about syncing folders or a specific file?
Sync/updating downloaded files is already in the apps since quite a while.

The new 2 way sync is "only" about folders and we enhanced it with 3.30.5, which will be soon in Fdroid release.

@joe-average-user
Copy link

joe-average-user commented Nov 20, 2024

You missed the point. your 2-way-sync implementation broke the sync as a whole. In fact every folder seems to get resynced continously, no matter what direction. I would say you misinterpret the sync-database in some point(field).
This does not happen on a fresh install, only in update path.

Oh, and forgot another important thing: the sync then ignores the setup when it only should sync with WLAN connection, not mobile. This is reported several times here and I can confirm this problem.

@tobiasKaminsky
Copy link
Member

@joe-average-user can you explain in more detail what is broken?
With latest 3.30.5 we tried to fix the sync:

  • all existing background jobs have been cancelled
  • we now only use one background job to sync, which can be enabled/disabled and has a time interval in settings

So after upgrade, it should be enough to once/finally check if all your desired sync folders are in the settings list.

@joe-average-user
Copy link

Well, for understanding the problems, read the thread.
Most - if not all - of us cannot check your latest update regarding the issue, as anyone I know needed an adhoc fix for the problem which only worked by removing the app, re-installing and re-adding the synced folders. Then the problem was gone. So I and probably others cannot tell you if 3.30.5 fixes it as we were not able to await this fix (the bug burned down all your mobile data volume as it disrepected the sync-during-wlan option completely).
Iow, all people constantly updating (by fdroid like me) were burnt...

@flixmart
Copy link

flixmart commented Nov 25, 2024

As per above I was badly having the problems on two devices when updating to 3.30.1 (FDroid). Both where manually "repaired" initially and are online again afair since 3.30.3.

Both devices have only been updated through all versions since (no re-install), one to 3.30.4, the other to 3.30.5 since yesterday. Both have never again fallen back to unwanted syncs.

In the meantime, even the two way sync seems to work as intended.

@tobiasKaminsky
The extended update notice and HowTos (OP of this thread) are helpfull ;-)

Do I understand correctly, that two way sync atm is hardcoded to WLan only, despite already having sync-conditions options for auto-upload? Couldn't two way sync be made to function simillarly?

@AHeinlein
Copy link

I just wanted to give this a try, as I am still on 3.29.2 and have blocked updates on F-Droid since, waiting for this to be resolved. But F-Droid refuses to perform the update, it says the new version has been signed with a different key and that I need to uninstall and reinstall. Basically that means I am forced to do the "fix" the hard way... :-(

@alperozturk96
Copy link
Collaborator

alperozturk96 commented Nov 25, 2024

@flixmart Yes. TwoWaySyncWorker is only working via Wi-Fi for now.

@tobiasKaminsky
Copy link
Member

I just wanted to give this a try, as I am still on 3.29.2 and have blocked updates on F-Droid since, waiting for this to be resolved. But F-Droid refuses to perform the update, it says the new version has been signed with a different key and that I need to uninstall and reinstall. Basically that means I am forced to do the "fix" the hard way... :-(

That is strange.
FDroid is responsible for building and signing it and so far I never saw a problem there (I am also using it).
Can you check on app details from which source it is installed?

@spaceChRiS
Copy link

@flixmart Yes. TwoWaySyncWorker is only working via Wi-Fi for now.

Are you referring to 3.30.5 explicitly or also to earlier revisions? Because with 3.30.4, it used 15 GB of my wife's mobile plan before I deinstalled it after all other countermeasures from this thread had failed.

@mase76
Copy link

mase76 commented Nov 25, 2024

When I put files into a synced folder via filemanager, they are not synced. NC does not know that files.

@alperozturk96
Copy link
Collaborator

alperozturk96 commented Nov 25, 2024

@flixmart Yes. TwoWaySyncWorker is only working via Wi-Fi for now.

Are you referring to 3.30.5 explicitly or also to earlier revisions? Because with 3.30.4, it used 15 GB of my wife's mobile plan before I deinstalled it after all other countermeasures from this thread had failed.

Since version 3.30.2, the Two-Way Sync Worker only uses Wi-Fi. Could the 15GB usage be due to Auto-Upload? Was Auto-Upload enabled with mobile data for some folders?

Additionally, we fixed the re-download issue in v3.30.5. However, if only mobile data is active, the Two-Way Sync Worker does not download anything.

@spaceChRiS
Copy link

Was Auto-Upload enabled with mobile data for some folders?

[...]

However, if only mobile data is active, the Two-Way Sync Worker does not download anything.

Auto upload was active for the photo folder, but only allowed for wifi (and this worked for years now), plus she did not add 15 GB+ of new photos that day. And, I was able to reproduce with 3.30.4 that if I switch off wifi, the unstoppable syncing process kept running. It even re-spawned several times after deleting the app cache and other tries to stop it.

Unfortunately, I had to deinstall and reinstall the app to prevent more damage to my wife's yearly mobile plan¹, therefore I can no longer test this behavior.

¹It would anyway never have succeeded, even if I had kept the phone connected via wifi for several days: The entire nc folder was added to the sync list, way more than 100 GB of data (I wrote 100 GB before, but this was outdated information 😉), which is more than the free space on the device anyway.

@alperozturk96
Copy link
Collaborator

alperozturk96 commented Nov 25, 2024

Was Auto-Upload enabled with mobile data for some folders?

[...]

However, if only mobile data is active, the Two-Way Sync Worker does not download anything.

Auto upload was active for the photo folder, but only allowed for wifi (and this worked for years now), plus she did not add 15 GB+ of new photos that day. And, I was able to reproduce with 3.30.4 that if I switch off wifi, the unstoppable syncing process kept running. It even re-spawned several times after deleting the app cache and other tries to stop it.

Unfortunately, I had to deinstall and reinstall the app to prevent more damage to my wife's yearly mobile plan¹, therefore I can no longer test this behavior.

¹It would anyway never have succeeded, even if I had kept the phone connected via wifi for several days: The entire nc folder was added to the sync list, way more than 100 GB of data (I wrote 100 GB before, but this was outdated information 😉), which is more than the free space on the device anyway.

Thanks for the information. We were aware of the unstoppable syncing issue and addressed it by stopping the scheduled periodic worker at app launch in v3.30.5. We expect this issue to be resolved in v3.30.5.

As for the mobile data usage still occurring in v3.30.4, we will look into it.

@bobemoe
Copy link

bobemoe commented Nov 25, 2024

I was hit by this too right at the beginning and uninstalled after my monthly allowance was consumed! I am watching this thread closely and waiting for it to become stable enough for me to reinstall.

One thing I have read is that sync is now limited to WiFi. I have no WiFi and do all my sync on Ethernet (or mobile). I think this needs to be amended to allow sync on Ethernet too (and a toggle for mobile) or it wont even work for me when I do reinstall? #13847 (review)

@tobiasKaminsky
Copy link
Member

One thing I have read is that sync is now limited to WiFi. I have no WiFi and do all my sync on Ethernet (or mobile). I think this needs to be amended to allow sync on Ethernet too (and a toggle for mobile) or it wont even work for me when I do reinstall? #13847 (review)

Adding an option to also run it on mobile data, is possible.

@joe-average-user
Copy link

Ok,, with 3.30.5 you managed to break the auto-upload alltogether.
Now the status is that some folders get auto-uploaded and others dont. Namely the first folder (Camera) does not get uploaded any more. I tried everything besides de-installing and re-installing. It does not help. Camera folder does not get uploaded any more, whereas pictures from Conversations (XMPP messenger) do, but screenshots dont. Auto-upload is now really completely broken. Please everybody, check_all_ your auto-upload folders if they really still work.

@ltguillaume
Copy link

Ok,, with 3.30.5 you managed to break the auto-upload alltogether. Now the status is that some folders get auto-uploaded and others dont. Namely the first folder (Camera) does not get uploaded any more. I tried everything besides de-installing and re-installing. It does not help. Camera folder does not get uploaded any more, whereas pictures from Conversations (XMPP messenger) do, but screenshots dont. Auto-upload is now really completely broken. Please everybody, check_all_ your auto-upload folders if they really still work.

I don't see this behavior, but I only have custom folders set up for auto-upload. Perhaps this is related only to automatically detected folders?

@RikkiBC
Copy link

RikkiBC commented Dec 2, 2024

Ok,, with 3.30.5 you managed to break the auto-upload alltogether. Now the status is that some folders get auto-uploaded and others dont. Namely the first folder (Camera) does not get uploaded any more. I tried everything besides de-installing and re-installing. It does not help. Camera folder does not get uploaded any more, whereas pictures from Conversations (XMPP messenger) do, but screenshots dont. Auto-upload is now really completely broken. Please everybody, check_all_ your auto-upload folders if they really still work.

I'm not seeing this myself either, but it sounds like a new issue that would be best in its own bug report if you're experiencing it.

@tobiasKaminsky
Copy link
Member

We did not changed any thing auto-uploaded related.
Is Camera or any other in 2-way sync list?
Else I think it is indeed unrelated.

@joe-average-user
Copy link

2-way sync is turned off with the corresponding button. I have problems in understanding where a 2-way sync list is to be found? My understanding so far was that turning on 2-way sync affects the "standard" sync list - as it did this when the option was broken...
I really do not want to turn it on ever again, so I fail to see if there are any further config options coming up after that.

@tobiasKaminsky
Copy link
Member

@joe-average-user which version do you use?
You should be able to see it in settings -> internal two-way sync.

There you can also disable it.
The list will be shown beneath the "interval".

Image

@Novemberheks
Copy link

You missed the point. your 2-way-sync implementation broke the sync as a whole. In fact every folder seems to get resynced continously, no matter what direction. I would say you misinterpret the sync-database in some point(field).
This does not happen on a fresh install, only in update path.

Oh, and forgot another important thing: the sync then ignores the setup when it only should sync with WLAN connection, not mobile. This is reported several times here and I can confirm this problem.

Same here. And it cost me a month's worth of mobile data and completely filled up my space on the server. I'm not going to use it on my mobile devices anymore until I can be absolutely sure these issues are solved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature: sync meta Issues about repository, contribution, development stable-3.30
Projects
None yet
Development

No branches or pull requests