-
-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[NOTICE] Version 3.30.1 Causing Sync Issues / Synchronising All Files #13872
Comments
The github versions don't install on devices that install from F-Droid / Gplay / ?. Older versions 3.29.1 available as APK on F-Droid does NOT install citing invalid package .... |
@noci2012, F-Droid builds are signed by F-Droid. Builds published on GitHub are signed using our key. This means that one must, unfortunately, choose and stay with one source for continued updates. In some cases, the versions differ in some aspects due to some other constraints, e.g. Google Libraries. This is the case for Google Play and F-Droid builds. Downgrading is not officially supported, as far as I'm aware. To install an older version, one should first remove the newer one. |
I'm curious why the same files were repeatedly synchronized ad infinitum? I don't think the Technical Background currently addresses this. In my case, Nextcloud downloaded 15 GB of data from a server containing 1 GB of data. Others have reported the same. |
I see in the F-Droid build log: com.nextcloud.client:30300290 2024-10-22 15:20:18,028 INFO: Building version 3.30.2 (30300290) of com.nextcloud.client . It seems that it takes 3 to 4 days until the build is distributed. Maybe we should help F-Droid to speed up the distribution process. I some knows more, please help out. |
I don't think so, because I have version 3.30.1 at the moment on LineageOS and it sync all files once (using almost all the battery), but at the moment it is fine. I checked it on another phone and it seems, that it sync the same file several times, but not ad infinitum. (Just my personal experience with 3 clients, so nothing to really relay on) Possible that the repeating sync start is caused by a timeout. With a lower server space usage, the sync seems to be repeated less often. |
Mitigation (i.e. reinstall, but explained really verbosely) is for technical users only. Many users don't know their password or what a password is (an example from my circles where they just want to see pictures by swiping). For them re-logging into the app via a computer with a QR code through an obscure web UI path is impossible. You littered every single user's phone with gigabytes of difficult-to-remove data. This is a major mess. Especially, that is comes just months after another major synchronisation problem. I already received complaints that Nextcloud duplicates all files into its cache and filling up the phone eventually (details in #13400 (comment)). Please, create a proper, automated solution for this problem ASAP. Otherwise you offload solving this chaos to every single technical user who chose and distributed your application. |
Data can be simply removed via "on device" and then removing the folder that you do not want, choosing "local only".
I am all for it, but do you have an idea how? |
In the short term an "Empty local storage" button would be a good interim solution. I can communicate to press this button and this problem goes away. This would also conveniently fix #9128, too. As a next step I could imagine a Trashbin-like system which, after a configurable amount of used local storage (e.g. 2% of device storage), it'd warn the user to press this button. One more step further, the application could be re-architected in a way that it outsources the internal storage handling into a regular Linux directory which the user can manage themselves. This would fix #13400. Even if you agree with this direction, I see this would have many edge cases and is not a quick solution. |
Back when this issue arose, it could be safely assumed that no one started using this feature just yet. Even if they did, it would have been a far more decent solution to have said "sorry, we made an error, and to solve it for everyone automatically, we have reset the two-way sync feature and removed the local copies. For those 2 people who were using it already, please set up the two-way sync again". Notifications like that should also be possible within the app. You could then also use this to inform people of new features. |
Even when not in Wifi, 3.30.2 is syncing the same files over and over again, eating my monthly mobile data completely within a few hours. Removing the files from the sync did not stop it. I had to delete the file database to stop the sync. The app seems to be completely broken now. |
Hi, I just updated to 3.30.2 from f-droid and I saw "check listed folders in settings -> internal two sync", so I went and checked, the app took a really long time and then showed me a list of a LOT of folders, I could swipe for minutes (EDIT: it's so long I did not reach the bottom yet, I think this is just a list of all folders...). It says "Pas encore, bientôt synchronisé" (my app is in French, it means, "not yet, soon synchronized") for each folder. I don't know what that means, I don't know when it will do that and how to avoid it (EDIT: It started syncing after an hour or so after the update). I tried to press on the cross but it just make the app not responding (EDIT: it takes roughly 1 min 30 to get the app responding again) and the time it would take to press that cross for each folder even if it was working would be insane. I'm scared by the idea of a two way sync being triggered on data I never wanted to two way sync. Apart from the issue of my phone storage being filled, my data is important and I don't want data from my phone to corrupt data on my server. I never selected those folders and I hope they won't sync, but I don't know what "not yet, soon synchronized" means. What should we do? Is there a way to prevent the app from attempting any sync for the time being, so that the issue can be fixed? I like being able to sync my photos to nextcloud and being able to sync some files manually, but I didn't know auto two way sync was a thing on android, is there a way to disable automatic two way sync? EDIT: For context, in my case none of the folders were wrongly synced (yet?), my only issue is that all my folders are listed under "settings -> internal two sync" and marked as "Pas encore, bientôt synchronisé"/"not yet, soon synchronized" and I don't want them to sync if they have no previous version locally (which would mean that they are not supposed to be synced) EDIT 2: I waited after tapping on a cross on one of the folder, I did it twice. It took 1 min 30 sec to register. I don't know how many folders and files there are, but if I had to press them one by one and wait for 1 min 30 sec each time, it would take days (Edit: 10 full days). EDIT 3: That's it, random files started to be downloaded. I put my phone into airplane mode from now on... I might have to reinstall and re-setup everything, which would mean a lot of work since tens of Go of files were synced and would have to be re-downloaded (mainly auto synced photos and videos) and other apps use identification from the main nextcloud app. I have multiple users hit with the bug. I'm going to wait a bit in the hope that we'll have a button to just remove all files that don't have any local version from being synced. EDIT 4: Thanks to a notification, I now know that 15000 items are queued for sync, and it takes a full minute for each item to be removed (I'm using a samsung galaxy s10 with the latest update available) which would take 10 full days with an autoclicker. Since there is no button to juste remove them all in one go, I have no other option but to reinstall the app, reconfigure everything and download all the files I had again. I was able to remove all the items with an autoclicker on a galaxy s21, the removal on that device of each item in "two way sync" was almost instantaneous. Most user won't setup an autoclicker, and I expect that removing all items by hand would have taken half an hour at least. Even in that case a single button to remove everything would be necessary. TL;DR: Please provide a single button to remove all files that don't have any local version yet from the "internal two sync", if you have just been hit by this bug, a button like this would stop any further damage without removing properly synced files. If that's too complicated, just a button to remove all items from "internal two sync" in one tap. |
@timkrief There's only one proper solution: note down any automatic upload configurations you have, then uninstall and reinstall. |
@ltguillaume I finally did it and I'm in the process of re-syncing all the files that were (intentionally) synced before. It reveals a lot of flaws with the sync system altogether, but it's not related to this issue. I'm still settings things up, but it seems to work for the time being, the issue of having all the folders added to two way sync is not happening anymore, but if I do add things to the two way sync system, they will sync again and again even with no modification, so I do have to make sure there's nothing in the two way sync page. I would like to say that when going to the "two way sync" settings to see if there is no issue, it's quite troubling to have a completely blank screen. When I had the issue, I had a blank screen for a minute before all 15000 folders appeared in the list, so I couldn't tell easily if the issue was still there or if everything was fine. Many user go to that screen and think this is a bug. I think it should be a priority to add at least a single line "no two way sync folder yet" or something to make sure the experience is smooth. see for instance: and:
Originally posted by @danepowell in #13693 |
This still happens with the new (cleanly installed) version? Wow, I didn't expect that and I don't know if the devs are aware of this. This should be addressed in a new issue!
Agreed, this is a UX 101 violation 😛 |
IMHO only half of the bug is fixed at best. The 2-way sync seems to completely ignore my wifi-only and charging-only settings and ate up my data-plan. This does not seem to be addressed in the 3.30.2 version at all! |
I may be missing something, but are there even settings for this? To my knowledge, those settings are only implemented for auto upload, not for two way sync. Apparently 3.30.3 does force two way sync to only be done via Wi-Fi, but if you're running Nextcloud via F-Droid, then you haven't received that update yet. |
I have been following this issue for some time now. Fortunately I read about it before the automatic upgrade from F-Droid triggered, so I am now still at 3.29.1 and waiting until it is safe to upgrade. But I am confused whether this is already the case (or will be once 3.30.3 hits the F-Droid store). Has anybody tested if the upgrade now works as intended, or are we assuming everyone has already been hit? |
To the Nextcloud developers, with all the information I have, this bug is/was essentially a Denial of Service "attack" across most users of Nextcloud Android. A bug can happen, as a dev myself I have full understanding how that can happen. But IMO, the correct response to a bug with this severity is to disable this feature fully, temporarily, for all users by immediately rolling back to 3.29.1. Why do I consider this bug a servere Denial of Service:
So this necessarily includes a large amount of "normal" users which (like others stated before):
And I know, these upcoming points are not something you could change, but are still true:
And new reports seem to suggest that this new internal feature can still cause issues even through it is expected to be fixed. This brings me to the conclusion that IMO, for now, there is one thing to do now, before trying to fix this bug: ASAP release a new version based on 3.29.1 (before 2way sync) with a notification explaining to "normal" users what happened & (if so) what they need to do now (can of course link to this GH issue for the occasional nerd). Also, but this can be deferred, it would be nice if a next version could delete the files the app created while syncing. However, I can understand that this might be a problematic step to take if users might have modified those files in the mean time. (For clarification: I suggest reactions with 👍/👎 are referring to the idea of the immediate roll back to 3.29.1 & not for other points I made, so this can be used as a poll.) |
Could you please please please just release a new stable version with everything two way sync disabled completely, then start building up this feature properly in the dev builds? |
Well, simply believe me. Whatever you read about "solutions", the only thing that does work is removing the app and re-installing, then disable 2.-way, and then add your syncs again. |
I use v3.30.4 and I've just encountered this issue (AFAIK) for the first time. So it seems not really fixed at all. I was connected to wifi though, so no damage done, but anyway… |
Where is that? I've now disabled two-way sync via the button, but it continues to download files… Edit: Ah I need to enable 2-way-sync, then click on all Xes there to remove them from the queue? I did that, and disabled 2-way-sync again. Aaaand it still sync again. Hint: At least for CalyxOS you can pause an app entirely, so it won't re-start. |
You will have a lot less troubles if you simply believe me it is broken. You can only restore original bahaviour if you follow my above advice. |
Is this about syncing folders or a specific file? The new 2 way sync is "only" about folders and we enhanced it with 3.30.5, which will be soon in Fdroid release. |
You missed the point. your 2-way-sync implementation broke the sync as a whole. In fact every folder seems to get resynced continously, no matter what direction. I would say you misinterpret the sync-database in some point(field). Oh, and forgot another important thing: the sync then ignores the setup when it only should sync with WLAN connection, not mobile. This is reported several times here and I can confirm this problem. |
@joe-average-user can you explain in more detail what is broken?
So after upgrade, it should be enough to once/finally check if all your desired sync folders are in the settings list. |
Well, for understanding the problems, read the thread. |
As per above I was badly having the problems on two devices when updating to 3.30.1 (FDroid). Both where manually "repaired" initially and are online again afair since 3.30.3. Both devices have only been updated through all versions since (no re-install), one to 3.30.4, the other to 3.30.5 since yesterday. Both have never again fallen back to unwanted syncs. In the meantime, even the two way sync seems to work as intended. @tobiasKaminsky Do I understand correctly, that two way sync atm is hardcoded to WLan only, despite already having sync-conditions options for auto-upload? Couldn't two way sync be made to function simillarly? |
I just wanted to give this a try, as I am still on 3.29.2 and have blocked updates on F-Droid since, waiting for this to be resolved. But F-Droid refuses to perform the update, it says the new version has been signed with a different key and that I need to uninstall and reinstall. Basically that means I am forced to do the "fix" the hard way... :-( |
@flixmart Yes. TwoWaySyncWorker is only working via Wi-Fi for now. |
That is strange. |
Are you referring to 3.30.5 explicitly or also to earlier revisions? Because with 3.30.4, it used 15 GB of my wife's mobile plan before I deinstalled it after all other countermeasures from this thread had failed. |
When I put files into a synced folder via filemanager, they are not synced. NC does not know that files. |
Since version 3.30.2, the Two-Way Sync Worker only uses Wi-Fi. Could the 15GB usage be due to Auto-Upload? Was Auto-Upload enabled with mobile data for some folders? Additionally, we fixed the re-download issue in v3.30.5. However, if only mobile data is active, the Two-Way Sync Worker does not download anything. |
[...]
Auto upload was active for the photo folder, but only allowed for wifi (and this worked for years now), plus she did not add 15 GB+ of new photos that day. And, I was able to reproduce with 3.30.4 that if I switch off wifi, the unstoppable syncing process kept running. It even re-spawned several times after deleting the app cache and other tries to stop it. Unfortunately, I had to deinstall and reinstall the app to prevent more damage to my wife's yearly mobile plan¹, therefore I can no longer test this behavior. ¹It would anyway never have succeeded, even if I had kept the phone connected via wifi for several days: The entire nc folder was added to the sync list, way more than 100 GB of data (I wrote 100 GB before, but this was outdated information 😉), which is more than the free space on the device anyway. |
Thanks for the information. We were aware of the unstoppable syncing issue and addressed it by stopping the scheduled periodic worker at app launch in v3.30.5. We expect this issue to be resolved in v3.30.5. As for the mobile data usage still occurring in v3.30.4, we will look into it. |
I was hit by this too right at the beginning and uninstalled after my monthly allowance was consumed! I am watching this thread closely and waiting for it to become stable enough for me to reinstall. One thing I have read is that sync is now limited to WiFi. I have no WiFi and do all my sync on Ethernet (or mobile). I think this needs to be amended to allow sync on Ethernet too (and a toggle for mobile) or it wont even work for me when I do reinstall? #13847 (review) |
Adding an option to also run it on mobile data, is possible. |
Ok,, with 3.30.5 you managed to break the auto-upload alltogether. |
I don't see this behavior, but I only have custom folders set up for auto-upload. Perhaps this is related only to automatically detected folders? |
I'm not seeing this myself either, but it sounds like a new issue that would be best in its own bug report if you're experiencing it. |
We did not changed any thing auto-uploaded related. |
2-way sync is turned off with the corresponding button. I have problems in understanding where a 2-way sync list is to be found? My understanding so far was that turning on 2-way sync affects the "standard" sync list - as it did this when the option was broken... |
@joe-average-user which version do you use? There you can also disable it. |
Same here. And it cost me a month's worth of mobile data and completely filled up my space on the server. I'm not going to use it on my mobile devices anymore until I can be absolutely sure these issues are solved. |
Nextcloud Android Client Version 3.30.1
Dear Community,
It is with great regret that we have to inform you that a recent version of the Nextcloud Android Client, specifically version 3.30.1, has introduced a bug that may cause one or more of the following issues:
Please note that manual intervention will be required by users who have upgraded to the affected version.
Updating - How To Disable Two-Way Synchronisation
We have carefully investigated this issue and released new versions which aim to address these issues.
With Version 3.30.4 we have introduced a new way to manage two-way sync. To access this redesigned menu open Settings → Internal two way sync.
This menu allows users to globally enable or disable two-way synchronisation. Disabling this will ensure that no two-way synchronisation will occur in the future. As before, users have the option to selectively remove individual folders using the × icon next to the folder name. For advanced users, there is also an option to set the interval at which the Nextcloud Android Client will attempt two-way synchronisation.
We have also added a button to disable two-way synchronisation for all folders. This option will appear in the overflow menu (⋮) if two-way synchronisation is enabled for at least one folder. Furthermore, this will attempt to cancel any transfers in progress and update the database to exclude those folders from being synchronised again.
Removing Local Files
Some users have experienced increased storage usage on their device. Allocated storage space can be freed by removing local files: open the On device tab and long press on any file or folder. Then open the overflow menu (⋮) to Select all files. When all files are selected, re-open the overflow menu and select Delete. When prompted, select Local only. This is important, as selecting Delete in this dialogue will also remove all selected files from the server. The removal process will take some time depending on how many files have been downloaded.
How to Upgrade
The Nextcloud Android Client can be updated via your preferred installation method. All versions are also available on GitHub for download.
As of 05/11/2024, version 3.30.4 is available from F-Droid. This version includes all major fixes and the redesigned management interface.
Unfortunately, the Google Play release will take some more time to be available as we are currently blocked by Google. This is, however, tracked in a different issue: #13871.
We will update this issue as soon as the releases are available.
Mitigation
We strongly recommend that all users to upgrade to version 3.30.4 or later using your preferred method to resolve these issues. This version includes several fixes and changes that should resolve any issues related to two-way synchronisation. Please refer to the explanation above for information. If this version is not available for you, or if something else is blocking this update, there are several options to mitigate any unwanted behaviour.
Option 1 - Manual Removal
From version 3.30.2, users have the option to quickly check and stop two-way synchronisation for specific folders. To access this menu open Settings → Internal two way sync. On the following screen one will see all folders scheduled for two-way synchronisation and have the option to exclude folders using the × icon next to the folder name. If a blank page is displayed instead, no folders are scheduled for two-way synchronisation.
Option 2 - Reinstalling
For version 3.30.1 to 3.30.3 (outdated)
The simplest option is to reinstall the Nextcloud Android Client. Due to the technical nature of the issue reinstalling any version above 3.30.0 will resolve the issue. The process of reinstalling an app varies from Android version to Android version. However, the general process will be similar to the following:
Please note that any local data of the Nextcloud Android Client that is present on the mobile device will be lost during the removal process.
Option 3 - Reinitialise Local Account
For version 3.30.1 to 3.30.3 (outdated)
Another option, if you prefer to keep the current version installed, is to remove the current user account and it add again. This should have a similar effect to reinstalling the application. To re-initialise your account in the Nextcloud Android Client, follow these steps:
Please note that any local data of the Nextcloud Android Client that is present on the mobile device will be lost during the removal process.
General - Enabling Data Limit and Warning
Many mobile network operators charge their subscribers an additional fee when they exceed their data allowance. To avoid these charges, most versions of Android offer some kind of safety measures. These include a Data Limit and Data Warning. The latter notifies you once the specified amount of mobile data has been used up. The former allows you to set a hard limit after which Android will disable mobile data and prevent you from using any more. Also note that you will not be able to use any apps that require an active data connection, such as Internet messengers.
Settings to enable and configure the Data Limit and Warnings can usually be found in the device settings.
Related Issues
Technical Background
Internally, the Nextcloud Android Client uses a database table to keep track of all files. This table has been extended by two columns to accommodate the new two-way sync feature:
internal_two_way_sync_timestamp
andinternal_two_way_sync_result
. The latter of these columns is not relevant for the purposes of this investigation.We use the timestamp to keep track of two things. Firstly, the date when the corresponding file was last synchronised. Secondly, whether the file should be synchronised at all. If the timestamp value of a file contains a valid timestamp greater than
0
the file will be scheduled for synchronisation. In this case0
means the file has never been synchronised. A value of-1
indicates that a file should not be synchronised at all.On a clean install of the Nextcloud client, all records in the
filelist
table will be initialised with a timestamp value of-1
. When upgrading from an older version, the database is migrated to fit the new scheme. During this migration, the timestamp column is added to the table and initialised with a default value ofnull
. This must be handled carefully when interpreting database values, as improperly handled null values are known to cause a variety of problems. In this case,null
represents unknown; the file has never been synchronised and the user hasn't made a decision about whether or not it should be synchronised.In a recent update, a change was made to mitigate null pointer exceptions during file synchronisation. Due to a small oversight, this also meant that all null values would be interpreted as
0
. While this fixed the null pointer issues, it unfortunately introduced a new problem.When a user updates to an affected version, the column is filled with
null
, which is interpreted as0
. After a while, a background worker will look up all files that need to be synchronised in the aforementioned table. Since all files have a0
timestamp, it will try to synchronise them all.For technical reasons, we also store the root folder, which contains all other files, in the same database. This folder is represented by a record having
/
as its filename. Therefore, all folders and files are scheduled for synchronisation.As it is normally not possible to select the root folder for two-way sync, we have excluded it from the synchronisation process in the patched versions.
Due to the nature of our apps and release channels, rollbacks to previous versions are not possible. Instead, we have released newer versions that fix these issues.
Version History
null
conversionThe text was updated successfully, but these errors were encountered: