Large files (>10MB) on S3 primary storage are lost when uploaded to group folder

[infra-monkey]

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

Steps to reproduce

1. Upload a large file to the group folder
2. Try to access the file
3. Get an error saying that the object cannot be found

Expected behaviour

Object should be present

Actual behaviour

I can see that The file is written on the backend

Nextcloud logs show

 Exception Failed to read object urn:oid:769058
Could not get object urn:oid:769058 for file __groupfolders/6/24-01-01 14-59-28 1225.mov 
fopen(httpseek://): Failed to open stream: "OC\Files\Stream\SeekableHttpStream::stream_open" call failed at /var/www/html/lib/private/Files/Stream/SeekableHttpStream.php#49 
fopen(http://s3.empire.lan:9000/nextcloud-data/urn%3Aoid%3A769058): Failed to open stream: HTTP request failed! HTTP/1.1 416 Requested Range Not Satisfiable at /var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php#72 
ServiceUnavailable Could not open file

Uploading to a regular user folder works perfectly well.
Moving a large file from a user folder to a group folder looses the file also.
However, copying a large file from a user folder to a group folder works. The file is accessible from the group folder.
Files <10MB don't seem to show the same behavior.

Server configuration

Operating system:
docker container

Web server:
nginx

Database:
postgresql 15

PHP version:
8.2

Nextcloud version: (see Nextcloud admin page)
29.0.7 & 30.0.0

Group folders version:
17.0.4 & 18.0.2

Updated from an older Nextcloud/ownCloud or fresh install:
Updated from older versions

Where did you install Nextcloud from:

Are you using external storage, if yes which one: local/s3/smb/sftp/...
primary storage is S3 minio

Are you using encryption: no

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/Saml/...
OAuth2

Client configuration

Browser:
Firefox/chrome/brave

Operating system:
Linux Leap or Fedora

Logs

No error during upload, but when trying to access the file I get this :

{"reqId":"AH29rbeboMLyOpfwxiEM","level":3,"time":"2024-10-01T05:48:57+00:00","remoteAddr":"10.27.1.1","user":"mattermost-bnrda-ix5z9ksay38488pmnefdehnmer","app":"PHP","method":"GET","url":"/remote.php/dav/files/mattermost-bnrda-ix5z9ksay38488pmnefdehnmer/BNRDA%20Video%20Upload/24-01-01%2014-59-28%201225.mov","message":"fopen(http://s3.empire.lan:9000/nextcloud-data/urn%3Aoid%3A769064): Failed to open stream: HTTP request failed! HTTP/1.1 416 Requested Range Not Satisfiable\r\n at /var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php#72","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0","version":"30.0.0.14","data":{"app":"PHP"}}
{"reqId":"s6MMhsgognRjj6U8YDz8","level":3,"time":"2024-10-01T05:48:57+00:00","remoteAddr":"10.27.1.1","user":"mattermost-bnrda-ix5z9ksay38488pmnefdehnmer","app":"PHP","method":"GET","url":"/remote.php/dav/files/mattermost-bnrda-ix5z9ksay38488pmnefdehnmer/BNRDA%20Video%20Upload/24-01-01%2014-59-28%201225.mov","message":"fopen(http://s3.empire.lan:9000/nextcloud-data/urn%3Aoid%3A769064): Failed to open stream: HTTP request failed! HTTP/1.1 416 Requested Range Not Satisfiable\r\n at /var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php#72","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0","version":"30.0.0.14","data":{"app":"PHP"}}
{"reqId":"AH29rbeboMLyOpfwxiEM","level":3,"time":"2024-10-01T05:48:57+00:00","remoteAddr":"10.27.1.1","user":"mattermost-bnrda-ix5z9ksay38488pmnefdehnmer","app":"PHP","method":"GET","url":"/remote.php/dav/files/mattermost-bnrda-ix5z9ksay38488pmnefdehnmer/BNRDA%20Video%20Upload/24-01-01%2014-59-28%201225.mov","message":"fopen(httpseek://): Failed to open stream: "OC\\Files\\Stream\\SeekableHttpStream::stream_open" call failed at /var/www/html/lib/private/Files/Stream/SeekableHttpStream.php#49","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0","version":"30.0.0.14","data":{"app":"PHP"}}
{"reqId":"s6MMhsgognRjj6U8YDz8","level":3,"time":"2024-10-01T05:48:57+00:00","remoteAddr":"10.27.1.1","user":"mattermost-bnrda-ix5z9ksay38488pmnefdehnmer","app":"PHP","method":"GET","url":"/remote.php/dav/files/mattermost-bnrda-ix5z9ksay38488pmnefdehnmer/BNRDA%20Video%20Upload/24-01-01%2014-59-28%201225.mov","message":"fopen(httpseek://): Failed to open stream: "OC\\Files\\Stream\\SeekableHttpStream::stream_open" call failed at /var/www/html/lib/private/Files/Stream/SeekableHttpStream.php#49","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0","version":"30.0.0.14","data":{"app":"PHP"}}
{"reqId":"AH29rbeboMLyOpfwxiEM","level":3,"time":"2024-10-01T05:48:57+00:00","remoteAddr":"10.27.1.1","user":"mattermost-bnrda-ix5z9ksay38488pmnefdehnmer","app":"objectstore","method":"GET","url":"/remote.php/dav/files/mattermost-bnrda-ix5z9ksay38488pmnefdehnmer/BNRDA%20Video%20Upload/24-01-01%2014-59-28%201225.mov","message":"Could not get object urn:oid:769064 for file __groupfolders/6/24-01-01 14-59-28 1225.mov","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0","version":"30.0.0.14","exception":{"Exception":"Exception","Message":"Failed to read object urn:oid:769064","Code":0,"Trace":[{"file":"/var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php","line":308,"function":"readObject","class":"OC\\Files\\ObjectStore\\S3","type":"->","args":["urn:oid:769064"]},{"file":"/var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php","line":283,"function":"fopen","class":"OC\\Files\\ObjectStore\\ObjectStoreStorage","type":"->","args":["__groupfolders/6/24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/lib/private/Files/Storage/Wrapper/Jail.php","line":288,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->","args":["__groupfolders/6/24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/lib/private/Files/Storage/Wrapper/Quota.php","line":153,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Jail","type":"->","args":["24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php","line":283,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Quota","type":"->","args":["24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/lib/private/Files/Storage/Wrapper/PermissionsMask.php","line":97,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->","args":["24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php","line":283,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\PermissionsMask","type":"->","args":["24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/lib/private/Files/View.php","line":1134,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->","args":["24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/lib/private/Files/View.php","line":962,"function":"basicOperation","class":"OC\\Files\\View","type":"->","args":["fopen","/BNRDA Video Upload/24-01-01 14-59-28 1225.mov",["read"],"r"]},{"file":"/var/www/html/apps/dav/lib/Connector/Sabre/File.php","line":457,"function":"fopen","class":"OC\\Files\\View","type":"->","args":["BNRDA Video Upload/24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":85,"function":"get","class":"OCA\\DAV\\Connector\\Sabre\\File","type":"->","args":[]},{"file":"/var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"httpGet","class":"Sabre\\DAV\\CorePlugin","type":"->","args":[["Sabre\\HTTP\\Request"],["Sabre\\HTTP\\Response"]]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php","line":472,"function":"emit","class":"Sabre\\DAV\\Server","type":"->","args":["method:GET",[["Sabre\\HTTP\\Request"],["Sabre\\HTTP\\Response"]]]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php","line":253,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[["Sabre\\HTTP\\Request"],["Sabre\\HTTP\\Response"]]},{"file":"/var/www/html/apps/dav/lib/Server.php","line":370,"function":"start","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/html/apps/dav/appinfo/v2/remote.php","line":19,"function":"exec","class":"OCA\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/html/remote.php","line":146,"args":["/var/www/html/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php","Line":75,"message":"Could not get object urn:oid:769064 for file __groupfolders/6/24-01-01 14-59-28 1225.mov","exception":{},"CustomMessage":"Could not get object urn:oid:769064 for file __groupfolders/6/24-01-01 14-59-28 1225.mov"}}
{"reqId":"s6MMhsgognRjj6U8YDz8","level":3,"time":"2024-10-01T05:48:57+00:00","remoteAddr":"10.27.1.1","user":"mattermost-bnrda-ix5z9ksay38488pmnefdehnmer","app":"objectstore","method":"GET","url":"/remote.php/dav/files/mattermost-bnrda-ix5z9ksay38488pmnefdehnmer/BNRDA%20Video%20Upload/24-01-01%2014-59-28%201225.mov","message":"Could not get object urn:oid:769064 for file __groupfolders/6/24-01-01 14-59-28 1225.mov","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0","version":"30.0.0.14","exception":{"Exception":"Exception","Message":"Failed to read object urn:oid:769064","Code":0,"Trace":[{"file":"/var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php","line":308,"function":"readObject","class":"OC\\Files\\ObjectStore\\S3","type":"->","args":["urn:oid:769064"]},{"file":"/var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php","line":283,"function":"fopen","class":"OC\\Files\\ObjectStore\\ObjectStoreStorage","type":"->","args":["__groupfolders/6/24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/lib/private/Files/Storage/Wrapper/Jail.php","line":288,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->","args":["__groupfolders/6/24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/lib/private/Files/Storage/Wrapper/Quota.php","line":153,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Jail","type":"->","args":["24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php","line":283,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Quota","type":"->","args":["24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/lib/private/Files/Storage/Wrapper/PermissionsMask.php","line":97,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->","args":["24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php","line":283,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\PermissionsMask","type":"->","args":["24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/lib/private/Files/View.php","line":1134,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->","args":["24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/lib/private/Files/View.php","line":962,"function":"basicOperation","class":"OC\\Files\\View","type":"->","args":["fopen","/BNRDA Video Upload/24-01-01 14-59-28 1225.mov",["read"],"r"]},{"file":"/var/www/html/apps/dav/lib/Connector/Sabre/File.php","line":457,"function":"fopen","class":"OC\\Files\\View","type":"->","args":["BNRDA Video Upload/24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":85,"function":"get","class":"OCA\\DAV\\Connector\\Sabre\\File","type":"->","args":[]},{"file":"/var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"httpGet","class":"Sabre\\DAV\\CorePlugin","type":"->","args":[["Sabre\\HTTP\\Request"],["Sabre\\HTTP\\Response"]]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php","line":472,"function":"emit","class":"Sabre\\DAV\\Server","type":"->","args":["method:GET",[["Sabre\\HTTP\\Request"],["Sabre\\HTTP\\Response"]]]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php","line":253,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[["Sabre\\HTTP\\Request"],["Sabre\\HTTP\\Response"]]},{"file":"/var/www/html/apps/dav/lib/Server.php","line":370,"function":"start","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/html/apps/dav/appinfo/v2/remote.php","line":19,"function":"exec","class":"OCA\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/html/remote.php","line":146,"args":["/var/www/html/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php","Line":75,"message":"Could not get object urn:oid:769064 for file __groupfolders/6/24-01-01 14-59-28 1225.mov","exception":{},"CustomMessage":"Could not get object urn:oid:769064 for file __groupfolders/6/24-01-01 14-59-28 1225.mov"}}
{"reqId":"s6MMhsgognRjj6U8YDz8","level":3,"time":"2024-10-01T05:48:57+00:00","remoteAddr":"10.27.1.1","user":"mattermost-bnrda-ix5z9ksay38488pmnefdehnmer","app":"webdav","method":"GET","url":"/remote.php/dav/files/mattermost-bnrda-ix5z9ksay38488pmnefdehnmer/BNRDA%20Video%20Upload/24-01-01%2014-59-28%201225.mov","message":"Could not open file","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0","version":"30.0.0.14","exception":{"Exception":"Sabre\\DAV\\Exception\\ServiceUnavailable","Message":"Could not open file","Code":0,"Trace":[{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":85,"function":"get","class":"OCA\\DAV\\Connector\\Sabre\\File","type":"->","args":[]},{"file":"/var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"httpGet","class":"Sabre\\DAV\\CorePlugin","type":"->","args":[["Sabre\\HTTP\\Request"],["Sabre\\HTTP\\Response"]]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php","line":472,"function":"emit","class":"Sabre\\DAV\\Server","type":"->","args":["method:GET",[["Sabre\\HTTP\\Request"],["Sabre\\HTTP\\Response"]]]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php","line":253,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[["Sabre\\HTTP\\Request"],["Sabre\\HTTP\\Response"]]},{"file":"/var/www/html/apps/dav/lib/Server.php","line":370,"function":"start","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/html/apps/dav/appinfo/v2/remote.php","line":19,"function":"exec","class":"OCA\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/html/remote.php","line":146,"args":["/var/www/html/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/var/www/html/apps/dav/lib/Connector/Sabre/File.php","Line":463,"message":"Could not open file","exception":{},"CustomMessage":"Could not open file"}}
{"reqId":"AH29rbeboMLyOpfwxiEM","level":3,"time":"2024-10-01T05:48:57+00:00","remoteAddr":"10.27.1.1","user":"mattermost-bnrda-ix5z9ksay38488pmnefdehnmer","app":"webdav","method":"GET","url":"/remote.php/dav/files/mattermost-bnrda-ix5z9ksay38488pmnefdehnmer/BNRDA%20Video%20Upload/24-01-01%2014-59-28%201225.mov","message":"Could not open file","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0","version":"30.0.0.14","exception":{"Exception":"Sabre\\DAV\\Exception\\ServiceUnavailable","Message":"Could not open file","Code":0,"Trace":[{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":85,"function":"get","class":"OCA\\DAV\\Connector\\Sabre\\File","type":"->","args":[]},{"file":"/var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"httpGet","class":"Sabre\\DAV\\CorePlugin","type":"->","args":[["Sabre\\HTTP\\Request"],["Sabre\\HTTP\\Response"]]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php","line":472,"function":"emit","class":"Sabre\\DAV\\Server","type":"->","args":["method:GET",[["Sabre\\HTTP\\Request"],["Sabre\\HTTP\\Response"]]]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php","line":253,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[["Sabre\\HTTP\\Request"],["Sabre\\HTTP\\Response"]]},{"file":"/var/www/html/apps/dav/lib/Server.php","line":370,"function":"start","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/html/apps/dav/appinfo/v2/remote.php","line":19,"function":"exec","class":"OCA\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/html/remote.php","line":146,"args":["/var/www/html/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/var/www/html/apps/dav/lib/Connector/Sabre/File.php","Line":463,"message":"Could not open file","exception":{},"CustomMessage":"Could not open file"}}

Web server error log

Web server error log

Insert your webserver log here

Nextcloud log (data/nextcloud.log)

Nextcloud log

Insert your Nextcloud log here

Browser log

Browser log

Insert your browser log here, this could for example include:

a) The javascript console log
b) The network log
c) ...

[provokateurin]

Not groupfolder specific, this is an S3 issue.

[fcharlaix-opendsi]

I have the same issue with Nextcloud 29.0.7 on Debian 12, PHP 8.2 without Docker, using OVH Object Storage (their backend is not MinIO as far as I know).

On a Nextcloud 28.0.9 with Group Folders 16.0.10 and MinIO, the issue is not present.

[kesselb]

@fcharlaix-opendsi would you mind opening a support ticket?

If the problem occurs with 29.0.7, but not 28.0.9 then a relevant change is: #46013

A way to test that could be to comment the moveFromStorage method in ObjectStoreStorage to use, as before the patch, just the default implemention. Note this will reintroduce the issue that were fixed by it.

[infra-monkey]

@kesselb in v30.0.0 commenting out the method and restarting the container seems to fix this issue.
I'll do more thorough tests tonight

[Mika-6-9]

Hi @infra-monkey
Have you been able to do more tests?
And especially conclusive tests? 😄

Thanks

[infra-monkey]

@Mika-6-9 I have uploaded successfully 40+ big files since. No issue.
As I didn't have any problem prior to updating to 29.0.7, the change seems indeed to be the culprit in my case.
I don't use other external storage so I was not impacted.

Any particular use case to test?

[infra-monkey]

I still have a weird issue.
Upload file to root folder. Then move it to a group folder.
I get an error notification but the file is available in the group folder. After the move, the file was still listed in the root folder and disappeared after a refresh.
No problem with a copy tho.
The error is:

Error	webdav	
TypeError
OCA\Files_Versions\Listener\VersionStorageMoveListener::handleMoveOrCopy(): Argument #3 ($source) must be of type OCP\Files\File, null given, called in /var/www/html/apps/files_versions/lib/Listener/VersionStorageMoveListener.php on line 101
Error	PHP	
Undefined array key 769579 at /var/www/html/apps/files_versions/lib/Listener/VersionStorageMoveListener.php#97

moving between subfolders of the same group folder works correctly.
moving from the group folder to the group folder fails in the same way.
moving from a group folder to another group folder works also.