Deleting / Re-creating Bucket With Versioning Does Not Enable Versioning through Web Console

[markive]

We are testing MinIO, and the first test we did was to create a bucket without versioning (to look at file structures on disk). Then we wanted to test versioning so we turned it on. We found that our bucket wasn't creating files with version IDs (versionID = null) when uploaded through the API.

When we created a new bucket with versioning turned on from the beginning everything was as expected.

We therefore deleted the original bucket (by deleting all objects and versions first) all in the console and re-created it through the console with versioning turned on. We then found we got the same incorrect behaviour. So it seems like we have 'bricked' that bucket name. I was able to confirm there is no reference to the original bucket name on the file storage.

Another weird behavior is that objects uploaded through the API have this problem, but objects uploaded through the console web app do get a version ID for the first object version.

The only way to resolve the issue is to manually create the bucket using the command line:
mc mb --with-versioning local/mark-voffon-test Bucket created successfully 'local/mark-voffon-test'.

Expected Behavior

When you re-create a bucket with versioning, it should work when using the console web app. It should have the same behavior as the command line.

Current Behavior

Objects behave like they were first uploaded to a bucket with versioning turned off, and on a second object upload you get a version ID.

The console web app isn't re-creating a deleted bucket with versioning turned on the same as the command line.

Video link demonstrating the issue: https://youtu.be/lw41JBWDMYU

Steps to Reproduce (for bugs)

1. Create bucket with versioning turned off.

2. Upload file through API (we used c# and node scripts to test)

3. You will have file with versionID=null

4. Turn on versioning for the bucket

5. Upload same file through API

6. You should have versionId set by MinIO

7. Upload new file through API

8. You will have file with versionID=null (this is wrong, bucket should version new files from now on).

9. Delete files and bucket completely through console web app

10. Re-create bucket with versioning turned on from the beginning

11. Follow steps above

12. Behavior is still incorrect

13. Delete files and bucket completely through console web app

14. Create same bucket with mb command with versioning enabled

15. Follow steps above

16. Everything works according to the documentation.

Regression

I assume yes
We are running the latest version as of a few days ago.

Your Environment

We are running this as a Dockerised container in Azure Container Apps

• MinIO version used (minio --version):
  minio version RELEASE.2024-11-07T00-52-20Z (commit-id=cefc43e4daa4cbb490ef6726ea374e26a93eb85e)
  Runtime: go1.23.3 linux/amd64
  License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html
  Copyright: 2015-2024 MinIO, Inc.

• Operating System and version (uname -a):
  Linux rdrive-minio-ea--fh4gz0c-768ff885d5-8hxtt 5.15.164.1-1.cm2 Profiling endpoints for mcs #1 SMP Sun Aug 18 19:16:21 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

[ramondeklein]

I have created the following script (it assumes alias test and will destroy the bucket test):

mc rb local/test --force
mc mb local/test
echo "test 1" | mc pipe local/test/test-object
mc version enable local/test
mc ls --versions local/test
echo "test 1" | mc pipe local/test/test-object
mc ls --versions local/test

This is the output:

Bucket created successfully `local/test`.
 0 B / ? ┃...┃7 bytes -> `local/test/test-object`
local/test versioning is enabled
[2024-12-05 11:19:35 CET]     7B STANDARD null v1 PUT test-object
 0 B / ? ┃...┃7 bytes -> `local/test/test-object`
[2024-12-05 11:19:35 CET]     7B STANDARD aafb8da3-d1dd-4ae8-af4e-dd25281eff37 v2 PUT test-object
[2024-12-05 11:19:35 CET]     7B STANDARD null v1 PUT test-object

Versioning works fine. I'm not even sure if you can actually disable versioning when uploading data, but could you tell me which SDK you are using and how you're uploading the object that isn't versioned? You also may want to post the trace mc admin trace -v <alias> when you upload the object.

[markive]

@ramondeklein to clarify, the issue I am experiencing only happens when the console web app is used, not the command line as per your test.

I have recorded a youtube video that illustrates the steps and where the inconsistency is: https://youtu.be/lw41JBWDMYU

Do let me know if you need any other information.

For my testing I am using the latest c# sdk for .net 8. But we have also tested the latest node sdk. We don't believe this is the issue, but there's something wrong with the console web app.

Do let me know if I've posted this issue to the wrong repo. Thanks!

[marktheunissen]

Hi @markive I've followed your steps and cannot reproduce, yes I used the console. Could you please repeat the steps usingmc instead of your custom code? See below. Also, you can follow the verbose server logs using mc admin trace -v ALIAS and look out for calls to PUT /bucket/?versioning= with XML bodies like <VersioningConfiguration><Status>Enabled</Status><ExcludeFolders>true</ExcludeFolders></VersioningConfiguration>

And also calls when you use your SDK code vs. mc client. Can you spot any difference to the call bodies?

Create bucket with versioning turned off. ✅
Upload file through API (we used c# and node scripts to test) ✅

mc cp sample-cradle.jpg local/mybucket/sample-cradle.jpg

You will have file with versionID=null ✅

❯ mc ls --versions local/mybucket/sample-cradle.jpg
[2024-12-06 16:50:49 AEDT] 142KiB STANDARD null v1 PUT sample-cradle.jpg

Turn on versioning for the bucket ✅

Done in console

Upload same file through API ✅

❯ mc cp sample-cradle.jpg local/mybucket/sample-cradle.jpg

You should have versionId set by MinIO ✅

❯ mc ls --versions local/mybucket/sample-cradle.jpg
[2024-12-06 16:51:18 AEDT] 142KiB STANDARD de72f05e-e9e4-44f2-b3f2-94377b963ae7 v2 PUT sample-cradle.jpg
[2024-12-06 16:50:49 AEDT] 142KiB STANDARD null v1 PUT sample-cradle.jpg

Upload new file through API ✅

❯ mc cp sample-cradle.jpg local/mybucket/sample-cradle-2.jpg

You will have file with versionID=null (this is wrong, bucket should version new files from now on). ❌

❯ mc ls --versions local/mybucket/sample-cradle-2.jpg
[2024-12-06 16:51:29 AEDT] 142KiB STANDARD c7581aae-8045-41d2-aeac-f971b656e842 v1 PUT sample-cradle-2.jpg

As you can see above the version id is correct.

[marktheunissen]

You can also compare the API calls made by the console and the calls made by the mc command in your mc admin trace and see that they are the same. In fact it's the same client library (minio-go) so I don't see how enabling versioning on the bucket in the console could make a difference.

[marktheunissen]

We are running this as a Dockerised container in Azure Container Apps

Just run it locally and see if you can reproduce on local machine with minio server ./data or whatever

[markive]

Hi @marktheunissen, thanks for looking into it. I don't have a linux system other than a containerized environment so I don't know how I can run mc locally to test that. I could run it under windows, would that be a fair test?

I think focusing on the sdk shouldn't be the issue (we also tried with Node). Since versionId is controlled by minio, it should not be possible to successfully store an object (200 response) with versioning turned on through any means which results in versionId=null.

This is the request made by the .Net SDK, it's a PUT always, and in fact all the SDKs minio provides don't support POST:

Here is the API request that the console web app makes, it uses a POST even when uploading new versions of an existing file:

[markive]

I think the weirdest part of the behavior we are seeing is, why when a bucket is re-created does versioning not work, but when it is created for the first time with versioning turned on does it work perfectly? That's very inconsistent behavior and I'm 100% using the console web app for those operations.

[marktheunissen]

You can run mc on Windows and it just connects to any MinIO service that can be reached on the network

[marktheunissen]

why when a bucket is re-created does versioning not work, but when it is created for the first time with versioning turned on does it work perfectly?

Is this reproducible through mc commands? Like mc mb to make a bucket, etc? If you could provide that it would be very helpful. It might be that some bucket metadata is not cleaned up but we would need a script or something that clearly shows this.

[cesnietor]

@marktheunissen could you please try to reproduce this on the UI and with MC? and identify where the issue is happening. Thanks.

[marktheunissen]

@markive Can you please run MinIO on your local machine and see if this is reproducible with mc?

[ramondeklein]

@marktheunissen Did you try to reproduce this? It's a relative quick check for us to reproduce this issue.

[marktheunissen]

@ramondeklein Yes. You can read the steps I took in the comment I made in this ticket. Here is a direct link: #3483 (comment)

OP @markive did not provide the custom code used in the original bug report. @markive also said it didn't matter because it was happening with nodejs and C#.

So, without the custom code, I just used mc to perform the steps as given in the "Steps to reproduce" in the bug report. I could not reproduce.

Neither could @cesnietor.

Please be specific about what additional things you would like me to do, to try reproduce, given that I've already done the steps and shown the result above (#3483 (comment))

[marktheunissen]

#3483 (comment)

In this comment, @markive is trying to compare the Console web backend API, with the S3 API, and trying to reason about the differences - however this is a red herring, as these are two different APIs and it doesn't matter that one is PUT and one is POST.

[marktheunissen]

So it seems like we have 'bricked' that bucket name. I was able to confirm there is no reference to the original bucket name on the file storage.

This should not be possible with a properly functioning filesystem, and it does not reproduce in my SNSD testing. We don't keep previously used bucket names on a list or anywhere that might result in them becoming "bricked"

How did you confirm that there was no remaining reference? Did you look for all hidden files and folders including in /.minio.sys?

I think the weirdest part of the behavior we are seeing is, why when a bucket is re-created does versioning not work, but when it is created for the first time with versioning turned on does it work perfectly? That's very inconsistent behavior and I'm 100% using the console web app for those operations.

I went through just this specific part of the steps again. This does not reproduce. Recreating a bucket with the same name works as expected.

@ramondeklein note the environment being used here: "We are running this as a Dockerised container in Azure Container Apps"

The filesystem/environment is my primary suspect for this behavior, I don't know much about Azure Container Apps but it's like a serverless Lambda-type environment.

Is it spawning multiple instances of the MinIO backend? Maybe it has created 2 or more replicas of MinIO expecting them to be stateless, and is simply load balancing requests to them. So the behaviour is weird and seemingly random, because you're actually talking to two or more different independent MinIOs? We just don't know, which is why I asked for reproduction on a SNSD first to isolate the issue

Unless @markive comes back with more info I would close this as user environment issue.

[markive]

I am traveling today, but I will respond asap, it could be an environment issue, but Minio console/API/mc is not behaving consistently on it, or throwing errors, so it's worth pursuing, I think the environment I have set it up is a modern natural environment set of choices for people using Azure.

[marktheunissen]

Thanks

I think the environment I have set it up is a modern natural environment set of choices for people using Azure.

Maybe for general web apps on Azure, but it's not a supported or recommended environment for production deploy of MinIO.

[ramondeklein]

@marktheunissen In my previous assignment I have used Azure Container Apps too and it's just a bunch of Docker containers, but the persistent storage part is the part where it becomes interesting. But this seems like it's an Azure Containers Apps issue not playing nicely with MinIO.

@markive Although we would love to make it work on all platforms, we simply cannot check each platform that our (open-source) customers are using. We simply lack the resources to do so. I hope you understand, but it would take too much time to recreate this on Azure Container Services ourselves and ensure we use the same as you have. This works fine using vanilla Docker and/or Kubernetes (also Azure's AKS), so we will close this issue.

I do have a lot of Azure experience from the past. If you can provide me a Terraform script (or ARM template) that spins up an environment in a single namespace and provide the steps to reproduce, then I could take a look. For now I'm closing this issue...