Not able to launch the minio UI ,it is giving the error "cmd.InsufficientReadQuorum"

[ravisingareddy]

Not able to launch the minio UI. Giving the below error in the minio pod.

Error: Storage resources are insufficient for the read operation .minio.sys/buckets/uiv-adapter-certs/.metadata.bin (cmd.InsufficientReadQuorum)

Steps followed.
1.Helm upgrade is done from RELEASE.2022-09-17T00-09-45Z to RELEASE.2023-09-30T07-02-29Z
2.Minio UI login is working fine.
3.now did a roll back again.
4.Minio UI is giving error after entering the credentials.
5.There are 4 minio pods running in the system.

Complete log is attached:
API: SYSTEM()
Time: 05:00:51 UTC 06/11/2024
DeploymentID: d560b84a-7d6e-4f18-a8eb-e5509848f20d
Error: Drive https://doc-uiv-minio-3.doc-uiv-minio-svc.doc-uiv.svc.cluster.local:9000/data/server, path (.minio.sys/buckets/uiv-adapter-certs/.metadata.bin) returned an error (decodeXLHeaders: Unknown xl meta version 2) (*fmt.wrapError)
20: internal/logger/logger.go:259:logger.LogIf()
19: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
18: internal/logger/logonce.go:135:logger.LogOnceIf()
17: cmd/erasure-object.go:526:cmd.readAllXL()
16: cmd/erasure-object.go:594:cmd.erasureObjects.getObjectFileInfo()
15: cmd/erasure-object.go:220:cmd.erasureObjects.GetObjectNInfo()
14: cmd/erasure-sets.go:932:cmd.(*erasureSets).GetObjectNInfo()
13: cmd/erasure-server-pool.go:795:cmd.(*erasureServerPools).GetObjectNInfo()
12: cmd/config-common.go:33:cmd.readConfigWithMetadata()
11: cmd/config-common.go:55:cmd.readConfig()
10: cmd/bucket-metadata.go:141:cmd.(*BucketMetadata).Load()
9: cmd/bucket-metadata.go:168:cmd.loadBucketMetadata()
8: cmd/bucket-metadata-sys.go:412:cmd.(*BucketMetadataSys).GetConfig()
7: cmd/bucket-metadata-sys.go:226:cmd.(*BucketMetadataSys).GetObjectLockConfig()
6: cmd/bucket-object-lock.go:48:cmd.(*BucketObjectLockSys).Get()
5: cmd/metacache-set.go:123:cmd.(*listPathOptions).setBucketMeta()
4: cmd/erasure-server-pool.go:1292:cmd.(*erasureServerPools).ListObjects()
3: cmd/erasure-server-pool.go:1205:cmd.(*erasureServerPools).ListObjectsV2()
2: cmd/bucket-listobjects-handlers.go:238:cmd.objectAPIHandlers.ListObjectsV2Handler()
1: net/http/server.go:2084:http.HandlerFunc.ServeHTTP()API: SYSTEM()
Time: 05:00:51 UTC 06/11/2024
DeploymentID: d560b84a-7d6e-4f18-a8eb-e5509848f20d
Error: Storage resources are insufficient for the read operation .minio.sys/buckets/uiv-adapter-certs/.metadata.bin (cmd.InsufficientReadQuorum)
8: internal/logger/logger.go:259:logger.LogIf()
7: internal/logger/logger.go:385:logger.CriticalIf()
6: cmd/bucket-object-lock.go:56:cmd.(*BucketObjectLockSys).Get()
5: cmd/metacache-set.go:123:cmd.(*listPathOptions).setBucketMeta()
4: cmd/erasure-server-pool.go:1292:cmd.(*erasureServerPools).ListObjects()
3: cmd/erasure-server-pool.go:1205:cmd.(*erasureServerPools).ListObjectsV2()
2: cmd/bucket-listobjects-handlers.go:238:cmd.objectAPIHandlers.ListObjectsV2Handler()
1: net/http/server.go:2084:http.HandlerFunc.ServeHTTP()
critical: "GET /uiv-adapter-certs/?delimiter=&encoding-type=url&fetch-owner=true&list-type=2&prefix=": {}
goroutine 15209957 [running]:
runtime/debug.Stack()
runtime/debug/stack.go:24 +0x65
github.com/minio/minio/cmd.setCriticalErrorHandler.func1.1()
github.com/minio/minio/cmd/generic-handlers.go:559 +0x2ea
panic({0x24d0ac0, 0x654c5f0})
runtime/panic.go:838 +0x207
github.com/minio/minio/internal/logger.CriticalIf(...)
github.com/minio/minio/internal/logger/logger.go:386
github.com/minio/minio/cmd.(*BucketObjectLockSys).Get(0xc013e34848?, {0xc013cbf2d5, 0x11})
github.com/minio/minio/cmd/bucket-object-lock.go:56 +0x1cc
github.com/minio/minio/cmd.(*listPathOptions).setBucketMeta(0xc0031ddb30, {0x512c000, 0xc0140260c0})
github.com/minio/minio/cmd/metacache-set.go:123 +0x66
github.com/minio/minio/cmd.(*erasureServerPools).ListObjects(0x7f7be3febf18?, {0x512c000, 0xc0140260c0}, {0xc013cbf2d5, 0x11}, {0xc013cbf322, 0x0}, {0x0, 0x0}, {0xc013cbf2e8, ...}, ...)
github.com/minio/minio/cmd/erasure-server-pool.go:1292 +0x272
github.com/minio/minio/cmd.(*erasureServerPools).ListObjectsV2(0x512c000?, {0x512c000?, 0xc0140260c0?}, {0xc013cbf2d5?, 0xd?}, {0xc013cbf322?, 0x11?}, {0x0, 0x0}, {0xc013cbf2e8, ...}, ...)
github.com/minio/minio/cmd/erasure-server-pool.go:1205 +0xdc
github.com/minio/minio/cmd.objectAPIHandlers.ListObjectsV2Handler({0x4cf48a8?, 0x4cf4898?}, {0x5128030, 0xc0140a02c0}, 0xc01408d200)
github.com/minio/minio/cmd/bucket-listobjects-handlers.go:238 +0x63a
net/http.HandlerFunc.ServeHTTP(...)
net/http/server.go:2084
github.com/minio/minio/cmd.httpTrace.func1({0x5128030, 0xc0140a02c0}, 0xc01408d200)
github.com/minio/minio/cmd/http-tracer.go:238 +0x1af
net/http.HandlerFunc.ServeHTTP(0x2426a20?, {0x5128030?, 0xc0140a02c0?}, 0x4?)
net/http/server.go:2084 +0x2f
github.com/klauspost/compress/gzhttp.NewWrapper.func1.1({0x5128030, 0xc0140a02c0}, 0x0?)
github.com/klauspost/[email protected]/gzhttp/compress.go:391 +0x347
net/http.HandlerFunc.ServeHTTP(...)
net/http/server.go:2084
github.com/minio/minio/cmd.maxClients.func1({0x5128030, 0xc0140a02c0}, 0xc01408d200)
github.com/minio/minio/cmd/handler-api.go:277 +0x4cc
net/http.HandlerFunc.ServeHTTP(...)
net/http/server.go:2084
github.com/minio/minio/cmd.collectAPIStats.func1({0x5128030, 0xc0140a0210}, 0xb3?)
github.com/minio/minio/cmd/handler-utils.go:356 +0x106
net/http.HandlerFunc.ServeHTTP(0x9a?, {0x5128030?, 0xc0140a0210?}, 0xc013e36530?)
net/http/server.go:2084 +0x2f
github.com/minio/minio/cmd.setBucketForwardingHandler.func1({0x5128030, 0xc0140a0210}, 0xc01408d200)
github.com/minio/minio/cmd/generic-handlers.go:472 +0x22d
net/http.HandlerFunc.ServeHTTP(0x5127fd0?, {0x5128030?, 0xc0140a0210?}, 0x10?)
net/http/server.go:2084 +0x2f
github.com/minio/minio/cmd.addCustomHeaders.func1({0x5127fd0, 0xc017653f08}, 0x4cf4860?)
github.com/minio/minio/cmd/generic-handlers.go:547 +0x3c2
net/http.HandlerFunc.ServeHTTP(0xc013e36d40?, {0x5127fd0?, 0xc017653f08?}, 0xa4?)
net/http/server.go:2084 +0x2f
github.com/minio/minio/cmd.setRequestValidityHandler.func1({0x5127fd0, 0xc017653f08}, 0xc01408d200)
github.com/minio/minio/cmd/generic-handlers.go:460 +0xe06
net/http.HandlerFunc.ServeHTTP(0x40?, {0x5127fd0?, 0xc017653f08?}, 0x64ac9f0?)
net/http/server.go:2084 +0x2f
github.com/minio/minio/cmd.setHTTPStatsHandler.func1({0x512a790?, 0xc01798b6c0}, 0xc01408d200)
github.com/minio/minio/cmd/generic-handlers.go:298 +0x119
net/http.HandlerFunc.ServeHTTP(0x10?, {0x512a790?, 0xc01798b6c0?}, 0x0?)

[prakashsvmx]

3.now did a roll back again.

does that mean it was downgraded back to old version ? downgrades are not allowed

adapter-certs/.metadata.bin) returned an error (decodeXLHeaders: Unknown xl meta version 2) (*fmt.wrapError)

This indicates downgrade.

[ravisingareddy]

you mean there is no rollback/downgrade supported for minio ?is there any other workaround to bring the minio pods back?

[prakashsvmx]

Yes. downgrade is not recommended
(may be only under certain conditions with strict supervision and confirmation from MinIO engineers via subnet.min.io ) !

[ravisingareddy]

thanks.is there any work around now to bring back to the normal state