/lib64/libc.so.6: version `GLIBC_2.34' not found Error when restoring physical backup on ARM Servers

[kv-dhyey-moliya]

Report

I am seeing following error on mongodb-operator logs and when restoring backups on ARM Servers.

2024-09-19T06:57:35.443Z	ERROR	Reconciler error	{"controller": "psmdbrestore-controller", "object": {"name":"restore-2024-09-19-03","namespace":"mongodb"}, "namespace": "mongodb", "name": "restore-2024-09-19-03", "reconcileID": "add65b59-601a-4ad0-81fa-0ade99c22894", "error": "reconcile physical restore: resync config stderr: /opt/percona/pbm: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /opt/percona/pbm)\n/opt/percona/pbm: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /opt/percona/pbm)\n stdout: : command terminated with exit code 1", "errorVerbose": "command terminated with exit code 1\nresync config stderr: /opt/percona/pbm: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /opt/percona/pbm)\n/opt/percona/pbm: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /opt/percona/pbm)\n stdout: \ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).reconcilePhysicalRestore\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/physical.go:134\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/perconaservermongodbrestore_controller.go:200\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1222\nreconcile physical restore\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/perconaservermongodbrestore_controller.go:202\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1222"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222

More about the problem

This error only happens with ARM Servers, and does not happen with AMD64 Servers.

Steps to reproduce

1. Create a MongoDB cluster with ARM Architecture nodegroup (I used t4g.xlarge)
2. Create a manual backup on S3 storage using type: physical.
3. Restore the backup. The error will be logged in operator logs.
4. Not able to perform any further operations (stop/start/backup/restore) on the cluster after this point.

Versions

1. Kubernetes - 1.29
2. Operator - 1.17.0
3. Database - 7.0.12-7
4. PBM - 2.5.0 and 2.6.0

I tried with pbm image percona/percona-backup-mongodb with versions 2.5.0-multi and 2.6.0-multi. Both versions have this issue.

Anything else?

Workaround to this error,

1. Manually delete the whole cluster (data will persist on the PVC and backups will persist on S3 storage)
2. Re-deploy the cluster, but with AMD64 Architecture nodegroup (I used t3.xlarge)
3. Perform the restore operation again. It will succeed without the error.
4. Pause the cluster, and change the nodegroup to ARM Architecture (I used t4g.xlarge)
5. Resume the cluster.

[hors]

Hi @kv-dhyey-moliya, did you use 7.0.12-7-multi image as well? Like psmdb was started on ARM nodes. Am I right?

[kv-dhyey-moliya]

Yes I used "multi" images, with ARM nodes.

[kv-dhyey-moliya]

The db works perfectly on ARM, and logical restore also works perfectly, only problem is physical restore there I am getting this error.

[hors]

@kv-dhyey-moliya It is a pbm issue. PBM arm was built using el9 and glibc 2.34. PSMDB image was built using el9 and glibc 2.28. When the operator performs a physical restore, it copies the pbm binary from the PBM image to the psmdb container (the operator runs special pods for restore). That is why it can't work with psmdb container.
Logical backup and restore should work. It will be fixed in the next release of PBM 2.7.0. I think we will have a release next month.

[kv-dhyey-moliya]

Thanks for the detailed update. 🙌