Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

/lib64/libc.so.6: version `GLIBC_2.34' not found Error when restoring physical backup on ARM Servers #1656

Open
kv-dhyey-moliya opened this issue Sep 19, 2024 · 5 comments
Labels

Comments

@kv-dhyey-moliya
Copy link

Report

I am seeing following error on mongodb-operator logs and when restoring backups on ARM Servers.

2024-09-19T06:57:35.443Z	ERROR	Reconciler error	{"controller": "psmdbrestore-controller", "object": {"name":"restore-2024-09-19-03","namespace":"mongodb"}, "namespace": "mongodb", "name": "restore-2024-09-19-03", "reconcileID": "add65b59-601a-4ad0-81fa-0ade99c22894", "error": "reconcile physical restore: resync config stderr: /opt/percona/pbm: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /opt/percona/pbm)\n/opt/percona/pbm: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /opt/percona/pbm)\n stdout: : command terminated with exit code 1", "errorVerbose": "command terminated with exit code 1\nresync config stderr: /opt/percona/pbm: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /opt/percona/pbm)\n/opt/percona/pbm: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /opt/percona/pbm)\n stdout: \ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).reconcilePhysicalRestore\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/physical.go:134\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/perconaservermongodbrestore_controller.go:200\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1222\nreconcile physical restore\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/perconaservermongodbrestore_controller.go:202\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1222"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222

More about the problem

This error only happens with ARM Servers, and does not happen with AMD64 Servers.

Steps to reproduce

  1. Create a MongoDB cluster with ARM Architecture nodegroup (I used t4g.xlarge)
  2. Create a manual backup on S3 storage using type: physical.
  3. Restore the backup. The error will be logged in operator logs.
  4. Not able to perform any further operations (stop/start/backup/restore) on the cluster after this point.

Versions

  1. Kubernetes - 1.29
  2. Operator - 1.17.0
  3. Database - 7.0.12-7
  4. PBM - 2.5.0 and 2.6.0

I tried with pbm image percona/percona-backup-mongodb with versions 2.5.0-multi and 2.6.0-multi. Both versions have this issue.

Anything else?

Workaround to this error,

  1. Manually delete the whole cluster (data will persist on the PVC and backups will persist on S3 storage)
  2. Re-deploy the cluster, but with AMD64 Architecture nodegroup (I used t3.xlarge)
  3. Perform the restore operation again. It will succeed without the error.
  4. Pause the cluster, and change the nodegroup to ARM Architecture (I used t4g.xlarge)
  5. Resume the cluster.
@hors
Copy link
Collaborator

hors commented Sep 19, 2024

Hi @kv-dhyey-moliya, did you use 7.0.12-7-multi image as well? Like psmdb was started on ARM nodes. Am I right?

@kv-dhyey-moliya
Copy link
Author

kv-dhyey-moliya commented Sep 19, 2024

Yes I used "multi" images, with ARM nodes.

@kv-dhyey-moliya
Copy link
Author

kv-dhyey-moliya commented Sep 19, 2024

The db works perfectly on ARM, and logical restore also works perfectly, only problem is physical restore there I am getting this error.

@hors
Copy link
Collaborator

hors commented Sep 19, 2024

@kv-dhyey-moliya It is a pbm issue. PBM arm was built using el9 and glibc 2.34. PSMDB image was built using el9 and glibc 2.28. When the operator performs a physical restore, it copies the pbm binary from the PBM image to the psmdb container (the operator runs special pods for restore). That is why it can't work with psmdb container.
Logical backup and restore should work. It will be fixed in the next release of PBM 2.7.0. I think we will have a release next month.

@kv-dhyey-moliya
Copy link
Author

Thanks for the detailed update. 🙌

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants