efs-plugin - Can't list or write files to EFS mounts

[ba1ajinaidu]

/kind bug

What happened?

I have a liveness probe which creates and deletes temp file in a EFS mount point, it's failing frequently. Upon checking I found I can't run ls or create any new files in the EFS mount point( It just hangs, have checked the troubleshooting docs but no luck). Restarting the pod where EFS is mounted and efs-csi-driver pod doesn't help. efs-csi-driver pod for that node has these logs

Defaulted container "efs-plugin" out of: efs-plugin, csi-driver-registrar, liveness-probe
I0923 14:21:41.286610       1 config_dir.go:88] Creating symlink from '/etc/amazon/efs' to '/var/amazon/efs'
I0923 14:21:41.287573       1 metadata.go:65] getting MetadataService...
I0923 14:21:41.290268       1 metadata.go:70] retrieving metadata from EC2 metadata service
I0923 14:21:41.300072       1 driver.go:150] Did not find any input tags.
I0923 14:21:41.300455       1 driver.go:117] Registering Node Server
I0923 14:21:41.300626       1 driver.go:119] Registering Controller Server
I0923 14:21:41.300710       1 driver.go:122] Starting efs-utils watchdog
I0923 14:21:41.300891       1 efs_watch_dog.go:243] Skip copying /etc/amazon/efs/efs-utils.conf since it exists already
I0923 14:21:41.300980       1 efs_watch_dog.go:238] Copying /etc/amazon/efs/efs-utils.crt
I0923 14:21:41.301670       1 driver.go:128] Starting reaper
I0923 14:21:41.301833       1 driver.go:137] Listening for connections on address: &net.UnixAddr{Name:"/csi/csi.sock", Net:"unix"}
I0923 14:28:58.665766       1 mount_linux.go:244] Detected OS without systemd

Found other relevant issue here, but i'm not seeing any resource issue on the pod.

What you expected to happen?

How to reproduce it (as minimally and precisely as possible)?

Anything else we need to know?:

Environment

• Kubernetes version (use kubectl version): v1.29.7-eks-a18cd3a
• Driver version: helm chart - aws-efs-csi-driver-3.0.8, app version - 2.0.7

Please also attach debug logs to help us better diagnose

• Instructions to gather debug logs can be found here

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale