PVC annotations prevent restore when CSI driver is different on destination Cluster

[rbasore]

What steps did you take and what happened:
I am restoring from a cluster with CSI Driver vmware to CSI Driver Netapp and the following annonations are being restored to the new cluster preventing Kubernetes from properly scheduling the PVC.
volume.beta.kubernetes.io/storage-provisioner: csi.vsphere.vmware.com
volume.kubernetes.io/storage-provisioner: csi.vsphere.vmware.com

What did you expect to happen:
I would expect velero to remove these annonations

Anything else you would like to add:
I am able to delete the PVC and recreate it and the restore works as expected

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[sseago]

It looks like we're already doing that right here: https://github.com/vmware-tanzu/velero/blob/main/pkg/restore/actions/csi/pvc_action.go#L153-L164

Are you running velero with the EnableCSI feature? Are you using csi snapshots/datamover or fs-backup?

It looks like the current logic only clears the annotations when we're using CSI snapshots or datamover, so this bug may be specific to fs-backup. It may be that clearing these annotations should be done in a regular pvc restore item action that doesn't depend on CSI being enabled (i.e. the fix here may be to move the removePVCAnnotations call from the csi pvc action to https://github.com/vmware-tanzu/velero/blob/main/pkg/restore/actions/add_pv_from_pvc_action.go

[rbasore]

I am using fs-backups, we deployed velero to use -node-agents so we could scale the amount of data that we have backed up. I do not currently have EnableCSI Feature enabled, I have tested with this feature enabled and still using file level backups. We are using the kopia uploader.

[sseago]

@rbasore ok, thanks. That's helpful. From looking at the code, I think we're only removing annotations when using CSI snapshots. My suggestion above to move the removePVCAnnotations call from the CSI PVC BackupItemAction to the generic PVC BackupItemAction should resolve this. So it looks like this is an actual Velero bug here -- but it's one that should be relatively easy to fix.

[rbasore]

Can you give me a little more information on how I can get that implemented in my environment? We are deploying the code with HELM today.

[sseago]

Can you give me a little more information on how I can get that implemented in my environment? We are deploying the code with HELM today.

The fix will need to be applied to the velero BackupItemActions in the velero codebase to be included in a future Velero release. We're right at the Feature Complete stage for Velero 1.16 -- this is a simple enough bugfix that we should be able to get it into 1.16.

[rbasore]

Thank you for the update. I look forward to upgrading to velero 1.16 :)