How to combine snapshots (incremental backups) with backup deletion because of TTL?

[sgielen]

Describe the problem/challenge you have
I created a velero Schedule earlier that would perform a full volume backup every day, with a TTL of 3 days (72 hours).

This has been running for quite a while, but I just found out that all three stored backups are much smaller than the volumes themselves.

I found out that this is because a Backup created from a Schedule is incremental by default.

This means that my backups are useless: after 72 hours, the initial full backup is deleted by Velero, and the subsequent snapshots are worthless. In my opinion, this (default!) behaviour isn't clearly documented even now that I know about it.

How can I configure my Backups, even those created from a Schedule, to be non-incremental, or how can I force the plugin to create a new full backup before my the last full backup is deleted because of TTL?

Describe the solution you'd like
A method to create a Schedule for which the Backups are full, not incremental. Or even better, a configurable max age of the last full backup after which a new full backup is created instead of an incremental one.

For now, I will work around this by creating a CronJob in the cluster that simply runs "velero backup --wait", without using the schedules feature.

Anything else you would like to add:

Environment:

• Velero version (use velero version): Client v1.5.1, Server v.1.4.0
• Velero features (use velero client config get features):
• Velero-plugin version: ?
• OpenEBS version: v1.11.0
• Kubernetes version (use kubectl version): Client 1.19.2, Server 1.18.3+k3s1
• Kubernetes installer & version: k3os
• Cloud provider or hardware configuration: local arm64
• OS (e.g. from /etc/os-release): k3os

[sgielen]

My workaround is to create this CronJob which automatically creates Backups, just like a Schedule would, but without the backups being incremental...

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: full-volumes-schedule
  namespace: velero
spec:
  schedule: "37 4 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: velero
          restartPolicy: OnFailure
          containers:
          - name: full-volumes-schedule
            image: ## Velero image here ##
            command:
            - sh
            - -c
            - 'PATH="$PATH:/" velero backup create volumes-full-$(date +%Y%m%d-%H%M%S) --snapshot-volumes=true --ttl 72h --volume-snapshot-locations aws-bucket,default'

[mynktl]

Hi @sgielen
As of now scheduled backups are incremental backup only. However plugin can be extended to create full backup only for scheduled backup.

[sgielen]

Hi @sgielen
As of now scheduled backups are incremental backup only. However plugin can be extended to create full backup only for scheduled backup.

Thanks for your reply. Just to double-check: Are you aware that all backups in Velero have a TTL by default and that incremental backups become useless after this TTL expires for the initial backup?

[sgielen]

In other words, the current default plugin behaviour is very dangerous, as it gives a false sense of security. I would suggest to change default behaviour to make a full backup always, and then add support for incremental backups back in when it is safe to do so, e.g. when periodic full backup is implemented based on the TTL setting.

[mynktl]

Thanks for your reply. Just to double-check: Are you aware that all backups in Velero have a TTL by default and that incremental backups become useless after this TTL expires for the initial backup?

Yes, I'm aware of the backup expiry details. We have mentioned this information at https://github.com/openebs/velero-plugin#creating-a-scheduled-remote-backup also. Are you referring this document? or any other document?

I would suggest to change default behaviour to make a full backup always, and then add support for incremental backups back in when it is safe to do so, e.g. when periodic full backup is implemented based on the TTL setting

Existing default behavior of scheduled backup is incremental backup. If we change the default behavior then it will the existing users also. Instead of changing the default behavior, It will be better to add extra configuration parameter, to volumesnapshotlocation, to disable incremental backups for scheduled backups.

[mynktl]

Hi @sgielen
This one came up in the our discussion and we are looking at how best to support this feature. Will keep you posted!

[hergi2004]

Hello mynktl, any new or upcoming release to fix the issue with velero incremental or do we continue the path sgeilen mentioned above of cronjob. Thanks

[tcrst]

Thanks for your reply. Just to double-check: Are you aware that all backups in Velero have a TTL by default and that incremental backups become useless after this TTL expires for the initial backup?

Yes, I'm aware of the backup expiry details. We have mentioned this information at https://github.com/openebs/velero-plugin#creating-a-scheduled-remote-backup also. Are you referring this document? or any other document?

I would suggest to change default behaviour to make a full backup always, and then add support for incremental backups back in when it is safe to do so, e.g. when periodic full backup is implemented based on the TTL setting

Existing default behavior of scheduled backup is incremental backup. If we change the default behavior then it will the existing users also. Instead of changing the default behavior, It will be better to add extra configuration parameter, to volumesnapshotlocation, to disable incremental backups for scheduled backups.

What's that configuration parameter that disables the incremental backups on the volumesnapshotlocation?

[NicoG60]

Hi @mynktl , any update on this issue?
I'm currently spinning up a cluster using openebs and velero and would like to check if the CronJob workaround is still the way to go?

[miguelcostaUI]

My workaround is to create this CronJob which automatically creates Backups, just like a Schedule would, but without the backups being incremental...

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: full-volumes-schedule
  namespace: velero
spec:
  schedule: "37 4 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: velero
          restartPolicy: OnFailure
          containers:
          - name: full-volumes-schedule
            image: ## Velero image here ##
            command:
            - sh
            - -c
            - 'PATH="$PATH:/" velero backup create volumes-full-$(date +%Y%m%d-%H%M%S) --snapshot-volumes=true --ttl 72h --volume-snapshot-locations aws-bucket,default'

I was trying to use this cronjob to create periodic full backups but it is not working for me. Can't get it to do anything. I'm using the image "velero/velero:v1.11.0". The cronjob creates the jobs.batch but the pod does not run. Any advice on how to make this work or how to debug it?

[woloss]

I was trying to use this cronjob to create periodic full backups but it is not working for me. Can't get it to do anything. I'm using the image "velero/velero:v1.11.0". The cronjob creates the jobs.batch but the pod does not run. Any advice on how to make this work or how to debug it?

They removed shell from image. I tried v1.6.3 it does include bash, but looks like no velero cli, maybe something like v1.4.0 will work. Another way, make own image with shell included.

[saxcorp]

Hello, i prepare to use velero backup my full cluster to minio Storage. Is this problem is still active ?