Option for delayed deletion of files from backing store

[ericmerrill]

We are looking at creating an option between delete immediately and delete never - more specifically, we are planning on implementing a 'Delete after X duration' feature.

The idea being, much like the 'never delete' option, that you use the bucket if you restore from a DB backup of a certain age. But we only keep snapshots a certain amount of time, and I'm never a fan of 'never clean up' setups. So we want a retirement strategy for those files, once the related DB snapshots are gone.

There are limitations of this. Namely, just like delete immediately, it is not compatible with multiple sites using the same backing store, and we will need to implement a table that tracks files that have been deleted from mdl_files, but are waiting to be deleted from the backing store. We will then have a cron task that uses that table to go and delete the actual files.
We will also need to check that table when adding a new file to the store, but if a hit is found, it means we won't need to re-upload it - just remove it from the to-be-deleted table.

We already have some developers looking at this, and out plan is to open a PR, but I wanted to open this discussion now, incase there are some glaring problems with this plan that either I'm missing, or would prevent it from being integrated into the community plugin.

[brendanheywood]

thanks @ericmerrill

Namely, just like delete immediately, it is not compatible with multiple sites using the same backing store, and we will need to implement a table that tracks files that have been deleted from mdl_files, but are waiting to be deleted from the backing store.

This is already done, this is in tool_objectfs_objects with a location state of OBJECT_LOCATION_ORPHANED

Almost all of the logic here should already be in place, the only difference is adding logic around whether it is done right now or done after X days. So I'd expect the patch here to be quite minimal

[dvdcastro]

Hello,

We have implemented this in #549

Let us know how we can help to move this forward.

[brendanheywood]

Well this one has slipped through the cracks for just over a year, we will be looking at this shortly

thanks @dvdcastro