cPanel & WHM Version 82 has been released, and brings a slew of great updates. Take a look at what is included, and then upgrade today!
 

Ability to control retention for S3 Backup Destination

Sebastian shared this idea 5 years ago
Open Discussion

The new S3 Backup Destination is very helpful - thank you!


However.... using it in production is expensive. Each day we add nearly 100GB of new backups to S3, and yet there is no provided option to remove old backups.


Can we please have a way to manage retention on Additional Backup Destinations?

Comments (17)

photo
4

I would like to keep only the most recent backup on my local server, and a weeks worth on the remote server.

photo
2

Much needed. We have plenty of free space in the remote storage there would like to keep many backups, but only limited space locally.

Ideally, we want to keep 2 daily backups locally (for fast access) and 7 daily/4 weekly remote.

photo
3

The cost of storage is vastly different as is the need to store for different time periods. I would love to be able to store long term offsite files and a few onsite files. I think this feature is well worth it.

photo
2

You can set up auto-delete rules within Amazon S3. They call it Lifecycle Management. Set the rules up on your whole bucket, or individual folders for when files should be automatically deleted (e.g. 35 days old).

Instructions are here:http://docs.aws.amazon.com/AmazonS3/latest/UG/lifecycle-configuration-bucket-no-versioning.html

photo
1

This is useful. Thanks for the link! All I need now is to be able to automate conversion of items to "reduced redundancy".

photo
4

I noticed that my local retention rules are being applied to S3.


Example:

I set WHM to retain 7 daily backups (they are stored on a separate drive) and also transfer them over to S3.

My S3 bucket has a lifecycle setting that permanently deletes backups after 31 days.


I would like to be able to set remote retention rules to a different value than local rules, not the same!

photo
1

This is important. Local and remote resources are not always the same and bandwidth costs are an issue for some.

photo
1

I need option for only monthly backup to FTP.

photo
1

Custom retention for S3 would be nice, but cPanel does already apply local retention rules to S3. Only the number of day's I've specified to retain backups are being retained in S3.

photo
1

Retention can be quite complex (as versioning affects it too) but is easily managed per bucket using rules.


I think it's a bad habit to delete backups from S3 manually (as WHM does with its retention). Also it wouldn't be useful to be able to change the rules for S3 retention (or glacier archiving and versioning for that matter) unless the policy set in AWS has the leading role, eg. when you change the policy in AWS then those changes will reflect in WHM also. Even then, Amazon is rapidly developing new features all the time like Glacier, encryption and even more retention options. To be useful (and prevent conflicts) those all need to be implemented in WHM and kept up to date.


It's better to set it once, configure it well and forget about it. If you watch their news you'll see when a new feature has been implemented and whether it's worth using it in your policy.


Let's not forget there still isn't support for the not even so new authentication system that uses AWS4-HMAC-SHA25 (protocol v4) and is the only available authentication type in newer datacenters. More important things have to be done right now.

photo
2

People pushing AWS retention rules don't really get one of the problems.

On local server i don't have unlimited storage, so i have to set cPanel to delete backups after 1-2-3-x days.

On S3 i have unlimited space, but cPanel will still delete them based on local delete rules ( 1-2-3-x day), so i can say goodbye to AWS retention rules.

Also i don't really care what AWS has/can do.

You could setup Apache/PHP on your own, yet if you are using cPanel you expect to do them for you.

cPanel's backup should be a standalone application which doesn't require me to login to other places, and and add some manual config.

photo
1

You always can push cPanel for a feature request but you need know how to work around your issue.

For instance, my cPanel backup retention is set to 1 day and I keep 30 days of backups in S3.

Like I said before, you just need to enable versioning and object life cycle in the bucket policy.


With versioning, If you delete an object, instead of removing it permanently, S3 inserts a delete

marker, which becomes the current object version, then you set the life cycle of objects to expire in 30 days. Now you can choose what to do with expired objects, delete or archive them in Amazon Glacier.

photo
1

How are you getting this to work? Because I can't. I enabled versioning and lifecycles but my weekly backups still disappear. At the moment, cPanel is storing my weekly backups in an S3 "Backups" bucket. It puts the backups in a "weekly" folder, itself containing a sub-folder which changes name every week, for example it' s now called Backups/weekly/2016-09-20. I have versioning enabled and objects set to be moved to the Glacier storage class after 6 days (i.e. before cpanel gets a chance to overwrite the "weekly" folder). Even so, every week my remote backups are removed and I can no longer see the previous version. I have no option to "initiate restore" on any single object, the option is greyed out, presumably because S3 doesn't see the object as a new version of the old one, but as an entirely new one (?) After all, it has a different name: it's no longer "https://s3-eu-west-1.amazonaws.com/backups/weekly/2016-09-13/accountname.tar.gz", it's now "https://s3-eu-west-1.amazonaws.com/backups/weekly/2016-09-20/accountname.tar.gz" so as far as S3 is concerned, it's a completely different file. What am I missing here? Because it sounds like it's working for you. I badly need my backups retained for more than 1 week and obviously the ability to retrieve them from the Glacier. At the moment I am not even sure if they have been moved to the Glacier at all, because you can't just "browse" the Glacier, you have to initiate a restore job, and as I said I have no such option available. Thanks.

photo
5

We have set up our S3 bucket with a few rules to make it easier to manage costs. This can be done on a bucket-level inside the S3 management console:


* Move files to glacier after two weeks

* Remove files after 100 days


The move to glacier helped us greatly reduce cost while still retaining backups for a long time.

photo
2

Is there any update on this? And any changes to this should affect not just S3 backups, but other off-site backups as well.

photo
1

No update yet, but we are definitely aiming to spend some significant time on the backup system in 2017. As soon as anything specific to this is done, I'll let everyone know!

photo
1

This would be ideal. We run a VM running on limited SSD storage. We only have enough space to keep 3x backups locally and ideally we want to keep them on S3 for longer. Obviously we need more than 3 day's worth of backups so it involves manual work to retain them. The ability to set custom retention periods for S3 would be ideal.

photo
1

This would be a nice convenience feature. Right now I just have a bucket policy erasing all backups older than x number of days. It works but is not as "friendly" as a built-in cPanel feature.

photo
1

I'll expand the original request to simply be that remote destinations should be able to override the default retention with a custom one. For instance, we are testing adding B2 but S3 has been our main backup destination for the last few years. Our retention is 5 dailys, 3 weeklys, and 3 monthlys. Local backups are deleted due to lack of space. I want to add B2 (since it's so cheap) instead of replacing S3 since I want more redundancy. But it would be nice to leave the normal retention in place for the new B2 but switch S3 over to only accepting and retaining dailys (to shave cost). Being able to override retention policy at the destination level makes sense to me and allow for tiering across unrelated destinations. The comments above about being able to 'program' the behavior within S3 & Galcier is noted, but I want my server to manage it, not the storage provider.