Automatically delete old files from AWS S3

aws s3 automatic cleanup

We usually store a lot of files during our work. Fast internet and data availability support this very much. Unfortunately, a very large part of the data we collect is unnecessary or out of date after some time. Some people view their datasets and periodically delete unnecessary things.

What about the cloud? Since the price for storage has become very low, many of us collect data not only at home, but in the cloud. Why?

Because it is cheap, convenient and we have access to it from anywhere in the world 🙂

Nowadays, we need to filter the data well to find something that is useful to us. If we don’t, we’ll be drowned in a flood of information. With the stubbornness of a maniac, we collect the necessary data, keep it, and after some time it becomes unnecessary and obsolete.

The same applies to logs stored in the cloud. Storing them is also cheap and convenient, only … only logs also have validity. Let’s not keep data longer than we have to. For some systems it will be 30 days, for others it will be 90 days, and for some systems it will be a year or several years, if we are required to do so by law and regulations.

If you have any data in AWS that you would like to automatically delete after a certain period of time, then this article is for you !!

Automatic deletion of data from the entire S3 bucket

On the AWS (Amazon Web Service) platform, we can easily automatically delete data from our S3 bucket. We open Amazon S3 and select one bucket from the list, on which we want to enable automatic deletion of files after a specified time.

amazon web services s3 - simple storage service

Go to Management and click Create lifecycle rule.

aws s3 - management

We give the name of our rule. Select the option saying that our changes are to apply to all objects and select the checkbox that appears. We must be aware that all objects older than the number of days listed below will be deleted.

s3 - create lifecycle rule

If we want the files to be deleted after 30 days, select the “Expire current versions of objects” option and enter the number of days after which the files will be deleted.

aws s3 - lifecycle rule actions

In summary, we should see something similar to the picture below. When everything is correct, click on Create rule and our automatic file deletion rule will be ready.

aws s3 - timeline summary

Automatic deletion of data from one folder from S3 bucket

Everything works nicely, but what if we wanted to have different retention set for each group of files? We won’t be creating a few buckets for every flu. Managing so many buckets would be a nightmare. There is an easier way. We can limit the deletion of files to a specific folder or subfolder only.

I strongly recommend that you check this option on a test bucket if you are just learning, or make a copy of the bucket you are implementing it on, just in case.

If you make a mistake, you could lose your data.

If you are sure of what you are doing then you can limit file deletion to a specific folder only.

The whole thing is very similar as before, only the difference is that this time you select the “Limit the scope of this rule using one or more filters” option and enter the name of the folder to which the rule is to be created.

aws s3 - create lifecycle rule prefix

Summary of Lifecycle rules

amazon web services s3 - podsumowanie lifecycle

These are the basics, we can extend our automation much more. The basics will help you avoid unnecessary expenses and keep order by automatically deleting old logs or outdated data.

If the article has given you some value, or you know someone who might need it, please share it on the Internet. Don’t let him idle on his blog and waste his potential.

I would also be pleased if you write a comment.

If you do not like to spend money unnecessarily, I invite you to read other articles on saving money in the cloud.

Leave a Reply

Your email address will not be published. Required fields are marked *