 Welcome to my talk my name is a lot of back and today I'd like to talk to you about Why and how you should nuke your AWS account? Let me start with a short survey By a raise of hands. How many of you ever forgot a running easy-to-instance in AWS? Okay, so you can see it's a shared problem. Hopefully after this presentation you can buy one of these from the money I'm gonna save you Of course, I mean the Jack it's not the car So I work on a project called open shift cluster manager Ocm for short we installed a lot of open shift clusters on GCP and AWS mainly on AWS because of Red Hat's offering and many of them sometimes Because we test a lot we use a lot of versions sometimes clusters just fail from the leading and we leave them running in AWS so try to delete and running open shift cluster in AWS you have to deal with a lot of dependencies. You may have seen it if you ever try to delete an easy-to-instance You delete it you forget to delete the volume or the disk or anything else So try to understand all the topology of all the resources in AWS It's kind of hard and you need to understand whatever resource depends on another resource And you need to be aware of the right ordering of deletion. So for example when I started doing this manual work I tried to delete a VPC for example and deleting a VPC blocks blocks me from I'm sorry deleting VPCs blocked by deleting the subnet and then I couldn't delete the subnet because the IP address was Being used so it became a real problem and when I first started Doing this manual work. I figured well, I should better automate it because I'm doing it more frequently and Once I did it over and over again, I understand I understood that I also need to handle multiple regions and I need to of course be aware of all the Dependency graph and I need to repeat it again and again. So this became kind of a agonizing task and I Thought to myself. Well, I should probably avoid reinventing the wheel in this case and I thought somebody probably already resolved this problem before So I did some research and I found there were actually several tools for handling this kind of situations And one of the best found the best one I found was AWS nuke, which I'd like to share some thoughts about With you today. So what was interesting? I went online on their github? repository and it's an open source project and the Interesting part was that they're dealing with kind of the same problem that we're dealing with only with Kubernetes clusters so they're installing Kubernetes clusters and Having the same problem with cleaning up their AWS account and we are dealing with with open shift clusters So it was nice to see and I'd like to share with you how you can Use that tool to clean up your AWS account and reduce your spend So to use it you start with Defining a config file, which is basically a file. You set up which regions you want to clean Some AWS resources are tied to a specific region. Some are not our global So we just write global as a region and then you list the accounts that you like to clean And then you just type AWS nuke minus C and then a path to the config file and then minus minus Profile and that contains the AWS profile That the AWS nuke would be using to connect to your AWS account and clean things up So if you download and run this tool, you would see an output which kind of looks looks like this What AWS nuke would be doing it would be connecting to your AWS account It will scan all the resources that you have on your AWS account And it will list whatever it find finds for which is candidate for deletion and by default It will not clean anything. It will just list all the resources if you do like it to do a cleanup You would need to add a minus minus no dry run flag that will prompt you if you like to Actually do the cleanup process and if you hit yes, then it's gonna do the actual cleanup Do take in mind that some resources require some waiting for them to get deleted So what the AWS nuke would be doing it's gonna try delete some resource if it's gonna fail or Find it again on another scan It's gonna reattempt deleting it and it will try over and over again until it deletes everything that you asked it to delete There's also sorts of filtering options for that tool like in my example I'm trying to delete only for example ELBs or EC2 instances as three buckets So I can list them all under the target section of the Config file I can also ask the AWS nuke to filter out some resources So for example in this specific example, I'm avoiding cleaning up my own credentials My own admin credentials to the account so I can use that also as filtering out These kind of resources. There's other types of filtering options like by value by regular expression date I encourage you to go online and read about them So we started integrating with this tool and everything went Buttersmooth We have a nightly test suite that runs overnight It takes roughly two sometimes three hours to run and then when developers came in the morning they Come they see the results and by that time the cleanup process was cleaning all the still resources On the AWS account and it worked good for a while and then we hit the wall What happened was that developer came in the morning. He saw some environmental failure Sometimes it was something that a developer fixed and they wanted to rerun the test suite now imagine Tests are running creating clusters on AWS and at the same time that the cleanup process was deleting them So obviously that became a problem So we started thinking how can we avoid deleting resources that are actually being used at the same time And we figured we couldn't really know if a resource in AWS is actually being used So we started thinking how can we avoid this problem and One of the things that came to mind was well We don't want to rely on the test side the test can fail it can crash It can run from my laptop and I can be offline. I couldn't really rely on asking are you using this resource? There's no one to talk to at that point. So We figured well the test runs for two three hours sometimes four But never more than five hours. So if the test or goes over five hours, it will time out and will terminate So we figured everything that is older in AWS older than five hours. We can safely remove it We have a dedicated AWS account for the test So it's it's safe to remove the problem was that some AWS resources has a field of creation timestamp, but not all of them and We want to overcome this problem and we figured how about we'll add a first seen timestamp and while we are scanning the resources and if we do it frequently enough we can add this tagging to each AWS resource and Running this frequently enough the first seen timestamp will be pretty much close to the creation timestamp of that resource So what we are using? We are using the AWS client that the AWS nuke is using to connect and talk and scan the AWS account We are using that client to also tag the resources while it scans them So the AWS nuke will be connecting to the AWS account It will start scanning all the resources while doing so it will tag first seen timestamp on all of them and then Once it finds a resource that is older than five hours It just adds it to the resources that our candidate to get deleted and that worked great for us we are Working with this method for a while it reduced our spent from looking like this every time we hit a quota issue or We got an alert from AWS that we're spending too much money And then developers came and started manually deleting stuff We went to this so it's very lower lower spent and much more Untrendy I would say So let me summarize one thing would be Trying not to invent the wheel If you're working on a business problem Or another problem that is not directly related to your business. Maybe someone else already solved it AWS nuke is a great tool for cleaning up AWS resources It's open source. So you can contribute as well. We are contributing to this project and then everybody Benefit from that So I encourage you to look into it I'll take any questions now So we have some filtering according to time Oh, yeah, so the question was there's all sorts of filtering options in the AWS nuke one was The question was if there's any options to filter by text. So there is an option to filter By value for example and in this case This one's for a gap for example, this is a filter that is Filtering out my user. So it's basically a text filter Without tagging Okay, so the question was how to delete resources without tagging Okay, so if you don't want to use tags, you can consider one of the other options For example, you can use a regular expression or a date if you like There's many many options and if none of them suits you That's the fun of being an open source project. You can contribute to it really and and we will benefit from that as well Any other questions? Okay, guess not. Thank you