 Hi, everyone. Thank you very much for tuning in today. Today's talk is about attacking terraform environments. By the end of this talk, you'll learn ways to attack and secure your terraform environments. This talk would cover a lot of things that probably not known before. One thing to say before we start, terraform by default is not as secure as you would think. You have to put a lot of work to secure it. What I'm aiming for is by the end of this talk is to learn about all the things that you can know so that you can secure your terraform environments. Let's start. A little bit about me. My name is Mazin Ahmed. I'm an application security and offensive security engineer. I founded Full Hunt. It's a security startup that solves the attack surface visibility problem for companies and organizations. Also, I'm an occasional but 1,200. I have been acknowledged by Facebook, Twitter, LinkedIn, Zoom, Oracle, and many other companies. Of course, I'm in love with cloud security. Anything related to cloud security and of course, different things related to security are my passion. That's why I put the effort into having this research made. The agenda for today is first, we're going to go over things, a background on terraform. What's terraform? How does it work? Why do we need to learn about all of this? Why is it important? Attack, victors, scenarios. Attack, victors, and scenarios. Of course, the most important part is the demo, which would be really cool. The recommendations at the end and then we'd have room for questions. First of all, what's infrastructure as good? Back in the years when cloud was starting, water was becoming a thing, if someone would like to deploy something, they would go to the console, take their time in deploying their resources from the console and if they would like to do this again and again and maintain that, they would have to log in every single time to do that, which is a bit of a painful thing and does not scale. That's why infrastructure as code became a popular thing in the DevOps and DevSecOps communities. Why? Because you would be able to write all of your resources and the cloud as code and then you would be able to deploy many times and at the same time, you would allow room for peer review so that the people in your team and different teams would be able to review it. It's amazing. It allows that part. Of course, one thing that you can say that once you have this being set up, you would be able to run it through CI, security scanning for compliance and you can automate many things without really having to have someone to review everything being deployed. Of course, when you deploy this in using Git, you would be able to have an archive of everything being deployed. It's just an amazing technology that everyone loves, hopefully. Who is Turfome? Turfome is an open source infrastructure as code software made by HashiCorp around 10 years ago. Users define their code in a language called HCL or HashiCorp language and then deployments and the entire infrastructure management happens through Turfome. Whenever you write code, you would pass it to a Turfome build binary and then it will read the state of the current infrastructure and then it would decide what changes should be done. Why I'm talking about Turfome today, instead of all of Apollomi or different cloud formation or different cloud orchestrator, because Turfome is the most popular IAC orchestrator in the planet. This year, a couple of months back, they have released Turfome 1.0. It has been a big thing for everyone that is working with Turfome and of course it passes 100 million downloads, crazy numbers. Another thing that is cool about Turfome, once you have it, is you can write the same infrastructure code and have it integrated with different providers. You can write code that allows you to deploy to AWS or Azure or GCP or anything else. It's not only these six technologies and six providers that are there. They support more than a thousand providers as of today. How you would be able to do this is, as a practitioner or a user, you would write your code. Then once it's ready, it would go through Turfome plan and Turfome plan is the way that Turfome would be planning your deployment and show you what would be changing. Then it would go through the apply. Once you click on a run apply, it would be deploying right to your cloud provider. The thing is, when you are running apply and plan on apply, this means that you are checking the drifts between the current code and what is there on the provider. If you are applying changes, this means that you have right access on different things on your cloud provider. Who would have this? It would be Turfome. Setting up Turfome would grant it high permissions that allows it to do everything that you need to have it to do. While it sounds curry, it's the default thing. We have to just set it up in a way that could be secure and not rely on the default things and try to think better about it. As I said, with great power comes great responsibility. A single mistake here can lead to the entire compromise of the cloud provider or the cloud account. Imagine your entire AWS account being held for ransom. If someone compromised it, they would go into your cloud provider. They would be backing up all of your S3 buckets. They would delete everything. They would have it somewhere else. Then they would ask you for ransom. Imagine this. Ransom is not a thing that you would see only on your internet network or in your laptops. It's something that can be done also in the cloud. It's still not a thing today, but I can imagine that this would be happening in the upcoming years. As I said about the Turfome states, the way that Turfome works is it would be storing all of the changes and all of the additions that it made in your cloud provider in a file that acts as a database called Turfome state. Every time you do a change, it would check it against that file and then would update it whenever there's anything. There are many ways to have your Turfome state being deployed. One thing is you can have it locally. Second way that is I think the most common way by everyone that I know is using a remote backend of S3 so that you would specify a bucket, you would specify a key, and then which region would you like to have this bucket at on. Turfome would be automatically creating this bucket for, would be automatically pushing the changes and the state file there, and whenever it updates, it would synchronize with that key. So attack number one or thing number one, if you have get object permission on the S3 bucket for sorting the state file, then you would have access to secrets, access keys, and database credentials. So everything that is being deployed for the organization through Turfome would have a copy of it in the state file. So there's something like an RDS database would have their credentials stored somewhere and that place would have a copy in the state file. Same thing if you create an IM key, access key, then you would have it there or any other secrets that you would define or the provider would be defining for Turfome. So whenever you get access to an organization and a cloud account, try to search for if they have a S3 bucket that looks like it's storing secrets for Turfome. It would have the entire instructions for you to go further in the environment. Another thing is states on Turfome Enterprise. Turfome Enterprise has, Turfome has an enterprise offering called Turfome Enterprise and the normal Turfome that people use is a CLI binary that people would be using for deploying infrastructure, but with Turfome Enterprise you would have SSO, team management states management in different ways and configurations and different ways for your integrations. And Turfome Enterprise is amazing, but still the same problem of storing the states files exists on Turfome Enterprise. And how it works is you would deploy Turfome Enterprise as an instance. It follows a sort of hosted model. Also they provide a cloud hosted model that they would be hosting it within an offering called Turfome Cloud. But let's focus on the self-hosted one that you can host in your infrastructure. The states can be stored in many ways. Postgres UL, AWS RDS, if we wanted this same instance desk and many more. If you get access to any of these places, these places would have access to the state file and then you can copy this state file inspected for secrets, use these secrets for gaining a step ahead. Another thing to say, of course we talked about Turfome Enterprise is deployed as an instance on the AWS account or in the cloud account. And if you compromise an instance, this is the best place for attack resistance. And there are many reasons for that. One, once it gets deployed successfully, no one really touches it. Another thing is upgrades and maintenance really happen. It's not a common thing that in a DevOps engineer we go in and log in and access it to the server to see different things. And this, something is broken. Then there is an actual upgrade that would happen. This is the only time that someone would go in and try to SSH to this instance. So this is one thing to add. And of course, because Turfome Enterprise would have permissions to deploy to your cloud environment, the company would have to provide it with proper permissions to deploy online. So it would have probably admin access to your AWS accounts or any account that is being used for. If you get access to this VM or this instance, then you would have high permissions in doing whatever on the infrastructure. You can create new users. You can backdoor users. You can read buckets, which is something that is not really common, but of course you can do that. And you can do everything. And one thing that is funny here is let's say that you are in engagement and the security team detected that the database was compromised because of the state files and all, they would ideally go and rotate the database keys. And how would they rotate the keys? They would rotate it through Turfome. That's a typical way, or one way actually. But if they updated to Turfome, then you would still have the latest keys being synchronized on the Turfome instance, which is really cool. Let me pause for water. And as I said, Turfome Enterprise has high permissions. And it's not something that you should really put on the internet. But there are hundreds of organizations around the world that have Turfome Enterprise exposed to the public. If there is a single zero day here, I'm pretty sure this would be a nightmare for all of these companies. These are filters that you can use for just checking it out by yourself. Okay, we talked about how you can use this in your advantage with when compromising an AWS account that has or an AWS user that has access to the S3 bucket that stores the state files. But let's say that you are not really able to compromise the AWS account and you get access to a developer machine and this developer have access to Turfome Enterprise or Turfome Cloud. What things can you do here? Turfome have exposed a full dedicated API that can be used for automating functionalities and tasks. This can go from everywhere from updating users to provision plans to add comments and everything. And one thing that they have is an API called state version output. And this state version output API allows you to download the state file that you would have needed to compromise an actual IM user or all of this. So now if it wasn't really possible to do this, you can just go to the API and ask it for your state file nicely and it would respond with the actual state file in plain text, which is nice. Of course, this would also contain sensitive data with clear text credentials and everything that could be fun. Going further, a couple months back, Alex K. released a blog post talking about Turfome plan RCE. It's a problem that Turfome has it for over the years, but it wasn't really talked about within the Turfome primates. This blog post has become a viral post about how this attack could be done in a practical way. And they were talking about how you can use an emulation provider or an external data source that could allow you to get code execution on the Turfome interface or the instance that is running Turfome. And this can result in the compromise of the entire infrastructure when someone submits a PR that is not approved. What you're going to do is if you find, let's say you landed on a developer machine and this developer machine has GitHub token that allows it to push PR on the account that is being in the repository that hosts the Turfome code and this one is integrated with the Turfome interface like the normal setup. What would happen here is if you as a developer check out a future branch and then push code with the malicious payload or the attack that talks about the malicious provider that you're going to talk about, then you will get code execution on that server. You would get shell access. And this is really simple and of course it's not really intended and this should never happen in real life but it has been applied between usability and security here. It's hard to find a way to secure it and at the same time it's hard to not allow something like external data sources to be used. Chris for Alex K for having this book was written and shared to us. And this is an example of an actual proof of concept. If you'd like to test it out and you are authorized to test it, go and copy the code that I have here on my screen. Of course the slides would be public on my website and my blog and just use this code and then push it in as a future branch and get the repository that is responsible for hosting the Turfome code and then see the magic happens. You would get shell access on that machine. Of course this is a big problem here but the thing is other platforms are also vulnerable. It's not only a problem for Turfome interface. Anything that relies on Turfome built would be vulnerable to this. Eric Osterman in 2018 tried to introduce a fix and it was rejected. The fix was to use to restrict users from querying Turfome plan within Atlantis to only a subset that is being specified. And this would grant security here and could be a temporary fix but it does not really resolve the actual problem from Turfome. Ideally you should not really be able to run code on Turfome interface instance and get full access there or get access into the machine or the instance that is running Atlantis. Or even the developer machine that runs Turfome plan. This is not the right thing. Moving ahead, we talked about the attack that you can upload and you can upload an external data source like this one to gain access there. But would it be fun if we can do this in a way that is not really detected from the PR? Let's talk about the evil provider attack. Publishing providers within HashiCorp's Turfome is automated. This is made to encourage publishers to push code to Turfome. But the thing is trust but verify. We cannot really just push code and have it being trusted to be used by the community and by Turfome without really verifying its content or even having a way to detect attacks here. So let's say for example that one of these providers or one of these 1000 providers that are being used by people it's not only these 608 providers have been compromised and they have been bagged or and you're going to talk about how. And if this happens and someone just say Turfome in it and Turfome plan, what would happen is the attacker would be compromising the company. It would be chaos. But you know I don't want to have this being as a theoretical attack that does not really happen in real world and someone is discussing it in public. I would like to show an actual proof of concept about this attack from zero to the end. So let's start. First, let's try to set up a provider. I set up a new GitHub account. I called it even Turfome. Of course I didn't want to use my personal GitHub account because I know it may get suspended. We're going to talk about this one in a bit. And then I developed a simple provider and this provider was pushed into even Turfome such as Turfome provider E. Of course this is my picture. Anyone can know that this is the code that I have. And then I backdoor this provider with this code. This is a simple provider that would grant me reverse CCP shell upon the execution to my server. And then I pushed it to the Turfome registry and it was accepted. It was approved. I had my category as cloud automations because it automates my hacking here. And I was showing a bit what happened. Once I had the provider being approved by Turfome, I made a PR to push it to somewhere. And it looked like this. It's just the same way that we would be pushing providers in our companies. And now the fun part is the demo. Popping shell in Turfome cloud hosted by Turfome. This is app to Turfome.io. And you would wonder from here is how is the detection here? Probably in a well-organized environment that is running a good EDR solution or antivirus solution, this attack would be prevented or detected, but it wasn't the case. I pushed the provider that I built into VirusTotal and no provider really detected it. And back to the attack that we showed here, pushing the provider that grants me a reverse CCP shell got executed correctly within HashiCorp infrastructure, which is really bad. Imagine what could happen afterward. I haven't really exploited this further. I only slept at this part. And I tried my best to make it as obvious to the security team to see how the team response would be like. And before I continue here, I want to say that this attack is against the terms of use by HashiCorp, running this in an account, my reslocked in account description. So how is the security team response? Within hours of my exploitation, they detected the evil provider attack and contacted me directly to chat about my findings. And then we discussed various ideas and thoughts to introduce a fix for the thing that was presented here. And HashiCorp informed me that they are really thinking about ways into having this being mitigated for their customers. For the time being, there is no patch. There is no fix to introduce. And there is no way to prevent this. I appreciate HashiCorp's efforts in handling and analyzing the research. One thing to add is I was planning to make the responsible disclosure the next day of my tests. And before even going to the next day, they were the ones who were contacting me. And I really appreciate them being this proactive. One thing to add here is they are also thinking about starting their bug bounty program. At the current time, they don't have a bug bounty program, but this is one thing that they also are planning to have. Now, going to the recommendations. First, be careful. There is, like you would have hard times in maintaining a secure from environment. So being careful is my mere recommendation here. Another thing is, why is it like this? Okay. Another thing is, when using SV as a remote backend, try to have a bucket access policy to prevent other users other than their from instance to have access to your instance. This way, you can prevent like unauthorized access in certain ways. Of course, there are bypasses for this solution, where if you have another user that has access to modify the the bucket access policy for this bucket, then they can delete the access bucket policy and then they can access the bucket here. But at least if you can use this as a protection, this would be a good way. Another thing is to continually update and review your terraform instance. It can be easily forgotten in the noise. And if there is an attacker or an actor that lands on your terraform instance, as like mentioned earlier, this is a great place for attack persistence. So it can be easily abused badly here. And then it can be easily for defenders to forget about it. So, yeah. Another thing is, be careful in permitting people to have right access, even feature like feature branch access for your terraform repository. Because it can lead to direct code execution and there is no way to pass it. I'm waiting for a patch to happen. We had a group and I were talking about different ways to detect the attack. Meanwhile, the actual patch is there. But it's just ways to detect it. There is no way to prevent that moment. Another thing is to maintain a terraform instance in an isolated VPC that no one really actually said on the Internet. Do not be exposed to the Internet. This is bad. The next zero day that would happen, someone would scan the Internet. I'm probably compromising organization because of this mistake. And it would be that. And I think this would be lastly, state files are sensitive. Like they contain a lot of data from database passwords to SSL keys, SSL certificates, access keys. Read them as sensitive data. And of course, a good idea is to set up CI check to detect rogue terraform providers. This would aid in the exploitation discovery whenever someone is trying to exploit your terraform setup. This is one thing to add as a defensive measure. One thing that HashiCorp security team asked me to add is the terraform class security model. This is a document that talks about different ways that you can use for modeling your terraform class security while at the same time it can apply for your terraform interface setup. And now we are reaching the end of the talk. Final thoughts. Terraform is amazing. I love terraform. It brings several security features when implementing IEC with terraform, but still with great power comes great responsibility. Protect your terraform environment. Stay safe and set up persistent monitor like full hunt to discover shadow IT, misconfigurations and mistakenly explore services. Of course, it would not really work out for all of the terraform attacks we talked here. But yeah, just one thing to add here. And lastly, the questions.