 And thanks for having me. Really excited to present this year in the cloud village. I've presented in many cloud villages and DEF CON in general, but the cloud village is a new one for me. So super excited about this. As Jeff mentioned, the title of this presentation is identifying toxic combinations of permissions in your cloud infrastructure. So this is a mashup of a lot of research we've done over a few years now in doing risk assessments for organizations and largely focused around AWS, Azure and GCP. So we'll give you some examples, some more stories, things like that to really help you think more about a lot of these complex permissions in the cloud. And with that, I'll go ahead and jump right into it. So really won't talk too much about myself, but rather instead I'll just tell you a funny story, how I got into cybersecurity, you know, hanging out at DEF CON for many years that there's always really fun and interesting stories about how each and every person has gotten into cybersecurity. For me, I was actually working for one of the stock markets and at that time, I was a Unix system administrator in an Oracle DBA. This is around circa 1993. And I was always one of those folks that's always kind of tinkering around with stuff. And there was a tool from Dan Farmer. You may remember Dan in the old school days of DEF CON 1 and beyond. And Dan authored a lot of different tools out there for doing security assessments, things like that. And he had a tool called Satan, which was an acronym for kind of a security assessment toolset for Unix platforms at that time. So I downloaded it, ran it full bore on the network and knocked over our server in the middle of the trading day on the trading floor. So obviously it was not a good day for me. I was called into the CSOS office at that time, although they didn't call them CSOS. Back then they called them VP of Information Security and he effectively said, listen, I'm here to offer you one of two options. You're either fired or you can take a promotion. So I really couldn't go home and tell my wife had been fired. So I took the promotion and asked questions later, but he effectively said, listen, I've got some auditors on my team, but nobody really thinking about the network from a hacker perspective. And you already managed the firewalls, DNS and Vine and a bunch of other stuff. It'll be your job to now protect the network and build out a team doing that. And that was how I got into cybersecurity. So hope you enjoyed that story. All right, so getting into the presentation then. In terms of presenting this talk and a lot of this different types of content, over the last few years, really kind of begins with really begging the question of like, what am I really looking for here? How do I identify toxic combinations of permissions? How do I look at that from a proactive, a live and a reactive perspective too? Furthermore, I'm a big believer like Dan Farmer in the best way to really understand the risks and vulnerabilities in your environment is to do various forms of pentesting and the security assessments, obviously in a planned and both a black box type of scenario and all of that. But in understanding how a lot of these recent breaches occurred is really breaking down the TTPs of those tactics, techniques and procedures to really understand how they occurred to better understand how to protect our own cloud infrastructures. In addition, we'll kind of map that to what we've modeled out in terms of a cyber kill chain, but now for the cloud. And then we'll start to break this down not only in terms of AWS and Azure, but also GCP. And then we'll wrap up with some additional resources. So at the end of the day, what am I really looking for? There's a lot of different aspects you can look for in cloud infrastructure, specifically around identity and account activity, not only in terms of human accounts, but remember service accounts outnumber human accounts today five to one. And that's gonna increase over the next few years. They say by 2023 service accounts will outnumber human accounts 20 to one. So it speaks a lot to the automation and at the pace at which we're building in the cloud that there was like plethora of automation going on. And that's gonna obviously continue. There's also a lot of new services in the cloud that you may or may not be familiar with. I think a lot of people who have been working in the cloud already are familiar with the nomenclature surrounding EC2 instances or VMs within AWS. S3 buckets were storage within AWS. Azure Blobs and Container are storage within Azure. But there's so many different services coming out each and every day. Keeping up at the pace of this is really, really difficult. Also, when I think back at my days working at the stock market, you know, circa 1993, we would have different teams that would build out the network, the servers, loading the operating system, loading the databases. These things would take a week, two weeks to deploy end to end and we'd have different teams deploy that. In today's world, many organizations give developers access to the whole stack. So in 15 minutes or less, they can deploy all of that and begin their work. But also that power becomes, you know, less and less focused on separation of duties. And how do you start to distinguish what should I give developers and what should I not? And how do I further control that especially in a production standpoint? Also how you access the cloud infrastructure has changed. We still have things like Secure Shell and RDP. But now in AWS, for example, you have AWS access keys, which give you access from the command line interface. So developers and administrators tend to use that for managing and doing things within the AWS environment. And then as you start to think about the risks and the cyber kill chain, lateral movement, privilege escalation, exfiltration, all of these things is starting to understand of all of these different permissions and policies and more, how do I really start to map those out from a cloud perspective versus how we traditionally look at it from a network perspective? So what we've learned in doing over 100 risk assessments in the cloud for various organizations is that there are tens of thousands of permissions in the cloud. AWS, by default, if you don't modify it for a user, you're giving that user over 10,000 permissions. Same thing for Azure as well, over 10,000 permissions. And also for GCP, thousands and thousands of permissions. And again, unless these are restricted, the probability that a user is gonna have admin type permissions where they could use that to destroy the infrastructure, you're accidentally giving them a lot of those permissions and roughly 50% of those permissions allow that capability, whether you realize it or not. And furthermore, since we look at each and every identity in the cloud, we find out that on average 95% of the identities in the cloud have these permissions, yet the majority of them don't need it, nor should they have it. And what's really alarming is that, you know, what we've talked about up into this moment is really permissions that have been assigned. But what are permissions that people are actually using? What does their activity really expose? Well, their activity exposes that they're using less than 1% of the permissions assigned. So clearly there's a huge least privileges problem and there's a huge issue with over-permissioned access in the cloud. The cloud security alliance did an awesome research paper around 11 prominent breaches. Most of these organizations are household names, not for only us in the cybersecurity space, but just even to your mom, dad and other people that you know. And one of the systemic threads across all of this was really over-permissioned access. And really, you know, really exposes the fact that least privileges are more important than ever because across all of these different breaches was a lot of permissions that the user or service account had never used, yet were used by the attacker for performing the breach. There's some really powerful commands within the cloud. A single line or script can definitely destroy a portion of, if not all of your cloud infrastructure. So as we start to map this out to a cyber kill chain, but now in the context of cloud infrastructure, we can take all of those things that we've mapped out traditionally in terms of reconnaissance, infiltration, privilege escalation, lateral movement and exfiltration, but now look at it through the lens of the cloud. A lot of organizations have short up their security in terms of the perimeter. So there's a lot of different, you know, virtualized perimeter type products out there that help you identify what may be exposed to the outside world. But in terms of the permissions, and you start to take a look at these toxic combinations, they've got tens of thousands of permissions and someone's able to initially penetrate my cloud infrastructure, even as a normal user, are they gonna have a lot of excessive permissions that they can exploit that that normal user never actually uses? And the short answer is yes. And that leads to many other things. Within the cloud, you also have this concept of role-changing or cross-account access. You may have multiple subscriptions or accounts or projects in the cloud. And cross-account access, for example, in AWS allows you to set up roles and policies such as the STS Assume Role to allow you to actually jump from account to account or subscription to subscription and move around in this cloud infrastructure. And as you might expect, if misconfigured can allow also an attacker to move around and not only perform privilege escalation, but lateral movement. And this really starts to kind of bring out the fact that role-based access control, how we traditionally approach this, is really kind of becoming old school because it's really kind of based on assumptions. And you really need to look more and more at the activity of what's going on in the cloud to really bring that second ingredient to the table to better understand, gee, I've got a lot of over-permission folks in the cloud. I need to really prune this back. And remember, the ability for read-only in the cloud, read-only to storage, that alone can allow exfiltration, right? So even read-only access can be quite dangerous. So if we start to break down some recent breaches, and then we'll start to kind of dive deeper and deeper into this, there was one really prominent breach with a financial services institution that once their web application firewall was breached via a server-side request forgery, once access to the IAM role was obtained, allowed the attacker to start to exploit the over-permission access, all those toxic combinations of permissions. And we'll break down those permissions for you a little bit later in the presentation, that at a high level, allowed the attacker to uncover and map out over 700 other S3 buckets that were in the environment that had never been used before by that particular role. And furthermore, with that read-only access, determined that one of those S3 buckets contained more than 100 million customer records, and with that read-only access, allowed that attacker to download those records. If we start to kind of dive deeper into this, what is the cause for this? Well, a lot of it stems initially from default policies. So you may have your JSON policy here with things like Resource Star and these different permissions that are actually allowing broad or administrator access in the environment. But by looking at activity, we can better distinguish, for example, what permissions have been assigned versus what's being used. This starts to help us uncover a lot of these unnecessary permissions being assigned to everybody in the cloud. If we dive deeper into this and we take a look at this now from AWS access keys, for example, in this particular example, this was a prominent security vendor. I won't mention the vendor by name, but one of the developers had actually developed code, embedded the AWS access keys in that code and had posted it to GitHub. And an attacker had found that these were sitting out there and furthermore uncovered that a developer had actually made a backup of the RDS database for Snapshot and used those API or AWS access keys to then access that backup snapshot, which actually uncovered user accounts, password hashes and being a long-time employee of VeriSign way back, I would argue even the bigger issue being exposing the certificates and the private keys for every one of those web application firewalls around the world. So at the end of the day, more than 13,000 customer certs, private keys and the password hashes were exposed. And AWS and Amazon in general, recommends as part of their well-architected framework that these keys be rotated every 90 days. But when we do our risk assessments for organizations, we find many times these keys are not being rotated. And one of the key reasons for that is, they end up creating keys for everybody in the environment, but some of the people are not even using them. And so those keys sit out there forever and unused. So this isn't a silver bullet. I mean, if you rotate access keys, it doesn't prevent developers from posting them out on the web accidentally embedded within code. But although it's not a silver bullet, it is kind of one of the pieces of the security hygiene. So ideally, if you're rotating these keys every 90 days, maybe by the time a developer posts it, someone else discovers it's sitting out there, you've already rotated the key and maybe that old key is unusable. So it's just one of many things. And we'll talk more about other things you can do for mitigating these risks in the environment. All right, I'm a big believer lapdian farmer in really getting to the heart of the matter and really providing some real world examples. We said that there's 10,000 permissions within AWS, but what are a few examples of that? So one of those is how you define your virtual firewall policies, something called a security group within AWS. And if you give a user all 10,000 permissions, which we frequently find, they have access to this permission, which is authorized group ingress, security group ingress for EC2, meaning they can change the firewall policy for your VM. So think about this for a minute. In an old school perspective, would you ever give an average user or a developer the ability to change firewall rules? Obviously not, right? But by giving this default permission to every user, that's effectively what you're doing. Also, if you start to think about this from an insider threat, another dangerous permission is IAM create user. This allows a user to create another user ID. It may be for another developer or it may be a dummy account used for insider threat. So when giving out this permission, you're inadvertently giving all users the ability to create other user accounts. So from an administrative standpoint, would you ever consider that in a normal environment? Obviously not, right? There's a reason there's administrators, right? But effectively you're giving that permission and that ability by giving them that permission. We also talked about role-changing or cross-account access. And this is related to the update the Sumeral policy and the STS Sumeral. By giving out this permission, you can allow developers and other people to change how people can move around from account to account or subscription to subscription. Also quite dangerous, especially if they're going from a development account to a production account. So this can also be quite dangerous. And then furthermore, the IAM put user policy allows them to modify policies to change access to resources and other things. So again, just four of these 10,000 permissions. What about GCP? Well, there are similar policies and permissions and tasks within GCP or Google Cloud. So I've mapped these out for you as well. I won't go through them in detail, but by virtue of the presentation and probably the recording, you can always go back and review these. But again, in terms of external access, insider threat, lateral movement, and privilege escalation, I can map these out for you as well if you have a Google Cloud environment. So what about Azure? So within Azure, we're all familiar with the SolarWinds breach. And one of the things that came out of that was a lot of customers coming to us with part of our risk assessment saying, hey, I wanna make sure I'm not exposed within Azure as well. And Microsoft had a really great article, CVE, and everything around this around tracking and monitoring role assignments right. For people to change role assignments. So that's a really important permission we consider within Azure that you should at least monitor, whether you've got some kind of IDS logs, however you're monitoring that with your SIM, really, really important amount of that. You're gonna have people that are normally using this on a regular basis, but tracking that and having an audit trail of that is super important. And we'll talk more about audit trails later. And then preventing it, get to least privileges. For each and every one of your users, review what permissions are being used based on their activity and use that to get due away with all the permissions they've never used. Most of these cloud infrastructure environments provide privilege on demand, permissions on demand just in time access. So if you remove a permission that someone uses once a year and they suddenly need it, that's exactly why just in time permissions exists. Allow them to go ahead and request that permission and it gets added. But it is a balancing act, but does it make sense to leave those other 9,000 permissions they've never used, including admin permissions sitting out there? Probably not, right? So you have to kind of weigh the cost of getting to least privileges versus removing that one permission they might use once a year. All right, so bringing this back to the cyber kill chain. We talked about aspects of infiltration, privilege escalation, lateral movement and exfiltration and discussed under each and every one of these some examples of that. We also kind of gathered some key metrics across these three key platforms to kind of bring out where we find the most prominent risks. So within ADBS, we find that more than 65% of all internet prices have an EC2 instance or more with access to all S3 buckets. Whereas that ideally should probably be a lot more restricted for the actual access. Do we want somebody to gain access to the environment by accident to have access to all S3 buckets or narrow that by really restricting that broad access to what is really ideally needed for that particular VM and that particular workload and environment? In addition, more than 50% of the enterprises have identities with privilege escalation. So we talked about briefly privilege escalation but they have those super admin permissions and roles. And again, usually the number of administrators compared to developers and everybody else is usually less than one or 2%. So should 50% of the people in the environment have those admin permissions? Probably not. Within Azure, we find more than 85% of enterprises have over permissive users and service principles. A lot of this stuff's actually been orphaned and left behind after a project. So a lot of this stuff isn't even being used anymore. So again, looking at the activity will help you identify these leftover service principles, roles and more across all of these cloud infrastructures to say, gee, a lot of this stuff is old. It hasn't been touched in a year or two. Let's go ahead and decommission it. And then more than 70% of the subscriptions have identities both users and service principles with over permissive contributor roles. Remember, identities with a contributor role can significantly increase the risk because they have the ability to delete, delete critical infrastructure. So again, you wanna be careful with those contributor roles and we find them excessively used, yet, or over permissioned but underused within a lot of these environments. And then in Google Cloud, we find more than 85% have user managed keys for service accounts that are not rotated. We talked about that in the context of an AWS environment and more than 80% have service accounts with over permissive owner or editor roles either directly attached or inherited from a folder or organization. So again, just speaks to the least privileges. All right, so the other half of the presentation, how do I fix all of this stuff? It's really, really difficult to do this manually when you have tens of thousands of permissions and when you start to think about this if you're a multi-cloud or hybrid cloud, now your problem goes up exponentially. How you perform this and the experience behind it really requires a good understanding of the authorization model for each cloud infrastructure. Do you have different teams that manage different cloud infrastructures? Most likely, right? So how this is done may be done by different teams, but it needs to be done in a cohesive way. And it does require deep visibility, right? We're not just looking at what permissions have been assigned but then identifying not only those toxic combinations of permissions, what permissions are people actually using based on the activity? So really a lot of this can be pretty daunting. There's a lot of tools and approaches that you can use. Within AWS, you've got CloudTrail, Access Analyzer, Access Advisor, some people are using CloudWatch. Within Azure, they kind of merge together. Microsoft Graph and Azure Graph and Security Center is actually pretty awesome now. And then within GCP, you have a variety of Google Cloud logs. There's a lot of different products, both commercial and open source that are out there to help you analyze these logs because they can be quite extensive. You could easily have millions of entries on a daily basis. So how do I analyze these logs? Let's dive into the next rock layer here. Within AWS, you can kind of log into the Cloud portal and in doing so, you can view each event within the history. But again, if you've got a million events, you'll never make it through in one day, right? So if you download the JSON, for example, you can start to kind of comb through this data and leverage some of these open source tools or some of your own automation to kind of go through these. But let's break down some of the elements within, for example, one of these events. In this first example, we have one where it lists out the access key ID. So this is revealing the authentication method, which means it's using AWS access keys, not the username and password credentials like you would within the web portal. In addition, keep in mind that by default, there's no multi-factor authentication with access keys. That's why hackers love to attack using this method. So maintaining, rotating these, keeping tight control around these access keys is just like how you probably handle secure shell today. Also, you can see what activity is being performed by the event name, in this case, list buckets. So based on a role policy permissions, in this case, listing S3 buckets for your storage in the environment. And then furthermore, the source IP and the type of browser being used, which are kind of traditional things that many of us may look at in other environments. But again, you know, are key elements within one of these events that you may find within CloudTrap. This example is one that instead implies that there's something nefarious going on in the environment. Here we have an event that looks similar, but is a little different than the other. In this case, we have the same access key ID, the same AWS access key. And list buckets being performed, but now it's coming from a different IP and a different browser. And while, although this is a really simple, fundamental example, it does start to set some light on how you look at this data, and then furthermore, how you can identify anomalies. AWS authored a really cool pre-provisioning tool, kind of creating a clean room, almost like you would in forensics, right? So all of us have seen CSI shows where there's a clean room, they're all wearing like white clothing and everything as they take a look at evidence they've gathered from the field. From a digital perspective, right? Now many organizations and law enforcement and others do this in terms of mobile devices and laptops and things like that. And now in the Cloud, you can create a clean room. And there's templates sitting out there, such as a CloudFormation template, that'll help you quickly create that and be prepared if you need to do forensics in the Cloud. The Cloud Security Alliance authored a really cool paper surrounding this that gets into how you can fundamentally have a playbook around your clean room forensics and how you can take this along with a CloudFormation template, spin that up and actually perform this in an isolated environment. As we jump a little bit over into Azure more, now within Azure, within the web interface, you have the ability to go straight to the command line for Bash or PowerShell, making it really, really convenient. And here we can actually see what each of these two environments look like, whether it be Bash or PowerShell. And a lot of organizations and administrators and developers and other people will use this, but again, be careful of some of the potential dangers here in terms of security hygiene. In terms of your Azure CloudShell, Bash or PowerShell, it can keep some files within your storage account within your subscription. And if you make any kind of mistakes and you're using the PowerShell commandlets and there's some error logs, they may end up in the Azure folder in your IMG or image file system. And one of those, as highlighted by Netspy, is a new Azure VM commandlet that can be vulnerable so it will log their credentials when creating new virtual machines. In their example, they showed how, by looking at one of these files, that you could find different things like an administrator username and password. So again, you wanna be careful with the security hygiene surrounding this. The risk is if someone gained access to a normal user account, could they then open up, for example, PowerShell, go into this, look at the logs and see if they could find a privileged account like an administrator account. Security center within Azure has improved dramatically and there's a lot of good data here. This will depend on the type of subscription you have but at the bottom right, you can see that there's a section for most prevalent alerts and this will tell you some of the most prevalent alerts or risks going on within the environment and you can drill deeper into that. This will expose for you a particular event like this one here, the failed secure shell brute force attack going on when it occurred, what frequency, is it repetitive or not? And you can drill into one of those individually to see what accounts they're targeting. And furthermore, from what IP and other things that may help in terms of your analysis. In this presentation, we're really not talking about Office 365 but just be careful that traditionally within the exchange online environments that a user by default, any user may be given PowerShell access. Now this is for the default user and it's just for their user account, not like an administrator with more broad PowerShell access. But again, if that's not needed for the user, you may wanna turn off PowerShell for most of your exchange users, your exchange online users that is, if they're not using it. Because if any person's exchange online account is breached, you could effectively maybe use PowerShell to do automation, to further target other accounts down the distribution lists, delete, send emails, mark emails is on red, things like that. And you wanna look for evidence of that within your unified audit logs. Keep in mind that the unified audit logs may only be saved for seven or 30 days depending on your level of subscription. So if you need to go back six months, you may not have that data. So some great PowerShell scripts available from Microsoft to actually extend your unified audit logs, save them in a blob of container and definitely consider starting doing that moving forward. We've mentioned briefly around anomalous behavior and this can really be leveraged very well in the cloud to identify a number of things. For example, when a critical resource is accessed for the first time, you're gonna have a lot of users by default with a lot of excessive permissions and excessive access to resources they've never used. So identifying when a user is using a new resource can become really important in terms of identifying initial IOCs or indicators of compromise. Also, if they're using that permission for the first time, signing in from different or multiple IP addresses, a strange time of the day or multiple attempts to gain unauthorized access. In terms of mitigation strategies, ideally you're also gonna find you probably have a lot of inactive identities. These may still be people that work for the organization but have never used their account within ADVS, Azure or GCP. This is low hanging fruit in terms of mitigation and locking things down, either disable or remove those inactive identities to cut down on all of that excessive access and permissions. And then you can set a policy, it may be 90 days or something different in terms of high risk permissions. Does Dave the developer need all of those 5,000 permissions of the 10,000 that give him or her access to any of those super admin permissions? Probably not. So you can remove that and again, you can base that on activity. You've never used them Dave, I'm gonna go ahead and get rid of them. And then just really starting to get more and more to least privileges and then leverage your just-in-time access if somebody says, hey, I need access for something new. Well, that's what it's designed for. And then in terms of your hygiene for cloud permissions activities that is, again, taking a look at the high risk permissions. If you're not sure what they are, because there's 10,000 permissions overall, there's some great tools that will help you identify that. What we know in doing our risk assessments is that on average, for example, in AWS, 5,000 of the 10,000 are considered high risk for admin kind of permissions. So roughly half, which is a lot. Determine what high risk permissions have been assigned to identify what and where. And then furthermore, revise those to remove the unnecessary permissions. How do you do that? Again, we usually point organizations to looking at their activity saying like, listen, they haven't used 9,500 of these permissions. So you can probably get rid of all of those and then leave them with the 500 permissions they normally use. And then the anomaly and outlier detection and rinse and repeat. So just winding things down, some great additional resources. Scott Piper runs summit route.com and a blog. Great stuff around AWS security. Rhino security has some great reverse engineering type blogs. The AWS forensic readiness one that I mentioned earlier, there's the link there. And for Microsoft, they have the Azure Hacker Hunting series. So last slide. Stephen Schmidt in a letter to a senator following a prominent breach said this, even if a customer misconfigures a resource, if the customer properly implements a least privilege policy, those relatively little an actor has access to once they are authenticated, significantly diminishing the customer's risk. And what we're really talking about here is really that ongoing cycle around least privileges to get rid of a lot of those excessive permissions and toxic combinations. And with that, thank you so much. I really appreciate it. Thanks.