 Hello, welcome. My name is Ron Soto and I'm here today to speak to you about detection challenges and cloud-connected credential abuse attacks. So let's get started. My name is Ron Soto. I've been in the industry for around 10 or 15 years. I used to work at Prolexic currently working as principal security research engineer Splunk and like I said before I used to work at Prolexic then I worked at Gaspita, a little bit of time in Akamai. I co-founded Hack Miami and Pacific Hackers. I also created my own CDFs and some of them you may have heard of them. One of them was command and control and the other one is no quarter CDF. So let's talk a little bit about this how the cloud permeates or permeates inside the perimeter. I think it's pretty clear at this point in many environments even in small businesses where I myself I used to be a system engineer have my own little IT company and I used to actually install Microsoft SMB server. I remember setting up exchanges which sounds crazy but yeah I used to set up exchange servers in offices of 12 to 15 to 20 people or even crazier things such as a blackberry enterprise server as small businesses. So because that's basically that was the way things were supposed to go and we didn't have the evolution of the cloud. Nowadays you really don't need a domain controller for example in small businesses and many large enterprises you still do although we see things such as off that taking over a lot of the the market but basically as you can see here even Microsoft has made a huge move into the cloud so we have Office 365 exchange online you no longer need to have a Microsoft SMB server or an exchange server or if you don't want Microsoft you can have pretty much most of it by using Google Apps and then you can have email presentation messenger, telephony, video, conference you name it so these are some of the examples of how the cloud now gets into inside the perimeter many of those all servers many of those infrastructure which some of them referred now as mutable infrastructure have gone away. The infrastructure at the cloud level is referred as immutable where usually the providers basically apply the setup and updates and patches to it. You can also see in this slide that I actually placed a graph where other examples of things that we use almost on a daily basis such as comb repositories for example like GitHub or GitLab or we have things that we use for deployment of technology or development of software such as CircleCI or any other technology that is applied to the same end goal. So here's an example of basically how the cloud and the perimeter can basically merge together we're going to talk about the what we call a converge perimeter and in this case we're looking at two couple of examples in this example we're looking at Amazon Transit Gateway or a shirt being it as you can see here we basically are seeing environments where this type of service basically allows the cloud real estate to be part of your perimeter and for all intents and purposes for your users this is invisible. They may be placing code, saving code, moving files, doing all kinds of operations, logging into a server via SSH, copying files across the WAN, pushing updates and basically they won't be able to realize that some of those instances that they are interacting with are actually not part of your formal perimeter they are indeed lease property or lease or rent the property if you want to call it, from the cloud providers. So in this case two clear examples of how this can happen this can also happen in an informal way which I will cover later on in this presentation. For example I call this the hotpotato why because there's always seems to be a not a clear idea it's very clear for the providers but for the customers it's almost like a hotpotato is it mine is it not they're not pass it around the truth is that many of these providers do have a line they they draw into why your responsibility is as a customer that have plenty of cloud real estate and what they provide to you and depending on the service model we know some of the service models or the most popular service models from the cloud providers are things such as software as a service platform as a service and infrastructure as a service in contrast with on premise which is basically when you own by everything you have it in a way where you can do whatever you want but once you start mixing up these things you have to be alert and be conscious that some of this stuff you might just not leave it to the provider there seems to be this false notion that because for example we're talking about like I said before a mutable infrastructure where many times the cloud providers do actually a great job updating hardening the resources that still does not guarantee the fact that this infrastructure can be vulnerable can be attacked I'll give you an example many years ago there was something called hardly and I was working on a company that basically we're looking at 30% of the internet at times and what happens is then the amount of servers even with embargo I repeat even with embargo because we were within an organization where there were embargoes for disclosure such vulnerabilities and this in particular was on their embargo even with the time on embargo the amount of service we had to patch it was almost impossible to finish it in time and the reason why I'm saying this is because these things happen and will continue to happen so when you have a number of different technologies you may have containers that run out buying for example you may have servers around a window you may have Windows servers out there as well which you can set up an Amazon or an Asia definitely it adds to to the mix it adds to the the attack surface considerations and in scenarios like this even as these things are mutable even as the co-provider to their best to protect them and harden them you're still responsible for it that your data is yours applications that are running in there that may be attacked as well because there's there they became vulnerable because of the disclosure or even in zero day these are your responsibilities and these are some of the considerations that we're going to see today as I walk you through some of the attacks and that I was able to research during this year and how to address them so coming back to this right we are talking about a converge perimeter converge perimeter is basically the cloud real estate that you're using leasing or whatever form or commercial use that you're doing what you're giving it plus your perimeter perimeter technically ends either in the border or your win or your internet gateway or routers and then once you have things such as the examples just gave you that you have a converge perimeter you basically if we if we're looking back at this figure where if you have real estate from a provider lost your perimeter and your users are formally or even informally accessing cloud resources then what this means is basically that this is part of your perimeter is a converge perimeter even though you are leasing or renting cloud real estate so the reason why I name it like this is because we are gonna look at certain type of attacks which is basically in our views of reuse credentials which is how Microsoft has looked at it and the reason what this happens and hence the title of this line is because basically when you have as I am bringing you up in my past lines this pair of technologies right all kinds of operating system versions all kinds of applications you need passage through this entities for many reasons functionality continues delivery continues integration deployment of technology you need mechanisms or in this case tokens or credentials or secrets that allow you to establish authentication authorization it's a sort of a trust and in this infrastructures that are now more complicated because you have a converge perimeter and you have to interact with all kinds of disparate technologies so yeah it is a feature this secrets this tokens or certificates for example you may have federation services which we're gonna look at later on you can use also certificates for for authentication are things that are used to facilitate the operation in passage many times of data many times of code many times of deployment of technology so here's where we are focusing at this point because obviously these are the key elements that we're going to be looking at because by obtaining this type of secrets token certificates attackers can do a lot so coming back from what I just said when you are in a federated environment and you use this type of technologies obviously as you can see it here is a graph where basically we're looking at disparate technologies but this disparate technologies can indeed communicate by using let's say you can use a federation for example what you use either off tokens or you can use SAML which is a a mechanism of basically authenticating between entities and what this shows here is that this is the realities of what we're facing today when we're looking at the interaction of cloud technologies and all kinds of operating systems mobile systems remote offices you name it you need something that will allow the interaction operation and passage and most of all you need the users at the end of the day the operators to use efficiently technology if we were to put or force for example what the factor of indication for example between many of these internal systems and external systems every time this basically will become suboptimal and and basically all the benefits that we use we take from things such as continuous delivery continuous integration or even deployment of technology that that basically knows it that basically was lower to a point where you have to consider is this even worth it so let's look at some key points on federated environments since we're going to be talking about abuse of cloud connected federated environments things that we need to look at are for example formal connections of perimeter and cloud real estate resources what I call the converge perimeter they help this this this the reason to bring it up in a way why is it that we're looking at federated environments and why are they important well they they they allow the increase of cloud utilization so as we said before move on move on on premise resources to the cloud they increase this utilization also provides the ability to increase the resource availability or even geographical reach we're talking about where they go and circumvally elastic technologies you can expand or shrink as you need it you have to have a local data center or you can have the minimum you can have inside the perimeter but take advantage of this and this obviously it's not the same thing to use or to create environments that you may need at the moment and then pay for usage that that is a wonderful thing that has helped a lot of companies and like I said before we're looking at standards that allows the passage of data identification authentication that's why we're looking at things such as tokens certificates even passwords and API keys of course formal federations are usually the ones where there is a to put it in a certain way a formal technology in place that identifies and authenticates entities between each other and that implements obviously and a stricter control of access of this resources and facilitates the use of it like simple sign-off for example at the at the GUI level and at the user level but you can also have what I'm calling formal federations the formal federations are places where because of the usage of so many cloud technologies in the state of at rest credentials secrets certificates API keys that are inside your perimeter for example by developers even though you don't have a formal federation an attacker if such attacker access those resources they can people into the cloud and even in some cases depending of how your setup is they can even people from the cloud to your perimeter and that's what we're trying to focus here today that we had to understand the risks of the expansion the the the subject of the converge perimeter so the converge perimeter risks scenarios some of them obviously we have credential leakage in public repositories we have for example use of vulnerable components from cloud and for example open source libraries or containers usually downloaded embedded they go through either for deployment because they're hosting a database or they're hosting an application that's needed or in some cases in development where they had the perfect environment to host a specific application or a specific API that's needed they don't get scan they get downloaded from public places and here there's that is a big risk you can also take a look at this brochure of cloud apps and infrastructure that may lead to internal access and I will give you an example of a campaign several campaigns actually rather that we have observed where where specifically this scenario interesting enough provided from cloud to internal access the reuse of federated credentials attacks such as a golden sammel discovered by cyber arc or things such as off talking hijacks or past the cookie for example we'll see some examples of it with actual examples and then we can see in the case of federations people think from cloud providers from one cloud provider to the other I actually seen that I researched it and then from there try to move into the converge perimeter the internal resources so here's some of the examples of cloud connected credential abuse here's a an attack which is not that difficult what you see on the top left is Mimi cats that was run against a windows host and then on the right you see for purposes of approval concept you see how I was not logged in I downloaded or rather discover reveal me cats against a windows host I was able to get the specific cookie then by the the same bra browser functionality inserted a cookie and there I was I was able to basically log in back this is not new but it still works I did it recently and this is something that can be done by somebody that has the possibility of access to in this case an employee so this is the first example here's the second example which was part of the research I did with Jenkel one for the net scope and here Jenkel actually did an amazing job by identifying things such as the off talking hijack from GCP basically as you can see in this slide you can extract the token and then once you strike the token this can be done basically by either access to an endpoint or the physical possession of a device that has obviously the SDK or the CLI for GCP and then all what you have to do is refresh the token and just so you know MFA does not protect against this because if the token has been authenticated already you can keep refreshing it for example there's another case scenario of off tokens hijack here's a one that had a lot of I guess chatter doing part because it's used in in the apparent use in the campaign during saw winds according to CISA and here's the I took this graph from this Cycnia advisory cyber art did also an amazing job and one of their researchers actually created a tool that basically strikes the sample assertion creates a force sample assertion and then passes it so once you do that as you can see in this graph this is a federated environment and in this federated environment we have basically likely a domain controller and in this domain controller we have active directory federation services you set up a federation and then from there you communicate and make it easy for users to basically use either office 365 or exchange online and what happens is if you are able to basically forge and it's not easy to say that it's not easy to forge it the providers I got him very smart they have become especially very restricted on the attributes because that's where you can change things such as the duration or the user so when you look at the sample assertions there are things such as attributes and then this attributes that you should be checked based on the identity provider basically to prove in a certain way before that they you are allowed access and the reason why this attack was so successful is because if you this can still happen by the way if you are taking sample assertions not likely with this providers this providers actually make it very difficult for you to do a sort of a golden sample attack but if I know that there are other open source and other even internal cloud setups and you are not strict on the checking of sample assertion interviews this attack will work and it's very dangerous because it basically allows access to every single resource that is dependent on the interaction with such assertion so here's another tool that obviously had to do although it's a tool that is ran against a likely active directory environment however can provide a lot and make work in for their environments let's come back to the industry Microsoft is still the leader Microsoft is for the most part the enterprise operating system active directory and there are tools such as a DFS dump which are not only you have to compile it every time to run it which makes it incredibly difficult to detect but they are great at obtaining keys that will help you forge certificates or even use this keys to forge sessions or make requests for further identification and recon of the active directory environment where you are so these are things that you had to consider because very hard to detect and we do have a little bit of a way of sort of looking at this from a certain perspective however this is not a civil bullet against tools like this so here's a like I told you before some of the actual campaigns that we have seen whereas you go basically full circle and how you go full circle wall okay so let's say you have exchange online or you have yeah exchange you let's start with exchange this list let's say you have an exchange online you can many times we've seen environments where there is no multifactor authenticator authentication so credentials actually remember we talk about federated active directory right passage authentication identification through all many environments so we have for example a exchange online right we have tools that can we don't even need tools we can actually if this is for example was a misconfigure exchange server you can actually use PowerShell to elicit a response that will give you a sort of a directory information usernames in some cases even depends upon how it was configured way way a bunch of information that may allow you to target users or you can just simply cardboard for brute force it and let's say you're able to brute force it and because this is obviously a federated active directory environment most likely the user repeats that's the same password for all their services so all what you need to do next is to find the VPN of this company or RDP servers that might be exposed to the internet so you are technically able to log in because you have obtained the password right so you have obtained the credentials you were able to log in an exchange online you were able there are other tools that you can use or even do it by a PowerShell and obtain for that information depending on what level of privileges you were able to obtain this user credentials and this user in particular and then from there you can attack log in and attack extract for the credentials use things such as emicats for example that we looked at it in the past the cookie example or adfs num which we also looked at and then forge credentials and then here's the full circle you go back to the cloud now you're in GCP you're in AWS you're in an Azure and this is what I wanted to bring up to your attention today this is happening right now unfortunately many times the expansion of the cloud and the connection of such cloud that has brought a lot of benefits you know nothing better that you know about the old times I went to my OW UA I remember I used to put all the UA's we know we know multi-factor authentication or even VPN it was it was seen as how cool we can access it and check our email we don't have to go into the office right that was that was actually the the original purpose of these things well there the times have changed and these things can no longer be exposed to the internet without some sort of protections or we seen actually cases where hardcore keys API keys passwords in a code had led to breaking through either VPN this happened this just happened by the way this happens at colonial pipeline they broke in via VPN that they didn't have multi-factor authentication or you can even exploit the VPN like it has happened sometimes and there are and I can tell you this because I see it and I've seen it there are tons of RDP servers that don't have multi-factor authentication and many of them are federated and with that then obviously you just had to execute your tools the best to your ability and then if you wanted to be in the perimeter either because you access via VPN or by RDP then you can expand further into whatever real estate cloud properties this vulnerable environments have which could be in any of the main providers that we all know and were aware of so as you can see it's almost like a circle you can go full circle and now let's say I was able to get in into or even break let's look at the other side let's say I was able to break into a certain a logging server that is part of a connected formally connected environment by one of those services that we just talked about you will be able to basically access servers or or or even endpoints that will be reachable because of this extended network and converge perimeter so this is basically something that complicates the current situation that we have and sometimes it gets even more complicated with where we're licensed not clear of ownership between the providers and the clients and now we're seeing an increase in this type of attacks that obviously require the use of federated cloud connected environments which I would say are the perfect environment to do this type of attacks all right so with that in mind how do we approach this attacks is there anything we can do yes we can do we can do a lot we can approach this attacks and this is something that has hopefully has started to happen in many enterprises but because bringing back what I said initially but because there's this false notion that somehow your provider is gonna protect you or you don't have to worry about it it was not very clear and he didn't help to be honest and this was one of the things that I realized as I was doing my research the cloud providers they want you to stay with them as far as analytics in as far as even obtaining locks from them is incredibly difficult it was it has become easier lately but before it was like talking different languages schemas and that poses a challenge for system admins for AWS administrators Asian administrators there is it companies are trying to avoid marrying to a single vendor so when every single vendor does not want to play with the other all what this does unfortunately I have to tell you because I've seen it myself is in pairs and security other customers because when you have an environment that has multiple vendors the best ways for all of them to collaborate the best way for all of them is to provide the easiest way to disability into their environments and that's what I'm trying to approach here the approach that we did here is we already know we can observe and analyze the endpoint right so give you some of the things that I've seen and related or associated in this attacks things such as the destruction and certificates or keys tools like the certutile.exe on common processes running in servers all of us all in that may give away for example the user a BFS dump things such as registration or or registries rather used for privileged escalation or the use of mini cats which for the most part a lot of the vendors can detect pretty good and then we can take a look at the cloud providers and in this when we look at cloud providers obviously we had to we had to focus on the specific of every technology right there are no universal technologies for federations some of the standards like the disarm all the off are being used by by these providers however the products themselves using in different manners that is why I'm putting some examples there of some of the things that we look at and I'm about to show you things that are even more specific that we believe will help you detect this attacks because it's challenging I can tell you if we take away one side of this slide you will not be able to detect this you will not like the past the past the cookie for example or the off I look at both longings from the long on perspective it looks like anybody and sometimes you can use for example security groups even using security security groups depending on the policies that you have you may not be able to see it and even worse if you don't have visibility at the end point level so as we have expanded into a converge perimeter we need to expand our visibility and that includes the cloud real estate so we have things that we need to look at and we need to get alerts from it if you don't if you're not getting alerts like for example if you have an ultra 65 that has excessive lowland errors then you're missing out you you you may be under attack right now and you don't know but consider this if you haven't all 365 from a federated server where where passwords are repeated then the the scenario that I just gave you it is likely to occur it's even worse as we go higher privileges do you have the soup admins you have admins per one or admins per land segments I seen that layers of administrators these are things that you had to consider that if they get compromised they may indeed provide access to all these resources here's a breakdown of the actual detection searches that we in my I developed this with my with my teammates at the Splunk threat research team and this is far what we do so here's someone of the TTPs that we're looking at endpoint for example they use a third util.exe which basically allows you to see the execution of it which is usually very rare this is not something that happens commonly on common processes an endpoint which may give you the presence of the execution of tools such as ADFS dump for example registry keys used for privileged escalation and persistence this is a T1546 012 then we have things such as Mimikatz which is weakened attack pretty good this is credential access which is T1003.001 this is obviously my third TTPs and we even have no one that is actually mimikatz via PowerShell in event code 40703 remember for you to have this you need a system on policy and you need a GPO that would allow you to get the event IDs that you need most of these event IDs are not audited by the phone so you need visibility but the visibility requires several steps just like the cloud you have to select a sometimes you need to sort of sync your log somewhere and then they need to get picked up by some servers that then senses somewhere else and this this can be actually cumbersome you had to do your homework at the endpoint as well you have to have a strict system on policy with all these consequences you know there will be a lot of logs they may have to be rotated there's a log lifecycle that you that I had to consider and of course with EVTX type of auditing you have to know the right GPOs and apply them and after that obviously you need a place to retrieve this doesn't have to be it can be okay or any other sim where you can build your own sim right but the bottom line is the events will be the same the logs will be the same so let's move on now let's take a look at the actual cloud TTPs so for example we have things such as AWS semiaxis by provider user in principle we had this one is a giveaway AWS semia of the update entity provider this usually when some of the attributes are modified this is something that is not very common and that you should have an alert for any means obviously to be locked property you have things such as excessive single sign-on logon errors service principle and other 65 that's also an indent in itself that needs to be monitored or the addition of a service principle there's also an event that needs to be considered an event to be monitored by itself and of course all 365 new federated domain remember what I told you about people team from cloud to cloud I was able to do that by by basically adding a new for the rated domain so these are the type of detections that will give you a way to address the abuse of federated credential abuse so here's a an example of a the search I created at using SPL code in Splunk and basically what this does is give you obviously at the latest stuff so or rather sanitize it to give away some of the the sensitive stuff in it but basically what you get here is when somebody went and said hey let me change the the sample provider and that was me by the way trying to replicate some of the attacks and this is a way to not only investigate but in combination with all of the other searches at the end point and at the cloud level then we can get a very good disability not saying we will detect 100% of everything but we get a good shot of detecting this type of attacks the abuse of federated credentials in cloud connected environments here's another example you can find this if you're interested if you're interested in looking more the code we have a github where you can actually see the actual code this can be translated into other technologies other languages but again I just showing how this translate into that technology and just like I show you the attacks I wanted to show you the detections and the investigations and with that in mind I'd like to thank you for coming today and watching this talk and we're gonna open for Q&A remember you can find me at RodSoto.net my Twitter is RodSoto and if you have any questions please let me know you can email me as well at rodatrodsoto.net and I hope you enjoy this presentation thank you