 All righty, well good morning good afternoon or good evening wherever you guys may be today I am jurist and with me. I've got Patrick and we are here to talk with Sean Metcalf about his talk hacking the hybrid cloud We've both taken some time to watch this. Hopefully you guys have too. I thought it was an excellent talk Incredibly comprehensive and did a great job of Kind of giving everybody even if you don't have a lot of knowledge about active directory or the way that Some of these deployments work of kind of bring you up to speed So if you are kind of joining and they're looking for a talk to watch and you haven't caught this one yet Make sure you go back and check that out Sean why don't you take a moment to introduce yourself? Tell us where where you are how you kind of got into this and how we got to where we are today Sure thing. I'm Sean Metcalf pyrotechnic three on Twitter. Hello everyone. I've spoken at a few def cons in the past I really enjoy the security research element of Technology and IT I've been working with active directory since pretty much when it came out in the year 2000 First is operations admin engineer Designer architect what what what have you and then pivoted pretty hard into the security aspect in say? 2003-2004 but I'd say probably the interesting thing to me was looking at the security aspects of these systems along the way A lot of times you have people that are administrators and engineers and they understand the security elements of it They understand the challenges, but they just don't necessarily have the the backing to push into that direction So certainly for me since I was coming up through the Microsoft area I didn't feel like the security space or infosec was really a space for me because I had done worked with Microsoft Windows and most of the security people in 2020-10 and certainly behind are like oh Microsoft Windows. That's a toy OS like what what do you know about security in that area? so after I was had been worked with active directory for a long time and I started seeing these attacks go from Hypothetical to reality. I said hey, there's really something here And I started putting together some talks to help people better understand active directory security and then in 2017 I did a talk with Taya about Hacking the cloud and and what that actually means and at the time Taya was working at on the azure red team And so I learned a lot from from Taya about you know how these things interconnect Certainly looked a lot at azure ed in officer 65 at that time and then as I was Continuing that journey in the past year. I identified some some issues or concerns that I had with hybrid cloud So these connection points between your on-prem active directory and azure ed or officer 65 or azure or and beyond and Certainly Durkian Malema has done he did a great talk last year at DEF CON and has been publishing some great information about doing just that from a comprehensive attack perspective, so Certainly the research of others have been inspiring my research recently and things that I've been looking at excellent Sean so I guess also just for starters. Can you kind of like Explain in a nutshell what like a the hybrid cloud? I think people kind of understand the cloud and people understand on-prem Can you kind of talk a little bit about what you mean with the hybrid cloud? Sure thing so it's good to have a good definition or it's good to have a solid definition of what it actually is because Like you said it can be confusing. So for me at least at least the definition I put together for this talk is hybrid cloud is when you have On-premises infrastructure, so you have something like active directory and then you have something in the cloud it could be You've extended your on-prem environment into cloud I as infrastructure is a service so you're using the cloud sort of as a date an extension of your data center a lot of Organizations are moving into the cloud and using the Azure or AWS or even GCP is as their data center now So that's one part of it another part of it is when you have all your on-prem infrastructure That's handling your typical things like active directory log-ons things like that But then you have services that are up in the cloud. So you're sad side. So software is a service you have things like Salesforce or you have work day or you have office 365 or things along those lines where you have some connection points And so certainly from the office 365 On-prem ad side there's a connection point called Azure Active Directory Connect and there's some interesting things that have occurred You know with that connection point between your on-prem in that cloud environment. I pointed out some back in 2017 Dirk Jahn's public published a whole bunch of information about that others like Adam Chester's has identified some things where the additional connection points of what Microsoft's been doing to simplify this authentication flow between On-prem and the cloud and I think that's really one of the things that I wanted to dive into as part of my talk was Discussing the roles and how they're typically over permission and also looking at these this authentication flow It gets very complicated and that's why I closed out the talk with a bunch of different slides where you have Three different cloud environments and you have your on-prem environment You're working through the the federation of it and and how they all get interconnected in very interesting ways The I am story there the identity access management story is very complicated when you have multiple cloud environments and it gets difficult to track what all these roles are and Typically they get over permission just like they do on-prem So we've the other thing I wanted to establish as a narrative of my talk was that we have virtualization on-prem We're using VMware primarily Predominantly and then there's also hyper V to some extent and then in the cloud we have effectively virtualization writ large Be it Amazon AWS which started with Zen uses more of Amazon's Nitro now to as you're leveraging a custom version hyper V core to GCP using Zen to straight up on in their environment. So If this concept of virtualization of the way that you attack or could attack virtualization on-prem is very similar to how you can do So in the cloud I think that was an important part of what I was talking about and I even mentioned how you attack Could attack on-prem ad Domain controllers as well as those on-prem ad to make trailers hosted in the cloud So regardless where they are they're vulnerable and then the attacks across those different platforms can have some similarities But also some pretty significant differences Excellent Also, is there really any kind of Difference or what should people think about if they have their ad environment in Azure or if they have it as an on-prem? ad environment their differences they should think about sure So my my friend and colleague Demetrius will he his first question is do you trust the cloud because everything starts with that trust? So We usually presume or assume that our customers trust the cloud because they already have things in it But we still ask the question that to what degree or to what level do you trust the crap cloud because you have to have that trust? I have concerns certainly about placing on-prem active directory domain controllers into cloud environments I as environments regardless of what they are because ultimately there's a significant amount of trust The with the cloud provider the other part of that is oftentimes you'll set up your I as environment AWS or Azure Where your server admins are the ones that manage that entire environment? Well, your server admins who manage that environment can run commands potentially on the domain controllers in those environments so if an Attacker can compromise one of those server admin accounts and they can Compromise your on-prem ad and I walked through the for the scene of talk with Amazon over Federation a way that you can link your I am role in in AWS with your on-prem ad group and then you can have an on-prem regular user That's a member of that group that gets compromised because he's a regular user And then leverage that account in order to compromise The domain controllers hosted in AWS Just because that one account was compromised it has that extended role access and privileges And then because it's in the cloud that can roll right back again and compromise the full on-prem active directory environment So there's some very interesting things that happen with that So I also talk about one interesting one from Azure AD to Azure Because of this sort of backdoor that Microsoft put into it and it's talked about but it wasn't clear that if you were global admin Azure active directory that you could then go jump over to Azure and have full control of that environment So I walked through that as well because these sort of connection points are interesting But a lot of times operations is like let's let's go. Let's get it done And they're being told by the executives. We have to we have to get into the cloud our number one goal right now Is to get in the cloud? I mean we've had a surge of interest and and of companies that have moved to the cloud We've been very busy talking and Consulting with customers that are moving to the cloud because they're like are we doing this right? Because operations gets pulled along and thrown in say you got to you got to migrate all these systems in the cloud And then you have a security team going I just don't know what I'm supposed to do with this like What are we supposed to know? What is what because the thing is on-prem as well understood, right? We at least know what our egress ingress points are at least we hoped we hope that they do Pentesters and red divas are good at helping to point out what those actually are We have a decent idea of auditing and logging and while things aren't great At least there's some knowledge of where some of those weaker points are in the cloud We have to leverage whatever the cloud provider gives to us. So Azure AWS GCP They have different capabilities. They have different controls and then the nomenclature the names for all these things are different across all of them So what one thing that may be called one, you know, something may be called something in Azure It may be called something different AWS and something else in in GCP so what I find myself doing is saying, okay a subnet whatever that may be called in this cloud environment That's what you need to do Well, let's have a zone where we segment off this one type of System from the others and then we make sure that we can track and control what systems that can connect with it I will say that one of the things that we have the opportunity with in the cloud now is the ability to actually Implement network segmentation which we've failed for years on on-prem right because now we're setting everything up We're building these we should be building these zones a zone for to make controllers a zone for servers a zone for clients The zone for other things and then control the communication between them There's some ways that could be done on-prem, but the cloud gives the capability and ability now to move forward in that direction Cool. We've got some coming in From chat here So mugs asks Azure services evolve quickly. Do you see a ad evolving rapidly or is it pretty well-baked at this point as a fundamental underpinning of PAAS and software as a service services generally speaking Right, so as your active directory and the officer 60 flat 5 cloud That's it's kind of an extension of that is evolving very rapidly Microsoft keeps hiring developers and adding new features and it's funny because Mark Morrow who I did a talk with about the cloud last year at black cat I was at Redmond with him earlier this year for an Azure ad identity event and As Microsoft was talking about some of the new features. I looked over and I was like, yeah, so like active directory, right? He's he's not in his head So I think there's going to be some more more parity between the Azure ad side and the ad side even though Azure ad is not active directory at least not the same because it's the Authentication is very different. The management is very different But when you look at the way that the features are being deployed and configured MFA is getting baked into it conditional access I consider is kind of an identity-based firewall you can control who can access from where with what sort of credentials be at mfa or or Passwordless so you have some of that control capability built baked into it So I see it just growing and the challenge is that the cloud changes so frequently I mean myself trying to keep up with it It's it's a lot. I mean microsoft sends out emails to officer 65 customers I think every week saying these are the new features Azure Active Directory is growing and developing maturing and will continue to I think on the ias side So azure aws gcp I think pretty much the platform is relatively stable from the perspective of hosting Virtual machines instances whatever you want to call them and the management of those will will grow and develop and the The orchestration around that will grow and develop The I am the ability to manage them will grow and develop, but I think the core hosting components are pretty Static, I think that the security elements and controls will will grow and develop and improve in that area as well Excellent So with regard to security or hardening r0 wants to know What's the best ad sync scenario direction for hybrid environments? Is it azure to on-prem or on-prem to azure or does it not matter? So typically you're going to have your on-prem environment sync to a cloud environment And last year I identified that there were a lot of these sync tools There's a different sync engine for just about every platform I mentioned azure ad connect for the microsoft azure azure ad Office 365 component But oftentimes it's going to go that way because your core identity is going to be rooted in active directory on-prem at least for most organizations As organizations would go more cloud first The the challenges that I've heard customers say well, we're going to go migrate from ad to azure ad And I'm like it just doesn't work that way So you don't have LDAP you don't have kerberos. You don't have nt ntlm for authentication You don't have group policy. So everything's very different. So it's not going to work the same way And there's different controls and capabilities there So the identity is basically still going to be rooted in your on-prem ad unless you go with a large federation provider That that then becomes your identity provider your idp for that Identity for for your users At that point then there's some other things you could do But I think for the most part we're going to see on-prem ad at least for the next few years syncing up to the cloud or some some sort of weird amalgamation of combined identities for years there was this concept of like a metaverse sync engine where you have different identities in different places you may have employees in in your on-prem HR system and you may have Contractors in some other system and then you have this kind of collected understanding of identity across those I think that with federation that gets simpler and yet with the cloud it gets more complex because The identities need to be managed even even more so now And it's already been talked about several times about how identity is the new vulnerability or the new attack surface for organizations because we've we've seen that when a Admin identity gets compromised then the whole environment can get compromised And in the cloud that gets even more complicated because you don't necessarily know what users on-prem could be mapped to Other things in a cloud environment as far as what rights they have there You could check them out in ad and they're a member of the right groups and they can do all these other things But they may be configured as an admin and work day or an admin in Salesforce And that's not going to be clear Of course the same thing can be true for or said for vmware on-prem or another application But it just that that problem just gets exacerbated so to speak into the cloud environment as well So Sure I'm curious What do you think you know for an organization that is kind of maybe thinking about moving towards more of a cloud strategy? Do you think that you know because like you said you've got upper management kind of saying hey This is this is what we've got to do we've got to move to the cloud wave our hands around say move to the cloud Is is is that kind of move towards the cloud in any of these kind of different scenarios? Do you think that that kind of move from a strictly security perspective? Is that more risky to do that than sticking with on-prem and how much more risky you think people are taking on by doing that? I think it's a balancing act Honestly, it's and it's a tough one. So the the issue is that when This train his cloud train is moving and operations and security in organizations Just kind of has to get in line and and just get on board with with the cloud train The challenge is that there's a knowledge gap and with the cloud. There's a huge learning curve I've certainly seen it. I'm sure others have first of all. You can't just download some software or trial and run it on a vm You know and on on your existing system It you can get a trial for 30 days But very often you're not going to learn what you need to you're not going to learn what you need to in 30 days On a trial so it's going to cost some money And then you have to find the time to work with this new paradigm of an environment whatever they may be So there are some some kind pretty big Barriers for entry as far as I'm concerned with it And so the challenge is with the security of the cloud is you have to kind of understand the security capabilities of that cloud environment The other thing I mentioned in the talk is that We talked to a customer where their executive didn't want to Have all their eggs in one basket So they signed up with the big all of the big three cloud providers amazon, aws microsoft azure and google cloud platform Now their eggs are in all the basket and it's even more challenging from that perspective because operations and security They've got to figure out what that looks like and how to normalize across all three and of course there's Programs or applications or systems that will help you kind of merge all that together in a way that makes sense But then you have one system that over arches and controls whereas control over all of these So I think it's a it's a complicated issue I think that certainly there are a lot of security benefits that that can be gained from the cloud But you have to know what you're doing and know how to do it right I mean if you're going into office or 65 and azure ad you could absolutely as a It was a relatively new company or relatively small company go cloud only and not have the on-prem ad Assuming that you don't have all the applications that are tied into it I did an x director security assessment in the last year where they had One x directory forest that had to stay around because the application that they needed for whatever they were doing That it was in that forest It was hard-coded to the name of the active directory forest And that application the the company that made it didn't around anymore. This is 10 15 years old So there's no way that they can move off that environment until they can figure out how to how to How someone else come in and recode that application potentially But there's some weird scenarios like this that are challenges, but ultimately the migration from on-prem to the cloud I think the challenges that are going to be applications and authentication flow Some of the applications aren't going to support Federation I I've told customers look tell contracting now any new applications have to support federation Because this is the time to actually get those on board it so that way you're not in a situation 10 years from now Where you can't move things now as I said the managed ad For the major cloud providers azure aws gcp Those are there to support those applications But as I talked discussed in the talk There's some interesting things there because now you're providing an ad environment Managed by the cloud provider But you're giving the application owner effectively the the admin rights to that environment And they're not going to be as cognizant of how to well to protect those credentials potentially as someone else made So there's some interesting things with the managed ad environments as well from a security perspective So I say the my answer is it's complicated But I think eventually It hopefully will get less complicated and be easier to actually on board these These security tools like microsoft security defaults for office 365, which are enabled or are turned on by default now Once you have a new tenant What are you having having given that you've now kind of taken a big picture look at all of the different platform options there Do you is there one that kind of sticks out to you as one platform is doing it better than the others? I'd say from a managed ad environment and I looked at azure ad domain services microsoft's managed ad in 2017. I looked at amazon aws as well in 2017 I like the fact that amazon aws had all the different delegation groups that they have had The initial goals of those two environments were quite different so that explained why they were they were configured that way There's some interesting aspects around amazon azure active directory domain services microsoft's managed ad and how they sync Which could be some interesting things for pen testers or red teamers I would say that there's there's certainly elements of each I like I like how amazon is delegated a lot of different things I like gcp's approach to some of the delegation that they're doing As far as just saying okay, we're just going to give you access to this component. For example azure ad ds and amazon aws both pre-create A fine-grain password policy and say here you have rights to modify this Gcp's just like go ahead and create as many as you want, you know kind of do your own thing So there's definitely things I like about them the different ones But I'd say probably amazon aws is probably a little more mature from from the delegation perspective and and the You can set this up and and create a trust From your on-prem to to this managed ad environment and leverage this as an extension of your on-prem ad Good fine. I was going to combine two of the questions that we got here jurors from both Mugs and crunch bang How much better off are we if we don't sync credentials from the on-prem to the cloud platform and rely on ss Oh, and if you're unwilling to do that, what would you be missing out on using something like the ad pta? Good question. So The the question it really comes down to authentication and the risk profile of the cuss the customer themselves. So For your single sign on from your on-prem to your cloud It's typically going to be federation But that means you're going to have to manage and maintain your own adfs servers or microsoft active director federation servers Which are a pain they are not that easy to set up. It's it's difficult to get right get right There's a lot of different things you need to get correct So then you're looking at paying or octa or one of those companies that simplify that process for you Which is going to cost money If you're a smaller company you're probably going to do something like password hash sync at least to the microsoft platform where You're going to it's azure ad connect is going to have the permissions in ad to pull the hashes for all the different users that are synced Go ahead and hash those and send them up to azure ad and then when the user logs into azure ad It compares The password that the user is typed to the hash function that it runs through The on-prem ad hash function plus the azure ad hash function to see what that password actually is The benefit of this password hash sync is that microsoft gets a ton of password data from Security researchers privately law enforcement the dark web stuff like that So at least if you're doing password hash sync you have a pretty good chance of knowing if that credential has been compromised So that that is a nice benefit and if there's any sort of problem We're like ransomware just crushes your on-prem ad and your users can't authenticate You can flip a bit and the users can authenticate directly to azure ad So that way at least they still have access to their collaboration environment So a couple of nice benefits there pta is interesting from the perspective of providing access Without having to do password hash sync or or federation I think that ultimately password hash sync is probably the direction that everything's going to be going in There's certainly concerns as i've highlighted that other people have identified With the different options. Like I said federation is hard and a federation server has to be available to the internet So you have a federation server that's Basically a web server that's listening to requests from the internet We have customers that have been had Nile service attacks basically because accounts have been locked out with older versions of adfs So there's some challenges there. You got to get up to adfs On server 2019 to enable smart lockout and really have that Protect against this denial of service accounts are locked out. So I think it's a it's definitely a complicated story and it's going to be per Per organization, I'd say the larger organizations pretty much already have federation because they have sass apps that they're using And they they pretty well understand that the smaller organizations are going to use password hash sync or pta cool, i'm gonna Kind of combine maybe a little bit of a look forward along with with a fairly good question that we got from chat You mentioned in your talk that you know cloud is in cloud security is a Large area. It's growing area. It's an opportunity for people that that that are in security You're looking to further specialize to to see some success over the next few years I think the challenge for that is how do you how do you get in there? And how do you mess around with this stuff? How do you learn this kind of thing? um specifically You've been asked if you were if you're going to set up a hybrid training lab for uh red teamers of some kind But you know beyond maybe that you know If somebody wanted to learn this in their spare time on on the side What's the best way for them to to to get into this kind of stuff? um, I I would say that the best way to get into it would be To use a trial. I mean that that'll get you 30 days of experience. I would say read the documentation There's videos out there as well. We're kind of walking through new features That's a good way to learn and understand what's there For me, there's nothing better than actually doing and playing around with it I mean for speaking specifically to azure active directory in office 365 You can get an account for about 20 a month. I think that gives you office 365 But they have cheaper ones than that. I think that um, you can use But again, that's that is a barrier to entry because there's a cost with the cloud That's not there for a lot of the on-prem things So you can there's different trials that you can sign up for So you have one account that might be $8 or $10 a month and then kind of have trials for the others And from what I've seen with some of my testing you can create a bunch of different accounts And then have those kind of last for a period of time a short period of time Before they kind of just get pulled out because they they're not subscribed so It's it is a challenge. I would say that Every every organization should have a test account And if your job is dealing with the cloud any way she perform Ask your company to make sure there's a test account so you can better understand this because if you're going in this direction Companies go in this direction The most important they make and do is to make sure that the operation security people Understand the elements and ramifications of this The the cloud to me is the wild west. We just have not explored enough of it and gotten to the point where we Truly understand it. We're not testing it as well from a security perspective from a red team pentest perspective And I think from the security perspective, I don't think the controls have been configured the way that they should be For most organizations just because there's a lack of understanding new things come out new features are there and they just don't get implemented So so kind of given that if if if you can't mitigate the risk kind of technically and practically It didn't here here comes the lawyer to solve problems with lawyers and contracts um What kinds of things do you think that somebody could build into an sla to kind of protect the The client in case kind of you've got a you've got a compromise at the by the cloud service provider level Uh, that's a great question. I'm not sure That's probably above my head as far as what that would look like um And there's mitigations for for that sort of thing. I mean certainly from the iside amazon aws has some components around How you can configure your own encryption key? Uh, that would mitigate that pretty strongly. I would think um, I believe gcp has the same capability Uh, as you're probably the same capability as well. So there are some options there Encryption is easy key management is hard. So if you're managing your own keys that makes things a little more difficult Um, but I think that those are some mitigations that could be done There's certainly regulatory requirements based on things But the thing is that these cloud providers have all said we're we're we meet all these uh, criteria. So It does make it challenging. I I don't have a good answer for that. Um The cloud providers compromise at a pretty major level Which probably will happen at some point I I don't know that it'll be the big three because they spend a tremendous amount of money on security But a cloud provider compromise I think is going to happen at some point It's just the nature of what we do like the intel processors were compromised It's interesting in the past few years So, uh, when you go when you have hypervisors that are kind of the core to what you're doing And then people are going to look at the hypervisors, which they have And I think that's why microsoft has something like a hundred thousand or two hundred thousand dollar bug bounty for, uh, azure core hypervisor bugs Well, I think uh, it's kind of one last question This kind of relates to how some of this stuff operates out of the box, right? Is uh, any of these cloud services that you that you Have taken a look at meets any of the kind of HIPAA requirements out of the box when it comes to Using active directory creds to try to link those together with any any electronic medical records programs that you know of or So I don't look at the regulations per se I so I can't speak to that or really how they work across the different cloud providers I can tell you that the the regulations There's there's always the regulation of how something's supposed to be because it's generally understood. That's the best way to do it Based on active directory security systems that I've done on-prem and what I've seen Uh customers do which have been HIPAA compliant or PCI compliant When you have a a domain in an active directory force, which is a PCI domain and then you firewall that off But I can compromise a domain controller in one domain to jump over to a domain controller in that other domain Then that breaks the whole concept of PCI for that So these things have to be separated the security controls have to be better understood From the from the reality of it. And that's why pen testing and red teaming I think is important And certainly security assessments like what we do, but the the pen testing and red teaming Showing what these issues are and walking through how they could be Compromised despite controls that people academically think are are good I think that's a big part of it And I think that any aspiring or junior pen tester a red team are watching this right now should absolutely start learning the cloud as best they can I mean again with the constraints of what it costs But there's a lot of documentation out there, especially from microsoft And there's videos about walking through what these features are and what they do and then Start figuring out some ideas and some some hypothesis of what this might mean and then Spin up an environment And and see what it looks like in a trial and play around with it And then your trial is going to expire and then give it a couple more months set up another trial to Play around with it and hopefully the cloud providers will provide some sort of student or you know You know beginner access where they can actually just start playing around with these things and get a better understanding of it I'm sure that there's some learning programs that i'm not aware of that that someone will say afterwards Uh, but that would be very helpful Cool. Well, hey, uh, we are just about out of time But I want to I want to thank you for putting together your talk and doing it pre-recorded Which I know is is definitely a different different Different experience that most of us are used to kind of getting up there and doing it live But then also kind of playing around with this with our kind of experiment of doing the live q&a I know I had a good time watching your talk. I thought it was great And I hope everybody in the in the in the chat go out the same thing If anybody needs to get ahold of you or you know, kind of follow you for more of your developments Where can folks find you on the internet? Twitter's the best at pyrotec 3. That's p y r o t e k number three Good stuff well, thank you very much again and Everybody everybody hanging out in chat, you know, we're gonna take a 30 minute break So you can you can watch the next talk if you haven't watched it yet And we'll have a new crew of uh goons and the next speaker up here in just a little bit Thanks a bunch. We'll talk to everybody soon Thank you