 I'm going to announce Chris McCoy. He's a good friend of mine. He's been at Cisco for 12 years. He's in this super spooky group called ASIC, like he mentioned. Nobody knows much about that village. I mean, much about that portion of Cisco. He's going to explain quite a bit about what they do and then also open up some other information that I think you'll enjoy. He's a principal engineer for Cisco. He actually is going to be next door. He's been helping with setting up and running the village with us. So give a warm welcome to Chris McCoy. Thank you. Thank you, Joseph. Thank you. Thank you, Joseph. So yeah, my name is Chris McCoy. I am with ASIC, the Advanced Security Initiatives Group in Cisco. We're primarily based out of Knoxville. We have people in Austin. We have people in RTP and all over, really all over the world. And what we do is we are the primary white hat hacker organization for Cisco. So let me go ahead and get into this. I'm actually going to show you a real security evaluation that we did on application-centric infrastructure and the results that we had. And anyway, let's just get into it. Hopefully you can see this. These are the greets that fly. Nicholas, thank you so much. Everett, thank you so much. Amit, you know who you are. Omar and the rest of the ASIC crew. Terry, I couldn't have done this without you. So thank you. And the audience, please save your questions until the end. I have a lot to get through and I have a lot I want to show you. So just wait until the end and I'll answer any questions you have within reason. Okay? All right. So disclaimer, I am speaking for myself and not Cisco. I'm not officially on here on behalf of Cisco. I am in a Cisco group. This information is three years old. So these vulnerabilities that I'm sharing with you have been around a long time just to give you an idea about when we got started. We got started on ACI just before it just right after Cisco acquired NCME, which is the business unit that this came from. And so this is, a lot of this stuff has already been disclosed. It's already been released to customers a long time ago. And if this stuff still affects your network customers, it's because you aren't patching. Please patch. Okay. And this is not a rebuttal to the Black Hat 2019 talk. There was a talk called Apex Adventure in ACI Wonderland by Oliver and Frank Locke. The findings might be interesting to those that have seen that, but if you haven't seen that before, it looks like I need to stay away from that speaker. But the findings may be interesting if you've seen that talk. But if you haven't seen that talk, then I'll actually go through this whole thing. Security and trust organization is who I'm part of with the advanced security initiatives group. And so we're security focused. We try to find security vulnerabilities as early as possible in the product life cycle. So we have, usually we work in about three, three or four team members. And we just hammer on a product. It could be for a month. It could be for six months. It's basically like a red team that lasts a long, long time. Looking for vulnerabilities. And we discover those zero days and we get them patched out and get our customers protected. That's what we're here for. I also wanted to tell you that networks are not that boring. I've made an entire career out of hacking networks. It's amazing. There are so many things and so many opportunities for red teaming networks. You have no idea. Apps and operating systems, I actually find a little bit boring. I like the plumbing. Because the plumbing has a lot of opportunities for attack surfaces and all kinds of things. And I'm going to show you one. Okay, so I'm going to get into software defined networking, SDN. SDN means a lot of things to a lot of people. It has a bunch of different definitions. This is really what it is, kind of boiling it down. So boiling down software defined networking, it is really about a central point of management. That's what you want. You want to optimize your network for physical virtual machines and containers. You also want to do some kind of policy enforcement. So that you have what are called micro segmentation. Where only certain services can talk to other services. And I'll get into that a little bit more. But really what you need is you need programmability. You need to adapt the network to the endpoints, the agile forwarding path. And you need to have some kind of discovery. You need to know where things are located. In ACI, in Cisco ACI, this is made for a data center. It is designed to scale up to one million endpoints. And an endpoint could be a VM, it's an IP address, MAC address. It scales up to that large of a data center. It also has the property of multi-tenancy. So you have many different customers that are all connected to the same data center network. And they all want to be able to control their own policies. So it's a multi-tenant environment. You also want to control how endpoints communicate. What TCP ports are available from one source to a destination. You also want to be able to insert middle boxes. You want to do things like load balancing, real firewalls, NAT. You know, if you're doing carrier-grade NAT, maybe you need something like that. And so the answer to all of this is the application-centric infrastructure, or ACI, Cisco's ACI. And so this is the way the fabric looks. The fabric is made up of switches that are in the rolls of leafs and spines. So this is what's called a CLOS network, if you've ever heard of this, a bipartite graph. You have a spine. And the spines are really responsible for routing traffic from one leaf to the other. The leafs are fully meshed up to the spines, and the spines are fully meshed down to the leaves. And the idea is that you can load balance across the spines. And this is just one layer. You have very large networks like a Facebook where they have multiple layers of leaf and spines. They also have other layers down here too, where you have maybe top-of-rack switches. But typically this is what it looks like. And then attached to the leafs, you have the controllers. Those are called your A-picks. That's your advanced policy infrastructure controller is what an A-pick stands for, A-pick. And attached to your leafs, you've got your servers. You've also got something called V-leafs. This is your hypervisors that actually are running the code, the ACI code, to be able to do the fine-grained policy that I was talking about earlier. You have virtual machines and containers that are running them there. You've got your firewalls. These are your services, your layer 4 through 7 services that you're doing for load balancing. And then you have external networks of some kind. You have to connect this somewhere, right? You have to connect to a router and switch to those routers and switches that go outside. You know, you need to sometimes maybe connect to the internet. You need to connect to your internet. And you'll have an external network. So the thing to keep in mind is for the most part you don't attach to the spines. That's not always true. But you attach to the leaf and you'll have things like services for leafs that you attach to. So everybody with me so far, this is an example of what these switches look like. They are rather high-powered. The ones on the top actually do power over ethernet. So you can attach things like phones and cameras to them. The one on the bottom, that has multiple 25 gigabit ethernet connections, and it has, I believe it's either 100 gig. Yeah, I think it's 100 gig uplinks that it has. So this is pretty high-powered stuff. You know, many, many different connections like I was talking about earlier, up to million endpoints. Okay, ACI, they have, so I'm a CCIE. I've been doing networking for about 25 years. And I had to completely unlearn networking to really understand how this thing worked, because it was just boggling the heck out of my mind. I just couldn't understand it. But at first, really, I had to unlearn it and then relearn it to really understand what's going on. When you have VLANs, they aren't really VLANs per se. I mean, they're not bridge domains. They represent what are called endpoint groups, EPGs, endpoint groups. And endpoint groups are grouping of endpoints, and they're allowed to freely communicate inside of the endpoint group. So all of your endpoints that are inside of your endpoint group, they can talk to each other without any kind of policy, any kind of constraints. You can go from one endpoint group to the other without any problems. So the idea with this is that, you know, we're running out of IPv4 addresses, right? I mean, people have to make these very large networks and you have to be able to constrain your network so that you need to be able to design your network so that you don't have to be constrained by IP addresses and VLANs and all those things. And that's what EPGs are all about. Contracts. Contracts are what allow you to talk from one EPG to another. These are basically saying you have a provider and consumer relationship that says, I am providing TCP 443, HTTPS. I'm providing this to anybody that wants it. And then if I have a contract that comes along and it consumes that endpoint group, then that endpoint group can use that TCP 443. And if you're not part of that contract, then you don't get access to that port. It's the way it's supposed to work. Inside of the fabric, the fabric is completely layer three. Even when you're doing layer two connectivity, when I say layer three and layer two, when I say layer three, you're actually going through a router. When you're going through from one endpoint group to another endpoint group, you're always going to be routing. And the way they do this is they use an overlay. You've probably heard of things like VXLAN. If you haven't learned VXLAN and you don't understand VXLAN, you really should because everything is going that way. And they're starting to even use VXLAN in the edge of your network and your access layer using Cisco has software to find access. And I'm sure there's probably a lot of other ways out there. But as it turns out, it makes it really easy to be able to do policies. Please save your questions until the end of the presentation. Thank you. Okay, your links are IP unnumbered. We're using ISIS internally. That's the routing protocol that they use. And one of the things about ISIS that makes it really interesting is that it runs on a link and it runs at layer two. And it doesn't actually give you an opportunity to be able to inject things across broadcast domains. So if you're on a switch, what this means, if you're on a switch, you're not able to actually get access to that link to be able to inject things into ISIS. So it's a little better. But it's always layer three routed. And they advertise what are called VXLAN tunnel endpoint IP addresses. So whenever you have a frame to send, it goes into the leaf switch. The leaf switch encapsulates it in VXLAN and routes it across the fabric to the destination leaf switch. The destination leaf switch then takes that encapsulated Ethernet packet frame, strips off the VXLAN stuff and sends it on its way to where it needs to go. And they use ISIS. It's been tuned for densely connected fabric. So of course you have many connections that are going all over the place. And it's been tuned so that it works. And they use IC multicast. If you don't know where things are going, they use anycast. So one of the things I'll talk about is a little bit later on is that the spines are not dumb. The spines are actually really smart. They're the ones that do all of the tracking of who's connected where. Okay? We talk about tenants, VRFs, and bridge domains. VRFs, and they call this a context inside of ACI. But it's basically when you get boiled down to it, you've got a VRF, which is a routing table. You have a bridge domain and the bridge domain contains subnets. And those subnets can be anywhere on any switch, any leaf switch that's in your fabric. Okay? And you have multiple of these since you have multiple tenants. Okay? So endpoint discovery. So when a device talks to the fabric for the first time, it will maybe send DHCP, send ARPs, send something. It doesn't know who it is or who it's talking to. So what happens is the leaf switches are constantly listening on these things to just do discovery. So when a VM, for instance, sends a packet for the first time, the leaf switch will pick it up and it does a source look up to see whether or not it knows where that leaf or where that source is. And once the endpoint is discovered, it sends through a proprietary protocol called COOP. That's the Council of Oracle's Protocol. And that's how it does location learning. So just in this example here, I'm showing something different where we have a VM that's running in vSphere. It's running in VMware. It talks to the vSwitch. The vSwitch then sends it on to the VXLAN endpoint. So this is supposed to be agnostic. Pretty much VXLAN has one out. There's another encapsulation technology out there called Geneva. But there used to be NVGRE. And I think Microsoft has also kind of moved away from that. But once the ingress learns that the MAC address and IP address of the arriving frame, it sends that, excuse me, along with the ingress port vLan and vXlan to the switch and the leaf forwards that endpoint address to the spines using the 0MQ protocol. That's what they use. And then that information is spread out or it's distributed to all of the spines. So all of the spines know at any one time who's connected where. Hopefully that makes sense. This will come important later. And then when you send the information, when you send the frame to the egress leaf, the egress leaf then is able to learn the source address, MAC address, and all that other information when it gets to where it's going, unless you tell it not to. There's a flag in the frame that says don't learn this. So what is this for? What are we going with this? So just showing you this is the life of the packet. This is how the packet works. It's sent from the vSwitch to the vTEP, the leaf switch, then if you're using vXlan, it actually swaps out the vXlan information for its own internal encemi vXlan protocol that goes across the fabric. If it doesn't know the destination, it sends it to a spine using an anycast IP address. And it's just basically a range of IP addresses that are allocated to all of the spines and all the spines are listening on that IP address all the time. Then in hardware, they're able to do that lookup to find out where that actual destination is. And then it gets sent to the destination leaf switch. The leaf switch then strips off that information and replaces it. If they're using VLANs for your endpoint groups, it could use VLANs. If you're using NVGRE, which isn't as popular anymore, they may use that. But that's basically how it all works. Excuse me. So then we get into VRFs. I'm going to just zip through this a little bit. But this just shows how tenants, VRFs and bridge domains and endpoint groups all work together, how the frame source's information is kept. It's actually relayed from one leaf switch to the other through VXLAN. And the APIC controls the allocation of VNID labels across the fabric. Labels, by labels I mean things like your actual VNIDs, that's your VXLAN network ID, and your source groups and all those other things. And it does policy enforcement either at the source or the destination. And whether or not the policy has already been enforced has signaled through a flag in the label and this will become important later. Central Management ACI, they use a REST API that's load shared through the APICs for everything. In other words, you can put a load balancer in front of this thing and it would work just fine. It doesn't matter which APIC it shows up on. It uses what's called sharding, which sounds really dirty to me. But everything is sharded. So they have, in other words, they have different instances that run on different APICs. I'm going to get into the security a little later on, I promise you. But they use this API for everything. They use it for the REST, they use it for the GUI, they even use it for the CLI. So when you're on the command line interface and you're SSHed in, they actually run a charoute jail when you're SSHed into the box. And even when you're talking to, even when you're configuring things through the CLI, that even gets translated up to the REST API. So they really have only one way in or out of this system, which is good because it only gives you one thing to secure. And they also have a Python SDK called Cobra that they use to manage. So let's actually get into the interior of this thing. This is how it all fits together. These are your switches. These could be either leaves, these could be spines, and then you have your APICs. And your APICs somehow magically connect to and talk to the leaf switches or the spines. And the APICs run these processes inside, and these are called data management engines, or DMEs. And the DMEs have their own internal proprietary protocol called intrafabric messaging. And the primary front door through this whole thing is through engine X. So the DMEs are responsible for a little part of something called the management information tree or MIT. And all communication happens through engine X. So if you're going from the outside end, you're going through engine X. And they communicate to each other from the switch, from the APICs, all through IFM, and it happens in between the VTAP IP addresses and how they secure that. I'll get into it in a little bit. This is just a GWIS diagram of how this all works. What's important here is that they have the intrafabric messaging that comes in over TCP IP. So internally this is all proprietary They have the doer, they have this model that tells what they need to do in this management information tree and they have a replicator to replicate it to other DMEs if you're doing sharding. And then they have a persistifier which stores to an SQLite database. And each of these has its own database. And they also structured it so that if they're sensitive data, they encrypt that data. If it needs to be encrypted, it will be encrypted on the box. I'm pretty impressed with it otherwise. Communication of policy in ACI, they use DME to DME communication through IFM and they use TLS using mutual authentication. So there is authentication not only of the client but also of the server and the certificates that they use are loaded on the switches in the apex of the factory. So the certificates that are used, they have a very long expiration date and the way they work is they use that certificate to authenticate the client and authenticate the server. I think you guys can see where this is going. The abstract policy on the APIC that they use, the endpoint groups, they actually take that and they make it concrete and they enforce it in the leaf switch using the DMEs. So for device discovery in ACI, if you were to plug an APIC into anything, the only protocol that it speaks at first is LLDP. That's Link Layer Discovery Protocol. If you were to plug into a switch, the only protocol that they use is LLDP. That's all we saw. So I was like, oh great. What's the tax surface going to be, obviously, the LLDP? That's one of the main things that we're going to be using. One of the things that I'm really kicking myself in the bud in but for is that the folks that I was telling you about earlier that did the black hat talk, they found a buffer overflow in LLDP missed. And I feel really bad about it and I wish we had found it earlier but hey, it's patched and there are there are if you're listening to this video, please patch your fabric. Custom TLV, they have their own little proprietary TLV that they used for discovering the fabric so the fabric can discover the APICs. And when the switches see these custom TLV packets, they will open the underlay to the device and it's connected to it. And one of the things about LLDP which makes it interesting is it doesn't travel over switches. It only travels inside of a broadcast domain actually a collision domain. It's only from one point to the other point. There are multiple MAC addresses but one of the MAC addresses that they actually use here will only travel one hop. It won't go any further than that. And what that also means is that unless you really screwed up with your hypervisor configuration, you have to be bare metal attached to be able to tinker with this. You can't do it from a VM. Makes sense. Okay, so here's the real deal. Here's what we did. I've given you all the background. You know about DMEs, you know about LLDP and all those wonderful things. How are we going to do this? How are we going to actually own a data center? Here's what we're doing. Sure, let's own the data center. So in this case, in this situation, you have a device that maybe got owned. It's a bare metal device. Let's say it's not a hypervisor or maybe it could even be a hypervisor where somebody escalated privileges and got access to the hypervisor or maybe they were in a container in the escalated privileges and got access to the hypervisor so that they can send directly to that leaf switch and be able to talk LLDP. So that's the situation here and what we want to do is we want to attack the APIC, which is more than one hop away. And remember this is a layer 3 switch fabric and it has overlays and normally we can't even talk to the APIC because the APIC does not process packets. The packets flow from leaf to spine to leaf. They don't go through the APIC. We don't even have a way of talking to the APIC or touching the APIC so we need something to be able to get access to that. It's the central point of management for the entire ACI fabric. Anything that an admin can do, they can do from there and it makes it for a very tempting target. It's a single target. It's the controller in your entire network. If you have a million endpoints, they're all being controlled from the APIC. So central management, we looked at the GUI and the GUI actually looked pretty good. They use an off-the-shelf framework. I forget which one it is right now. They probably have already changed it anyway because this is from... When we did this eval, it was on 1.1 or 1.2 was the version and I think they're up to 4 something now. So they've been around a long time. So they had cross-site request forgery protections. That's one of the first things we'll look for. They were very well protected from XSS but not for the reasons you would think. Every time they took any kind of data from the user for any field, they would limit all of the fields to basically alphanumeric characters, spaces and things like that and that's all you could do. You could never do anything like brackets. So we thought, how can we do this by looking for some fields that may not be user input? So we did XSS actually cross-channel scripting on LLDP. So the idea is that you may do social engineering attack and say, hey admin, I'm having a problem. I'm not getting connected. Can you make sure that I'm connected to something? Can you go look at my interface and look at LLDP for me and see what you see? Because I'm sending out LLDP and if you say that it's really attached to my machine, I'd really like to know. So you get somebody to look at it. In the meantime, notice on the right-hand side we have local users. We're going to run the script that just sends an LLDP frame, single frame. Boom. Did you see it? Just by taking advantage of the access that the browser had we're able to create our own user. There's a user now called Hacker. And we just have this open at the same time. That happened in real time. So now if you look, we have access to pretty much everything on the APIC. Well, that's fun. So let's go a little bit further. We need a route. We need some way of getting to the APIC. Let's say the APIC is behind a firewall, which hopefully they are. Getting to them, but we do have access to the fabric. So let's see if we can get access to that. So there is something called the infrastructure VLAN. If you've used ACI before, there's this little checkbox on a port that says give access to the infrastructure VLAN. That infrastructure VLAN is the crown jewels to your fabric. Don't give it to just anything. And the reason why is it gives you access to the underlay. Even though it's called an infrastructure VLAN, what they did was they set aside a VLAN for accessing the underlay. When we got into this, we knew that there was ISIS, we knew that things were routed, and we were like, how the hell am I ever going to get my IP address into ISIS? I was like, how is this going to work? Well, as it turned out later on, we found out that they do DHCP. When you have access to the infrastructure VLAN, that's one way. Another way is they actually learn your IP address, which I'll talk about. It's automatic, so they do zero-touch provisioning. Any time you ever attack a fabric or a network for the first time, look at how they do day zero configuration. Chances are you might find issues there. Okay, so communication is a layer two. We saw LLDP and we know that they're a custom TLV, so we wrote a SKP plug-in to generate these TLVs. And there's no authentication before them, so what happens if we tweak and replay? We find that it says, hey, you're a controller. You have access to the controller endpoint group and you have infrastructure, so that's what we did. And this is what it looks like in the underlay. So you have the A-pics. The A-pics are usually dual-attached to two leaf switches. And then they have a set of subnets, or there's a subnet that's set aside on that leaf switch that is advertised out and this whole thing is on the underlay IP address range that you will specify when you first set this thing up. And you'll notice that it's all completely routed over ISIS, and if we're the attacker, we can do the same thing and represent ourselves up as an A-pick. So we're spoofing ourselves to the network and saying that, hey, we're an A-pick and it believes us and it allows us onto the fabric. Okay, once we're on the underlay, we have a much larger attack surface, right? We can get access to all of the different A-picks. We can get access to the hypervisors that happen to be attached to the infrastructure VLAN. And all of the switch in A-pick DME ports are accessible. We can get to them, but they're protected by TLS, okay? They're using mutual authentication. We need access. We need some kind of way of getting access to that TLS connection. What do you think we did? Anybody guess? We need a cert and we need a private key. We could steal it from something. That's one way. What's that? Spoofing. Spoofing? Well, it has to be issued by Cisco because it's got a trust store and yeah, you can steal it. The DMEs, they communicate policy throughout the fabric. That includes all of the users and credentials on the A-picks, okay? But we don't have access to the DMEs because it don't have access to TLS credentials. We don't have the certs and the private keys. So, we looked at the certs and the certs showed that they were issued by Cisco and we know that they are issued by Cisco at manufacturing time. And they have an expiration date way, way, way out. And there's really nothing else that was really special about them. And so, we tried another private key and certificate that was issued by a Cisco certificate authority. What do you think we used? If you have... If you're doing attacks against a PKI, keep a cache of certs and private keys available. You can extract them out of these things. It's not that hard. It's hard, but it's not that hard. It's just basically on flash. It's kept by Cisco. It's a private key and it's issued by Cisco and it comes from a Cisco PKI. Not all things are that easy to get off. We actually have a module called the TAM, the Trust Anchor module, which is tamper resistant and it's actually programmed somewhere else that's not in a contract manufacturer facility. And they ship those chips to be embedded on a device. Those are a little bit harder to get to, but if it's something that's on flash, you can get to it. So, we use that. It's a private key and this is a video from soup to nuts of owning an APIC. And so, we're going to get those credentials. We're going to stick those on the APIC so that we can get access to SSH. The first thing we do is we're going to listen for LLDP. And you could see that we're now sending LLDP and we have access to that endpoint group and these are all of the VLANs that we have access to. Now, from that point forward, then we can ping our default gateway that is on the infrastructure of VLAN. And once we do that, we've been learned and everybody now knows about us. The spines, the leaves, they all know about that IP address that we're reusing because they redistribute, basically, ARP into ISIS so that we're able to get discovered. And then, from that point forward, then, so this, that's basically what I was telling you. And now, we'll connect to the APIC. The same thing, we're going to create a user with our username and password. Then we're going to be able to SSH in so when you have a username on the APIC, you can SSH in. You can get access as admin but not root. So we wanted root access to the APIC so that we could do something and I'll be showing you in a little bit. And the way we did that, there was a open internal service that you could only get to if you were logged in locally. It was not an external service and it had a command injection vulnerability in it. And that service ran as root and that gave us access to the APIC as root. And then once we had that access, then we installed rootkit. I'll show you that. Okay, now we've completely withdrawn our LLDP frame so we no longer have access to the underlay because we don't want to attract attention to ourselves but we want to get access to the APIC. The APIC has a VLAN that it sets aside called in-band network so that gives us access to the APIC and we just add it directly in using our port which is 1337 and we logged in and that gives us a bind shell. And that's something that we positioned ahead of time. And then we run script dev null and that gives us access to an actual sudo terminal so we can run anything we want. That's kind of a nice trick if you didn't know about that. Type ID, we see that we're root and you'll notice that there's no processes there to listen. Part 1, 3, 3, 7, they're hidden. There's also no there's no directory there called rootkit and we just log in and there's the subversive rootkit. The claim to fame with the subversive rootkit which is really neat if you haven't played with it I strongly suggest you do it, it runs on Linux. And it does it's hooking through the debug registers of Intel. So you don't actually hook anything you don't modify the kernel in any way. It uses the debug registers to be able to hook the different system calls and one of the system calls they have of course is right and you're hooking that in delete speak by just turning lulz mode on and lulz mode off and you can see the other things you can do or you can hide files, you can hide processes but of course we didn't hide the kernel even though we probably the kernel module even though we probably could have with me so far. Okay, so after that oh, that's we're done. So this is how it worked in Summary. So as a bare metal attacker we spoof the APIC VLLDP we access the infrastructure VLAN we use the static IP you could have also used DHCP and that gives us access to the local switch that we inject our IP and ISIS that's automatic that was something I was like how the heck are we going to do this? It turned out it was a lot easier to do than I thought we gain access to overlay one that's the name of the infrastructure VLAN we steal a cert from an existing switch or use the cert or key from a phone the access to the data management engine the one that we're interested in is called the infrastructure engine and we connect to the shard leader for AAA we create a local admin user we then connect to the APIC over the overlay one via SSH we SSH in skill the root user SSH keys which we actually could have done we did that through a command injection vulnerability that gives us access to the APICs and the switches we install a root kit remove the admin user and now they're owned these are the kinds clap if you like for our customers you may be wondering like why am I doing this why am I talking about Cisco in this thin depth I want you all to know that we work really hard the ASIG team works really hard I'm blessed every day that I get to work with such great people it's just, it's absolutely amazing some of the smartest people on the earth are in ASIG I just love it, I just go to work every day I just love it okay, owned, yes so we have access to the APIC but we want access to the endpoints so in CME they have a VXLAN it is mostly standard I think that they use a weird UDP port but there are a couple flags in there that says whether or not policy has been applied the source policy applied flag so once you have access to the to be able once you have access to the infrastructure vlan you have direct access to be able to send in CME VXLAN encapsulated frames routed through the leaf switches and then they can get to their target and you just say hey I've already done source policy don't worry about it and then the destination leaf switch then we'll forward it onto the destination so it's just as simple as that so once you have that access for instance once you send something to it you can spoof your IP address so maybe you may respond some time somewhere off your network maybe you can get tumbled somewhere else so then you can respond to it that way so then that's basically all there is to it it's quite easy so here are the fixes that they did XSS obviously is very easy the XSS vulnerability was actually a DOM XSS vulnerability it was not stored it was not reflected or anything like that it was in the DOM so make sure you sanitize don't just rely on your back end to provide safe data to your web app they use the single page web app it was all done through single page so sanitizing content and escape anything you put into the DOM context of course matters obviously strict mode this is one of the mistakes that we made they couldn't turn this on without breaking all the existing fabric so they had a flag it was a nerd flag that we probably couldn't really talk about very much to even to our customers because we wanted we didn't want them to know exactly how we just said turn it on it's one of those things because if you don't have it turned on then you're able to do the LLDP and then able to get to the underlay and so strict mode when you have it turned on it fixes the LLDP underlay access issue and it wasn't unfortunately backwards compatible and that's why it had to be a flag that you turned on one of the things that just went out recently was they made it so that it was on by default because this has been three years now and people probably should have had this on by default now so when the host claims to be an apic over LLDP the switch will require a random bearer token to be inserted into LLDP TLV to open the underlay and the way it learns that token is through connecting to a DME that's on the leaf they have an apple that they put on the port that says you can basically connect to the leaf but no further okay once you connect to that you'll have access to that one DME and then that's authenticated using mutual TLS using the certificates that have to be put in a whitelist ahead of time once you have that access then you have access to the underlay and only then okay another thing they did was they strengthened the subject distinguished name checking for certificates this is unfortunately very common guys especially in embedded products just don't assume because that they're using a PKI that they're not actually checking things you'd be surprised they don't get a let's encrypt certificate try to connect to something and then it may say oh hey this is on my trust store I'm going to allow it they may not do the subject checking make sure you look for that because it's been in five products I think now so far no reason an IP phone should be able to talk to the data management engines the admin registers these APICs and switches this is actually part of just what they do when they first set up the fabric and then you'll put in the serial number ahead of time or you'll actually just say accept this and it puts that into a whitelist that's actually used for checking this this check happens during the initial TLS exchange and then IVX LAN spoofing and unfortunately by design it's part of the use cases that hypervisors be able to send directly to the leafs that's the Vleaf use case so unfortunately there's not really much you can do it's just you really want to protect the infrastructure from untrusted hosts make sure you do this there is a new feature called the cloud sec that's on the newer switches that actually does encryption from Vtep to Vtep and anyway it's something that I'm kind of excited about to see but it uses something very similar to IPsec between the Vteps okay so lessons learned don't expect hardware to play nice someone else's box is plugged into your switch as user input don't ever trust unauthenticated information regardless of where it's coming from you'd be surprised even if you're doing LLDP you can actually do XSS through LLDP who knew context sensitive sanitization before inserting into the DOM PKI's public infrastructure is especially in hardware it's really hard because once you've done it you're done it you're stuck you can't revoke certificates you know you don't have OCSP you know the open certificates as protocol or CRL's you know once you've once you're stuck you're stuck okay you can't really check those kinds of things don't abuse protocols LLDP is strictly for discovery it's just like Cisco discovery protocol discovery link local discovery protocol any questions now how am I doing on time actually got through that faster than I thought yes the new Cisco certs that are in on core are they're going to be taken into account I don't know you could send me a drop me an email or drop me a Twitter and I'll see if I can find out VXLan yeah VXLan is data plane in the certificates that they do is for control plane or management plane so they're kind of different they're orthogonal but they're different any other questions the script what the script does is it connects so when we yeah it gives you a PTY a pseudo terminal so normally when you like if you have a bind shell that's listening on a socket doesn't really give you a terminal so if you try to run VIM or something like that it won't work so if you run script dev null that just gives you a PTY and it doesn't save it anywhere which is really nice so it'd be surprised how many little tricks that you learn you know just from watching other things yeah I can't run anything how do I just run yeah script dev null yes sir are there any similarities between ACI and UCS the APIC uses UCS server it actually is a re-imaged UCS server that has its own certificate an ID for it but other than that it's just a UCS server good question any others oh the question was where did we run this we use testbeds so we set testbeds up ahead of time that's actually probably one of the most time consuming parts that we're not actually doing work is just getting testbeds to work and for those of you who are doing like red teams or cloud pen tests it's the biggest I hate doing cloud pen tests because you're always dependent on somebody else to set up like a dev environment and hopefully it's exactly the same as the production environment and most of the time it isn't but what we do is we go through and we try to set up or we can we try to set up an on-prem environment that looks fairly complex you know in real any other questions any of this interest to you if you like this kind of work please apply go to jobs.sysco.com and type in ASIG I think there's only a couple jobs down there right now but we're always trying to hire good people any other questions no questions about interfabric messaging, TLS leaf switches, spine switches connections, nothing like that thank you thank you for coming