 So, we're going to get this next talk started. Let's get a big round of applause for them. We've got at least one new speaker. Two? Yeah? All right. Hey everybody, three for the price of one. My name is Gina Matthews and I'm a computer science professor at Clarkson University in way, way upstate New York. Knows Bleed New York because we like to affectionately call it right across from Canada. Ronnie Bull is both a PhD student with me at Clarkson and also a faculty member at Utica College. Caitlin Trumbull is a senior undergraduate who's been helping us run all these attacks that we're going to tell you about. In particular, we're going to talk about VLAN hopping and art poisoning and man in the middle attacks in virtualized environments. And this is a bit of a follow on from the presentation we did last year at DEFCON. So, here's a little roadmap what we're going to do. We're going to talk about the context of the problem for Layer 2 network security in virtualized environments. The problem in multi-tenant environments, especially in cloud services. We're going to talk about the test platforms that we used and all of the different virtual, the hypervisors and the virtual network devices that we tested. And we're going to talk about the specifics of the attacks. We've got some great videos and demos to show you. Specifically of the VLAN hopping and art poisoning this time. We also have videos and many, many details of the map funding and DHCP attacks that we talked about last year. If you want to look those up, and finally some conclusions. So, the key question is when you have virtual machines that are hosted in a multi-tenant environment, how safe are they? What can they do to each other over the network? They're essentially connected to a virtual version of a physical networking device. And the question we're trying to answer is the Layer 2 network attacks that typically work on physical devices. Do those work on their virtual counterparts as well? And if so, what are the implications, especially in a multi-tenant environment? Cloud services that rely on these virtualized environments could be vulnerable and that can include data centers hosting mission critical or sensitive data. This is not the only class of attacks from co-located virtual machines. It's kind of an old story. You're vulnerable to those that are close to you, but in a public kind of multi-tenant environment, you're not quite sure who you land next to. We've done some other work in the past of the types of things that virtual machines can do to each other as have many other people, but in particular today, we're talking about the attacks over the network. So, here's a pretty simple diagram of the setup we're worried about. We have a physical host here, and on it we are hosting a bunch of virtual machines. One of them, that's a nasty, malicious one, that can do things to the other victims, and you can see they're sharing a virtual switch or bridge, and that's passed through to a physical network interface. And to kind of jump ahead, the bottom line is that we do see in our research that virtualized network devices have the same ability to be exploited as physical devices. Sometimes we see more vulnerabilities in physical devices. Sometimes, in certain cases, we see less than a physical device, and we're going to show you that for the attacks we're talking about today. Another interesting thing we've seen is that some of these attacks have the ability to weave the virtualized environment and affect the physical networks that they're connected to as well. So, that's another interesting bonus for you. What are some of the consequences if a malicious VM successfully launches a layer-to-network attack in a multi-tenant environment like this? What are some of the things that they could do? They could capture all the network traffic coming from the victims that are co-located with them. They could redirect traffic. They could perform man-in-the-middle attacks, denial-of-service attacks. They could gain unauthorized access to restricted subnets, and they could affect performance, which is, I guess, a sub-example of denial-of-service. So, here are some of the test scenarios and results that we're going to be talking about. We're going to give you a little update on MAC flooding, especially because we've changed over our testing platform completely since we spoke here last year. We're going to talk about VLAN hopping in quite a bit of detail. Also, art poisoning and some man-in-the-middle attacks. Here is a lovely picture of the test environment that we were using last year. It was pretty much built from pieces and parts that we could salvage. Rest in peace. You've served us well. Thank you very much. Here are four complete details, some of the hardware specs and the hypervisors and the virtual networking devices that we were testing when we spoke to you last year. All of those details are on the DEF CON 23 CD. We have nice new shiny hardware. Thank you to college for that. And it allowed us to basically repeat all of the experiments we did before in just a much more controlled environment than our pieces and parts, and also extend the work. So here is some of the gory details of the test environment that we used this year. You can see a lot of different hypervisors, a lot of different virtual networking devices, and you can see some details here of the hardware specs. Again, all of the gory details are available for your reading enjoyment in the DEF CON docs. Okay, so to kick things off and also just to provide a good transition between what we did last year and this year, I'm going to give you a little update on the MAC flooding stuff. First, you know, here's a simple diagram of the scenario in which we're doing MAC flooding attacks. Again, we have two VMs attached to a virtual switch. We have a victim, the target VM, and we have an attacker that's patched through to a physical interface that connects to a physical switch and the internet beyond. And the target VM is just doing a set of regular pings, and then we're going to see what happens to the performance of those pings when the attacker launches a MAC flooding attack. And here in this graph, which is something we showed you last year, you can see from like zero to 65 or 70 or so what was happening when it wasn't under attack. So low latency pings, no trouble. You can clearly see when the attack gets launched, and you can see just after 240 when the attack stops. You can see clear impact on the network performance for the victim VM. And this was done on Gen2 in a Zen-bridged interface. And this gives you just a sense of, you know, we're just doing a lot more systems and platforms this year. So this is the same test performed across not just that Gen2, Zen-bridged, but also to the Zen OpenV switch, Citrix Zen server, Hyper-V standard switch, Hyper-V Nexus 1000V, the Proxmox-bridged, VMware ESXi, and a control physical switch, a Cisco 2950. And so some of those, when the attack starts, don't, aren't affected. It's a little difficult to read from this graph. Maybe it's a little easier to read here. It's still not incredibly easy. We realized at the last minute there might have been a better way to do this one. But from this, if you look carefully, you can see that the Hyper-V standard, the Hyper-V Nexus, VMware ESXi, were not vulnerable. And I'd like to make the point before we head into some of the rest of the things that all of these Layer 2 vulnerabilities that we're discussing were targeted toward the virtual networking devices, not so much the hypervisors themselves. What we're testing is what happens with the virtual networking devices. So it's interesting you see a mix, right? Not everything's vulnerable, not everything is safe. And we're going to see that trend continue across all of our tests. It wasn't uniform, which things were vulnerable and which things weren't. At this point, I'm going to pass it over to Ron. He's going to tell you about the VLAN hopping. Okay, so we've performed a few VLAN hopping experiments on the environments as well to see what we could do about getting frames onto unauthorized networks. We found some interesting results. So some basics about VLAN hopping attacks, just allowing unauthorized access to another virtual LAN on a packet switch network. Two methods of doing this currently, switch spoofing and double tagging, where switch spoofing is Cisco proprietary using Cisco protocols, and double tagging is an exploitation of the 802.1Q protocol. So basic virtual LAN information, we have a basic frame, Ethernet frame here, and what we do to get VLAN going is we stick a four byte tag in there, or VLAN header, with 32 bits of extra information to support the VLAN ID to show where that frame is supposed to go on the correct networks. So switch spoofing. Here's a CDE listed about Cisco switches that if they support 802.1X security allow attackers to bypass port security and gain access to the VLAN via spoofed Cisco discovery protocol messages. So a little bit about the Cisco discovery protocol. Basically it's a proprietary protocol that Cisco has on their switches that allows two or more switches to identify each other's operating systems, automatically negotiate connections, things like that. So if we can have a vulnerability in that protocol and actually exploit it, we can get on a different VLAN. A couple more CDEs and vulnerabilities for switch spoofing. Cisco Catalyst 2900 Virtual LAN switches allow remote attackers to inject 802.1Q frames into another VLAN just by forging the ID tag. And a quote directly from Cisco from one of their white papers about dynamic trunking protocol. Basically if it's enabled, you can send a fake DTP packet out and switch that port from auto into a trunking mode. That's pretty effective across some of the environments. It's important to note that on most Cisco switches out there, at least some of the earlier ones that are still in production today, DTP auto is the default setting. So you want to make sure you harden that. Okay, so again, this is just another Cisco proprietary layer 2 protocol allowing you to do that automatic configuration of trunking connections between two switches. Pair this with CDP and you pretty much have an automatically configured network. So some of the consequences of this. Basically the attacker can have a full trunk connection to the switch. They can make themselves into a trunk, spoof themselves as a switch, which allows them to have two-way communication on that VLAN. They place themselves directly on that VLAN allowing the eavesdrop on the traffic, send messages to and from systems on that VLAN. Okay, so here is a switch spoofing demo. Basically we did this in the ESXi environment. The setup is we have a virtual machine within the ESXi environment and it's going out through the virtual switcher bridge. ESXi is more like a bridge than a virtual switch, the standard switch on it. We have that connected to a physical Cisco 2950 switch and the port is set to DTP auto just default configuration. And then you see there's another system on the other end that's connected via trunk link with a VLAN 20 and one support over that trunk link going into another hypervisor environment there. So let's take a look at this demo here. And all these videos are on YouTube. The links are on the white paper on the Defcon CD so if you want to watch them later. Okay, so on the left-hand side we can see the configuration for the Cisco 2950 switch. On the right-hand side we see the ESXi network settings for the Kali virtual machine that's on that system. We're going to first take a look at the interface settings for the ESXi server, in this case it's called Luminara. I can see it's connected and it's just set to VLAN 1. It's just on the default VLAN of that switch. Nothing was changed besides the defaults. It's not set to a trunk port or anything. We're just going to verify the trunk status on that. And we can see that it's using 802.1Q encapsulation. It's not trunking and the mode is set to auto. So it's just default configuration. Nothing was changed. To go over onto the virtual machine we're going to take a look at the network interface here. We can see there's just the basic network adapter. It's on an isolated VLAN test environment we created. It's a separate network for that. Which is over there on the switch on that port. So if we dig a little deeper into these details we can take a look at that network and we can see down there on the bottom we set up that VLAN test network and the Cali virtual machine is the only system on that network. And if we look at the properties of that we can see that this connection has no VLAN ID set to it. So it's just on the default VLAN for the ESXi virtual switch. Now we're going to go over to the virtual machine itself grab console access and we're going to start the attack. In this case we're going to just use the Yersinia program so we're just going to type Yersinia minus i to get into interactive mode. Got to bear with me a little bit. Okay and then we're going to select the default network interface on this. So just hit enter. And then we'll have to launch the attack. So we're just going to go up and select the DTP or Dynamic Trunking Protocol and then launch the attack press 1 and we can see it successfully went into trunk mode. So the dynamic desirable effect was affected. Over there on the switch side we can see the line protocol went down it came back up again and if we check the status on that trunk port we'll see it indeed went into trunking mode. You see it's now instead of VLAN 1 it shows the trunk. If we take a look at the full status we can see that it's DIP set for trunking and auto and A2.1 Q encapsulation. So the system was effectively put on as a trunk now we can access any VLAN we want that's associated with that trunk. So in this case any system that was on VLAN 20 or whatever. So that was the switch booping attack. In the ES Actify environment and see. Okay so results across the board here we can see that the physical Kali 2.0 control system we set up the physical switch we tested it with a control to make sure it worked first that way before we proceeded into the virtualized environments. You can see it pretty much worked under every system that used bridge networking instead of a virtual switching networking. So if you're using an environment that has a bridge virtual switch this is going to work. The open source Zen with Linux bridging it worked in the ESXi environment as you saw and it also worked in Proxmox which also supports bridging as well. So any environment that used the virtual switch it did not work because basically those that DTP packet when it was sent through it was blocked for getting to the physical device. Okay so mitigation in the physical world you just disable unused switch ports don't give access to people to actually get connected to those ports. Disable CDP and DTP for those ports that you really need it set up on. Restrict the amount of trunk ports you don't just want to have ports set up to be trunks if you don't need them. Although all the hypervisor environments if you read their documentation they specifically say that for best practices set up a trunk port for connection to the virtual switch so that way you can actually support VLANs into your physical environment and other virtual environments. So really just limit these to access but in the virtual world it's really hard to do this so if you're using bridged environments that this could be affecting your network. Okay let's talk about double tagging now here's the CVE for double tagging the 802.1q VLAN protocol allows remote attackers to bypass network segmentation and spoof VLAN traffic via message with two 802.1q tags so we saw before that we have a normal basically this is the normal traffic this is the scenario we used here where we have two switches a trunk connection between them supporting VLAN 1, 2 and 3 you can see some two machines on one side connected to that first switch one's on VLAN 1, one's on VLAN 2 and on that second switch we have VLAN 1, 2 and 3 so the goal is to get some frames across between the two switches from that system on VLAN 1 and try to get it over to VLAN 2 or 3 instead of VLAN 1. So here we can see the normal flow of traffic we're just using a single tag where it goes through VLAN 1, goes into that first switch goes across the trunk connection still with that VLAN 1 tag and then it goes into that system that's VLAN 1 on the other side basic normal flow of traffic and we can see that the other VLAN 2 and VLAN 3 machines do not get that frame it just gets blocked so here's a comparison of frame types the first one is a standard Ethernet frame without VLAN support the second one is that standard VLAN tag with one 802.1q tag in the middle or header in the middle and the third one down on the bottom is doing 802.1q double tagging so this is where we're actually doing a frame within the frame with multiple ones so in this case here we can see the effect of double tagging we have VLAN 1 and VLAN 2 on the one side that tag has a 1 and 2 in it so we have two headers in that frame it goes into that first switch it doesn't get passed on to that first system connected to that first switch because we're still reading that VLAN 1 tag we're not seeing the VLAN 2 tag as soon as we go across that trunk connection that VLAN 1 tag is stripped off passing over the frame with the tag on the side it goes directly to that system on VLAN 2 and doesn't go to the VLAN 1 or VLAN 3 here we can see the same thing with 3 just a different example in case of the consequences an attacker can send packets to any targeted VLAN as long as it's supported over that trunk connection and as long as they have access to that native VLAN the target on the axis is isolated it's in a native VLAN broadcast domain this is not a good attack for eavesdropping because basically it's a one-way attack on another VLAN you're not getting anything back to you so it's good for DOS attacks it's good for maybe a one-way covert channel to another system on another network but that's about it so we have three different scenarios we ran with this the first one was a control scenario we had the two physical switches which this was known to work in we had a physical attacking system connected on a native VLAN and went through that first trunk link to the second switch and we were able to access this video right now, it does work here you guys are more than welcome to go click the link but we have two more videos I'd like to show they're a little more effective than this one so the second scenario we did was two virtual switches so we took out that second 2950 and instead we're going from one virtualized environment through the physical switch into another virtualized environment so in this case we have an attacker VM, it's on no VLAN just on that native VLAN on that virtual environment going through that virtual switch connected through a trunk link via best practices on one and 20 into the Cisco 2950 through another trunk link connected to another virtualized environment which is going into a virtual switch there on an access VM on VLAN 20 so in this case, in this demo we have the attacker systems on Zen server and the target systems on Proxmox okay so again on the left hand side we can see the switch configuration for the 2950 that's in the middle of these two two systems port 31 is the network Zen server port 29 is the Proxmox system yes I do like Star Wars the middle system is Zen server and the far right system is the Kali system on Proxmox so the Proxmox system is our target the middle system is our Zen server attacker what we're going to do is take a look at those interfaces just to check out the settings so port 31 is our attacking system except for trunking, notice the mode is on I didn't do auto because I actually specifically set this to trunking mode for best practices and we can see the 802.1Q encapsulation so what we'll do here now is check out the settings for the attacking system and we can just see I have it just set up on a basic network interface port, there's nothing special about it there's no VLAN tags on that system or not it's just a straight connection into that server on that VM and that switch, the Proxmox system you can see it's tagged for VLAN 20 for that client on that system so we're going to run the attack from the Citrix Zen server VM Kali there to the Proxmox system which is also Kali on the Proxmox system we're just going to run a simple TCP dump command and we're going to grep out for ICMP traffic to try to just pull out that ping packet we're going to try to send through so just basic TCP dump with some verbosity and grepping for ICMP and then over on the Zen server system we're going to set up your Cinead to launch the double tagging attack and just selecting that default interface again so this time we're going to select 802.1 Q protocol and then we have to do some setup on the bottom there so down in the bottom you can see there's tags for input VLAN and then the target VLAN there's also the source IP address and destination IP address so we want to go from VLAN 1 to VLAN 20 and then we want to go from a source IP address it doesn't matter so we spoof the address of 192.168.5.5 and then the target system 192.168.1.11 so we have to just input that in there for the settings and we're set up there so we can see up on top we're using ARP 192.168.1.11 so we see the ARP protocol there and actually went through we're going to launch the attack sending out that ICMP protocol or ICMP frame and sometimes you have to keep launching this attack a little bit what I found is when I started doing this testing nothing happened and I sat there for a little while longer and just kept pressing the attack button and then finally what happened is I'll sit here in a second took a little bit of time so if you're doing this at home boom, we see all the traffic so there was a little bit of latency going from one virtual environment through the physical switch into another environment but we can see that the frame was actually successfully forwarded from VLAN 1 through a virtual switch through another virtual switch to VLAN 20 on the other side and we moved the address of 192.168.5.5 going to the Kali1 host name which obviously ETC host file on that Kali system is resolving the local IP address of the host name on that system so this was successful we were actually able to do the double-tagging attack through a physical switch into another virtual switch okay so we have one more here in this case we decided hey if we could do it with two virtual switches let's take it out let's go through one physical switch and the second virtual switch will successfully do that double-tag so we don't have three switches now we're down to two, one's a physical, one's a virtual in this case the physical Kali system is the attacker and we're going into a Microsoft Hyper-V guest on a Cisco Nexus 1000V has anybody ever tried to set up a Cisco Nexus 1000V? how fun is it? right? oh yeah okay so let's take a look at this one same pretty much setup we have on the left hand side and there's a port one which is connected to that physical Kali system which is in the middle we can see it's just on VLAN1 no trunking going on there so this has a trunk connection between the physical switch to the Cisco Nexus 1000V on the Hyper-V environment so we'll check out that interface as well and we can see it's set for a trunk as per best practices and 802.1Q encapsulation again we have VLAN1 2 and 10 supported on here in 20 now we'll go over and check the configuration for the Nexus 1000V which supports the same Cisco commands pretty much for identifying trunk connections so here you can see we have Ethernet 3.1 is the trunk connection connected to the Cisco 2950 and we have VETH1 which is that Kali system connected on VLAN10 in this case and that's connected to that Cisco Nexus 1000V so in the middle here we have our attacking system which is a physical system and then on the far right we have our Hyper-VVM and again we're just going to run TCP dump a little bit of verbosity and grep for ICMP traffic forward this a little bit so we can speed it up and then on the attacking system we're going to run Yersinia again we're pretty systematic in our approach this testing to make sure we did it across the board and it was the same everywhere we did it now our physical Kali system we have a multi home so we have four network interfaces on this so we can do different testing on different networks so in this case we choose Ethernet 2 because that's the one that's actually on that VLAN test network so we just went in there and changed the configuration of that interface and then we're going to go up and choose the 802.1Q protocol again and set up that attack down in the bottom we've already seen that so I'll forward through that a little bit here just verifying the IP so we're going to that IP of 122.168.1.199 in this case is the virtual machine in HyperV and we're going to start launching this attack now so here's the same thing just launching it we can see the spoofed IP address going to the destination IP of 100 802.1Q and the protocol's ICMP and if we just start launching that thing again took a little bit of time to get this thing to successfully go but in the end we see that it actually works so it went through the physical switch from a physical attacker through that physical 2950 into the Cisco Nexus 1000V onto VLAN 10 on the target virtual machine so it actually worked so we're seeing that we can actually do double tagging and switch spoofing attacks in virtualized environments which is pretty scary so overall results of this the first set of charts there on the left hand side show the open V switch Linux bridging pretty much everything worked for going in so this isn't coming from this is trying to get into an attack on the side of a virtualized network so it worked on every single environment except for the standard V switch for Hyper-V we find this is because when you're launching your senior you're actually spoofing MAC addresses on the attacker side and what we saw last year we were discussing our MAC flooding attacks those also didn't work in the standard V switch for Hyper-V because the standard V switch has some protection for MAC spoofing against it in the server 2008-2010 environments so what we expected against this attack was that MAC spoofing safety mechanism so if that wasn't there or if you could do this attack without spoofing MAC addresses you could probably get away with it in that environment over on the other chart we see what was successful, what could we launch the attack from which environments could we actually send something out from in this case we saw it worked in the bridging environments so anything that was a bridge except for ESXi in this case worked we also saw it work in the standard Zen environment with open V switch so pretty much all the open source software was affected by this attack mitigation techniques, standard ones for physical devices is limit your access to VLAN 1 or just eliminate VLAN 1 all together maybe label it something different so it's not easily identified restrict access to switches by MAC address which can be spoofed so that's a little bit difficult it's kind of a double edged sword there but the heart of this attack is having access to that native VLAN as we saw and even the native VLANs within the virtual switches which may not be identified so you really need to look at hardening these switches if you're going to be using these in production environments okay so we're going to pass on to Caitlin now for ARP spoofing ARP spoofing ARP is an address resolution protocol it's a layer 2 network protocol that maps the physical MAC address to the logical IP address each system on a network has an ARP cache and it stores the information from discovered nodes on that network ARP cache the common entries in ARP cache is the default gateway and the local DNS the process for ARP is an initiating system sends out a broadcast request to the entire network it asks who has a certain IP address a node on that network who has that IP address responds with its MAC address and the information is stored into the ARP cache okay so this is just a demo and on the left hand screen is the router or the default gateway in the center is the target machine and on the right is the attacking machine on each of them we have the ARP cache displayed and you can see the IP address of the attacker and the target over on the default gateway and you can see the MAC address displayed with it showing each separate entry and then over on the attacking machine we run our ARP poison script which is just a basic python script and that's going to run it's going to go through and poison the ARPs so we're going to go over on to the target and the default gateway and check our ARP cache and you can see as it comes up in displays with the two IP addresses that the MAC addresses are now the same so on the default gateway both the target and IP address are showing that the MAC address is the same so that was spoofed into tricking the MAC address then we're going to go and we're going to run a ping just out to Google just to get some packets flowing get some traffic and that's going to finish up the script and run through there once the script is finished it's going to restore the target machine so we're going to go over and stop the ping and restore the ARP cache and check that over and then you can see on the target and on the default gateway that the ARP cache is now displaying as it was in the beginning so the target was restored and then we also ran a TCP dump script so that's going to show the traffic that came through and then you're going to be able to see Google's IP address showing traffic to the IP addresses that we captured packets from Google and then for the results you can see it worked on all the platforms ARP spoofing mitigation on Cisco switches you can make use of the DHCP snooping and dynamic ARP inspection validation it's not supported on the Cisco Nexus 1000V and then there's also ARP watch which runs as a service on a Linux system and monitors the network and changes for ARP activity okay so to kind of wrap this up and put it into some context we can see that the results do show that virtual networking devices can pose the same sometimes even greater risks than their physical counterparts so that's an important thing for all the users of multi-tenant environments to understand which systems were vulnerable varied pretty widely across the tests there was no one best system tests everything was vulnerable to that lack of sophisticated layer 2 security controls similar to what is available on integrized grade physical switches greatly increases the difficulty in securing virtual switches against attacks like these so what is the bottom line impact then to someone running in a multi-tenant environment a single malicious virtual machine has the potential to sniff all the traffic passing over a virtual switch can pass through the virtual switch and potentially affect the physically connected devices as well allowing traffic from other parts of the network to be sniffed as well so that's a very significant threat to confidentiality, integrity and availability of data that's passing over a network in a virtualized multi-tenant environment there's a lot of very mission critical stuff being housed in virtualized multi-tenant environments these days as we know so what can users of these systems do well educated users can question their hosting providers which virtual switch implementations are being used which attacks are they vulnerable to are they doing any additional mitigation strategies against attacks like these they can also audit the risk of the workloads they're choosing to run in the cloud with this environment with this understanding in mind they could consider extra security measures on their own or request additional security measures from their hosting provider users of encryption, additional service monitoring more detection of traffic attack traffic that could be present alerting there's a lot of things that could be done what are some of the next steps for us last year was our first time at DEF CON this is our second time we're a pretty small team we made improvements this year we did more attacks than we did before on more regular hardware and we're really happy with that but there's a lot more we'd like to do at the institute for apples to apples testing of virtualized environments we have kind of a long history of this we've also done what kind of malicious things kind of virtual machine that's co-located do if it hammers the disk or it hammers the CPU or it allocates a ton of memory or all sorts of malicious things we've done apples to apples testing of different kinds of virtual machine migration technologies and what's going in the clear and what's not and how long it takes and we think there's a real place for that kind of apples to apples independent testing of those things if you're in a situation of wanting to deploy some of these technologies both proprietary and open source it's really hard for you to bring all of them in house and run through all of these kind of tests so we've seen a lot of great feedback about oh thank you for doing all of these tests and pointing out where the problems are we've also seen vendors fixing things we'd love to see industrial partners to participate in an institute of this kind whether it's places that have technologies they'd like tested and would like to see our results and have a chance to say okay yes this is the results in the default environment but if you turned on this thing then the results would be completely different we would love to partner with people like that we'd also love to partner with people who are using these technologies in production environments and would be interested in these results we have leads we would really just like to do more testing in production environments basically using best practices out of the box proprietary and open source technologies over time we've seen some of the work that we've done result in vulnerabilities that are then patched so we're really proud of that but we understand that in production environments many times there's additional secret sauce or additional monitoring or additional conditions that are put in place we actually got some great leads last year after our talk that we still need to follow up on but this is us this is a pretty small team and the biggest bottleneck we have is the need for more great students like Caitlyn to be funded to do this kind of testing so if anyone would like to help us sponsor a few students we promise it's good educational value so this is e-mails for Ronnie and I also all the details if some of the typing and the demos you didn't quite get slowed down zoom in on the videos they're all there all the great details and all of the publications presentations, demos are all at Ronnie's website also like to give a shout out to Nick Moranti who helped us get our AV going after some last minute troubles and I guess now we'll take some questions Hi so I see you did a bunch of testing on the hypervisors the virtual switches that come with the hypervisors did you try this with anything like open daylight or VMware NSX these kind of SDN solutions that sit on top of the hypervisors not yet but we'd love to I did start looking at some SDN stuff but we've seen that there's a lot of there's a lot of vulnerabilities in the controllers out there like where you can actually take control of the controllers so by implementing SDN we're looking at implementing more security flaws into these environments when we're trying to reduce that footprint I looked at with the DHCP stuff we did last year we presented on that I've found a python script just by using simple python and scappy you can write a simple script you just daemonize that will find you know rogue DHCP servers on the network and fix that so I think by using smaller scripted demons that we could actually deploy without putting more complexity in these networks would be a better solution than trying to throw the monster of SDN on top but many people are using SDN so it is a good place for us to start exploring but that's a whole other research track right on thank you great talk I can see you tested against Zen you tried against Amazon inside the Amazon environment I know we haven't tested it within Amazon I think that is not in their terms of service but we did have some leads within Amazon which we'd love to follow up on I'm not sure we haven't done that yet we'd love to so have you done any cam table overflow attacks against the virtual switches as well that was last year's talk so if you look on the deathcam media server it's all there demos and everything so along with you were talking about the 2950 physical switch so that's running I'm assuming like iOS 12.04 or something like that were you able to sort of do that attack on like a newer version of the iOS like a 15 dot something let's go back up to the thank you I don't want to go to details here I want to go to that picture I'm not exactly sure what the model of that switch is but if you look up here on the top of the rack I can get there again on the back there's two switches up top there those are brand new Cisco's I accidentally tested it against those and it did work so I don't recall which environments but yeah it did I was able to switch because it's not using the 802.1 Q encapsulation so if a switch doesn't support that your double tagging is not going to work there's other encapsulation types for trunking that are supported now on the newer devices awesome thank you hello I'm from Brazil first of all thanks for the great presentation was wonderful have you got the first part I'm from Brazil first of all thanks for the presentation I have worked with a lot of companies in Brazil a lot of environments from small to large and I have never seen anyone of them with ARP spoofing preventions implemented none of them and sometimes large environments with load balancers and to upload the applications they are encrypted from the client until the load balancer and from the load balancer to the back end usually take off the encryption and create confirmation going on and based on your experience have you seen in you talked about this have you seen what kind of solution have you seen if you have implemented in the environments that you have tested or worked with let's go to the ARP slide here if you guys look at the slides online we have images that didn't show up here because of this but there's some good stuff in there for the ARP so basically you're looking at ARP watch because the Cisco's have this dynamic ARP inspection and DHCP snooping that allow you to look for road DHCP servers and different ARP problems going on there but ARP watch is a Linux utility you could demonize run a server on your network and actually look for malicious ARP activity on that network it's about the only thing we really looked at but it does work pretty well but I'm not pretty sure if I was clear what of these kind of of mitigations have you seen implemented in actually implemented in the components and that's where we'd really like to do a lot more we really haven't done much work in production environments we are just looking at vanilla out of the box stuff according to best practices and then looking at what the mitigations could be we don't have a lot of visibility into how often people are doing these mitigations in production environments for us it's interesting to hear you say that you've seen lots of environments and nobody's doing that so we'd love your email and we'd love to follow up that would be really helpful for us I see we have a line of more questions and we got the kind of yank sign from the goons we'd be very happy to talk more with folks I'm not sure where the right place to go and meet people is I guess if you come up here you can go where we go and we can follow up thank you so much thanks everybody