 Hi everybody, I'm Gina Matthews and this is Ronnie Bull. I'm a computer science professor from Clarkson University in way, way upstate New York, almost Canada, knows Believe New York as we like to call it. And Ronnie is both a professor type at Utica College and also finishing up his PhD at Clarkson. And we're going to talk about Layer 2 network security in virtualized environments. So first a little roadmap of what we're going to do today. First we're going to do a little bit of context, a little background to set the stage. We're going to talk about some very basics of virtualization, multi-tenant environments and cloud services. And we're going to talk a little bit about networking basics, both physical and virtual. Then we're going to talk about the platforms that we use to do our testing. The hypervisor platforms, the virtual networking device is that we tested and things like that. And then we're going to get into the good stuff, the actual attacks and the results that we obtained. In particular we're going to look at Mac flooding attacks and DHCP attacks in virtualized environments. We're going to talk about some mitigations for these attacks and some next steps and conclusions. All right. So this might be starting a little basic for some folks but just to make sure that we are all on the same page. Virtualization 101, right? We used to have a bunch of physical servers and now we've converted these all to virtual machines and they are living happily together on one physical box. They're co-located together on one physical virtualization server. And that's one step in the plot thickens. The next step in the plot thickens is that they are likely sharing a virtual networking device, a physical networking device, out of this one virtualization server. And the third step in the plot thickens is the virtual machines that are co-located together on the same physical box may not belong to the same trust domain. So this is the classic example. If you look carefully you can see some virtual machines that belong to competitors. Some belong to Coke and some belong to Pepsi. And what the question is, what can they do to one another since they happen to land co-located together in the same physical machine? If one of these VMs is malicious, what type of attacks could they do to the other VMs on the system? And just to emphasize the importance of what we're talking about, multi-tenant cloud services like this model are all over the place. You know, Amazon EC2, Microsoft Azure, Google cloud services, lots of, you know, less high-profile places do the same thing. And there's a lot of mission critical applications being run in this way. Most of these environments are running some form of Zen, whether it's open source Zen or Zen server. Some of them are using VMware or Hyper-V. And they're pretty much all sharing network connectivity between the tenants. I would like to emphasize that we did not do the testing of our attacks in EC2 or Azure or any of these places, but we'd love to. So if anyone in the audience is from Amazon or Microsoft or Google or any, you know, hosting environment and you'd like us to do some testing in your environment, no actual VMs would be hurt. We would love to come and do that. So come and talk to us afterwards, right? Okay. All right. So the key question before us is since all of these client virtual machines are essentially connected to a virtual version of a physical networking device, they're sharing it. Do the Layer 2 attacks, network attacks that we all have known and loved for so long still work when they're applied to their virtualized versions? And yeah, cut to the chase. Yes, most of them do. But we'll get back to that in a minute. I did want to emphasize that this is not the only class of attacks that co-located VMs can do to one another. This is by no means the only source of nastiness. Our research group at Clarkson has done other things where, you know, we hammer the disk from one VM and we see does it tank the performance of other VMs on the system or hammer the network with just a lot of traffic, not the kind of network attacks that we're talking about today, just allocate a huge amount of memory, you know, spin crazy on the CPU, all sorts of different things. And other research groups have demonstrated, you know, amazing things like stealing people's encryption keys through cache effects and other side channel attacks. So this is by no means the only class of nastiness that virtual machines can do to one another when they happen to land next to one another in the cloud. And I would say that it's an old lesson that you are vulnerable to those that are close to you. That's why we're all a little extra nervous being here at DEF CON, right? We're normally all spread out. We're all together. Okay. So the bottom line is that our our experimental results so that virtualized network devices do have the potential to be exploited in the same manner as physical devices. And that's going to be the heart of the demos that we're going to do today and show you the specifics of those attacks. And in fact, some in some environments, those attacks can even spill out of the virtualized environment on to the physical networks they're connected to. There's some good stories about that. Matt flooding and Citrix Zen server basically allowed eavesdropping on the physical network traffic. It basically flowed out, flooded the CAM tables on all the physical switches around also. Yeah. Okay. So in that what if slide, I had a malicious VM and you know, say what can that malicious VM do? In particular today we're going to talk about, we're not going to demonstrate all of these, but here's the classes of things you can do. Capture all the network traffic from the victim VMs. That's bad, right? Redirect the victim VM's traffic someplace they didn't intend. That's bad. Perform a whole variety of man in the middle attacks. Do denial of service. Gain unauthorized access to restricted sub networks and generally tank their performance. So those are all examples of nastiness. Okay. One more bit of just level setting review before we get into the details of the attacks. Quick review of some networking basics. So what are physical bridges? There are devices that connect two or more segments at layer two. They do separate collision domains. So they're maintaining MAC address forwarding tables. And they're forwarding the requests based on the destination MAC address. Here we see, you know, the layer two Ethernet frame. Basically the decision is being made based on that destination address. And here's the classic bridge picture. It's very simple. If a frame is sent in one domain and it's destined for something in that domain, it stays put and doesn't cross the bridge. If it's destined for another domain, it will be forwarded according to the information in that table. So virtual bridges are very similar. You can kind of think about them as the simplest form of virtual networking. In particular, we were using the virtual bridging support that's built into the Linux kernel and bridge utils user space package. And we actually found that in some cases, it did better than some of the enterprise switch solutions. So that was an interesting surprise. Then we also have switches. So physical switches operate also at layer two or higher. You can think about them as multi-port bridges. They also separate collision domains. And they also have a dynamic table, a CAM table, a content addressable memory table. It's very similar to that bridge forwarding table. And it's also using the destination MAC address to figure out where to forward each frame, what port to forward each frame onto. And there's just, again, a kind of classic switching picture showing that the switch will keep track of which MAC address lives off of every port so it can successfully send only the frames destined for that MAC address out that port when all is well. And virtual switches are much like their physical counterparts. They are a more advanced form of virtual networking than bridging. They can emulate layer two and hire physical devices. And they can provide additional services such as quality of service, VLAN traffic separation, and performance and traffic monitoring as some examples. Okay, so with that kind of context set, I'm ready to hand over to Ronnie, who's going to tell you all about the cool attacks. Okay, we're going to discuss some of the attacks that we experimented with in these virtualized environments, specifically the MAC flooding attack and the DHCP attack. And there's a few scenarios for each attack that we're going to go over. So this is the test environment that we use. It's a big rack of servers that were just dedicated for this work. And here's some of the hardware specs for this particular experimentation. We had three servers running open source Zen on Gen 2 Linux. One of them was running Linux bridging. The other one was running open V switch 1.1.0. Then we had open V switch 2.0.0, a Citrix Zen server 6.2, Hyper-V. And then we had Hyper-V free. So we had the free version and the actual full MS 2008 server, Hyper-V, and then VMware vSphere. So you can see the different specs here. They were fairly similar, but in reality with the testing we're doing, the hardware specs have no meaning to this because we're actually attacking the virtual switch. So let's talk about the MAC flooding attack. I see we have people up here. Okay. So MAC flooding basically is we're just taking a bunch of frames, random frames, we're flooding the switch with it to try to fill up that CAM table. When we fill up that CAM table, the switch goes into a hub mode, which basically forwards all the frames on the network to every device that's connected, breaking all those collision domains. No, no, no. Keep going. Okay. Good stuff. So here's an image of a MAC flooding attack occurring. You can just see a bunch of random MAC addresses going on here. Just flooding that network out. That's my computer, is it? All right. You all know how this works. Are they doing a good job? Our brand new speakers. Thanks, guys. Good job. Go back to that TCP flood screen. Whoa. Get off it. Get off it. Away. Okay. So for our scenario, this is the basic network diagram we have. We have a target virtual machine running Kali Linux and then the attacker is also running Kali Linux. It was just pretty easy to set that up. The virtual switch all connected to the physical interface. It's connected up to the physical network, you know, switching infrastructure, then a gateway router going out to the internet. So let's show some demos. It depends on the virtualized environment. So if we go back up here to this slide, oh, I'm sorry, it wasn't there. Here we will, the MAC flooding summary, I guess I'll show the demos after this. Open source download Linux bridging. We also had the OpenV switch. We had Citrix Zen server which was also running OpenV switch. An older version, 1.4.6. Microsoft Hyperview, the Hyper-V virtual switch and then the free version also with the Hyper-V virtual switch and then VM or vSphere. So that kind of spilled the beans there. But let's take a look at some of these demos. Hopefully they'll work. It's a reduced resolution and I had to test them out here, but they may not, you may not be able to see the text too well, but they are on the CD. So if you have the Def Con CD, you can view them. So here basically we're seeing the Linux bridge mode. We have a virtual machine running Gen2 Linux, running Zen and just kernel bridging. You can see in this first window here we're just showing the network interface, showing the configuration and bridge mode, showing that the actual interface providing for the virtual machines does not have an IP address, it's just bridged. Then we have two virtual machines. The one in the middle is the target and the one all the way over is the attacker system. So on the attacking system we're just going to fire up Wireshark and we can see we're just starting to see ARP traffic, general broadcast traffic going across that. What we'll do is we'll apply an HTTP filter and start running the Mac flooding attack just using Mac off, which has been around since 1999. So you can see the Mac flooding attack running over there on the side and then we can see Wireshark and in Wireshark we can see a bunch of mail form packets flooding the network. This is just an indication that something's going on here. Now if we apply the HTTP filter we just want to view to see if we can see the target systems web traffic, see if we can get some plain text information out of that. So we really shouldn't be able to do that in a switched environment, but this is a bridged environment. So it's loading up the filter here, filtering all the past packets and then over on the target system we're just going to start surfing the web. And we can see that really nothing interesting is happening in the Wireshark window. So basically what's going on here is Linux bridging is preventing the Mac flooding attack from actually being successful in this case. So let's take a look at what happens when we actually do this on OpenVSwitch 2.0.0. So again in that first window that's the hypervisor, the host, it's just Gen2 Linux and again I'm just showing off the basic networking configuration for this system. It has a management interface set up with a dedicated IP and then it has a second interface set up for the use for the virtual switch for OpenVSwitch. And then we can see here the configuration for OpenVSwitch. There's a couple VMs that are attached to it and running. So again in the middle we have the target and over on the side we have the attacking system. Highlighting the IP addresses there. We're going to open up the web browser on the target system and again we'll start running Wireshark on the attacking system. So we're very systematic in our approach here. We use the same VMs all the time. Same process. We're seeing the ARP traffic. We apply the HTTP filter and then we're going to start running the Mac flooding attack. So flooding the network. And then we'll start surfing the web over on the target system and pretty much immediately we start seeing the HTTP packets coming up in Wireshark. So at this point we are acquiring all the network traffic from that target system. The stuff that should be prevented through the collision domain protection. So in this video we go a little more deeper and we start looking at some of the packets here after we've gathered enough of them. So let's just gather a few more here and then it's a little hard to see up there but if you watch the video later you can see where we actually started inspecting some of these packets and we're able to pull out some of the clear text, all the header information. But I go into showing where you can actually look at the text on one of the web pages and view it in Wireshark. So it is actually passing the clear text over. It's just showing that demo here. Okay, that third video is going to, for the sake of time, I'm going to kind of skip that third one but it's just showing it on OpenVSwitch on Citrix Zen server and it does work. So it's just using an older version of OpenVSwitch. So feel free to look at that on YouTube or check it out on the CD as well. Okay, so I've already showed the slide but basic summary is that it worked in any OpenVSwitch environment and the performance was impacted on every single environment. We can't talk about VMware because in the VMware end user and license agreement it specifically states they are not allowed to publish or talk about in public venue. Performance benchmarking results without prior consent from VMware. But when you look at what happens to basic networking over that connection when the MAC flooding attack is occurring, you can see something like this. So this was performed, this test was gathered on a bridge interface. The first 60 ICMP requests, there was no MAC flooding attack running so we got a baseline there and you can see it's really low and what I was doing was just pinging a server one hop away from the virtualized switch. So we were just going out on the land to a server out there and just pinging it to see what we were getting. So you can see it's a really low latency there for the first 60 ICMP packets and as soon as we start pinging up and then all of a sudden that flood has just really happened and we got a lot of variation in latency there, pinging out in almost 130, 140 milliseconds. And then as soon as we cut off that MAC flooding attack we can see it drop right down to nothing, back to the baseline again. So you can really see that the performance does get impacted here on the network when running this attack. Okay, so we did our due diligence and we reported this vulnerability. We notified cert.org, we got assigned the VU number there. Haven't received from cert.org for a CVE yet. So we were hoping we would hear something from them. We did report it to OpenVswitch and they immediately responded with a patch. So any version of OpenVswitch from 2.0.0 and higher has been patched to alleviate this or mitigate this attack. So feel free to test it out and if you want to go to that link the slides are available on the CD as well. You can actually read the whole details of that patch and everything that it entails and actually see the source code as well. Okay, so traditionally these attacks on physical networks are mitigated by using port security to limit the actual MAC addresses that can accumulate on a port. So they actually don't hit that threshold where the CAM table gets overflowed and we start flooding that network. We can also allow authorized MAC addresses. We just specifically say you're the only system that connects to this port and that will help mitigate this attack too so that we can't make that CAM table limit hit. Or we can just disable unused ports but this stuff doesn't really work on virtual switches. Port security is available on the Nexus 1000V the Cisco but it's the paid version and you need to run that either in a VMware environment or a Hyper-V environment. So it's very costly to get into that level of protection. So these other virtual switches hopefully they can be brought up to speed even especially open source versions to actually offer this kind of security because we are getting into the virtualized world here and this is very important. Okay, let's discuss DHCP attacks. So the basic of the DHCP protocol we can use this to allocate network information IP address information to a bunch of machines on the network. It makes things a lot simpler for administrators. Things like subnet mass, default gateway. All the essentials you need to actually be on a network can be allocated through DHCP. The basic client server protocol the client broadcasts out for an address the server responds with a lease information or an offer and then the client broadcasts by saying yeah sure I'll take that and then we have this lease established we can actually have an IP address and all the information we need from that server. DHCP allows for options so if you take a look at RFC 2132 you're going to find there's about 255 different options available to pass through DHCP something like time server domain names, ARP cache all sorts of good stuff. So really to perform a DHCP attack we got to put a rogue DHCP server that we're in control of on the network and it's going to compete with a legitimate DHCP server to actually provide address information network information to clients on that network and this really has a 50-50 chance of actually being successful and if you actually put multiple rogue DHCP servers on a network you're going to be able to increase that probability and this is very simple we were using DNS mask you can set this up in about five minutes or less to actually do this stuff on a basic center SVM. So we have a few conditions here and a few scenarios we ran range that the legitimate DHCP server is passing out and you try to mimic that on your rogue server and try to do some duplicate addressing attacks we're just going to cause denial of service so basically a denial of service conflict in that and here's just a graphic illustrating we have a legitimate DHCP server and a malicious one and the first two clients get their addresses from that legitimate server but we can see that third client got it from the malicious server and it's in conflict with the first client so we have a problem okay a rogue DNS server if we have a rogue DNS server if we're running a DHCP server we can provide DNS information primary DNS server information for those systems on that network so if we're in control of the DHCP server we're running something like DNS mask well that also doubles as a DNS server so now we can use that to actually provide false or poison DNS information to the clients on our network and really the effect of that is we're directing that traffic to where we want it to go so we can actually start stealing information or harvesting information if we're directing to web servers we're in control of so basically this is a rogue DNS server attack where we have on the top a legitimate server client ask for Gmail it gets back the IP address for gmail.com legitimate address but if we have a malicious server on there we can direct it to wherever we want it to go okay the incorrect default gateway was another attack we tried so we're providing just incorrect information and basically just calling causing a denial of service attack for those systems on the network that is trying to get outside the network to different subnets or different resources outside of the LAN a malicious honey net was another attack we used so instead of giving just a bad default gateway address we actually ran up a router system on another VM pointed the clients to that router system and then routed the traffic through that and kind of mirrored the production network within the malicious honey net by doing that so this is just an image that's showing that we have client A going through a regular default gateway and this is happily getting to where it needs to be but client B associated with a rogue DHCP server got a bad default gateway and its traffic was directed to that honey net for collection and finally we did remote execution of code so when we started this research on DHCP attacks it was right around the time when shell shock was big so we were trying to figure out what we could do with that so by making use of the DHCP options we found that we could use shell shock to actually run remote execution of code on these systems and what happens is we have a lease time on DHCP so every time that the lease refreshes itself that code is ran so we could kind of use this as kind of a mock up cron job on the systems so the effect really you know this could be catastrophic this could be harmless we could use it to set an MOTD we could use it to scp our shadow file out to some email address or server and we can recursively delete the system as it's booting up right after it gathered this IP address information so the basic protocol is the same you know we're doing that client server exchange for our IP address information but at the end here we're passing that option 100 or option 114 and we're throwing out the shell shock attack and in this case we're passing the RM-RF4 slash with that shell shock attack to delete the system so the test environment for this is the MAC flooding attack where we're just reproducing the stuff on the same environments that we had built there but we did need to expand the virtual machines beyond the Kali system so we fired up four CentOSVMs one was configured is that DHCP and DNS server then we had a simple router system to do that malicious honey net attack and then we set up a basic web server to use for doing our poison DNS attacks and we'll show you another demo here in a little bit of another way we use that and then our final system was just a client system left vulnerable to shell shock so we could actually do a proof of concept with that so the actual scenarios we did for the remote execution of code you can see that we passed that option 100 and that's the exact code that we ran with that so we were just doing a proof of concept fairly harmless attack on here and we just echoed out testing shell shock vulnerability if you can read this it worked to a file in slash temp called shell shock we also tested to see what kind of access we had so we passed the ID command instead of just been echo we just did been ID and that passed in and actually when the client received its IP address information from the DHCP server it showed that it was running as root so with the poison DNS server we just basically gave it the poison DNS server information and then passed it to spoof gmail mail.google.com and www.gmail.com to steal the user credentials and then in invaled default gateway attack we passed 1.1.1.1 so it just basically denial of service the systems and then the malicious default gateway we gave it that 1.20 address which was our system our router system set up to pass it out to another honey net so how do we monitor for this how do we actually see that we were really having a 50-50 probability between our legitimate and our rogue servers we just used TCP dump so here's a simple further information for the DHCP handshake if you will and it produced an output like this where we just saw the broadcast traffic coming between the clients and the servers and we can actually if you look into this you can actually see that it was about a 50-50% chance that you would hit that rogue DHCP server so what I was just doing is running DH client service I'd run it once I'd watch TCP dump output I'd kill it run it again and it pretty much bounced back and forth so 1.92, 1.68, 1.2 was a legitimate server and 1.3 was the rogue server and you can see here it was just going back and forth back and forth so in order to do the ID test just to verify we were actually running as root once we ran the shell shock command we just did the user bin ID and it produced this output on the client when we ran DH client and just showed that we were running as the UID of root the GID of root so we were in the root user root group and to summarize these attacks it worked on everything every attack, every scenario we did worked across the board there was no protection against these attacks default out of the box for any of the hypervisor environments that were experimented with so we have a few demos here for this how are we doing that time we're going to skip the poison DNS server one just for time since it is fairly basic we're just running that DHCP server I want to show you these shell shock ones and hopefully they come out good ok so here we have a shell shock system which is just that client system that's vulnerable to shell shock and on the other end we have the rogue DHCP server so if we look at resolve.conf we can see that the system originally grabbed its IP address from the legitimate DHCP server and temp just contains yum.log so it's just a standard installation there's nothing else in there we're going to restart the DHCP service or the DH client service here to pull a new address and try to grab one from that rogue DHCP server and just wait a few for the process to occur while it's waiting take a look at the dnsmass.conf file we can see the DHCP range was just specified by a one liner here and we can see we did 200 to 254 and we did a least time of one minute to cycle through the leasing just to see what we could do quickly so we can verify that it was actually refreshing we set the default gateway to 192.168.1.1 and if we go down a little further in that config file we can see the shell shock attack here that we passed out that echo command testing shell shock and trying to write it into temp so over here the client pulled the dns information from the rogue DHCP server so we know it actually got an address from that rogue DHCP server and the command should have ran on that client system so if now we check the contents of temp an interesting thing is every time the system either pulled from a legitimate DHCP server or the rogue one it actually pulled the same IP address because it just requested it each time so there we can see the shell shock file was written to that client through DHCP and if we cat out the file we can see that it actually contains the information we wanted to place in that file and we can see it was written as the root user as well and over here you can see on the side in the dnsmass.com file I'm just comparing showing that what we actually pushed through actually went through now if we just wait about a minute or so so you're going to see on the video I'm just going to keep trying the date command just to see what's going on here but we're going to see it actually got written again it overwrites the file it just takes a couple seconds here because what happens is when you run it actually doesn't happen on the exact minute so if you run the DHCP client command it goes through the whole process of getting the lease and everything and sometimes that goes beyond the minute and then it actually hits it so I was sitting here kind of being impatient running the command and it does end up going across the next minute to push the shell shock there so we go back up we look at the lease time as one minute here and if we just do the LS over on the other side again we can see that it indeed wrote it 1436 and the next time stamp was 1437 so I was actually able to write that every minute if I wanted to okay so that was just our basic proof of concept attack just to see if it worked so then we asked well what can we really do with this I mean that's great we can write the temp we can put a file in there what do you do but how can we actually use it to leverage it and actually gain full root access to that client system so what we did is we figured maybe we'll try to play with some SSH keys so the same scenario same setup we have our client system and we have our rogue DHCP server over here and we're just going to try to SSH over the system first and we see it's prompting us for a password so there's no key information in use right now on that other system so next what we're going to do is we're just going to look at that the host file on the system and we see it is using DNS mask which uses the host file to actually provide DNS information on the system so we can see that we have a web server running on 1M21681.10 that's providing the name of web as well as the Gmail stuff that we were spoofing before and if I SSH over to that system and we look at the VARWW HTML directory we can see the contents and then there's an index.html and a file called badkey that I put in there so this is my web server I control out there in the world and it's my server I'm going to try to use to host that key that I'm going to push over and there if we cat it out we can see the public key that was generated using the keygen SSH keygen command so we'll leave the web server and go back over here and look at the DNSMask.com file there was blank on the tail there but let's see so at the end of here we can see now I'm using a little bit more intricate command I'm actually using curl to pull down that badkey file from the web server and then I'm redirecting that to root.ssh authorize keys on the client system so when we run DHCP this is the actual command it's going to be using so it's going to go out there it's going to try to download that key file and place it directly in the root user's authorize keys file so we'll go over on the client we'll check out the authorize keys file and see if it's there there's no authorize keys file currently on that client system so it was never placed in there and then what we'll do is we'll try to obtain an address from that system but first over on here we're looking at the root directory on the rogue DHCP server and we can see there's a public and private key in the .ssh folder and if we cat out that file the public key will see that it's the same public key that is on the web server that's being pushed down to the client through DHCP so going back over on the client we'll just re-run the DHCP service to try to pull a new address from the rogue server and after a few little bit of time here we should get our information and we hopefully we'll see that the authorize keys file is placed in the .ssh folder and there it is written by root look great we wrote the file but doesn't work right so if we go back over let's cat it out we can see here that we got double entries in what happened is I believe when I ran it at the client I ran it at the tail end of the minute it wrote it once and then because I was redirecting and doing a pen instead of an overwrite it just added another entry of that key so good housekeeping when you get in the system clear up that authorize keys file should be good to go so again over here we see we're using that curl command redirecting and that's why we got that double entry and then we should just try to test the ssh into it the same command that we used the first time so instead of just typing it all again going into 182.168.1.226 which is the shell shock system and it went right in so it didn't prompt for the password this time so our authorize keys file was pushed over there and it was effective for getting us root access to that system so if we ID we can see that we are the root user on that system exit out try it again and of course the last log should say the rogue DHCP server so it was effective so how can we mitigate these attacks enforcing static addressing so a DHCP server is right out of the question but it does get cumbersome if you have a really large environment depending on DHCP for address allocation this could cause some issues we could also use DHCP snooping on switches this is usually effective in the physical world but again it's not really there yet in the virtualized switched environment there is an option on some physical switches like Cisco and HP we could also restrict access for specific MAC addresses to those systems but this is highly restrictive or DHCP server authorization in Windows environment so Windows 2000 server enough if you have an active directory environment and you're running a whole Windows DHCP servers you can actually do DHCP authorization but really who wants to run a whole Windows environment so we could also explore techniques so ideas maybe defining filters to identify these DHCP requests and transfers say hey this is the real legitimate DHCP server if I see any other addresses broadcasting on this network we're just going to ignore them we're going to route those clients to the correct place so most virtual switches these days especially open view switch allow you to actually integrate with like open flow so this is a good way to move forward in this to actually try to use software defined networking to prevent some of this from occurring one thing we did find especially in the send OS environments that when I started running the shell shock test I was able to write the temp find because it was Chimod 777 it was world writable it worked great but when I started doing that SSH key attack it didn't work and I was wondering well I have root access why isn't it working down to SE Linux so a lot of system administrators disable SE Linux right out of the box it's enabled in sent OS and red hat distribution so what this was doing is it was preventing it from occurring it was seeing this network service trying to write to a non-world writable folder or file and it just said no you can't do it so it never worked until I actually disabled SE Linux and I could write my files anywhere I wanted to so take this into consideration SE Linux will help prevent from writing like this authorized keys attack because you can't write to anywhere except for temp but it does not work for any of the other DHCP attacks that we were trying out or any of the MAC flooding attacks for that matter so looking ahead we intend to move forward into looking at V-landing V-land hopping attacks specifically to see if we can actually start hopping different V-lands in the virtualized environments so we're trying to separate domains logically separate and isolate these systems through V-landing on the network and most virtual switches offer this capability so by using things like double tagging or switch booping can we actually break these V-land separations on these networks so we'll have to see maybe we'll come back next year and present our research and results on this okay so Gina all right so was that scary a little bit okay so to wrap this up all the layer 2 vulnerabilities we discussed were targeted towards the virtual networking devices not the hypervisors in particular so that's a good thing to keep in mind every virtual switch product is available on every hypervisor and as you saw the results show that the virtual networking devices can be just as vulnerable as their physical counterparts to these classic attacks MAC flooding and rogue DHCP and all these kind of things we're going to continue to work on this as Ronnie said it's good to know that Zen server and any other solutions using open V-switch are vulnerable to these dropping out of the box right now although there is that patch so you should get that and all the environments that we tested were vulnerable to every type of the DHCP attack manipulations that we did so that's a really good one to know as well so a single malicious virtual machine running in a multi-tenant environment has the potential to sniff all the traffic that's going over that virtual switch and you saw some of the devastating effects it can have it's obviously a significant threat to confidentiality integrity and availability of data that's passing over these networks in virtualized multi-tenant environments the cloud service environments that are being used for tons of production stuff right now and all these results definitely indicate that a full assessment of Layer 2 network security in multi-tenant environments is warranted and important and we hope that some of the takeaway actions people will have is to really question their hosting providers about what they're running are there any mitigation steps that they're taking ask for that, push for that encourage auditing of the workload and we'd be happy to help with that I think we mentioned auditing the risk of the workloads that you run in the cloud thinking about these issues and you have no idea who you just landed with in these multi-tenant environments and consider some extra security measures some for the users and some for the service providers so increased use of encryption would obviously be good and additional service monitoring and additional steps to detect these threats to notice this kind of traffic happening and alerting so we'd be happy to take some questions that's the next thing we really want to work on we did notice that when the MAC flooding was running it did break the VLANs so it broke the VLAN separation right on the bat when it turned into the hub mode so in Wireshark you were able to actually see all the VLAN tags on all the frames on whatever system you were sniffing we didn't specifically look for that at the time but we did see that we did see traffic on other VLANs especially when we ran on Citrix Zen server and it flooded the upstream switches and then we could start seeing all the traffic on all the VLANs from those upstream switches as well so it really got pretty bad we flooded close to 100 switches and put them all in hub mode okay thanks if you have questions come down here to the front and if you are leaving please leave through the back doors not the side doors please it'll just help with the congestion out in the hallways thank you on the VMware test you guys indicated they weren't vulnerable to the MAC flooding did you test any of the workloads with allowed forge transmits set or just their default default ESXI virtual switch and we just did the same exact test on every platform so it was just what you saw in the demo with the MAC flooding attack so nothing beyond that at the point people have suggestions of specific configurations that would be interesting to test on those products that would be another thing we'd be happy to hear on the linux site which virtualized next did you use sorry on the linux site which virtualized next did you use like e1000 versus vertio and also did you try testing kvm versus zen we didn't use kvm versus zen because we were really looking at open v-switch so the switching mechanism was the important part here and if you're using kvm or zen open v-switch anyway so the hypervisor doesn't really matter it's the virtual switch implementation we were using open source zen and we were just using straight excel command so no over the top kind of higher layer interaction for management it was just straight command line excel okay did you try any different virtualized nick drivers like vertio versus e1000 like you were saying no we didn't thank you though so my question is do you have a flat layer 2 network only have you guys tested this across a routed switch network where you'd be implementing something like ip helper addresses for clients to a specific address rather than relying on picking up dhp requests across a flat network no we were just looking at the specific LAN itself and then we were moving forward to the vlan attacks next so I mean if you guys had a multi-tenant environment you would typically never be picking up off of a local dhp server you'd be routed back to your corporate network to your own personal dhp address or dhp server so unless you had a dhp server that was spoofing the dhp server address this should not be happening across a multi-tier network right if you send us an email with us some specific things you think yes we would really love that and if you have an environment like I said we'd love to test more environments in my network we use dhp guard on the physical ports do such technologies exist in the virtual switches I have not seen anything on the virtual switches yet maybe when you get to the higher the paid Cisco versions there might be some implementation of that but I haven't seen it yeah the nexus not the free version though any information about ec2 infrastructure as a service any information that you've gotten either from documentation also I'm from Potsdam wait a minute go ahead you should come visit us at the labs the only thing we really know about ec2 is that they use Zen and they use openv switch but then they have their higher level controls on top of that as well that they've implemented other than that it's kind of safely guarded really trying to get into there but it's been really difficult yeah I've been through some information sessions with AWS and they definitely make you sign not disclosure thank you though so it sounds like you did this like you said all on one flat layer 2 network so in a multi-tenant environment they also may deploy just as a bunch of layer 2 networks I'd be surprised if they put the same tenants in the same layer 2 although it's possible I just wonder if you thought about that and the other thing a lot of these attacks are possible in a physical network in a typical physical network but I totally agree with you that these attacks could be prevented in a virtual network they have a lot more ability to look for that kind of thing the stakes are higher because in the physical world you can control you can't as much there should be more control actually did you test against ARP spoofing attacks can't hear you too well did you test these things against ARP spoofing attacks and if you did also if virtual switches can have an ARP proxy I'm still not quite understanding can you come up right after yeah we'll talk after yeah I was wondering if y'all were using the standard VMware switches or the distributed switches it was standard we were starting to set up the environment for the Nexus to do the distributed switching but we haven't gotten to that point yet we're requiring licensing and funding to do that but we're getting there yeah but we would need to set up multiple VMware environments and then do the distributing there but we're getting there okay thanks everybody