 Hello my name is Tim Pearson and I'll be one of your instructors for this course. I've been a technical trainer and a consultant for the past 25 years in both security and in virtualization. The contributing author to the well-received VMware vSphere and virtual infrastructure security securing ESX in the virtual environment. I've been the noted speaker at many industry events including Infosec World 2010, hosting the Virtualization Security Summit, as well as venues like Inatec, GISSA. In addition, I was a named instructor of the year from EC Council for a large pool of their instructors in 2009. In addition, I've also been involved in many military venues including venues at the Pentagon and numerous nuclear facilities addressing both security in the U.S. hair and in Europe. I've also achieved 29 industry technical certification from vendors such as Cisco, Microsoft, Nobel and even EC Council. Lastly, I'm a bi-weekly panellist of the podcast Virtualization Security Roundtable which can be found on TalkShoe.com and iTunes. Hello my name is Dwayne Anderson and I am very pleased to be part of the VMware Advanced Security course. Now I have been able to be in security for quite some time. Back in 2004 I got started in the security industry. I started out actually doing some penetration testing and some forensics work. I've been able to go out and I've been able to test the security infrastructures of many companies and governments across the globe. It's been a lot of fun actually. One of the things that is impressive is that I actually get to break things and I don't have to worry about fixing them. That's a lot of fun. You guys ever done that? You broke it and now you got to fix it? I don't have to fix it. That's a lot of fun. On the forensics side, I've actually even been an expert witness in the courtroom. I wouldn't say that that is a lot of fun. I would rather have not done that, but sometimes you have to do what you have to do. I'm hoping that some of my real-world experience will come to play here as we go through some of this. I'll be sharing some of the chapters here, working through some of this for you guys so that you can not only learn how to implement a lot of security measures with regard to VMware, but you can also do a good job of testing that environment. We want you to be able to see if you can break your own environment so you can make sure that you are secure. So pay attention, enjoy, and I hope to make this a really good learning opportunity for you. In this chapter, we're going to be taking a look at the virtual networking components for the ESX administrator. We'll take a look at the ESX networking components, how virtual Ethernet adapters actually work, how virtual switches work. We're going to compare and contrast a V-switch, which is a virtual switch, which is similar to a physical switch, and how it is actually different from a physical switch. We'll take a look at the spanning tree protocol, and if you're actually running that in your environment, in the virtual environment, you may be actually slowing things down. We'll take a look at the V-switch isolation, virtual ports, uplink ports, network port groups. We'll also take a look at virtual switch correctness, VLANs in our VMware infrastructure. We'll take a quick glance at NIC teaming, how load balancing works, failover configurations. We'll get a 1000 foot view of security features, and then we're going to roll up our sleeves and dive in deep. We'll take a look at managing the virtual network, encryption and certificates, and then finally round it off with Linux file system. We'll discuss the kernel, processes, accounts and groups, and then finally permissions. Now, first off, we need to start off taking a look at the networking components, because the hacker is going to get into your network. Guys, this is where he's going to get in. There are certain key virtual networking components that we want to take a look at. Naturally, we're going to be focusing on things like our virtual NIC, our virtual switch. In addition, VMware, with their vSphere 4 product, has introduced something called distributed virtual switch. Okay, well that's nice, Tim, but it'd be nice to know what it does to understand the problem that it solves. Well, if you have multiple hosts, they all must have the exact named switch in order for things to, well, let's put it this way, flow correctly, the motion to work correctly, things of that nature. With a distributed virtual switch, you don't need to worry about that. It's handled internally in the database and on the system. So that was really, really nice feature that they came out with in vSphere 4. We also have something called a virtual port group. Now, there's actually not really a mapping in the physical world to something like a port group. You could think of this almost as a template. You could think of it as an RJ45 template if that helps, because you can take a certain number of ports, isolate them and say these ports that I've plugged this virtual machine into, this card into, or what have you are going to be used for this particular function. So it really makes it nice for you to group this. It's almost sort of kind of like a VLAN where you do kind of the same thing. Now it's important to understand a little bit of nomenclature before we get started with the course. It's important to understand that this symbol right here denotes a virtual device. You can see this right here is a virtual network card. This is a virtual switch. This is a physical, hence no icon, a physical adapter, all right? Virtual switch, virtual switch, and so on. Now it also needs to be understood that the way we've had this drawn right here, this entire thing is one ESX host. This is a VM, this is a VM, this is a VM, and this is a VM. Now when I typically teach a live class, I'll ask the class, now how many VMs do we have in here? And naturally I'll get an answer somewhere in the neighborhood of, oh we've got four, Tim. Well, yeah, but we have four, we actually have five, all right? The service console is actually considered a VM and not really considered it is a VM. This provides us a lot of flexibility and isolation as well. Now you notice that we have multiple network cards, physical cards coming to one virtual switch here, but you notice there is no uplink port on this one. A virtual switch can have an uplink port, an uplink port is actually defined as connecting to the outside world or it can actually not have one. In this case, the only way of getting access to VM3 is through VM2. So this may be a firewall, it may be a proxy, it may be some kind of filtering device. So I wanted to kind of give you a little bit of an idea of the nomenclature and our environment before we move any further. Now it's important to understand how virtual Ethernet adapters actually work. In reality, there are six different types of adapters. Some of them are used for virtual machines, some of them are used for other purposes. We'll take a quick peek at them and why you might want to choose one over the other. The one that probably has come out latest is this VMXnet, which is a para-virtualized device that only works if VMware tools is installed in the guest operating system. Now we're going to get in a little bit later in the course on some things you may have heard about para-virtualized devices and possibly things that you may have heard about VMware tools. Oh, I don't want to install VMware tools because people have gotten in or broken into the system because VMware tools is installed. Well, I'm going to try and separate the myth from the reality on that. VMXnet 3, its claim to fame is it's built on the same architecture as the VM next, but look at this guys, it provides 10 gigabit per second speed. Now only ESX40 has that and it does require an upgrade to the hardware. Oh my gosh, you're going to have to go back get an order new hardware. Well, you know what the hardware upgrade is? Punch a button. I mean, it's as simple as that. It takes about three, maybe four seconds, okay, maybe five seconds to do a hardware upgrade. The machine typically has to be off to do this upgrade, but it generally is very painless upgrade. We have a VLAN which emulates the AMD adapter and then this is oftentimes referred to as the lowest common denominator. The E1000 generally always works. The reason that VMware chose the E1000 as its adapter of choice is almost every single operating system already has a pre-installed driver for that E1000 card. Then let's move on to the other adapters and check those out real quick. The other adapters, the first one we want to talk about is our window into that service console, which in the graphic previously was our fifth VM. Now we call this the vSwift card. Now the vSwift card is the one that's only used by the ESX service console. In addition, we have our VMKNIC. It's actually a direct connection to the VMWare kernel. It's used for things like vMotion, high-speed connections, NFS, iSCSI clients, all of those types of things. Now it's important to understand that every single one of these adapters, regardless of whether they're a vSwift, a VMKNIC, or the virtual machine adapters, every one of them has their own unique MAC address. They are strictly a Layer 2 Ethernet adapter. For all practical purposes, the operating system in the virtual machine doesn't even know that it's running in a virtual machine. It thinks that it's communicating with a live physical adapter. Now on this slide we're going to take a look at how virtual switches work. Now virtual switches are actually the key networking components. You can create up to 128 virtual switches for each 3.5 host and up to 248 virtual switches for each ESX 4 host. Now the virtual switch is interesting because it's a built to order device. That simply means that as the machine is booted or the switch is created, it actually is created. So we don't have extra code that's running in the system not being used for anything. It does use a Layer 2 forwarding engine. Well the Layer 2 forwarding engine is kind of the key part of the system and actually what gives VMWare a lot of its speed. What it does is it only reads in enough of the header of the Ethernet to make a decision about what it's going to do and then doesn't read anything after that. I kid around in my classes and say, this is kind of like working for a union. Well that's not part of my job. I didn't bid on that particular contract. I don't do that contract. So consequently, this is exactly how I'm going to handle it. So if it helps to think about it like that, then that's fine. VLAN tagging is the notion of adding a particular identifier to an individual frame to know where to go for security reasons. Now we tag a VLAN. Now the switch is also responsible for stripping it off because the VLAN itself, the information, if it arrives at the operating system virtual machine, it's not really going to know what to do with it. It's addressing information you can think of it as. It's the label on the UPS box that you might get. And then naturally we have filtering units that basically keep it away from this area and into this area. We have, in addition to that, our Layer 2 security checksum and segmentation offload units. Now the part that I thought was really cool about VMWare is they made this all modular in nature. They did this so future developments would be very, very easy for them to add things on, kind of like a building blocks as we build a home. All of this dynamically built at runtime, as I mentioned before, utilizing only needed components. This paged the way for us to provide third-party built-in modules. A third-party module or built-in API called VMsafe, and we also mentioned that we have VShield Zoned, are also built into ESX4. Now let's take a real quick look at VMsafe. This is simply an overview of VMsafe, but in reality what VMsafe is, is simply a set of APIs. What is an API? An API is an application programming interface. It's our window into the programmer's world, so to speak. So for example, in VMware's case, when they want to provide some functionality to us or a developer, they'll develop something called an API. We would utilize that API, that application programming interface, in order to build some component to provide communication to and from their particular piece of software. So the API restricts or allows what we would like, or possibly what VMWare would only like us to do with their piece of software. In addition, it opens up and allows third parties to develop security appliances. Things that the API allows us to have access to are memory and CPU. It provides detailed monitoring of guest VM, memory pages, virtual CPU state, network packet filtering, things like monitoring in and out of the virtual switch. It monitors within a dedicated to security VM. It also allows us to do things like process execution handling. The API that enables complete monitoring and control of the process execution inside the guest OS. In addition to that, one of the really cool things about the VMsafe API is the storage aspects. The VMDK file, which we'll talk about just a little bit later, you can think of as your virtual disk. Now this allows us to take the virtual disk and actually mount it on another storage device. So we would be able to take the virtual disk off of one virtual machine and mount it to say on another virtual machine, almost like passing it around if we needed to grab information off of it or something of that nature. Let's take a real quick look at our current VMsafe partners. You can see they range all the way from ultra networks down to web root. Now there's pretty big names in here as well. As you can see, IBM's in here, RSA is in here, a number of pretty good sized corporations. It's important to understand that we may want to be part of this API or we may decide not to be part of this API. There are certain vendors that chose to become what's referred to as vendor agnostic, which simply means that we can have our product run on multiple vendors. Catbird is a good example of that. Catbird allows us to run on the VMware product, allows us to run on the IBM product, allows us to run on the Microsoft product and allows us to run on the Citrix product. This way it provides them some flexibility, but at the same time they lose some of the really fine-grained details that that API gives you. The virtual switch versus the physical switch. Now let's take a look at this real quick. The SX server virtual switches have a lot of characteristics in common with the physical switches. First off, they maintain a Mac port forwarding table. We look up each max frame, each frame's MAC address when it arrives, and then we make a decision about how we're going to forward that frame to one or more ports for transmission. Now we know that the physical switch, if it doesn't know how to make the decision, it just simply forwards it to anyone. Well in reality the virtual switch doesn't need to do that because it already knows, as I'm going to show you in a little mock-up I'm going to do here in a couple of seconds. It avoids unnecessary deliveries. In other words it's not a hub. Supports VLAN segmentation at the port level and it also can be configured to access a single VLAN as well. I wanted to discuss exactly how a CAM table works. We're going to compare and contrast how it works in the virtual world and then we're going to also do that by understanding how it works exactly in the physical world. Now we all know that every switch that I've crudely drawn over here on the right has a CAM table and the CAM table is its lookup mechanism. We have a port assigned to every one of the devices that are connected to this. The CAM table makes that association by its MAC address. Now I'm going to label my MAC addresses on here A, B, C, and D. I've got very short MAC addresses in this particular environment. By the way, how long are the MAC addresses on Ethernet adapter? Okay, I know I heard somebody out there say 48 bits. That's right 48 bits but in reality it's not 48 bits. It's 48 bits is assigned to the entire card, a portion of it is assigned to the card, another portion is assigned to the serial number on the card. This is what is referred to as the OUI and so I have a portion of my MAC address, the entire MAC address as an example, half of it is the OUI, the other half is the serial number. I typically get the question in my classes, well Tim, is every Ethernet card globally unique and the actual real answer contrary to popular belief is, no it's not. Every Ethernet card doesn't need to be globally unique. The OUI itself is 2 to the 24th power or about 16 million in change. With that said, if it was globally unique, in reality 3Com has only really made 16 million cards. You're probably saying, well 3Com probably has more than one OUI and in reality they do but that would totally mess up my demo so we're not going to mention that. Every one of the manufacturers has a serial number assigned in conjunction with that OUI so they may start with serial number one, run up to 16 million and start over again. If we were to take just simply a router, the packet comes in the router, once it comes in the router the Ethernet header is completely stripped off and thrown away. We readdress that with another Ethernet after we get out of the router. Oftentimes I get the question, well what's the MAC address on that web server, Jim? Okay now think about what you're asking me here. The MAC address changes on every hop through the internet so in reality it's pretty irrelevant what that MAC address is. So we don't need to be globally unique, we only need to be unique on the network segment that we're on. Now if you have more than 16 million cards on one network segment guys, you've got bigger problems anyway. So let's take a look at what might happen in our environment if we very first turn the computer on. Naturally our CAM table that has these associated MAC addresses are going to be completely empty. We're of course going to have our various port numbers but nothing's going to be populated inside of the MAC addresses. Let's assume here for just a second that I'm going to take a frame and I'm going to pass it into the switch. Now naturally the CAM table is empty. Let's assume Al Gore just turned on the internet. He just flipped the switch on the internet. Nobody knows about anything if that's correct English. No one knows about anything. So we've got to pretty much guess about where to send this and that's exactly what the switch does. It does a technique called flooding. Flooding is the notion that it doesn't change anything on the packet. It simply forwards it. I should say frame. It simply forwards it. It forwards it out to every port except the port it came in on. So in this case A would be learned automatically and would be populated in the CAM table. It would then forward it by flooding it to B, to C, and to D. Now B, C, and D, let's assume for just a moment that it's addressed to B. B would be the only one who really responds. C and D would just simply throw it away. Now when B responded, B is populated. Then eventually C is populated and D is populated. At this point in time everyone knows about everything else. In other words all the partners in this switch know about all of the other partners. The convergence switch or the convergence register I should say is set. Thereafter we no longer do this flooding technique. The flooding technique as you recall was sent to everyone. So consequently now and only now is really the only time a switch is secure where we have that one-to-one mapping. Now let's say for example we have an individual we've hired that brings in his iPad or let's say he brings in some device hooks up a hub or a switch in his cube and installs some other devices. Even though we've told him not to, we've now introduced an E and an F into this environment as well. Well naturally the E and F will be populated as part of this large CAM table. Well every switch has a finite number of entries that it can hold. Most switches like the one that we use at home have about 1024 entries. Alright 1024 entries again if you got more than 1024 computers at your house you've got bigger problems in most families but 1024 entries is more than enough. Although their application to a good one by example is something called Mac-Off from Backtrack that gives us the capability to generate thousands of MAC addresses every second thus filling up this CAM table and naturally un-setting this convergence register. Now what happens is we are back to the flooding guys all we need to do is hook up a switch right here excuse me hook up a sniffer right here and simply sniff all the traffic on the entire network or rather everything is attached to that particular switch. So this is how we circumvent a switch and get it to coerce an attacker coerces it to come to him to be able to read all of the traffic on that environment. Now here we're going to be talking about the spanning stream protocol and why it's not needed. We discussed earlier how everything is kind of tied together and we put things together with a CAM table. We also discussed how having another switch introduced into that environment causes problems because things are added to that CAM table we kind of got to move things together. Well if we have more than one switch we need to have one person who decides okay this person goes here and this person goes here. Well in reality in VMware we solve that problem because your switch is a virtual switch. If we go down to one of our favorite electronic stores and purchase something we purchase a switch that's maybe eight ports 24 ports 48 port whatever we're going to purchase if we need to have like say 96 ports and we already have a 48 port we're not going to throw the 48 port away we're simply going to buy another switch and tie the two together. Well in the virtual world you just make a bigger switch remember it's built to order as we discussed before so consequently there is no need to have two switches tied together and the part that controls who's the master and who's the boss in this is a spanning tree protocol and the spanning tree protocol sends around frames to determine this. Well it's not needed all it's doing is just simply creating traffic in your environment that is absolutely useless you know possibly even slowing your system down. There's no need for the spanning tree protocol because you can't hook two switches together because virtual switches can't share the physical ethernet adapter there's not really a way to fool the ethernet adapter and you're doing some kind of a loop back or similar situation they could call some kind of a memory leak or cause a leak between the virtual switches.