 How's it going? My name is Aaron Rosen. And my colleagues are Eric, Dan, Savatory, and SOMIC. And we set up this OpenStack quantum lab, basically demoing some of the new features that we have in Grizzly. So I think we have a few extra labs. If people didn't get them, you can raise your hands and we can try and feed those out to you guys. But on the piece of paper, you'll see there'll be an IP address. And there's two different ways to access the lab. The first way is you could just SSH directly to the IP address. And the username to do that is Nasera. And then it'll prompt you for the password. And the password is Nasera again. N-I-C-I-R-A, right here. So just because these are internet-accessible labs, they're running in our OpenStack cloud that we run back at. Oh, now VMware. And so don't start using one of them. Give a copy to someone else because you'll be clobbering. You'll be working on the same setup. So we were able to give basically one lab set up to each kind of, for each chunk of tables here, you've got one on each side. So four per row. So share with your neighbors then. We have a set of extras that if you really don't like your neighbor or something, raise your hand and try to get an extra one from SOMIC or Sachin. So I also want to have each SOMIC solvatorian of them introduce yourselves and kind of wave your hand. So these are the people that you can kind of bug with questions if you get stuck, et cetera. Aaron will try to move forward at a reasonable pace that most people can keep up with. And then we'll be jumping in and trying to help our people are getting stuck. So if you're one of those helpers, just raise your hand, please. Wave it. Come and grab one of us or raise your hand and we'll come find you. It's probably easier. Cool. So the two ways to get there is either via SSHing if you have a shell on your computer or the second way is you can get there through a VNC console. So if you just point your web browser to the IP address on the piece of paper, and the password again is Nasera. And this will bring up a VNC console into the machine that we're going to be running the demo from. So I'm going to wait a couple of seconds and let everyone get logged in. If you guys have trouble logging in, just raise your hand and someone can help you. Cool. So is everyone pretty much logged in? Anyone have trouble? Cool. So while you guys are going in, I'm just going to give some overview about what we're going to be talking about. Along with the lab, there's also an etherpad link right here, which would be helpful to follow along with. I have it on the second slide, too. So basically in this lab, we're going to build a multi-tier application. I'll get into those details more later. But what we're really going to do is we're going to show off the new features that landed in Grizzly. So some of those features are security groups. And previously in Folsom, Nova, we were leveraging Nova for their security groups. And we implemented them in Quantum because it made a little bit more sense, because Quantum is a networking component. Security groups have to do with the network. Security groups in Quantum are a little bit richer. We also support egress filtering. So for instance, if you want to block communication from your VM going out, the Quantum security groups provide a way to do that. We'll also be demoing the load balancer as a service. So basically, you can provision a load balancer programmatically, specify to your pool members, and show load balancing requests. Another weak point that we had in Folsom was metadata with overlapping IPs. Nova network doesn't allow overlapping IPs. So in Grizzly, we implemented a metadata proxy agent to allow this to work. In addition, there's a DATP agent scale out and various other improvements and cleanups. So once you get logged into your lab here, this is the topology that we have set up. You'll land on this desktop, which is accessible via VNC and SSH, and also RDP if you have a RDP client. Once you land on that, you'll want to access this node, which is the external client that's on the internet. And we're going to be provisioning all of the stuff CLI from that client. We have two hypervisors running Nova Compute. On these hypervisors, they have the quantum open V-switch plug-in agent. And that's responsible for building the layer 2 fabric in order to span layer 2 over layer 3 segments. And there's also the controller node, which runs all the API endpoints and the network node, which controls the L3 stuff, the metadata agent, and DHCP. So let's go ahead and get situated and get on this node, the external client. So if you're using the VNC, you'll have to open a terminal. And then, yeah, so if you go to this etherpad link, the slides are also on them. Can you guys read this? There's no way you can read that. To the slides as well? Or is that on that page? It's on that page. OK, great. Yeah, so that's the. Does anyone need another second? Yeah, so once you access this etherpad, the slides are located right here. You should be able just to click on that. This etherpad link is also on the quantum section of the etherpad links for the Havana summit stuff. So it's also there. You know what it is. Cool, so the first thing we're going to do is we're going to hop on to that external network node that I have here in this topology. So his address is 10 127 1.201. So the two ways to get there, through the VNC, if you open a terminal here, you should be able just to type that IP address and then hit Enter. And the password again for this is Nasera. The second way is if you just SSH to the IP address on the page. And once you land on that landing host, which is this desktop host, then you can SSH directly to that. Does anyone have any problems getting in, getting to this point? It's cool. So what we're going to be working on today is we're going to try and build this multi-tier application. So basically what this application is, is we have two web servers, a database server, a jump box, and a load balancer. So the way this works is Tenant is going to go ahead and provision some security groups. These security groups are going to be self-referential security groups. So the first security group we're going to create is for the jump host. So the jump host, what his job is, is basically you'll be able to SSH to him and then get into all of your instances. So the idea for this is so that all of your infrastructure and your instances don't have to be public-facing. So they don't need a public IP address, and you can just get to them from that. So I'm going to start going through how we're going to achieve this and build this. So the first thing that we're going to do is we're going to make this command quantum net create public net dash dash router externally equals true. So what this does is this tells quantum that we're creating a L2 network that a router can uplink to. So this network usually maps to physical IPs in one's infrastructure. So this network is what we're going to use to allocate the floating IP addresses out of. So if you go to the terminal, the easiest way, one way that you can go about doing this is you can copy and paste from the etherpad to here. So once we did that, that just create the L2 broadcast domain. So we all up to using the VNC, you should access the etherpad from within the VNC window then you can copy paste. You won't be able to copy paste from your actual laptop, for example, directly in. So if you're using the VNC, you can go to bookmarks and Firefox, and you can also access that page. So this just creates a L2 network that's attached to the edge of your cloud to the internet. That's just the L2 broadcast domain. So is everyone up to this point? Was able to get in and run that quantum command? No, not yet. We'll do that in a sec. Cool, so I'm assuming we're all at this point. So the next thing we're going to do is we're going to associate a subnet with that network that we just created. So to do that, we're going to run this command, quantum subnet create. We're going to specify the L2 broadcast network that we want to associate with a subnet. Use your name for the etherpad. Let me see if it works for me. So if you can't access the etherpad, there's another link that I have right here via Bitly. If you can access it via that. Yeah, I think so. Like if someone wanted to look at what the quantum.com looks like. Sure, so once you get to this link, there's the IP addresses of all the hosts. You get SSH to those and look at the configurations if you want. They're all in Etsy Quantum and Etsy Nova going to this. Yeah, so that means someone else is using that same IP address. If there are two people using it, it's going to. So if you type exit, if you type exit, yep, and then type who. See, there are multiple people who are there. If you type exit one more time, what IP did you SSH to? Mate 5, yeah, so someone else also is accessing that. So what you can do is just say public work one or just add a number to the end of it or to the end of these commands, like name public subnet one, public network one. Someone else is already on there. Yep, that is what that pool of IP is, that subnet. Yeah, so someone also is on this lab. It's me, like they took my IP, too. Someone's on your lab? Yeah, or someone just guessed the IP. Excuse me, what is this, Eric? Hey, Eric. I think Eric said he had a couple extra ones. So just go ahead, make it move forward with the lab. Okay, yeah, let's just continue moving forward. Cool, so once we've created that network, the next step we're gonna do is we're gonna just create a private subnet that we're gonna use to connect our VMs, too. So the first thing we're gonna do is do quantum net create private net, and then after we do that, we're gonna associate that private network with a subnet. We can talk about the config files later on, if you wanna talk about that later. Here's another, okay, yep, sure, no problem. Cool, so at this point, we should have two networks, a public network and a private network. So the next step that we're gonna do is we're gonna create a external, or we're gonna create a router. So to do that, we're gonna do quantum router create, and then the router name. What is, what do? So quantum net create creates a layer two broadcast domain, so that just creates a network. So like an isolated L switch. So if you look at this first slide, creating a network just creates this first switch right here. And then when we add a subnet to it, that associates that network with a subnet. So that puts the IPAM on that network. So after we create that network, the next thing we're gonna do is create the router, and then we're gonna attach the router. We're gonna set that network's gateway interface to be router one. Sure, let's talk about the provider network stuff at the end after we make it through the tutorial. Cool, so one thing to look at is once we run these commands, this is what the topology is gonna look like in quantum. This is the virtual topology. So this 1.1.1.1 slash 24 is actually a physical router somewhere in the internet. So from your external host, his IP address is 2.2.2.2. So if you ping this box, this represents just a physical router out on the internet. And the 2.2.2.1 IP address is the IP address that any host that's attached to this router is gonna use to NAT with. So when he goes out to the internet, what people are gonna see is they're gonna see 2.2.2.1 as the source IP. 1.1.1.2, sorry. Yep, 1.1.1.1.1 is a physical gateway that's in your data center. 1.1.1 is already known and already provisioned by the service provider in the lab. So if you ping from your host, you should be able to ping 1.1.1.1, or if you trace route for it, trace route to it, you'll see that it hops through a router to get there. Do you believe you changed the password that you say something like that? I know. Does your mic go off? No. Do you want me to write it? I'll just get you to type it in. Plugged it, because I wanted to... Can you just tell us to go black because we're going to the court like that because we are reporting? Okay. We're good now. So if you want to go black, this is your goal. Okay, gotcha. Cool, so at this point, we should have the router created and we should have it uplinked to the public subnet. So the next step that we're gonna do is we're gonna uplink the private subnet that are the private network that we've already created to that subnet. So to do that, we're gonna do quantum router interface add and then we're gonna specify the router that we wanna add the interface to and the private subnet. So this is the command we're gonna be wanting to run. So after we do that, the topology is gonna look like this. There's gonna be a gateway IP address to the local network. So any VM that's attached to this private network is gonna be going through this gateway, 10.0.0.1. So was everyone able to run that? Okay, cool. So the next part that we're gonna do is we're gonna switch back over to this etherpad here and we're gonna go ahead and we're gonna start provisioning the security groups. So the first security group that we're gonna create is the SSH security group and what this group does is it allows port 22 for TCP to be able to go into the host. So the command that we should run is this. If you're trying to find the etherpad link to the code pad, it's this right here. So after you created that security group, we're gonna add these two commands and after we do that, we're gonna go ahead and demo how the metadata works. We're gonna show that we can insert an SSH key into the VM by doing a SSH key gen. So once you run that command, it'll generate a public and private SSH key and then we'll upload it to Nova via this command here. These steps already passed the presentation or the presentation just showed the topology. I can show the separate there. So we're just doing that just to show that you can ping it just to allow ICMP into it. You could choose to not do that if you want. So this will just allow ICMP and port 22 for TCP to that host. Yep, exactly. There shouldn't be. Yeah, you should be creating VMs. So once we all get that security group rule created, then we're gonna go ahead and launch some VMs. Yep, except the default when you run the SSH key gen. I'm gonna try and SSH this house and catch up to you guys really quick. Yeah, the variable is from Nova for Keystone already set into your environment. Cool, so at this point, after you run SSH key gen, it'll generate a key for us and then we'll update it to Nova via this command here. So the next thing we're gonna do is we're gonna go ahead and boot a VM on this network and we're gonna tell it to be part of the SSH group and we're also gonna tell it to inject this SSH key via metadata. So if we go back over here, this VM that we're booting up is this jump host right here. So in order to do this, we're gonna have to determine the network ID of the private network. So you can do this by doing quantum net show, private net ID. So when you do that, it should display something like this. So the next thing we're gonna do is, oh, okay, cool. Is anyone stuck on a certain section? Like is everyone generally up to the same point? I am now? You are? No? Do you need help? It's too slow. It's too slow? Can you do this via SSH or are you using the VNC? SSH. Okay. So if you raise your hand, if you're behind, we can kinda like come and try and help you or if it's just too slow, we'll just slow down a little. There's any, oh, back there. Anyone else who's stuck? One, three, yes. So my machine is using 10-0-0-2. Yep, so the, yep. So one is the gateway. Yeah, so if you do dash, so we're gonna go ahead and just boot a VM just to get everyone to that state. So in order to do that, we're going to copy this ID here and since we're gonna reuse it, we're just gonna save that in a variable. So if you copy, so this ID here is the ID of the network. So once we get that ID there, we're gonna go ahead and boot a VM on that network. So that command is this one. So you can see here, the parameters that we're passing to boot is we're gonna tell it the image that we're booting is this a Cirrus image. We're telling it the flavor is one. So this basically tells it the size of the machine. So the number of cores, the amount of RAM and disk. And then we pass it the network ID and that's the network that we want it on. This key name is the SSH key. We're telling it that we want it part of the SSH security group. And then this is just the name of the VM is Jumpbox One. So after you run that command, if you run NovaList, you should see the status of the VM. And it says it's currently active. It's been booted. Okay, so one thing to note what's going on. If you do quantum port list, you'll see that there are a few IPs already there. So 10.001 is that router interface IP. 10.002, that's the IP address of the first VM we booted up, which is the jump host. 10.003 is the DHCP agent. And that port basically is used to serve DHCP for your network. And 10.1.1.1.2, that's the IP address on the router interface that we're gonna be using to NAT with. So if we look back at this diagram here, this should make that more clear. Yep, so whenever you create a subnet, unless you specifically tell it that you don't want DHCP, there'll be a DHCP agent that is spawned off that serves DHCP for that network. So back here at the original slide, this shows the topology of what we have set up. So on the network node, there's a metadata agent and a DHCP agent, and also an L3 agent. And the L3 agent is what's used for the floating IPs. The DHCP agent serves DHCP. Cool, so is anyone stuck? If you raise your hand. Anyone else stuck? Are we all up to this point? Cool, so the next thing that we can start doing is we need to get that ID for the VM that we just booted. So if we look over here, the first command we need to do is quantum port list. And when you do this quantum port list, if you pass dash C ID, dash C fixed IPs, it should make it a little bit more readable. But we want to grab the ID of this port right here, the dot 1002. So once we get that ID, we again want to set it to this jump box port UID variable, and then we want to associate a floating IP with that port. So to this create floating IP command, what we did is we passed the port ID and then we told it the network that we wanted this floating IP to be allocated out of. So as you can see here, traffic that goes to 1.1.1.3 is going to be mapped to 10.0.0.2 and vice versa incoming. Is anyone, is everyone up to this point? We have the SSH key. Okay, later on, yeah, you're right. Yup, so this 10.1.1.3 is from the floating IP pool that we created earlier during this step. So once you assign that floating IP, we should be able to ping 1.1.1.3 and then we should also be able to SSH to it and you have to specify the username and the username for this is Cirrus. So if you do a novel list, can you do a quantum port show with that UID? Yeah, just copy that. Just go there. Oh, can I see the commands you ran? Probably be the easiest one. And then quantum security group show with that ID. More security dash group dash show and paste that ID in. Oh, sorry, security group ID. Can you try pinging it again? I'm not sure why that's not working. So, hey, can you make the font figure there? Okay. The instructions don't have a dash c, so a lot of people don't have a dash c. Okay, so one more time, in order to figure out the IP address of, so when you specify quantum port list, if you specify dash c fixed IPs and then space dash c ID, that allows you to have less things shown here so you can find the ID easily, more easy. Sure, and if you add dash c device owner. So an easier way actually to do this is probably a quantum port list dash c ID, space dash c device owner, and we'll want to take this UID right here where the owner is compute. Sure, when you do novel list, that displays the instant UID right here, and if you take this ID that goes along with the port, if you do quantum port show, this device ID will match up. See, 26F, 26F. Cool, so is everyone up until this point or is anyone stuck? Okay, cool, thanks. Cool, so is anyone stuck? Yes? Anyone else? Yes? Jump forward to the ID. So do you know the list? Okay, let me spray it. One thing is that in order to get to these VMs, you have to be on this external client. So if you look over here at the topology, this desktop VM isn't reachable via the internet. You need to be on this external client to get there. I think you're already there. One thing is if you uploaded the SSH key from the first machine rather than the second machine, you're not gonna be able to SSH to it without entering a password. So just for notes that the username is Cirrus and the password is Cubs when, like that. Cool, so, yeah, sorry. I'm not sure, sorry. I think there is a bug related to that that's open, but it should have worked when you put that floating IP there. I'm not sure what's going on, sorry. I don't know, I'd have to investigate more what's going on, sorry. Cool, so are some people at this point? Yeah? Okay, so we're gonna go ahead and boot the two web servers to continue this on. So before we boot these web servers, we're gonna create a few more security groups. So we're gonna create a database security group and we're gonna create a web security group. Yep, yeah, so if you type exit to log out of that image, we're gonna go back to the external client. So we're gonna create these two groups. So the first one, quantum security group database and then quantum security group web. And after we did that, we're gonna add some rules to that. So to the web security group, we're gonna add this rule that allows TCP port 80 to come ingress to any member who's a part of that security group. The next rule that we're gonna add, it's gonna allow TCP port 3306 to be accessed by the database servers for anyone who's in the web group. So this uses a self-referential rule. So the nice thing about this is you can continue to boot web servers and put them as part of the web security group and they'll all automatically be able to access the database security group. So more rules will be entered in automatically for you in order to allow communication to your database group. Yes, it's for the database security group and what it does is it allows anyone who's in the web group to access the members of the database group on port 3306. And the last thing that we're gonna do is we're gonna add these two additional rules that are similar to that, to the similar self-referential rules that says that anyone who's in the database group will allow the SSH group to SSH to them. So this allows our jump hosts to access all the instances. I'm not sure what's on the etherpad anymore. It was on the etherpad. Yeah, sure. So once you SSH to the public IP address on the sheet of paper, you'll have to SSH to that additional jump box which is the external client. So if you look at this topology here, you can see you're gonna be landing on this desktop host. But this desktop host doesn't have connectivity to the outside world and to the internet. So you need to get to this external client. From this, you'll be able to access any of the floating IPs that we'll associate to. So once you land on the jump box, you'll have to SSH to 10.127.1.201. So after we created these security group rules and security groups, the next thing we're gonna do is we're gonna boot a few VMs that leverage these rules. So if you still have this in your environment from the previous command, we should be able to paste these next three boot commands and to boot these VMs for us. So when I do NovaList, we'll see that I have, I just booted a web server one, web server two machine, a database server one machine and they're in build state. And here are the IP addresses that they have received. And after running NovaList again, they've gone to active state. All right, so just to keep things moving, I'm gonna go ahead and set up this web server. So the first thing we're gonna do is we're gonna go to our jump box. So once we're at our jump box, we're gonna SSH to our other instances. So as you can see, we have two web servers that's 10.0.0.4 and 10.0.0.5. So if I SSH to 10.0.0.4, it's gonna prompt me for our password. And again, the password is CubsWin. Yep, this is the password right here with the smiley face. So once we get to that machine, we have this simple dummy web server to run. So we'll just go ahead and paste this command in here and then we'll exit out from this machine. And we're gonna go ahead and do the same thing for the other web server. So what happens here is when a request comes in on port 80, it's gonna echo back which web server it was, web server one or web server two. So once you're back on the jump box, you should be able to debut again against these IPs and see the response. Web server one, web server two. So the next part of this is we're gonna put a load balancer up in front of these in order to load balance these requests. Is anyone stuck that can raise their hand? You can send someone over. Cool, so the next step that we're gonna do is we're gonna go ahead and provision a load balancer. So to do that, we need to get the subnet ID of the private network. So we can do that by quantum subnet show private subnet. So we're gonna go ahead and exit out of the jump box and run that command. And then again, we're gonna go ahead and save this ID. After we get that ID, we're gonna go ahead and create a load balancer pool. After we get the pool created, we're gonna go ahead and add our two web server members to be part of this pool. So to do that, we'll type nova list in order to figure out the IP address of the members. So if you did this out of order, your IPs may be different. So in my setup, the two web servers are 10.0.0.4 and 10.0.0.5. 10.0.0.5. So we'll go ahead and add those members to the pool. So we'll run these two commands here that add these two members to be part of that pool. So at this point, we have a pool that contains two members. After creating this pool, the next thing we're gonna do is we're gonna associate a health monitor with the pool. So what this is gonna do is it's gonna check the status of our instances if they're still running. And if they are, they're gonna keep them in the pool and if not, then it's gonna stop serving requests to them. So this is useful if one of your web servers dies behind the load balancer, then requests will no longer be forwarded to them. So after we create the health monitor, we're then gonna associate it with the pool. So again, we're gonna have to copy that ID and set it up like that. So to associate, we run this associate command here. It says it's pending create because it hasn't yet been created yet. So after we do that, the next thing we need to do is we need to create a virtual IP address. So that's our VIP IP. So what this does is any request that goes to this IP will be specified to the correct pool member. So we'll go ahead and run this command to do that. So what this did here is any request that goes to 10.0.0.7 is gonna go out to the pool members, 10.0.0.4 and 10.0.0.5. The next thing we're gonna do is we're gonna associate a floating IP with this VIP. So to do that, we need to copy the VIP UUID. And the ID of the VIP is located right here. The port ID is right here, so port ID. So if we copy this and then lastly, we associate that with the floating IP on the public network. So again, what this does is any traffic that goes to 10.0 or 1.1.1.4 will be directed to 10.0.0.7 and the same thing will happen in reverse. So if everything worked correctly, you should be able to curl to this IP address and you'll see that it load balances between the two servers. I do it once, web server one, the next time it's web server two. So once you have that working, one of the cool things we can demonstrate to show that the health monitor is working is to actually delete one of the web servers and see that the other web server is only gonna be the one responding. So if you do nova list, that'll display our instances that are running and then you can just pick one of the web servers to delete and that'll eventually get deleted. So I'm gonna go ahead and delete web server two. So after I do this, it'll take a few seconds to get detected and when I curl to this IP address, only web server one should respond. Yep, the timeout was set to be three seconds. Yep. Give or take from whenever the VM is torn down. So as you can see when I curl to it, web server one is the only one that responds. So the next thing to demo, I like to demo is the metadata stuff. So as I was saying in Falsum, we weren't able to support metadata with overlapping IPs. In Grizzly now we can support metadata with overlapping IPs. So the next thing we're gonna do is we're gonna set up the exact same network that we have now to demo the overlapping IP ability. So to do that, we're gonna go ahead and create a public network and a private network, exactly the same as we had done before and we changed the names to be private net too. And so now if I do a quantum net list, you'll see I have two networks that overlap in IP addresses. So the next thing I'm gonna do is I need to create a new router and the reason why I need to create a new router is because you can't uplink an overlapping subnet to the same router. Otherwise the router wouldn't know how to wrap between the two. So I'll go ahead and create that new router and then I'll go ahead and uplink the gateway of that network to the public network and then I'll add the subnet to be attached to that router. After that is done, we're gonna go ahead and boot another VM on this private network. So we'll need to determine the ID again and we'll boot a VM that's also part of the SSH group so that we can SSH to it. So after a minute or so, if you do NovaList, you'll see this another VM is booted, VM2 and you can see there's actually a bug that happens here that says that it's associated with the same floating IP but that's because Nova doesn't have the overlapping IP stuff and this is something that we have to fix upstream. So the next thing we need to do is we need to get the port ID of VM2. So if you do a quantum floating IP list just to see what the ID isn't and then you do a quantum port list and then we'll go ahead and associate that port with a public IP again. So now if we should be able to again, SSH to 10.0.0 or 1.1.1.6 without a password and if we type if config we'll see that his IP address is 10.0.0.1 and again if we SSH to 10.0 or 1.1.1.3 same thing, no password and he has the exact same IP address. So both of the instances have the same IP address but they have a different floating IP address. So previously metadata would not work if the VMs overlapped in IP so now that they can work. So in order to determine the port ID it's kind of a pain via the CLI so you need to do a quantum port list dash C fixed IPs list. So when you look at this you're gonna see that you have two instances with the same IP address. So you need to figure out which one of these is not the correct one. So if you do quantum floating IP list you'll see that we already have one of the 10.0.0.2s IPs associated with the port ID so we need to find the other one. Oh so once I got that ID then I associated that IP with a floating IP address. Yeah so I was always saying previously this isn't actually the two instances IP address if you do nova list. If you do a quantum floating IP list this will be displayed correctly. It's just a bug in the nova list that we're gonna fix upstream. So quantum floating IP list will show you so previously you had two IPs here you had the IP address one IP address goes to the jump box and the other IP address is to the VIP. So what I did is I created the exact same topology again and then I associated that with this IP address here and this is a different VM. So quantum allows you to have like overlapping IPs in order to. So Dan's gonna go ahead and show what's going on behind the scenes on these labs. All these labs are running on our MVP cloud back at VMR. We have about a hundred hypervisors that's running MVP and that's runs open stack that we use to dog food all of our stuff on and that's what we're using to deploy all these labs on. Where did I get the port ID? I got that from this command. I had to find the ID that matched the port. So I'm gonna go ahead and switch the display adapter to Dan's computer. I think it's on the other display. Okay so you guys definitely feel free to keep mucking with the labs in the background. I'm just gonna kind of show you a bit about the infrastructure that we're using to run these labs. So I think of this as the manager operator tool for MVP. So this is all running quantum on quantum, right? The quantum plugin that's being used at your layer in the lab is the open source open V-switch plugin. And then our cloud is actually running on the quantum MVP plugin. So this is the MVP manager. And you can see the state that MVP's created here. So this is an internal cloud that we're using for more than just the labs. It's also for kind of internal engineering, dog food as he mentioned. It's, we actually have two internal clouds. This is the Folsom based one or is this the older? Oh this is the Essex based one still? Okay and then we've got another one that's Folsom and moving to Grizzly. So they'll be Folsom for all of about two weeks I think I'm moving to Grizzly. So you can see here that we've created, you know we've got about 90 hypervisors in the system. We've got a couple gateways. That's kind of like our equivalent of the network node, how you get in and out. They're all, you can see they're all phoned home and connected to the controller, which is a good thing. We've got about just under 6,000 logical ports in this setup and about 1,600 different logical switches. So what you want to do here, if I go over to the search page, I can kind of query all the components. And so there's just a bunch of labs here. But if I for example wanted to, you know you told me you were having a problem with lab 24, you know, because that's the lab that we handed to you. I could come in here and look and actually see what's going on underneath the covers with your lab. So like that diagram that Aaron showed earlier, each lab setup is essentially three different networks. You could think of these as being three different VLANs, but you know because it's using the NICERA technology and open V-switch, these are actually created with overlay tunneling. So they're kind of, you know, you have the isolated L2 network, but without the limitations of VLANs. So, you know, here's the external network, here's the management network, here's the data network. And then you can see here, here are the ports that you're all running on. So whoever's on lab 24 is probably a pretty happy camper, because everyone seems green. You know, you can zoom into individual ports. You know, we can do things like look at the statistics of, let me, my mouse seems to not be very happy. You know, we'll be able to see if, you know, we can see what hypervisor it's running on. For example, this is server 137. We can, you know, if we had a troubleshoot, we could see the Linux device that's on that hypervisor. So you just get a lot of good management tools for this kind of stuff. So other than that, I don't think, unless there are any questions or if there's anything that people want to see in particular about this, that's all I kind of had. I just wanted to kind of give people a sneak peek of what's going on behind the scenes. Yeah? Service node, well, you should have gone to my MVP deep dive on. I think that's just a problem. No, so as you know from my MVP deep dive, the service node, there's two different ways that you can, so when you're having these layer two segments, but you're tunneling, you need some strategy for how you handle broadcast and multicast replication on a logical network. And one approach is just to have the hypervisor do it itself, source replication. Service nodes let you offload that to another host. So have a dedicated host for service node and broadcast, or for multicast and broadcast replication. This is one of the reasons that you can use overlay tunneling without having a dependency on something like multicast, like in the traditional VXLAM model. Our services can be implemented in different ways. In some cases, a service may need some centralized component, in which case, yeah, it would be located at a service node. But what we strive to do is actually implement all of the services as much as possible at the edge in the hypervisor itself, so you don't have to kind of waypoint through something. So that's what we refer to as distributed services. Any other questions? Aaron, was there anything else you wanted to show? Otherwise, we're happy to just hang around and answer other questions or help people through the labs.