 Both teachers, PhD here, and there was a guy who started the Minnet project, throws the first lines of code and everything. And Brian has been doing it as a student and then as a teaching assistant, and now they both work on open networking, developing and maintaining Minnet at the same time. And we also have Brandon, Michiel and Vimal here who worked on the Minnet in the past. And also, keep in mind, this is your unique opportunity to date events for all the tough questions that Bob is asking in the seminars. So why don't you like that? Thank you very much, Giannis. And so, yeah, as Giannis mentioned, there's a lot of team Minnet here, which is representing, which is pretty great. And probably some of you also should give credit to, actually, Disney McKeon, who really deserves credit for pushing the teaching side of Minnet by adopting it for use in his courses. In particular, CS 144 and CS 244, which are the undergraduate and graduate computer networking courses here at Stanford. So today I'm going to talk about, I feel like I'm handled by this, am I going through some speaker or something or I'm just going to just talk about it. I'm going to talk about network emulation with Brian and Bob. And so I guess the question you want to have is, what is network emulation? And I think it's important to define it just for the purpose of this talk. And network emulation, for the purpose of this talk, at least, or running on an emulator, means really running unmodified code interactively on virtual hardware on a regular PC. And this can give you potentially some nice things. It's convenient just running on a PC or laptop. And it's really cheap. You don't need any extra hardware just running on your laptop. And this is in contrast to running on a hardware test bit, something like a custom test bit that you might build out of a rack of servers and a rack of switches of wiring them together, or maybe a shared test bit like EmuLab Virginia, which you can connect to your internet and reserve slots. And as long as it's not for SIGCOM, you'll probably be able to get some work done on it. Or as long as it's not for SIGCOM, you'll be able to get some work done on it. And this is also different than a simulator. A simulator like NS3 or OpNet is a piece of software that also simulates the functionality of a system. But usually it requires that you modify your code, and usually it doesn't run it interactively in real time. Well, sometimes a code that's an abstract simulator can run much faster than real time. But usually if you want to do a complicated simulation, it probably takes a long time. And it doesn't necessarily operate interactively performance. So why is emulation awesome? Well, emulation is awesome, I will argue in this talk, because it allows you to do this, to move apps seamlessly between your laptop with this emulated hardware and a real hardware network. So the whole idea is that we take a real hardware network of switches and hosts and applications, maybe SDN controllers, and we shrink it down so it can run on your laptop. And we do this in such a way that you run exactly the same code. So the same apps, the same post apps and server apps, client apps run on the emulator as run on software. The same SDN apps, so controller applications that you might write to do custom routing or some interesting new network application using OpenFlow. The same application works on the emulator that works on your regular hardware. You can take your hardware network that you have, like say you're in IT services and you have a hardware network, it's like the Stanford Backbone, you can model it and your emulated network can do experiments on it and then hopefully it will work the same way as the hardware network or your configuration of the emulated network will work the same way as hardware. And it's very analogous to what people do in mobile development, which is that you may not have an iPhone, but you may be taking say CS193P here at Stanford, which is iPhone programming, or you may not, it turns out debugging on a phone is really a pain, it's got a little tiny little screen and you have to set up all this complicated infrastructure to introspect into it while it's running. So what people typically do is they use a mobile emulator. So Apple's Xcode includes an iPhone emulator, the Android development platform includes an Android emulator. So even if you don't have a handset or you don't have a handset of the right screen, Android has all these different screen sizes, you can emulate it on your laptop and then test it out on your laptop and it really should work the same on the device. So this is the same idea for computer networks and distributed systems that we really want emulation to have seamless mobility apps and that's why emulators are awesome. And so why might you want to do this? Well here at Stanford you have a lot of students who want to learn about networks and they may not have a huge network lab or they may not have a huge stack of hardware. You may want them to be able to do assignments just using their laptops so emulation can really help there. You might be a teacher doing a demonstration, say a professor doing a lecture, you want to demonstrate some networking thing. You want to make sure that your networks are harmful like some security attack where you redirect the HCP or have some rogue services on the network. You probably don't want to plug that into your real network because then everyone is on the real network and will be like, oh gee, what's going on? You do this safely on your laptop but still do the demonstration line. You can also, instead of just having PowerPoint slides this is how TCP works and this is its dynamics and this is the buffer occupancy, etc. You can actually run the real software so a company might also want a network emulator saying the reason they want mobile emulators that's that they're developing a product and they want every developer to be able to run the system and develop and figure out how it's going to work not just on a single computer but on an interesting network that has multiple switches, multiple clients multiple servers, you can have them start up doing web services or CDN or maybe multiplayer game or any sort of thing that you might want to design it's a distributed system that has multiple clients, multiple servers multiple switches and you really care about how they interact so this is why emulators are awesome because they're really Swiss army knives that allow you to do all these sorts of things and satisfy lots of people at Stanford at Silicon Valley, whoever wants to do something with a network, you don't need the hardware you can do it on your laptop but when you have it working on your laptop you can move it to a hardware that just works so the question we might ask is how do we build this network emulator that we want to and once we build it we really want to satisfy some certain design requirements we want it to be scalable we want it to be able to simulate networks of interesting size we want it to be easy to use and we want it to have some degree of performance accuracy so first I'll talk about scalability so in our network emulator we want to model hosts, switches and links and possibly network controllers that we're doing at CDN and we really want to be able to model networks of interesting size with practical performance so by practical performance I mean you can run it and it doesn't take forever and you can type at it you can make web requests and your server is delivered in a reasonable amount of time it just works sort of like a real hardware obviously it's not something for nothing you're constrained by the limitation of the actual hardware that you're model flexing but it should have reasonable performance so we should get networks of interesting size so obviously you can just do a loopback connection on your laptop and connect to a web server that way there's no switch between them we want networks of interesting sizes like the sort of thing you might have in a network lab where you have maybe 40 workstations and some number of switches so to start out with here's a very simple test network that we might want to model it's just a sort of pedagogical reasons how do we build this up there's the simplest possible network that has a switch in it we have two hosts a client host running Firefox and they're connected by a network switch so how do we model this one obvious way of doing it is to use virtualization so you're all familiar with virtual machine monitors like VMware or Xan or KBM or VirtualBox you can run these on your server and then you can run virtual machines inside it which act like real machines and so here we have a couple of our sample network which I just showed you in virtual hardware so here's our client VM running Linux kernel, running init running some background dns running bash, running Firefox or client program and also in our server VM running ultimately the top HTTP also running Linux kernel and the key thing is that the key thing is that they each have a IP address so the host VM on the left has a host IP address 10.0.0.1 and host VM on the right has IP address 10.0.0.2 so they have different IP addresses it's very important for a network application that you have different IP addresses and different clients can connect to servers different servers based on their IP address and the way this is implemented in the virtual machine monitor so this is a hosted virtualization solution where we have kind of this domain 0 which is the main operating system where we're running a regular operating system a software switch in this case an open v-switch which consists of a kernel module and this v-switch db and the way the system works is the virtualization system creates these virtual interfaces inside the VM and then exposes them to this domain 0 host as tap interfaces tap 1 and tap 0 and then the switch can connect to those and can decide how it wants to route packets between the two interfaces so once that comes up tap 0 is routed by switch maybe goes to tap 1 into the other VMs 8.0 and you can have a client-server interaction so this is great and it actually works but the problem is it's not really scalable so anybody who has run virtual machine monitors on their laptop has seen this that once you have like 3 or 4 VMs running on your laptop the performance really kind of goes down significantly well in fact I was at I think a networking group meeting and I was sitting next to Yanis and his laptop was there he was doing some work on it and his laptop was doing an overdrive it was like radiating heat in it sort of sounded like a small jet engine because the fan was spinning at maximum rate I'm like Yanis what are you doing he's like no I'm doing open play development and so he was doing the virtual machines and so he had like one virtual machine for host one for server, one for switch it was complicated it was like oh you must have a really big network like well no actually it's just this little network and the problem is that he had all these virtual machines and it really ate up all the performance all the memory in the CPU of the system and it wasn't really scalable we tried to do open play tutorials using virtual machine environment we found that making networks of more than like 10 nodes really just didn't work and even if we optimized the VMs it was really hard and so the insight I wanted to share with you all is that you don't really need most of this stuff and it's all Linux what are all these other processes for do I really want to run multiple parts of it multiple copies of it do I want to run my printer even all I really care about is the client process and the server process and so it turns out that Linux has a capability called lightweight virtualization or the Linux container architecture where you can virtualize different parts of the system so you may have seen this in Unix from years ago like the charu file system call allows you to virtualize your file system so that different processes can be attached to different views of the file system well similarly Linux has this feature called network namespaces which allows a single process or process group to be attached to a virtual interface so what we can do is we can take Firefox put it in a network namespace network namespace 1 attached to this virtual interface the rest and then it acts as a client we can put HTTPD in its own network namespace attached to its interface with a different IP address and it also has a different routing table it has a part table it has the whole Linux networking static virtualize so that it can really act as if it's its own machine but all it really is is a process and so the insight here is that every operating system you know every modern operating system and probably most operating systems virtualize the CPU already they have this idea of processes they have this virtualization feature already so processes are really easy they're really cheap we can make lots of them and if we have network namespaces each process can act as a virtual host or virtual server and the rest of the system we just do the same thing we have a single Linux kernel we just run the same virtual switch and I'll talk a little bit about these pairs of interfaces which is that we're going to use what are called virtual ethernet pairs so the Linux kernel has this abstraction called virtual ethernet pairs which are virtual wires and if you create a virtual ethernet pair it creates two virtual interfaces and everything that goes in one virtual interface comes out the other one and vice versa so they're really virtual wires everything that comes in each zero goes through the Linux kernel that comes out these two everything going into each one comes out these zero we can have any software switch we like just open these two virtual interfaces connect them as raw ethernet sockets and just read packets in and write packets out and so we can implement any kind of routing or switching in software and so we've drastically simplified our system we basically just have three processes now in a single Linux kernel and this gives us much improved scalability so to recap lightweight virtualization in Linux and network namespaces links are modeled by virtual ethernet pairs and switches are software switches for example open vswitch and if we want upload controllers they can just process it as well they don't have to be in namespaces although they can be if you want so I've shown you this virtualization or this lightweight virtualization architecture which I claim solves the scalability problem but the question is does it actually work and so we're going to give kind of a preview of the mainnet system a network of interesting size and so this is actually going to be a network of 100 switches and 100 hosts and the switches are all connected in a chain from switch 1 to switch 100 and every host is connected to its respective switch and just to make this a little bit more interesting it's going to be an sdn slash openflow design where the switches are all openflow switches and we'll connect them to an openflow controller and so this is a network of interesting size 100 hosts and 100 servers so imagine you're starting with hardware you need a rack of 100 hosts a rack of 100 switches and wire them all together it might take a whole lot of time how would you possibly connect to your external like say you wanted to have it run a server within the mainnet to communicate to external yes you can we're going to talk about that later and maybe you can demonstrate it so let's see so let's see we're adding a bunch of hosts added a bunch of switches we're going to link up each host to its respective switch and you switch to the switch next to it and then once we're done creating all the links which are all these virtual ethernet pairs we're going to configure all the hosts start with the openflow controller and start each of the 100 switches so the question is does it work and so we're going to check connectivity we started with the mininet command line on your base which I'll talk about more later but there's a command called ping-all which will basically tell each host to try to ping every host using ICMP and see if this works and if it works at least we'll see if it gets up to 100 so host H1 is trying to ping all the hosts up to H1 and it didn't take the whole amount of time and it just works and so in a few seconds we're able to create this kind of big network so we claim that we get interesting scalability for a virtual network in your lifetime so unless we start connecting to that sequence yeah, so I connected them all yeah, connected them all sequentially oh, sequentially in terms of the topology yeah, or yes yes you can create any topology you want you can create any topology you want so in this case it's what I call the linear topology but you can see how the hosts how the switches are connected so you can specify you can obviously make any topology you want both in Linux and in mininet but I just wanted to show you that this is a network of interesting size and this is the topology we're actually creating and it does actually work so the interesting thing about it is for switch H1 to talk through H100 it has to go through switch all up to like 98 other switches in order to do that so I thought it was an interesting to specify this programmatically or with a big file we'll talk, we'll show you in a bit so that was that was scalability actually let's talk next about ease of use so the question, this sort of answers your question how do you specify this so I've shown you the Linux mechanisms I pointed out the commands to actually do it but now I'm going to actually show you how to do it in Linux using the commands I specified so actually go back to that here so each of these instructions in Linux there's a command to configure it up so configuration commands for to set up network namespaces like net-ins great virtual ethernet pairs use the idling command to set up an open-v-switch for VSVS cut-all to set up controllers we'll in this case run the standard reference controller if you can run whatever program you want so I'll show you the set of commands to actually create the simplest possible well almost simplest possible network so here's this very simple network a single switch, two hosts connected to it and two active IP addresses connected to an open flow controller and I'm going to show you for walking through the commands to do this so this is how you do this using Linux commands so I've described these Linux features there are commands to configure them here are the commands so first we create the host namespaces using ipnetns to see we add those namespaces h1 and h2 we create a virtual switch using open-v-switch, ovsvs cut-all we create links, iplink add creates two beef pairs and they have two interfaces the interface on the left will be in the host the interface on the right will be in the switch so we have h1, e0 is on one side, s1, e1 so this is h1, s1, h1, e0 is going to be here, s1, e1 is going to be here we add another link for h2 to s1 and we create it's called h2, e0 in this namespace and s1, e2 for the switch then we move all the host ports in the namespaces by default if we just do iplink add it creates them in the root namespace which is like domain 0 but we need to move that one virtual interface into the host namespace so we use iplink set put h1's interface into h1 h2's interface into h2 and then we want to connect the ports to ovs so the software switch can route their traffic so we add the switch ports to open-v-switch and then since it's an open-flow switch it doesn't do anything until we attach it to a controller we have to set up an open-flow controller so I tell it that it's going to be on the local host running on the lubebat connection and then I'm going to run this ovs controller and tell it to listen on the tzp port and then finally we can configure our network so this ipnet and sexec allows us to run a process in the namespace and we're going to configure h1's interface as 10.1 and h2 as 10.2 and then we'll bring the interfaces up and then we'll try to pin between them so I'm going to demo this now and we're going to demo it since I'm terrible at typing as I discovered just today is I'm going to cut and paste so I'm going to cut this in and paste it in here okay that's the first tab and actually I cut off the bottom here so that was up to ovs.n4 right okay so now you can see we've connected we've created this bridge s1 as two ports connected to it also as an internal cpu port which we're going to and it's connected and it's specified an open-floor controller for it which if you hear this again it's even connected that's good so let's do the second half of this we'll configure our network and test it and so you can see the ping actually went through we can do more pings so we built that network with h1 at 10.1, h2 at 10.2 and we're pinging from h1 to h2 and you can see it works all of it is a simple matter of Linux commands and now you can just do it so how much of state is per interface and is there any global state like for instance for ICMP implementation in Linux is there any global state or is it all per interface state or are it's the answer is it depends so the question is how much of Linux is networking subsystems virtualized and so my colleague Eric Biederman or former colleague at Arista Network Eric Biederman implemented the network namespace subsystem in Linux and it's very interesting it changes every internal Linux call and structure so that it has a namespace feel so pretty much everything in Linux proper is virtualized and can work in a network namespace now depending on your kernel subsystem the setting may be global or local so certain things for example so over time everything is going to be virtualized eventually but there are certain settings that like are still global like for example in the past I think IP forward might have been global I think so that's a bit inconvenient because if you turn it on for one namespace I could turn it on for all of them and it's a really useful feature because that way you can build an IP router just by using Linux routing IP forwarding and setting up a static routing table you can easily build routers it used to be but I think they changed it now it's virtualized to her namespace another thing that used to be global was turning off IPv6 so if you turn off IPv6 it just turns it off for every those are the one Linux kernel it's a single setting another thing that wasn't virtualized was mbtcp that if you run mbtcp it's globalized so the key thing to remember is you have one Linux kernel for all these VMs and so if you want something that's virtualized and slice for namespace you just have to make sure that it does in fact work and so I think mbtcp may have made it so you can have mbtcp on in one host and off in another so the strangest thing right is that that way if you turn it on mbtcp in a namespace it actually turns it on for your underlying host right which you may or may not want but everything should be virtualized it's just a matter of time or all these kernel models get up to speed so there's nothing more insidious than just on or off there's nothing that's like well you know you keep some calendar oh yeah so talk about global states so for mininet the whole basic idea is only virtualize what you need so I'm just virtualizing network subsystem I'm sharing everything else sharing the user space sharing the pin space I'm even sharing the underlying file system now you can virtualize that other stuff if you want in fact you can virtualize all of it self linux containers and there's a tool to do this LXC but you know you have to make a disk image for your file system and you have to configure all your things with the right users and we think it's just a lot easier to share all that stuff by default and if you need a custom version of your custom pits or users just do that just add that on top of it so anyway here are all the linux commands you see just to make the simple network you can do it although it is a lot of commands you might ask yourself the question gosh you know wouldn't it be great if we had a simple command line tool or API that did this for us automatically and we could easily create the qualities of varying size up to hundreds of nodes run tests and as icing on the cake wouldn't it be great if it was included in Ubuntu and you could probably guess the answer and we built the system it's Mininet and it sort of does all of these things and so here's how we do the same thing in Mininet that we just demonstrated building the same network and this is using a simple API simple configuration API made by written in Python so all of Mininet's functionality is probably the packet forwarding and apps and everything that's just Linux stuff it just runs but the configuration is using a Python API so the things are slow you know it doesn't really have to be fast to set up your network although you want it to be reasonably fast but Python's not actually doing packet forwarding it's OpenVSwitch that's doing packet forwarding in a Linux kernel Python's not doing routing it's OpenVSwitch and your OpenBlow controller and those can be whatever it is can be Python it can be C it can be Java and Python is for configuring it and it's basically issuing the same Linux commands that you saw it's just doing it in a more convenient way so we have a number of abstractions in the Mininet API so you see this network abstraction which is a container for nodes so we create a network object we add some hosts to it which are host objects we add a switch to it which is a switch object we attach an OpenFlow controller we add some links which are link objects we start up our network and then once we have our network started up we can send commands to our host which our hosts are actually bash processes running in namespaces so we can send shell commands to them we can start up our web server we can make a request from it using curl we have a Mininet command line interface which I showed you a little bit of it which we can invoke on the network and then we can shut down we can kill off our web server and we can shut down so this hopefully is a lot easier than issuing all those Linux commands having an API is really good having abstraction is really good can we make this even easier and the answer is yes we can we can simply have a command line command that just does this for you and we have the MN command and by default MN just makes that really simple T-node network and we have built-in tests that you can run with this ping-all test we have pre-fabricated topologies which are parametrized so you can specify a tree topology with a particular depth and fan out we can specify link properties that bandwidth is 10 megabits per second and we also have this Linux this Mininet CLI so I'm going to show you both of these to you right now so so here is we'll first start up Mininet and run the ping-all test to check connectivity creates this very simple network checks to make sure that each host can talk to every other host and just shuts it down and it happens in about 30 seconds so now we'll take a look at our parametrized topology which is this tree topology and so it's depth of 3 and fan out of 3 so it's going to make 27 hosts connected together by 13 switches and the links I've all set to be 10 megabits so if I do give a command so we started up a Mininet CLI I have a Mininet command in the CLI called hyperb which checks runs hyperb between two hosts so we can go from H1 which is on the left the leftmost host to H27 which is on the rightmost and if all goes well to bandwidth there's a bottleneck link at the top which is the root of the tree should be about 10 megabits a second and I'll show you a little bit more of the Mininet CLI so we can start up X terminals for these two Mininet hosts so here we have Mininet host H1 on the left it has its interface it's 10.1 H2 on the right which has 10.2 I could ping between them but I'm not going to do that I'm going to do what my web server example will do we'll run this Python simple HTTP to run for 8 we can run Wireshark to look at the traffic on our Mininet host so we're going to look at this interface H1 needs 0, let's see what happens and while that's starting and then hosts and we'll actually start up the firebox web browser we'll be able to run in our Mininet host and so we're going to make an HTTP request from firebox to prime on our client to our server and you can see it all came up in Wireshark here all the HTTP things and we should see the end of the TCP connection inactive inactive so hopefully that shows you that you can build a system that's easy to use that scales well and is easy to use so the question we might have now well what about performance? it turns out that there are a lot of things where how much time something takes matters or not only the raw performance of how much time it takes but also the idea that we don't want one misbehaving host that's using a lot of CPU to interfere with the other ones or slow down the switches we want to be able to specify bandwidth limits for our links and hopefully we can meet those limits and so it turns out we can do this in Linux with a couple of additional commands that's right okay with a couple of additional commands we use the Linux's traffic control sub system which is invoked using the TC command to limit link bandwidth and to add other things like link delay and possibly loss if you want and we also limit CPU performance so that hosts or virtual hosts are isolated from each other using Linux CPU control groups in particular CFS bandwidth limits CFS is Linux's schedule and bandwidth limits allow us to say for every amount of time like every 100,000 microseconds don't let the CPU use more than 20,000 microseconds of CPU time and we implement that using the CG commands and so we're going to show you the same network that we've been looking at before for performance modeling we're going to limit this post to 20% of the CPU and we're going to limit this link to 10 megabits per second and add 50 milliseconds of delay and here are the commands to do it we use TC add a couple of QM disciplines one is the token bucket filter saying we limit the link rate to 10 megabits, the other one we'll be using NetM to add 15 milliseconds of delay then we're going to do ping and IPerf then we're going to limit CPU bandwidth basically set the period to be 100,000 microseconds and say that H1 can use 20,000 out of that 100,000 microseconds 20% of the CPU and then we'll run this command while true do on H1 and then we'll add that process into that so we'll actually try this out so now we're doing the ping we've added about 50 milliseconds of latency and now we're going to do the bandwidth test which should be 10 megabits and now we started up 10 megabits a little bit less now we started up this while true do and we put it in this really CPU container using CPU classifying and if we look at top we should see this bash time wasting process and it normally would use 100% of the CPU because it's a while true do and it's limited in the CPU container to only use 20% of the CPU so this is one misbehaving host can't mess everything up for everybody else so that's convenient and so the question is so here are the links to manage do it can we do it in Mininet? so when we add a link we can specify the bandwidth and delay and various other parameters as well and when we add a host we can call it a CPU limited host and we can say only use 20% of the CPU now the question you might have is well does this help us and what is performance accuracy anyway and so we thought about this a lot and we want to argue that performance accuracy what it really means at the end of the day is not how detailed your model is or how much introspection or whether it's based on the specification of your network or your CPU or whatever but rather whether the results batch hardware that is to say do the results on your simulator match comparably controlled comparably configured hardware now it's not something for nothing so if we have a 1 gigahertz system and we slice it into 10 virtual hosts those virtual hosts are going to be 100 megahertz so it'll be slower hosts but links similarly we don't have infinite link bandwidth the links might be slower as well we might have like 10 megabit links rather than 10 gigabit links but it should work the same if we configure a hardware system with this configuration and the emulator and run the same experiment it should work about the same so how do we test this out well I did my dissertation on this and what's commonly done with emulators to test their performance accuracy is you just run a bunch of programs you run them on hardware and you run them on the emulator and you compare the results and if they match things are pretty good although past performance doesn't predict future performance but that's at least a good sign it almost always means there's something wrong with your emulator and you haven't modeled the performance correctly or the hardware or there could be something wrong with it or maybe there's something wrong with the hardware too so that's another thing it works on the emulator and you see this obviously if you do development and industry where it works great on the simulator works terribly on the hardware it's actually a bug in the hardware so there's a bug in one system or the other so that's a really good point well we did a couple things we sort of did the traditional thing of creating micro and macro benchmarks and we wrote a technical report on that which you can look at if you're interested but thanks to Brandon Heller and Nick McCann we had a bigger and better idea and this was to sort of crowdsource our testing to basically test the system on grad students and so for CS244 which is the graduate networking course for the third project Nick told everybody to tell the students in the class to pick any research published paper that they like that they think is interesting and try to recreate the experiment that was done in this paper and run it on the simulator and see what happens and so this does a lot of really interesting things first of all computer science is often very much like cold fusion somebody does an experiment and it works once unlike social sciences we have a sample size of one yet we claim complete generality so if you try to reproduce an experiment well guess what you can't publish it anymore because it's not original work it's not considered novel this has been done before so as a result things are not routinely reproduced it's a big problem so we wanted to get the students get into the students mind this idea that experiments should be reproducible and also test the system and test the Minina system by getting lots of experiments that we could run Minina is performance accurate and produces the same result produced in the research literature now if the things don't match if the things match well then we need to make sure that the experiments set up correctly and that we're not just lucky so that's thing number one but if they don't match then it could tell us several interesting things one is Minina is broken and we need to fix it or the performance modeling that we're using in Linux isn't working that would be really good to know two is that maybe we've hit the performance limits of the system that once again the experiment would basically not be configured correctly somebody's like oh I have all these 100 gigabit links they're not getting full bandwidth it's not working right the problem is that the simulator doesn't have the capacity and the experiment really isn't configured correctly for it it could also be that the experimental result doesn't hold on 10 megabit and only holds on 10 megabit links and then the most interesting thing is well maybe there's an error in the authors of the original results or maybe there's something different that they didn't consider and so that's something we'd like to see a whole lot more of in computer science because most things that are published it's just a PDF, we have no way of reproducing it except trying to manually get a stack of hardware and do it most of the time results are rarely reproduced so we think they should really be done obviously you can do it on hardware you can do it on hardware test fed we think it's something like emu lab or genie is great for reproducible results we think simulators are good for you to share your results and to reproduce them and so the nice thing about if you go to our blog for CS244 and reproducible research you'll see all these experiments and if you click on any of them or look at any of the individual posts you'll see all the instructions with all the data and actually AMIs Amazon machine images which you can load on PC2 and actually run the experiment and reproduce the same diagram and we think that all research papers should be runnable which is that if you click on a diagram in the paper you should be able to get a link to all the data and basically hit a button and say run this experiment see if the results are the same and generate the figure and so we want to really encourage everyone to do this so check out reproducingnetworkresearch.wordpress.com and you can see all these experiments and we hope you'll do this regardless of what platform whether it's MiniNet or or NS2 being reproducible research is really important for your science so that's the end pretty much of my part of the presentation but now we get to the good part and I'm going to hand it over to my student colleague Brian and Connor Thanks Bob so we're putting together this talk that's all good we were sort of wondering what might be something interesting we can show you that MiniNet can do so if you've got your phone or your laptop out you should see a network we've got a little access point here it's called MiniNet and if you can connect to that network we can single switch in two hosts we'll call those H4 and H2 now we make this a little more interesting by adding another switch another host and then using that host as a NAT we've got my host operating system which has a connectivity to the wide area ahead we can make this topology a little bit richer by adding a third switch it gets a little bit interesting here because in legacy layer 2 networking you create a spanning tree and one of these links would be disabled but for anyone who doesn't understand openflow they know that if you're a controller to handle it you can build it so we're running an openflow controller that's going to manage this network with three switches even though there's a loop in the topology we've also added a fourth host just to make things interesting on top of these hosts we can layer some steps to the hosts that are connected and it'll also serve as a DNS server and the DNS server will serve up these DNS replies for you finally I've taken USB to ethernet dongle and connected it to a dealing router here running in bridge mode so what we've done here is we've plumbed in a standard legacy piece of hardware into a virtual network running in Linux on my host operating system which we've given connectivity to the network so we've taken the black box networks that we've seen and we've shown that we can really plug these into the other Linux faculty finally one more interesting thing the openflow controller is not doing shortest path routing it's actually by default routing through this bottom switch always this bottom switch has a bottleneck link with this top switch and that's going to cap all traffic that goes through it at 5 megabits per second I've also added a faster link that by default is not used but the controller has the ability to dynamically switch hosts across that link for access to the network so I'm going to switch over quickly we'll see we've got some hosts connected here which is fantastic so we've currently got about 40-ish maybe more hosts and we can see here our link utilization so across the bottleneck link which the 5 megabits per second link it's probably not working well at all we can see that we're not exceeding that 5 megabits per second so if you're also downloading videos you'll realize that you'll probably detrimentally slow down the network for everybody so everyone just checking the email will probably be fine but it's not going to be great if you want to see more but we can if you want to try out the fast link we can switch some people over this video it's working that's a great video by the way that's Nick Thiemster's video from his online SDN course where he's talking about he's giving a demonstration of main head which he used in the course so Vimol you want to try out the fast number this is your computer we're going to announce internet connection and we'll see that if Vimol starts to download something he'll be able to use this faster link we weren't seeing anything there before because Vimol is using it Vimol is not downloading anything anyone else want to share the fast link with him no you can try me a mellow iphone I'm right above it mellow mellow iphone mellow iphone mellow iphone sorry I'm 84 284 I'm sorry I'm sorry I'm sorry I'm sorry we're using here to manage it so this is a custom controller I think that was for the demo I think that was for the demo I'm sorry it's not weird it's a fast link so nothing on the it's loading the video oh it's loading the video yeah it might be the wireless that buff on the video well yeah we are connected to a second time it's going to be weird I'll get your free version of the package here. Oh, really? What? That's their phone number. All these other demos work, so one of them had to break. Right. We haven't tried it with this many people. So we're starting with a mileage maybe, or perhaps we've matched out the company that has the capacity point of the piece. So yeah, we'll see what happens. And it should get, look, it is running on a virtual machine on all these guys on a virtual machine in Brian's laptop. But I really think there should be a capacity that just works. Yeah. So what's probably the wireless? What's probably the wireless? It's not the capacity, it's the wireless. You think it's the wireless? Yeah. Yeah, everyone's downloading the video. We're going through two wireless links. We're going through the AP. And then a wireless thing. We're going through a wireless link to standards. That should be that AP over there. So there should be a brief description. No, I have a good question. I know. We could be colliding, so. I have the whole thing by Marshall. So we should see the fast link. Yeah, so we should be able, we shouldn't see that the fast link is capable of running on a virtual machine. So we can, we can do that. And I can go back and show you quickly how easy it is to build something like this. So we can continue this. So three or one more time. The point of this demo was to show you that meaning it can be used as a black box emulator environment, it can also be used as a component in a larger network. You can still run real traffic over emitting that network. We're really just running standard Linux programs on top of virtual hosts, which are just as we found out processes in network namespaces. And those can be plumbed together into the Linux operating system, just like you would plumb a VM or any other virtualization support. To show you another, another mini-ed example, here's our demo topology. We add a few switches at the top. We connect those switches in a triangle. The links that we connect the switches in, we can add bandwidth and delay parameters. Then we add some hosts that will act as our web server. We've given them a default route here just so that they have connectivity to the network. We add links to connect them to the bottom host in our topology. We add a new type of node called a NAT, which just extends the host that we had in Mininet. A NAT also, when you configure it, will use IP tables to set up the traffic forwarding and masquerading, and then it will set up, it'll also just put the IP forwarding bid on for that network namespace. And then we've given it some configuration parameters, and finally we've got our DHCP server, which is another subclass example of a Mininet host where when it's configured, we'll start up DNS messages. Finally, we add our hardware interface to our last switch, and we're off the road. So before we conclude it, we just wanted to open up the forwarding questions about Mininet in general, or the demo order, I think. I was going to ask at the end, you had a hardware link at the last bit. It probably makes sense that you had to do certain things in a certain order. There's a lot of how points to order in some ways. For setting up this topology, we could have... Well, there are some dependencies. We have to create a switch before we can add links to it. Some are obvious. But the hardware link could have been added right after we created the switch. Now, the demo controller does depend on this order a little bit, because we use the open flow ports to do the switching over to the fast link. So the fact that this link is added last means that it is on port two of those switches. You can also specify the port. You can specify the port. I'm relying a little bit on order here, but inherently no. It doesn't matter. As long as you've created a host or a switch, then you can add links between them in any order you want. As I follow up to Janice's questions, I will ask the previous speakers to submit questions for our first one. When you talk about accuracy, you're referring to functionality or to performance? Both, actually. I was talking about performance accuracy. If you just put these things together, you get functional accuracy. You get something that is a reasonable interleaving of the events. But the problem is, what you often care about is some degree of performance accuracy. You'd like to be able to do a load balancer. In order for a load balancer to work, you have to work in time. If you change traffic on one link, on one link, the traffic should increase and you should be able to measure it and be able to determine, this link is congested. There are a lot of examples that you can see on our blog. Things where latency, how far you are away matters. It could be a lot of things, but a good example is latency. There's a demo of a DHCP attack where we have a DHCP server that is remote and slow to respond and then a local rogue DHCP server that we attach to your local switch and he responds more quickly. The rogue DHCP server attaches to you. You get a rogue DNS server and you get sent to some random website that you didn't want to go to. Something as simple as that can also be something more complicated like we want to do. This long rogue where actually your TCP bandwidth depends on your latency and your latency gets longer and longer depending on how your switches are. If you look at our micro benchmarks, we look at a lot of this micro scale behavior. If you do things right, you can really see a lot of TCP dynamics micro scale TCP dynamics with this approach and also larger scale. Our demo didn't work so great, there are errors whether we're overloading our poor D-Link or whether there's iffy Wi-Fi going up or whether they're interfering with each other. Ideally, this is an example of our experiment didn't seem to have good performance accuracy and why. This falls up to Brandon Heller's PhD thesis which is what does performance accuracy mean in an emulator and how do we achieve it? There are two things. One is that you should match hardware and the other is what if we don't have the hardware, what do we do? The approach that Brandon takes which I think is a really good one is to come up with the idea of network invariance. Things that we know must be true or must be false. If we can monitor our simulation over time and make sure that all the things we think must be true are in fact true. For example, if your input queue is full then you should keep getting packets out of your switch at a constant rate with constant spacing. Things like this are true. We have a lot more confidence that our simulation is running correctly. We don't have 100% confidence but we certainly know that if our axioms all of a sudden become false we know that the simulation is bad and we need to reconfigure or the emulation is bad. Question? Does it modern you when you mentor parameters that it can't handle? If I enter a latency that's best in the speed of light what happens? I think that just gives you an error but the basic answer is no. You're adding delay So submitting just gives you an error and it says no, I can't do it. Actually many of itself would put in something that doesn't let you set the links to be particularly large and people complain on the mailing list, why can't I make my links large? I want it to become magical faster if I increase this number. Increasing the number will not make your computer actually faster. This is an emulation that runs real time. On the other hand some work with Vimal and Antonin Baugh and Stanford have come up with an implemented time dilation which says that we basically slow down the clocks of the virtual machine so that every second of their time is actually 10 seconds of real time. In that sort of situation then we can get 100 gigabits. But generally you can't violate things like the speed of light. The parameters are just rejected. You can do that in a simulator. In a simulator you can make virtual time run events happen before the thing that caused them and create all kinds of parameters. In the end of the day, is it really open flow between controller and switch? So could you restate your question? So controller, switch. Are you speaking open flow protocol between? Yes. Yes, so we love open flow. I'm worried to share. We think SDN is going to change the future of networking and one of the things I needed as a network researcher working on open flow is a way to build these SDN apps. I didn't have a stack of hardware and VMs didn't really work very well and so that's why I built many of that because I really wanted to build interesting systems using open flow. But you can also just use legacy networking I mentioned how you can just use IP routing to make a Linux aesthetic IP router in Linux. Open v-switch you can just run it in bridge mode. You can use the Linux bridge. You can use any, in fact any of the applications of if you wanted legacy routing there's also route flow which implements legacy routing on top of open flow which you can also attach. So I'm not huge on legacy either in an ID, but it turns out it's a really useful application of Mininet for mainly for demonstrations and also for people who still run legacy networks and wants it experimentally. Can I ask my last question? Sure. So one of the things about using an emulator is that you can use or try to navigate for when you use a simulator is that you can modify different kinds of scenarios. And you know about that, is there any way to fuse those to say you want to modify your scenarios and track your patterns and stuff like that? Yeah, definitely. So the whole point of having an API and having these parameterized apologies is to make it so that you can run the same experiment repeatedly with different parameters and change the link bandwidth, change the link delay and the question is, well how do you the nice thing about an emulator is you can run real traffic like you guys were all able to stream a video and connect a real web server so how do you do that? And so I have an idea of doing this and that is that you can basically record the real traffic and play it back. So you can record the real traffic using something like TCP and then play it back but there are problems because yeah it won't necessarily work. So I think you sort of have to so that probably won't work. You use real applications like Firefox or Perl or whatever and you just script to them, you just run them. So the nice thing about running real Linux software is any software you can come up with, it just runs. So people keep asking me on the mailing list how do I generate traffic on Mininet? Thinking that there's some screen that has a traffic matrix that they're entering in it we don't actually have that as an interface to Mininet. Instead to generate traffic you just run applications that do things and frankly if you just care about generating traffic and you don't care what the bits are you probably shouldn't be using an emulator because you can probably calculate the results analytically using math and Mininet's doing a whole lot of work. It's actually copying the bits out of your process into the switch over the virtual Ethernet pair into another process. It's doing all that work. If you don't care what the bits are probably an abstract simulator is a better approach. Great. Thank you very much. Thanks a lot.