 So, hello everybody, I hope you all had a good lunch. We're glad to be here, thank you for that. And my name is Jakub Libosvar, I work at Red Hat. And my name is Rodolfo Alonso and I work at Intel. All right, so what are we going to talk about? We'll go briefly through the security group overview and then explain two different implementations of firewalls using the OpenWay switch. And then we will present some tools that you might find handy when you're debugging the issues. So, is there anybody who doesn't know what security groups are in Neutron? Raise your hand, only yourself. Okay, so I'm going to skip this slide. I made a beautiful picture, I spent some time on it. All right, so this is how the hybrid plugin works in Neutron. This picture shows two TAP devices and they are plugged into the OVS but not really. They use some different mechanisms because we need to filter traffic that goes in and out from the instance. So, at this picture you can see that the TAP device is actually attached to Linux Bridge and from this Linux Bridge it goes through VAT pairs and then they get into the OVS. So, those who work with networks will probably be doing this right now. We want to get rid of that and achieve this situation. Sorry, it doesn't work. Okay, this is how it should look like. Attach the TAP device directly to the OVS Bridge. You can see that there is a little difference between those two images, right? I'm going to hand talk to Rodolfo now. He's going to tell you something about his implementation with OVS. Well, first of all, I want to talk about just only a bit about what is a firewall and the evolution of the firewall in networking. The first one was the packet filtering. There are a bunch of rules, a bunch of static rules just to classify according to, for example, the network address, IP address or protocol and port. This is very similar, for example, to our home routers. But, of course, this is the easiest way to make a firewall. Then you have the second evolution, the second stage of a firewall which is a stateful packet inspection. That means not only you can match on those parameters like MAC address, IP address or protocol, but also you will track the whole connection from the source to the destination. For example, for TCP, you will have your TCP and connection protocol, you will have your synchronized packets, your acknowledged packets and you will be able to track the whole connection during the life of this packet or this connection. And then, of course, you have the most evolved firewall which are application firewalls very related with the packet inspection. Those packets go beyond the fourth layer in the OC model and grab all the data inside the packet and can analyze, for example, which application are you using, which protocol are you using, but this is beyond our firewall because we cannot manage this. This is more related, for example, for SFC using NFB combined in OpenStack. This is above our firewall. What is inside a firewall? I think most of you know what a firewall does. For example, a firewall needs to allow you ARP and network discovery if you are using IPv6 message and, of course, also a firewall needs to give you the possibility to use some services like DHCP, ICMP or IEMP. Also, a firewall must prevent against ARP spoofing by, for example, filtering MACA addresses and, of course, also DHCP snooping by allowing only trusted DHCP servers to serve these IP addresses inside the network. Of course, as Jakub will talk later, a firewall can manage connections using connection tracking and the last part, but the last is a firewall must manage user rules. Let me talk to you about the first implementation we made for this firewall using OBS. First of all, against the IP tables firewall which is already implemented in Neutron, those new firewalls are using natively OpenFlows. All the packet filtering is being done inside OpenB switch. The first one, use LayerAction, which is an OpenFlow command, an action you can use in OpenFlow. How are we using LayerAction to implement this firewall? It's very simple. When you create a rule inside the firewall, you are allowing, for example, some packets, some traffic going inside the firewall and then going inside, for example, a virtual machine. But, of course, you need the traffic to go back to the source of this traffic. How can we manage this not creating manually new rules just to allow the traffic to go back to the source? Very simple. Using this LayerAction, what you are doing is reflecting all the traffic, for example, changing the destination... Excuse me, when a packet matches this rule, instantly it creates a new flow, a new rule inside OpenB switch, allowing the traffic going back to the source. It's very simple. What we are doing here is just changing the source and the destination MAC, or changing, for example, the IP destination and source, or even the protocol. This is very simple. The implementation of this firewall using the tables inside OpenB switch is also very simple. It's not very difficult to explain. For example, if you have a packet coming inside the integration bridge in a stack system, in a compute system, you will match first, because this is a match in OpenB switch, you will match the serial table, and then you will classify all this traffic. If a packet is a traffic going inside a virtual machine, this packet will go to the input traffic table, and you will use your user rules to allow or to drop this packet. The same happens when a packet goes from a virtual machine inside the OpenB switch. If you want to output this packet to another virtual machine or to an external source outside the host, you will have your user rules inside these tables, and you will allow or drop these packets. This is a... Well, first of all, sorry for this grab. Why? We are going... Are we going to present also, because this is the name of the presentation, we want to present as you the benchmark. The testing benchmark workbench is very simple. I am going to explain this. We have a traffic generator connected to a 10 gigabit port, and the traffic going into this NIC interface is going through a patch port to the physical bridge. The physical bridge sends the packet inside the integration bridge, and the integration bridge sends the packet to the virtual machine. Then this virtual machine has a reflector, a traffic reflector as simple as just sweeping the MAC address, the IP address, and the protocol, in this case, UDP port, and sending back again the traffic to the... Sorry, the packets to the traffic generator, and this is how we made the test. In this case... Sorry, I cannot explain how using the firewall performs better than not using firewall. But we ran out of time, because it's very time-consuming, and I cannot explain this, but you can see these three lines, these three curves are very close, so you don't have here a lot of performance improvement. Now, what we can see here, this is an obvious DPDK user space 2.4 implementation, and we can see here the first line, the one in the top is... Sorry, traffic going inside the virtual machine switch without any firewall, so we are reaching the top speed of the NIC interface, which is 10 gigabit. But comparing OpenVswitch 2.4 with IP tables to a DPDK version with LearAsian, we have around five times the performance improvement, which is, I think, a lot. And please, let me introduce again to my colleague, he is going to present the implementation of the firewall using contract. Thanks. So, as Rodolfo said, this implementation uses a contract table for getting the traffic back. That means this becomes a stateful firewall driver. At this image, you can see the instance A and instance B. Instance A will represent an egress traffic, which is traffic going out of the instance towards the switch, and instance B represents egress traffic. So, how does it work? The integration bridge that those top devices are attached to, it first goes to the base table, well, it distinguishes between those two kinds of traffic. So, either it processes an egress pipeline or an egress pipeline, and in egress pipeline, there are actually two tables. The first one is called egress base, and there you have spoofing protections and some implemented filters for not allowing running, for example, DHCP server, on an instance. Also, when you get passed through this egress base, it will perform a CT lookup, that means it will look where there is already a connection that exists on this hypervisor, and then it goes to the egress rules where the actual rules from security groups are implemented. And at this table, you already have the information from the contract table. So, you can say that you allow a traffic that goes out for kind of traffic that was already established as an egress traffic. There is one table that stands somewhere in the middle, because when you get an egress traffic, you still can have an egress traffic for the other instance. I will first describe the egress. That means when you get to that egress accepted table and you see that your traffic goes out of your network, it will go out somewhere in your switch and let it be switched like a normal action. And in case when you have connection coming from the outside, towards the instance, it goes again to the base table and then it sees that it's not an egress traffic for any instance that you already have attached to your bridge. So, it goes directly to the egress pipeline where similar actions as in the egress, what I describe, are. That means in egress base, you won't have any contract information. So, you can perform spoofing rule ARP, IP, MAC, and once you pass that, it will again do a lookup into contract and go to egress rules where, again, actual security group rules are implemented and they get either accepted or dropped. And once they get accepted, they do something called commit, which basically takes this kind of connection and puts it into the contract table so it can be later used. And there is a special case where you don't have any connection to or from the outside and it goes directly from one instance to the other instance. So, it won't get outside. It's on the same network. So, it goes from instance A through the whole pipeline to the instance B using the single contract table. Maybe you can ask how do we segregate those connections that have the same IP addresses or ports. So, there are zones in contract table that are used. And in this implementation, we basically use one zone per network. So, I will show the example how the rules actually look like when you have some security group rule created for your pipeline. This example shows ICMP egress traffic that's supposed to be allowed coming out of the instance and nothing else. So, you can see that actually two flows are in open flow table per one security group rule. Those highlighted red things that you can see there means the state of your connection which means for new or established connections we actually go to some other table which is also highlighted. This is the table number 73 and it's that middle except egress table where similar rules are there later. And this is because of mechanism that's implemented there for dropping connections that are not yet in the security group, in your security group. That means once you remove those two highlighted rules it will actually get you dropped your connection that you don't want. And in this accepted you can see that new connections from contract are committed as I said before. It's the second line and all other established connections don't do the commit to avoid unnecessary load. Now you can see the comparison between IP tables, firewall driver and OBS firewall driver. This benchmark is a little bit like not how it's supposed to be done because the IP tables is actually done on OBS 2.4. So I think OBS 2.5 will perform better so the performance increase that you can see on this graph probably won't be that high but surely it will be better than the IP tables. So I'm going to say... Because the first thing you are going to do, I'm pretty sure about this, is once we finish this presentation is download the code and test this, we are going to give you some tips about how to use OpenVswitch and how to debug it. We are pretty sure that OpenVswitch is a very good backend for networking in OpenStack but also you can read in the last OpenStack survey dating August last year up to 62% of deployments using more than 1000 nodes are using now OpenVswitch. This is the most commonly used backend for networking using Neutron. So I think making this kind of features to be used in OpenVswitch is a good option. Also OpenVswitch performs very well. The release cadence is very high and the community support is astonishing. I can assure you that working with OpenVswitch is easier than using other kinds of backends in Neutron. Let me show you some small... Let me check first of all the time. We are on time. Let me show you some small commands to debug or to use OpenVswitch. For example, the first one. Once you create or you stack a system in a hosting, for example in a compute node, what do you want to see is the network inside this. So for example, you have this open OBS CT also that will show you the bridge already created inside the OpenVswitch and inside the bridge, the ports which are connected to the switch, you can create several bridges inside the OpenVswitch and connect those switches between them and also connect, for example, virtual machines or physical nicks to these bridges. Of course, if you want manually to create a bridge or to delete a bridge or, for example, to list ports, you have a lot of options. We are not going to show all of them, but you can manually add a bridge. It's that simple. Add bridge and that's all. Using OBS CTL. To delete a bridge is that simple. Del slash br and then the name of the bridge. If you want to see the ports inside the bridge, you can just use list ports and that's all. If you want to go a bit further and see what is happening inside, for example, inside the bridge, you can dump the flows which are currently going through the bridge and you will see, for example, this. You will see for each flow, you will see the time of this flow. You will see the table which is matching these flows now, the number of packets. You will have a lot of information. It's very handy to use this command, but also you can go and step forward because if you want, for example, to see the data plane flows, that means all the flows crossing the open V-switch, crossing all the bridges, you can use this OBS-APP CTL command that will show you from the very first of the, sorry, the first part of the traffic to the end of the traffic of a flow, what it's really doing. For example, the first one, you can see the first part is the import is coming into the open V-switch through the port number seven. You can see, for example, which is the connection track state and other parameters like, for example, the MAC address, the IP address, which VLAN does it have, and also which are the actions being applied to this flow in the switch because you have several, you can, this flow can match several rules. So you will see which actions are being applied to this flow in this moment. And this is very handy if you want to debug. Instead of using, instead of dumping the flows or looking at the data plane of the open V-switch, what you can do is almost like simulate what a flow will be managed inside the open V-switch without injecting any traffic inside the switch because it will simulate the different flows, sorry, the different tables and actions a packet will have according only with the specification. For example, you can simulate injecting one packet in the import one and with the VLAN tag 14.04 and you can see all the bridges and all the flows that this flow will match. It's very nice and if you are debugging using open V-switch, you will use this a lot of times. And of course, how to enable those firewalls in Newton. Using open V-switch in Newton, as we said, is very simple. It's fully supported and to apply or to use those firewalls is just a matter of modifying one config line in one config file and you will have, once you restart of course the agent managing this, you will have the firewall up and running. The second, the first one, the connection tracking firewall is already measured in Newton code so if you download, you clone the Newton repository, you will have it. For the second one, the OBS action firewall driver, you need to install a networking OBS DPDK and then you will also need to modify the ML2 config file for the Newton agent. So I'm going to say a few words about the future work that we plan. So first of all, we would like to test connection tracking in user space with DPDK so we can have that nice performance that Rodolfo showed you with learn actions with DPDK. We also need to come up with a plan to migrate existing deployments that use IP tables firewall driver to the OBS. One of the things that could improve performance of OBS firewall driver would be implementing conjunctions. That means when you have any remote security groups you can basically put them all together and have one flow or set of flows or all those addresses that you would like to get your connection to. Currently, what firewall driver does when you update a port, it basically flushes down all the rules that are on in the open vSwitch and adds the new ones which makes a little time there where it actually runs without any flows so we can implement bundles and perform this operation of update like flush and add as one atomic transaction that means we will not get any time without any flows. Currently, we have only a few tests for firewall which we call functional tests that do the tests in isolation of the firewall so what we want to do is to implement better test coverage by using full stack which basically will test all integration with open vSwitch agent and we saw bugs there just because we don't have this coverage and also my favorite we will introduce a new box with all this stuff that we want to do so we can fix them and keep us busy. I will, I will, I will introduce him. Alright, so this is the end of the presentation now it's time to ask questions. So no questions. Really, no questions? No questions, no worries, questions. I went to an earlier presentation that presented a blueprint for logging accepts and drops in security groups which is something that we've been looking for for a while. How would you support that kind of a blueprint? Well, I didn't see this this blueprint so sorry, I cannot ask for at least me. I cannot ask for this. It depends where the logging is performed, whether it's somewhere in the agent or if it's in the firewall itself. Well, I imagine it was going to be in the IP tables because that's the only place that would know about the accepts and the drops. But those firewalls are running inside the Neutron agent. So there is a process already running this firewall. So if there is any kind of logging, this would be applied on the agent. This is not a separate, a parallel process. It's running inside an existing process and fully supported process in Neutron. So if you increase or you modify the login for example in this process you will have these new alerts or these new logs in this one. But first of all I need to read this blueprint. Maybe we can put it on the list of future work. Some of your benchmarks, did you it wasn't clear if you're tracking bandwidth or just connections per second? Sorry, yes maybe I didn't explain this. Can we just to it was megabits per second? What was 1000 users mean? Yes, I didn't explain that. Sorry, thanks thanks for the question. We created only one manual rule in the B-switch and to test sorry, to test how it performs with several users. For example we made a test with only one connection then we met with 10 connections 100 1000. The method we used for that was incremented the source port the UDP source port. So that was the way we created new users or new connections against the firewall. For example, with contract it's very simple to track that. With learn actions, every time you have a new user you will create a new connection, in this case a new user, you will create a new rule. So that's a bit of overhead if you have a lot of connections in a firewall. I don't know if that explains what I was trying to get at. So it looks like the tests weren't on connections per second. It's not the setup rate. It looks a little bit... A fixed number of connections we didn't have time to put all the benchmarks but we decided to put the 1000 connection benchmarks and it's megabits instead of the last bit. It's megabytes. This is a mistake. This is not megabytes, it's megabits. And the X axis is bytes per packet because we tried several sizes of it. Thank you. If you would be interested in more information oh, okay. So there are links. There are links to more information. Thank you very much for your attention. Thank you very much.