 Good morning, everyone. Welcome to this talk on enhancing OpenStack Firewall as a service for real world applications. My name is Sriram. I'm part of the management and orchestration team at Juniper Networks. We're working on neutron plugins and VMware integration and a few other aspects for the last few years. I have with me two of my colleagues, Sharath and Chandan. Together we're going to cover a bunch of topics today. Some of these are ideas that we believe are good practices from real world firewall deployments that we have seen and how we can bring those good ideas back into OpenStack and share it with the community and provide much richer functionality with respect to firewall in OpenStack. What we're going to do is start off with a quick overview of what is a firewall and how it compares with security group and then quick touch up on the firewall as a service functionality and then we will move on to the three ideas that we have which we think can be useful in various scenarios. To give a background, the difference between a security group and a firewall as a service is really if you take a very simple OpenStack setup with a bunch of compute nodes, controller and a network node, the security group is something that is realized using IP tables and it is on compute nodes and it is basically securing the traffic coming out of the VMs ports. The firewall as a service on the other hand is IP tables again, but this runs inside the network node and to be specific on the router or the namespace that each network is connected to and again it is applying policies on the router port. So it is securing traffic which is going from one network to another network. So how does this firewall as a service work in an OpenStack environment? So as a tenant you create a firewall and then you can also the horizon steps are basically to create rules and policies then you create a firewall instance and then associate those firewalls to one or more routers to all the routers and the moment you associate a firewall to a router the IP table rules are configured on the network node and your traffic is starting to you know the traffic flow is monitored and secured after that. So we have a few ideas that we think will really help the you know the overall firewall as a service functionality and what we're going to talk about today is how do we schedule a firewall policy? It is a very common functionality in real firewalls to apply a firewall rule at a particular time of the day and how we can bring that functionality into OpenStack. The next one we have is leveraging firewall logs. Logging is another very critical component of firewalls and how we can take out the intelligence from syslogs and the logs that are generated from firewalls and make use of it is something that we're going to talk about. The last one is an interesting topic where the ability to choose a particular router slash firewall. Like I said the firewall as a service works on router or the namespace and IP tables. What we want to extend that and make it more capable and how you can do that is something that we want to talk about. Okay so I'm going to hand it over to Sharath and he's going to cover the scheduling aspect and follow it up with a demo. Hello everybody. I'm Sharath Chindra. I'm a tech lead with jennipa networks so I work on OpenStack Notron plugins and my primary focus is on the firewall as a service area. So in the current proposal like in the current idea so what we see is in the field one of the most widely used features by network admins is the ability to schedule firewalls at certain time intervals of the day. So this enables them to restrict access to certain websites and access to their remote servers, SSS servers, FTP servers at certain parts of the day. So this actually frees up their bandwidth and improves productivity and gives them more return on their investment. So we see that IP tables library both on CentOS as well as Ubuntu supports the ability to schedule firewalls and by default it uses UTC as the time zone and to take an example let us say you want to create a rule and you want to schedule a rule from morning nine to five on all the weekdays you'll try out something like this. So the command is like IP tables and you define the chain and the match condition is time. You follow it with like the time start. You'll say I want to start my rule from nine o'clock till the time stop is five o'clock on all the weekdays from Monday to Friday and the action that has to be taken on the rule. So this is basically the format in which you can define a schedule on the IP tables. So these are the various parameters that are available to define a schedule. So one is date start which is a date and value on the date stop and if you want to just specify the time intervals you can say time start and stop and you can mention the day of the week or month of the week using Monday's and weekdays. So to take a few examples let us say you want to have a rule which runs only on the weekends. You can say weekdays is Saturday and Sunday or you want to have a rule which runs between two date time intervals. You can say date start date stop or two time intervals you can say time start time stop. So to get this feature working in OpenStack we had to do an enhancement in a few areas. One is the horizon UI, the other one is a neutron client and firewall service plugin and the firewall agent. So we are visualizing the enhancement in this way. So basically when you navigate to the firewall section on OpenStack you will see one additional tab where you can define your schedule. Basically that is the place where you go and define your schedule saying this is a schedule, this is the name, this is the various parameters for the schedule and then you can actually go to the firewall rule section and when you are creating a rule you will be given an option to actually select the schedule that you want to associate the rule with. You can say this rule can be associated with 9 to 5 schedule or you can choose the schedule that you want to associate the rule with. So I will be showing that to you shortly in the demo. So this is my OpenStack installation. So this is my OpenStack installation. As you can see in the firewall section a new tab has got added. So this is the place where you define a schedule. So I will give it a name. So I am taking two parameters right now. So one is both are daytime values. So this defines today from 10 o'clock till 11 o'clock I want the schedule to be active. So I define the schedule here. So now when I create a rule, so let us say I define it between 2 networks 20 and 30. So this is the enhancement that has been done to this UI. As you can see this is the place where I can go and select the schedule that I just created previously. So I am associating it to the test schedule. So my rule has got added. So actually in practice what will happen is let us say you have an active firewall and this rule is already associated with this firewall. So in the back end the firewall plugin will receive this command and actually push this schedule onto the IP tables. So for this demo we are taking a shortcut like basically we are capturing this information in the service plugin and dumping it into your log so that we can see what is happening in the back end. So as you can see, okay I will just show you the, let us look at the Jotron client. So a new command has been added and you can see that. So I hope you can see this. So basically what you are seeing is like the firewall schedule rule that has got defined in the UI has got added to the back end database and Jotron client is able to list it. So you can execute all the commands. So these are the commands available create, delete, list and show. So you can say show, test, schedule, build, give details about the test schedule. So what just happened is like when you created the firewall rule, so we are taking a shortcut here. So what it does is it sends this information to the firewall plugin and at that point we are capturing it and logging it in the back end. So as you can see this is the command that has got captured. So you can see that IP tables hyphen a, the source IP is 20 and destination IP is 30 network, protocol is TCP and hyphen m time. The date start is the time you have given and date stop is the time you have given and the action is allow. So basically what we are showing here is that we are able to receive the information from the UI and the next enhancement will be to actually send it to the firewall IP tables agent so that it can push it onto the IP tables. So this will end the demo and I'll hand it over back to Chandan. Hello everyone, I'm Chandan from Juniper Networks and today I'll be speaking about the proposal of adding packet logs in OpenStack firewalls. So if you look at the firewall implementation in OpenStack currently, the way it works is like the tenant starts by creating some rules for restricting or allowing his packets and then what happens is he aggregates this rule into a policy and that policy is associated to a firewall, right? Now, what he has to do is now next step is to iteratively go over the rule and verify that everything is working. So he has no way of understanding how his firewall is actually working behind the scenes. So he has to actually send some packets, verify if things are actually what he wanted. So this is the iterative process. He might have to adjust his rule, come back and redo the rules. So this is what we see as a problem and I think adding firewall logs to the feature set of firewall as a service will help us a lot in resolving these kind of issues. So this is where we are trying to propose the new idea. I have seen some kind of work being done on firewall logging but we are trying to present a much broader picture of what can be done. So the problem that we see currently is there is no way of logging any of the activities that your firewall currently does. So the tenant does not get to know when his packet is getting dropped silently by the firewall or even when the firewall is working perfectly fine. You don't see how effective it is. Do you have duplicate rules or do you have false positive like you are too restrictive on your rules? Those kind of intelligence are not provided back to the tenant. So these are the features that we can provide by implementing firewall logging. Again in terms of troubleshooting you can have firewall logs which can help the tenant in debugging a new rule that he implements on the firewall. And I mean these are some of the very basic use cases that you can directly benefit by implementing firewall logs. There are some high level use cases which can be implemented with firewall logs but they include some other components also. So one or two that I have mentioned here is monitoring. Monitoring can help with threat analysis. You can have correlation of threats. That means whenever the attacker is actually targeting some part of your infrastructure you can look at the historical data or you can look at the other infrastructures that you have in the same data center and that have faced the same kind of threats and effectively improve your firewall policy by adding new rules. You can have report generations and auditing going on your firewall logs but for all of this to happen the basic necessity is to have the firewall logs collected first. So this is what we think can be brought in with this idea. Find tuning of rules can be done. That means most of the time what happens is people are paranoid. They start with a very, very restrictive kind of rules and as and when the requirement is there they try to open the ports. But sometimes unintentionally you might be blocking some legitimate packets. In the similar way you might have opened your firewall just to make sure something works. So all those things can be caught by the audits of the firewall log. So this I mean till this point we have just looked at what are the benefits. Now let's look at how these things can be implemented on OpenStack with the default implementation of firewall. So here you can look at this slide and see how the OpenStack firewall normally works. You have the tenant who is speaking to the neutron server and the neutron server is actually having the APIs. These APIs are what the tenant will be pushing his configuration to. So once he is pushing his firewall configuration through this API the API will actually pass it on to the neutron agent or the firewall agent which will implement it on the router namespace using IP table configuration. Now what we want to do here is to enhance our neutron API, the firewall API and give the tenant the ability to add logging features so that he can enable logging for a particular rule so that all the logs are generated. So once the APIs are enhanced you will have the agent pushing the IP table configuration to enable the logs. We'll see in the next few slides how to do that with IP tables. Once the logs are collected or generated what we introduce here is a log server which can be a centralized log server and it will actually collect all the logs and then we can probably write an analyzer to generate the alerts and reports. But these are not within the scope. The idea is actually about creating the logs first. So once the logs are analyzed, alerts and reports are generated it will be given back to the tenant and now the tenant has a view on both sides. You have a cause and effect relationship, whatever rules he is pushing and what is the output. He has a view of both the sides. So these slides actually show the default implementation of IP table based firewall logs. So as you know the default implementation of firewall as a service in open stack is based on IP tables. So we did a little bit of looking around and found that IP tables can very easily implement firewall logs. It already has the feature available. Here I show a very simplified command. The format is very well known. You have the IP tables minus a chain and then a match condition and you also have the ability to limit the number of logs that are generated by this logging command. And then you have a log target and again you have a log prefix which can be used to identify your particular log message. And then you have this log level. So these are some of the options. There are various other options that I have not mentioned. But at the end you can see there is an example that has been given and we have collected some logs in the next screen. You can see I am pinging one of the IPs on the top and in the bottom you can see that you have the log format in the log collected. You can see that there is the log prefix actually comes as part of the logging and can be used to identify this particular log. So this is helpful and this can be easily implemented within open stack. Okay. Now that we have seen the IP table part, the configuration part of it, we want to talk a little bit about how we intend to implement this logging feature. We were visualizing this log feature to be implemented as two different parts. One is per rule logging when you want to, this is targeted more towards like debugging of your rule. So you are starting with a new rule and you don't know what will be the impact of your rule on your current infrastructure. So what happens is you can enable logging feature per rule and this will capture every packet that matches that rule and log it. And then there is a catch all rule which is going to be catching all the packets that did not match any of your firewall condition or rule condition and this will be more targeted towards generating logs and creating alerts and things like that. So here is a mock up of UI. So you can see that we have done a little enhancement and given a check box which will actually allow firewall logging to be enabled for a particular rule. And this is the one I was talking in the previous slide about debugging a rule. And here is what we give as a simplified, again a mock up screen, simplified way to give a feedback of what we have collected, what all the firewall rules and firewall logs that we have collected and giving it back to the tenant. So what happens is now the tenant will know what he has done on the firewall and how it is impacting his packet filtering. So I just want to summarize. So here is what we plan to do. First of all, we see there is a possibility of improvement on the firewall as a service with firewall logs which can help in debugging of the situation, threat analysis, and you can fine tune your rules. Firewall logging can be very easily integrated into the open stack because the default implementation that is IP table already supports it. We just have to work on making things available to the end user. We introduced something called a centralized log server and log analyzer but can be a simple sys log server or we can come up with better options. And we will need some enhancement on the UI side to make these things work. This is just for exposing the features which is already available. And I think we have seen one attempt of doing this by one of the contributors and we are trying to get in touch with him and collaborate and see how far we can take. We want to take it at a much much higher level because he was just talking about collecting the logs. We want to expose the log back to the user so that becomes more usable. So that's pretty much what I wanted to cover in terms of logging. So I'll give it back to my colleague Sriram who is going to take the next. Just to recap, we have covered two topics so far. One is scheduling firewall rules and how we can use logging to generate information and intelligence out of the network. The next thing we are going to cover is the ability to choose a router and a firewall. To understand what is the context for this, let's look at the challenges that we have today with the current solutions in OpenStack. So I don't know how many of you attended the DVR presentation yesterday but in a non-DVR case, your router and firewall is hosted on a network node. It faces a typical challenge of performance and scale because it's restricted by the hardware configuration that is available on the network node and it's also the single point of failure. And HA in Neutron is not something very easy to implement. And in a DVR case, the proposal is to move the routing into the compute nodes. You still have some limitation which is bound by the compute nodes capacity. So you still have some challenges. It does solve the performance issue a little bit. And the third challenge that we have seen is how can we leverage some of the rich features and functionality that networking appliances provide? For example, next generation firewalls nowadays can do very smart application-level tracking and enforce security policies in a very dynamic fashion. How do we bring those capabilities into OpenStack where the default IP tables are supporting a very simple five-tuple kind of security policy? So the solution we are proposing is a combination of things. When you look at the new network operating systems and modern networking hardware, it's more and more Linux-based. So it's possible to leverage them and make them as a network node themselves. You can run agents on them and they are purpose-built for networking functionality and they are built from ground up for performance and resilience. So you don't have to reinvent HA or performance with those appliances. So we can offload, routing, and firewall wherever needed into those devices. From the real-world perspective, you can bring in differentiated services. You can have some tenants realize the benefit of these advanced services and make them pay for it. Or you can have tenants who want probably a lower quality of service and go with IP tables approach. That flexibility can also be built in OpenStack given its complete open nature. The idea is really we can have all the solutions coexist. It's not that you can go with one or the other. There is a possibility to support different combinations. So I'm just taking a simple case here of a non-DVR and to highlight the problem. Let's say you have a setup where you have two network nodes, network node 1 and network node 2. And the network node 2 happens to be a 2x powerful CPU and 4x memory compared to network node 1. When you try to create a router or firewall rule, you don't get as a choice as a user where that router lands up. It could land up on network node 1, which is not as capable. What if tenants wanted to choose and are ready to pay for it? The ability to host their networking services on a much more capable system. So there is a precedent in OpenStack for this. It's not something that doesn't exist today. In OpenStack, you have the ability to choose a DHCP agent and this functionality has been there for a while. As an administrator today, you can attach a specific DHCP agent to a network. And this gives you those benefits of redundancy performance because you have the ability to choose the right DHCP agent for you. And in this screenshot, we have controller, which is a network node and a controller and a dedicated network node. And I'm just highlighting in OpenStack that you can choose it today. That's what we are getting there. So we want to make it available for routers and firewalls. There's a back-end application so that we know it can be done at each lab. We can choose a specific one for each and every router. And the firewall as a service, because of the scope it works, the next thing we want to do is associate to a router at the router creation screen itself. And again, what we have suggested here is you can treat in the default behavior also. There are routers where tenants want to create their routers on a specific particular network node or a firewall. It could be physical or virtual. They have the ability to choose. And what we want to do further from here is depending on the capabilities of that router and firewall, expose advanced capabilities to customers and end users. The additional benefits, which is basically if you want to move to an NFV kind of a model, you don't have to rip and replace everything. You can bring up nodes dedicated with virtual form factor firewalls and routers appliances, allow users to move one step at a time instead of moving one shot and grow your system over a period of time to a more richer SDN-based solution. So those are the three topics we wanted to cover. So we had some more ideas on performance and scale improvements. You want to cover that? I think we have time. So we wanted to cover this in case we had a little bit of time left. So basically this is like in the current implementation of firewall as a service, we see that there are a few performance issues which can be fixed very easily. So this is like looking at the existing problem and proposing a solution. So let us take the use case where we have four networks, network 10 and 20 are connected by router, 30 and 40 are connected by another router. So the first problem is in open stack, today when you create a rule, there is no validation done. You can actually start creating invalid rules. So basically I'm creating a rule from 70 to 80 network which is not there in my topology. So which actually succeeds in open stack. It doesn't do any validation. So the second problem that we see is let us say you create a firewall policy and associate three rules to it. But rule one is specific to router one, rule two is for router two, rule three is for router three. So even till kilo what we have seen is all the three rules are pushed onto all the routers. So which is basically router one doesn't even care about rule two and rule three. So that is the problem that we are seeing and that we have seen. So the proposal is like I mean what we see is there is no rule validation and unnecessary rules are pushed onto all the routers which actually is like a security hole waiting to happen and effects performance because when a rule is evaluated it has to be it has to scan across all the rules that are present in the system. So coming back to the solution what we see is what we can do is when a invalid rule is added to the system it can throw up an error to the user saying I don't see this particular network in my topology. So we can reject it right there when you add a rule. So that is the first enhancement that we can do and the second thing we can see is let us say I have three rules rule one is from network 30 to 40 rule two is from network 10 to 20 and there is a third case which is called any rule which can go to on the all the routers in the system. So you associate it to a policy and you create a firewall and associate the policy to it. So when you do the deployment the rule specific to the 10 to 20 goes to it rule specific to the other router goes to it the any any rule is a special case which actually gets copied to both the routers. This is the optimal way in which we can actually utilize the rules. So these are the two things that we observe in the current open stack implementation which can be improved. So that's all we had for today. Before we go to the Q&A we have a few other things to quickly talk about. Just yep so Chandan and I have written a book on open stack networking which was released just couple of days back. We have some free coupons you know we'll choose based on the Q&A session which are the best set of questions. These three coupons for complete the book will be free but we have discount coupons which anybody can use at packpublications.com. So any questions? Yeah that is true but the case is like let us say you have a rule defined and you never created the network. It's like a rule which is which can be pushed on to any router like currently. I mean till you create the actual network that rule is invalid state. If it will be great you can speak it on the mic I think it gets recorded. I don't know. Any other questions from anyone? So we told that the first part of the problem is to start logging. I think it was very evident that we will use the IP table construct or a configuration to start logging and then what we are proposing is when the IP table starts logging we will use syslog configuration to push it to a centralized server. So once you have the log configured on the centralized server then it's up to us. We can write an agent or a looping call which can from time to time grab the logs from the log server and depending on the prefix we had a log prefix right. The log prefix can be used to identify such say which tenant is going to this logs generating which which tenant is generating these logs. So depending on that we can filter out the log and show it to the tenant. So the logs can catch the tenant ID or some identification information for the tenant so that when you show it on horizon UI you can filter out the log specific to the tenant and show it to him on the screen. It might it might be totally outside of Neutron yes yes yeah as I said see part of this solution is within Neutron but if you look at the bigger picture if you want to support the bigger use cases you have to bring in see the log server does not have anything to do with Neutron or the analyzer does not have anything to do with Neutron right yeah yeah we yeah we can either leverage them or if you think there if we think there's value in creating our own then only we can look at creating something like that right we can always leverage whatever is there. So we have to make it pluggable that's the theme of Neutron right so the open stack in general we have to make it pluggable and customizable and leverage solution that already exists. Handle the deletion cases where a router is deleted and there are like rules firewall rules associated with it. Yeah so currently it is handling kilo also so when you delete a rule there will be a callback to the firewall plugin. So there is a callback which happens and we actually can take care of the rule deletion. So at least yeah so basically the rule can be moved to a inactive state but in that case we will be ending up with the okay so if you take this particular solution what I propose is if you have a bunch of rules associated with a router and the router gets deleted these rules are now in an invalid state so they can be moved to an inactive deactivated state there is an active or the inactive state for a rule. No I think yeah yeah no no he's saying the router is deleted the router you can delete the router see the rule can remain but it's not active so basically firewall is active okay let me tell the scenario we have a firewall and a rule and a router and you have filtered the rules and applied to the router like you have rule one which is given to router one now you go and delete the router one now on the firewall this rule which is which has got pushed previously what should happen to this rule this is invalid right now right because router has gone. Yes rule can exist yeah rule can exist when you say the rule will not be deleted in the yeah I think when a router is deleted you don't need to do anything to the firewall or the rule it can exist if there could be other routers or if it is a last router it can still exist when you create the next firewall or next router and you associate that time validation needs to happen no the rule will exist in the UI but it will become an inactive state in the UI yes exactly depends on the exact state at that point if there are other valid routers present it should stay multiple you have to validate I think we are done we are okay yeah I think we are thank you and thanks for coming and listening to us and we'll be available right here for if you need some other help or questions or discussions thank you