 Hi, good morning, everyone. My name is Sadeek. I'm working for Red Hat as a technical support engineer, primarily concentrating on cloud and virtualization. OpenStack is one of the products that I regularly work with. Within the OpenStack, I give more attention to work on problems related with NOVA, Neutron, Cinder, and other core components. Today I'm going to talk about how Neutron builds network topology for your material application. So agenda for today, I would first give you a brief overview of Neutron. Then I would explain what are the different native components within the Neutron and what are the different external components that Neutron depends on. And I would explain what is a network namespace in general and how Neutron uses network namespace to create application topology. So without understanding what is a network namespace, it may not be able to follow my presentation completely. Then I would explain how a network topology of an example multi-terror application looks like and what are the steps building in that network topology. Then I would try to correlate that application topology with Neutron. That means what happens inside the Neutron when we try to build the application topology. When we create the network, what actually happens inside the Neutron to create that network for us? When we create a router, when we create a load balancer, when we create a firewall, what actually happens inside Neutron to make that possible for us? Then I would explain why am I doing this talk here? What was the motive behind this? So Neutron overview and components, most of you may not need an overview of Neutron, but for people who are new to Neutron, let me just spend five minutes for them. Neutron is a project within the open stack that delivers networking as a service. What does networking as a service mean? All of you here will be aware how difficult it is to build network topology for your application using conventional hardware. You need to worry about physical switches. You need to worry about physical routers or software routers. You need to worry about physical load balancers, firewalls, or software load balancers or firewalls. You should know how to configure them. You are responsible for the long-term maintenance, security, and management of all these devices. So it's also expensive because most of the devices or resources that you are going to provision may remain underutilized most of the time. It's also time-consuming because you have to spend a lot of time from planning to implementation. When it comes to networking as a service, we are going to abstract all these into software where the user is going to get an interface to build network topology for his application without worrying about underlying hardware or infrastructure requirements. So it's also easy and convenient. This is also QC. If you know what are the network details, what are the requirements for your application topology, you just need to collect those details and submit all those details to a form and click Apply. So once you do submit all these into a Neutron API, you should have your network's provision or network topology provision within a minute or two, depending upon how complex is the network topology. So Neutron is responsible to create, manage, remove networks, routers, load balances, firewalls, VPNs. So we are going to explore how Neutron is going to achieve this. So as I explained earlier, Neutron has different native components and different external components that Neutron depends on. All these components are going to work together to deliver networking as a service. So let's take a look at what are those different components and how do they work together to deliver networking as a service. So these are the components that we have inside Neutron. We have a Neutron API server which basically provides an interface for users and services to interact with Neutron. We have Neutron DHCP agent, which basically facilitates DHCP IP addressing for tenant-private networks. So when a network is configured to have DHCP IP addressing, DHCP agent comes into picture to create... to spawn DHCP server service for the tenant-private network. It's also responsible to manage... So when a user makes a change into the tenant-private network, then DHCP agent makes that change into the DHCP server service. We also have Neutron Layer 3 agent. Neutron Layer 3 agent is basically responsible to enable routing between the tenant-private networks or between a tenant-private network or an external network, as well as floating IP management. We also have Neutron Load Balancer as a service agent. It basically helps us to provision a load balancer in front of a... and form a cluster behind the load balancer using instances from a tenant-private network. We also have Neutron Firewall as a service agent. It's basically responsible to... it helps us to create rules and firewall rules and policies and apply those firewall rules and policies on tenant virtual routers. We also have Neutron VPN agent. Suppose there are two different open stack sites connected through Internet, and there is a tenant who has private network on both open stack sites. It's possible... Neutron VPN agent basically helps the tenant to connect those two networks together through the Internet, has two advantages. There is a direct connection between the two private networks, and the connection through the Internet is secure by creating a VPN tunnel. And we also have Neutron Layer 2 agent, which basically helps us to manage flow of tenant-private networks, packet flow of our tenant-private networks through a shared physical device. And Layer 2 agent is also responsible to manage the flow of packets within a compute node or within a network node. So all these components are going to communicate with each other using an advanced message queue in protocol. It can be queued or a bitm queue or any other service that complies with MQP. So why I'm saying this is because in later slides I'm going to use all these components and I'm going to explain how these components are going to work together to create topology for our application. So there are some external components also within the Neutron. This is not the complete list of external components. So we have a network namespace. We have OpenViewSwitch. We have DNSMask, which is basically responsible to create the DSCP server for tenant-private network. We use LibreSwan or OpenSwan for VPN agent purpose. We also use KeepLivd to enable high availability between tenant virtual routers. And we also use HAProxy or any other load balancing software for load balancing service. We also use IP tablets firewalls to create firewall access service. We also use DNAT and SNAT for floating IP management. We also use a lot of network user space components and allow to depend on the underlying operating system. So what is a network namespace and what is a namespace? A namespace allows isolation of a group of resources to its own space and all the resources within that namespace runs with the illusion that they are the only processes within the system. So this can happen without the namespaces they are not knowing each other without creating any conflict and while they still use the same parameters like IP address, port number, sockets or any other parameters. So if you take a look at the picture that is on your left side you will see that we have three namespaces namespace 1, namespace 2 and namespace 3. All of them are going to run HTTP deposes listening on the same IP address having the same port number, same interface names and they can run each other and serve different websites. So this is the basic concept behind namespace and grouping network related resources in such a way forms network namespace. So we are going to use a network namespace extensively when we try to build our network topology. So coming back into the so we have been talking about neutron components what is the function of each components and how do we physically deploy these components. So we have ideally an open stack deployment should have a set of controller nodes, a set of network nodes and a bunch of compute nodes. The set of controller nodes basically runs the API services database service as well as the message queuing service and the set of network nodes are going to basically run DSCP agent and all other neutron agents. All the compute nodes are going to run computer related services as well as internet. Our focus today here is on network nodes and compute nodes and we are going to look at what is going to happen instead of this compute node and network node when we try to build network topology for application. So this is a brief overview of neutron let's get into application network topology. So there is a tenant X and there are two open stack sites open stack site A and open stack site B and the tenant A is going to build a multi-tier application that spans both open stack sites. So let's take a look at what is the network topology of this multi-tier application on site A. So when we come to site A, the tenant has three tier applications where he has a web server cluster and he has an application server cluster he has a database server cluster and due to security reasons all these clusters cannot run on the same network they need to be isolated to its own private network. So this means the tenant has to create three private network on site A and the application server and the database server need to be connected together and the application server and web server network also need to be connected together as well as the web server network and the external network. So when we connect them together we need to create the routers and apply those routers between those networks. So when we apply a router this means all network communication between those two networks should be by default allowed. So we don't want to do that and we want to only allow legitimate traffic between those two networks. So to make sure that we are passing only legitimate traffic we need to create firewall rules and policies only the legitimate traffic and apply those firewall rules and policies on the routers. So we do this by creating a firewall and applying that firewall on the router. And we also have a load balancer in web server network that sits in front of the web server cluster. So we need to make sure that we have to provision a load balancer. So let's take a look at what happens on site B. The site B basically has a single network, a remote network that I want to call. So the remote network is basically responsible to send some backups or any other data from the web server network so that he has a replication of all the data on the remote site. So this means the remote site has just one network, one private network that is created by the tenant and there is a router that connects that private network into the external network and basically these two sites are geographically separated and we need to make sure that the web server network and the remote private network is connected together by creating a VPN tunnel. This makes sure that the communication between these two networks are secure and there is a direct connection between these two networks. So this is the example application topology that we are going to build. So we are going to do two things. We are going to look at how Neutron is, what are the steps to build it and what happens in Neutron when we try to build this application topology. So how to build it? There are multiple ways to build it. You can either use a horizon dashboard, you can use CLI, you can build it via API and you can use, you can create a heat template which basically uses API in the back end to build it all at once. So the choice is yours and during this presentation I want to stick to using horizon dashboard because there is the easiest way and that may be the first thing everyone want to start with. So before starting to build, let's take a look at how actually the Neutron topology that we have. So we have two network nodes for high availability purpose and all the Neutron agents are going to run on those two network nodes and Neutron led to agent has already created, we are in the obvious bridge, it's basically a centralized location where all packets arrives and then those packets are redistributed to different entities. Then these two network nodes are connected together using a tunnel network through BR term and these two network nodes are also connected to the project using BR EX obvious bridge. So this is the network topology we have upfront and we are going to look at when we do each action, when we try to create a network, what happens inside this Neutron topology. So the first step, whether we have to create three networks on site A and one network on site B and we need to make sure that we are going to create instances and the database related instances are added into the database private network and application server instances are added into the application network and like that. So the steps to create a network is straightforward. You just must know what are the details of the network like network name, network ID and gateway IP address and optionally allocation pool or DNS server. Once you have all these details you are going to log into Horizon and click on create network and once you enter all the details and you create a network you have the network created. So what actually happens inside Neutron when you try to create this network and start an instance to that network when you create the network nothing happens it just updates the database and background things happens only when you create your first instance to that network. So suppose we are going to create an instance to that network basically Neutron API server is going to tell the DSCP agent to provision that network provision the DSCP services for that network. The DSCP agent basically create a namespace for each network on both network nodes that means we have three networks and we are going to have six namespaces and two namespaces for each network. This is because we have configured our DSCP agent with DSCP agents per network so that it spawns two copies of the DSCP server services for each private network for high availability. So once the namespace is created then Neutron DSCP agent is going to work with layer 2 agent to create an interface to that namespace and attaches that interface into the BR Indovius Bridge and then it tags that interface using a VLAN ID that is specific to that network. This VLAN ID is basically used to segregate the packet traffic inside a network node or a compute node for a tenant private network. So once this interface is created then the interface is configured using an IP address from the tenant private network. The IP address is basically going to be the DSCP server IP address for that specific tenant network. We are going to have two different IP addresses one IP address on node 1 one IP address on network node 2. So once the IP address is configured then DSCP agent is going to spawn a DNS mask process exclusively for that network to each namespace. The DNS mask process is basically a duplicate copy that is what is going to run on each network node. So how the high availability works both the DSCP service service for a tenant network is going to respond to the DSCP server for DSCP discover from the instances which is running the DSCP client and both the DSCP server will respond with a DSCP offer and the DSCP client is basically responsible to accept only one of the DSCP offer and drop the other DSCP offer and it then sends a DSCP request specifically to that DSCP server. So even if one of the network node goes down the other network node would still be able to respond to DSCP request. So this is what happens inside DSCP so after provisioning the DSCP server we need to make sure that the layer 2 traffic for the tenant private network is flowing through the network node to make sure that we are going to add a flow into the br turn on the network node that translates the internal VLAN ID for that network to external VLAN ID or external GRE or VXLAN tunnel ID. So once this flow is there then the packets from the DSCP server is free to flow outside the network through the physical shared physical device. So let's take a look at what happens actually inside the compute node at this time when you start an instance for each interface that an instance has into a Linux bridge and the Linux bridge is actually attached into connected to the br in obvious bridge using a patch PR cable and the end of the patch PR cable that is connected to the br int has a VLAN tag associated with it. So when the packet reaches the qvo x end the packet is going to get a VLAN header added into that from there the packet is going to be moved down to the brtun tunnel bridge and once the packet is on the brtun tunnel bridge then there is an obvious flow inside the brtun that basically translates the internal VLAN ID to external tunnel ID or GRE ID last. So once this is done then the packet is going to move out to the DSCP server for DSCP request or for any other requests to the desired physical device. So we have the DSCP server up and running the next step is basically to create a router and connect networks using the router. So the steps to create the router is straightforward we just click create router and you have the router ready and you click on the router and make sure that the correct networks are attached into the router. So I don't want to spend a lot of time here because this is self explanatory the interface tells how to do that. So basically I'm going to look at what happens inside Neutron when you try to provision routers to... So I'm going to first take the Neutron routers that's going to attach the web server network and application server network and the application server network and the database server network. So Neutron API server sends a request to Neutron layer 3 agent to provision the routers and Neutron layer 3 agent basically going to create a namespace for each router on both network nodes. Once the namespace is created then Neutron layer 3 agent is going to work with Neutron OpenView switch agent to create an interface to create an interface into that namespace and the interface is tagged with VLAN ID so then the interface is configured with an IP address from a pre-configured network which is expected to pass through the VRRP traffic between those two namespaces. So once the interface is configured with this IP address it's also going to be tagged with an VLAN tag it's also going to be attached into the VR in OBS bridge and once the interface is up we have the VRRP traffic moving between these two namespaces. Then again Neutron layer 3 agent is going to work with Neutron OpenView switch agent to create two other interfaces and one of them is expected to hold the default gateway IP for one of the network and the other one is expected to hold the default gateway IP for the other network. This is going to happen on both network nodes there is no IP address configured right now just the interface is created and attached to the namespace. So once this is finished then layer 3 agent is going to create a configuration file with the details of the gateway for each private network and on what interface it needs to be configured and all the details and what is that VRRP traffic interface and things like that then once the configuration file is created by the layer 3 agent it's going to spawn a keep AliveD process using that configuration file and once the keep AliveD process is spawned one of the namespaces will be elected as the master node or master for the virtual router and that keep AliveD process is going to bind the default gateway IP for that private network to both the interfaces and that would start listening and and through using that namespace that namespace on the active master node we are going to enable routing between the virtual networks once the network node goes down the VRRP traffic should stop happening and the other node would understand that the master node has died and it would take over the IP address and act as the active active router for that router this is actually what happens when we create a router so we have created two routers that means we have two namespaces configured like this so the next thing is to configure create load balancer for web servers so when we create a load balancer we need to know we need a health monitor so that it can check the health status of the backend nodes we also need to create a pool and attach a VIP to the pool the VIP is going to be the IP address where the load balancer service is going to listen and we also need to specify what are the backend members going to serve the cluster services so once we do this we do have the load balancer provision and we have formed a cluster behind the load balancer using nodes from the private network so let's take a look at what actually happens in said neutron when we provision a load balancer in front of a cluster so the first thing that happens is neutron API server sends a message to the load balancer as a service agent to provision the load balancer and there is no high availability for load balancer instances right now so it's going to choose one of the network nodes to spawn the namespace specifically for that load balancer instance then once the namespace is created the Elbas agent is going to work with the layer 2 agent to create an interface to that namespace and configure that interface using the VIP that we specified while provisioning the load balancer and once that is finished it's going to be tagged using an internal VLAN tag ID for that private network and later after that load balancer agent is going to create an haproxy.cfg configuration file that explains what is the VIP and what port the service on the VIP is going to listen and what are the backend nodes that is going to deliver the service within the cluster on what port the backend nodes are listening for the service once we have, once the Elbas agent creates the configuration file it's going to spawn an haproxy process using that configuration file once the haproxy process is spawned then we have our load balancer ready and listening for the VIP so it's actually what happens inside Neutron when we try to create a load balancer so earlier we explained how to connect those private networks together and now we are going to explain how we are going to connect the web server network to the external network so basically the steps are same to create the router but instead of attaching both the network as an interface you need to set a gateway for the router using the external network once you do that you will be able to provision floating IPs create floating IPs and associate floating IPs with instances so we are going to look at when we connect web server network and external networks together what actually going to happen inside Neutron so the first thing we are going to create a router and connect using that router to the external network the first thing that happens just like I explained when we connect the router is going to create two namespaces and going to configure the interfaces for keep alive VRRP traffic then it's going to create other two interfaces to configure the default gateway IP address from the external network as well as from the web server network so the only difference here is that one of the interface is going to also carry the floating IPs so the floating IPs are basically configured as an IP alias on that IP address so if you have 10 IP addresses then we have floating IPs then we have 10 IP alias configured on that interface so this is the only difference when we come to when we come to connecting a private network to external network and for the floating IP management there is also a D NAT and SNAT rule inside the namespace that translates the floating IP back and forth to the private network so now let's go ahead and create a firewall so when we create a firewall we first need to create a firewall rule that specifies what services need to be allowed between the private network you need to allow database services on what port the database is running and once you allow all the services you are going to create a lot of rules and you are going to create a policy using those rules and you are going to use those policies to create a firewall so once you create a firewall so the steps are three steps create rules create a policy using those rules and create firewalls using those policies once you do that let's take a look at what actually happens in serenutron there are no namespaces created or anything like that when you create a firewall so this is actually going to be applied on existing virtual routers and when we create a firewall here we are going to have we are going to apply related IP tables rules all the rules that we created are going to be translated into corresponding IP tables rules and those IP tables rules are going to be applied in the namespace between the two interfaces and once the IP tables rules are there the interfaces are going to route only based on those rules so let's take a look at how do we create a VPN and how do we basically what happens in serenutron when we create a private VPN so to create a VPN we have to first create an internet key exchange policy and once we create the internet key exchange policy we have to use an IPsec policy using that IKE policy then later we are going to use create a VPN service using those two details and once we finish then we are basically going to create a site to site connection so here we have a site A and site B and on site A we are going to create a connection to site B specifying what is the actual gateway, external gateway IP for the site B what is the private network behind that gateway on site B we are also going to create site to site connection to site A where we specify what is the gateway IP on site A and what is the private network behind that that need to be connected together so once we create the site to site connection on both ends what happens in serenutron so we are going to create a VPN tunnel between the web server network as well as on the remote network and once we do that we already have on both network nodes the queue router namespace corresponding to that virtual router running up and running so on that namespace we are basically going to spawn an open swine or Libre swine process using a configuration file which explains what is the remote network, what is the private network and gateway IP for the remote network so we are going to run this open swine process on both site A and site B so once we have it running then this process that actually runs inside the namespace is going to capture requests for the remote private network and encapsulate that packet and send it to the gateway on the remote site so once the gateway on the remote site accepts that request it understands this is a tunnel traffic and it removes the encapsulation and sends the packet into the private network and it's behind the gateway so this is actually what happens inside the VPN agent so this is all I have to explain to you what actually how Neutron builds network topology for this example multilateral application there can be a lot of different ways to create your application and there are a lot of it differs from deployment to deployment so where did I go just one minute this is the example multilateral application topology that we have created using Neutron and basically if you look inside Neutron this is what actually happened if you are going to use a conventional hardware you are going to do all these things individually you are going to do this on firewall you are going to have a load balancer, physical load balancer you are going to do this manually and all these are abstracted into the software and you get aware to provision this topology within a minute or less than a minute this really brings the advantage of Neutron and how we can use this to provision topology for our application so that's all I have today and thank you for listening and if you have any questions please stick to the example network topology that I have explained here because there are different ways to build an application topology so it takes a lot of time to figure out if you are going to talk about a different topology thank you very much for your explanation I want to this technology can I use this environment on obvious previous version of the OpenSarc and Juno Icehouse I mean the question is not really clear this explanation is specific of Kilo 1.0 I used Juno to create this if there isn't anything new that came up in Kilo I may not have covered it I was sticking to Juno to prepare these slides thank you yeah obviously so I explained to you there are different ways to build it if you use a heat template to build it if you have prepared the heat template you will get it in a minute or less than a minute the manual way is for us to learn things how to build it not specifically to build a production if you have to provision 100 instances of this topology I mean the manual way of configuring is not scalable so we use a heat template or we use a script or anything like that directly hit the API and get it done for us so if I use a script or something like that it's not possible for you to understand what happened to the beggar just get finished in a minute or two thank you okay so I didn't get the last sentence yeah definitely you can have more than 2 network nodes so you can have 3 network nodes if you want and if you want 3 copies of the dacp instances to be running then you are going to configure dacp per network is equal to 3 so there is a configuration for virtual routers whether you want 2 instances of the virtual router to be running or you want 3 instances of the virtual router to be running or more than 3 and you can have the agents also running active active on all the 3 nodes so it's all communicates together using the message queue so which agent gets the message first it's going to execute that so it's possible to have 3 nodes also Michael Delzer American Airlines what's the pros and cons of using an external DHCP service like Infoblox as opposed to trying to use the software product within OpenStack you are asking what software is being used for the extended what are the pros the positive aspects or the negative aspects of using an external DHCP service like Infoblox as opposed to trying to do it within the software if you are an audit compliant company because the way Neutron was does not allow external DHCP to be used simply because I think something is changing in kilo to remove security groups for instances so coming to the point Neutron does not allow external DHCP service to be used because of the default security group rules it has that prevents any IP address that other than allocated by the Neutron to be it will not allow that IP address communication to the instances so the advantage is we have we are going to choose a specific IP address to any instance we are going to bend that to the instance and we are going to apply firewall rules and policies based on that IP address so it is not possible coming to the advantage and disadvantages I think I don't know if you have an existing DHCP service and if in kilo I think we have the port security that we can run a port without any security group we can leverage that so it depends it may differ from environment to environment thank you I think I was expected to talk about that in the compute slide I missed that I don't know why I missed that because I didn't write a line for that so actually the security groups works firewall is different and security groups are different basically so it's just like you have some valuable items and you lock the drawer after keeping the valuable items there then if you go out of your home the firewall is actually the main door and security group also is your drawer so security groups are going to basically work on the compute node on the Linux bridge that we create that clearly says what traffic need to be allowed to this specific instance yes basically so you are going to tell what are the services that is going to be allowed to this network using a firewall and using a security group you are going to say what are the services that is allowed to this specific instance any other questions east-west firewall basically comes into floating IP communication correct between two nodes in the same network that basically works with security groups only this firewall is only allows or prevents communication between two private networks so following up on the earlier question with DHCP couldn't the neutron plugin act as a proxy for DHCP system within the network and then it would know what the address was so it could use that in its security policy okay is that something that we are even looking into it may not work still do you know what it is I am asking is this something that we are examining as a future possibility for the DHCP security groups is what is creating problems when it comes to DHCP so we are going to allow creating a port that does not have any kind of security groups attached with it a plain port if we have that then we can have we can use any external DHCP services for that okay thanks for listening and if you have any questions I will be available here let's try to catch them