 Good afternoon everyone and thank you for joining me for this last session. I Know this is probably one of the last things spending between you and Relaxed evening, so I'll try to make it as enjoyable as possible. My name is Alex and together with my colleague Chavi we have both developers network virtualization company Midokura and we presented for this afternoon a session on creating containers for network services so the idea of creating containers for network services or otherwise said containerizing network services is To enable certain neutron services such as for example load balancer firewall Dynamic routing that is BGP or VPN Making them containers to make them To have them as easily to deploy as possible for example in your data center So the question was why containers is that in many of these for many of these services We're actually using existing software for example for L bus we may use HEPROXY for VPN we may use LibreSwan or OpenSwan, so creating containers for these services is is because we may easily package these Applications and make them available at different compute nodes now What do we look at when we want to create containers for these network services? Well, the first thing is because One of the things is that containers and these Network functions have a similar lifecycle. So for example We don't have to have a full-fledged guest operating system running in order to for example support firewall or VPN But on the other hand We want for is for example these containers this this services to start up as fast as possible the second reason is High availability scalability. So with containers it's possible that We can Move one service from one compute host to a different compute host Deep for instance if you if we have a problem with we have to take it out for man for maintenance Another reason is resiliency so By enabling these network functions containers is Possible to monitor the health and if there is a problem then we simply start a new container a different host Also using containers. It's possible to use for the same function multi-vendor solutions. So for example if you want to use BGP and Or firewall you may use different applications depending for instance for customer for tenant You can use an open source software for a different customer a different tenant You could use let's say an enterprise version, right? and if the customer is willing to pay more for way to pay the license for that version and Finally we want to make Deploying the services as easy as possible for cloud operators We want to provide them tools for management to migrate containers from across different compute hosts to Set up different policies to manage these for example the host the set of hosts where these containers are deployed now How do we look at implementing these service containers? So starting from the top you can see that At the first layer we have different possible services that we can implement the service containers for example taking from Neutron We can use load balancer. We can use firewall. We can use VPN as service and so on the second layer is the open stack Neutron layer and These we have a Neutron plug-in that usually interfaces with whatever network utilization software use and The third layer is for now Let's say an upstart layer that we call service containers And it is of this layer where we provide the actual implementation that translates between the network function and network service above and An object that represents that service At different compute hosts, right? So these for now abstract objects called service containers will represent for example the type of service that you have to you run their configuration and we're supposed to be deployed and how they interact with with a virtual network and Finally at the lower layer The lower layer layer, excuse me, we have different compute servers and It is where where we deploy these services as containers, right? So these compute hosts. They will be connected probably through an interface with the software the networkization software that you have there and They will interact with with the other components with the other instances the other virtual components in your virtual network and so Now to recap a little bit. So the key key requirements. We looked at when we started implementing service containers work scalability, so We wanted to make sure that we can deploy these services as containers out compute hosts So that is where as you add more capacity to your other center adding more computes and supporting more instances the System also scales out in terms of the kind of services you can provide Then we want to also have it high availability. That is whether it is a problem either the container level or the compute level These services can still miss a transition from one host to two different one. Oh Sorry Then we want to monitor the container health That is we want to inform tele operators whether it is a problem with a service Also to make sure that Whatever network utilization software use it's they could be packed for instance the Health of these containers and otherwise move them to a different host we want to make possible for operators to manage these containers from the container level to groups of containers by applying policies to them and These policies may include for instance container affinity to certain hosts to certain groups of hosts to affinity to other containers or they could have faith sharing with certain other services or Certain network devices or ports for example, so how did we implement containers in mid-onnet? And from now I start to just explain a little bit what we did in order to enable enabling service functions in mid-onnet using containers so Just a brief introduction is on it is another realizations software started by minda kura and has been open sourced in 2014 and It provides For example that layer 2 layer 3 virtualization Stateful and stateless not firewall and so on and since release 5 1 we've added support for VPN as a service Which we implemented using these service containers? Now just a brief overview of how mid-onnet works So we can easily understand the the next step that where we go into how we implemented service containers so at the top we have Open stack a neutron which interacts through the middle of plug-in with a controller layer we also call it a mid-onnet cluster and this layer is responsible of simply translating The higher level northbound models from from neutron to our own Mid-onnet model or southbound layer model in that we store in our own database for this we leverage to keep her and At the low layer at the low layer we have different computes and on each compute we install different software they call a mid-onnet agent and This is the main software that deals with the packet level functions in network talization so the key difference What distinguishes me on it? is that We tried with me with this mid-onnet agent to push the intelligence of the edge that is by Converting the higher level neutron models into our own mid-onnet models that we store in a southbound data race It's possible for the mid-onnet agents to download these let's say enriched modified models and Apply their own logic in order to make packet processing distributed as distributed as possible so in this example if Certain VM or instance from one compute sends a packet to a different compute the mid-onnet agent residing a dead ingress host Will apply whatever transformation is needed? such as the packet seems like it's flowing through the virtual topology that we have in green on the right side and After all this transformation, or we also call it a simulation is completed The modified packet would simply be tunneled to the egress host in this case where we have the second instance or the second VM VM 2 In a little bit of more detail. I just want to mention that With in mid-onnet we leverage the OBS kernel module. So we have OBS in the kernel and On the top in user space we put the mid-onnet agent, right? So what happens is that when a certain instance sends a packet? Me know that they intercepts the packet runs the simulation on it and then depending on whether that packet has to aggressive Or the same host or a different host We'll send it back through through the kernel and I would be tunneled at the other host or be Would be simply be up to the same host If the VM is on the same physical compute Now with this brief introduction on mid-onnet and how it works How did we implement service containers? So Starting again from the top. So we see we have open stack neutron. We have now our plug-in and Here, let's say we have a service this service can be L-bass can be VPN service can will be in the future dynamic routing of course and this is something that you either create from neutron or it from horizon from the CLI from highs and for instance and Because our plug-in interacts with the mid-onnet controller that we have there on the top What we do in the controller is simply translate the service that you create from neutron Into a bunch of objects that reside in our southbound model and these objects in this case When you create a new service we create a port for it where we connect the container We also create a service container object and this container object encapsulates the information we need for that container that is encapsulates for example the service type Probably it's version of what kind of software it has to run and As well its configuration. So for example when we launch the service container, well, you know, we have to run This application it will run like this. This is its configuration and so on and Finally, we also create a third type of object in which case we call it a service container group and The purpose for this group is simply to allow cloud operators to have a better control of group groups of service containers and this is for reason of applying for example scheduling policies or affinity of the containers for different computes or how to manage their life cycle and I will show you later a demo of how this works So this was the first step doing the translation now the second step that we we took was to do some software modifications software modifications were one to add some additional container service support in our mid-on-net software running with a controller nodes and Also to add support for starting this container At ease manager life cycle or the compute hosts So we see that we have a container service Running of the controller and several container services running at the compute so In a couple of slept steps the way this works is for example the first step is you create a service from Neutron, right? this gets translated into our southbound model through the controller and will store these objects in our southbound database then The controller would notify for example, which compute host would have to start the The container so we do this was scheduling algorithm The compute starts the controller step number three and then the container step four will report via the southbound database It's status. That is what it's running if it's healthy, it's okay and so on and Final step the controller monitors the container status and takes action Whenever something is wrong for example, the container is down the compute is done and It or it for instance the operator has modified the policy or has manually rescheduled the container and so on so now I'd like to demonstrate to you how this works and Before I do so I just wanted to take a couple of steps explaining a little bit of a little bit of setup so for this setup I on my laptop here, I started an open stack environment and I'm running a controller an open stack with open stack with Neutron the Mid-on-net Neutron plug-in and I also running the Minonet controller software and I also have two computes actually I have three computes because the Minonet agent is also running on the controller node so I can play a little bit with with the containers I can show you how I can move the containers from one compute to the other compute and The virtual topology I'm trying to demonstrate here is a simple one I have two networks they call them Mercury and Venus and they also they all have a router This would be the tenant router and they're connected to a public network and I'm trying to enable VPN on these routers such that the instances that I am running On each of these networks can can talk to each other through VPN, right? So In the implementation of Minonet what would happen when we enable VPN on these routers is the we attach to each router in the virtual topology a Container an IPsec container in this case To which traffic will flow from the tenant network and through the public one and the same the other the other tenant, right? Now a little bit of information about what translations we do So for for what whoever is not familiar with how VPN works in in neutron and opens that Right. So when you create when you set up VPN in neutron What you do is first you create a VPN service and in this VPN service you say What is the router on which we want to enable it and what is the private network that for each enable VPN and then For every VPN service that you have you create an IPsec site connection That tells what is the peer router and what is the peer network behind that route? so through the translation more like explain a little before so we're going to create the service container and service container group and of Course we also create a router port where we'll attach the container And we're also going to add all the rules that are necessary to redirect traffic flowing through this router such as for instance Clear-text traffic coming from the private network would be redirected to the container Right such that the software running in the container will be able to encrypt it But also for instance the IPsec traffic come up from outside that is the blue traffic as well as for instance the red Ike traffic that's necessary to set up the Ike security associations, right? So in the end we gonna end up with something like this So with having in blue the clear-text traffic that flows from the VM from the instances to the router gets redirected to the containers and Then in the containers the in this case we use Libre Swan and The Libre Swan software will encrypt it according to the configuration send it to the peers and This is the red traffic the encrypted traffic that we see so for this demo I have here open stack you you can see my my virtual topology let me know if it's not big enough and I can make it bigger and We also have the Two instances one of the twins is actually over there right so here I'm logged in into let's say with the instance connected to the mercury network. I can see it's IP address Right in this case. It's 192 168 12 and I for instance right to ping Let's say the second instance right it doesn't work because they are both in private networks behind routers and In order to make it work what I have to do is enable a VPN connection between them so for this First I'd like to go I have here Let me try to make it this bigger The three hosts I showed you previously in my physical topology So on the top I have the controller and then the other two computes on the bottom So on the top because I have running the mean on that controller here. I can start We have our own let's say CLI to interact with Southbound model this is needed for now at least because you don't have access to the container models from directly from neutron yet, right? so here we can see The three hosts that we have running and I try to emphasize a little bit information about what's related to the containers So we see here that we have the three hosts They are live and we also have some container configuration that we can set up later And this is needed for instance when operators apply. They want to apply policies For the container services And now if we try to list the number of containers we see that we don't have any at the moment So to set up VPN. Let's go back to horizon and see here I already created not to waste any more time the Ike and IPsec policies and Let me now set up the VPN. So for this for instance, I'm going to set up a VPN service for the mercury network All right, I'm enabling on the mercury router and the local subnet is 1 0 slash 24 I'm going to do the same for the Venus and Now at this point both VPN service are impending create mode. That is because I'm gonna get this bigger Are impending create mode because we haven't yet set up the IPsec side connections so for this we go to the IPsec side connection tab and here it is where we configure the peers That is for each VPN service that we added we configure to which Peer site it should connect So the first one would be mercury. Let's say to Venus Sorry, so we enable this connection on the mercury router We use the policies that we define these are just policies with a default configuration and here we enter the IP address of The peer router that is the Venus router and the last step is to say what is the peer subnet that is the subnet behind the Venus router in this case is 192 168 and 2 0 slash 24 And a pre shared key. Let's say my secret key so Now we can see that the VPN service for the mercury site is in active state and if we go back to our CLI and We list the containers we can see that we have one container running so this container is an IPsec container That is given by its type We can see what's the port that's corresponding to the container. So we created a new port for it And also the host what it has been scheduled So for example if we list from here the routers that we have So we can see that the router one is the router for the mercury site We can see that we have an initial port That is connected to the container and this is the port that it's plugged So when we did a translation from the neutron model to the Southbound model that we use in middle net we created this new port for the new service container and We connected this port to the router and we connected the container there. So now let's do the same at a new Site connection between Venus and mercury. So this is on Venus. We use the same policies This is is connected to the mercury router and the PSM net is 10 slash 24 and the same pre-shared key Hope everything is the same so It works so now the second VPN services in active status and if you go back To the middle net CLI we see that we have two containers now so one for each VPN service because we have two routers and The second container in this case is running on host one So let's hope the pink works and I don't have to redo this Yes, so now the pink works between two sides So I'm gonna leave it running for now. So We see we have two containers on different compute hosts. So if we list the host so host one is compute one host two is compute two So I said that one advantage of running the of implementing the services containers is it? Well, then there is a problem. It's easy to simply move the container from one host to the other host So let's try to do that now. For example, let's go to compute two where we have container zero and Let's stop the agent running there, right? so the name of the middle net agent is middleman, so I'm gonna stop that service and let's List the containers again, and now we can see that one container has been migrated from host two to host zero And if we go back To the pink we can still we can see that still running now There may be a couple of packets lost this depends on the environment in my case my laptop here is pretty slow especially with 12 gigs Allocated to running open stack on all these computes. So you will see that the TTL is quite high and It also depends because you use JVM on having a warmed up JVM. So normally the packet loss and Failing over the containers Should take very little time, but when I'm doing this the first time usually takes a little bit more now I said that another thing interesting about using containers is that We can allow operators to easily manage the way the deploy service in the network. So whereas Scalability high availability is that we allow you to for instance use the same computes so for instance you add more capacity providing services simply scales out and When there is a problem, we can simply transition containers from one host to the other host We also can provide operators with for instance features like policies and as I said previously in our controller nodes what we do have is this container service and Part of the responsibility of this container service is to handle the container Scheduling that is choosing the compute host where we run these containers Now in the client implementation we have so this is a project that we started about four or five months ago is That the controller nodes coordinate between themselves in sort of active passive way So we only have one controller node running at a time But if there is a problem you take it down for maintenance another controller node just takes up the scheduling function And they they are also restart tolerance So you can restart it and nothing in your scheduling of the existing containers will change If controller nodes are down the containers won't be supervised if there is a problem But otherwise when you start a new one, you simply see where the containers are and just work now One of the the functions of this container scheduling is of course to determine where we run these containers This is one. The second is if there is a problem and a container just a particular container has a problem We want to fail over that container to different hosts If a host is down we also monitor the host status and if a host is down Then we take all the containers that you have there and we try to migrate them to different hosts And finally we want to allow operators some granularity in manipulating these containers So for example, we allow an operator to Perform let's say per container scheduling that is say I want this container to run a threat host also to apply some scheduling group policies What kind of policies well? One type of policies is affinity policies and these tell us to which groups to which group of hosts We can run a certain container and for example We support anywhere affinity where you do you choose from all the hosts that you have available in a cloud from all computes We also have right now implemented host group affinity and this will allow for instance to choosing from a certain subset of hosts and Probably even more let's say Important for certain types of containers is poor group affinity where you have the possibility to specify that the the Compute host to which containers to run are tied to whether some ports are located there and this is for example useful When you have let's say Containers supporting BGP right you want to run your BGP containers But your gateways and the gateways are defined by where you have your uplink ports If you move your uplink port from one gateway to another for instance when you take a certain compute down for mental maintenance Then you'd like also that the containers that are there to migrate to where the uplink port goes right, so when you use Poor group affinity you simply say I want my containers my BGP containers to run where my uplink ports are and Then as these for instance uplink ports move from one host to different host The containers would also migrate to with the ports and In addition to to the affinity policies that we support. We also added some selection policies now Because this is a young project what we've we only have added two so far and probably the first one was simply a weighted policy where you just Are able to allocate a weight metric to each compute host and say What is the probability that host will be chosen? to run a container and Where the metric of zero means that we should not run a container there So for instance, we'll see during the the demonstration that when we set the metric to zero We can't run a container the container be migrated And excuse me, I also have we also have a list policy in which case this is a live policy and The list policy takes feedback from the host and tries to allocate containers to the host that have the fewest number so Let's start a ping here again. I Just want to demonstrate a couple of these of these policies because we don't have a lot of time So for example, if we list the containers now, we see that we have One container running a force one the other container running host zero. Let me start the third Compute and in the meantime, let's say I want to take host on host one for maintenance, right? And I only have host zero left. So What I want what I can do is simply Practically migrate a container container one in this case from host one to whole zero so For this host it's container one. We simply set host Host And we can see that the container has been migrated now just at the moment when I Entered container list. It was in starting status and we see it hasn't been allocated. It's an interface yet but if I'm refreshing now you see the container has started and you see there what it's network interface and We go back to to our ping here and the ping is to the running And in terms of policies that I told previously that we control policies through the container groups So right now in implementation have at the moment We create a new container group in a translation for every serve VPN service you create So now we see that we have here to container groups and the policy is the least policy, right? But For example, we can change this policy to weighted for certain containers And then we can see how this policy will affect the containers. So So let's change the policy for both groups to weighted and In order to apply certain weight for instance, let's say we have We have here our host list and we want to make host to The host with a higher container weight, right? So let's make for instance list this host 10,000 and And Let's say we don't want to run any containers the host zero is for a controller. We want to stop running containers there so what we can do is We take host zero we set its container weight To zero and we expect of course, this is probabilistic the probability of choosing a certain host is weighted by the value That we entered but it's not by any means certain and this is the other policy or the other policies What will allow flexibility for this? So now we should migrate? containers from host zero to any other host and The scheduler should choose a host based on the policies that we entered and according to the weight, right? so We can see that now both containers are running at host 2 The ping is still is still going and If we go to host 2 and we list the interfaces then we see here that we have these Are the interfaces where the containers are connected now? I'd like to I'd love to show you some more, but unfortunately I also have to leave a couple of minutes for questions so Of course, there are many other policies we can think of we implemented only two we didn't have time to add more If you want to try it so everything in this releases open source You can just you have to set up your own open-stack environment. We have on we don't work quick start. You just It's a script you downloaded you right? That's what I used to set up my Open-stack environment here. You can download packages for 5 1 that is the Because the the quickest are scripted is for 5 0 so 5 1 you'd have to download and install packages, but it's quite easy from building on it or and of course if you run into any problems Get to us on slack and we'll try to answer them so now I'm open to any questions you might have. Thank you. Yeah Good presentation. Thank you. So can you elaborate a little bit on how you got neutron to integrate with those containers? Yeah, so we haven't worked on the neutron side yet anything related to this. So the idea was to for now was simply to leverage the southbound model that we have and Use that at least initially in order to implement these containers so yeah, our idea was for instance to support VPN in in middle net right and in addition to this we thought of all these challenges related to high availability scalability and Giving the operators some control over where they run for instance this third-party VPN software We use Libre Swan right, but you could use Quara for instance for BGP or whatever So that is how we started with the idea of containers. That is let's wrap these application as containers They don't let's don't make them full VMs because they don't maybe they don't need to write They don't need full isolation So we'll have which the southbound malls that we had to implement this initial version at least Right and this is what we had so far and this is how we did it Of course maybe in the future this is these ideas are a bit separate from The containers work they're doing now in neutron So it's not the same container orchestration, but it's more related to let's say encapsulated containers Certain services, of course, it's very possible that this could be taken off frame You know implementing in neutron and adding support for it in neutron But it also for now at least in our version It depends a little bit on the kind of code that we have in order to orchestrate these containers that is schedule them Get status from them health checks and so on Hey Alex fantastic presentation. Thanks. It's a really neat capabilities you may have said this and I might have just missed it the Policies that you're able to enforce and the scheduling that you do. Are you leveraging any other? open-source schedulers or have you guys written on yourselves now for this one we implemented ourselves our own scheduler All right, so if there are any more questions. Oh Excuse me So I think I've seen this is the third session with a similar idea And so far no one's really thought of what the API or I haven't seen any ideas on what the API from neutron might look like Do you have any thoughts? Have you thought ahead about what you'd like to do? Would it be something you want to let the tenants do is it something you'd reserve for this site operator? What are you thinking about about how the tenant would engage that? So initially we thought this would be a at least an operator site because Probably for the for the tenants this would be the service They don't ask for container in this case. They ask for the service and the service is VPN or firewall or whatever For them the fact that behind there is a container that would provide them. Let's say 99 point something Availability it should be transparent right but for the operator. It's not so probably of course It'd be would be awesome in this case to have some neutron support for it. So In this case operation go to our CLI to manage them We haven't put any thought in in an API for this yet. As I said the project is pretty young and We it started from a need of Implementing VPN it it wasn't just driven by we want to have containers But of course we'd have to think about it I don't know if you explain this so that these containers provide the network service How does it interact with the middle net agent on the individual host there? You do the pack a simulation, right? I just want to know what's the What's the integration? So at that layer even it interacts with with the middle net agent in the same way of VM interacts the only difference is is that Because we have this richer lower layer models that we use we just Tell the agent to what kind of transformation it needs to apply in this case for instance the what we needed to tell the agent was that hey for Traffic got coming from the private network and going to the network to the peer network Just route it to the container and there we have the container we have Libre swan while running and Libre swan has been configured know that packets Going between these two networks they should be encrypted with this configuration and the configuration is the one you you set from neutron But otherwise from the agent side is just as a VM. So there is no difference there It is a transformation in the mall. That's important not The agent we haven't done any work on the agent to do some particular as the end operations Okay, so if there aren't any more questions. We're the time anyway. Thank you very much for attending this last session