 Hello, everybody. Good afternoon. Thank you for staying for the very, very last session. So you must be load balancing enthusiasts. Welcome to the deep dive into elastic load balancing using the Octavia Active Active Dipology. My name is Dean. I'm from IBM Research in Haifa, Israel. And with me here is Alan from our Cloud Dev Team in Northern San Jose. Now I'm going to, since this was advertised as a beginner session, I'm just making sure that everybody knows their terminology. People listening a few sessions ago to the heat in Elbaz, probably that should have been a prerequisite to our talk. But so essentially load balancing just to motivation and terminology. We have, I assume, I have a service on the cloud. And I'm using horizontal scaling to provide that service with a pool of backend servers. And that pool of backend servers, the members of that, each backend server is called a member. The entire thing is called a pool. That provides high availability. So essentially, if one server fails, we don't have a service failure. And it provides performance. So the load is shared between all these backend servers. Of course, the performance is elastic. So I can have a large pool if I have a lot of, if my service is very successful. And my service is not so successful. I have a few members in my pool. And that's sort of essential. From the user's perspective, they use a service IP or a VIP. They don't know which backend server actually answers and serves their content. And for that, of course, we need a component that splits the traffic across all these backend servers. And that component is the load balancer. So load balancer does distribution of every new connection over these backend servers. It avoids failed servers. So it doesn't forward to any server that failed. It avoids overloaded servers and distributes the load among those servers. So it provides a performance. It's important to note that the load balancing of the load balancing as a service is not the pool manager, OK? So it's completely independent. You do need a pool manager to manage your pool. Heat, Celine, or whatever, or what have you, this is actually going to add and remove your members. The load balancer will use whatever backend service you have and will report broken ones since it's aware of which servers fail. So for that, it has a health monitor and it has stat collector so it can perform all this availability and performance optimization. The last piece of terminology is the LB algorithm. So we need to load band something, number of connections, CPU, the current Octavia load balancing supports, things, at least number of connection, source AP, ground robbing, these are the types of algorithms that are supported right now. And another important term is affinity. Affinity means that packets that are similar to each other go to the same backend server. So obviously, we want all packets from the same TCP flow to go to the same backend server. But usually, we want more than that for performance purposes. For instance, if we do SSL termination, we would like all packets from the same source IP to go to the same backend to have abbreviated handshakes for the TLS and improve performance. Of course, we will allow them to the same HTTP sessions or same users to go to the same backend service, et cetera. Now we come to the load balancing as a service. So OpenStack has load balancing as a service. It has two versions of the load balancing API. I'm focused here on load balancing as a service version too, because version one doesn't exist for me. And so load balancing API defines these logical components. We have our load balancer, which is equivalent, in a sense, to the VIP. We have listeners, so we can have multiple protocols. We can have one listener listening on port AD, for HTTP traffic, another listener on another port for HTTPS traffic, and so on. And we have pools of members that actually implement the backend. We have policies associated with the listeners deciding which pool to go to as a default pool, or we can have layer seven policies to distribute to more pools. And these pools can actually be shared between listeners. And also we can have different load balancing algorithms for each pool. For each pool, also, we have health monitoring to check, so we can have different health monitoring. Of course, it depends on the protocol. The current API uses the LBAS prefix, and that's just the name of the logical component under your regular create delete list show, and so on. Octavia is not a load balancer. Octavia is a load balancing as a service. So it tries to provide an operator load balancer. The actual load balancer is HAProxy under the cover. And HAProxy is, of course, very powerful and useful. And what Octavia does is essentially it creates a VM for each load balancer instance, installs an HAProxy image there, configures that HAProxy, and uses that for the load balancing. This is all very pluggable. So Octavia calls that VM in on four. It doesn't have to be VM. It could be container. It could be something else. It doesn't have to be HAProxy. The current reference implementation is a VM with HAProxy running on it. And also, since Mitaka, we have active standby, so Octavia actually creates two VMs for each load balancer instance. One is the active load balancer, and one is the standby that takes over using VRP when the first one fails. Now, it's important to note that, as all this picture I showed, like the Octavia connected to all of these and four, but actually the data pass is separate. It's completely separate for each of them. The only thing that's common is the management network. So the black network on top is just a managed network. All the data pass actually goes to tenant networks. It's completely separate. Octavia itself under the cover. Of course, there are a lot more components. Lots of plugability. It's a very clean architecture. Everything is a driver. You could replace whatever you want. And we actually did that for this work. So if you don't like any part, just write your own driver and replace it and be happy. OK, so there's some four that can actually do much more than just plain old load balancing. We just don't just want to distribute traffic. Actually, HAProxy is really great. It can do a lot of things. It can not only load bands, which we expected to do, but it can do things like SSL termination, offload that task from the backend service. It can do things like there are seven policies and content switching. It could do cooking insertion for session affinity and stickiness. It could do compression. It could do all sorts of things. Since Mitaka, we now have Sanctis Steven L7 content switching. So Octavia is constantly adding support for all sorts of things. There's still things that are not supported like compression. And in principle, we don't have just to support what HAProxy does. We could actually put whatever we like there. So we have a framework that actually spawns a VM and put something, some image on there. So if you have your image doing more, doing caching, maybe doing more stateful firewall, maybe doing something else, doing some your complicated rewrite, re-encoding, whatever you like. This is, of course, not going to be supported by Octavia right now, but you could try to drive it to do that. And it's nice, because you could do all sorts of things that you can offload from your backend servers. Now, of course, the more you do, the more resources you use. OK, so let's take a step back and think, again, why we actually wanted a load balancer. So we said we wanted a load balancer to provide high availability. But now we have a single point of failure in the load balancer itself. So we went on and have active standby. So now we don't have that single point of failure. And the other point was performance. But what if my service is fantastic? I have lots of users coming in. I am adding and adding and adding backend servers. This is great. And then I found out that my load balancer cannot handle all that load, because that's a single VM. Maybe it cannot handle all of that stuff. OK. So this is a problem. Now, I have a very cool solution, active-active topology, which basically means that I'm using elastic load balancing and using a pool of them for to handle all my traffic. So I have a bunch of them for as a pool handling the load balancer duties. This is very nice. Now, the only component I'm missing is some component that will take my clients and actually distribute them over these on four. So I need to split that traffic. This is great. But sort of, I think I'm stuck in a recursive loop here, and I don't know the stop condition. So I'll stop. I'll come back to this point later. So from the abstract of this talk, our motivation was to cost-effectively provide load balancing as a service for cloud workloads. So basically, we have customers that expect to support their elastic workloads on the cloud. So they can be very small workloads, and they'd be very large workloads, and they wanted at a good price. So what a good price means is that the small guys, they expect it to get it for free. And the big guys, they're willing to pay, but they expect it to, however huge their workload is, that it will still support it. So this means that from a load balancing as a service perspective, we must use as little resources as we can when we have a single load balancer and a very small load balancer, but when we have a big load balancer, we need to be able to add resources and add resources and make this happen. Now the existing Octavia implementation doesn't address that problem, because essentially there's one VM doing the work for each load balancer instance. So that could be too small for huge workloads. And for small workloads, that's not cost-effective. We could solve it in different ways. Octavia is working on moving the load balancer, maybe have an implementation on containers, which is more cost-effective. But still, we need to handle the huge loads. Also, with the active standby mode, we have one VM standing idle. So we are at 50% utilization anyway. OK, so this is why we introduce the active, active, and plus one topology. This is in the final stages of the blueprint. When I pitched this talk, I was hoping that it's already there. Apparently, people are working hard on Mitaka and are slower. So I'm new to this open source stuff. I couldn't get it through, but I'm working on it. So what we have in this is NO4, as all of them active, acting as a pool so they can handle a large workload. And essentially what we do is we split the traffic in two stages. The first stage is what we call a distributor. It's a simple stage that splits the traffic to all them four. And then we have a second stage that does the regular load balancing, the advanced modern load balancing between them four and the backend service, all this session affinity, SSL termination, and so on. The plus one is a standby M4, very similar to the standby in active standby mode. So this standby is just ready to replace a failed M4. When a M4 fails, the traffic just goes to that standby. The failed M4, it gets recycled, it gets restarted again, becomes the new standby for the whole thing. And this obviously has better utilization. And we can extend this to N plus K, although not currently supported by our code. OK, a few words about the distributor. So you should think about the distributor as not as a load balancer per se, but more like a gateway router. So essentially, it should have the same high availability attributes as a gateway. So really, all the load, all the workloads, all the traffic goes through it, but it doesn't do much. Seems like hardware is a good match. Seems like native Neutron would be a good match in the future. And it's not so smart. So it really is a multiplexer for IP connection. It just needs to balance something not very accurately. All the hard work is being done by M4. So it's more like ECMP. It could be ECMP, actually. The only thing that we must keep is the affinity. So we must make sure that TCP flows go to the same M4 just because we want to have SSL termination at M4. And M4 is a real proxy. It terminates the TCP connection. There's no reason why this distributor cannot be shared. So in terms of cost effectiveness for the small guys, it could be shared. For the big guys, we can have dedicated distributors. SSL termination is only happening at the M4 level. So there's no real security concern there. But if you are fanatic, you can have your own distributor. It is multi-tenant. Again, think of a gateway route. And if you have your global load balancing already in place and you use DNS for your global load balancing, you could potentially just plug that in and use directly. This is not what we do right now, but potentially you could just do distribution by DNS. Our implementation of the distributor is an SDN software distributor. What we do is a really simplified one-arm direct routing topology. So that means that the distributor is co-located on the front-end network together with the M4. And the M4 do not advertise their VIP. All the traffic goes to the distributor, and the distributor does essentially L24-ing to the right M4. So it just replaces the Mac of itself with the Mac of the M4. And it's also direct server return, meaning that the heavy traffic actually doesn't go through the distributor, goes directly through the gateway to the clients. So this means the distributor is pretty lightweight. It doesn't do a lot. We implemented this distributor as open-flow rules. We wanted to be very generic and wanted to just do SDN, pure SDN so we can plug in either software or hardware. We are actually now experimenting with hardware-based open-flow rules and using hardware to implement these open-flow rules. And essentially, we're using version 1.5 of the groups in open-flow to select based on the hash of source IP and option of the source port. The implementation, the reference implementation currently is an OVS VM. It can be any open-flow switch, as I said. It is multi-tenant. And we didn't add high availability yet. But it's essentially using exactly the same code that spawns now and for. So we are going to add the VRP stuff and the HA to that. If you are more interested in the specific rules, this is how the rules look for the forwarding. Very simple. What you have in yellow is your hashing selection method. What you have in light blue is the fields we are selecting on. So we're selecting on IP source. And in this case, also TCP source port. And when you do a selection in a group, it selects a bucket. And each bucket has different actions. In this case, all of them have similar actions. They just replace. They plug in a specific MAC address into the destination internet. And just push it back into the network. Now let's go to elastic load balancing. This is the next step. We have it implemented, but really used sort of a proof of concept, not ready code. We used an afford pool that is managed by heat. You could manage it by any other pool manager. You could use send in. You could implement your own. We are actually working now on re-implementing this using the Octavia way, using a cluster manager driver. So it will be a pluggable, and you could replace that whatever you like. And essentially, the features that we need, we don't need all the features of heat. We don't need all the features of send in. We need something fairly simple. And it's also attractive in the sense that you don't want to be dependent on another service. Obviously, if you have a back-end pool and you are managing that back-end pool by send in or by heat anyway, so the service exists there, so you might as well use it. But we are trying to make Octavia self-contained, so we are trying to have at least one implementation that does not depend on anything else. Also, the implementation itself uses the pluggability of Octavia. So we just modified the compute driver instead of using nova drivers directly to create compute nodes, to create the M4 nodes. We actually call heat to do that for us, and it comes back with the nodes. Heat creates the initial population of these M4. It deletes failed M4 and recreates them. It removes the M4, overloading the windows, overload, underload, and it does all of that. We use Cilometer just because it was the easiest to plug into heat to monitor these M4 and tell us when they are overloaded and tell us when we actually need to add and remove stuff. All the load-balancing work is still and all the control of the load-balancing as a service is still done by the Octavia control. So that means Octavia still, whenever heat spawns in UM4, Octavia still goes on and configures that HAProxy does all the actual work. And Octavia actually still does all the health monitoring that it does now. So it knows if UM4 is alive. It knows the stats. It knows if it's overloaded and so on. So Octavia knows all of that stuff. So it really does question, do we need Cilometer at all in our scenario? But since Octavia is already monitoring all of that stuff, maybe it can just fire these alarms webhooks directly. And of course, if we have an internal implementation of Glaster Manager, we don't need to go through Cilometer. It doesn't make sense. The existing health monitor does not have out-of-scale actions, but it's easily added because the monitoring is already there. Finally, we added code to make sure that when you add or remove an UM4, we are going to add and remove forwarding rules to the distributor. So the distributor has to use this in UM4. It's not like VRP, where you just take over single IP. You have to tell the distributor, and it needs to add it to the group. Now, the tricky part there is handling the affinity. So I'm not going to go into details in this. Maybe at the end, if we'll have enough time. But there's a big difference between the N plus 1 and the out-of-scale. So the N plus 1, there's no problem of affinity. If we use hash, the plus 1 just takes over an existing one. There's no problem of reshuffling. If you out-of-scale and you increase and decrease your cluster, you have a problem. Both when you go down and you're going to go up. You can do things like consistent hashing to make that better. But you still need to handle this. And the way we handle it right now is just do connection tracking to make sure that existing TCP connections don't break when you do out-of-scale up and out-of-scale down. And with that, I'll hand over to Alan. Hi, everyone. So maybe let me just talk a little bit about IBM Cloud. And so as we learned, let's say, the IBM Cloud was actually built on open technologies just to ensure interoperability and also flexibility. And also, let's say, from the session that I learned at the beginning of this week, so IBM actually started to contribute to the open source community since 1998. That was pretty early. I didn't know that. And actually, let's say, those are cool. Open technologies are becoming the core technology in IBM Cloud. For example, containers, Cloud Foundry, OpenStack, obviously. And for IBM Cloud, a bunch of Cloud offerings are really running OpenStack operating system. And there are a couple of examples. A couple of years ago, we released OpenStack services, which is a private Cloud hosting on a software data center, WorkWild. And we have container services. We have compute services. Now they are running OpenStack environment. And talking about OpenStack, so by the way, I'm new to OpenStack. But I think I love this community, especially this is my first summit. I love all of those sessions here. I think I will try to get more and more plug into this community. And talking about OpenStack itself, so basically, IBM is one of top contributors into the community. So that's why, as I learned, I also probably know you guys have heard from one of IBM sessions that we're going to really plug into OpenStack deeply and trying to shape this project well to hope this particular project will be flexible and that we can bring in more talents from the community. And this is kind of my personal understanding about a cloud. So basically, as a cloud user, what is really essential kind of a component in the cloud? So I would say this is workloads. Because my career started with applications. So I wanted to develop application and turn those applications as a service. And they'll be happily running in summer. Now cloud is coming to the picture. We're really running the services in the cloud. And there's a workload. Now there are a couple of aspects really to a certain application developer. I wanted to have some features really supported by the cloud, for example, as a high availability. I wanted to have my application really as a service. It is never, I mean, it's just a never, but at least have a minimal service outage. So I have some level of availability. That's number one. Second, let's say when my end user of my application getting increased, and certainly, let's say now, I mean, the data traffic will increase. I wanted to solve the performance issue. So hopefully, let's say the cloud actually can help us to solve the performance, to scale up all those instances to handle more and more traffic. And what we learned from Dean's explanation earlier, we need a low balancer, just for these two obvious reasons, high availability and performance. And next, let's say for low balancer itself, we don't want a low balancer to be a single point of failure. And that is so obvious. So that's why we need to have low balancer HA. And to look at the existing activity infrastructure, we have an active standby. Now, active standby probably is not good enough to handle more and more workloads. So we wanted to have the ability to automate more and more workloads, so we wanted to have an active, active HA model. And lastly, with that active and active HA model in the picture, so we can possibly just enable per paper use, those kind of features to be cost effective. So next, let's walk through the demo, the work that we have put together. It is not a final implementation, just for demonstration purpose. But the work is very fun. And I will say Octavia has a very strong architecture, allowing us to really make some changes to have that active, active HA enabled. And I have a web link down there. That's the YouTube link. If you're interested, you can go through that web click. It is about, let's say, 13 minutes. It's pretty interesting. So before that, let's just walk through what kind of demo scenario that we have. So let's pretend that, let's say, we have two small businesses wanted to open up, let's say, two e-flower shops. One is only wanted to sell ref flowers. And the other one is actually wanted to sell the blue flowers. In each shop, we'll sell different kind of red flowers and different kind of blue flowers. And then we will have each kind of flowers hosted in one of the workload or VMs. If you look at the right-hand side, if you focus on the red one, which will actually go through a lot of web page hitting the red flowers. And the right-hand side, there are three instances with a different IP address, dot 3, dot 4, dot 5. And those are kind of displaying different flowers. And obviously, that will be red. And each and every flower shop will actually return the backend IP onto the web page so that we exactly know which type of flower and what kind of IP address it is hosting in that workload. And we use, let's say, heat plus salameter to auto-scale a cluster of the m4r VMs or the cluster of load balancers. If you see, let's say, focus on the left-hand side, basically the starting point is with only two m4rs because we wanted to have the basic HA that's active-active. We just have two instances. And when the load coming to that cluster is getting increased, and salameter will be filed, and the heat will actually just scale out that particular instance and join that cluster. And on top of that, left-hand side, square of the red m4r, you see a VIP. That's a 20.0.0.12. And that is the virtual IP for the cluster of load balancers. And the next thing that we customize is that because we wanted to make sure when you access that VIP, we got to make sure which m4r VM we are really hitting. So that's why we kind of, let's say, tweak the HA proxy configuration a little bit just to inject the m4r ID into it. So when we actually hitting that website, we're getting which m4r ID we're hitting back. This is only for demo purpose. It just wanted to show that, let's say, we're hitting a different instance at the back end. That's the workloads. That's the red shops over there. And also, we're hitting different m4r VMs. OK? So this is essentially the web page. I mean, it's a small web page for these two eFlower shops. And just for the lack of time, we just focus on the red shop for now. And there are a couple of items are interesting. I wanted to point it out. First, let's say, on the top, on the address bar, you can notice this is the virtual IP. And that's 20.0.0.12 that we just saw. And the next thing we wanted to want you guys to focus is on the top of that image, you have a backend IP, 10.0.0.3. And this is the m4r ID I was referring to. So when you hit that cluster m4r, and this particular m4r ID will be played back on the web page. So we know from this web page, all those three key information are visible to us. So next, we refresh, send another request to that virtual IP. And we will see several changes on the web page. So first, obviously, that red flag got changed. And we are hitting different back end, the different workload. And that workload is running in a different IP address. So previously, that was 10.0.0.3. Now it is changed to .4. At the bottom, because we have two m4ras. So within the cluster, there are two instances running, active, active again. And the m4r ID gets changed. Next, the one thing is not changed, certainly is the VIP. Sorry, I forgot to mention that. Now next, let's say, let's do this again. We send another request. So another customer probably not coming in. And then you will see, we use the same VIP on the top. And something changed on the back end IP. And the flower got changed. And the m4r ID stayed the same, just because the hash we used is based on the source IP and port number. And sometimes the hash doesn't actually hit the same. Sometimes it hit the same m4r ID. It's not guaranteed. So this is kind of, let's say, wanted to just unfold what we created using Octavia. We created two different low balancers, one for a red eFlower shop, the other one is for eBlue. And then you have different virtual IPs. And this page gives you kind of an admin view of the networking topology. When I say admin, because you don't see the back end workloads here. You don't see all the eFlower shops anywhere here. That's because those are in tenant networks. I mean, those are in the tenant views. And this admin view got to give us some very interesting information. First, let's go through all the networks that we have. And obviously, this is not the latest in the Mitaka release because our work was done actually between Liberty and Mitaka. So during that time, let's say, we were just pick up the latest stable Liberty. So that's why this particular screen may not be refracting to the latest Horizon GUI. So first, we have management network. On the management, that's the same management network for Octavia. And the second network is actually for red shop. That's the neutral network for that red shop. And certainly, let's say the color is a little confusing here. It is painted as green. But actually, for our demo purpose, it is really for the red shop. For now, let's pretend we are color blind. So the middle one is actually not green. It's red. And we have on the far right, that's for blue shop. And let's pretend we are color blind again. That's blue. And this is kind of the magic that we have here for active-active. That's the distributor. So the distributor is kind of the one actually with the truck in a two different and for a cluster. And two soft, let's say, for red and for blue. And that's why you can see the distributor is the special one getting plugged into three different networks. Management network, red shop network, and the blue shop network. And the instances on the top that you can see, those are the infrared instances. And for red shop, you have a cluster. For blue shop, you have a cluster too. Next, let's say, since we are using heat, and certainly we have a heat template to keep tracking all the infrared cluster. And for heat, if you're familiar with that, there are a lot of resources under that heat template. And there are some key stuff I wanted to just point it out very quickly. One is you have a heat auto-scaling group. And this is for infrared cluster. And you have two salameter alarms. One is for scale up. One is for scale down. And correspondingly, you have two scaling policies. Now let's switch it to a different view. And this is kind of giving you a very nice diagram, or you can say this is a topology. Now let's switch to a table view. Again, this is a heat resource list. Essentially, you have that auto-scaling group. This is maintaining a cluster of infrared. You have for the scale up alarm. When the infrared VMs are kind of getting stressed, this alarm will be filed. And then the scale up policy will be kicking in to give you another infrared instance. So on and so forth. And then scale down alarm is there. Correspondingly, you have a scale down policy. That's all you have for the heat template resources. So next let's take a closer look at what's in there for salameter alarm. So for people that are familiar with all the salameter properties, probably on the right hand side is the most important thing. So we have alarm, which is actually based on AVG, means that's average. And you kind of have a type of threshold alarm. And the threshold that we set is 40%. And the period we wanted to give is 120 seconds, which is two minutes. And one thing you don't forget to highlight is the evaluation period. We set to 1. And the state means, let's say, salameter basically can bouncing back with three different states. Unknown is actually, you don't have sufficient data. OK, you have sufficient data, but the alarm does not actually meet all the conditions. Then the last one is alarm, meaning, let's say, condition is met. You set the alarm. When the alarm is fired, then the alarm action that you are out will be invoked. So for the folks, actually, don't quite get the salameter alarm yet. That's OK. I give you this particular statement. So this is just alarm fires when the average usage of the CPU consumption greater than 40% over two minutes. That's kind of a quick description about this particular alarm. And the same thing, let's say, for scale down alarm, we have defined as an average static alarm. And the threshold is set to 10%. So you have a period. It's 120. And the statement is that you have the alarm fired when the average of CPU utilization less than 10% over two minutes. So next, let's say, we really start the stress. We have a utility, and that utility is just sending all the ping requests aggressively. And this is kind of the page that we show here to track the status of the salameter alarm. So you can see that the CPU utilization is getting increased over 40% as we defined in the salameter property. And once that salameter gets fired and for a VM will be added to the cluster, and obviously that's by key engine. So next, let's say we take that stress away. Let it just cool down. Then the salameter will track those CPU utilization getting dropped. It is below 10% within two minutes. And then existing in for a VM will be removed from the cluster by key engine as well. I mean, this is kind of very straightforward. Now, this particular chart is very interesting that we really capture this chart. We run the test for a couple of hours, and we see where this is really working. And essentially, let's say this chart proves it is working. And you can see on the line, that particular line is just explaining, let's say, you sending the load aggressively, increase the load, and decrease the load. And the green one is actually a track of how many and very VMs that you have. And initially, starting from 2, we wanted to have active active, just to instance as the minimal. Then when the load increase, you would see all the number of infrared VMs getting increased as well. And the load gets down. And all the infrared VMs cool down. And they essentially got destroyed by key engine. And this is another interesting chart that we wanted to see with that particular approach. And pretty much the balancing is at two levels. At the back end, since we're using the round robin algorithm for the load balancer, the first, let's say, script, it is actually just capturing. When you hit that back in server, so the load is equally distributed. And the second script is actually just to capture how often we're hitting each channel in front of VM. It turns out it's almost, let's say, equal or given. That pretty much concludes the demo. And if you're interested, go to that YouTube link. And it should be very interesting. Now I hand it over back to Dean. I think we ran out of time for a few seconds. So I'm not going to go into containers, unfortunately. So I just wanted to just mention in one sentence that there is work being done on activity to move to containers. And this helps in terms of cost efficiency. And Active Active actually allows you to use containers because the default configuration could be a lot of containers. And so it gives you better spread and better availability. And maybe even saves you some of the stuff in terms of the availability just because containers are faster. This is really the end. So I'm open for questions. If you do have any remarks that you don't like this design, this is your last chance to go over to these blueprints. Don't do it. But so that's really your last chance. And I really hope we get this thing done. Yes, please. So my question is practically you told that these mobile answers are practically VMs. These are glance images, right? So what I'd prefer and those are instantiated from like a novel instance, is it? Yes, right now. So Octavia Code distribution actually creates those images for you. It goes on and fetches and creates and builds the images for you for the basic implementation. But essentially you could do whatever you like. You could put whatever you like there. We are thinking that there's some sort of maybe being able to specify flavors so you can maybe define these two flavors, different typologies and different types of form for it. So the kind of agents that you have, what has to be running inside of the VM that is configuring the HAProxy or the correct load balancer is just provided by you when you are provisioning that kind of machine? This is a separate component of Octavia. So Octavia has a controller, spawns these VMs per load balancer. The load balancer itself, what Octavia now spawns, it spawns a generic thing. And it happens to be an image of HAProxy and it goes on and configures it. But if you replace that driver, you can do something else if you want to. But if you just download Octavia, it will do all the things for you. So it has all the scripts. So it's not packaging HAProxy inside, I believe, but it's actually fetching that image and creating it for you. So if you just take the code, you will get all the things you need to have an implementation running. So I had a question I saw in the slides. You said there's demo code. Is this being developed with the OpenStack community process? Is it available in Garrett today? You should know the answer for that. So actually, as I said, I'm new to this and I was trying to push code and then I pushed a little bit of code and people said, no, go away. You didn't put a blueprint yet, so you can't push any code. So I have all this code waiting and I said, oh, blueprint, no problem. People would just say yes and they have no objections and just say yes and it didn't happen. So this code is purely developed for OpenStack. It was developed for OpenStack before we knew actually what will do it was an IBM and probably will not consume it by IBM if it doesn't go to OpenStack. So we're totally focused on that. So I'm research, I don't care, but he's deaf and he actually has a team and they're working on actually coding stuff and having live items and so on. I'm happy just coming with the design and say, oh, it's possible and I go away. But no, I'm not going away. We actually committed to this code. So this was just a joke. Yeah. Just to clarify for myself, as of today's Mitaka, Octavia, does it support as a cell termination, right? It does. Yeah, and where does it store the certificates? In Barbican. In Barbican. So that is supported already. Yeah, yeah. This is already supported. I think actually supported from Liberty. I'm not sure it completely worked out, but yeah, it works from Liberty. Yeah. Thank you. Yeah. When you scale down, does it drain connections off the load balancer before terminating the VM? So yeah, this is a valid point. We don't have that code in yet. We actually don't have any code to drain connections even when you just do an upgrade to and for and things like that. So yeah, that's a missing piece that we need to add. Thanks. Any more questions? Well, thank you for staying late.