 All right. Good afternoon everybody. I hope you enjoyed your lunch and To help with your digestion. We are talking about today about neutron and exactly we are going to talk How better it got in the past year? So I am Salvatore the guy over there the young guy over there is Aaron. He's not married if you want And we both work with VMware as you know VMware is very evil We want to sell you a lot of proprietary stuff and for this reason today We are going to talk to you about how good the open source bits got into open-stuck neutron allegedly so What happened in the past year so a lot of code in past 12 months a lot of code come and went we had like more than Probably 1500 commits a lot of lines of code were removed We added twice more lines of code and that and after all that crap Did we manage to make it just a little tiny better? maybe so in order to Realize whether we actually made it better or not I could talk here for two hours Maybe even three about all the changes that we made but without any numbers It doesn't make sense. This is the same thing when somebody comes to you and say Neutron doesn't scale. What the hell do you mean by scale? So I mean by scale you can mean a lot of stuff for this reason we made a set of an experimental campaign where we want to measure exactly how The agent and spawn specifically for this talk the layer-to-agent scales in its interactions with the plug-in the server side over the message queue bus and When the load imposed by the server increases we measure the load in terms of a number of Interfaces that have to be Concurrently wired so for instance there we have you know the agent when it wires one interface at a time is Under a light load when it wires 10 interface at the same time the load is a bit tiger So we measure the load both in the layer-to-agent and the DHCP agent Basically the layer-to-agent you know for configuring an interface has to do all the wiring like applying the appropriate configuration applying security groups and for the DHCP agent has to ensure that the Mac IP Association is added to the proper DNS mask configuration So the methodology of our analysis is a rather simple We increase the number of concurrent server being booted from 1 to 20 We didn't step of two. I mean we didn't step of ones But here we are just reporting a part of the results and in order to make sure that we did not get any outliers We repeated every experiment 20 times. So for instance the last experimental set we booted concurrently 20 VMs for 20 times with an overall number of VMs 400 VMs And then for each metric that we defined we studied the mean the variance and a Midian so but which are the metrics that we Told were meaningful. So the metrics are three for servers three for instances and three for ports Valuable metric for an instance at the time that the instance needs to go into active state So, you know when you boot an instance in OpenStack, you have a various states Then the instance goes into spawning and finally it goes into active and since there is a networking component In the time to go to active we wanted to measure what that networking component time is and exactly that's t allocate net T allocate net. It's the time that Nova spends chatting with neutron for configuring networking for an instance and finally the last parameter is the time needed to ping an instance, so it's the time from when I do Nova boot to when I'm actually able to ping Floating IP associated with that instance and of course there are a lot of components involved here There is a component due to spawning the instance There is the other compute component for associating the floating IP and finally there is another component for actually waiting The instance that goes through the boot process and as networking ready This is for instance if we go a little bit more into detail and look at the port Matrix we have three time three three matrix which are really specific to neutron components The first is time processing time to process to time to wait for processing an instance Which means how much time that interface is sitting there on the integration bridge or an instance on a story an integration bridge on host before the layer-to-agent even considers the idea of Wiring that port then the time to bring an interface up Which means how may how much time goes from the moment where the vif is plugged from Nova in the integration bridge Until the neutron layer-to-agent takes that vif processes it wires it and therefore That bridge is read that sorry that bridge that interface is ready to communicate and finally It's time the time DHCP, which is the time that is needed to Notify the DHCP agent about the new interface and Register the new interface in DNS mask as I say they're there on registering new interface is pretty much adding a new entry to the host file in DNS mask and Let's go for results straight to results So what you see here in the diagram is the difference in the time to go up from Havana to Juneau the red line is Havana the blue line is Juneau thankfully and As you can see there are also I also added the the bars for the standard deviations Which are the pink ones for Havana and the light blue ones for Juneau So what we can say is that well, it's a lot better and those are not lies. Those are true results and Yeah, because you might say this guy is lying This guy just want to tell us that neutron is a better I actually know that neutron is just a huge pile of rubbish It might be a huge pile of rubbish, but it kind of works now. So Kind of so What you can see is that from Havana the Havana one now now it's a year is gone Havana It's not supported anymore. We can actually say oh man. That was really really bad I mean when you went up to 10 is 10 concurrent instances it could really not wire anymore properly because the Time to wire an interface started to grow up in a quadratic way Which is really not scalable if you have like a high-density server where you can have like hundreds of VMs on a single server You can't really run the OBS agent with this But we do know we have a linear trend with a slope which is not so bad For instance with 20 concurrent instances, we are still under 30 seconds for wiring all the instances So, I mean the means that the instance it takes more time to wire wire like in 25 seconds Which is probably not great But it's still acceptable if you consider that you have 20 instances also This test have been run on machines Which have just four cores and eight gigabytes of RAM not because we don't have machines I mean VMware can pay for us for giving us good machines for doing this test We just wanted to do something which could also be compared Against the test that we run in the gate just for giving you a credible reference The second measure is the time to allocate a network So these measures are actually taken not in Nova in neutron But in Nova. They represent the time that Nova spends talking with Neutron for doing all the API calls for Allocating a network. So this is where you know my friend Aaron here did a very good job He mostly did all the patches that allowed us to bring this time down and you know and the interesting thing is that the trend the growth trend of the of The line for Ravana for you for Juno is just less than a sixth of the growth trend that it was for Havana, so the scalability is a lot better. I put in this diagram also the median for a for a consideration that The difference between the mean and the median in Havana was huge while you know They are pretty much the same in Juno. The difference here is ML to ML to add in Havana bad database locking problems and The median the the mean is really skewed because of some outlayers Which were some particular requests which were timing out and starving because of these locking issues in ML to but all these problems have been now solved in ML to and the the error the mean and the median are pretty much the same thing Finally, I have one last diagram for you This time is a bar diagram and the buyer diagram reports failures Which may be either VMs going in error or doing the boot process or VMs timing out so never able to being reachable from ping and the red the red bars are from Havana and you can see we can see quite a lot of failures for Havana like Over a quarter of instances failure failed when we run 20 concurrent instances And the blue bars are the failures that we observed in Juno Aron, where are the blue bars? I'm out there. Yeah Well, we actually saw absolutely no failures in our test when booting instances concurrently with Juno Which is a good result actually the problem that led to all these timeout failures in in Havana is a Very simple problem that since Nova and Neutron were not coordinated Basically any Nova will start booting an instance and the network was not yet ready What happened is that you know the instance then tries to get an IP the network is not ready This and say well, I don't care about it anymore I don't want an IP anymore and it's not reachable So this is something that has been fixed by a running cooperation with dance meet from the Nova core team And now we have not just Neutron which scales a lot of better than what used to scale in Havana, but it's also an awful lot more reliable Still, I mean, I agree. It's far from being perfect, but at least you can stop throwing Throwing stones at it. Maybe probably I'm sure there are a lot of things that still don't work and Yeah, so let's analyze a little bit the progress together so I Think when we started with four some, you know We didn't even have an engine probably a proper engine to Neutron It was more of a pedal thing. Then yeah grizzly started to work somehow Havana was probably just good to sell fruit or to the young to go around with the three wheels Ice house. I think ice house was really reliable, but probably slow instead, you know with Juno We got a nice and comfortable reliable Toyota Camry. It's not fast It's not shiny, but at least, you know, it's reliable like you can do like 100,000 kilometers or even 200,000 kilometers with it and So this is pretty much a good analogy of think of the evolution that we need for the open-source components that are running within Neutron so before So this is just a summary of the improvements that we made. I think we pretty much already discussed all of them Before we know going on I just want to mention that We in this analysis, we just looked at the layer two agent. We've done analysis also for improvements in the layer three agents But unfortunately we had to select what to talk about in this presentation but even the L3 agent we Did a lot of improvements which actually brought down a lot the time needed for synchronizing the Status of the L3 agent with the status of the server for wiring floating a piece and so on. Oh One last diagram This is the time for bringing an instant field blue line is the time for bringing an instance into the active state in Juno And the red line is Avana So we have a little problem here The blue line is higher than the Avana line So now Juno is a slower than Avana, so well we made it worse and it's all is fault And now because it's is fault. It's is task to talk and explain you why this happening Come on go go cool so as Subletory was pointing out one of the components that we added in ice house actually was this event reporting system between Nova and Neutron, so now Nova Neutron now talk to each other or Neutron now talks to Nova So if you use horizon, I'm sure everyone's seen this problem where you associate a floating IP with your instance And then the floating IP doesn't actually show up on the horizon UI So the reason why this happens is Nova returns cache to IPM information about the instance So it doesn't automatically get updated when you associate that floating IP and the way that it actually updates That information is there's a periodic keel script that runs on all of the compute nodes And the amount of time it takes to actually heal the cache Unfortunately is based on the number of instances that are running on that server So if you have a hundred instances on a server, it'll take a really really long time for that cache to get healed So typically what you'll see is someone who associates a floating IP with the Neutron API Then they'll go ahead and query the Nova API And then they'll be kind of confused why the floating IP doesn't propagate all the way to Nova immediately So in order to remedy this problem Whenever a floating IP is associated with Neutron, Neutron then informs Nova To heal its cache So what would happen is someone goes ahead and associates a floating IP to Neutron Neutron then tells Nova API API, hey, this instance Changed then that gets dispatched to the compute node and the compute node then goes ahead and heals the cache As you can see in the system, we're kind of dispatching the event all the way to the compute node So one of the things that we might try out is actually pushing this logic a little bit higher into Nova API So we can do this quicker Another so when you query Nova API now now you see the floating IP and it's done a lot quicker than it used to Another problem that Salvatore was pointing out is in the gate We would see oftentimes where you would boot a server but that or boot an instance But the network yet isn't ready for that instance and unfortunately some of the DHCP clients and the guest Will attempt a couple times and then give up so what would happen is you would go ahead and boot an instance You would query Nova it would say it was ready But then you would actually try an SSH into the instance and nothing would work the network wasn't already ready for it So we leveraged the same event framework that we used to update the cache in order to solve this problem So now the workflow internally what happens is a user goes ahead and talks to Nova API to boot an instance That gets pushed to the scheduler the scheduler then Dispatches that to one of your compute nodes and the instance is started in pause state And the network is actually attached to the instance. So then what happens is neutron goes ahead and wires the network Whenever that happens and then the back end to neutron will then notify The neutron API that the port is updated is now an active status. So whenever the database status changes from Like down to active then neutron API then tells Nova API that the port is now plugged and then Nova compute will unpause the VM. So the VM will come up with active status with a working network These are controlled via these settings Some plug-in implementations in the back end aren't able to expose an operational status or it might take a really really long time for The port to get wired So these are the flags that you can use to tune the timeout or you can even disable it if you want and this actual event System that's used in Nova is actually going to be used for Cinder and several other things so things don't have to pull as much they can get notifications when something changes And in fact Congress is actually looking at using the same system And if anyone's interested in hearing more about that there's actually a design summit at 440 in room 124 So I don't you ready to voucher these things actually works Yeah, you ready to swear it It actually works because indeed it has actually been running in the gate for already six months So it's been running in the gate. It's been running in a few production deployments So these event reporting system is apparently apparently a bit reliable I have to say we have to fix a few bugs into it for ice house So we can say that now for Juno is really like a hundred percent. I'm not sorry not a hundred percent Like 90 90 something percent reliable. So and beyond doing this this increased the lot the Reliability of neutron when it comes to scalability. We had to work on the layer to agent the layer to agent Is the problem that it was terribly slow in Wiring interfaces wiring an interface is a process that involves Detecting the interface on the integration bridge it involves associating an appropriate Configuration to that interface on the bridge so that it gets into the same broadcast domain Which is for that the tenant network to which the interface belongs belongs and then applying Security groups to it security groups in the case of the layer to agents are applied to IP tables So it means just applying the relevant IP tables rules So the problem is that there was an inefficence in the interface with the server was not really efficient Because for fetching the interface detail we fetch details one by one So if you had like a bust of 20 new interfaces, we made 20 calls to the server At the time of Avana the server RPC server on the neutron server was not even Multitraded or did not allow to have multiple workers. So this became a serial process So we optimized it one by allowing multiple Workers on the RPC API interface and two by bulking the request for retrieving interface details So if you had the six new interface to wire We asked for in a single API call for the details for six interfaces Then there was another behavior really annoying that the security group calls were preemptive So I was processing an instance it came a burst of security group update requests and that burst of security update request Starved the whole process for wiring instances Since the ML 2 plug-in in Avana was also sending a update to request like Crazies, so he was sending an update port to request even if the port was not updated at all so this caused it like a burst of Of requests that the agent added to process and it preempted in preempted Normal interface wiring then another problem that we had is that even if we have obvious DB Which is a very smart thing and can tell you exactly what's changing on the bridge. Nevertheless, we kept pulling it like, you know Like ships, you know, you have something very smart that you can use it and we kept using the dumb thing and also we also did the kind of superficial analysis of Devices to process you can Read there the software was so dumb that kept wiring the same interface over and over and over and over It just in some case not always So we fixed a lot of things a few things were just bugs Other things required a bit of rearchitecting of the L2 agent for instance instead of having the L2 agent add like These mechanisms were security group calls were handled in a different queue than Well, we're not handle at all in the queue were handled just different thread in different threads from the main Port wine in course now. We have like an event loop An event queue everything is added to this event queue regardless where from where it's coming from so that it gives you like predictable Processing times and you in theory. We are not doing that yet, but in theory you will also now have multiple servers processing Elements from the queue so that you can actually do parallel processing in a safe way and well, we fixed security updates for security groups and Most importantly we started leveraging the OBS DB monitor the OBS DB monitor tells us exactly when there has been an event on the OBS bridge, so we just if there is no new interface wired We don't keep pulling it for no reason. We just start processing interfaces when they're actually available on the OBS bridge and And another thing that was really important for using neutron at scale is that security group Where not really working beyond a certain sky scale mostly because the payload the for The messages exchanged by the server and the layer to agent growth add an exponential growth so this was really a problem in the design of the communication protocol and This was fixed by Miguel angle. Are you in the room miss girl? Are you anywhere in this room? Disappointed I expected him to be in the room. So Miguel did this Analysis on his blog and this is the difference with the new RPC Which is the blue one compared with the old one, which is the red one when the number of port increases you can see that this is just with the Security a security group with the basic rules like you know the default security group and you can see that with the old RPC The sides of an RPC message went easily like half a megabyte and the time for processing For processing an RPC Like likewise at an exponential growth So now you can imagine this is just for 40 ports Imagine something like the Nova default security group, which is applied to all basically In most cases all the ports in your deployments, which can be easily thousands of ports Imagine the time that will be required in the case for processing these messages Thankfully, this has been sorted as well in June. Oh, so we are much happy about that And if you have want more detail about this particular announcement you can That's that's the link to Miguel's blog where it describes everything in detail. So Let's move away from the alert to agent and let's discuss a bit the improvement in the layer tree agent So one problem that we had there that it take quite a lot of time to process in the router So if you created a router added the router interface It might take a few seconds quite a few seconds before you could actually use that router interface If you created the floating IP you got your floating IP back But that floating IP was not yet working because it was not yet wired So we changed a little bit the logic for router synchronization tasks Using the same principle that we did for the layer layer to agent using a priority queue in this case We are already processing elements in separate threads. So you have all the changes for All the messages for router updated are added to an event processing queue the other side of the queue There are like different worker threads and every worker threads picks and pulls an event from this queue And this has greatly reduced it Router processing times another thing that has helped Processing routers time is fully leveraging the concept of IP name spaces IP name spaces are fully isolated and therefore also the IP table stack on each Name space is isolated. Therefore when you are applying IP tables In the name space, you don't need a global lock You just need a lock relative to the name space where you're operating this little change has allowed us to apply concurrently IP tables to different name spaces So a lot of operation which were previously serialized can now be executed in parallel Have you used the like I think you view most of you have used like a neutron with avana, right? So when you created the floating IP with avana there was a little problem You create a floating IP you get an object in response and that object contains the IP address of your floating IP But you have no idea that floating IP is working is not working simply because it was missing an indication of its operational status whether Indication that will tell you whether the back-end be it the L3 agent or whatever back-end you're running as actually wired a floating IP so we added the concept of operational status for floating IP as well What happens is that when we create a floating agent is initially in status down because obviously we assume that's not being wired then a Router updated message is sent to the L3 agent This routed up in updated message contains the information of the new floating IP When the floating IP is wired and wiring a floating IP means basically just you add the IP address information to the gateway port and then you add the appropriate the NAT and a source NAT rules when all of this happens the layer 3 agent Signals back the server that the floating IP has been wired and its status goes to active The dual process is obviously when you disassociate a floating IP the status goes from active to down any error that occurs in the process like Something bad happens on the layer 3 agent on the layer T agent is not available at all The floating IP goes into an error state which reflects this condition. So this is basically it and other enhance other enhancement for I wish we had time to talk about these enhancements, but They will hardly fit at the time for this session. So we have multiple RPC multiple rest and RPC API workers, which means that both on the external side the RPC API the REST API and On the internal side, which means the message queue We now have multiple workers and therefore we are able to process requests in parallel This is something which has also helped a lot with reducing the time to allocate Network info from Nova and we the graph that we've seen before and We have better IP address recycling. So the IPAM module, which is like baked in the neutron logic Kind of did a lot of query area a lot of database query every time you address you Allocated in IP address to you know those queries were meant to make sure that the information for Available and allocated the IP range We're already in sync where actually you probably don't need to do that every time You need to do that only where your IP address pool is Approximately it's exhausted The IP address are close to finish So and that's where we do a better IP address recycling and we have reduced the law to the pressure on the database Though then another problem that plagued the neutron with Avana was The error that most of you have probably seen in the logs lock weight timeout exceeded Lock weight timeout exceeded is actually just to be a little bit more technically is not a database problems Not a new term problem is actually a problem with neutron not using event light in the proper way What happens is that in the middle of a transaction? You have a part in the middle of DB transaction You have a particular statement like even just a log statement which causes even to let to yelled If even to let then yelled to another transaction It starts another transaction which can deadlock with the previous transaction that you executed And this typically happens when you have the lock for update statements because you know Maybe the previous transaction has locked the record then yelled to another transaction that wants to update the record And that our transaction is stuck because it's waiting for the other thread to release We have removed a lot of them I think we've got to the point where in Under heavy workload the occurrence of this kind of error is now to reduce to a minimum When using the ML to plug-in we still have some of these errors when you use the load balancer service plug-in But those problems are being sorted out with the work. It's being done for load balancing in kilo and And well, I think that's really all now and since I have really no voice anymore I'll let that on finish the presentation Cool So kind of in summary that as you can see from the results that we did the L2 agent has significantly improved in scalability and performance Security groups now can now be used in larger deployments And we don't have that exponential problem because of the wire format that was being used The Nova and the new the Nova to neutron interface the communication there is now much more improved and reliable As you can see in our tests there are no failures booting 20 or so VMs concurrently whereas in previous times in older releases We would have a lot of problems there when operations were being done concurrently And we also now the instances when they're go to active-state the networking is also wired And a lot of the resources now we've tried to improve the operational status of them So for instance now floating IPs actually reflect if they're up or down Still there are a lot of problems that we have in a lot of area for improvements For instance the OBS agent and the Linux bread agent still pull on the bridge Constantly to find new vifs. We do have the new logic using OBS DB monitor to get those Statuses or those changes so we don't have to pull but we haven't yet enabled that by default But that's something that we'll be doing soon The Nova and the neutron interface is a lot better, but there's still a ton of work that has to be done there Currently if you query Nova or Nova API that actually makes a call to a neutron So unfortunately if neutron is down that also brings down the Nova API endpoint So that's something that we want to continue to improve and furthered a couple in this next release A Little bit has been done around improving DHCP, but that's still kind of a weak weak point Another problem that we kind of have is even though the networking is wired The IPM formation has not always propagated to the DHCP agent in time in time So making that Reflect or and have an operational status to know when the IPM information actually makes it to the DHCP agent to further improve This orchestration is something that we'd like to also improve Final thoughts things are a lot better, but we still have a lot of work to do We still have a lot of questions on how we were going to handle data plane scalability for instance If we use and the current implementation the hybrid OBS driver Requires using OBS and Linux bridge together and that has some data plane Ramifications and performance costs there that would we'd like to address. There are some people who are working on Programming OBS flows to do security groups instead of using IP tables. So that should be a good data plane scalability improvement that can come and There are some questions on maybe improving the neutron API to be more usable For Nova to help improve that integration that would be for instance the question like you have known When you turn like you are kind of forced into a topology Where you need to have a network that go up links to a router and then the router as a gateway Whereas there are a lot of application of cloud actual cloud application like no the pool You know no blue the runs in the gate where they don't give a damn about the router the floating IP They just will like give me a VM that's able to do something that I can reach through the network So this is all a matter of improving API usability so that it can be the API can actually be Leveraged by more cloud applications. So I think that it's all for today that we add I don't see many people sleeping. So we probably did a good job So if you thanks for listening and if you guys have any questions, there are crime Thank you very much That's there are the microphones for the questions over there or other or you can Use one you can try and speak up if you have enough voice or you can try use one of these I wanted just to ask if you check these performances basically only on IPv4, okay? I know that in a van is not there, but also maybe it is would be interesting also to know what is happening in Juno regarding this. So is the performance You mean between IPv4 and IPv6? So the checks that we made actually are in a way, which is basically independent of the IP versioning scheme because the only difference in this in in this workflow using IPv6 is The DHCP configuration that either you don't do it at all if you do slack or you do it using RA DVD If you do the HP state full with IPv6 the wiring part on the agent You know for bringing the port up with the IPv6 is Pretty much unchanged. Obviously. There is a different part for IP tables where instead of using IP tables before You'll be using IP table v6. We didn't do the measurements. I expect that It should be pretty much the same order of magnitude, but I want to bet on it because we neutered you can't never know You know and you you can ever expect what's going to happen. So Hi, sorry to a good talk a very simple question. You didn't specifically talk about the setup So I take it it was a neutron and obvious DB directly Some servers or can you elaborate on this? I I probably did not state with enough clarity So we try to keep a setup which is as close as possible to the one that we run by default in the gate To provide a measure for comparison and therefore I didn't mention this that explicitly is the ML 2 plug-in with the OBS mechanism driver running the OBS neutron agent and On a machine like it is I say this with four cores and eight gigabytes of RAMs Obviously, if you run this is not a production machine obviously This is the neutron node, right and and we you know for Focusing exclusively on a single agent I run the server on one node and just one compute node just to make sure that all the Instances that I launched ended up on the same agent and so that if I launched 20 instances on the on the neutron on the Nova side I got 20 ports being wired on the same Compute node but what I didn't doing this test is an horizontal scale test because I mean It also involves like a lot of hardware doing this but an horizontal scale test I know that there are people out there that have done this kind of testing So not testing the stress testing scale on a single node to see how the component scale But doing a scale test across thousands of node to see how do you overall architecture scales? Which is another interesting testing to do I? May have missed the very first slide, but do you have any snaps of? Where ice house is relative to Juno is all the improvements in the last six months or where there's some already an ice house We did the measurements from Avana to Juno. We didn't do measurements for Avana from ice us to Juno I mean that is something that I have or I still have all the infrastructure ready to do the test We are already have a lot more results than the one that we have here And we are going to publish them in the next few days or weeks But it's a good thing to do also the same kind of analysis for competing ice us to Juno I mean I have to say that the largest chunk of improvements probably went in in the period from Avana to ice house and And then the people from our eyes I ice house to Juno. We stabilized a lot along these improvements Hi, my question was more on the on these tests that you did for concurrency and The various granular measurements what kinds of tools that you use to repeat these tests? Okay. That's a good question so I I Wanted to use OS profiler. That was my initial idea I started to use OS profiler and I found myself so close to the meeting time that I had to do To the to the summit time that I had to do something quicker So what I do is I developed a tempest scenario test for doing this kind of test And it's just a tempest scenario test that boots VM concurrently and measures the The time for instance time to ping a VM time to a cessation to a VM time to bring the VM into active For measuring things into the agent in a reliable way. I just instrumented logging. I instrumented logging. I fed all the Logs all the logs. I pushed all the logs to see slug and then I had another process reading the extracting this information from syslog Processing this information and doing statistical analysis Pushing all everything into a CSV file, which I then imported in Google Docs where I get I got I created the graphs That's pretty much it. So it's a lot of under made process, which I think the data analysis part It's something I can keep developing Maybe it's probably useful for the data collection part I think that if I keep doing this kind of thing in the future are probably switch twice profiler And do you think that could be integrated into the gate? Integrated look integrating to the gate Testing which is not functional is always an odd thing because Because the problem is you would like ideally to do something like a Gate test in which you say if your patch is slowing down whining time by more than 10% Then your patch is not good The actual truth is that defining a baseline a baseline for that is very hard and also Another thing is that you a single measurement is not reliable You need to get enough measurements to have data which are statistically reliable. So you can Surely monitor the current condition Like in a continuous way, but not as a gating process I think I don't think it's a great idea to do that as a gating process because you know like In some cases for instance Aaron did the patch that slowed down things like you know increasing the time of VM to get Too active but by slowing down things he made the software a lot more reliable You know slowing down is not necessarily a bad thing Great talk. I have two questions One is one of your child we show that the old but you know, there's a tea up or something to linear why Yeah, I'll go I'll go back to the slide and you tell me which slide are you referring to? This one something like that. Yeah, the previous one this one. Yeah, why why do you know it's a linear? Now before you ask a question the second one is there. How do you handle the situation? They said that if there's some sort of OVS Let's say flow is not at correctly and when you boot up a new tongue instance, right? Is that waiting passively or is doing some sort of proactive checking? Oh, if a if a flow is not set correctly Like if you mean like if there is a problem like data plane in OVS, you mean flow not set correctly What do you mean? Right? Like it's some flow is not added on to the some other bridges Okay, you mean like if there is any sort of error in the wiring process, right? That's one example. Yes Yeah, so when there is an error in the wiring process the thing is there are errors which are Caught by the OVS DB interface and then there are OVS failures Which might not be caught because for instance if the OVS kernel driver crashes for instance That's not a failure that we catch in that case. Unfortunately. We are not able in any way to report yet That condition back to the server But in the case where there is a failure while processing interface, so for instance you call No VSD be command that the WSD be command fails We report we report that failure in the port status So you will have a VM which I guess now doesn't boot at all, right? The port would go to because if the port is not wired the VM doesn't boot previously What happened was the VM boot went to active you wanted to use it and bang did not work Then you had to go to the neutron API Check the port status and you saw that your port status was down Which probably made no sense because if I have no network There are very little use cases for a cloud VM to be used unless you are one of the kind of guy that can do everything through a VNC console Right the full process actually the way that it works is in ML to the agent will detect the port is plugged in by Pulling on OBS TV then he'll set up the flows and then he'll send Message to the neutron server saying that the port is now active and then that will trigger the message to know if it's on positive instance Which OBS and which kernel? It should work on any version or most modern version. Yeah, this is all done in it's okay That's whatever's on the gate. Is that what you were that's a good question because we did this test on kernel 3.2 2.58 for Avana and On kernel well the kernel for trusty. It's the same kernel trust it runs in the gate I don't remember exactly the version and the corresponding default OBS versions which are shaped by canonical now It's true because there have been also a lot of improvement in OBS in the middle and this is helping a Lot when we call OBS DB because the version that we had in Avana was a multi-traded The version that we have now is multi-traded. So this is helping as well So the infrastructure is helping as well to give you an actual comparison actually run Run Avana in the environment for which was designed and Juno in the environment which was designed But this is the the infrastructure. Let's say the data plane infrastructure is helping as well Excuse me just I Better understand now why you removed the log for update. I thought it was for Galera compatibility, but now I understand why The other reason why sorry, why did they remove what as a lot for updates? The log for updates call. Ah, okay, like alchemy But I think that a lot of part of the code was Very relying on this look for a date to manage a resource resources concurrency So I wonder if it doesn't introduce a lot of Race conditions. No, we made it. We made it in a way that doesn't reduce luck We changed the code in a way that if we could rely Rather than rely on lock for update if we could rely on database integrity For ensuring the same kind of behavior we did it In no case we implemented the as we needed a solution for distributed locking We in other cases we went to Optimistic approach which means if you do an operation that when you normally needed to lock It's because you are competing for a resource which can be taken by somebody else In that case if the resource that you want on database is used by somebody else You land up having an error made an integrity valuation error from the database And so what are we doing now if you get an integrity valuation error while trying to retrieve an IP for instance What you do is oh, sorry somebody took the IP before me and you try again So we've added a retry mechanism, which to an extent you say while you're retrying. It's not good I mean it's much better to have like a sort of consensus if you want But that would have implied the adding some form of distributed locking and when you come to distributing computing Constructs like this to even just simple distributed locking either you do it good or you don't do it at all And so we decided the decision was whether we need to you start using something like memcash d or Using something like zookeeper to have proper distribution across multiple API workers or just multiple threads Do the retry mechanism the retry mechanism? I can say it's a bit pedestrian if you want, but it works And we prefer that at least for the recycle while we think how we should do proper distributed coordination And I mean it doesn't it doesn't matter, but this is what all the other open stock projects do Any more question Apparently not three two one zero. Thank you very much and have a good day