 Okay, I think we're ready to get started So we're here to talk about Interoperable networking in open stack via def core my name is mark velker on the other side car detective VMware And I'm also a member of the def core committee. Hello. Okay. I am Kyle Mestri. I'm a distinguished engineer at IBM And I'm on the open stack technical committee. Hey, and I'm Russell Bryan I'm a senior principal engineer at Red Hat and I'm on the open stack technical committee and also the open stack Foundation board of directors So a little bit about what we're gonna talk about today For those of you that are new to def core It's a fairly new thing for most people in the community just to be kind of became probably most people's minds around the Vancouver summit So we'll do a little explanation of what def core is about just kind of level set for the rest of the conversation today And then we'll talk about what interoperable networking means and in terms of def core and in terms of capabilities of the stack exposes We'll talk a little bit about some history about how we got to where we are today and why we're in the mess that we're in and how We're trying to straighten it out We'll go through the Nova net and neutron conundrum and then we'll talk about what has actually been proposed to the board of directors And that will be voted on in in three months And then talk about how you can engage in that conversation And supply your own feedback about the the proposed capabilities Before everyone starts I was just curious is everyone a herd of def core or does everyone knows what it is Okay, or a little bit at least I mean I was just Everybody know what networking is it was winger actually but close Okay, so a quick level set on def core. We'll try to keep this short There's a lot of nuance here the dog gloss over but if you want to pick my brain find me in the hole And we'll we'll talk right off The def core committee essentially is a body that's created by the stock foundation's board of directors It's essential goal is to create what we call guidelines a guideline is basically a document That states the products that want to use the open stack name or the one use the open stack powered logo Have to expose certain capabilities to their end users So basically the guideline consists of a list of those capabilities The tests that you must pass in order to prove that you expose those capabilities and then also a list of designated sections and designated sections just basically say that if you're exposing a capability You should be using open stack code to do it and the essential thing there is just that you can't go Right open stack in Java and call it open stack because that's not what we are at the end of the day products that want to Use either open stack in their product name or that want to get that logo at the bottom of the slide there from the foundation Have to adhere to this sort of the The sort of end user case for this is that all those capabilities can be counted on if you're using any product that bears that logo So if I want to run a workload on multiple clouds or multiple cloud products as long as it has that logo I know there are certain capabilities that I can depend on being there and exposed to me So obviously that brings up the question of how do we pick what capabilities go into those guidelines though? We require vendors to expose whether a public cloud or a distribution or a managed service Defcore has 12 criteria for selecting those capabilities and this is kind of what that looks like. They're kind of lumped into four four categories It's if you look at Kind of what's on the slide here. You can see that these are mostly trailing indicators of acceptance So for example, we don't include something in Defcore guidelines until it's widely deployed So it's not like, you know, the minute a new feature comes out It goes into Defcore because it actually has to be accepted by the end user community before it goes in there Some of the more esoteric features that projects expose that a lot of people don't use are also probably not good candidates for Defcore for that very reason I won't belabor this because we could probably talk for half an hour on just these 12 criteria But I did want to at least give folks kind of an idea of what this looks like from a Defcore perspective And again, there's there's a whole Wiki page that you can go through that kind of explains each one of these a lot more detail and the links are at the end of the slide We'll put those up on slide chair for you to grab later So a couple of couple of important points first of all Defcore only covers capabilities They're actually exposed to end users So there's a lot of things that open stack that are considered admin only APIs for example Or there are things that happen kind of on the back end that the user never sees So for example, you wouldn't see Defcore making requirements around say RPC messages because end users never see those for the same reason when we get to Provider networks in neutron. Those are generally an admin only thing So if a user can't actually use a capability, it doesn't go into the Defcore guidelines. This is really about end users kind at the end of the day If I'm if I have a tenant account in a cloud I should be able to run all the tests that are required for Defcore and that's all I should need Having more than one way to do something is actually okay Like the classic use case we see for this is when a project has more than one version of an API So maybe if I want to upload images into a cloud, I could use you the Glance v1 or the v2 API That's that's actually okay from Defcore perspective. What we care about is that they meet those 12 criteria And that they're they're actually exposed to end users And the the sort of line there is it's okay to offer users choice We just want the the choices that they have to be dependable We mentioned that Defcore is already a trailing indicator Kind of one of the big things to understand there is that every guideline that Defcore produces covers three open stack releases The most recent three open stack releases So that means that a capability won't go into the Defcore guidelines until it's actually been present in three open stack releases That gives a time to mature and and meet all those other criteria as well Also, when we add new capabilities to the guidelines over time There's actually kind of a waiting period. So we have to put them into advisory status for six months And then the next guideline they can actually be required The purpose of that is basically to give vendors a heads up that something is coming And it's going to be required in the future and give them time to make adjustments to their products if they need to So if I'm a public cloud, you know, I may have to change configs or or get ready to expose new things to user That could sometimes mean pretty involved work on the back end I may have to you know deal with storage issues or I have to deal with network redesign issues those kinds of things So the purpose of the six-month period there is to give all the vendors a heads up and also give users a chance to chime in as well So in terms of interoperable networking realistically for most people today what we Probably have to have to have interoperable networking is basic L2 L3 And external connectivity so those are really the things that we kind of focused in on for this first round Defcore this is the first time that we're actually gonna have any requirements around networking in Defcore So, you know baby steps, right? We can get to more advanced stuff later, but this is where we have to start And just mention again, we care how these are exposed to end users We don't really care how they're implemented on the back end So we really don't care if you know you're building networks with VX LAN or VLANs or whatever else That's an implementation detail that should be completely hidden from the user We basically care that that API call works and returns a thing that users can use The problem is that there's lots of different ways to provide connectivity in open site today There's provider networks, but that's an admin only construct You can directly attach instances to an externally routable network Users don't necessarily have a way to find out that that is an externally routable network unless each vendor tells them so So if I'm writing a workload that's signed around multiple clouds that can be kind of challenging There's floating IPs Which is kind of a matter of debate even in the greater industry outside of open stack Amazon has kind of been getting away from that model and DigitalOcean actually just recently announced that they're adopting floating IPs So there's there's kind of a lot of debate in the industry at large not just within open stack And then we also have a new API proposal in Neutron, which Kyle talked about in just a minute One more thing to kind of levels out the conversation here when we talk about networking Remember, we're kind of starting at baby steps, right? And we're also looking at stuff that is very widely deployed in open-sat clouds today So we're not including IPv6 in the discussion today just because Although we can all see that IPv6 is an important thing that we should be paying attention to it's just not widely deployed in a lot of Open-sat installations today. So that's kind of on the sidelines for the time being and again That doesn't mean that clouds can't offer IPv6 It just means that they have to offer IPv4 as well We're also focused on L2 and L3 here, so we're not going to get into the L4 through L7 services All right, so let's talk about a little bit of background some history So there was a time where Neutron wasn't a thing or a quantum or any such Alternative to Nova Network and so, you know when open stack started we had Nova Network and it was simple And in quite limited, but it just worked which was which was fantastic And it was not at least intentionally plug all and it's just software So it turns out you could go put your own code in there if you want to but that really wasn't the goal It was you know here and we're just a very basic straightforward way to provide network into your cloud and You know that was that was the way we did things. So if you jump the next slide here, you know, what's a Nova Network did it had You could do flat networks Optionally with open stack providing DHCP on the on that network Or you can do really really basic tenant isolation using B lands and the tenants didn't have any control over that But we could do some some super basic tenant isolation. There's no no support for overlapping. I address support That's a big thing in Neutron, you know every tenant may want to create create networks using the same Private IP address space, but you couldn't do that in Nova And there's no, you know, no no support for integration with your physical fabric That's you know some and and well many other things that Neutron can do That just Nova didn't do it was just Just just just really simple and it used IP tables for security groups Which is pretty common in the Neutron case and it had both a central and a multi-host mode for providing external connectivity But that was it there was there wasn't there's no sort of a concept of SDN controllers no integration with any sort of you know In these newer alternatives So if we think about Nova Network from an interop perspective It was actually really good from an interop perspective Which is part of what makes this whole thing difficult because like, you know Through the development of open stack We sort of have to balance this innovation angle with the fact that we actually want to at some point provide some sort of reasonably standardized thing that you that you that you get everywhere you go and you use open stack So we kind of we have this this balance, but So, you know in the Nova Network days you had, you know, very little options Which was great for interop, but but the reality is people actually want more choice The industry is coming up with new and better ways to to do networking to do virtual networking and you know, we want to make that possible with open stack and So the reality is that the neutrons the way forward and so now we need to sort of take that innovation But still figure out how to provide Something that that is standardized and and can can be included in and the rules we use for interoperability Okay, so so now we can talk a little bit about neutron now that we have the background of Nova Network somewhere right as Well, and I think I think one of the one of the biggest things that That neutron provides is the ability for tenants to create and connect their networks and subnets which which is so this is like Mark was saying this becomes a user-facing API So this this is something that the def core now wants to get involved with so we can kind of figure out a way of What of all of these different API's we want to we want to put under the purveyance of def core I think a little bit but but you know the problem is drivers can implement this in many different ways on the back end Maybe not so much a problem as long as the user-facing API is there and the semantics of the API are honored By all the different drivers. I think so that's pretty good One of the really big things that that the team kind of focused on over the kilo on Liberty cycles and really a lot of it was not in Liberty was documentation around this Because it turns out that with regards to Nova Network There was a large swath of people that really liked the flat mode of Nova Network And we're not aware that you could you could do that with neutron as well And so that's one of the things that we ended up documenting was Was that particular mode in particular? So so now people who were using that and maybe wanted to deploy a new cloud and wanted the same behavior of their flat Network could do it with provider networks on neutron as well So that was that was actually really really huge thing And it was odd that there was like a communication gap I think there for a long while that we finally solved at the Vancouver Summit and yeah There's just this this big assumption that just because neutron could do these more advanced and more You know it more advanced things that were exposed to users that that was the way you use neutron When it turned out you can actually scale it back and do something just like just like Nova Network Exactly. Yeah, now it's now it's well documented and we've seemed to be clearing up some that misconception. It was really huge Actually, yeah, so then beyond that though Neutron does provide you a pretty rich API to to build things like networks Subnets and ports and you can as a user using the API you can construct some pretty Complex topologies using that and build different things and that's great There's a lot of people that really like that that use that to build these topologies allow their Users of their cloud products to do that. There's another group of people that that still Really want something really simple. So for example, if you if you want to boot a VM and maybe get connectivity For that VM both inside and outside There's a lot of things with neutron you have to kind of do to get that up and running You know, you have to create networks subnets. You have to boot the VM. You might need a floating IP things like this So one of the things that that we that we looked at during Liberty Was this what's been termed get me a network and it really is just as simple as Look, you know, I just want a network. I just want a network for my mission I just want the VM to come up I wanted to have a connectivity and just get me a network and so we we iterated on this a bit during Liberty We never I mean we merged the spec the work You know, we I think there was a version of the patch towards the end that came out But it was more of a proof-of-concept type of thing So so that that that really is something that we're looking at for mataka now And I think the team is really prioritizing that to get that done Our own infrastructure team I know really wants that the open stack in for a team for example because one of the info clouds They're building they just want to be able to put VMs right on public networks without using floating IPs And so this would allow them to do that in a nice user-supported way So so that's what we're going to do and this kind of leads into the next thing Which is this discussion of the technical trade-offs of floating IPs You know, what is a floating IP or a FIP as it's kind of knowing around I really love the FIP terminology better personally But so what is the floating IP? So it's an IP that's assigned on an external neutron network And and you can actually use NAT to get to it So what that typically means is if a tenant as you create if you create a network Typically your those will be like some sort of private address space that that network will get from that from the subnet That's associated with that network So you can get outside because we do you know NAT to get out But if you want to get back into that VM say you want to get traffic into the VM You can assign the floating IP address to that as well. And so the reality with floating IPs is They are great for some workloads. A lot of people do like them I think like Mark said digital ocean is now Going to support them at the same time as maybe other providers are not going to so there's really no There's no consensus around yes, you absolutely have to have them for your cloud or you don't I think So I think that this kind of leads us into the fact that there are lots of different ways to do networking If you sit in on any of the neutron design summit sessions You might notice that there are lots of different ways lots of vendors different ways of providing connectivity I think so there really is a demand kind of for this get me a network spec though I think so so we're gonna figure that out But I think like the slide says even when we get that in it's gonna be a long time before we could kind of mandate it in Defcore right and and mark kind of went over why that is so yeah It goes back to the point that you know We have to have something the capability asked to be exposed in the most three recent releases of open stack before we can Actually put it in the standard and even once it is has been present for three releases It still has to meet all the other criteria So people actually have to have adopted it and used it in the wild Before we can actually stick it in there so it could be a while before something like this actually makes its way into the Defcore guidelines, right so that's so even if we get it in it's you know Realistically gonna be the the P release before assuming it's actually deployed in everything so and then the floating IP conundrum as well I think an early version of one of these patches for Defcore had floating IPs in it and very harmlessly was put in there and a Bunch of people liked it and then the negative one fairies came out and just you know destroyed it I think you know that was it because there was a there was like people didn't like the floating IPs eventually I think so the so the reality is is I Don't well. I don't want to spoil anything from your future slides, but yeah Yeah, the future of floating IPs. We'll talk about that soon sure All right, so speaking of what's currently proposed. This is where you can find that thing So the link at the top of the page there is a link to the spec that was just presented to the board of directors on Monday of this week And that it's proposed spec so we have now come out to the community said all right We've done our scoring. This is what we think needs to go in there Community please come comment and there's three months before the board will actually vote to approve it So it's basically there's three months for you the community to chime in and and you know put in your your two cents into the bucket And I'll have a link in a minute to review where you can actually go do that Basically it includes four buckets of capability of capabilities is advisory. Remember that we have to have the advisory period before it Actually comes required So there's basically L2 operations. This is stuff like creating networks reading that listing networks updating networks deleting networks Same thing for L3 We also include routers in that which allow tenants to create routers and you know connect isolated L2 networks We also have security groups and floating IPs But let's talk a little bit more about floating IPs and in a minute or two here L2 L3 and security groups the early feedback that we got when we were going through the scoring phase on this was pretty People pretty much were okay with all those capabilities. There's a little bit of discussion about routers, but not very much So those seem like they're pretty likely at this point to land as as required in a feature spec Floating IPs on the other hand was a bit more controversial So if you look you breathe the proposal that was on the last slide floating IPs are currently in there But it seems pretty likely at this point that there's not enough consensus and that they'll probably Wind up falling out of the spec before it gets up for approval So that means two things One if you have an opinion about whether you like floating IPs and think we should be able to depend on them or not You should probably go comment on that spec So, you know the like Kyle said that the negative one fairies are out on floating IPs If you're a plus one fairy now would be good time to speak up So just like a survey who's in support of floating IPs anyone to people I'm raising my hand actually so there's Russell Yeah, anyone else really no one. I don't hate them. I don't know if you Yeah, that's a good question Last bullet on the slide So basically there if we look at how people are actually doing this today There's enough different ways that different products actually exposes when I say products I mean public clouds distributions managed services Runs the gambit, right? We actually did money Taylor actually went out and surveyed all like 15 public clouds that he has an account on and put some Information up about those we went through some of the individual products as well And we'll select a some user data from the user committee as well when we get through there it's so like fragmented right now that there really isn't a External connectivity standard like floating IPs was our best hope And if that isn't widely enough adopted then probably nothing else is right now What I will say is that You know the top side of that is that we probably won't be able to have a external connectivity Standard way of connecting for quite some time for at least you know 18 months if not more The the sort of flip side of that is that kind of galvanized the community behind Wow, that's that stinks And we should fix that so that's where a lot of momentum has come behind the get me a network spec For example as possible route to that there's also been some discussion about how we could have a maybe a new API call in Neutron That goes into that again and figures out okay. Well the way that this particular cloud is set up is that You know it uses floating IPs so when I say get me an IP that has an external connectivity I will actually go create a floating IP and assign it to that instance and you know so it's kind of another layer there So like I say basically the the work that we did here and kind of unraveling that whole story and actually going out and looking at What people are doing today was actually it sparked a pretty healthy conversation about how to how to do interoperable networking in the future We figured out that you know L2 L3 security groups There's actually a pretty good consensus on that we should be able to set that up to where people can rely on those capabilities External connectivity is hard and we should go fix that So now we're kind of in the face. We're having a good discussion about how we we go and do that So again that you know like we just say there's there's a good conversation about that going on now And we'd be interested to hear more from from people about how that all should work And so just to reiterate here about why we need that kind of feedback It's not like the user survey can really even ask about this in a way that makes sense, right? And users don't actually know in a lot of cases how their provider has set things up in a lot of cases if we go look at you know Say red hats distribution or canonical distribution or VMware distribution There's actually different ways that an individual customer can set those distributions up So it's really hard to like send out a survey and get good feedback on this kind of thing but at the end of the day Vendors who want to call themselves open stack or want to be able to use that logo Whatever goes into the spec they're gonna have to expose right now their customers can switch it off unless it's a public cloud Right where the customer can actually do that But they're gonna have to expose that capability. So it's kind of a big deal for for the vendor perspective And users on the other hand want to be able to depend on certain capabilities And this is the route that gets them to be able to do that Any project that bears that bad any products that bears that badge will have these capabilities exposed If we choose something that has technical implications on the other side Like, you know, one of the arguments against floating IPs is that they rely on that and people don't like now because it's not performant Right in their minds for their use cases. So if we were to go with floating IPs People are probably gonna have to live with that, right? Because it's hard to do floating IPs and something else within the same compute cloud possible, but complicated for the administrator So, how does supply feedback? So this primarily affects open-stack products, which ultimately offend and affect and use maybe offend affect end users So one way one channel do this is if you're using a vendor product, whether it's a public cloud or distribution or I don't know consulting shop for all I care It's ultimately pretty good to lean on your vendor a little bit and tell them look This is what we want. This is what we don't want right because ultimately that trickles back to us as well Use cases are also super helpful Some of the feedback we got during the early phases of drawing all this up was you know If we include this capability then use cases X Y and Z are broken So that was good feedback for us to be able to understand how things are being used and that makes us helps us make better decisions Throughout what capabilities include Usage data is super super helpful There's a tool called ref stack that people use to run these tests when they want to go for a logo certification process with the foundation Anybody who has tenant credentials to a cloud should be able to run those same tests So if you are a vendor or a user of a cloud and want to go run those tests yourself You can actually go do that those actually get reported anonymously to a site called well was rust act on that until a couple weeks ago I think now it's open sec org slash rust act But that actually gets reported into a central repository and the more of that data that comes in the more We can correlate what's actually out there in the wild so we can see that you know these five tests We only have five results in the entire repository that they passed on that tells us that that's probably a poor capability Opinions are good, but their opinions right so just because me a vendor doesn't want to implement something or doesn't work Well with my product or whatever it is Is not necessarily a reason that it shouldn't be an awful right if the whole rest of the world is doing this And I'm not I'm not interoperable that by definition right so I'm gonna be out of compliance with spec And I either need to fix that or decide I'm not gonna be called open sec That said opinions are valid data points for us So especially if they're backed with some rationale like not just I don't like floating IPs, but I can't use them because X Y Z So feel free to submit opinions. Just be careful to recognize that they are what they are All right, so now we get the point about how to actually supply feedback there's a review All that review says right now is the one line change that changed that spec We showed earlier from Review status to approved status that won't actually land until January at the earliest when this goes up before the board But in the meantime that we use a good place to collect all kinds of feedback So that's where people can chime in and say, you know, we need X Y or Z We don't like this capability. We don't like that capability. That's also where in future patch sets We may drop some of those capabilities. So I think someone actually already submitted a patch this morning to drop floating IPs out of it So, you know, that's that's a good place to sort of funnel feedback You can also grab me or one of the other deaf core committee members in the hallway You can grab one of the TC members in the hallway talk their ears off about it I think my email is actually somewhere in the slide. So I'll leave you to find it Operators meetups. We try to stay in touch with the operator community as well So if you're attending those sessions, then those are good places as well We rely pretty heavily on the user survey when we're looking at things like what Toolkits and client APIs people are using So if you look at the most recent user survey results, I think lib cloud j cloud and Was the other one fog were the three most common API toolkits that people are using That gives us good data because if we find out that, you know 60% of people that are using and something other than the upset clients are using those three things We can go look at those three things and say wow that one doesn't support the lights v2 at all So maybe that's a strike against clients v2 going into all the guidelines. So please answer the user survey if you can I already mentioned rest act And if you're working on over a neutron in one of those technical communities a good place to give feedback is within that technical community as well Couple resources there like I say we'll post these slides later if you're new to def core There's kind of a one-on-one deck that you can go look at There's also some links to our published criteria The definitions that we use and we're talking about things like what a capability actually is The guideline itself and then the previous guidelines as well If you want to see what's more to prove today and what products very logo actually have to expose today I think that is about it. So we've got a couple minutes for questions Anybody has any Right. So generally speaking API extensions are not something that we put in the guidelines because they are very frequently optional In neutron's case like all kinds of stuff is extensions. Yeah, it's actually like super widely deployed Yeah, I think that that particular concept of extensions and neutron isn't isn't really a good Indicator of where the line is between what's like in reality like a core feature versus a more Well, I'll say less used feature. And so that's just that's a really good example of that It turns out to be it seems like it's more of an implementation detail than a something that really communicates Something that's you know required or not. I mean, yeah So I mean ultimately it comes down to what people are actually deploying in the wild Generally speaking extensions aren't widely deployed in a lot of cases in cases like you know, they're other projects, too Neutrons in the one but I think in neutron's case some of those extensions really are very widely deployed open stack more generally And some you know, some of the other projects have started this and new trends talked about it quite a bit But hasn't implemented it yet as this idea of getting rid of extensions because it's confusing for this exact reason and Just making, you know, this is the API. It may be that that feature is disabled and when you try to use it You know, it may just read and reject the request and and hopefully we even make it discoverable about what may be Disabled or whatever the case may be but getting rid of that idea of extensions could sexually painful for everybody So I think that's really kind of the direction that the open sex going overall So every almost every open-side project has is an abstraction over something right in neutron's case What they extract is a bunch of different kinds of ways to do networking. It could be physical switches with VLANs It could be SDN controllers could be whatever else, right? Stuff that goes into the drivers is how That specific thing under the hood Realizes the concept that you're asking for so when I say get me a network What I'm actually doing might be, you know provisioning VLANs on a switch. It might be setting up open flow tables You know, it could be any number of other things. Yeah that Yeah, so the way to the way to think about Yeah, the way to think about neutron conceptually is sort of it's kind of two pieces of kind of really zoom out, right? So there's the API layer there's that there's the common implementation of the API that exposes the concepts and users and administrators That are common to in theory common to to whatever the back end is that you've chosen And then there's the the back-end drivers that are sort of invisible to the end user that implement that so But that the part that's required like for this is that top layer So everyone standardizes in the top API layer that the code and the API itself and then it's you know Your choice of the of how the back end. Yeah, and to the point You know, it is possible that vendor like fun running a public cloud for example I might implement neutron have these API's and then also have some other API's that I exposed the other means that are not Neutron, that's fine. This doesn't prevent me from doing that, but I have to expose the ones that def core requires The implication general being that you know, most providers are probably not going to choose to have to offer two ways to do the same thing Because that's a hassle for them It's possible some might but As an open psyche user I can depend on that one being there maybe else so The way it generally starts out is the deaf core committee itself starts doing its own homework about what they think is widely deployed we look at the user Surveys we look at what's supported in client toolkits and those kind of things Step two is we kind of like draft a straw man proposal and go talk to the PTLs And the PTL is general bring in some of the other technical community members We also generally bounce those kind of things off of end users as well That eventually makes its way into a patch Literally a review and Garrett that people can go comment on And that's actually provides the scores for each one of those 12 capability. We say yes, this is widely deployed. Yes This is used by tools. Yes, this is used by clients That goes up for debate and people are welcome to comment on it We are going to start rolling up to both the TC and the user committee kind of a report of that Every every cycle as well And through that feedback process there, that's what eventually goes into the standard that we present to the board Once we do that and it's in that proposed standard Then it kind of gets wide publicity like I'm sure in a week or two You're gonna see a blog on super user about this and some emails to the ML and that kind of thing So that's the that's kind of starts that three month period where stuff can be thrown out before the board even gets to vote on it And then once that's done obviously we have a six month advisory period as well make sense So yeah, we should talk about pressure relief valves. So there's a couple of those If there's a capability that is required and there's good reason for it not to be required There's a thing that we can do called flagging a test The reasons that we would do that are like we screwed up on the scoring and it actually isn't widely deployed And we thought it was or maybe tempest is broken and that test just won't actually ever pass for anybody, right? So once we flag a test then it's not required for the rest of that guideline We may drop that flag in the next cycle like if it's a tempest bug in the bug gets fixed Then we'll drop the flag, right? We do eventually retire capabilities from deaf core and the way we do that is By deprecating them. So just like with any any API in open stack. There's deprecation cycle and we'll kind of do the same thing In a lot of cases the project will choose to discontinue something like, you know We may require clients v2 today and glance decides to write a new API and deprecate the old one So as a lagging indicator, we'll generally follow the projects So when a project says this thing is deprecated, that's the point where we'll probably deprecate it from the the guideline Yeah, the nice thing about about removing things if you think about it in terms of like these being requirements on a vendor or Or someone providing a product is it's you're really making you're actually making their job easier when you remove something So there's like less controversy around that I think I mean It's it's a step back away from interoperability and a larger set of things that you can count on So there's a there's a bad side to it But in terms of the people that you're holding accountable and saying you must be able to do this To be able to use the open stack name and logo like they're not going to argue with you making it easier on them So which is kind of nice, but so that's why you kind of you see a much more You know robust processor on adding things because that's raising the bar and that puts pressure on everybody And it's great for the whole community, but you know that that's increased pressure So that's why there's a whole lot of focus on the process for adding things Anybody else? Okay. Thanks for coming. Yeah. Thank you very much