 So let's go ahead and get started since we're at the top of the hour. Okay, well hello, YouTube and thank you, Gonshom. We want to welcome everyone to this week's episode of Open Infra Live. This is our new weekly hour long interactive content event series streaming live to our Open Infra YouTube channel on Thursdays at 1400 UTC. So today we are getting a very special behind the scenes look live at the PTG for the OpenStack Technical Committees meeting today. So we appreciate you all letting us join so Gonshom, I'll go ahead and turn it back to you. Thanks Ashley. Thanks everyone for joining. And in this PTG session, we'll be discussing about the release cycle things. First of all, I'll share my etherpad so that you can see the notes that I hope everyone can see the etherpad. So this is our today topic at line number 39. Let's move open stack to release cycle. So open stack is six month release cycle. And we are going to discuss about how we can make it yearly cycle or not. So I'll let Zulia to give some background about this topic and then we can start the discussion. Absolutely. So this topic really came out of a weekly social call that the Iran community runs which is geared at basically social interaction trying to keep us human relatable. And we were discussing one of our new features and one of the operators because the Iran community has several operators to also contribute. Basically said, that will be a very long time, if ever, if they were to ever get the upstream future that we just dropped or that we just added or though being a late release. And so we kind of got them to elaborate a little bit more and express things further. They started down this path of expressing how they felt demoralized by the six month release cycle, and that they were basically unable to ever catch up because their minimum turnaround time for certifying a new release of open stack was six months. That tended to be more like eight or nine months. So basically they are somewhere like six cycles back. And that means they'll never ever catch up unless they are able to make drastic changes in open stack or how they operate. And if you think about how that sort of problem space, that's really a business problem. But it should have some feedback into the technical process and development, because if we're moving too quickly, or we're trying to still move too quickly, then we're doing them a disservice and making it harder for them to adopt and use our software. And we're actually increasing their operator burden as an operator, because there may be some things or fixes that they absolutely need, but they now having to keep his back ports and they're having to build development around it when they could just get the latest version, but they can't get the latest version, because it has to go through the certification process. So this operator express this, and then additional operators kind of start chiming in like yes same same exact problem. It takes us six plus months to certify, and it's a risk management issue. And then developer start chiming in with frustration about the release cycle myself included. And a lot of this just that it's a driver of a lot of frustration both sides of development and operators that we try and scramble as much as possible to get a six month release out with latest latest new features. And when those features may never actually be able to be loaded up and used by some of the people even developing them for their own environments because they'll probably have to cherry pick them down and run them locally which increases their technical debt which makes it harder to certify things. You can get hopefully get the picture at this point. And I hear, I see Terry wants to speak. So by all means Jerry. Yeah, I just wanted to give some perspective on how we basically solve that discussion last time we raised it because it's not the first time we've been we've been discussing this and probably because it's not like there is no great solution to the problem and but maybe the trade off needs to be revisited and it's it's important that we ask that question regularly. Last time, the way we we solved it was by saying that basically fast forward upgrades would allow deployments to skip releases and to skip certification of certain releases. And therefore the pressure to upgrade would be less. I suspect that from the feedback you got that did not really happen or like people feel pressure to upgrade to every single release or like something is not working with the plan that we basically come up came up with two years ago or three years ago. I think it's part of it and I, I think it does come down to how organizations perceive risk management. Like I've talked with some non open stack folks over the years, it banks specifically. And they always go major version, major version, they never skip releases, or they never jump anything. Which means that they are basically stuck in this certification process constantly and having worked in litigation services for myself. A long time ago, we basically did the same thing. And we did it because we were burned by firmware, we are burned by software. You can't test everything in every matrix. So we ran into issues and as such we decided we had to slow down to do the right thing to keep our services reliable for our customers. And we at that point just couldn't skip anything. It increased our pain dramatically. And that's a real troubling. I mean, I like, I've certainly been burned by that as well in the past, but just because other other projects didn't take too care there and and make it difficult, I guess doesn't mean that you should lose all hope in the world and, and assume that it's always going to be that way but I guess I'm really not sure how like it's one thing to slow down to slow down but I'm not really sure how that actually ends up with people getting anything faster, unless, unless people are just like so dead set on deploying every release and you know they're working on Okada and they refuse to go through, you know, any skipping at all and so, you know, if it if it always takes them seven months to deploy a six month thing they'll never catch up but I mean like that's just that, you know, kind of bonkers. So if we if we double the time, or have our release velocity. So how are they going to get things quicker, like features and things. So. Okay, I guess there's a couple things we should probably consider open sacs not seen as the shiny new. It's more seen as a stable stable reliable useful platform. So they don't feel the need to, they don't feel the need they absolutely must consume things that quickly. So that's the it's a double edge sort of sorts there. But that's the people really need these things and there are a ton of Okada clouds still out there. There's a ton of rocky clouds still out there there's people slow are solely upgrading, but it's just if it takes them forever to get to the next version. And the solution as developers is to slow down. That does free some of our capacity up to be one more coordinated or possibly do intermediate releases and allow some of the operators that do need those new features to get those features much faster. It just it does come down to everyone's risk tolerance level and how they can mechanically move through their processes. And I, I can see the pain point that you bring from other users but I also see. I mean, this is like there's no right answer. If you go yearly you have the other issue of if I want to line something an open stack, and I'm an organization that 100% relies on upstream doesn't like to cherry pick things down. It's a different way to year now before I see my thing arrive or show up in the upstream stuff so now the people that want to move quick are the ones that have to be the ones that sit with back ported stuff. Meanwhile, the kind of slower people, are they actually going to start getting faster. I mean, like I said, Terry mentioned fast forward upgrades was like a thing or like a framework that was trying to help them accomplish that. And it didn't. So is the problem, the resources or is it like the framework like maybe even if you did three year release cycle so it's still like, it's still not people not upgrading. It doesn't sound to me like FFU. I mean FFU is just a plan anyway it's not like a thing you download and install. But it sounds to me like there's some sort of fundamental misunderstanding about whether it's safe to do that or not, or, or whatever I mean basically I think what we said by by declaring that plan was that we understand that people are going to do this we're going to do our best to not burn you and just randomly change something in the middle of of a thing that doesn't fit. You know, these things like we expect you're going to run database migrations and data migrations for each thing, but outside of that, you know we're not going to completely hose you. That should be our declaration that, you know, if we break it, we, we own it so it's it's safe to safe to take this approach and that that gives us both ways right it gives us the fast way and the slow way and so I guess somewhat tested to I think that's like how the red hat open stack product, like, I think it relies on that or does something like that. So, yep, and and and triple O I think has that stuff built in so yeah you could you could certainly choose your releases carefully to align with somebody else that happens to be doing a lot of testing and, you know, maybe do a maybe do a good job but you know even even we don't like always have the same granularity between our jump points so I don't know it seems that if I worry based on what the way Julia described it and maybe it was just the wording but I worry that it's a perception thing and not like well we tried this for six cycles and you know we got burned three of them so we we don't want to do it anymore. I don't think it's it's necessarily gunburned by us, but I think it's a business processes have become such that especially when the business is basically built upon the platform that the risk must be managed very carefully. And so then you end up in, you must only go major version to major version or coordinated release to coordinated release. There's, we could conceivably allow for a more flexible release model where we go okay everything's coordinated once a year, and things can move independently and we keep a sort of sort of a tested matrix. And that way there's still the delegated delineated. This is open stack as a release named release and then there's okay there's these projects are moving at x speed with these version releases and these fixes or changes. And as long as projects as long as projects test at that matrix them, I would think it's okay but again this is it's not necessarily anything technical pattern or process is going to fix it's going to say comfort level of the business that uses the software. I've heard this from operators last couple of years where it's just there and they become so risk adverse that any major thing is perceived as risk and thus is deferred or drawn out as much as possible, so that they are comfortable with it. And usually those processes take months. So, I mean I literally had a platform I was using a prior job. We were two years behind. Part of that was businesses risk management the other part was our certification process to get that that hardware and software combination into the into the data center. So, it's not like we're the only people with this problem to As an operator, I relate a lot with what Julie is saying. So, in a large organization that that deploys a lot of open stack projects, we basically say that every day is an upgrade day, because we are always trying to keep pace with all the releases of the different projects. And we try to do this by project so try to not overlap upgrades. So, this is very hard. However, I also understand the other side that when your release maybe will not solve the problem for all the other clouds that are still running or cut the rocky. For me, the question is why these deployments running these old releases still keep doing it. It's mainly because they are afraid to upgrade. And it takes a lot of time to test all of these things. One thing that as an operator I eat a lot is that the deprecation period of something in open stack. It's usually to release the cycles. Am I correct? And of course, if you lose the boats, basically then it starts to be very complicated. And when we are upgrading and we see that some of the configuration options were just renamed, but they still keep there. This creates a lot of friction for operators to to upgrade. Because then they need to deal with all these different things is not just upgrading packages. It's all these different configuration options that they need to figure out, even if it's in the release notes. But this is very, very hard for large infrastructures with a lot of projects. So for me, easing the continue to try to simplify the upgrade process will be the key for all of this. I tend to agree with that. And I think a big part of that is the fact that we've been very liberal in how you deploy open stack. We don't have like an open stack deployment tool, and that is the tool to use. And I think the other projects like Kubernetes has a QBDM and that's just like something that just like takes care of everything for you. So you don't have to know about all the details and everything. But having said that, I think the sticking point to what you're saying is maybe an idea could be something where, you know, you're doing upgrades, you know, projects can automatically. But that's the thing, right? Like if a project could rewrite its config file to have all the updated values, that's great. But guess what? Everybody's using config management at large scale and your stuff is going to overwrite it the second that the project does that, right? So it doesn't help that much more. I don't know. It just seems to me like, you know, open stack is work. It's not, you know, just put it on and it just does its thing. It is and it's not necessarily technical problem. It's a more of a human and perception problem that we have. And ultimately comes down to risk management in the business and their tolerance for it or intolerance. And at the same time, I know as a developer contributor, the six month cycle bothers me to no end. It's frustrating, but What's frustrating about it? Like is it the coordinated release process and the lack of really basically get told we have to do X, Y and C and it's like, seriously, we're doing it and we're adhering to trying to coordinate as best as possible. But you didn't you didn't look at the thing and see our other other things and our processes and how we're actually different cycle really cycle model. And it There's a lot of a lot of context that gets lost in that. And it's, it's for me as a as a project leader, it's frustrating to Yeah, so from from our release management perspective, one thing we could do is encourage everyone to do intermediary releases and then only do a coordinated release every year. Because that would have the benefit like we've we've not listed all the benefits of switching to one year releases. It's not just like reduce upgrade pressure or certification burden, but also from an upstream perspective, it would result in less stable branches could basically support older open stock releases for a longer time because you would you would have less branches to maintain and all our efforts to have like longer term specific releases be more like a long term support and others be short lived is not really worked either. Like everyone is maintaining a bit everything. So in the end, if we did Like what would be the main throwback if we adopted something right there would be only one coordinated to release every year, but we would still encourage projects to release more often so that features can be put out for the people that are waiting for features they could upgrade just one component and get that feature pretty fast. And then you could you would still have like a good cadence in terms of engagement of your teams, but you would be free to define it however you want rather than have to adhere to these six one cycle which might be too, too often or not often based on on the stage of the project and so I guess the drawbacks we would still have with that model is is the tested upgrade matrix because you would basically have like two styles of upgrades people that would do all the intermediate releases and people that would upgrade from stable release to stable release. But I guess that's already the case for like all the project that are already in intermediary releases release model. So what's your, like, what would be the main drawbacks of that, like, trade off, I guess. I have a small kind of question of your my theory like I, I like that idea but how does that work when it comes to integration, because if let's say, you know, ironic wants to have a quicker pace and really specific feature. Now all of a sudden like I don't know ironic 12.0, which was not the coordinator release version it's 11.0 is compatible with Nova 11, but not exactly anymore. How do we figure that part out like can you make these intermediary releases but they have to be still compatible with the rest of the stack or So take the Nova case because Nova consumes everybody right so you've got, if you've got basically two versions of Nova that people can come from right. You've got to you've got to test upgrade from the last coordinator release to master and the last intermediate release or whatever you know your six month unnamed thing is to to master. And then you've got all of the projects that Nova depends on that also have those as well so, you know, sender starts to go off and, you know, has released a thing but you know we need that new attachment API for our new code but we have to be able to operate you know with you have not upgraded the old one yet and maybe you came from the older older last named release of Nova so you haven't run the data migration yet that fixed the connector in our database for the new, you know, sender it like super much The testing matrix, whereas right now we have, you know, everything coming from last coordinator release to master and, you know, last master but that delta is really small and I just, while I have the green box around me I wanted to unpack one other that Terry said which was about the number of stable branches I that argument doesn't resonate with me at all I guess because that the ability for us to support back a certain amount of time does not really match to number of stable branches to me. But it's just Delta right like if nothing has changed in the Nova console service, I can backport stuff to, you know, full some no problem right but but if if stuff has changed is when it starts to become a problem and so you're still going to have two years worth of change. You're going to have fewer, you know, get cherry pick commands to run but that's not the difficult part the difficult part is. Okay it was a big conflict here I think I've got it resolved but I'm really not sure and the tests that run on this two year old release have been pared down so much that I'm not getting a lot of coverage anymore because of all the external dependencies and risky so I guess I'm not sure that that really gets us much in terms of, you know, more more stable branch support. I actually agree with what you just said rearing stable branches. Still branches are my nightmare of sorts. The delta is not that much it's everything else breaking. Well, yeah. That's purely time based right like yeah that stuff be breaking. Yeah, we go if I try and go to Queens it's good luck. If I need to go to train. That's usually pretty easy, pretty reliable. But yeah, it's, it's all the other external dependencies changing and breaking. And the, the test, you're right about testing model that's absolutely something but it doesn't necessarily have to be required or mandated. It could be you're on your own voyage at that point and the community could be doing the best possible job, or the community can have the tenants, or the project have the tenants where they, they do not break things or they communicate how they break things or change things. So it really comes down to what this project do, and how do they do it. And as a project that has been doing this for a while. It's actually not horrible. We, we, we really live live and operate by tenant of not breaking things. Well, I think we all do. I really, I really don't think that all the projects going off and kind of doing their own intermediate schedules and confusing the, what the picture looks like is going to increase the clarity for operators and I also think that none of the projects that I have a strong say in what I recommend that we claim to, you know, have these intermediate releases and a coordinated release and not test the upgrades because that's where the stuff gets really hairy and it just 100% will massively increase our testing footprint and our CI infrastructure requirement footprint because grenade jobs are expensive and, you know, we're going to have to, we're going to have to blow that out a little bit in order to have any sort of confidence for operators that, you know, this, this works. So, I have an idea which maybe kind of relates to some of the stable branch discussion that we were talking about or the stable policy. Maybe we could allow projects to backport features and allow them to have to make like not a major release so the middle bit is that I'm completely forgetting right now. It's a minor patch update. And so they can go ahead and do it up and release that. And so as long as they continue to integrate properly with the rest of open stack within that coordinated release then they're fine, which then allows projects to ship all the features that they need. And not ship anything that breaks any other part of open stack and if they want to ship something that breaks another part of open stack then they have to do it in a way that's backwards compatible so if let's say ironic wants to publish something that breaks the version of Nova, because it has a different API or something then Nova needs to have some sort of compatibility, so they can interact it with both, or something like that and they can only remove it. Once the next coordinated release would happen and so that way project to move at their own velocity and they have that kind of like two points in time in a year where they got a full reset on being able to start with stuff that break, because they'll they'll assume that it'll be the same kind of serious project that's running. I don't know if that. I don't know if I explained that in a good way or somebody can tell me if that was a clear. I think I lost you there. Isn't that what we already have with projects that that go with a non coordinated release they release when they want and they can get stuff out. I think they can't, they can't backport and add like features to older releases. Oh, you mean backport features to stable branches. Maybe I'm not too. Maybe I'm not into the whole stable branch stuff in this specific case but I'm assuming they're kind of releases off the stable branch or like for intermediary releases, you basically just release features and whatever, like whatever you want and then we pick the latest intermediary release to be part of the coordinated like six months release. So you can do that what you can't do right now is backporting features to like a previous stable branch, a previous release. That's more like a choice I guess it's it's a bit orthogonal to this discussion but it's it could clearly be changed like we we just said we just communicated that stable branches would only contain very safe patches so you should be able to upgrade those really safely, but we could also say that they're more like moving objects and but then the pressure to backport things would get potentially messy. As everyone would want their pet features back to their pet 20s. In some cases like, like no IPI wasn't in things we have like, like, it's not possible. Well, for API later things to backport right and RPC and database and objects, so I mean, if the goal is is not to make Okada, you know, live forever. I don't think that backporting features back there is going to help that right because that's that's what people are just going to do they're going to. And yeah, they're going to be like look I'm on Okada I really really need you to backport this thing from Zena and sweet now I can stay on Okada for another 20 years. Yeah. It's not a good idea to enable that. But there's there's there's somewhere we need to find a happy medium to in all these different contentious things. And that's kind of why I wanted to bring the discussion. So noting. Oh, sorry. Yeah, if I like to say like moving to feel for that topic but if you have a little bit of the things related to go ahead. I actually had a comment in the YouTube stream. Basically, if I'm interpreting the comment correctly. Basically, they end up putting efforts to upgrade on hold when they're building something on top of open stack, because they need a stable and changing platform. And then they run situations where they cannot afford the time and resources to do the upgrades with the regression testing while they're doing the other work. So it's kind of there are many people with multiple moving pieces all at the same time, but that that's exactly projects. That's going to be exactly the same thing with one year releases except the releases are going to be twice as big and have twice as many destabilizing things per release right so they're still they're going to be even more afraid to upgrade when they need things not to move when when we've got a year's worth of stuff packed into each scene right potentially potentially I mean it everyone's risk level and tolerances completely different. I don't see how that doesn't. I mean that that all that does is raise risk, it just makes the risk less often, in my opinion, I rather than having smaller stepping points that you're like okay this is a reasonable chunk that I can, you know, consume and upgrade to I feel like you'll end up with a huge chunk of stuff. So instead of a six week upgrade cycle twice a year you'll end up with one 12 week cycle and everyone will hate you. I think the problem is people are running six plus month upgrade cycles. No I understand doing their regression test. But I think this is where the where things, the disconnective work, a size of units and effort versus the way a entity being a business or an operator operates is if they, they can't necessarily consume the smaller chunk because of policy. Like I know, I know people that live in organizations where you cannot do a minor release, you must do major releases only. And you can only do major releases once a year. And these, these things seem like arbitrary crazy things, but how the business has chosen to manage its risk. So that cascades into everyone's daily life of how do I manage my cloud. And when the, and when that moves to two years should we slow down to two year cycles. I mean, we've provided a way to not have to consume every every granule, and we've, you know, we've said that we will own it if it breaks. Obviously we're not going to meet everybody's everybody's really I mean I have at least one customer who can't hit a 12 month certification cycle. And it's very specifically just not going to happen. And they're doing FF use so it's not the six month thing is not a thing but I mean like that's a significant chunk of even a slower coordinated FF you to FF you cycle. One year release isn't going to help them either in terms of fitting in all of their certification stuff. I guess I just not sure where it ends. I think the conundrum ends up being once an operator is at such an infrastructure size where we're talking thousands of machines. Their risks starts just dropping to nothing. They have no risk tolerance. There's goes to the roof and they have no risk tolerance because we have two of two people with thousands of boxes here. Right. I mean, Muhammad and Belmiro have a few boxes each, I think. Yeah. Time check we have around 20 minutes left so you're discussed a lot about upgrade and stable thing. So let's cover a few other points we have on etherpad. I'll go one by one and if I miss anything, feel free to interrupt me. So first is on line number 160 about the foundation marketing effort. So just something the current release model creating a support foundation marketing effort or it's just idea like it can be more aligned and can be helpful. I think that something that will be hard. I mean, from the marketing point of view, I don't know if it's going to be, you know, probably not a thing to say but you know marketing will always have things to talk about throughout. But I think one of the hard things is when you have such a long release cycle. It's kind of hard to keep people engaged and have some sort of like exciting news or like hey this thing came up or this thing just happened or we're going to do this thing. So it kind of just slows down the things you can talk about and especially if we're not also releasing new features then well you know there's not much to talk about other than like, here's this thing that we got released eight months ago that we're talking about today because we don't really have much to talk about. I feel like that's one thing the other thing is also the PTGs are also going to like be much more split out and we lose that continuous cycle of like staying up to date. That one is the marketing version of my comment below about release notes right the release notes becomes so large, double the size, your marketing thing becomes half as often and twice as much right so you probably, you probably only get to like put the a certain number of bullet points in front of whoever reads the marketing stuff every time you do that right so you you just literally, you can't be like alright guys well. Instead of reading this one sheet, you know we're going to ask you to read two sheets because we've slowed down our thing right now they're going to. There's going to be one sheet and like half as much stuff is going to get mentioned to people because we're doing it half as often. We actually see that today with ironic we have lots of smaller features that go in and when it comes to marketing it we get three or maybe. Yeah, kind of do a high level mention of rough of things but it's like the list of release notes is already a book for us. Yeah, yeah, so I mean it. If we end up doing intermediary releases then we could we could like explain some of the features that were released during. Like I wouldn't I wouldn't sweat it too much with the marketing part because it feels like first like today we're pushing less changes than we used to back in 2015 where when we had like features all over the map so in terms of quantity of features to like present in at the at the end release marketing meeting. I don't think that's that different. It's more like how do you keep the engagement throughout the year than the massive amount of marketing news you have at the end of the year. So it would be more that like people lose sight of open stock for one year and all the people that think it's dead will will only get will wait a year before realizing they were wrong where rather than six months. And the other thing is like what when you do those. Those intermediary releases we could have like if there is one feature we want to we want to hype or project we want to show that they are making quite progress we could we could basically talk about that project in particular, but that brings us back to the testing metrics and if it's really a no no and there is just no way we can make it work then then then I would definitely not support switching to one year releases if we can't have like intermediary releases to get features out to users and get keep the news cycle on exciting and people in the cadence of doing things. It's difficult to do one year releases without nothing in between for keeping the excitement outside but also inside. That news cycle is not just important for engaging people via marketing stuff it's also important for keeping the community engaged and and whatnot right I mean like this is why this is why two weeks sprints are an agile thing right to like keep everybody moving keep everybody synchronized keep the the goals, the goals and the achieving of the goals kind of within. Within you know you're kind of like temporal frame of reference and everything and a year is a long time you know I mean to go between. There's milestones, and even if you see like the number of full time contributors are reducing the first day by day so if say for example in my company I say okay I have one year for open stack to do all these features and all. They will obviously ask me okay let's do some of the other stuff for downstream. So that's the very much valid point for non full time upstream data. Yeah, and thinking a lot as well as like well. Sorry, I know we're talking about the marketing stuff that definitely I think is important is. You know, people, most folks that are working on open stack or salary and they're working on things. And so, if it takes a year if they're working on something and you know they decide to move on and do something else. If they're working with a year, you might end up with a bunch of stuff that's half baked, whereas with six months it's more likely for things to be driven and finished up in that six months cycle but I don't have related that is. Yeah, that's true. In other respect we see, like there are few comments on the pad like for the, how much process we might need to change if you move to the one release cycle, like deprecation policies or stable policies you already talked about, and maybe about the testing and how much like we support as per the stable versus extended maintenance and all this stuff. And obviously the ETL and TC term that's also leadership is one of the big question for us, because in every cycle we face the leadership issues for many projects, and in many of the project peoples or ETLs are volunteering with their extra effort. So with one year release needs a one year commitment from them. So that's also one aspect. Maybe leadership because we can have more issues with this one year release of maybe less so that's my point I think we might face more issues in that. Yeah, I mean that's that's a very reason that's a reasonable point but you know no one's forced to stay for a year if they decide that they want to move on and things over they can. So I think that that should be the. You don't like take a site you don't like take a stand for ptl thinking I'm going to quit halfway through right you're like, am I going to do this right now it's a six month thing and we're not able to get people to even commit to that or or stick around for it. I think it's a big problem I think that you know it's going to be a it's going to be something where we have to fundamentally change the mapping if if we're going to, if we were to do something like that and then of course TC is twice as long and you know we know how that went so yeah I feel like coordinate it again if we do in intermajority releases you could have like a ptl that is tied to a shorter cycle doesn't necessarily have to be one year. Again, I'm not sure it. It's definitely a problem if we just do one year releases, but if we manage to somehow make intermajority releases work across the board then it's it's not that much of a problem because the only thing special about the coordinated release is that it's it create we maintain a stable release out of it it's the only thing that makes it special. Well, it's also the high impact. Most important potentially, you know, getting all the ducks lined up on the important release so. Yes, please, please pressure because that's the important one and you need your future in that important one. There is like friction everywhere. I'm not saying it's the miracle solution, but it, I just don't see the like moving from six months how we do things currently to one year. I don't see that working for all the reasons you mentioned. And I could see like a model where we would, we would have intermajority releases and one year stable releases I could see that work. And it has drawbacks, but maybe the metric testing metrics problem is just not something we know how to solve in which case that's also a no no. Well, so me as an operator now if that's what we start doing and I need to upgrade. I'm running a survey and I want to go to Victoria. Do I go to tip of Victoria, or do I have to go to the intermediate release then like victorious first release like, what's my plan at that point because right now it's very straightforward. You just go to the next release, and you're good to go. And also do you. Are you willing to upgrade just your sender to one of those intermediate releases to get a feature or in reality do you wait for the fully coordinated fully tested. You know everything matches thing because if you, if you wait, then all we've done is slow that down the fire hose. If you do. We've got all the potential, maybe additional documentation or gotchas with like okay well how do I do I need to tell Nova that you know I've got, or you know, now I've got a new a new sender and old Nova so what's the what's the impact there. That's part of the testing matrix thing to Terry is like. And sender releases a new thing that they expect to be used as part of their, you know, more aggressive intermediate release they've also got to kind of assume that people are going to run that not with the kind of somewhat matching, you know they they landed patches and Nova and sender to go together but they need to make sure that the documentation and the, and the prescriptions also cover what happens if you didn't upgrade your Nova because I only upgraded sender to get that feature right so that's like the human testing matrix part. Well, as an operator, it's an operator justifying intermediated releases, it will be very difficult. So you have the stable release. If you have something then in an intermediate release that you really want. It's a huge risk. I think most of the people will just go for the stable one. Which, which effectively means we're doing that additional work in the middle for not a whole lot of gain. And if everybody slows down to the one year cycle anyway we've we've just slowed down our delivery. Yeah, and one more point in, sorry, in that we have to think about the hierarchical dependency that also like for no other depending on the sender for one feature but sender depending on someone else for another feature so that also make it more complex. Yeah. So here's a here's a possibility and I'm mentally going back in time. A long time, I'm thinking, one of the best things I ever did in the project was we create a tenant of the user or operator should always be able to upgrade from any version we release. Now, that's hard to actually test, of course, but somehow we got into this pattern of we only want to support have upgrades from within last from the prior release or two releases or three releases back. And that meant that upgrade code got deleted. And that's not ideal. I think one of the things that we could do is say we, we have a tenant of you should be able to upgrade from, say, say Victoria or train to whatever the next F releases or whatever, and be able to do it seamlessly and it just work. I would love to be able to say that too. But I mean, I know, I know it's it's it's an ambitious thing but that would also give people and organizations a much higher level of trust in the software. It will also change how we how we make changes, but it won't give them more trust if we can't test it and and if we can't test it it's not going to work. Nova, Nova has so much larger of a problem of this than everybody else because Nova runs on thousands of box boxes and a deployment where most of the other services run on like three and there's data that has to be collected from those boxes. Sometimes during an upgrade when we need topology information that only that box can tell us, like, not not being able to assume at any point in the cycle that we have got information collected in the previous cycle or the cycle before that. You know, just means that like all of our database columns have to be nullable forever and we always have to do, you know, an if none check everywhere and then, you know, you think you can make a call an RPC call to a compute node and that it's going to have this available and and it might not I mean it just it just introduces a massive amount of debt into the code and when you run on thousands of boxes and each deployment that's really really really substantial. I also also don't know if that will even like resolve the whole, it takes me seven months to certify in the release of six months like that's if that's kind of what we're like, I think if you focus on what are we trying to solve is, are we trying to help out some of those developers that can't keep up. Or are we trying to help our developers ship code faster or like what's the root thing that we're trying to resolve. Well, I I think that Julia is exactly right that we want we want to not lose trust right we do not we do not want to give them any more concern over the upgrade process and and to me. Any upgrade for any upgrades are hard anything that's not tested we should assume is broken like anything and the larger we increase the scope of things that we say will work but that aren't tested is trust is trust goes down in that equation to me. Yeah, no that's a good point and I mean if people were feeling already pretty skeptic about doing an upgrade. And they run into an issue that hurts, they're probably not going to do it next time or they're going to try to dread it as much as they can because it is as painful. Remember last time what happened. And, and so, because we were not testing it, then someone shows up tire see and is like hey I'm running to this issue and we're like whoops. There's nothing there's no test that are testing the upgrade that you're doing. Looks like you're going to have to figure out how to fix the code or something along those lines and so I feel like that might negatively impact. For sure, the perception that upgrades are hard and make it even harder, whereas now I mean, I understand people have the whole certification stuff that they're doing and all this other things that, you know, but for the upstream code it just works and we have tests that are testing upgrades all the time making sure that they're working properly. No, they're still going to be small things that show up but I think that I don't know. I find that what we currently have is working pretty neat. I think my only concern here is maybe that maybe we just need to talk to that organization and see if there's something else we can do to help them. Maybe not be able to certify faster, or talk through them through like hey maybe you could follow the FFU path. And that's the sort of thing that you can go towards and maybe do a little bit more of explaining and talking rather than changing policy to adapt. I think we could literally just start releasing major releases for every release and technically they could upgrade to anything at any time at that point. Right, we can just tell them anything will work and then wait for it to break. So guys we have just I think one minute left for our session so I'd like to conclude us like how we want you to go further on this to want to continue it on tomorrow or we have, if we have called pretty much all the topics maybe we can think about the X and I don't from here. Like as Mohammed mentioned, going towards the operator and get the feedback. So, can I ask a quick question. So, two of our two of our largest operators are here. And this isn't a concern for them is this is this an education issue perhaps that we could address it that way. And messaging, I think that what what Mohammed was just saying actually is maybe maybe a good first step is to let's let's communicate look I don't know if you guys remember or no maybe you have shown up since the full FFU thing but FFU is a thing that we expect to work and we take responsibility to make sure that it doesn't break and that allows you to do the following thing we expect that to work if you want to not deploy every release, or certainly every six months. This is what we expect you to do. We expect it to work, you know, communicate what that means again and again communicate the support that we have for it and see if that helps people be less. So, I think it makes sense and we can start with the mailing list, let's discuss on and hopes meet up. If we still have hopes meet up things. But I think as a first step, I think we can conclude this on mailing list and keep continuing the discussion where we have operators and other operators channel also like hopes meet up or meetings. I think we can come up with some action items tomorrow on this so we can help. Yeah, I think so say tomorrow we can get the action items. So, yeah, we'll continue with that item tomorrow so we have one minute left so I'll handle to Ashley to wrap up the things and present for the next week. Great. Thanks. Well, thanks to you and the entire TC for including us today and letting us see a little bit into your world during the PTG. It was awesome to get behind the scenes look from some of those conversations and thanks for interacting with our YouTube audience. That was great. I'm also hoping to do and ask me anything episode with this group soon as well so stay tuned for that and be sure to join us next week for a recap of the PTG. We'll let y'all get back to your meeting. Oh, thanks Ashley and thanks everyone for joining and this ends our today TC PTG so tomorrow we are going to meet at 13 UTC for for our discussion. We have a break in between. So join us tomorrow for further discussion or anything you'd like to discuss what we have on the iPad. Thanks everyone and have a good evening and night. Thank you. Thank you. Bye bye.