 All right, we're good to go All right, welcome everyone My name is Sean dig. I'm on the TC. I do Nova Dev stack QA things And my name is Terry Perez at the TC and doing all his management stuff we'll be Mentioning in this talk the older changes that we went through over the last like six to twelve months of our Opposite governance of the stack projects governance and and what what we actually implemented there So we'll like introduce the rationale why we actually made those changes What the change actually was the direct consequences of the change and and where we're like sitting right now? and There's a whole lot of information here and people have had a lot of questions So this talk is really structured We're going to try to move through things relatively quickly and leave a Extended QA session at the end so that all of the burning ideas that you've got Things that you might be confused about things you want to talk about we can do then So let's just start with kind of why we got made changes this past fall You know if you look a little bit back at the history of open stack, you know, we started this five-year mission with two projects Nova Compute Swift object store And over time the number projects in open stack grew Some of these were parts of existing projects that split out to have their own separate teams become a little more modular Other things were new good ideas that people had that fit within the realm of open stack Like interesting things that you would want to do in or with a cloud over time We built process around that to deal with the fact that there were so many of these projects coming in and we had a formal incubation cycle And things matured and there was this ladder that they went up Where that left us was this picture, which is in every open-stack presentation ever And explains almost nothing because it's a big complicated mess, right? And this is where we had gotten there was a lot of projects in open stack and the process of adding something new meant because of all the coupling and all the interaction patterns a really sort of Interesting and hard puzzle game to figure out how this came in and interacted with everything else correctly all at the same time We have effectively gotten to a point where Our eventual consistency model was not working at the scale of the number of projects that we had And and that is what is what we call the integrated release that was the one and only product of our community was this set of core projects that we would release every every six months on a predictable schedule and As the as our community grew We encountered these products that this integrated release was too small and too big at the same time It was too small to express the the rich diversity of the projects that were Naturally appearing in our ecosystem and and that were part of our community on one side and and on the other It was already too big for anyone to consider installing all the pieces in any kind of open-stack deployment So the other Problem we we quickly had with this single integrated release concept Was that since it was the only answer that we were able to give we started up piling too many questions? Too many questions of which the integrated release was the answer So there were like release management or people working in packaging wanted to know is it part of the release? Is it something I need to Work to release at the end at the end of the cycle is it something I should strive to package to package open-stack in the end? People from QA or or the continuous integration would ask the question is it's things we need to Test together is it things we we that are so tightly Related that we have to test every single proposed commit Against the current state of development of all the other projects and the answer to that was the integrated release as well Then enthusiasts wanted to know is it part of the ecosystem? Is it part of what the this thing this community this summit you call open-stack? You would have operators wanting to know is this mature enough to deploy? Is it something that is left the experimental stage to the point where you can you can safely deploy it? Operators again wanted to ask a slightly different question which was is it stable enough? Is it is it something that you will change completely overnight below me or is it something I can build? On top of the API's you propose are you going to support those API's for a long time if you deprecate a feature? how long it will take and We still had a single answer to that was the integrated release And users wanted to know if this is part of the is this a part of every single Open-stack cloud that they will find out there Is it is it the minimum? Interoperability features that we are required to to interpret between clouds finally developers Well last but not least developers wanted you know if if the thing they were working on was considered valuable was considered first-class And and it's not just developers also their employers would just only fund developers working on Open-stack projects So only on project that were part of the integrated release So we had this chicken-and-egg problem where we were asking those projects that were coming up to be mature stable part of Tests to code take co-gated tested together and at the same time We were preventing resources to get to them because we were basically saying these are not first-class citizens yet and Finally cross-project teams would would try to survive over all this because everything we added to the integrated release We had this inception that cross-project efforts throughout Open-stack would have to support them So documentation would have to produce documentation for everything that is in the integrated release or release management will have to Basically handle manage the release process for all those projects QA would write tests I mean there was this assumption that that anything we add to the to the integrated release will have to spread support for it and that created this situation where those those teams are really constrained in resources and Anything we added just just put too much strain on them So we were at this at this stage by by by the Juno cycle we couldn't grow the integrated release anymore We couldn't reduce it anymore either Any project that we would add would have to be a positive answer to all those different questions that people use the integrated release as the single answer for and So there was a lot of pressure new projects still coming 100% of the technical committee activity was was just handing this incubation later and We could not just accept new project at the end of it So there was this bottleneck and time was really ripe for a reboot of the system. So this realistically what we're looking at was Refactoring the community, right? This is what you get to with any large software stack at some point you have to decide What? It's time to refactor. It's time to figure out what's working. Don't break that Figure out where your bottlenecks are and eliminate them and then go and realize some interesting new patterns had already emerged organically down underneath you and grab those things and replicate them other places and the beginning of this Refactoring the the key fundamental sort of switch in your head for this is changing the question of Is this open stack to are you open stack? Because it turns out Open stack is made of people Right we kept having this idea that that we were blessing Maybe functional areas but turned out what it really was was teams that were delivering an interesting thing and that had a community around them We have a set of tests In our mind going forward with the Big Tent about what are you open stack means? There's sort of four basic points The first of which is do they align with the open stack mission, right? The open stack mission is to produce an open source ubiquitous cloud platform That works for public private and hybrid at small and large scales all scales Is this a project which helps put forward that mission? Is it something that? is a sensible service that would add into that a Tool that helps move that forward something that consumes existing open stack in an interesting way that's part of That thing that we're all trying to do together There's the Do these projects these groups of people do things in the open stack way we have an open stack way It's the four opens right is it being done an open source license is the community being run in an open way We have this mechanism for Voting for leadership And the way that contributions work in projects is the development being done in the open through our toolchain Garrett the way our core review system works the way our upstream testing works as the design being done in the open just being done informs like this on Public archived channels and public archive mailing lists so that anyone has the ability to participate in that It's not happening, you know in someone's hallway somewhere at just one entity and Do these projects? Coming in do they ensure some basic interoperability with the rest of open stack? Specifically, you know the minimum barrier we're talking about here is if they have users They have to interact with Keystone like you clearly are an open stack if you do that and you should Try whenever possible whenever it makes sense not to duplicate something else there But sometimes there's more than one answer to a particular problem and that's okay And allowing that and encouraging that our community is fine And lastly, you know, we have this technical committee, which is directly elected by all the technical contributors if you want to be Are you open stack you have to say that we'll take guidance from the technical committee when it's important There's sometimes times when the TC has to Work across all the projects to solve a thing for all of open stack together So the the other side of the change is what we call tags And so on one hand we make the open stack tend to way more inclusive of the values new Projects that are make up our community, but at the same time we want to provide clearer Information about those projects so that you can more easily navigate this this this tent and Tags are meant to provide the information to downstream users of open stack They are not meant to be project badges that every project should collect to be a good open stack citizen all those tags are must be driven by one question that is useful for the consumers of that ecosystem and provide a precise answer to the question rather than lump all the All the questions into a single concept called the integrated release and hope that the TC will make the call if you meet all of them or not So the idea is we no longer need to have a single answer. We can be extremely precise We just need to find the right questions now, which is slightly different the the consequences of the direct consequences of those two to approaches Well, it doesn't change that much The on the fact is there were open stack projects beyond the integrated release already There were projects developing on on the open stack. They've mainly is participating in the design summit participating in the conference here Behaving the open stack way following following advice from from the rest of the community developed really in the open and They were just considered second-class citizens. They were not part of the integrated release So they would not get that design summit space as much design summit space as the blessed project But they were existing and we are just recognizing the fact that they were existing already and they were open stack already the the other Non-consequence is that it doesn't really dilute our ability to focus on on important things That's a common fear with with the introduction of this change. Is that like more project that means you will dilute more the Especially the cross-project efforts, but it's quite the other way around because cross-project efforts already had to Support all the integrated release and they had no choice in that They could not they had to support equally anything that the TC would add to the integrated release and that created a lot of strain and we Move that the other way around where those cross-project efforts now can choose to focus their efforts on a smaller set of project that they They feel more comfortable working with and that's again a change that was already started to be implemented like the documentation team no longer Created documentation for for all the projects at the same time. I think shine will touch on that. Yeah To a dueling flippers so An important part of this is the horizontal teams Moving to a more self-service model. So as Terry said, this is already starting to happen And it was one of these patterns that was emerging that we tried to capture and sort of formalize going forward The horizontal teams just couldn't support everything that we were that we were trying to do so instead started to pivot and be as responsible for building some sort of framework For which projects could provide their own service for that particular function documentation testing even dev stack integration and let the horizontal teams focus on Maybe a smaller set of things plus an easy way for anyone that's in our community to plug into it We have a whole bunch of examples of this that have already happened You know the the external plug-in interface to dev stack was one of these ones that that I was working on to make it Just super simple anything anybody wants to plug in. Here's a hook and a stable interface and Now you can have a very easy development environment That integrates your thing with all the other parts of open stack that you want to integrate with The documentation team, you know in this last cycle had previously been many maintaining API docs Themselves and their own repositories like all the Nova API documentation is now back in the Nova tree where it probably belongs And it's now a Nova ownership responsibility, which is what it should be right every project Needs to kind of own all all the parts of its artifacts that come out And you know can ask for help can ask for help with tooling from other from some of the horizontal teams, but Has to get into this model where each project is responsible for its own vertical to become a durable element This gets Even more interesting. We started talking about testing because as Terry said before our our old testing model was we take everything in open stack at every commit level and Throw it all together and run a bunch of integration tests and like thumbs up or thumbs down And that gets really really complicated to figure out what the thumb down was when we're now talking about 40 get trees You know, this is like to think of this as testing with clay, right? If you are a sufficiently skilled artist you can build really really cool things out of clay and That's what we were giving our community was a pile of clay this new model where we're kind of moving things into Back into the projects We are trying to go with more of this Lego model durable elements that we test in smaller units That then we can plug together and assume that other parts are already working There you are limited by what you can build with Legos, but a lot more people can do this And if we're talking about ubiquitous cloud computing The lots more people is a really important part of this So another another aspect is how do releases look in this in this big tent? So there won't be an integrated release anymore as such at the end of the Liberty cycle We'll still have coordinated release point at the end of the six-month cycle for the project that are willing to To commit to those those deadlines and those milestones But the main change is that we'll will move from managing most of them to Refine processes and provide tools for each project to be able to Produce those releases easier and that should make my job Smarter, which is a good news That the binding aspect is the release the development cycle We would still all use be using a six-month development cycle Even if some project might do intermediary releases where it makes sense But we would still mostly organize everything in six months development cycles between between design summit that one of the key I think question that is still opens is how do we redefine from a marketing perspective the communication around releases? because those used to be Predigested in like number of new features or number of bug fixes or number of contributors number of new projects and Those metrics don't necessarily make a lot of sense in this new world where you add new projects so we need to we need to revise the Marketing work around around release time because we won't be able to communicate the same type of information as Supposedly closed set of projects would follow you to again It's not it's not a complete change because we were adding projects before It's just recognizing that when we were communicating in number of features number of developers number of projects That might not have been a real metric to to judge this the health of up and stuck from Then present and future that's yeah So since we got rolling with this new big-tent model And we defined this much sort of simpler definition of what was open stat are you open stack and We've added a whole bunch of stuff We've added a number of things that were Look kind of like traditional open stack services rest apis that provides some particular point function you know things like Magnum in Congress and Magneto DB Morano and Mistral We've added The rally project which focuses on benchmarking and this is actually a very interesting case of you know There are some overlaps with the existing QA toolchain, but it takes a different approach to certain things And that's okay in this new model. We can have Related problems or even the same problem being approached in different ways By two teams that are you know working in the open stack way and producing good stuff We have some things that are different than we've ever had before Like there's a whole puppet open stack modules community that lets you There's a whole pipeline for building and installing a cloud using puppet They've been developing for a long time. It's very active base And now this is you know, they're definitely open stack and they're part of this whole thing I will expect to see other operator like elements coming in as well and You know another horizontal effort the security effort which was the traditional open-sack security group and Vulnerability management team kind of building a new thing and is building some new security audit facilities for Open-sack developers to be able to use These are all great things to have as part of our community They've always been part of our community and now they're recognized as such and they always wear open-stack Open-stack people open-stack groups. So it's just recognizing them for what they were already on the tags side We've been looking for questions that are interesting to to provide answers for We've been answering the question. When do I expect updates? How long do I have support? Is this something that will be released every six months by creating tags describing the release model? We've been also addressing the question of the long-term longevity of a project by creating a tag that describes if a team is Diverse so is is the team producing the project coming from a single company that might like change strategy and just drop it Or is it coming from a more diverse community and you can expect it to survive any any corporate roller coaster event The next the next questions we want you to to help answering is where do I start is I see you have this bunch of projects now Is it where where what are what are the bad with what is the base kit? I should I should probably be considering first before before deploying others Will my stuff break after I upgrade ties into the stability question What is the what is the commitment each project makes as to supporting features and APIs for over the long run? Is it is it the project that says well, I can break everything every release or is it the project? More mature project will say well, I will not deprecate your API forever And that type of commitment is not necessarily easy to extract from the current project documentation And will will create a tag for that to describe it more and make it extremely easy to access that information Next more good questions. We will be looking forward The questions that are useful for our community. We want more people involved We want operators directly involved in creating those tags maintaining them And defining the criteria that apply to them. We I talked about maturity and stability That's the first two examples that we want the ops to help us Define the meaning of what is mature and they are the best place to answer that question would and to to define what maturity actually means to them and Finally, we'll we'll have to make that more easily consumable especially by newcomers in the ecosystem Because not everyone knows that they need to look into the governance repository to see where the tags fall So we'll we'll probably work on a on a webs on the open stack.org website to exhibit that information for like general consumers newcomers to the ecosystem and Was that that brings us we are done Except we left plenty of time for questions because there always seem to be some there's a mic in the middle there Because this is being recorded to be great if you had a question if you went to the mic and asked it there Ready? Yes. Go for it. I had a question about Interoperability with regard to some of the deaf core stuff is one of these questions does a project Intend in the future to become part of the what will presumably be an expanding deaf core definition so today it does not and The way that the artifact that's currently in the bylaws around the TC's interaction with deaf core Is a thing that's called the TC approved release which Yes, hey, we even have a slide for it the TC approved release So there's specific wording in the bylaw that the TC has to produce this The There's a couple proposals out of what this is going to be in first iteration in first iteration This is probably going to be what was the integrated release and full stop and the The thing is of what's in the integrated release that the only The only thing this tag is important for is what the board will issue trademark programs on Right now of all the projects in this Over half of them are things the board has not Decided that is trademarkable So we feel like there is a lot of ground to be covered within the existing window Before we decide like something else needs to be added to that and that will probably be a pull model as much as a push But it's true that it's one of the meanings the integrated release had the because since that was the only answer we could give that was also used by the board as the the group of project that might make sense to apply trademarks on and Well, we'll we'll basically Refactor that into a new tag so that it's only used to mean that meaning rather than like have all those piled up meanings So God maybe first just kudos to you guys for working on this real challenge to scale this kind of community I thought refactoring was a great way to think about it Um You guys talked a little bit about The beginning of this some things that maybe the integrated release was kind of tough on it was the only answer to the question And you almost made it sound like well, maybe that was a bad idea why we do it but I think one of the big things that it got us is sort of a known quantity and I worry or I wonder what your thoughts are Moving forward with Big Tent that maybe documentation in one project versus the quality of the documentation or the dev stack Integration in another might not be as good you gave the the Lego analogy, which I thought was great But one of the reasons Legos work really well is they snap together very tightly so the One important aspect of that is we should be able to still know which projects are reaching a Satisfying level of documentation and currently we don't really have how would you quantify that though? Sorry How would you quantify a satisfactory level of documentation? That seems like a very difficult question to answer objectively is it not so There are fine lines in there, but reasonable people can say that this documentation is terrible right, so you know whether or not we're talking about like a Three-grade scale or like a you know, 20-grade scale like how you're gonna rate things or whatever I think it's You're right. There's certain things that require subjective human opinions here about what is this documentation sufficient for operators to Actually use this thing in a real way. Is it gonna be community-driven like people are gonna be doing likes on the thing like yeah Documentation is good on this one or this one's mature or this one has some longevity well So the part of the maturity tag when when we discussed it with the operators at the Philadelphia mid-cycle They they included the presence of a number of documentation guides as one of the metrics to consider a project operationally operationally mature and There are other things like a credibility the fact that you provide database migrations to over Over from one version to another a few other Key aspects the fact that you don't have to play with the database your with your own hands to actually be able to start Launching the project, but can do it with proper CLI or API access that type of stuff that doesn't really Is not necessarily obvious, but if if that group can refine a set of things that like Any mature project any project coding itself mature should should have that's I think a good Good way of looking at it and they will refine it They will improve it and I will encourage projects to actually reach that level of operational maturity which I think is We have to have a base standard and and currently some projects are under that yeah, and there's an integrated projects are under that Yeah, there's a dedicated session in the abstract about operator tagging That's on the schedule here this week. It's I think it's tomorrow and so anyone's it's just in that should dive in It's not recorded that's okay All right Next question How would the ATC status be? Defined in this new model so ATC is defined by the by the by laws by the open stack foundation by laws And it's a contribution to an open stack project So the there will likely be more ATCs That doesn't mean every ATC will get a necessarily I mean the fact that ATCs are invited to the summit is is like a Even staff decision basically currently we it's not even ATC is anymore now because ATC. It's like a few contribute over the last year and This time around only contributors from the last six months got invited so that we could reduce the the number of invitations so the There will be more ATCs. ATCs will all vote for the technical committee, which is like full circle with the constituency and and the elections and but the the rights of the ATCs as far as design summit go might might not just follow up not Every ATC might get to summit or something like this Great more questions I have a question you have a question go for it. What about programs Sean programs, right? so So as our open sack is made of people One of the real big drawbacks about the old model a drawback I don't know that we even really realized fully until we started to the new model was that we had this concept of a blessed Program like the compute program and then everything that had to do anything with anywhere near compute basically they had a global lock on that construct and You know one of the ways that this played out Suboptimally is some of the containers work that was happening where it's like well That's kind of maybe looks like compute. So it has to start and get through this set of gatekeepers Instead of the fact that it's kind of a different thing and it kind of a different API and it kind of should just Crop up on its own right Magnum like under the new model. We just said this is not a Nova scope This is this is Magnum has a thing which is about orchestrating containers within existing open stack stuff And so this program concept in these global locks on ideas Just goes away this is about project teams for groups of people that accomplish a thing and The fact that another group of people is accomplishing a thing that's near that thing It's fine, right? Yeah, and there are the old model. We ended up having Forcing some teams to work within other teams were you know artificial leadership They did not necessarily recognize the leadership of the group They were forced to work in and so that created unnecessary tension and and some project could not like lift the ground Leave the ground just because they couldn't grow inside an existing an existing program anymore So we basically aligned the structure with reality Which is there are groups of people that are working on an interesting stuff And we if they behave like an open-stack project and they've helped further the open-stack mission then yes They're one of us basically Yeah, so on the flip side of what you just discussed you also worry about Delusion of a particular area which has a current kind of global lock like we think Nova compute its synonymous You know what if we have ten different compute platforms or one per container technology and those kind of arise organically then you know It's not as clear to everyone who may not understand the nuances of the tagging system It may not have bought in from the beginning what the differences are and what's the right path for them And are we counting on you know vars to help them figure that out? I mean what is kind of the strategy around then what so we have a provision in the in the requirements for new project teams Which is that you should not unnecessarily duplicate or or compete for with an existing solution? So it's kind of a fuzzy requirement Obviously things that are lower in the in the stack like Nova Keystone would they would take a lot of It would take a lot of effort to convince the TC that we actually need a full new Implementation of the thing the glue that binds all those things together, which is the authentic the common authentication and But as you grow as you move up I your hub the stack like Metric collections we have already have multiple solutions for that and and maybe the market should decide which one is the best or Deployment we used to have a single deployment program And that prevented puppet modules from being developed within the the open stack community just because we had this deployment program that was focused on on on triple O and and It ended up not being the best the best idea in the world. So At least other avenues should have been pursued at the same time and this model Let's us make that judgment call where is competition beneficial or is it actually hurting the the open stack? Platform as well. Yeah, I mean at the end of the day, right the TC still gets a final like yes No on these things we have a set of guidelines so that it should be relatively smooth in most cases And but there's still a backstop of like no that would actually Substantially damage our community. We're not gonna let that you know live under this umbrella We haven't had to pull that trigger yet The stuff that's been coming in has been very clearly. Yep. This this makes sense like let's have it in but we reserve that as shepherds of the community and In one pop one point on the governance is that we actually iterate fast in the rules and the tags and the various things It's not as if we we had to get it right Forever in in and set it in stone and will apply it forever. It's not like the open stack foundation by those that takes Three years to change if you realize that they don't match reality. We can iterate fast on on those things and if Like Sean said we those guidelines are just guidelines for the TC to decide so you could totally See a TC in the next six months that will have a slightly different View on the rules that should be applied They they could they could basically edit that that list and add new things or remove things So it's not set in stone. Well, we'll see how it goes. We'll see We'll see the results. I'm confident that it's it's an improvement because we're basically matching the reality of our community rather than imposing From top to bottom a structure that is does not match how people work So I think it's an improvement, but we'll see With with time. We'll see if if that That's a definite improvement or if we need to iterate and improve again on the on the governance You know what else for the question we got a couple more minutes Just a quick one. Are these tags are pretty these ones or permanent? So the the tags The way the tags work is there if you propose a new tag There's a document within our repository which describes gives the whole context like here's the background Here's why you'd want it. Here's the The criteria for which it would be applied and here's the frequency that this should be revisited Some of these things are are sort of manual Things where it's like we'll do this sort of on a release cycle Others like the diversity tag There's some provision for doing it in a in a sort of more Dynamic way. Yeah, more dynamic way so that's part of Defining the the question that you want to ask You know a tag is really an answer to a question that's being asked and The timeliness of like how frequently that answer should be refreshed is part of that question and we try to Take that in so there's no global answer for that, but it's part of What a tag definition looks like is how often it it gets updated and will that be the TC who always checks whether the Requirements are fulfilled and the tag can remain So that the TC would probably still pull the trigger on the merge just because we own that repository But I expect a lot of tags to actually be refreshed by people that know the answer better than the TC members So if we define an operational maturity tag I I hope that the the ops would propose the changes and we would just well that that sounds about what the tag was Meant to be that corresponds to the tag definition. They apply the rules It doesn't have to be the TC members that do all the updating of the tags I hope we'll be able to decentralize completely maintenance of those tags. It's just that we're we're still enforcing that the Tags that are proposed are validated by the technical committee If only to make sure that they're actually answering a useful question Rather than you know, just be another batch that some project want to add to their collection So the yet is really to and to enforce that every tag is as a rational and is useful for the users and not just for process sake or And why I say that the TC approved for this tag is completely for the process because that's for communication for the board Okay, and who can propose tags? Anyone anyone anyone and and this is really a big part of this was you know One of our bottlenecks was all this information heading through only a technical committee We have a lot of smart people in our environment and they should All be able to participate in this mechanism. So, you know, if you're interested in helping step forward It's just a change on the governance repository and you just proposed it to change as a change in Garrett The tag definition is in the sub directory. He just proposed new one and I think we're out of time. So thank you all for coming