 Very nice Okay, so it's about 11. We're gonna get go ahead and get started Thank you all for coming out this morning and listening to us our talk is herding cats into boxes We're gonna talk a little bit about the release management processes And some of the changes that we're making I'll I'll get into some of the future stuff But first Terry is gonna tell us a little bit about the history of release management open stack So here my name is Terry Carras director of engineering for the open stack foundation and I've been handling release management since the beginning of the project and I'm very happy to delegate that task to Doug now we'll be talking about the history of Release management in open stack a few a bit of a release theory as well So it can be a bit technical then what changed during the Liberty cycle where we are at the start of the mitaka cycle and And where are we going next? So first of all releasing the The act of yeah me that's a no-working You can wait and I'll go for it It's not walking either Interesting Yes It's cute, right? So releasing is like putting a cat into a box. I mean I needed an excuse for this one I mean all software needs to be released and all cats love boxes But it's when you try to put a cat into a box that things start getting interesting I mean putting yourself the cat into the box or try putting multiple cats into boxes at the same time And that's it what release management is about Releasing in in open source is not that difficult. It's it's about putting a version number A tag on the current state of your source code Then you can have automation turning that into a source code table that you will publish but the act of releasing is mostly putting a version number on a given state of your git repositories and Then the There are two approaches to versioning you can go on the pre-versioning approach Where you basically decide early on which version you are going to produce in the end and you march toward that version and Release intermediary Developmental versions like alphas betas release candidates along the way so that External people to your project can consume that start testing Verifying that it's walk. It's working well with your use cases Until you release the final the final thing That works pretty well with a time-based release schedule obviously because you already know when you will be releasing so you can make up a Version number based on the dates The second approach is called first versioning and that's when you decide The version number you're going to produce at the time you release So the good aspect to it is that you can pick meaningful version numbers because at that time you know Exactly what will be in your releases that can let you use major version bumps if you change a lot of things or smaller version bumps if you didn't change that much the The issue is you will not be able to issue developmental Milestones along the way because you don't really know which version you're going to produce so you cannot make a beta or an alpha or release candidate of that of That thing you're just marching from the version you last released and you decide at the time you release the version you You produce so it's in the end. It's it's about You would only do full releases basically in the post versioning mode So what's up with OpenStack in OpenStack at the beginning? We settled for a time-based release model with a six months release schedule We use this year that sequence version numbering and we use pre versions so you we do developmental milestones along the way and Release candidates until the until the end release worked quite well for us Especially at the beginning where QA coverage was not so great putting out early versions of What we were working on out for? Packages to package users to test earlier and early on was really useful Swift was always the cat that would not fix in that box They they were a more mature project. They wanted to do more frequent feature-based releases and so they wanted to use their own versioning system and So they use this this x.y.z Version based on how much they changed they would increase major minor or all the smallest part of it It's not really cementing versioning. We'll come back to that We still wanted them to use pre versioning. We still wanted them to do a release candidates and That created actually tension between the release management team and the swift team because they could not tell us which version They were marching toward and we needed that to produce for these candidates It's worth noting that even if Swift did this different Release model they still released a version at the end of the cycle to correspond to the global Coordinated release we were doing for all the other projects In particular, that's the release. We would cut the stable branch from every at every release We cut a stable branch so that we can backport important bug fixes and vulnerability fixes and That the end of cycle release would be the one we would cut the stable release branch from on those stable branch We would use post versioning. So we would just increment the lower number 2015.1.1 for example for the Pre versioning model or 2.0.1 for the Feature base model. So that worked well As long as we shipped everything Inside the service so we shipped the originally we shipped we shipped the client libraries inside the Inside the service itself but at some point we Wanted to release the client libraries separately The idea was that you should be able as a user to have the client library installed and be able to use that against any Version of the service that happened before so we would have client libraries that would be backward compatible with earlier version of the service So for that we would do a single release channel. We would do Point releases on a single timeline and use semantic versioning on On the client library. So that means the new version would be compatible with every single service version We had before Semantic versioning is a major dot minor dot patch and it's it does the semantics of it are actually quite well-defined and Client libraries were the first to try to follow that model Okay, so the that use of post versioning for the Libraries grew out of our experience with the Oslo program and the the versioning system that we were doing there and Those those libraries started out as code that was actually in the the projects Moved into the incubator and they got copied around But they the intent was always to release them as libraries And so as we did that as the Oslo program sort of matured we ended up with more and more libraries The early ones were using alpha versions They were using the the pre versioning just like the server project So we would do a major release at the end of each cycle, but we would have alpha and beta releases in the middle of the cycle That worked okay until We grew to the point where we had many many more libraries, so as we We stand now we have about 30 libraries within the Oslo program and Somewhere in the kilo time frame we started shifting away from doing pre versioning and starting to do the post versioning so that Basically, we were able to release official versions of the libraries at any point and By dropping the alpha versions we eliminated some of the complaints we had from We eliminated Eliminated some of the complaints we had from developers who didn't want to use alpha versions as they were developing new features in their own projects We also removed the issue that we had at the end of each cycle of having to re-tag all of the libraries with a Final release version and then go back and update the requirement specifications in all of the other projects that we're using them so we shifted To the the the post versioning model and that worked pretty well However, as we started finding more and more libraries in other places we Being managed by other teams We started noticing issues with some of the the releases So we grew the number not just at the number of libraries, but actually the number of teams that were managing libraries So everybody sort of had their client library already each service project was responsible for having a client library to talk to their service But some of the projects started adding libraries that were not rest client libraries and that were used by more than one project. So As Those teams were managing the releases. We noticed that sometimes they would release them at a bad time in the cycle So maybe right before a milestone we'd end up with a new drop of a library that had a breaking change in it or something We also noticed that the version numbering that they were using was not consistent So they were not correctly using semantic versioning rules Which meant that consumers of the library didn't necessarily understand the ramifications of upgrading to the new version of the library So we decided to Recentralize all of the release management for libraries We were still managing the services Already, but we took back the library work and and put that on the release management team And in order to do that because we had so many libraries we had to streamline the process that we were using so we made some changes to Tooling and also to processes and ended up with a a standard Sort of management system that we apply to all libraries across all of the projects and open stack So they release as needed using post versioning The version numbers are selected via semantic versioning rules, which are well-defined We have some guidelines like don't release on Friday because none of us want to work over the weekend if you break things And then we manage all of this by actually reviewing a patch that handles the release request so before we had the review system in place you would do the release by pushing a tag and When once you push the tag it's basically there and you can't remove it You can't unrelease the thing so we we put a review process in front of that so that We can help enforce and make make sure that everybody's following the guidelines That process of managing the the review is Mostly automated although there is actually a human being that sits in the middle so the the author who wants to have a release of the library submits a patch to the open stack releases repository We looked that over and then approve it and it merges Then we run some scripts and that's where the human being comes in so the scripts Actually do that tag and they get repository They go to launch pad and they create a milestone so that we have a record of the artifact and it also updates open bugs Well actually fixed bugs that have not already been added to a milestone or added to the new milestone automatically And then we generate an announcement email So if you've if you're on the open stack announce email list You've probably seen many many of these announcements over the past cycle as we've been automating this stuff and it's Okay, so we we put this stuff in in play over the past cycle or cycle and a half and as we got that working the application developers and the service projects started Asking if they could have a similar sort of process so that they could release their projects more often so that was an Interesting thing that we hadn't considered doing before except for the swift case We had mostly wanted everybody to be following the standard six-month cycle But as we grew the number of projects we had we realized that Initially when we had a few projects, they were pretty tightly coupled and it made sense to work in lockstep But now that we have so many new projects under the big tent rules We don't need and most of them are not so tightly coupled We don't need to couple their release cycle quite as tightly as well. So this gives teams more flexibility on the other hand Releasing in the middle of a cycle and putting a new package out there that that's something We really want to reserve for the more mature teams So we want teams that have good QA systems in place and good Automated testing and that sort of thing to be the ones that take this this sort of role on and not just make Every project work this way so the Problem with with intermediary versioning is that you basically cannot have a common version if everyone is on a different Release schedule with different version numbers. So intermediary kind of requires its own versioning at that stage we We still had a lot of projects with the year that sequence model like every every single one of them except swift and And But we expect more and more projects. We had a high running during the during the Liberty cycle Switching to an intermediary release model. So that creates confusion for for users as they see things that are Released with the common version number and things that are released with a specific version number So they start asking is it part of the release is it is it is it really Liberty if it's version 2.5.0 and So the this confusion was was a bit of a problem and we started considering getting rid of the common version So there were two options. We could switch the project when basically they switch from the common version to a specific version or we could switch them all and at the same time and we were at the same time looking into The abandoning the integrity to release concept and so we didn't really need a common version number that much anymore The value of keeping the common version number But it comes from the comes from the integrity to release day days So we just decided to switch them all we give gave them each their own version number Based on the age of the project based on the number of of integrated releases. They went through So Nova for example is 12.0.0 in Liberty and the car is 1.0.0 in Liberty We tried to avoid giving them all the same Version number because then we would we would be slowly drifting They would be a slowly drifting away from the common version number and adding more confusion in some project would be 12.0.1 some other would be 13 Some would be 12.1.0 and that create that would create actually more confusion So just putting them out based on their age and and switching from the common version to that specific version And let everybody be confused with all at once, right? At least confuse everyone just once So what about stable branches for stable branches, we had another kind of dilemma Because we used to do synchronized point releases We would take all the stable branches in the integrated release and we would bump the version number for them at the same time was a very Time-intensive process especially for stable branch managers To stable branch release managers to to chase down all the different teams that were part of the integrated release Make sure that they backport all the important fixes in time etc etc and And not a lot of people showed up to to help with that. It was a very small team So we didn't look forward adding so many more projects with the big tent and having so many more projects having Stable branches and having to bump them all at exactly the same time furthermore the The what would be actually synchronizing the the release of because they would all be using Different version numbers now So what's the value of bumping them all at the same moment actually communicated the wrong message because it kind of told our users You should upgrade everything at the same time and in stable branches will only backport safe fixes so it you should be able to use any stable stable point version you want with any other element of I mean if you take the kilo release you can take some services to 2015.1.1 some others from 2015.1.0 And they will all work together quite well because we prevent breaking changes on those stable branches So the the fact that we bumped everything at the same time kind of communicated the wrong message to our user Which is you should upgrade everything at the same time, which is very painful So at that stage we say well, we should get rid of them completely. We should no longer do point releases On the stable branches. We still do stable branches obviously We still backport fixes and and vulnerability fixes to those stable branches, but we would not be releasing it again because anymore because there is no real value it's to Releasing a very specific version all the stable branches commits are actually production already unusable because we only backport safe fixes to that So what's the value of Releasing the stable branches But when we proposed that that back at the Vancouver summit There were a lot of people in the room saying we still need to come on reference points because we produce source code the distributions take it values other Packagers come up with something with it and having come on reference points across that spectrum all those people saying well I'm at 2.5.1. I'm using the code base at that version Let's you compare where where everyone everyone is so it it had value for the users basically so we could not really Remove them completely. So the alternative was we should just tag every commit because every commit is usable and But I've created another opposition Mostly from developers this time that saying that we would because we are using git tags to Mark the version numbers and if we push hundreds of tags in the git repositories Then you end up with a fairly populated tag space in the git repositories making it almost unusable by humans So we could not come to an agreement on that and we settled for an intermediary solution Which Doug will expose in a few minutes Is it your turn? No, it's mine So where are we now? At the start of the mitaka cycle. Where are we? currently Version numbers we said that we did not denormalize them and that's how the liberty release looks like it's Nova is twelve dot zero zero swift if two five zero Ironic is four point two point zero Neutron is seven dot zero zero, etc. That can be very confusing and And before we started denormalizing I wanted us to create a tool that would map the release series like liberty to The set of version numbers actually make up the of the components that actually make up the liberty release And so was thinking about a CLI tool that you would type that would give you the numbers and some kind of a way to make sense of this This all those components and all those different version numbers But at that stage we already had the releases repository where Doug mentioned earlier We review release requests. So we already have the mapping between the release series the git commit SHA's the and they the and the version numbers somewhere and So what Doug did is he took the the world repository applied a lot of Sphinx magic to it And that created this website, which is quite awesome So you have on this website the version numbers of all the components that make up the liberty release And what we did is we imported the history of the project as well. So you can go back in time and see what version numbers Were used for previous releases as well Which turned out to be an interesting exercise to try to get a git commit SHA to to austen release because we were using visitor at that time and and the release was not as as as Defined that it is today Anyway, you can you can look back into it if you it's all machine readable because the releases repository is Yaml So if you want to build your own tools on top of that or your own automation reacting to releases It's also possible to consume that as well release models we Spent a lot of time during the liberty cycle to standardize the release models that were actually supported by the release team And that's what we have for for me. Takya now cycle with milestones is the usual OpenStack model with one release per cycle It's pre-version you do a one-time based release at the end of the cycle and As I said before it's great for Project that are young or project that are so massive that they can't really test all the various combinations They can be used because you produce those Developmental milestones along the way that you can give to your users so that they can test their specific use case and check that It's not broken So it's good for those projects don't have that much of a good end on QA and It's also enforces a number of release management practices that we've been polishing over time like freezes or release candidates And so it's a bit of a package that is easy for a new project to opt in Especially when they're in in this young stage the cycle with intermediary model is is the generalization of the Swift model basically It's post version release as needed You still strive to do a final version at the toward the end of the cycle And that's the one we would cut the stable branch release from it's good for a more agile project Or a more mature project or smaller projects that have internalized the QA process enough in their team so that they're confident that they can release directly because they will know Not be release candidates or Developmental milestones with this model. So it's a bit of a trade-off you you directly release and if you are confident that this this is this will be working well and but that means you You have to have internalized that QA within the team rather than relying on your users to feedback the Pre-release feedback We have two other release models The independent release model is for things that don't really need a stable branch or not really open-stack services So we have plenty of deliverables produced by the open-stack project like infrastructure projects that are not really tied to the release cycle and so we can we model them until Under that model and the last one is the known release model for repositories that don't really need to be released like I know specs repositories or the releases review repository that we don't make releases out of those they are most mostly tracking the current state of things So that's the the tag. We've been using to describe them. I think that's your No, that's what Doug will expose now, right? So what we decided to do During the Liberty cycle a theory talked about the two extremes tagging everything or tagging nothing We fell somewhere in the middle So projects will tag releases as needed basically, but they will not be doing that in a synchronized way. So Just because a security fix lands in one project or a bug fix lands in another project Doesn't mean that we'll tag all the projects at that point, right? So Things that would trigger a tag that's still kind of up to the project team Security fixes obviously we want to get the those released as quickly as possible bug fixes That'll depend on the severity as to when when actually, you know, if we've got several things in the pipeline We might wait until they're all done before we do a release But that'll be up to the project teams They will really they will ask for those releases through the releases repository with a Request just like they do for things off of master and then the release management team rather than sort of tracking them down and making sure that they get those things done in a Synchronized way we'll be educating them about the the reasons they need to do releases off of their stable branches and Encouraging them to do it when we kind of notice that there are a few patches that are ready to be released And that sort of thing so we'll help manage that but we won't be prodding them to do the work too much We also are planning to make Some changes to the way that we deal with release notes for this cycle So since we're not going to have a stable maintenance release manager assigned to generating the release notes at the end of a synchronized stable release We've created some tools that will let us do release notes in a more distributed manner as patches are landed So Reno is a new tool that we've created within the release management team that Manages release notes as a small collection of a collection of small files within the source code repository And it it's branch aware so it can generate the history based on whatever you've actually checked out and that sort of thing And then it's integrated with our documentation tools so that the the release notes get published Automatically as tags are applied or as patches are landed We're also going to move away a little bit from Launchpad For change management tracking, so we'll still be using it for bugs We'll still be using it for blueprints, but we're going to try to move away from it for tracking what goes into a release Launchpad is a great tool for doing all of this tracking, but it's actually pretty complicated to use correctly There's a lot of permissions issues that we run into and name conflicts and things like that And by moving some of this stuff into the tree and building documentation using Reno instead of using launch pad We think that in combination with the release History site that we've got now We'll sort of replace the use of launch pad and make it easier and less work So the release management team is pretty small and having Individuals have to keep up with all of that stuff gets me a lot of a lot of the fast release management activity was actually reconciling the state of Launchpad trying to to represent what changed what bug fixes and features are in a given release and What actually landed in the git repository is and it used to be a lot of work to try to try that That was at least 75% of the work that we were doing was just keeping up with all of that stuff. So So we're also during the metaka cycle going to continue With the automation work. I mentioned that the release's repository is only semi automated right now And there's actually a human being doing the actual release we want to Get that to a point where when the release request is approved and merges that the the robots automatically do the release for us at that point and Then we're also going to try to work on some additional tooling to help liaisons for the release related tasks that they have So each project is supposed to have a liaison designated to work with a release management team And they'll need to be doing things with launch pad that we're not going to be doing anymore but they'll also need to Interact with us and make sure that we get all the right release tools Sorry release requests approved and things like that. So we're going to work on building some additional tooling for them Another way we're going to work to empower teams to sort of have self-service releases is by Dropping the synchronized milestone Process for the master branch during metaka. So we've definitely going to be doing that for stable branch and we're going to be talking about doing it for the master branch as well because Again, the projects want to have a little bit more flexibility We have some project teams like the Oslo team need to work ahead of schedule So they need to get features into their libraries before anybody can use them to implement features in the services and we've got other teams like The horizon team who works a little bit behind everybody else They have to wait for the services to finish implementing features before they can build the user interfaces for them Before they can finish the user interfaces for them So we already have teams that are sort of out of sync and we're just going to generalize that again to Allow teams a little bit more flexibility. So we'll have milestones on the schedule this cycle But they're just going to be dates and they're going to be reminder points for you should be doing these kinds of things around this point in the schedule rather than spending Days in a row up late at night tracking everybody down and sort of enforcing that that's something we did for them The release candidates already we already had staggered release candidates because not everyone was was getting At the bottom of their bug list They're the release critical bug list at the same time And so everyone was doing their first release candidate for the end of cycle release at a different time And that's basically generalization of that, right? Yeah, exactly Right. So for stable We'll be doing releases as needed So as a security fixes is landed We might just release that immediately as bug fixes land We might accumulate a few of them if we know we have some in the pipeline and then we'll do a release for for the master branch And that'll be per project, right? So Any any project that needs a release they can just basically do that when they need to do that and that That will also apply So for the master branch that will apply for projects using the cycle with intermediary Release model. Thank you It will not apply for the project using the milestone model So they'll still have the milestone dates they'll tag somewhere around that date and then at the end of the cycle will have a Similarly kind of synchronized period of time where we're trying to get them to do their final releases. Does that make sense? Okay Okay, so those are our plans for the metaka cycle Thank you again for coming this morning before we start taking any other questions that folks might have we wanted to mention that our Talk is dedicated to our cats Malcolm Duncan and Patooie Patooie They love boxes So we'll take questions now if anybody has any other questions. I think we have a couple minutes Eight minutes Maybe Tony I didn't turn that on right so Reno uses a collection of small yaml files that live in a particular Directory within the source repository and it takes those and compiles them into a larger release document and the yaml file includes different sections for different Criticality of notes like security fix or bug fix or new feature or things like that and It organizes the results based on the release series history So it knows based on where that file landed in the commit history what version that feature is in Or it fixes in and then it organizes within each version number based on the criticality Thank you How various projects dealt with the problems that switching from the year-based version to the sembrer that's less than the year How's that been dealt with yeah, so the distros add an epic value to the front of the Version number and that's kind of a hidden value But it allows them to increment a portion of the version number so that they sort in the correct word What about the problems that causes with things like? None of the server projects are published in a way that are pip installable right now as far as I know Maybe ironic is though some of them were published to pi pi, but it was actually unusable. Okay That's one of the reason why we did it all at the same time to that everyone knows that it's happening rather than spread the pain over several years It's true that some distributions have to to do some work like like introducing an epoch like Doug mentioned to Solve the credibility between the two but as are pretty well Defined problem space now Do you do you have any example of things that might break? Yeah, I mean, I know networking Cisco and some of the drivers have been published to pi pi All right, so you should probably have something like an epoch there as well The release team specifically said that epochs are not going to be supported. Yeah, we we decided not to do those and put packages Yeah, yeah, and I think it actually made it unpackageable by some of the other projects So that may be something that has to be dealt with with with version constraints or Exclusions in future releases of things Any other questions? All right. All right. Thank you all. Thank you