 Thank you Martin. Yeah, so as Martin said, my name is Doug Hellman I'm here to talk to you today about a tool called Reno, which we built as part of the open stack community to manage our release notes It's not really relevant to the talk itself, but I'll for those of you who might not know what open stack is I'll give you sort of the elevator pitch it is the cloud management platform infrastructure management platform written in Python that gives you features like you might find in AWS or Azure or things Like that and it's all open source and you can run it on your hardware in your own data center We consider Release notes to be an important aspect of communicating with our users Those of you who build libraries are probably used to explaining about what you've done in each release Or if you've got an application that you distribute to other users and don't just host yourself You're probably used to doing the same thing sort of explaining what kinds of challenges You might run into to do an upgrade or what new features are present or what other kinds of changes are there With open stack in particular because of the complexity release notes are a very important part of our documentation And so I'm going to talk a little bit about a tool that we created to help us manage those in a way that Let us maintain them more sustainably We created it early Early in 2015 as as our community was growing and we were seeing not just an increase in the number of different Services and components to open stack itself, but the contributors and the contributions that were coming in so we started out with Six or seven projects service projects each managing an API and a Python client library to talk to that API and then over time we grew But in the early days each project was sort of responsible for doing their own thing with release notes So we had a wiki and people would just go and manually put notes into the wiki Maybe they would keep up to date with what they were doing and frequently they really wouldn't so at the end of a release cycle We would spend a bunch of time going through all of the work that we had done and our release cycles are every six months So that meant going back through six months worth of work for all of those projects and trying to understand which ones were user-facing changes What added new configuration options? What might impact the upgrade and things like that? That turned into a real challenge for us as we grew in particular because Of the amount of changes that we were seeing so even at a point where we were still doing just six different projects that the changes ramped up very quickly and It also turned into a problem when we were back porting changes so our development model is that we do all of our development on the master branch and then we produce a stable release every six months and The stable release branch will see updates over the period of six to 18 months depending on how long it's maintained And those changes are limited to bug fixes So we call it stable because we don't back port features and we hope that that makes it a little more stable than it would otherwise be But those bug fixes typically come with changes that need release notes as well. So as We saw more and more of those branches created and more and more Patches and fixes going into them that turned into a problem for us, too And then we really hit a stride where we were growing as a community So we grew from six projects teams to over 60 different teams now and we have Those 60 teams managing 300 different or more deliverable objects. So that we talk about deliverables meaning An artifact that we build and ship to someone. So that's usually a library but it's often a service in some sort of package as well and The manual processes that we were using for six teams just we're not going to scale to 60 teams and and 600 or 300 different deliverable objects We did scale out the release team So we originally had one person managing all of the releases and and he was doing a lot of work by hand The team doubled in size when I joined it, but I'm extremely lazy So that really didn't actually doubled the amount of work that we were able to do instead of trying to do all of that Work by hand I started thinking about ways that we could apply automation. So as a good programmer I immediately tried to write a program to solve the problem that we had right so I Have not yet been able to figure out a way to automate writing for lease notes So I kind of set that aside and instead focused on making it easier for other people to do that work So making it easier to manage the release notes throughout the course of developing the project over a release cycle as well as Automating the publishing of those release notes So we include them within the package that we distribute But we also want to publish them on our website so that they're easy to find and at googleable and that sort of thing When we sat down as a team to look at Our different requirements for this project we kind of divided them into two different groups So we had some content requirements about how we wanted to manage content and some process requirements about how we wanted to make our process sustainable as we grew Peer review is a very important aspect of our community culture Everything that we do within OpenStack is peer reviewed So every bit of code is posted for review and normally reviewed by two reviewers at least Before it gets approved and merged We wanted to know we also wanted to be able to organize the content So I mentioned several different kinds of release notes like upgrades impacts bug fixes and incompatibilities and things like that We wanted to be able to organize our release notes as they were published into those different kinds of notes so that you could Go and read all about what you needed to know to do an upgrade in one section and not necessarily worry about new features in that section We also wanted to be able to change the release notes So we don't assume even with a peer review process that we're going to catch any mistakes or You know not miss something or have some other kind of error So we knew we wanted to be able to go back later and change notes in old releases And then it's very important due to the pace that we work at that we want to avoid merge conflicts So it can typically take a week or more to get a patch reviewed and merged even at a fast review period Some of them take much longer than that and every time you encounter a merge conflict You basically have to start over because you have to Rebase the patch and resubmit it has to pass all of the tests again And then you have to get reviewers to take a look at it again So we wanted to design a system that just completely avoided having to deal with merge conflicts as a contributor On the process side. We had some other kinds of similar requirements We use two different kinds of versioning for our deliverables. So libraries are Versions using semantic versioning which means that you increment the version number based on the kinds of changes that are in The new version not the number of changes. They're in the new version For those kinds of things We don't know in advance when you write a release note what the next version is going to be So we didn't want release note authors that even have to think about version numbers And we wanted them to be able to just contribute a note And it would automatically be applied to the correct version when the release was created. I Mentioned our stable branch policy We wanted to make sure that not just that we avoided merge conflicts on the master branch But that we also avoided merge conflicts when we backported patches Right now we use Garrett as a review tool and you can backport a patch basically clicking a couple of buttons in a web page And it will create the new patch on the new branch for you and Apply it automatically and as soon as you start talking about merge conflicts there you Raise the bar for back porting a bug fixed Very you know makes it much harder to do that. So we wanted to avoid merge conflicts there as well And then of course since the release team is very lazy We didn't want any manual processes involved in publishing the notes at the end of a release So we didn't also, you know, we didn't want to have the author of the release note have to worry about versioning We didn't want the release team to have to sit down and Mark version numbers on a bunch of release notes or anything like that either. We just wanted all of that stuff to be handled automatically We went through three different designs In the course of trying to figure out how to meet all of those requirements So the first one was sort of the obvious thing. We're already using sphinx for documentation We could just use sphinx for release notes You could just write your release notes in a document and include it in sphinx That has a couple of different problems So either the release note author has to know the version number so they put it in the right place in the right file Or someone has to go back later and apply the version number So that didn't meet either of those requirements for us And it's also messy for cherry picking changes particularly if you If you organize things in a directory structure in some way when you cherry pick a change back You then have to edit that change to put the new note in the correct place for the older branch because the version number system is different We also thought about using the get commit messages So just having the patch author write the release note right in the get commit message as they were contributing the patch that Would work mostly but the Sort of the audience for a commit message is very different from the audience for a release note So for a library it might be similar But really we wanted people to be focused on writing commit messages for the reviewers to understand what was going on in the patch and Not necessarily the user having to go and read the get history to understand everything that had changed So we wanted them to have either a condensed version or an expanded version that explained in more detail for the correct audience For those release notes the commit messages are also largely immutable So once it's in that public repository your notes are basically set and that's done We also looked at the get notes feature which is I don't know a lot of the detail about this one but it's sort of a parallel thing to the get repository and you can do a little bit of extra setup and Have notes attached to commits, but not be part of the commit And that was an interesting approach until we realized that you needed to do a bunch of extra setup with Garrett in order to make it work And the permissions were different and it wasn't really a reviewable thing So it fell down on our peer review criteria there and So we finally ended up with a system that we implemented as Reno where we use data files inside of the commit to hold the Release note content, but they're not Sphinx files. They're just in independent Data files and then we do some integration work to make those publishable Because they're inside the patch. They're reviewable So the reviewers can comment on them and reject them and you can make changes and submit a new version We do read the release notes data out of the get objects So they're not it doesn't matter where they are on the file system They're all basically in one directory together and we look at the get history to figure out which versions they go in because they're part of a Commit we can tell which version the commit is in and So that's what Reno does Reno Uses those data files and basically uses the get history as a database reading those particular files And then assembling them in whatever form you've asked for to build a release notes report It's a command line tool So you start out with the Reno add command and that you give it a slug just to sort of be able to identify what the file is Frequently people will use bug numbers or they'll use a short abbreviation of the feature We have a specification process which gives everything a sort of unique name So sometimes the the notes files will have those as part of the names as well And then Reno adds a unique identifier to that slug that you give it so that it can track the history of that file Across all of the branches and through all of the revisions that you have The files are placed in a subdirectory of the project like I said, they're all just placed in one directory together They can be organized a little bit under that directory. Some teams have divided them up into different Different kinds of organizations. It doesn't really make any difference for that Reno because it's not reading the file system. It doesn't really care what files the notes are actually in and The data file themselves are YAML files. So this is an abbreviated version of what one of those would look like The section the keys there are section names and those are predefined But configurable So if you have a project and we have several projects outside of open stack that are using Reno They'd have a different set of features that they want to use or sections within their release notes that they want to use You can change what's that what those are and then Reno knows what order to apply the sections and then which sections each note goes in within the section within the data file You have a list of restructured text blobs that are just sort of inserted in in the order that they're presented And using restructured text lets us integrate easily with Sphinx. It also lets us Do things like link off to bug reports or feature descriptions or specifications or that sort of thing So you get all of that that nice markup and we use restructured text rather than HTML or something like that because we do actually generate PDF documentation for some of our translated sites especially those behind the Great Wall in China It's easier for them to download a PDF and share it than it is sometimes to get to websites that are outside of China So the output from Reno we can take a look at an example get repository. So this is a typical graph showing the history with the beginning at the bottom and the most current commit at the top there and I have one stable branch that I've created just to be able to show what a cherry pick is going to look like There's two different versions are tagged So we have a 2.0.0 version that has four commits And if we assume that we're all good developers and care about our users We have a release note in each of those we're going to have four release notes for 2.0.0 And then version 1.0 is the first two commits there and that's going to include two release notes So if we look at the output of the report command for the 2.0 version We can see those four release notes with the I've just included the letter names there so that you can tell that they're from each of those commits and It does that I've included them all in the new features section just because that made it easier to fit on the slide But of course you can organize those into different sections as appropriate Reno figures out which notes to include by starting at the tag that you've given it As a start point and it reads the git history one commit at a time looking for release note changes in those files And then then it assembles those into its database for that version and it emits them in the order that they've been presented and It knows that those notes apply to that version because they appear either on the commit that is tagged or Before the commit that is tagged with the next version number or the previous version number, I suppose so As it's scanning backwards in history It's going to find the 2.0 version first and then it will find the 1.0 version following that It is so Reno does the scanning using a pure Python library called Dullich, which is an implementation of the git Library we chose that one because it's pip installable without any other dependencies, which makes it easier to manage and work on release notes On various different platforms. So there are a bunch of libraries that actually interface would get But this one because you can just pip install it it means folks that are on a Mac or on a Windows system They don't have to figure out how to install a C library in order to do the work with Reno and And using Dullich has proved to be Considerably more reliable than using the porcelain output from git So the first version of Reno actually ran git as a command and then read the output and tried to parse it And then somewhere in there the porcelain formatting changed and we were no longer getting some of the data that we needed in order To track changes to files. So we had to rewrite it using a library instead So if we look at the stable branch If we ask it to scan the history of a branch instead of asking for a specific version number We get two versions as output. So using git's sort of standard version nomenclature of tacking on a Number of patches after a tag we get a 1.0.0 dash 1 as a version And that's actually an unreleased version that represents the change that's on that branch after the tag and Then a 1.0 version that includes the a and the b commits So scanning a branch doesn't necessarily stop at the previous release So you can actually ask it give me the entire history of a series of releases on the on a given branch And that's useful Particularly in the way that we publish releases. So we name these stable branches So Rocky is the current branch that we're in and their alphabetical order. So based on What the naming convention doesn't matter, but they're in alphabetical order So you we have a page for each project for each series and we can say show me all of the notes for That series all together and you get all the versions that have been released all the patch updates and everything So I talked about Backporting changes from one branch to another So if we take that commit e and we copy it using cherry pick from the master branch onto the stable branch We get the note that goes with the code change and it ends up on that stable branch And then if we tag a new version at the end of that stable branch 1.0.1 meaning that it's a patch update with a bug fix in it We get the release notes willing for 1.0.1 will include a copy of the note that was in the 2.0 release That's in that e E commit and we can do that without making any changes to e because it has a unique file name for the release note And assuming the rest of the code backports cleanly you can just backport and approve the patch now That doesn't always happen, but we didn't want Reno to introduce his reasons to have backport problems. So we We're still not a hundred percent there, but we're doing the best we can And then the notes for 1.0.1 will look like this So that includes that e release and then the previous commit that was there that hadn't been tagged yet So we get both notes When we publish our notes, we publish them from one job that happens when we tag a commit And so we actually have as I mentioned we have a different page for each release series and we publish all of that together at one time When a user reads the release notes though, they don't read all of them They go and look at the version that they're installing and they want to see the release notes for that version or that series of versions And so having a note copied from one branch to another like this means that the user doesn't have to jump around and find the release Notes in different places based on where the fix was originally applied and then backported to That was a real problem for us with the wiki approach because people would backport the code and then forget to go and copy the release note around And so the as you after the initial release on a stable branch basically the release notes were not really being updated very often In addition to doing a backport like that I mentioned that we needed to be able to change a patch Change a release note in case there was a typo or something like that So the way that works is you change the the release note file at any point in the history of the branch After it's been added and as Reno scans through the history It finds the change first because it's starting with the newest patch in the branch and going backwards in history But it doesn't actually insert the content from the release note until it finds the add event where that file was initially created So it takes the content from the newest version of the file and inserts it at the location in the history of the original file And that lets us change the published version of the notes for an older release after that release has been cut We can't obviously change the content of the package for the 2.0 release So if you're looking at the release notes that are inside the package We can't do anything about that but we can change them what we actually publish online And so this is the output for looking at the 2.0 release after that change has been applied You can see that there's no typo in the release note anymore, and it's been changed outside of that It also understands that if you delete a release note file if for some reason the note doesn't apply at all And you want to delete it When it sees that delete it basically ignores the file so it doesn't add it later So if you have a completely incorrect release note that somehow passed through all of the peer review You can just delete the whole thing and it will be ignored All right, I've shown you basic command line output for Reno it It does also integrate with Sphinx. That was one of the key benefits for using restructure text So there's a Sphinx directive to insert the release notes for a given series or version into the output of Whatever your Sphinx documentation is This particular directive without any extra arguments is going to insert the full history of the current branch So that's really useful on the master branch because you basically get everything that's happening in the development series on to Publish the documentation for a stable branch you would specify that stable branch name Or if you just want to have a version or a couple of versions You can include a range or a set of version numbers in there as well So it's easy to organize your published release notes in whatever way you want Basically any arguments you can pass to the report command you can pass here through this Sphinx directive And then the output depending on what your styles are going to look like your CSS styles It's going to look something like this So you get version numbers inserted with headers and you get all of your sections With titles and then the release notes are inserted in order and we do maintain a consistent order So it's not necessarily Predictable as you're writing the release notes, but we're going to insert them in the order that I think it's sorted based on those unique IDs so that it's consistent so that if you publish a new version you don't suddenly scramble all of your release notes into a different order Right, so that's the tool the effect that had on the community was pretty impressive even beyond what we had anticipated so We've definitely increased the quantity of release notes that we're writing We are at something like 13,000 notes for those 300 different projects now, which is far more than we had Even you know relative to this this six projects at the beginning More importantly though, we're changing the culture. So in addition to Encouraging people to write release notes Some of the teams are actually requiring that new features come with release notes new configuration options come with release notes Less so for bug fixes unless there's a significant upgrade impact or something like that But that's not a pervasive thing throughout the community it's a few of the teams that are a little more a Little more interested and have a little more impact in the sort of thing that a release note would in document and I consider that a win for my own laziness because I don't have to deal with release notes at all at this point It's basically every once in a while There's a bug report against the tool and I go work on that but I don't have to do with the actual documentation So if you're interested in trying Reno out We would love to hear from you It is set up by default to work using the open stack naming conventions and version Conventions and patterns and that sort of thing, but it's very configurable I think almost everything that would be a useful behavior has a configuration option now But if it's not then we're interested in making it more usable for more projects If you're not interested in Reno, I'm still interested in having you write better release notes and actually write release notes So you might be interested in looking at a couple of other tools Town crier is a tool out of the twisted community, which I didn't know about at the time that we built Reno But I believe it has the problem of requiring you to know either know in advance the version numbers or do something at The release point to deal with version numbers and so it wouldn't have met our needs anyway But it's still a good tool and then blurb is a tool that Larry Hastings wrote for the Python dev community And it has a similar sort of problem and both of those teams are releasing one thing And so it's not really a big deal for their release managers to take a little bit of action at the point of the release But for us with the scale of the number of things we're touching we didn't want to have to deal with Any of those kinds of manual steps The documentation for Reno is on the open stacks website so docs.openstack.org And this sample repository that I use for this presentation is on my github account if you want to play with that and I'll be posting the slides for the presentation Online as part of the conference website And that's all I have so if anybody has any questions I'd be happy to talk If anybody has any questions please raise the hand Thank you for this presentation Just a quick question if I understand for now it's only on forget But do you have any plan for any other VCS and second part of the question would it be easy to contribute and well for other VCS Is it just like callbacks plus settings or is it like really forking the whole stuff Yeah, so I had not seriously considered other version control tools yet, but I'm interested in talking to you about what which one in particular are you interested in Perforce I don't know anything about that so it'd be up to you to do that I could help you there is I think a point at which we could add some logic that would let you switch back and forth between those two things in the scanner in particular I don't know whether Perforce's data model is similar to gets it really relies on being able to walk through the history and see the topological sort of the commits and that sort of thing so But but yeah I'd be happy to talk to you after about some of the details for that that would be interesting sure Okay next question please Basically the same but for me curial Basically the same answer yeah Okay, let's see over there It might be easier for mercurial because I think the data model is closer there yeah I think quite understand how it knew when to stop going back in history so you gave an example where you ran it on Rocky Yes And I went back and then got MB which are from master Yes How did it not go back to for example the commits that came before Queens or something previous like Sure What's the termination condition basically Okay so I very drastically simplified the example in the presentation so the open stack we're on the rocky release cycle right now that means we have branches all the way back through a potentially for some projects And it does a little bit of work to figure out where the previous stable branch came from and it stops before it gets to that point So that's not really It's very difficult to represent that in a slide And the hand waving is probably indicative of that but yeah it does have some smarts for doing that and that's you can turn that off so projects that don't have that kind of stable branch structure could basically ignore that feature and disable that but yeah Okay we have time for one more final question Thank you for the talk I understand that individual release notes are committed also to master so my question is how do you if you make a mistake in your release note how do you commit without force pushing to master Yeah so you we don't do any force pushing ever all the changes are iterative so you apply another patch on top and then Reno as it's scanning that history it sees a change to a file Because it's scanning backwards in history it'll see the change before it sees the ad so to hold on to the contents from the patch where you've changed it And when it gets to the point where you've added the file six months ago or a year ago or whatever then it will insert the note at that point in the history using the newer contents And it can do that because it's not reading the files that you commit off the file system it's reading them out of the get objects in the get history if that makes sense Thank you very much for the presentation and let's have a round of applause for that Thank you all very much for coming