 Hi, I'm your speaker at 1115. My name is Danielle Adams. I'm from the NodeJS Project and the OpenJS Foundation. I should live here in New York, so very happy to cross the river and speak to all of you people. And today I'm going to be talking about the work that I do on NodeJS as a NodeJS Releaser. So, yeah, let's get started. A little bit about me. I work at AWS. If you're familiar with cloud computing and cloud software, you might have heard of it. I work on a platform called Amplify. It's a platform for mobile and web developers to build their applications. I also work on the NodeJS Project, so those of you that are here, you probably have heard of it or worked with it, hopefully. And so I am a Releaser, and I'll talk about what that means, and a member of the NodeJS Technical Steering Committee. So I work on the project to help, or I work with the other leaders on the project to help, you know, make it a great project for everybody and sustainable, you know, and work on the future of the project. So today I'm going to be talking about the work that I do as a Releaser. So does anybody here know how many supported releases there currently are of NodeJS? Six. Okay. That's a good answer. Not quite right, but good answer. And then does anybody know how many releases have been done this year? Just shout it out. Okay. Do you say 50 or 15? 15. Yeah, it's a little more than that. So I'll go over all of that and what that means. We actually have four supported releases now, and we've done about 50 over the past year. So what is a release when I talk about that? I'm going to be saying that word a lot, so we're going to just breeze through all of the definitions so that you kind of understand when I'm throwing these words around that are a little bit more nuanced when it comes to Node. So a release is a NodeJS, excuse me, a release, and NodeJS refers to the code changes that have been made to the code base that will be built and released on top of in-the-node distribution channels. Code changes are evaluated by each commit. So NodeJS might have up to five active release lines. Like I said, right now we have four. And a release line is a semver major version that is maintained kind of simultaneously next to the base project, which you could just think of as like the main Git branch. Releasers, which is what I am, we prepare the releases and are responsible for testing and releasing the builds and deploying the releases and making sure that everything goes smoothly and that we have these nice releases that are released to the world and that everyone is able to use. So these are the different stages of the releases. So I think as most of you know, or if you've worked with Node, you might know we have three different types of releases. We have current, we have active LTS, and we have maintenance LTS. So if you have a current version, if you're using, so right now the current version would be version 19, this happens every six months, and there are maintained versions that are most closely synced to what's happening on the main branch. So you're getting all the latest and great features that have just been merged into Node, and everything's kind of tested out in the wild there. Active LTS is our name for kind of like the actively released on stable branch. So usually commits that have been released on the act, okay, I always get the words make up, the commits that have been released on the current branch usually get pulled into the active version, and then those, so we know that those features have been tested already and tested in the wild. And then we have the maintenance version, and so that is for 18 months, and then that only gets the bug fixes and security patches, so that's really kind of another stable version that we're maintaining, but doesn't get the greatest features. So this is a timeline visual for the releases. So first, we're going to use BR and AR before release and after release to refer to the different stages of a release. So about three months before release, the Node project starts coordinating that, okay, on this date we're going to have a major release. Usually in September and April of every year, they happen every six months, and so there's a lot of work that goes into that about deciding, okay, what's going into the release, and then also just making sure that it's very well tested. So a release candidate is created, and then after a couple months of kind of preparation, it's finally released on release day, and it's like a big deal, all the blog posts go out, it's, you know, everybody's excited. Okay, yeah, so I mentioned the preparation. Day zero is when the release is released, and then, so during that current period after six months, the release is changed to an active LTS, it becomes stable. That's only for the even versions of the Node releases. The odd versions then reach end of life, they've kind of reached their, so since all of the major versions are kind of mirroring what's in Dev, I'm sorry, in main, then the odd versions are no longer needed because we have a new current version that's been released. And so then after that, after, let's see, so yeah, so after a year of being the active LTS, then it's moved to maintenance LTS. And then after three years, it reaches end of life and the version has been deprecated. So this is just a scatterplot of all of the different commit sizes of each release, so I mentioned that each release, the way that it's prepared, it's evaluated by commit. So you could see kind of how I mentioned there's that first six months of the current release, then it's 12 months of the active LTS. You can kind of see with version 14, in that first kind of six months, it's kind of flat lines. And then suddenly after six months, it goes up. And that's because the releases, when it was first released, were more frequent. Usually the releases are about 100 commits each, give or take, and then after six months, those releases, because it's a more stable release, they become less frequent, but each release has more stuff in it. And so that's why we get version 14. We had a 600 commit release, version 16. This year, there was a 700 commit release. And so those releases are really exciting because they bring in all of the latest, greatest features that have already been tested in current. So I just talked about how we kind of have all of these releases out in the wild. So how do we maintain these at once? So release lines have two Git branches. Everything comes back to Git. The Git workflow is really great for managing a lot of the release stuff. So we cut a release. We create two different branches. We have one that's used for staging the changes on the next release. And then the other one is used to mirror what has already been released in the last release. So you could think of it as having... It's kind of like how a service would work. One is the staging environment. The other one is what's in production. And so it's called staging and pretty much just production. Node.js collaborators use GitHub labels to categorize commits by their sem for change. So it's easier to figure out which commits go into the release line. I just talked about how we have releases with hundreds and hundreds of commits. It's really important that the collaborators are working together to correctly label each of the GitHub pull requests because this is how everything is on GitHub so that when releases go through we can kind of make sure we're not reading everything so closely but we can filter stuff out that shouldn't go into the release. So just to kind of... This is another visual of a timeline. So every time a major release is cut it branches off of the main branch and then it kind of lives on its own forever until that version reaches end of life. And so this is how this works with the two branches that are used to maintain the release. So we have main and so the little squares are commits. So those commits will get pulled into the staging branch. Those are used to like test and test the build locally and whatnot and then while we're releasing we pull those changes into the production branch the main branch for the next release and then they sync up because they now have the same because now that code has been released via like a release. So how do we go through all of these commits? So I have my main branch and everything goes to main. So the Node.js Releasers we have a CLI tool called BranchDiff and that will make API calls to GitHub and it will pull in all of the commits that have been properly labeled to for how I want... for the commits that I want to bring into this. So for instance a label that gets used a lot is Sembrer Major. So we have commits that have been landed on main some of those are meant to be for the next major version of Node so we don't want to pull those into the current versions of Node because it's not a Sembrer Major change. So we use a denialist to Sembrer with like all these different tags including Sembrer Major and then so that the tool knows okay skip that commit that's already... that's targeting the Sembrer Major change. So we cherry pick each one and we pull it on to the staging branch so that we can test it. Mention, yeah, there's a lot of get magic here. So again I showed like a little bit how that works but how does this scale? There could be hundreds of commits I can't just pick each one by itself so while there might be hundreds of commits in a single release we have to automate as much as we can. So we can't automate that whole process or we can't automate it yet because every time we pull in a commit there might be a merge conflict or a test might fail or just something doesn't look right or whatever we have to kind of anything that comes in and it doesn't land cleanly the releasers have to be able to identify it and say oh this shouldn't go into this release and so that part is a little bit harder to automate so the human element is really key in that and so usually when we do run into merge commits, I'm sorry, merge conflicts we will ask the original PR author to create a back port to the release branch so what does that mean? A back port PR it's something that's pretty regular on the Node.js project so it means it is a commit that's already been approved to go to main but it doesn't land cleanly on to the staging branch so usually this happens with big refactors you know anything that has been built on or anything that's been worked on on top of a semver major change then it might create a merge conflict when you're pulling it into another code base or you're pulling it into another release line excuse me okay so let's see oh yeah so this is how we automate we take the list of commits that I saw that are returned from this tool and then we pipe them to cherry pick and then they are cherry picked on to the staging branch for that release so I make tests once the test passes everything's good right? wrong so that's just the beginning we've already pulled in commits but now we have to make sure that all of the commits are actually working together so a releaser on their device or on their local machine they can run a build of Node.js and then they will run the test suite against that build that they've created so I have my personal Mac I'll create a local Node build and then I'll run the test suite against it so yeah so what happens when something fails or the build fails we use git bisect to make sure we can go through all of the commits I won't go into too much detail about that because one of the other Node.js releasers is speaking later this afternoon so Beth is going to talk about what we have to do when we are trying to make sure that this long list of commits works together and so she's going to show you how to git bisect so I'm just going to breeze over that part but yeah so we do this to identify commits that break the build and the test suite and then we can use an interactive rebase to actually go back in time pull out the commits, edit them basically rewrite history to make sure that the list of the commits that have been pulled into the release line are all passing the build and passing the test suite so we run it again looks good right? not quite we have to do this several times so we have to iteratively do that until we have a passing build so it's not just once, it can happen you know ten times, quite a few times before we can have like something that we can start deploying so passes local test suite, great so now that I've passed the local test suite there are a couple steps first I think I mentioned release notes at some point we have an automated again another kind of CLI tool where you can see there's a markdown flag and it kind of takes the list of commits and it will generate a release notes kind of like commit list and it'll take all of the descriptions from the PRs and stuff and then it's up to the releases to kind of put together those release notes and also to kind of write some descriptions of oh these are the notable changes these are the breaking changes usually there's some dependency changes that we want to highlight as well so we'll put all that into the release notes and those will go into the change log as well just like any code change we propose a release on github yeah you can see here I've added that notable changes section we just put the release notes into the github description I have some fellow releasers and Node.js collaborators that have signed off on the release and then we run the CI tests that are kind of like out in the ecosystem so I've already run my tests locally so why do I have to wait for all these other tests that are also on CI well for people that have like it's kind of interesting because you know if you haven't worked on software that is packaged and released out in the wild you might think okay well I only need to run my tests in one environment but that's not really the case it's not a service so you don't know where the Node.js binary could be running it is very much like something that could be running anywhere so all of the CI tests which take they don't take too long to run but you know it is quite a bit of there is quite a bit of processing power there I'll say and so we have to run the tests across a bunch of different platforms and operating systems and then we also run the tests against a lot of kind of hand selected dependencies out in the Node.js ecosystem because we also don't want to break the ecosystem there's a ton of packages that people are using you know they're using them in their web servers and Node.js run times and a lot of the JavaScript build packages are built to run on top of Node so we try to make sure that everything that nothing's kind of like a huge breaking change against any sort of environment wherever Node might run so and then we build the release so I just talked about how we stage all of that information so first I'm going to create the release build this is just another CI that creates all the builds for all the different platforms we tag it on GitHub so we've like pushed all of this information to GitHub we tag we use like a tag object to link to the release nodes and then we promote and sign the releases so that they are verifiable on you know the distribution channels that are from Node so I just talked about all of this kind of like public public steps that we take for releasing but I didn't talk about any of the security parts about it so like we don't want to default to security by obscurity but you know when we are working on a security patch we want to make sure that we're doing it behind closed doors before we are disclosing that vulnerability so I'm going to talk a little bit about security releases and how they're different from our regular releases so yeah so like Node.js like many and sorry Node.js like many open source projects uses a public bug reporting tool for security researchers to submit potential bugs so that all happens that's I guess behind closed doors you know it's just a conversation between the Node.js team and a security release researcher once the bug has been verified as a vulnerability that's when the wheels start in motion about with the Node.js security release so because there's so many steps in that we actually have a designated security steward that is assigned to that release we have a rotating list of people that each take a release it's very much like hurting cats there's just a lot of different pieces to it and they all have to happen kind of asynchronously so we have one person that is designated to make sure all of the steps happen so that you know we don't miss anything so when we prepare the release that is a little bit different so I talked about in the regular release you know it's just the releaser we look at commits we pull stuff in it's really up to the decision of the releaser this is a lot more collaborative we have because there might be a couple security vulnerabilities that have been reported at the same time we have to make sure we're organizing you know there might be like one two three security patches in one release so we make sure we do that so that we don't have security not that we have a ton of security reports but you know we just want to make sure we batch those together so that we can you know it's clear okay this is how you can address like these XYZ then a date is scheduled so usually unlike the releases the regular releases which can take a couple weeks this might take several weeks or even months in order to coordinate all of the releases to be done before the release is done then we have to check that the fixes are actually ready for integration into the code base and that they fix the problem that was originally reported so this step is exciting because exciting and interesting because the OpenJS project was let's see so I'm going to talk about who actually works on these fixes so the OpenJS project was the first NoJS was the first open source project to be funded by Project Alpha Omega which was really exciting for the NoJS collaborators and leadership because that means we actually got external funding for some of the security efforts that we really wanted to focus on but everyone on the project is kind of you know they're very they're working on maybe the operations or the feature forward work and so this devotes a couple of people's time to making sure that we do proper security triaging stewarding releases and then just improving the overall security efforts of the project so this doesn't just include security releases and making sure we have proper patches this is just overall like you know recommending best security practices across the project and so yeah so we've been excited to be funded for this project this year and so this extends across the security releases and the work that's done to create remediations for any vulnerabilities and again like I said this takes the burden off the maintainers which you know there isn't really when you're working on operations or feature forward tasks there isn't really a deadline where it's like when you have a security vulnerability usually there is a deadline and you have to do within a certain amount of time and that's a lot of pressure on people you know who might be doing this in their free time or you know not really being paid for this type of work so we have the fixes in place usually that person that's working on the patches they are engaging with the security researcher to make sure hey does this fix what you saw there's some agreement that this is the patch for it and then we start the release process so oh wait we don't release yet first we do a pre-release announcement so you might have gotten an email or something that says there is a you know XYZ security release coming out on this date but it doesn't say what the vulnerability is it doesn't have a CVE or anything it just says we have a security update you know stay tuned so that's usually what this announcement is then we run the release process so that little step is all the stuff that I talked about in the first part of the talk so you know creating the PR pulling in the commits running the tests you know creating release notes making sure that there is coordination on the proper release notes because this is there's no other changes in that release it's only dedicated to the release the security patches and then there's a post we post a post release announcement so we have we might then disclose the security vulnerability and then we have a path to remediation which is in the form of hey upgrade your Node.js version patch your version you know this is the vulnerability if you think that you're affected so like I said a lot of that stuff happens a lot of the release process happens in public so how do we do this behind closed doors then we default to get everything that we do in public we actually also have a private space that we do that so we are reviewing code we have code reviews there's a lot of coordination between on just different what we should be pulling in to the security research sorry the security release and all of that happens on just like a private GitHub repository let's see so now that I've done this entire release what do I do next so I just spent weeks or maybe months working on a single release and do I get to rest well no releases as soon as we're done with the last one we start thinking about the next release because there's always a release coming out releases are scheduled years and months or months and years in advance there are no releases scheduled up until 2026 which is about four years so yeah so every time we are you know we're finished with a release we're kind of also planning for the next one and not just the next patch but maybe the next security release or the next semver major release and so as if you're kind of like yeah so you might have seen that each LTS version gets a code name that comes from the periodic table of elements and so in the English language there are no code names for Q and W so if you have any ideas please open a pull request you know because we also schedule the names very far in advance as well and so four minutes left so good doing good on time so this is also just to give you an idea of kind of like how you know I told you okay well we're scheduling in the future this is kind of a look at the past a couple years back this goes back to 20 no sorry not 2014 maybe 2019 but so you see we have version 14 is maintenance LTS right now and so that version has about we've done 45 releases version 15 which was that kind of in between current release about 15 releases and then version 16 which is still supported that has about 30 and so that will grow to the same height as 14 by end of life and then you have 17 and 18 which are tied because both of those have been in the ecosystem for about six months 17 was for six months and then it reached end of life 18 was released in April and so it's kind of hit the same number of releases and so that will surpass 17 and probably get to the height of 14 as well because it is an LTS version and then we have 19 which is just released in October and so that will reach about 15 versions and then it will hit end of life when version 20 is released in April so I just talked about all of this stuff that's like a huge coordination for a bunch of different people and teams across all different regions does this really benefit Node.js developers well I would say yes I think that there are multiple audiences multiple users for each version line or release line and so while some developers might want the latest greatest versions I know at my job like a lot of people are kind of on bleeding edge of technology but some people just want to upgrade every year every two years and so this release structure actually accommodates both of those type of users and as I'm sure you know Node is used widely across many different industries and different product types and so we want to make sure that we're catering to everybody and I believe that this model allows Node.js to have multiple release lines for multiple types of developers and platforms all at once so if you want to learn more the contributor guides that URL I'll leave this up for a second if anyone wants to take a picture the contributor guides have yeah what do they have the release guides they also have the back porting guides a couple other things some of the docs about how we do security releases and security stewarding the release group has a working group that has its own repository we meet once a month discuss releases and the schedule and what not to learn more about Project Alpha Omega I think that's the URL for a lot of the news and stuff for it and then yeah and then there's a link to my slides I think I also need to upload them to the public forum or whatever so and that's pretty much it thank you so much everyone for listening to me this is my Twitter account and then you can find me on GitHub as well so I appreciate your time thanks have a good day