 Welcome to Jenkins Docs Office Hours. It's the 22nd of March, and let's get it started. Topics on my list. She-Code Africa Contributon Project. Jenkins 2.277.1, the most recent long-term support release. 2.278.7.2 is coming. Meg, any other topics you wanna include on the list? No, are we, what about the Summer of Code, is that? Oh, that's a very good one. I thought the 24th was a milestone of some sort. Yeah, so Google Season of Docs, very good one, right? Application is due this week. Need to review and discuss the application, the ideas and the application. That's very good, yeah, let's do that. Sorry for the clattering of my keyboard, but I love my keyboard. Okay. I love click, I love keyboards that click, it doesn't bother me at all. Okay, great, so. I hate touch screen. All right, okay, so yeah, that's very good. Let's go through each of those, and I think there are things we need to, that we can help each other on with each of those, great. Okay, so first, She-Code Africa. Delighted to say that we've been accepted. There we are. Oh, right. And if I recall correctly, sponsors also lists. No, they don't show who their current sponsors are on this page, let's see. So mentoring orgs, accepted orgs, this is us. And, but I do have to show the tweets that have arrived. Jankin CI, whoops, Jankin CI tweeted about it here. Ooh. And we blogged about it here. Yay. And so it's a good story that we have to tell. And as even a cool thing is, okay, I acknowledge that I cheated, and let's see, where do I find the replies to it? I don't know, let's see. But if I look at mine, I was, I did a little bit. I, as a personal tweet, I can be a little more bold. So what I said was, you wanna help create a more diverse open source community? Contribute. And ask your company to contribute. Yes. And this has actually got, has received surprisingly positive feedback in terms of the response by people who read these things. So very positive. Interesting, interesting. And I don't know if anybody will act on it, but delighted that Shikot Africa's work is being highlighted and that we can help them in a little bit. So looking forward to it and their stuff that's going on as well. All right. So let's see, we should link to the blog post so that people are aware that we did that. So did that. And the pipeline projects, the projects idea are, whoops. Oh yeah, this is their form of our project idea. Ah. Okay. And when I last talked to Xenob, she was great with our idea that we would take up to three and was very grateful for the details. Thanks for your help, Meg, in going through this, reviewing the steps, assuring that we know how to do them, et cetera. So very, very positive. Oh, good. Well, she's a force to be reckoned with, isn't she? Yeah, and that's great. Okay. I still think we should think about hiring her when she's done with her studies, whatever it takes. Just me. So let's see, there we go. And now let's get this one back. So we've got this tweet recorded and Oleg did a very nice post on LinkedIn of it. Oh, goodie. So. Oh, let me go look for that. I'm not good about sharing stuff on LinkedIn, but I will find Olegs and share it on mine too. Yeah, and that's part of the fun of it, actually, is to watch, oddly enough, I think we now on LinkedIn have more followers than we do on Twitter. Interesting. Yeah, it is really quite cool. So let's find his, oh, maybe I don't, oh yeah, here it is, good, there's it. So if we grab a link to this one, copy link to post and there's the LinkedIn post. So just generally positive, it's really good and that also helps us highlight all sorts of progress. So now, Megan, in terms of that one, I guess one thing to safety check here is, is that what we intended was that we'll do twice a week meetings with those that were mentoring, one during sort of your working hours and one during Kristen and my working hours. Okay. If that doesn't work for you, so one during one, basically at the start, start of day in Africa and one after end of day in Africa. If we need to adjust it, we will, I don't know which day it will be, I don't know any of those things we'll talk about when the candidates are selected and then work through it to be sure that, hey, it works for all of us. Will there be on the one that I'm in, will there be somebody else there mentoring who like knows something? I hadn't planned on it, but we can consider that. My thought was we'd let you coach on things like writing style and content things where they submit a poll request and you say, hey, here is coaching on your specific poll requests. And then Kristen and I will handle things on, oh, hey, I did this step in the detailed instructions and it didn't work the way I expected. Okay, I'm good with that then. Does the Jenkins Stock Project have a style guide or anything like that? Does not. Okay, so. So you just have to apply your best judgment on style. Okay, good. In particular, because these changes will be going into plugins, it's even harder to apply a style guide because it will eventually arrive on the, after the plugin releases with the changes, it will arrive on jenkins.io, the website, but it'll also be visible in their Jenkins instance when they run it. So, okay, good. If I have mixed feelings about, I mean, sometimes we get obsessed with adherence to a style. It has to be good English syntax in some world. Well, and that's the piece where I think you are intensely valuable because you're so good at doing reviews of English and good technical content, right? So that's something that Kristen and I are weaker at, whereas we can help with the more detailed, oh, this is how you compile a plugin stuff. Right, okay. And I can, if I see something technical that I'm suspicious of, I can just alert you, keep my mouth shut. And I'll just check it. Well, yeah, just tell them, hey, check with Mark and Kristen during your session with the two of them. Right. Yeah, so I'm not shy about commenting on content and phrasing, but I'm also not the expert that you are. So my thought is we get, we get the best of both worlds is you're free to tell them, gee, I don't know about how that specific compilation question you're asking, ask Mark and Kristen. Okay, good, cool. All right, okay. And precise times and days to be negotiated, right? Because we have to see. Now, I guess one thing that I've wondered is if they get a lot of subscribers, do we want to consider taking more than three? And if they were all on the same project, it might be worth it, but I'd probably limit us upper bound to not more than five, maybe six, just because I fear we'd be overwhelmed if trying to mentor that many. Yeah, if we, if this group, if this meeting had more people attending regularly. Right, if we had a larger group of mentors, we'd be fine. Right. Yeah, okay, good. Let's, I would say if we, if we're just overwhelmed with really talented people, you know, we could revisit. Right. And we could, now, are we the only ones doing this? Like is Oleg gonna be involved in it? Oleg has offered to assist if there are technical details that need his assistance. And so he's willing to assist, but I think at this point, the things that we've outlined, the details precisely enough that I don't expect to need his help. Right. For instance, if someone comes in with a very specific plugin or a plugin that needs Oleg's exact skills, he's more than willing to help and has agreed to do so. Right, I just, yeah, but there aren't any projects other than the doc project being put in. No, this is, this pipeline examples, project ideas, the only project ideas so far. Now we could, we could, it's potential we could get others. It's half and half actually. Well, and that's specifically why I chose it. Right, the fact that this is, this requires that they do some development. It benefits them with experience with development and we also get a doc improvement in the process. Yes, yeah. Oh, I think it's brilliant. Yes. Okay, good. Well, we'll see. There's always surprises, but they'll learn from those too. Right. Anything else on Sheikot Africa? Looks good. What are the dates? Do we know when the dates are? So start, should start as far as I understand April one to April 30th. Okay. And I believe selection happens. So selection, I believe happens in late March and then the projects run from April one to April 30th. Okay. We are now officially in late March, I think so. Right, exactly. So any day, what is your- I assume they're still trying to get final funding to decide how many women they can pay during this period? Right. Okay, anything else on Sheikot Africa? It looks good to me. Okay, so next piece is Jenkins, the 2.277.1 LTS is a major update to Jenkins that includes form modernization and several other really cool changes. So configuration form modernization is the one that has been causing the most struggles in the community as far as we can tell, but there are others as well in it that are worth people considering and how they deal with it. So let's see. That is gonna blow screenshots right and left again, isn't it? See, actually it improves screen, it changes screenshot but improves them dramatically. Oh, yeah. I'm just, remember I'm maintaining code that's on, or courses that are on what 2.222 or something. Yeah, so you'll wanna get those up to 2.77, agreeing screenshots. Yeah, I can't do it right now. Ah, got it. Labs can't support it, but this just blew up in our faces, so I think Tammy just got a little more interested in it than she was. Got it. Yeah, so now there was a, we also did a Jenkins online meetup video that spends an hour and Darren Pope and I did a, let's see, where is the Jenkins online meetup? There it is, okay. So if we look at this one, this is the video that takes, we had Jesse Glick, Tim Jacome from the, Jesse Glick from CloudBees, Tim Jacome from the Jenkins community in the UK and Felix Coruga from CloudBees with me and Oleg going through a panel discussion, talking about it. Oh yeah, okay. Video, so Jenkins online meetup reviewed the changes and what they mean. And what we're seeing is that we're getting proof repeated proof that there are plenty of people who even with our best attempts to guide them, don't read the upgrade guide, don't read the install notes and just post questions. And the crucial steps are upgrade your plugins, update your plugins, upgrade Jenkins, update your plugins again. And that's atypical. That is not common for a Jenkins release to need that. But because of the significant UI improvements here, they need to make that change. Okay, are most of the plugins updated so they will work? Most is way too strong, many, many and certainly the most popular ones are. Okay. There is a Jira dashboard that shows the plugins that are known to not be ready for it. Aha, okay. And right now there's a little bit of angst in one of those. I'm gonna drag a page in here so we can see it and grab a copy of that. So one of those, this one here at the very top, the TFS plugin highlights a class of problem in one of the most popular plugins in the world. And that's one of the most popular plugins in plugins where the TFS plugin is no longer being distributed. It's no longer being distributed because the people who maintain it have stopped maintaining it and it's no longer being distributed because it's not open source software. It has a dependency in it that as far as we could tell at the time was not open source. It looks like it may now have been made open source so it might be able to come back but it's also got a security problem. And so users who were using Microsoft's Team Foundation Server have this plugin that has an own security problem and has not been updated for tables to do. So if they upgrade to 2.277.1, they have a bump. They have a, oops, it doesn't work. And the answer is you either remove TFS or you stay with the older Jenkins version. Because we've stopped distributing this plugin and so they've got, but here you see a number of people commenting on, hey, yeah, this is what's happening. Oh boy, it happens. Yeah, well, and some plugins, yeah, that's just the reality. Some plugins are, we've had other cases like that where they're using running a deprecated plugin and we aren't updating deprecated plugins to support the new user interface. They need to switch to use the modern plugins. Yep. The most, the visible examples there are things like warnings and let's see, finebugs, warnings, other old static analysis plugins. Others are no longer distributed, no longer, let's see, how do you say it? No longer, yeah, no longer distributed by the project. TFS is the example there. Any other, any questions you've got on 2.77.1? No. Okay. I go look at it, it sounds like you should know about it. Yeah, all right. So 2.77.2 is coming. The release candidate will be out in a few days and then after two weeks, we'll deliver the actual release and that delivery, let's see, I should bring up the release checklist because this is one of those things that I'm going to use more and more of these things. I like this particular thing so much, I am so impressed by it. It was created by Tim Jacome, our release officer and I have just been absolutely astonished at how effective checklists are in GitHub. Okay, so here is, let's make it big enough to read. Here is a release checklist for Jenkins 2.277.2. And so it's just an editable form, an editable page and I can go through and I happen to be the release lead for this one, but so I can go through and use each item in the checklist to verify have we done this step, have we done this step or strike it out to say, okay, we're not doing this step and it's intentional, we know it. Right. So the exercise here is really quite impressive and what it's done is it's captured the knowledge that Tim has, that Oliver Gonja has, that Daniel Beck has, that Olivia Van Nien have about the process we use to generate a long-term support release. Yes. So, oh, in fact, here I've got to fix while we're here. Everybody knows where we are on it and we don't have one little thing that slips through the cracks. Well, and it's okay, it's helped me to reduce the number of mistakes made during certain phases. It's also a good way to highlight, hey, this release.ci, there it is, good. Highlight the kinds of things that may have issues, et cetera. So this one, for example, says run the job on a particular server, but we don't have the infrastructure set yet, so I just strike it out. And now my checklist has this nice strike through right there. Yes, beautiful. Yeah, so it really, and oh, this is publish a pre-release. Don't know if we'll do that one, think about that. So again, the amazing power of Atul Gawande's concept of checklist. So, all right, so that's the easy part. Let me put a link into the checklist because that's helpful for me. Now, next topic was Google season of docs. Okay, so this one, we've posted a draft application proposal and ideas to Jenkins.io. We've got to do the application this week though, so I wanted to get your feedback on what's been discussed so far. So let's go find that on Jenkins.io and it's in community documentation, season of docs, season of docs status page. Okay, so here is the organization proposal and here is the, when the application draft. Oh yes, okay, good. So that was what we need to discuss and here are the project ideas. Okay, so first challenge is let's talk to the project ideas. These are, I've put the one that I think should be top of our list and the one we actually submit here first. So let's, all right, so let's grab that and say, okay, so organization proposal is there. Okay, so the organization proposal is to who we are, what we do and the proposal here is we want to expand the documentation that describes how to run Jenkins on Kubernetes. Last year's Google season of docs started this effort. Xenob did a fine job with it, but there is much, much more that needs to be done and lots more that we can do and we think that Google season of docs is a great way to fund a professional writer to do that. Okay, so the idea then is here's the concept it says, hey, we've got lots of presentations and lots of articles, but we need to expand the documentation describing Jenkins on Kubernetes and concepts, techniques, choices, et cetera. With work areas and topics, here are the things that we had identified before as, hey, these are the things we'd like to do and we need, I've got to do some more detailing on this. The complication here is that in addition to this, we have to propose a budget that Google will give to the Jenkins project that we then use to pay this writer. Would that also be budget to give them a lab environment they can play with? Usually what we do is we go beg the providers to let us use, to have the provider donate capacity or we beg the Jenkins project infrastructure to donate. Okay. In, as an example, Oracle has a thing where they'll give you $300 with the credit for a month. Amazon has the same thing, Google has the same thing. So there you've covered three months just by switching from one provider to the next. And this is not include the agent issues for Kubernetes, right? Oh no, it does, it needs to in fact, that's part of the, part of this exercise is people need to understand how to use Kubernetes agents, how to use them effectively, how to deal with the challenges that go with they're being completely ephemeral so they disappear immediately after use, those kinds of things, absolutely. Well, they don't have to be ephemeral. Kubernetes agents will, okay, strictly speaking, they don't have to be but it's certainly most common that they are. I should be right. It's not mandatory. You can connect a static agent to a Kubernetes cluster, it's just not typical. Unless you're running Windows or you're running specialized hardware. I thought there was something that you might want to for Maven to maintain that setup or something. Right, and that's sort of the, that's one of the interesting complications, right? Is, hey, what if I want to cash build artifacts from one build to the next? Or cash dependencies from one build to the next? Right. And that's a cool topic for this project idea. So is this, what is the organization of the eventual documentation out of this? Is this a separate set of documentation that's everything about running Jenkins on Kubernetes or is it subsections of the regular running Jenkins and then here's the extra stuff for Kubernetes? Subsections of the Jenkins handbook is how we'd envisioned it. And so what we see now, if we look at the content we have now that Xenob created, we have an installing Jenkins chapter, if you will, in the user handbook. And inside that chapter, there's a section on Kubernetes. Right. And then in the, in other sections, there will be similarly Kubernetes, Kubernetes specific items that we'll add there. So for instance, system administration we'll have a Kubernetes section. Managing Jenkins will probably have sections that are detailed specifically for Kubernetes. So like managing security and security. Yeah. Pipeline certainly is a good one to have something about Kubernetes because, actually that's here, let's do this because you and I talking through this is really a good thing to do. Okay. So Kubernetes documentation sections and sections and topics, and let's just capture them. So we've got installing with, and Xenob's already done installing with multiple methods where it includes Helm and Helm and operator. And I think one other technique. But then we've got maintaining a configuration with configuration as code. And then maintaining job definitions with JobDSL. I don't know if we want to. I know I am deeply in love with maintaining job definitions in a Docker image. It just works great for me. Let's see, that's one we should, maybe we put that here as maintaining a Docker image. A Docker image, image of the Jenkins controller with plugins. Great. Are there areas of Jenkins administration that are not affected by whether you're on Kubernetes or not? There are, certain. Should we list, is that worth listing the things that are not affected? So I would make the assumption that things are not affected unless we tell them that they are because most things are not affected by that. Okay. But there are special cases like agents in a Kubernetes cluster. And maybe we call it ephemeral agents in a Kubernetes cluster. And then the more interesting one of static agents in a Kubernetes cluster. This one, ephemeral agents is probably in the 95% or more of cases that users want. But there will be times, and hey, here's this special case, you need this. Right. What about folders? Are folders, do folders show up in Kubernetes or are they just another thing on a master? So they're just another part of the, they're actually a part of this job, this Docker image definition and the job definitions. So those two things cover folders quite well. How about high availability? Oh, that's an interesting one. Yes, all right, so. It's such a hot top. I mean, basically you get it for free sort of, right? Except when you know, actually you get it, but you need to know how to recut, how to set up. For instance, your pod may be saved, but maybe you want to be sure that you're saving your data off on something too, right? And yeah, I think that's a good data. Let's, well, let's talk about, there's one that webhook. So managing, how would we say it? Managing startup time. No, what's a better way to say it? Improving availability. Let's just call it availability improvement. Okay, all right. So here it's things like handling pod failures. Here's another is webhook, webhook relays for while Jenkins is restarting, those kinds of things. This is one that Gareth Evans has some cool things on. So good topics, yes. Okay, and is that, do we need one on recovering from a failure? Oh, yes, very good, yep. Yeah, very good, okay. We may also want something on, probably something on artifact management, right? So where should you store it? Where to put your artifacts? Artifacts and why? And what happens, can you run, say, Artifactory on Kubernetes also? You can, oh, that's another good one. And does that change anything? I don't know, maybe you don't see that at that point. If you're just using Nexus or Artifactory to store. Yeah, it's a good question. So I was thinking of things like, for instance, if I decide I want to run Jenkins, plus Artifactory plus Giddy on a single cluster, right? Those kinds of things. Right. What about, I've heard mention of running Jenkins on a traditional VM or even bare metal, but having agents on Kubernetes. Ah, interesting. Okay, so that one is a relatively specialized. Let me note it though, so. I heard, and Lima was saying that they're seeing a lot of that, they don't, people have got what they've got and they don't want to rewrite that. But they need more agents and they want to get, well, they want to throw in Kubernetes agents and that that gives them a lot of flexibility and power that. Good, good suggestion. Okay, so that's where of agents, let's call it Kubernetes agents, Kubernetes agents to a non-Kubernetes controller. Controller. Controller, yes. And that's actually something using Kubernetes for capacity, even if Jenkins is, even if the controller is not yet in Kubernetes. Right. And that's a good story because we're moving that direction with ci.jankins.io. Uh-huh. Our big instance, Damian Deportals working on it, where we've got a virtual machine that hosts Jenkins and he's looking at eventually it will be on Kubernetes, but even before the controller is on Kubernetes, we need the agents on Kubernetes so that we can transition the jobs before we ever transition the controller. So by adding Kubernetes resources, we can detect Kubernetes specific issues, resolve them, incrementally make our transition. So incremental, so a progressive transition from locally hosted, if you will, from VM hosted to Kubernetes hosted. Yeah. Good, okay. Thanks for your patience while we go through this. Oh no, I'm learning more than you are. If there, are there any special security settings issues? Ah, very good, right, very good. Securing Jenkins in Kubernetes. And there we might say common mistakes, sensitive areas, managing credentials in general. So managing, let's call it managing secrets, credentials, et cetera. And the choices there include things like separate Kubernetes secrets or separate vaults, separate credential storage mechanisms, separate, what do you call them? Credential repositories. Okay. Like CyberArc, like HashiCorp's vault, the Azure Credential Store, AWS and Google all have credential stores that they use, that they provide. So good, okay. But what about authorization, that's done, does anything of that change on Kubernetes? It should not, but that's a, it's a good one to authorization is a good one to point them to the standard common techniques like, you know, and the common techniques, LDAP, Google Auth, OAuth, let's see, AWS. I think we've got something for AWS that does OAuth. Active Directory is certainly in there. Et cetera. By the way though, that is authentication, not authorization. Oh, oh, sorry, you're right. Thank you very much. Good point. Don't you hate writers? You should go to lunch with the writers sometime. It's really fun, everybody pulls out their red pencils and hits the menu. Yeah, okay. So authorization, those techniques, so RBAC, project authorization, those kind of things. Well, RBAC is our proprietary one. There is a role, oh God, there's a role-based open source one that is not we can call it. Yes, and it's called. Role-based strategy. Role strategy, right. Role strategy, project, project, role, is it project roles? No, just project status. But you assign everybody to a project and then get permissions to the project. Yeah, okay. And others, yep. So for me, those I'm less concerned about because the documentation is already there and accurate and should work. Pipeline is one though that should be in here because it's really, part of it is, how do we use ephemeral agents? And part of it is pipelines on Kubernetes agents and there are some capabilities that Kubernetes provides that give greater flexibility for pipelines that aren't available without Kubernetes. Okay. What about backups? Ah, that's a good one, right. So that one is- Because we have such a good story for Jenkins about backups anyhow. Backup, yeah, well, so backup is using, using, yeah, yeah, there's a whole story there. So using- It's not a pretty one. Well, using cloud vendor storage backup techniques. All right, so things like what I'm used to is the free BSD concept of, I want a snapshot. And that same thing exists on all the cloud providers. You say I want a snapshot, what you get is this point in time backup that happens in a matter of seconds. Right, and it makes me wonder why we ever talk about backup. I started, how do we not have good backup? But it's because we, you don't need them there. It's really an anachronism, isn't it? You just need the snapshot. But if you don't take the snapshot, you have no backup. So there's an element of that where- The snapshot is your backup. And if your file system does not support snapshotting and there are certainly plenty that do not for performance reasons, right? It's, there's a penalty that you pay by choosing a storage system that has snapshotting. And that performance penalty sometimes may be worth it, sometimes not. And they're saving and restoring. Right. Can I restore from not the latest snapshot? Right. Good, okay. Did we mention HA, but did we talk about like configuring the redundancy that you might need to make HA work? This, in Kubernetes, it's not really a high availability in that sense. What it's doing is it relies on, it's more of a failover behavior where you say, hey, my pod failed that was running Jenkins. The storage that's backing it is still there because it's on redundant storage. Right. The fact that it failed is, all I have is the downtime while I bring up a new pod that has the same Jenkins accessing that same storage. Exactly, that's, it's like, we actually need a conceptual, what does, it's avoiding single points of failure on Kubernetes. And I think there's still a couple that you can't get around, right? Right, right. And so that's, thus the story is improving availability. Yeah, if I'm not a Kubernetes expert, if I'm just an old IT hack, I wanna know what my HA is, how much I can have and how I set it up and how it works. Now some of that is gonna be somewhat familiar and some of it with Kubernetes is radically different as you said, because I just stand up a new pod and it's there. But while you get into things like saving, if you wanna be able to restart a pipeline, you automatically can restart any declarative pipeline from the last stage. Right. If you want access to more than the most street, which you can lose real fast by trying it, retrying it, you might lose your workable stage. So you've gotta do some setup if you want older, where are those older stages on Kubernetes? I have no idea. And do you do exactly the same thing? Good questions, okay. I know nothing here. I'm so over my head. See, and did I capture? Okay, so we probably, I'm not, okay, thinking about this one. So backing up a little bit. Pipelines on Kubernetes and how to use them well. Yeah, maybe. Yeah, well, okay, one more. So we've got tutorials on Maven on Python and on Node.js. Should we consider, as a possible candidate, a, no, see a Kubernetes tutorial is probably just too involved to be a tutorial. Unless it's, but how does somebody get their feet wet on this? I mean, we're not gonna teach they can go elsewhere for Kubernetes training. But it seems like there, it ought to be, can we give them some high level tour? Maybe it's a demo, but what everything looks like on, if you know Jenkins, this is what it looks like on Kubernetes and you'll do this and this the same and you'll do this totally differently. And this is sort of a hype. Well, and so there's a concept we've got for that thing. How about, what if we were to take, so I'm gonna open it up here. Let's look at, here's the concept of the solution pages and what we do here is, for instance, we include links to videos that are potentially interesting on that topic and the idea then is, okay, here's this solution page that has recent blog posts about this topic, links to Jenkins online meetups, links to other people's videos and we don't yet have a solutions page for Kubernetes at all. So for me, that makes good sense that one of these blocks here should be Kubernetes. Yes, it's basically, but we're getting better and better at documenting all the trees. How does a novice figure out that there's a forest there and understand the forest? Right, and that's where, well, and that's where introductions recommend, you know, good practices, user story, user experiences, and videos, et cetera. Right. Good, okay, well, thank you for helping with this Meg. I'm about out of time here, but I think what I'm gonna do is take these things and start putting them into more details on the draft on the proposal. Good, okay, yeah. And Yarl, you wanna prioritize some of those things, they're probably nice things for the long term, but not the top priorities for this year. Right, and that's where we'll just, I'll do my best, you know, best guess estimate, try to propose, hey, here's what I think we should be doing and this is how it should be structured and why it will help the community if we have it this way. Right, right, marvelous. I think we're set here. Yeah, good, anything else, Meg? I'm good. All right, thanks, we'll call it in. Thank you very much. Take care, thank you, bye-bye now. Bye-bye.