 Hey Marcel, thanks for joining Jonathan likewise. Thank you very much Marcel just recently and not long ago upgraded and is using Java 11 in his in his production instance All right, I got it All right, so I think I think the the crucial questions here Maybe let's let's start the the Conversations Mike you had you had assembled some initial slides Do we want to look at those first as a first first conversation starter and I'll take some notes in the the notes while we're discussing Yeah, I'm not really sure how to See a Share my screen. Oh Hang on. So let's And it may be that we don't have you granted share permission. I did hit record. Oh good You did so you should so right next to the right here. I was dead here. Okay But you have turned on recording. Yes, recording is on Excellent. All right, so really this is just a very skeletal outline of what I thought Topics for this session would be Installing with where we are right now So there's currently Java 11 support since the 2.164 LTS Relatively I guess relative to the relative term, but there's 55 open issues regarding plug-ins and interoperability with Java 11 and 18 issues Currently open related to core Jenkins And more if you want to add anything else about the current state Yeah, yeah, so the one of the one of the points there is we are we are now in the Jenkins dot IO documentation recommending users Use Java 11 and we're guiding them to use Java 11 in the Docker images that we build and in other other Locations suggesting it still the Java 11 adoption rate is is Relatively lower, right? It's most of the Jenkins users are still running Java 8 How are we collecting that information really basing those numbers off of? I think that there was some stats Let me make some notes here so relatively low and I believe that part of the stats generation process stats Collected by stats dot Jenkins that IO had some Java 11 numbers in it and Tim Jacome had shared some with me. I didn't I don't recall what this precise source was So that's an action item. I could take if you'd like of find exact numbers relative to How we have we've been collecting that as just anecdotal I Know and it was it was actually numbers. This was a this was a reported number of Kate of places where Java 11 was Reported as the running JDK and I believe the JDK is reported by Jenkins to the stats collection Did that answer your question? Yeah, I did Thank you So I guess before we jump for is anyone else have any Comments or questions about the current state of Java 11 Support in Jenkins So maybe it'd be a worthwhile thing to poll here. How many of the how many of the participants are actively using Java 11 already? I know Marcel is Ivan are you using Java 11 or is yours are your production instances still on Java 8? Yeah, that's in that day chicken right now and We have a mix we have a few in 11 and a few in eight Great. Thank you, and I know Tim. Jacob switched over to Java 11 about Eight or nine months ago and has had good results Jonathan Java 8 or Java 11. We're still running Java 8 we have Underlying parts of our the code base that we're running building through Jenkins is does a lot of crypto and There's a lot of things that broke going from Java 8 to Java 11. We're looking into it, but we haven't really Even if we run Jenkins in Java 11, we still have to be doing our builds and stuff using Java 8 for now Interesting very good. Okay, so Jonathan running 8 and Has dependency now. Are you using the Maven job type? That has a strong dependency or using mostly freestyle jobs not Maven the Maven job They're mostly freestyle and jobs we've we're starting okay, I go over to Ivy. Ah Good sort of mavenize things No, that's perfect because that means you're you could run Java 11 and just build run your builds in Java 8 good Marcel you had reported that you were running Java 11 already Yeah, I might ready to 11 it was the Mario great that we have on the LTS version two months ago, I believe Great excellent. Okay, and then run shit anything you need to report in terms of Java 11 use adoption in your world Um Relative limited right now actually most are on Java 8 James or Emilio do actually James. Let's go to you first and anything on your sense of Java 11 adoption None as in it's all Java 8 Thank you waiting for waiting for Mike to finish up some stuff. I Like that Emilio, how about you? What's your Java 8 experience or Java 8 versus Java 11 in your production instances? No, it's exactly the same then James. We are waiting for for Mike Oh, it's yeah, bye All right, and some how do you both we're gonna throw me under the boss And I assume Kenny is yours the same story there. Let me get off you. Um, yeah, it's The same story as we're still Java 8 you Great. Okay. Good. Good to know. Thank you for give the impromptu survey. I know it's unscientific Etc, but I I suspect we've we've already heard a good representative sample just by that great Yeah, thank you everybody And what's what's next I'm not aware of anything off-hand and it's mostly because I haven't Maybe dug deep enough, but I'm not aware of any major Java 11 related Updates and for the September LTS But I'm not as involved day-to-day and In some of that so I don't it's quite possible. There's some things I don't know about I Think if anything it'll just be incremental You know, maybe some more plugins have been fixed some of those plug-in issues or core issues have been resolved But I don't know if there's anything actively in progress or targeted for that So I had started on that first bullet. I had started a mailing list thread Proposing that we would switch The switch additional defaults to Java 11 With the September LTS the particular examples there were the docker images. So today we ship a docker image that's based on Debian with so we have a docker image named Jenkins slash Jenkins colon LTS So it's ambiguously named in terms of it doesn't promise what JDK. It's running it doesn't promise what operating system it's running and my proposal was September might be a good time to upgrade its operating system from Debian Buster the current thing it's shipping to Debian bullseye So Debian from Debian 10 to Debian 11 and use that same place to upgrade that thing from JDK 8 to JDK 11 is the thing we're shipping So the the idea there was Daniel Beck had lobbied. Hey, please don't Change those kind of major things except on the edge of an LTS, you know a dot one LTS release So so my proposal there had been Interesting I would have said really do you want to throw all of those changes in along with LTS change right right sure you don't want to do it with a security fix but I would have thought a Don't know if there's a possibility of doing a release that is There's not a Jenkins release is just a container release Wait through and I think I think you've got a good point James And and that actually is a it was one of the so when the last time we did a docker change what we did is we did it sort of midstream and and one concept might be what if we did the switch from the image switch in a weekly first and Change the docker image weekly four or six weeks prior to LTS and then admitted we will change the LTS On a dot one after we've had four or six weeks of testing the same kind of change in weekly I think that might be sort of what you're what you're you're describing Yeah, it's you tend to get well The the experience I've had is you tend to get different users using LTS than you do do using the weeklies And they tend to have kind of maybe a different plug-in set people running the weeklies Either don't know about the LTS or or or their highly skilled Jenkins admins and they kind of curate their plug-in lists and remove plug-ins when they become No longer used on their instances whereas the LTS is kind of like that they might have been running for a lot longer by maybe people that aren't as diligent in kind of removing plug-ins when they don't get When they're no longer needed, they just kind of like stay there and they just upgrade stuff and it's kind of like well I upgrade upgrade Up to get kind of thing So I'm I'm I Would be cautious at saying that there's the same coverage between weeklies that you would have Is that would that be enough overlap? It would just be what I would be unsure about So the the notion there is while while it might be a I guess let me test it at sort of a different angle Would you object if weekly switched from Java 8 to Java 11 at a different time than LTS? Absolutely not. Okay, so so that's that's not a barrier It may it may it might be considered necessary, but not sufficient There may be still even much more we need to do before we dare to change the LTS from from Java 8 to board I if Can you correct me if I'm wrong Are we currently publishing Both Java 8 and a Java 11 image and it's just what is the default? We are publishing both a Java 8 and a Java 11 of the image and and the change would be It's actually a very good point today We have a thing named Jenkins slash Jenkins colon latest and the proposal was switch that to Java 11 But the problem is that leaves no docker image that's delivering Java 8 then for the weekly Should we not introduce then a Java 8 one for the weekly that I think yeah Yeah, I think I think that's a good idea that maybe we should declare all right to keep to keep transition Viable we say look we admit we were ambiguous We should have had an image all the while it was called JDK 8 and now we're going to create one call it Jenkins Jenkins colon JDK 8 For those who want to remain with JDK 8 And until support it's removed right until we stop doing JDK 8 Yeah, that would have the benefit that okay by default they switch from 8 to 11 and If they say no the default is unacceptable they have a place to return to if they'll change their configuration Or if there's issues right because I don't think anyone would say 11 is unacceptable We're mandated to run Java 8. I'm sure there's gonna be some odd that would but Yeah, it's just the yeah it upgrades and it all goes bang Okay, I think I think at least for me that seems like a viable thing Ivan. What do you think? Yeah right now if we release and the default for For the gate and also with we release LTS dash GT key 11 if I record currently so we can add another tag that is LTS or wherever we version we want to release dash JDK 8 and that's it We keep the default with 11 and we release another two tags one for JDK 8 and another for JDK 11 Right. Okay. Good. So so that that makes sense to you Marcel I think that would not affect your your system at all right because you've already made your base image Use the dash JDK 11 variant Yeah, I'm not affected with that great and and so I think that's that at least for me seems viable Mike Do you see any barriers to that kind of idea anything that we should be worried about? um, I mean the only thing that comes to mind would be if we don't Advertise and we've made this change loud and clear people might be a little confused or surprised when they just grab or base the image off of Jenkins colon latest I don't really know any way around that because I don't can't really Change that name pattern Of the Jenkins slash Jenkins colon latest So that there's only ever explicit JDK version so So as long as we do a good job of communicating across the mediums that We've made the switch into default but you're welcome to Still pick a JDK specific whether it's eight or eleven base image For any given release It shouldn't be a huge disruption even in the worst cases. I think Okay, good that make that makes sense to me all right, so so now That that touches and the slide has the next topic is java 17 Are there other things we need to discuss around java 11 for me? What what's been proposed feels viable And a answer is one of my big concerns, which was some of the threads On the on the initial september LTS idea said hey, let's just drop java eight At at september and for me that's too big of a drop This soon it would it feels like to me We need three or six months more than that at least To get the word out to get more people onto java 11 before we would dare consider dropping java eight Do others have a different approach on that? Do you feel differently than that? I'm I'm admitting I'm giving my biases to compatibility and retaining things rather than moving rapidly to java 11 fully I would say i'm pretty much on the same page as you mark with that approach I think changing the the fault image to 11 will help A lot of people maybe on to java 11 without even purposefully intending to you Which I think you know the bigger that base becomes The easier it's going to be to to drop the java eight support sometime down the road You know one thing I think you have to remember for some people like for example one of the reasons Like we have a technical reason for having to stick with java eight. We also have a Procedural one because we do a lot of work for the u.s. government And we've been through multiple rounds of compliance and stuff. I know there's better ways of doing that but The other one also is that with open jdk The active support ends in 2022, but they're continuing support for security to 2025 now um End of march 2025 so there may still be people who are going to be sticking to java eight for quite a while longer So i'm not sure I know that probably because like i'm just new to all all this stuff So it probably causes lots of pain on the jenkins build and test and maintenance side, but i'm not sure there's Just a thought on my side. Sorry. Yeah, so jonathan your voice is saying java eight support may need to Last quite a bit longer just because of the The requirements that that your organization and and organization like it have for a stability or or needs for jdk eight Yeah, i'm saying that that that's my I mean this but i'm not i'm just saying just in general just because of the fact that there's still some runway left on java eight There may be other people in the same position Where for whatever reason they're keeping it? I mean ironically, there's there's more availability on java eight than there is on java 11 Oh, i thought i know what do you what do you mean by that? So can you describe that a little more? Yeah, the end of the end of support for java eight lts Is at least may 2026 And java 11 from uh adopt jdk Java 11 is at least october 2024. Yeah, so so so they By moving to java 11 you end up with less support Or less of a date, but you know the the idea is once once you've moved to java 11 Move into 12 13 14 15 and stuff. It is a lot easier. Yeah So so i think that's why they've not got it. Um, i'll just paste that there In the okay, so for you Yeah, thank you Well, that's what i think that's a good point to you is that i think sort of along with the specific discussion around eight and 11 changing our mindset and approach so that these kind of things become more normal and less of a spectacular event That happens once every decade This a position that will be good for jinkins in the long run so that you know, we do this successfully with java 11 and Use that experience to sort of have a pattern of you know, we were able to go from eight to 11 and You know, it's now much easier to go to 13 or 17 or 18 Or whatever Down the road and sort of have sets of expectations on You know what a user can expect in terms of support and keeping current at the same time Like i'd like to see You know it become easier for jinkins to to keep up with You know changes in the java world in the long run Good point. Okay, so your your point is it's best to keep pace with the java world um I've been less concerned about the 13 14 15 16 because they weren't lts and so their life spans what if i remember I'd only like six months or 12 months of support Whereas 17 i think they intend and i haven't seen any variants from that of They really intend to make it a long term support release comparable to java 11 Yeah All right, so are we are we at a point where we're ready to talk to java 17? Or are there other topics around java 11 we should be discussing before we move on Okay agent images containers Good, um, what the alignment with them. Um, I i i'm completely unaware of What what what exists and how they are. Um, but I know we we have historically seen issues when you run kind of a uh jenkins controller on java 8 and an agent with java 7 and Things didn't work correctly. Um, I don't know if we've still got an admin monitor um to detect that kind of thing or also how the Images kind of if there are both 8 and 11 images for agents that would need to be changed as well Or not And they're I think you're correct. There certainly are so it's not enough for us just to update the the Controller images. I think we have to update the agent images with a similar naming pattern And and so that means the agent images would get a dash jdk8 suffix At the same time and they would get uh switch their default to jdk11 so Docker inbound agent colon latest would Before this change be oh no, okay now james help me think about this Should that be on the weekly transition or on the lts dot one transition because I use the same agents in both places, don't I? Do you I don't know because A version of remoting goes into a weekly and a version of remoting goes into lts and those versions should be different and ideally you want to use a version of remoting in the agent that much is The controller you don't want to use a newer version of Of something on on an agent that's on the controller I know you can use an older version on an agent than is in the controller Um, but I'm not sure it will necessarily work the other way around it might do um, I I don't know if there's I I honestly don't know what what exists. Ivan's kind of shaking his head and kind of telling me no, but Unmute and tell me For inbound inbound agents Uh, I think that the remote is in the in the image, but for all inbound Ions, let's just say each one The remoting is is copy when you make the connection Yeah, yeah that there's only we only provide containers for inbound though, don't we for um, I think we provide SSH agent images as well Mr. Court Yeah, so we've got we've got mike posted in the chat the link to a list of the images that I think would need adjustment and so the Jenkins slash agent is a base image, but then we it's turned into two images that are used for For actual agents right janken slash ssh dash agent Ivan I think that's the one where you said the remoting is actually downloaded from the controller to the agent That's it. That's Yeah, so that's the outbound and then but inbound bundles and right now it's only got two images defined One for alpine to be the minimum size and then latest And so we would revise latest and then we would need something Maybe latest dash jdk8 or something like it to say hey, we're we're we're touching this thing Good, okay Well that brings up a question. I have two which is What does this mean in terms of interoperability testing for builds on jink and zi Are we you know Do we anticipate needing to Do extra testing between mismatch agent versions than we are currently doing to like that's not really an issue Probably today. I'm not sure if we're doing any Yeah, that's that's that's actually clean You ask a fascinating question because I I failed to tell people that ci.jankins.io has been running java 11 for I think a year or more And so so we're already if I remember correctly, let me double check just to be sure But I think it's I think it's already on java 11 And has been for an extended period Are there are there any things like eths right now that run with uh an agent on one version and Jinkins like you know jdk8 agent the java 11 jankins the the eths Just used the java that's there from maven and they're spinning up local agent as far as I'm aware that there's a View that I think might launch a docker agent For some testing And that's a good question. I have no idea what those are Yeah, so so as far as I can tell I look at our agents on On ci.jankins.io and the ec2 agents are definitely running java 11, but then they have inside there a Target or a label named maven and that definitely uses java 8 And if you want to use or or if you want to use java 11 with maven, you have to call for maven-11 So we've we've already got a good mix of Running java 11 on the as the agent process But invoking java 8 inside of it to do the compilation okay so the ssh agent connector is using a From jankins java with some build no idea what it is d9 3 It's Interesting. Okay. All right Great Are there other other topics around java 11? What else have we missed? Those are all the ones that I'm Aware of or can think of at the moment Okay, so testing testing the waters then we said earlier And and I'm prone to propose a jankins enhancement proposal that would describe this more precisely That what we would do is Four to six weeks prior to september lts. We would roll The controller images for weekly so that they are using Java 11 instead of java 8 And in their default images the latest the image colon latest and the the others like that which have no java version number included in them We would add a jdk8 Image so that people have it to fall back to if they need it And then we would watch for those four to six weeks To see if there were any disasters detected that cause would cause us to roll back that change before september lts Then similar change in the september lts Now james, I think you had concerns about Too many the the changes I was describing operating system update java 11 being too many are there is there a subset You would prefer to recommend saying hey, what if we did this instead? I I think as long as there is another image so that When if if someone upgrades and it all something breaks for them It's it, you know, they can just go right. I'm going to switch to the This other image. I'm going to report the issue. So this is broken, but I can still upgrade It's kind of like Not a blocker for them to upgrade jankins because we want people to upgrade jankins So the the more The more times we can remove a blocker and that's if We it's going to be on java 11 now and you're going to get a new operating system and you're going to get a new dot one of jankins Oh, and if that for any reason doesn't work Let us know and there's this image is here You can move to these eight lines But we will be removing them at some point in the future some point in the near future So it's kind of like hey We've given ourselves almost like three months to Make sure that people can move across Maybe even less maybe two months maybe a month however long it is I like that. Okay. I like that a lot So what I put it in is if jankins slash jankins colon lts switches to java 11 Then they and they had a problem They could Switch to jankins slash jankins colon lts dash jdk8 this new image. We've given them an escape hatch Yeah And that should be a relatively easy escape hatch for them to utilize. Um, I would think I don't know if we can get any stats from docker on on the number of downloads So you can actually see if that image has even been used by anyone Which you know, if there's no downloads after a month, then it's kind of like well, okay Right. That's good. Great. Brilliant big big victory. We're delighted Yeah, I like that. That's a good idea The if the jdk8 image is no longer does not get significant uptake That gives us further ammunition to consider Hey, we've successfully transition transition from 8 to 11 Yeah, I think that's an important Um aspect of this whole process that we should Um, definitely be trying to get as much um statistical data around uptake And you're both java 11 and 8 So we can make some more decisions I mean having said that I'm kind of I'm I'm not the I'm the chicken not the pig in this kind of restaurant If you're familiar with the analogy Okay, so I'm not sure who the pig is but I feel like the pig and I feel like that if if things go badly wrong I'll I'll hear about it all sorts of ways But even as the in that role I still feel like we've got a a viable proposal To say hey, we're going to allow an escape hatch I I like mike's argument that we should be sure we have data to tell us How how much this the escape hatch is actually used and I'm not sure if we've got that but docker hub certainly does count downloads So they they've got they could give us some estimates cool Yeah, I'm sure we'll hear of all the failure cases, but I don't think I have no idea if the silent majority Is happily chugging along on java 11 I think I'd like to assume that but the more actual data we can get to pack that up Right Yeah Very good. Okay. Anything else on so we're we're about 45 minutes into this session We've got about another 15 minutes. We've got two topics um I would propose we talk java 17 next Just to see what we think about it, but I still want to get to the What kind of timeframe would we consider or envision for drop eventual dropping supportive eight? Okay, if we proceed through those two