 Welcome everyone. This is Jenkins Platform Special Interest Group. It's the 6th of May, 2022. Great to have everyone here. Topics I've got, open action items, required Java 11, Java or so agenda items, Java 17 support, Docker agent, support additions under consideration and Damian has some information to share there. Exit and restart life cycle change that Basel since you're here, I wanted to do a brief question and answer session there just to be sure because I had put this on the list and I'm not sure that what I've said here actually is needed given how long we've been running with this. So those were the topics that I had. Are there any other topics that should be added to the agenda? All right, then let's go ahead. So first is the guilt phase of the meeting where we note that I've got an open action item that's been a long-term action item, plugin installation manager documentation is not describing all the use cases for plugin installation manager. We had a pull request submitted to Jenkins.io that attempted it but had a number of factual errors in it. We closed that finally and agreed with the maintainers of the plugin installation manager that we would rather put the documentation on the GitHub site in detail and then have a pointer from Jenkins.io to the GitHub repository so that maintenance will naturally follow with documentation but no change and no change expected because I just don't have time to do that work right now. Kevin, if you and I get some time, I might put you through it but there's somewhat of an expert level here needed that I'm not sure it's ready for you yet to take on. Any question on that action item? Okay, next topic then. Java 11 are newer for Jenkins Core. So the JEP has been submitted. We've got what seems to be a good consensus decision that September 2022 makes good sense as the LTS release and then we need to update the communication plan, the deployment plan, which weekly release should end Java 8 support so that we have good time understanding that it's okay, that the tooling is working, that we can deliver releases, et cetera and we need a notification blog post. Basil, anything that you wanted to report in terms of tooling improvements and additional testing? No, I don't think so. Okay, and Tim, any concerns from you if we go ahead with the idea that September 2022 will be the LTS that drops support for Java 8? No, but I think I just add that we would drop support on the weekly a lot sooner probably to iron out any issues. Right, and I think I assumed this was like possibly June or July, right? It may not be instantaneous after the LTS baseline is selected, but not long thereafter so that we iron out the issues in weekly before we get to LTS. Probably in the next few weeks, yeah. Yeah. So what that implies is that I'll be changing the banner, the admin monitor in the weekly branch that is the master branch shortly after we branch for the next LTS to include a sooner date than September since those users of the weekly line will need to be notified that we'll be dropping support for Java 8 sooner for their line than we would for the LTS line. That makes sense, good, thank you, yeah. So notify weekly users immediately that we're dropping within a relatively short period. Now I assume that the best way for me to do that is to submit details on dates and whatnot into the JEP. Any objections to that technique so that we can review it there in a PR review? Okay, that's good. Great, all right. Anything else on Java 11, on the required Java 11 story? Are there, are there critical problems that we need to resolve before we make the transition in weekly? I mean, we still have to resolve the remoting issue with Java 11, there's still one blocker. Which one is that? I don't remember the bug IDF hand but it's in the Epic for requiring Java 11. So there's an issue with the remoting that cannot be resolved other than by downgrading to Java 8. See, so is it, I need to make this big enough that my eyes can read it. Is it the Java 11 agent disconnection one? Right, ah, right. The only known workaround to this issue is to downgrade to Java 8. Is it reproducible? Nope. Right, it's no, the other looks, so we got them to try WebSocket as well and they said that they didn't have the assurance WebSockets. Ah, right, okay, I see what you're saying here is this one says, oh, no, but then Karsten says he did try WebSocket and still saw an error. Okay, and yeah, all right. So, so may need more investigation here given the non reproducible nature of it. Is it a, is it a hard block or do we accept that we may, we may end up delivering making the switch without fixing this one? I think it would certainly be preferable to fix it, but given that it's not reproducible, we might have a hard time doing that. Ideally we would fix it. Yeah, ideally we fix it, but without a reproducer. Like we can try, but. Okay. But I don't think we should block it on it. It does have quite a few people watching it, but it's not terribly active. As it's been open for three years, but I guess people have had the workaround of downgrading. Well, and wasn't, was this one that initially there was a problem that WebSocket needed some changes and WebSocket got those changes and then, then now, so at least this one says, hey, WebSocket worked. The other one says, no, didn't work for them under heavy load. I mean, we should probably reply to that and check what the actual error was. It may be a different error because I don't think, I think this, so this error is a specific, so looking at the, I looked through the code of this didn't understand too much of it, but the code is a specific check in the, in the remoting protocol, whatever it's called, JNLP Connect 4, whatever the protocol is, as a specific condition that is added there as a check saying that it should not be and that it should not be in this state. And I don't think WebSocket has that same, that the WebSocket protocol is different. Okay, so made the request, feels like a viable request. Let's keep it on our list and keep the worry. We'll go, we'll proceed forward. I'll submit some proposed dates for the transition into the JEP. That proposal probably won't come until next week. Today's kind of busy for me, so I don't think I'm gonna have capacity to do it today. That work for everybody else if I'll submit it and make reference to Tim, to you, to Basel for a review of the proposed change in the JEP. Okay, great. All right, thanks. Anything else on Require Java 11? Okay, next topic, Java 17 support in Jenkins. Basel, you wanna give a brief, brief discussion there how things are going? There hasn't really been an update since last time, but the tooling has already been done and plug-in parent POM 4.40. So we are testing the bill of materials set of plugins with Java 17 as of a recent change. So we should be fairly confident that plugins are in decent shape. There still are some interesting tasks that are open in phase one to get core fully supporting Java 17. There's one very interesting issue that Alex discovered with Jelly that could be a fun bug to solve, which I've included in that epic, but there's basically a type resolution subtlety with how Jelly resolves types that affects certain uses of Jelly, including one of our build trend pages and I actually added a test to Stapler that fails on Java 17, but passes on earlier versions. And that could be a very good starting point for someone who wants to work on that. It's kind of a fun programming language runtime bug for anyone who's interested in that kind of stuff. So that's... So the other interesting part was that Groovy isn't affected. That's right, it's only Jelly views, but not Groovy views that are affected, which is pretty interesting. So that's one of the core issues that remains. I mean, that's probably the biggest open issue that I can think of. I'm not seeing the epic in front of me right now, but I think that's really one of the most interesting things we've discovered so far. Yeah, I mean, most of this other stuff is fairly minor, like the file leak detector socket support. These are very small edge cases. But yeah, the programming language... This one, right? Yeah, that's the one that I was talking about. So I mean, this one, if anyone feels like tackling this, I've got a unit test in Stapler and you can execute the unit test in half a second. But the difficult part of this is maybe the five minutes or five hours or five days or whatever it takes to come up with a solution to this, which is gonna involve kind of digging into the depths of jelly and how it evaluates expressions. Yeah, okay. So if you like exploring meta languages, here's your chance. That's great. Yeah, excellent. I cleared down my existing stuff. I would look at it, but I'll get to it eventually if no one asked us. Right. Well, and truly Java 17 support, we're in a good... For me, it feels like we're in a good state, right? The tooling improvements and the amount of automated testing, the growth of plugin testing, that many, many plugins are now testing with Java 17 in addition to Java 11 and eight. And when we drop support for Java eight, we can reduce our spend on infra again by not testing plugins with Java eight anymore. So there's a win coming soon. Well, right now in the bill of materials, we're testing them with 11 and 17, but you're right that in the individual repositories of each plugin, their own Jenkins files are typically using eight and 11, very few have added 17 to that. So what I imagined we'll want to do is encourage plugin maintainers to do a sweep and basically remove eight and add 17. And I mean, we could have them do the sweep now and just add 17, so they could have three right now, but I don't really see a big push, I don't see a big reason to do that since we feel covered enough by the bill of materials testing, at least for the top 100 plugins. Right. There's nothing stopping anyone from adding, in fact, for all of my plugins, I've done that just to see what happens, but by September, I think we'd want to give the very clear message that, please go and remove eight at 17 and we want you to do this now, rather than to wait. Well, and it truly does not make any sense for a plugin to be testing eight when Jenkins Core weekly, and then eventually Jenkins Core LTS no longer supports eight. They're just, let's admit that the product is moving forward. Good. All right. One other thing on this is that there's at least one bug that we can't fix to move to Java 11 on Core because we need to add some JVM arcs for spotless and those fail on Java 8. So we can fix that once we move to Java 8. Yeah, and in fact, I've already fixed that in the pull request that drops support for Java 8. If you look at that pull request, they did a couple of interesting things. I first of all, I fixed that to demonstrate that requiring Java 11 will allow us to fix that. And I also, in one particular source file, I added a Java 11 only API usage to also demonstrate that we can now do that, which we couldn't before. So that there's those two parts of that pull request are just demonstrating that. Well, and isn't that that Java 11 is also an opportunity to use UTF-8 for property files? Yes, that was a huge file. I didn't know that was option. So which makes localization much easier then. Well, yeah, Java 9 plus has a, I mean, Java 8 has already the API for property resource bundle. I think that's what it's either that or property bundle, I think. Yeah, whatever it's called. There's already a class for this in Java 8, which we're not using. But if we were using it, it just so happens that they have changed the behavior of this class in Java 9 plus, so that the behavior of it in Java 8 is to only open the properties files in ISO 8, 8, 5, 9, 1 encoding. They've changed the behavior of this class in Java 9 plus to try it open with UTF-8 and to fall back to ISO 8, 8, 5, 9, 1 otherwise. So if we were using this API, we would just get this improvement for free. But unfortunately, Stapler isn't using it. It's rolling its own implementation of this. So my, please. I was just gonna say there was an effort a few years ago to introduce XML-based property files because that could add UTF-8 because some contributors got so frustrated trying to edit these files, but other contributors didn't want to have to use XML so it didn't get accepted. But yeah, this would be huge. There's so many encoding problems. Yeah. So my thought was if we switch Stapler to use this standard Java API instead of rolling its own implementation, which we could do right now and there's nothing stopping us from doing it today, then we'd get for free this UTF-8 support whenever we switch our runtime to Java 9 or later. So to me, this is just some cleanup of Stapler, the fact that it's using its own implementation rather than the standard Java API for this. Thank you. Great news. All right. Very good. Anything else on Require Java 11? And we just talked through, I think we settled on Java 17. Anything else on Java 17 support? Okay. The next topic, support additions under consideration. So we'd had a request for Docker on Windows Server 2022 LTSC. Damian, what do you want to share with us there? That we started just yesterday to provide the required underlying infrastructure providing Windows Server latest version. So that will allow to execute Windows containers that are both Windows Server 2022 and the previous LTS as well. So that could get the required tooling. So as soon as we will be able to do that, we will notify the issue and so the contributor, whoever that could be, I could start opening pull request on Jenkins CI slash Docker repository. Well, and we've actually got an open pull request already and it's failing. Yep. So we will have to adjust the label on CI Jenkins Iodo. We'll have to check if we want to keep the two images at the same times. We don't have any reason not to for now, but we might. Well, actually, and I think we actually do have a reason to keep the two images because Windows has the lovely behavior with Docker images that you must run an exactly matching kernel. And so if we're going to continue providing LTSC 2019 images, we've got to have a place to build them. You have low level compatibility. Yes. I don't know what they're thinking, but that's their reality. Windows requires that you have to build it and run it on the exact kernel match. Yes, but you can execute a container 2019 on the kernel 2022. I can you? I had understood you cannot. Just tried yesterday on the sanity check. But okay, given what you said, we will have to pay more attention and we will go the road having both at the same time first or having a path to tests. Okay, interesting. Thanks for the sharing. Yeah, just at least now I haven't double checked. So you've got better data than I do, but my understanding from a while ago was they were tied to kernel version and explicitly. So it was it's much, much worse than the Linux world. But that's a good thing that we share that because that might go easily for that pull request. However, that might have issues on the release process because the Jenkins Weekly or core process package using Windows container. So we have to be careful about that part, but I want that part is managed by AKS so that should go easily. We have great. I've sent a link in the chat detailing what Mark has been referring to the sum. They've done a bunch of work on Windows 11 and Windows Server 2022 to make this a bit better. So some scenarios may work across versions now, but... Oh, okay, so the policy change. Yeah, it was changed as of the last versions of Windows. Thanks. So that means that what I tested yesterday switched on my back the process isolation to hyper-visualization that will explain why I saw it working based on the chart on the doc. Yeah, I don't fully understand it from a quick read. You need to read a bit to understand it. My container test rate yesterday started a nested virtual machine inside the virtual machine where I was running and it executed the container 2019 inside that nested virtual machine. Okay. Yeah, so I assume, Damian, I think you're on the right track. I was worried this was going to be much longer delayed. Anything else you want to share there in terms of progress on potentially adding support for Windows Server 2022 LTSC? Nothing more. That means we need to keep both version lines for a few months altogether. Okay. All right, next topic. This one was, I carried this over from last meeting because we hadn't had a chance to talk to it. There was a change made in 2.344. So that's now three weeks ago or two weeks ago that changed the lifecycle to use a simpler lifecycle for inside Docker images, right? So instead of, Basel, if I remember correctly, instead of some complicated Java code inside Jenkins that was doing the exit and attempt to restart, it just now stops and let's Docker restart it. Did I describe that correctly? Yes, that's right. And so I was worried when it was initially proposed, oh, this is going to be a total disaster. Somebody's going to be surprised. They're going to be very angry at this restart behavior, but we've had no reports, no noise from anybody that I've heard about this change of behavior. So I'm not sure we need a blog post or any documentation changes. I didn't detect any. Any guidance from you, Basel? Oh, we just shipped it in LTS yesterday, right? So we'll probably hear something the next week if someone is upset about this, but I think a lot of people were probably already running with a restart policy, you know, maybe not for local development, but I think people who are using Docker compose or Ansible or some other DevOps tool to set up their images, they might have been using that option already. So maybe that change just wouldn't affect a lot of people because they were already using the option. You know, I don't see anything wrong with adding an entry to the release notes, like where the upgrade guide to point this out and draw people to the documentation that we've already written in the Docker GitHub project Jenkins CI Docker. We've already updated the docs there to mention this restart flag. So I don't see anything wrong with adding an upgrade guide entry, just to point people to that and highlight it as a minor change, but I would tend to agree that if no one's complained about it that a blog post seems maybe a little bit too high level or just too much for something so minor. I may have totally misunderstood the question, but since I upgraded to 2.346, as soon as I install new plugins and click on the restart when it's installed, my Docker container just exists and never restarts. So it may not link at all, but I'm experiencing some issues with that these days. Interesting. All the documentation that's said to add the restart flag when you started the container. No, I didn't know about this documentation. The only thing I was doing each week, I'm taking the latest Docker image and from this week, it doesn't work. And okay, I would have had to have a look at the documentation, which I didn't do because I didn't know about that. My fault. Yeah, so that's exactly the change, right? And we documented it to say that you need to add the flag in such a scenario. So maybe my use case is totally mine and nobody else will do that. Just have a very recent restart each week and not looking at the documentation, that the changes you have to make in your installation for it to work. Okay, I will have a look at the documentation. Sorry for the... Yeah, I think that's probably the main use case where people would hit this is local development, right? When you're just spinning up Jenkins with Docker run on your local machine to try something out. I think that's the scenario where people would forget to add this flag and be affected by this the most. Rather than, for example, the production use case where they're probably using a tool like Docker compose or some other tool that they would configure and check every setting, make sure every setting is set correctly. That's something you probably wouldn't do just on a local machine to fire something up quickly. So... Okay, yeah. I'm not sure how we can advertise that better, maybe other than putting an upgrade guide entry. Yeah. I was just thinking of Sheikah the Shuka mark and maybe we should tell them that they should change the configuration if they are upgrading to the latest version to make the screen shots, for example. Yeah, and I think most of them are actually running, executing the war directly rather than using a Docker container because I wasn't confident that they all had Docker capacity on the machines. Okay, so good insight. Thank you. Good question. Wouldn't that be worth a minor but quick blog post just restating what Basil just described like, hello, if you're doing developments or tests, then if the container exits, that's a standard Docker feature, as usual points to Docker documentation that mentioned the restart flag and then say we have updated the documentation for Jenkins, we recommend you to use restart always all the time so your container will restart. That should be short blog post. That's fine to me. If you find, if you think it's fine, I'm willing to try to write something to help unless someone is motivated, of course. That would be great Damien. And I could be the proof reader as I'm having the problem. So I could just test if your blog post works for two people like me, okay? Okay. Yeah, I like that. Good. Yeah, there's still one more bug we have with the life cycle, which is that on M1 Max, the restart after a plugin upgrade doesn't work on M1. And I actually found the root cause of that bug, but I just haven't found any time to fix it. It's not a terribly complicated, it's a medium difficulty fix, not very difficult, but also not very simple. But I think once that bug gets fixed, we'll finally be over all of these exit and restart issues with system D, with this Docker change and with the M1 fix, I think we'll finally be in good shape with restarting Jenkins. We're almost there, but not quite yet. It's like something like the system called bindings is wrong as the net tool. For the M1 issue, yeah, the way that we're invoking the system call from Java is encoding the arguments incorrectly, basically. And Tim, with your M1, you haven't been bitten by this or you've just learned to live with it? I think I created the issue. Oh, okay, got it, I see. I tried to, yeah, I got very lost. It's been quite a while debugging it, Basil managed to get to the bottom of it, but the main problem I had with it was during acceptance test harness and the Vincent's re-implemented the restart logic in ATH about a month or so ago to use Exit and to have ATH manage the process. So it's rather than relying on Jenkins restarting, ATH manages restarting now. And since he fixed that, it doesn't really affect me as much. Got it, okay, so your M1 with its 30 or 40 cores, however many it has can do all this work for you and Jenkins is not doing the restart, ATH is doing the restart. Yeah, I was working around it by grabbing the pro, from grabbing from the logs, the process, copying it out, reformatting it and then starting it myself manually, which would allow the test to continue. That was my workaround, which did work, it was just painful. But Vincent, see me recently has fixed it, so I don't have to do that anymore. Excellent, all right. Yeah, ironic part about this bug is I found a way to fix it easily with JNR, which I had almost the same week ripped out of core. So to fix it with JNA is slightly harder because it doesn't give you any convenient wrappers to encode these arguments. You've got to do it yourself. And I did a Google search on how to do this with JNA and the only results I found in the Google search were the incorrect logic in Jenkins. So... All right, so... Similar, there's been similar problems before with something similar where I think other libraries had copied from Jenkins, the source hasn't got it wrong for something quite similar. Yeah. That's the benefit and the detriment of being a 15-year-old, nice, mature project. People find you and duplicate you even when you're flawed. All right, anything else for this meeting? Any other topics we need to discuss here today? Okay, then thanks. We'll stop the recording.