 Alright, welcome everyone. This is the Jenkins platform special interest group. It's the 4th of December. Thanks for joining us. So here's what I've got as agenda topics. Open action items and simplified labeling of Docker images proposed by Jim Crowley. Then Windows Docker images for LTSC 2019, Gareth, Docker image support idea, official versus unofficial Alex, a proposal for me to add Gareth as an image maintainer on the Docker repository and setting a goal to get the master branch working on ci.jenkins.io. Any other topics you'd like to add? Okay, then let's go through action items. So yes, I still have the action item. Yes, I think it's shaping now started the process this morning. Last, last evaluation checked various images for problems. And we have plenty of them that plenty of examples to use to highlight plenty of examples to use as test cases in the in the JEP like Docker or Debian nine not end of life open J9 needs a maintainer, etc. So I'll include those in that JEP, and I hope to get it done quickly. I'll send everyone what I'll do is send everyone a draft privately likely so that we, or would you Alex maybe you and Gareth kitchen give me insight should I submit it as a pull request and draft to the to the JEP repository and we just do it in the, in the JEP repository or the user worry that hey this is going to be so controversial that we should review it as a group first repository. And then like very few people like monitor that repository, I think so until you send out the JEP on like the dev mailing list or something like that I don't think it's going to have a lot of like a lot of people are going to have an issue with it. Okay, so you're okay with it being a draft PR and we admit it's draft, it will, it may need radical changes because I may propose something that is so controversial that we say that's not acceptable we've got to throw that idea out. Yeah. Great. All right, next one was sent those options for adopt open JDK Alex anything you want to share there. Yeah, so I have a, I should have a PR soon that uses the Yammer repository from adopt adoptium. So I'll create the PR and then we can have discussion on that PR just to see if people really care or not. So, and that was a question I had there I'm not, I'm not confident that I detect signs of life on Sentos operating system Docker images. Do we have active maintainers there that you feel like people other people are interested in it. Well I know that there is recently some discussion that the Sentos images are the most secure they have the, they have zero CV ease when scanned in terms of the image. So I don't know if that's going to cause more people to start using them or not. I'm not sure, but it's possible that that that more people might use them, since they have less security vulnerability supposedly. All right, so I just haven't seen indications of a maintainer you were a me who cares about and watches over them but that's good to know. All right. Okay. Just just so discuss within the context of the PR. Great. Excellent. Anything else on adopt then. No, not right now I think that's all. I had a flag on a prepare a blog post. I think I'm just going to drop this the window of opportunity for me has passed. Or use your feeling that we should do this and use it as the way chance to announce the deprecation of of install dash plugins.sh. I mean the other thing we could do is create a shim it like replace the contents of install plugins.sh and shim it to the plugin installation manager, and then people will be using plug installation manager just without knowing it. There are some differences between arguments between install plugins.sh and plug in the CLI tool, but I think it could be used as a there could be a shim implemented instead of doing any sort of upkeep on that script and just replace it with a shim just just one option. I like that I think that's very attractive. Instead of removing it, we leave it, but it becomes a simple call to a call directly to the plugin installation manager, bring people into the new world into the new tool without requiring that they change their scripts. So that's just one option. I don't know if it's we have a separate script right now that is used as a wrapper around the tool it just it just passes the options directly though so it's it's slightly different so this would be I don't and I don't know if we want to put the effort into doing anything with install plugins.sh instead of just saying hey move over so those are just some options I think if we do decide to completely remove install plugins.sh at some point I think I don't know if it necessitates a full blog post maybe a post on the mailing list that developers mailing list or the user's mailing list. I don't know if it needs a blog post. Okay. Good. So that one, I'm going to leave it sitting there and we may not need a blog post. Using a shim with that, is that likely to introduce any differences? Could sort of downstream users of that? Oh sorry. There are some differences in behavior between the CLI tool and the script but I think that some effort has been put in to reduce that. So we'd have to see if it was and that's why I was saying do we really want to put the effort in there to have some sort of translation layer or implement a translation layer and test it and stuff like that so that's kind of the trade off I would think. Yeah and isn't the trade for me there seems like we're choosing if we deleted entirely there will be some portion of the users who the first they see of it is when they attempt to do a Docker build and it falls over dead. Whereas if we do a replacement with known incompatibilities, they won't realize that they won't necessarily even detect that the change unless they were using one of the incompatible setups. Okay, great. Okay, so that's a that's a decide if a contributor wants to his welcome to work on that. And decide then right you me whoever if if you feel like it'd be valuable to assume that's a good idea. Yeah. Anything else on that plugin installation manager and install plugins.sh topic. No. Okay, great. One topic on Jim submitting refinements for parallelization. Alex, I'm not aware I haven't been following his work there anything you want to report there or Gareth anything you've got. I started looking at the PR I need to the it looks pretty good. I'll say that the the only question I have is how it fits in with the Jenkins file because that's kind of important how we actually invoke it so I need that's kind of the part that Jim says he doesn't have a lot of experience with and so I need that's kind of what he wanted me to look at so I that's kind of what I'm going to be looking at to see, you know, is it easy. It's going to be easy to call from the Jenkins file in a parallel way and stuff like that so that I have started looking at it I just haven't had a ton of time to dig deep into it. All right, super. This because there's the other PR to introduce the declarative pipeline as well. I was concerned about the declarative just because we are introducing more of this parallel stuff that may need to be scripted. But we can definitely look at that too. There is the capability of doing parallel and declarative so we just have to figure out if it's going to work with what we need to do. And the parallelism in gyms for multi arch must be a multi platform parallel right it's got to do parallel arm 64 parallel z series mainframe parallel power DC and parallel AMD 64 or Intel. Right. And I have a PR that adds those in the multi arc stuff. But it would need to be reworked. Once Jim stuff is integrated. Okay, got it. And we need to have a stable power PC agent. Oh is it's it's it's not it's not been holding stable. I haven't I haven't looked at it see if it's back up. It is definitely online now on Seattle Jenkins that I open. It's it's been been running quite well. Okay. Does it still have issues when you know I just like a kernel upgrade. That's a good question I don't know that I've done a kernel upgrades instead. Okay, that's that was what I was wondering is, is, does the kernel upgrade somehow cause problems every time so that was that was just a. That was what I meant by stability. Right and that one I, I don't know that I've done an operating system upgrade on it recently. So it's good question. All right, simplified labeling of doctor images I propose we put that one at the bottom. Since Jim's not here, we'll let him talk to it when he gets here if he's here and if he doesn't make it we'll leave it off the topics. Next topic was Docker images for LTSC 2019 Gareth you want to give us an overview there. No update really on that. We were holding off merging the PR until after the LTS had gone out, which so I believe we can merge that one now, which should give us the witness of co LTSC 2019 Jenkins image. That's awesome. We base that with the latest plugin manager update this morning. So it should be consistent. I think it's 220 it's using across all of the images now. Okay, and, and I saw that Oleg has, or that it has two dot three dot zero has released but we would do that consistently along all brand all all platforms right now two dot two dot zero is the the one we're delivering at the mall. Yes. Okay, great. Alex, any objections from you on that as a reasonable plan and we look at the PR I'm sure and finalize it ready to go. I think we're good. I'll actually I'll merge it because I'm happy with it. When the when it's it passes first thing in the morning for me. And then if I run it in the afternoon, we get quite a lot of failures on that belt. And we think that is down to Docker hub rate limiting. Interesting. Okay. Oh, because it can't pull or pull images. Yeah, it doesn't give you a rate limiting error, but we think that we're doing the Docker login too late in the process. The so the, it should be any Docker command though should be using the Docker config environment variable which should have the credentials. That was my understanding when I read through the information you shouldn't have to explicitly Docker login if you have the Docker config setup because it should use those credentials automatically. But maybe we need to look at that in more in depth. And that's how we do it on ci.jink instead of as we have a with Docker credentials or with credentials and it provides a zip file that has the the Docker config information the Jason file that has a username and password and so forth or API token or whatever is needed. So my understanding of the Docker config environment variables it should do any sort of authentic authentication needed. So it may just be that we are plan is not. We're not on the sponsor plan yet. Oh, but Olivier said that we definitely were. Oh, did he, I thought he had said that we may need to change the creds. Oh, I had missed that. But maybe maybe I can go back and look at the mailing list but I thought that was what he had said. Interesting. Okay, so that that's certainly a, a, a, yeah, an interesting topic that I wasn't aware of. Okay, so. Doctor commands should already be should be already using the credits that we have because we've got them. Yeah, Gareth you're you're absolutely seeing failures and and when we merge this PR we may likely see another failure on. Well, we will see a failure on the master branch anyway because that's that hasn't been working for a while. But so this morning when I rebased the PR. It went through all the bills passed. So I seem to get more success. I haven't, I haven't put this data into a spreadsheet or anything, but anecdotally, I feel like I get more success in the morning, my time rather than in the afternoon. So I'm European time zone. So I wondered whether that was because changes come through a bit faster in the afternoon and yet we start hitting a rate limit or something like that. So, and what sorts of steps should we be taking there then so there's this, this Docker login PR Gareth that you close now. Is there something we should do instead or Alex do you have a recommendation of what we should do instead how should we approach this. Well, the failures are happening during the build right. So, so that wouldn't be the Docker login stuff I don't think, because that the Docker build. There's there's never a call to Docker login the make files don't call that ever. Right they just call Docker build dash T whatever. So if it's failing during that time then it's not it has nothing to do with a Docker login call. I thought that Docker build would do a poll, if it didn't have a portion of the image, and that that that poll could be rate limited or or blocked. Right but what I'm saying is that the Docker login PR that Garrett had does not affect that right because that was that was part of the publishing to like the experimental. That wasn't part of the build process. What we need to do is we need to determine and see if there's a way we can figure out if we're being rate limited via the Docker API or something similar, and put some debug statements in to figure out if that's what we're actually hitting. If that's what we're actually hitting then we can determine how we can solve that problem but we need to figure out why if we are actually getting rate limited or if it's something where maybe the Docker cleanup is not being run successfully all the time and so are the images or the disks are getting filled up. So we need to figure out why it's it's failing at that point. I think Gareth's information about it being related to rate limiting would be great to investigate and to kind of zero in on that and determine if that's the actual issue we're running into. One of Jim's PRs had, he's moved some of the some of these sort of Docker login commands into like a common functions area. So it may be worth pulling some of that into a smaller PR so that we can start to use the same bits with some debugging in scripts. I know we went through kind of an effort previously when Publish Experimental was failing with trying to add Docker logins and things like that and it actually caused problems because we had the Docker config, because it overwrote the Docker config JSON file, which would make sense yeah. So that's kind of the reason that we haven't had the Docker login in process because the understanding was that Docker config should take care of that. So, but I think the rate limiting stuff would be good to look at and make sure we have the right credentials that are being used so that it's a non rate limited the sponsored plan or whatever the Olivier would know. That's something I can invest I can investigate it how if we can pull that certainly like where we are with the rate limiting backup. And you should be a trusted account now on that repository so you should be able to like make changes to the Jenkins file and a PR for debug purposes and stuff like that so. Also, Gareth is a maintainer now on the not on the Docker Hub side but he is on the he's been added to the the GitHub Docker packaging, I think it's Docker packaging group. Yeah, I can't merge, but I don't have no choice but I, I have noticed that if I change the Jenkins file, I think I'm down as a contributor or something. So, so, I can change the Jenkins file and it does pick up my bills, but I don't have no choice. Got it. Okay, so, so you so Jenkins file changes are processed. It's weird you should be. I'll look into that because Mark you're able to merge aren't you. Yes, I definitely am. Yeah, let me double check just to hear if you're in the same group as Mark on the team Docker packaging. So we'll have to look into that more because that's. Yeah, we should figure that out. So here's here's what I definitely have merged permission right I've got I've got the books. Let's pick one that's not got conflicts. Let's talk about this one. If I were going to merge this I solemnly promise I will not merge it but let's look to it also has conflicts. Okay. The Java ops one 1025. Oh yeah, that's a good man. Right. I think I don't think that one has any conflicts. And there yes so I definitely have merged permission okay but the Gareth you confirm you do not have merged permission. I do now. Okay, good. All right, question answered. Excellent. I'm pretty certain I didn't this morning. So that's so that's true of the GitHub side. Mark he does have permissions there, the Docker hub side would be something we would need to look at more if he needs permissions there. And I haven't had a sense Gareth that that you need that side as much right the work thus far has been in the Jenkins CI slash Docker GitHub repository hasn't it. Yeah, so what would be the circumstances where I would need a permission that just be when adding a new repository or something adding a new repository, deleting tags. Do you do you have access on the Jenkins for eval I noticed there's some of the open JDK images published there. Are you publishing those or is it's one of the one of the build scripts is publishing those and Olivier created the repositories for me. Yeah so. So go to repositories mark repositories okay. Sorry, up at the top. Oh, up here. Okay. And then there should be a drop down this, you can change to the Jenkins. Jenkins or Jenkins for eval. It depends which one would he want. Would you need access to these. Well I'm not sure that Gareth do you need access to either of these on the Docker hub side, probably not initially. But I'm happy that I'll request if it comes to a point that I require access but yeah. I'm happy. I'm happy not to have access. I know what you mean. You have you have full support to do that that comment. Absolutely. Okay. All right so so we've got Gareth the permission to merge that's good. Okay. So we need to discuss then on the windows Docker images and the Docker login topic help me out here so it's still, Gareth will, you'll do some investigation see if we can figure out why the failures are occurring on the on PRs and places we know that it's failing on master but this is a different class of failure, not the failure that's causing master branch break. Okay. Next topic Docker image support idea official versus unofficial Alex. Yeah, so Daniel Beck is very concerned about the number of images that we publish as an official list, because it takes a significant amount of time when during security releases and so forth that he has to wait until those images are ready. Before he can proceed with his process. So what I wanted to propose is that we adopt a similar method as to what the like adoptium project does where they have an official set of images and an unofficial set of images. And those would be that we determine which set of images we want to be official. And those would be the only ones that Daniel would have to worry about in terms of waiting for security release. And then the unofficial ones we could add as many images we want the unofficial list but that would be a possibly separate build process for those unofficial ones, such that Daniel didn't need to wait for that. Right now we have quite a number of images we have sentos we have Debian we have some Ubuntu we have open j9 versus hotspot. We have multiple windows variants, although we're we actually reduce that number. So, you know that and that can only grow right as we if people want specific other things like alpine or this or that right then it just it causes that build to increase in time. So this proposal would be maybe part of your jet mark where we have two tiers of support one is the official list and those are the ones that that have all the security fixes and stuff like that the other ones maybe have a later update for security issues and things like that. And you know the official ones maybe we reduce to just JDK 11 or something like that right because it is a controller image rather than like an agent or or anything like that, reducing that to maybe JDK 11 and then one Linux variant and one windows or something similar, but just reducing that set for the official list versus unofficial and unofficial we can have Amazon Linux and five different JDK's and all that stuff but that's not going to impact the the security process for Daniel. Would you see them going into different Docker orgs. That would be a question with adoptium does that. They have the the adopt open JDK right now anyway that's the official one that has a specific set of tags that are quote unquote official. And then they have sub repositories under that for like open JDK eight and open JDK 11 and other open JDK's and stuff like that, where they have lots of different tags available there. So that is, that might be the best way to go is to have a Jenkins official, or, or something like that where those official images are published. And, you know, that's the, the stuff that the security fixes will always be in the other stuff will have a delayed set of things. And we'd have to definitely pull in Daniel on this because if there are releases of Jenkins Docker images that don't have security fixes is that going to be a problem or not. But, but this is just kind of get the ball rolling on what we can, what we can possibly do he did have some initial interest in that when I put it on IRC, but we'd have to pull him in and make sure it was something that he he'd be okay with in the process. And I think that's a very healthy thing to include in the Jeff, because, because we've got, we've got a big challenge there right now it feels like we've got far more Docker images and we have maintainers to support the images. And so I think, as in good conscience, I feel like we need to reduce the number of officially supported images, just because we don't have enough bodies of us to maintain them adequately. And if people want, want to maintain an image, they should become a maintainer that should volunteer and come help us. Yeah, I agree. I need to drop off on to. Thank you. Super. I'll start that. I'll start that one. Great. Thank you. Cheers. All right. So, Alex, anything else with regard to the idea on image support that I capture your ideas in the notes here. Absolutely. Yeah, I think so. Okay, good. So, and I think, I think this is a good thing to feed into the, our review of a JEP and discussions about, okay, how should this thing work and why how do we get there it's, it's certainly going to be an interesting challenge as imagine I, my argument says, we should drop sentos because we should shift sentos to unofficial because it doesn't have an active maintainer. And I would expect an outcry from the community saying but that's my preferred image and they'd be right. Yeah. And I don't know the other, the other problem we have to is I don't think there's a way to it for us to really understand which images are how often images are being pulled. Meaning, or is there. Well, I, I, I think maybe image pulls not. I'm not aware of, of any no data on which tags are the hottest tags, right. Right. Yeah. But I think there is data. I don't think it's publicly visible, but Gareth has been granted access to the, the stats data that Andrew Bayer maintains. Okay. And inside that stats data, I think there may be things which will hint to us, not, not Docker specific, but just in general, hey, here are these operating system platforms and it really knows operating system. Okay. We're testing stats data to confirm which configurations are most interesting. Sorry, do you need to drop off. No, I don't, I, I'm good until whenever the meeting is done. I just had a text comment so. All right. All right. Okay, I think the image support topic is settled my topic on new Docker image maintainer I think we already proved we got that Gareth's got access. The next one was that the jink and the master branch has been broken for a number of months. Help me out Alex I don't remember what the break is right now is it is it something that is. It's just the publish experimental and I think the gyms PR is supposed to help with that. So we merged in a previous PR from Jim to do some parallelization stuff. And there was not a Jenkins file update associated with it. And so the wrong thing is being called right now in the Jenkins file I can look at that you can assign an action item to me to look at that. And I'll try and figure it out. I mean I don't want to pile things on you in this exercise for me that that feels like we can we can ask others to help out with that if, if that would, if that would be okay for you. Sure. Yeah, if someone else is available I'm, I'm happy to let them take a look at it. I'm also happy to help so either way I'm good. All right, great. Alex or Gareth, or Mark, best to gate and submit the PR to fix the master branch. And we may need to involve Jim just to, if there's any question about what we actually need to be calling. Okay. And now if, if we turned off publish experimental, would that be an acceptable condition there not really right because we're, I mean, but this is not publishing. Yeah, I mean, we could disable it for now, and then just have a PR that hasn't enabled that that we do the debug in that would make it pass you're not going to have images actually published to Jenkins for eval. They're not being published right now anyway, so it's kind of a six dozen one half times the other six of one half dozen of the other. Right. Okay. If we want the good feelings of it being green or blue, whichever it is on Seattle Jenkins that out. Right. Then, you know, just disable it for now is fine. I don't know how many people are actually using those images. And at some point we would want to make it so that the multiarch images are not experimental anyway. Right and that's that's that's a different way to make the multiarch images non experimental that means trusted will need to also have access to the to the to the multiple platforms right and right now I'm pretty sure it does not. So right so that that would be an extra step for later. Okay. All right. Last topic I had then was on simplified labeling of Docker images, Jim's not here. Are there other is anything you want to share there on that. Alex. No, I'm. That's kind of the part of the PR to that we talked about earlier so it's just something I need to review. Okay, all right. And I think I understood from the mailing list that are from a comment that Jim's proposal is reasonably well aligned with the labeling you're already using on windows for the Jenkins freeval windows images. That's correct yeah. I did have one other topic. Oh, go ahead. We should probably do a bug scrub on the various Docker repositories like there are 96 issues on Jenkins CI Docker. Some of them are from like 2017. That's the oldest so some of them are related directly to install plugins that SH so we can, I think close those as that script is deprecated. It's not a top manager. So we just need to do a bug scrub of all the Docker repos I think. And maybe that's worth another note of pull request scrub on the on the Docker repositories as well because there are a number of those which are older and it may no longer be viable. Correct. And that's not a top priority it's just kind of something we should probably look at. Yeah, well, one of the things that's helped me in the past is, should we show some graphs once a month from the Linux foundations tracking system. They've got a system that watches our repositories for us. We use it in the doc SIG, and it, it produces some interesting data to show hey how are we doing or not doing to investigate. The next. That's a good idea. It's just, it doesn't give much hope because truthfully in the doc SIG, most of the time, it's, oh, gee, shame on me that's not we're not reviewing things as often as we should, but at least it tells us the truth. Right. Just what we need is a JavaScript debugger. Okay. Anything else. No, that's that's all for me. Alex, recording will be posted separately recording is now off.