 Welcome, everyone. This is the Jenkins platform special interest group meeting. It's the 8th of October 2020. Remind you that we follow the Jenkins code of conduct. We try to be good to each other. I'm going to start sharing my screen so that we can take a look at the agenda and we'll work from the agenda. Okay, so proposed agenda items, open action items. We adopt open JDK 8 for Docker on Debian. Retirement schedule for Debbie and stretch so that's Debbie and nine in our doctor images. Report on the power PC 64 little Indian agent access effort. Windows server 1909 agent support status report on the Docker platform support proposal action item that I have. A status report on Oracle cloud any other items that we should add to the agenda. Okay, and let's go ahead. So action items. I've got the action item to open a Jenkins enhancement proposal for doctor operating system support and platform selection rules. And we've got two very good examples right now of places where we, we need to actively retire a doctor image. We're going to use those as examples as a way to test the rules that I'm proposing the two of you will be copied, I will, I suspect that I will first start with this thing as a, as a draft in Google Docs so that we can comment much more freely before I submit a pull request to the Jenkins repository. And that is, I've got an outline later that I'll outline for you but we've got to consider Debbie and nine has already ended its typical support life cycle is now into long term support. And the Debian project, when they release bullseye Debbie what is Debbie and 11 will stop supporting Debbie and nine. So and that'll be in sometime in 2021. And that operating system vendor support is, is ending on this thing. The other example is that operating vendor supports ended long ago on Alpine 3.9 and Oleg Nanachev did a quick upgrade for us to 3.12, but really our process and our rules for support should have caused us to do that automatically anyway. So, so that's, that's those two things I'll use as test cases. Now, test case additions as 390x and arm 64 will be on the, the, the other examples right so as 390x 64 as example additions. And because we've now got the, the graviton instances available for us from CI Jenkins.io for arm 64 and we've had the S 390x available for quite a while. Alex had the accident and investigate options for sentos and adopt open JDK. Now Jim you've been a good voice for us to adopt open JDK anything you wanted to share there. Yeah, we do have sento as support it's in the unofficial images. One of the major things that I've been working with the dots, since I started working with you guys over a year ago, or almost a year ago is pushing the unofficial images to official. So that, you know, it's a little more, I know you guys graciously started utilizing some of the unofficial images because it's really not that much difference. But we have, I worked with the testing team, I enabled this huge big pipeline to basically trigger all the unofficial images nightly and basically test them all. And that's what we kind of were missing on the adopt organization is a testing pipeline I made testing pipeline. Before if you wanted to move an unofficial image to official. It was this whole manual process that only basically one person had a control over, which was slow. So now hopefully we'll be seeing a bigger wave of unofficial images, moving to official images which includes sento as some of the devian images. A lot of the images I've requested had to go in line with what your images you guys support. Yeah, so are the, for instance, does does adopting adopt open JDK provide s 390 x official images now. Yes, yeah, adopt has s 390 power and arm for some time now. Excellent. Alright, so already provide s 390 x. So that that's good that gives me hope that I can put one of one of the noise sources in our platform plan is, we have some images of JDK and others based on adopt open JDK. And so this that sounds like your work with adopt will allow us to consider making everything based on adopt so that we can delegate there be worrying about operating system care to them. Excellent. Yeah. And that's one of the action items which I'll get to after you're done yours. I want to add. Great. All right. Let's focus on that sento s topic. No, it should be available. So, you guys should be good. Okay, next was Alex submitted a poll request with deprecation notice for install plugins. We have a preview already is now delivering delivered the the new plugin installation manager. And that script script, it's not a script that Java program resolves dependencies of plugins, it's in preview, and so it's available in in all our doctor images for core for Jenkins core and thus far I've pointed several people to it and said look, you've reported a problem. The solution is to use plug in installation manager, instead of complaining that install plugins that sh does not resolve dependencies. All right, I've still got the action having to prep a blog post on update center and plug in installation manager. Yes, that's still to do. Jim you said you had another action item to add here. One of the main items was in that PR I submitted a while back to get accepted about the parallelization and the, the rearranging of the CI build tools for the multi arch images. In that pipeline, we, you guys are using majority open JDK images. Now it seems like the shift and correct me if I'm wrong is now more majority of adopt open JDK images. And inside that script. I think we're still calling open JDK so I need to go through and prune the open JDK references and make slight modifications at the, the, the core of the scripts there just needs to point to the different base images and stuff like that. It should be a minor tweak for that. You know, also I misspoke on as on CentOS support adopt open JDK does provide pretty much all the architectures, but for any OS, but for CentOS, especially it's kind of pain them but CentOS themselves don't produce a S390 image. Yeah they produce pretty much everything underneath the sun except S390 and there's been rumblings. Now with you know the buying out a red hat and I know CentOS is slightly different than red hat. There's been rumblings of S390 image but right now with the currently, you know if you want to run CentOS based things on S390 use cleft OS CLEF OS, which is basically just a recapitalization of CentOS that runs an S390. Right. Okay. And now, is there a red hat enterprise Linux for S390. Yes. Yeah, red hat runs fine S390 in power. There is a UBI image, universal base image, which is what red hat pushes out. It's not called red hat OS or whatever it's not rel, but it's UBI which is based off of rel it just stripped out some of the more like proprietary stuff. And that works on pretty much all the architectures. So, so UBI is a good topic for conversation in the operating system support as a potential outright replacement for CentOS. UBI is allowed as an open to act as though it's an open source thing isn't it it's not actually open source it's fully commercial, but yeah you don't need you with UBI. You don't need a license to use UBI right. No, no, you don't need any license you don't need subscription manager you don't need anything like that. As long as I everything I read and everything I've done with UBI you don't need a subscription manager like you would need it they give you on a row or something like that. Right. So this is available for every architecture you guys support right now arm 64. I don't think it has arm 32 support, but I know it has power and s3 90 in the arm 64 I'm pretty sure. Super anything else on the paralyzation action item. Anything else that you wanted to say there. No, just need to clean it up a little bit, removing the open JDK references. I know you got it right. You got it right. Oh, okay. Any other action items that we need to flag here. Next topic then adopt open JDK eight for Docker on devion. So this was a pending pull request for a long time and is now merged. It was merged. But you might assume that your comments were handled and and addressed. Great. Okay. Yeah, I mean the PR was pretty much perfect. Alex just accidentally pulled from the wrong image once or twice, but he was able to fix us in the instant. Okay. All right. I don't. Is there anything else we need to note with regard to that. Particular action. I think we're done with it and we use it and benefit from it. So next is retirement plan and schedule for retiring deputy of nine. This is, we've got to find a way to notify people that Debbie and nine support is going away or it would be nice. I should say it differently. Alpine certainly didn't find any way to tell us 3.9 was no longer supported they just set it on their web pages. We could do the same. But there is there is also the option to say, hey, we're going to try to alert people that we're going to stop supporting this operating system because the vendor is going to stop supporting it. And yeah, here we go. So it's part of the part of the platform plan is includes research on other examples like Alpine, like the end, like about open JDK, because certainly they've got the same same challenge right Debbie and nine was supported by them and I remember correctly now they've they've switched everything to Buster and beyond. So any questions. Mark at one time, I know Alex, I think was Alex I was talking to glitter. Is there like a some sort of tool that scans Docker images looking for like vulnerabilities and stuff like that. Because, you know, the way you're mentioning was that the way we found out or the way you guys found out for Alpine being retired was, Oh, hey, you went on the website or someone went on the website and saw it, or someone rated issue. Is there not a way to be preemptive in terms of that where like you have some sort of scanning tool. I think I think I want to say was Alex that said there was a scanning tool he was looking into to scan Docker images for vulnerabilities. And imagine one of those vulnerabilities that would come up is the outdated OS, or, you know, some problematic like hey there's no support for this. And then we can get some sort of alerting to be private preemptive about it. And there certainly are there's Nick, and I think that was it. There, there, and I don't know how they pronounce that my Italian wants to say it differently than I think they pronounce the snake anchor. And there are, it's, it's an entire market space. And so there are many different competitors in that space. We just don't have any of them enabled so it's a good point and that's a that would be a nice addition to our CI processes to scan our doctor images for vulnerabilities from through one of these one of these tools. Exactly. Gareth, that's one where I thought, had you been using image scanning in your previous work with Jenkins X, or was that not not a. Yeah, we used, we use white sauce. White sauce was is the sort of one of the commercial ones that cloud bees had a license for that we could use. But you could, I think, and so I could probably do the same. Thank you. Yeah, and I'm confident there are more in that space and more coming because of the increasingly heavy use of containers and the worries that hey my container is out of date and I don't know anything else on retirement. Slightly different talk about retirement. Is there any movement on changing LTS from eight to 11. Oh, good question. Let's put that as a separate topic here and we'll discuss it so. So I know like was trying to push that a while ago. Well, so, so what we've got is, let's talk to what we actually have. So we support currently we support both JDK eight and JDK 11. Yeah. And, and I've seen no move. And I don't expect any move in the near term to drop JDK eight support. Either. I was thinking more of, I think when you do Jenkins when you do Docker pull Jenkins LTS. I think it's maybe I'm wrong. But I think it pulls down, you know, obviously the LTS release of Jenkins, but I think it's bundled with JDK eight. Yeah, so I think your question is, should we switch the default. I'm going to cough. That's a good question. And I haven't seen any discussion about switching the default image from JDK eight to JDK 11 but I think it's a worthwhile topic and it's that again fits with part of the platform support plan. It's a good thing for us to get into those conversations. I know that we shift should we shift people, the day will come when JDK eight support will end from from various people it may be many years away but it will end. And we'll need to transition people should we already make the default need to use JDK 11 with its longer lifetime include in the platform plan discussions. Excellent, excellent question. Anything else on on that topic. Yeah, I guess more one slightly other thing is that I would actually help with S3 19 terms of me you saw it running with hotspot versus open Jane nine with the release of JDK 11 we I think IBM was able to get the jet included in 11 hotspot by default. So if you guys did release multi arched images which I think you guys are probably go with hotspot as defaults, because I don't really see you guys using open Jane nine everywhere except that one open China image. But it would it would cause S3 19 to run normally, not super slow. Right, right. Yeah. Yesterday X does not like you know I you could still use a as the default you would just need to make sure if you're building for S3 90 to instead of using hotspot as the base image make sure you do the open Jane nine variant. But that's it but with 11 you wouldn't need to care you just build all hotspot. Right, right. Good point the. So, if we now in terms of your need if there were no S3 90 X JDK image would that be okay. If we just did S3 90 support with JDK 11 with that. Yeah, I think that I think that would, I think that would. It's nice to have have a but I don't think for the workloads most people are running is a requirement. I think I think just some sort of Jenkins on Z is a requirement. The reason I ask is open J9 is a is an open topic for me as part of the platform support where I'm thinking, one way to reduce our our platform matrix, the breadth and breadth of the thing is, should we should we not do open J9. But if we said who we're not doing open J9 that means we would say we can't do S3 90 on JDK eight and that that that could be a blocker you say no we've got to have S3 90 at least open J9 for that in order to do JDK eight on S3 90 X. No, I don't I don't think it's a blocker. As long as there is S3 90 support I'm sure someone out of the woodworks will come out and yell it yell it's like hey why don't we have Java eight support. Right. Okay, good to know so S3 90 X. Yeah, that's definitely that's definitely something I'm interested in that your proposal in terms of. Well, and this is this is a chance for you to survey your customers your target audience. It's perfectly okay if you've come back while we're reviewing that document and say, no we've got to have JDK eight and therefore that means we must have at least one instance of open J9 being supported. Good. Okay. I mean, yeah, okay. Yeah, I'll serve you that. And everything else on that JDK eight versus JDK 11 is the default image. Okay, our PC 64 little Indian so the story here is marks access to PPC 64 le agents has worked great. Unreliable runs, etc. Access from CI Jenkins.io is unreliable, and we don't know why. And, and I am just completely perplexed by it. There's a networking issue from AWS that my home, my home where my machines are don't have. We need to investigate further and Alex has, has started the investment started and the experiment to see if he can understand what's going on. My initial experiments failed. And it may be that maybe that there's some networking issue in the Azure data center where where we're hosting CI Jenkins.io. There may be some networking setup or we just don't understand what's going on and so it needs some investigation still. So, so, so I can trace it on my end with Rafael. Are you guys. You said you're running on Azure or AWS. We're so the CI Jenkins.io is on Azure. And I don't right now I don't have personally the time to do the investigation so I'm not sure there's any gain for you invoking Rafael. Right now but Alex, Alex may and might might appreciate any any insights that insights they have on how to diagnose where's the problem. Yeah, I'm not used to diagnosing cloud vendor to cloud vendor connection. Yeah, is it just connection dropouts the SH or what what's like the season biggest issue. Yeah, so what we see is the SSH connection is established. It runs. It, it continues to operate. The, for instance, the remoting jar does not seem to finish loading or finish copying from from the from from the controller to the agent. And it's not clear why we can see that there's there's in the log file there's clearly a connection. And that, and it says copying the remoting jar, but then nothing further now that isn't if I remember that is an SFD connection, or and that I don't, there is something about the transfer of remoting that jar that is using. I believe an additional connection. I, that's the limit of what I know about it. So it's the problem breaks down when you're trying to transfer this remoting jar to that server and something goes right. Okay. Yeah, in fact, I can even hear we'll just let's go live and I'll show you any because I suspect the agent is currently disconnected and we can see the log. And so that will that will helps us visualize it. Here. And the, the log is completely perplexing because it's clearly doing real SSH work on the remote side. And this is, hey, this is the correct machine type. Yes, we expected it to be a red hat Linux and it is. It's got a reasonable path, all those things. But then it hits this point and seems to progress no further. I don't, I don't understand why and haven't had the capacity to do the investigation but that's, that's the kind of thing that's happening. Is that is that link public accessible and now is that just for you. Yes, I think so. Yeah, so let me I'll just put it into the notes. Yeah, thank you. No, I appreciate it. Yeah, I'll get without on that then and try to figure out maybe it's some sort of stupid firewall or something like that. Is it ever work or is it works. It has worked on occasion, and Alex saw it work, and then it started failing again so it's, it's intermittent, which is of course the worst of all possible. Yes, exactly. It still works great for me. My agent is still perfectly well connected. And let's see if I can show that the agent from my machine is there and runs great so here we are it's called this one. It's a Sentos machine, it's log looks perfectly good. And it completed the copy of the, the, the remoting jar and did a whole bunch of other checks. So, so it definitely it's not something where I can say oh it's clearly an issue on the side. Yeah. Okay, it's I don't know where the issue is. I'll look in with Alex. Thank you for showing me. Yeah, great. Thank you. All right, next topic, Windows Server 1909 agent support. So, Gareth, this is the. Thank you very much. Deep and sincere thanks to Gareth. What we've got is November of 2020 will be the last Windows Server 1809 security readings. Yes, the Docker image, our Docker images are based on 1809 Docker agent images are based on 1809 Windows Docker agent Docker agent images 1809. So we're at the end of life for that for that operating system version, we need to get running on or building versions based on 1909. And Alex and Gareth are going to be investigating Alex had done some initial investigating initial prototyping to see if we could already build 1909 and as far as I can tell, he's confirmed that we need to change the base windows operating system we're using before we build the windows 1909 image. Gareth, anything you need to report there or you want to report about what you observe. No, pretty much that I know Alex has put PR up to build a base image of, I think it's 2004 to see if we can then build the slightly older Docker images on top of that. Because I don't think there are 1909 base images available on AWS and Azure. That's, that's for me is really a strange one because they're, I did see a Windows Server 1909 base image on GCP. And I don't understand why of all people Microsoft and their Azure hosting would not have that image that's really weird. So yes, anything else on that topic. I don't think so not yet. I just sort of started looking at it really so. Super. And thank you for doing it thanks in advance for doing that. That's a that's a sort of a dark corner for me where Alex really knows very well how the windows and agent windows images are built and I feel like I'm just barely learning. Okay, more details on on the Docker platform actually maybe what we do is let's take Oracle cloud as a topic here. And I'll give a free status report then we'll use some more time on the platform support proposal. So, we've been approached as a project or a cloud has approached the Jenkins project. They'd like to. They'd want to be more involved. And they've asked for a dish we've met with them a couple of times. Discussed. And they asked that, hey, would you sign a confidential disclosure agreement so we can tell you about things that are coming. And we can then work together on those before we release them. And so I've signed a signed the CDA with them. And we'll see where it evolves. Some of the things that are interesting for the Jenkins project there is bandwidth is cheaper in and out of the Oracle cloud, then it is on Azure, where we're paying a. That's interesting to reduce our Azure bill. And if they can if they're willing to donate so that the cost goes to zero that gets even more interesting. And other hardware related things they apparently have a virtual machine style, more of a virtual machine centered focus. And that could help us with getting extra compute capacity if we need it. Now I don't have anything more than that other than to say that yes we'll continue and just like GMR gratitude to IBM for being a cloud provider for for equipment we look to other cloud providers to help. Thanks very much. Any questions on Oracle cloud. So here's my topic then Docker platform support. So here's, here's my rough idea on how I want to frame the initial document, starting with what we have today in terms of a tabular survey of the image types we support. So core and SSH agent, the JNLP agent of various other images that the project supports, which JDK sources we support. And this should go here, which operating systems we support which JDK sources we support. And then, Jim, you highlighted one JDK version, if you will, and that's adopt. Let's cover. No, that's what it's hot spot versus is the JVM, I think JVM not JDK. There you go. Very good. Thank you. That's not spot J9. Very good. And those factor into Okay, what have we got what Oh, and I guess the other is official or eval. Okay, because we've got the Jenkins for eval project that is publishing images, any other things in the, how do we highlight what we have today. Did I miss something else looks good. Okay, then I thought the next step is talk about what we want. Oh, no, I had one more which was issues with what we have. The reason to put that there is highlight Debbie and nine obsolescence Alpine 3.9 and the support. The Debian open JDK packaging errors. That was several years ago that happened but it's a good example for us to highlight why we would prefer to rely on adopt open JDK rather than open JDK. Also, I think arm support I don't think open JDK. I think they're working on it but I don't think they have on support currently. Oh, and Alpine support right great actually that's a good one arm and Alpine non support. Also, by open JDK. Are they producing, I don't think the, at least the last time I checked, I think there was a proposal from the official Docker image publisher, you know the group that publish the official images. I think they stopped supporting open JDK I don't think they're publishing new new builds besides x86. I think it's just, they're pushing out open JDK x86 new new images but s390 power arm, and all that stuff they officially stopped. So you won't see any updates for the other ones. Good, good, good examples of issues to highlight saying hey look, this is what I like that because for me what we show is, hey, an image or a series of tables which illustrate just how complicated the things are that we currently have because of all the different mixes. And, alright, here's the, the issues that come from that complication, and then switch into a. Okay, here's what we want. We know we've got to do Intel. We want to add s390 as a form as an official one and power PC and arm. And then. So that's how this is processors. So I want to mention up what we have today or the other variants you do is slim. I know there's not only does adopt have slim images which remove some of the unneeded files from the JDK documentation stuff like that, but also you have Debbie and slam you have, you know this slim versions of your, you know the systems. Right, and in fact, Jenkins has that Debbie, what we call Debbie and slim but slim is not an operating system for Debbie and it's just a variant. Yeah. Buster slim and buster. And again, a place where we're being unclear and this, this platform support proposal can highlight. Hey, here are the things where we're absolutely unclear on what what this thing is because it's not just slim. It's Debbie and Buster, it happens to be the lightweight, the small version of it. So you guys not, I think I asked this in the glitch that it looks like adopt and pretty much everyone else is starting to drop support for AMD, sorry, arm 32 bit processors like arm V six or something like that which one V whatever you guys aren't planning to do arm 32 whatever its name. Okay, no, as far as well. So one of it's a good, good question and one of my arguments why I think we should not consider adding 32 bit arm is, in order to support a processor, we need to have access to that processor from CI Jenkins.io and from trust and CI. And, and you can't get cloud hosted 32 bit arm. Yeah, at least not as far as I think. Yeah, I think everyone's moving away from it. arm 64 bit is obviously the successor. Right and so, so my technique at home on my raspberry pies has been I just lie and label them as arm 64 even though they're 32 bit. All right, so Debbie incentives. And then Alpine question mark and there are certainly people who like it so. So, so the, the idea then in the document talk about what we want that those three things then lead lead to what the document will propose as guiding principles and guiding principles one is hardware or the cloud hosted equipment must be available to test and build the image. So that that's so or let's say this way, host of the equipment must be available to the Jenkins project, the test and build the image. I think we need people willing to contribute to maintaining people willing to be maintainers, maybe a new role where we have to say hey we need a maintainer for for the Docker repository. And then the other guiding principles I actually vendor should still support security fixes. So the example there is Windows Server 1809 last security fix means we would drop it from our list because the vendors no longer delivering or Debbie and nine, when it exits long term support would no longer be allowed. There are other suggestions for guiding principles there. And then the ideas the document describes what rules will use to implement those principles and lots of writing still to do and I intend to do that today on my day off it's going to be nice. Any other topics we need to discuss here in our meeting. I think that that concludes for today so Jim will work the action items. And I think you in advance for your work on Windows Server 1909 will call that done. Thanks everybody. See you there guys.