 Thank you everyone for being here. This is the Platt Jenkins platform special interest group. Let's look at the agenda We've got we'll review our open-action items We've got a topic on renaming agent Docker images. Oh leg. Do you want that to stay on the agenda for today? There is no of this for me except I'm going to implement it and it's coming soon. Okay, great All right, then power power 64 power PC 64 Agent access results and I can show a quick demo Then z390x agent terms and conditions. This is the result from the governance meeting yesterday or yes It was yesterday Google Summer of Code 2020 project ideas. We'll do a quick review there and Then we've got rework going on in the Docker slave Docker agent Jnlp and SSH projects that we can talk briefly Jim, I assume we want to give you time to talk about Docker image publishing improvement up efforts the update there Yep, and on get LFS. Do we need a do we need a time slot for that one in this meeting Jim? Very briefly, okay, great. All right anything else that we need to add to the agenda Okay, great All right, so then let's go through the action items. Yes I still have the open on action item to get the Docker JEP open for platform selection rules I think we're getting platform selection without having an implicit in an explicit set of rules and that's That's good, but it'd be better to have the rules stated Likewise we've got an open action item Oleg for a Windows support policy I assume you're okay Oleg with how things are evolving there that It's it's somewhat ad hoc, but we're we're proceeding Yeah, we still need to document it Right now we're doing major work of Windows service record and we started hitting great issues here and there especially Exotic platforms where you can enter on full load night exotic platforms like We're into server nano so not that exotic We'll definitely need to do something about it No real updates Okay You said Windows Server Nano great All right, then the Windows installer code signing with Olivia Bernin and Alex Anything that you need want to report there No, it's still in a holding pattern until PDF didn't figure out what's going on right Okay, and then Jim on the youth start of the infrastructure discussions for agents on power PC64 Ellie and S3 90 X should we take this action item out as progress is being made or what would you like? We can take that out. I think I think that's kind of going forward. So Great. Okay. Yeah Then we had review and discuss the buck Docker build rework PR and I have not I have not done a review of that yet Jim, I believe you've continued making progress on that and awaiting a rival of agents Yeah, so that's still on the to-do list Okay, great Okay, so Oleg you had said on the renaming agent Docker images It's coming And that's that's sufficient Then the next topic the power PC64 Ellie agent. So I've got here. I can show oops Let's see my Jenkins. Here's my Jenkins server with a Showing the execution history using a power PC64 Ellie agent as provided by Jim and His team member in Brazil So this job is the get client plug-in it compiled just fine on power PC64 Ellie it does it did show a test failure and the test failure was due to a The locale setting that I had on the machine So I have to remember for that particular test to set a locale that supports you TF64 or that you support you TF8 and I I hadn't so I've adjusted that now I'm Other jobs are my test jobs that are running on on this particular environment the test for various bug conditions Some of them are known to fail and they fail. Expectedly Here's the platform label or plug-in running successfully So it looked positive now. I've got an issue right now though that the agent is currently disconnected And I don't understand why it's disconnected because I am still able to SSH into the machine So this is me doing SSH to IBM cloud Jenkins agent following the instructions that We're provided and if we look at proc CPU info you'll see it's a power 9 architecture and There are eight cores That's a little weird You know why you're getting disconnected that is especially when you can SSH store in Wonder it might be a firewall rule that might have got added or something. I Yeah, I don't know given that it's using a jump post so I'm I'm I'm accessing it through a Through a two-step process, right? I jump from SSH That then SSH is to it again, and that's not a common use for a Jenkins agent So yeah, it'd be that the agent is having some issue. It could be I don't know what it is but just to note that Progress has been made and the connection technique May not be as reliable as I wanted Here's what it says right now. It just says hey unexpected stream termination As I'm doing this but this is this is the same command that was running for several hours yesterday and allowed me to get those Those job those results collected One thing I was talking to Raphael about my co-worker in Brazil We spun up the power machine for you He said that Excuse me For those who weren't on the email. This is like a temporary setup right now for the power machine And when we get to when I think there's gonna be a term sheet coming I know I said I don't think there was one, but I think they're only just modifying the the Z13 as sorry s390 Term sheet and just putting power up to the top They said when they move it I asked them about giving a public IP So you don't have to go through that jump post and he said yeah, we could do that or we port forward stuff This is the JG agent run on like a some sort of port or something or is it just SSH here? So it's it's both So well it it can use it the easiest way is for it to receive an SSH connection on Arbitrary port for instance I have I have you could be on port 22 or I've used it on other ports One of my machines that's on the public Internet intentionally is not on port 22 to reduce the script kitties who yes regular probe port 22 Okay That yeah, that's something we can talk to Raphael about Going forwards when we get a permanent home to make sure we get port forward or a public IP to hope To alleviate some of this disconnecting Yeah, so so now it would be nice I think if I could I may need Raphael's help to check the logs on the jump post Just to see if that jump post is saying something about why things are being rejected. I can also turn on Verbose here. In fact, why don't we just do that now? That's a good excuse to let us so one of the nice things about how agents start is I can just do this save it and Click launch the agent and it will show us what it's trying to do Love SSH the Password based authentication. Okay, that doesn't help much So it looks like it's blocking waiting for password prompt because the key Well, so I'll need I don't need to do analysis in front of everybody. Yeah, okay, but it's it's it was running It is at the moment It looks like I need to give it an additional configuration that it should Hey, that's the expected RSA key. Hmm. I'll need to do some more investigating so Mark to investigate Identify What is failing because we also had Ola Olivier I reported that he was able to connect as well Yeah, I saw that, you know, that's great news that he's able to connect Any other questions on the on the power PC topic case and I assume next step And let's see Olivia reported next step is to get it into the Internet integrated into the step to integrate into the puppet infrastructure For I assume CI dot Jenkins that IO and right now. It's not yet. I haven't seen any discussion from him yet on No discussion yet on Inclusion into trusted CI I Assume that's that'll be future work. Okay Then on the 390 so there the the document has been received so the terms and conditions and the governance board Gave their support on yesterday's meeting Sports Olivier signing it On behalf of so signing it I Wouldn't say I'm not trying to say on behalf of because on behalf of would might imply a legal entity and We're still waiting for CDF to get through the whole legal entity thing But the governance board agreed and I think Olivier is willing to do so and he's the info offer officer so He's the best choice of person to sign it any questions on the 390 terms and conditions Now Jim once that signed. What's the what's the typical lead time? Do you have any guess? I? Guess within the week So if you guys get it done this week, it'd be probably mid next week to get that machine Also depends if I could go ahead and ping them multiple times to get that up and running so It should be common practice in terms of getting the machine up It would just be getting the public IP that might take a second Excellent really good super Anything else on 390 and let's go on to platform of the Google summer of code 2020 project ideas from the platform sick So here we've had Let's take a look so jet the Jenkins project has been accepted into Google summer of code and And congratulations to Oleg and others who have done such amazing work to make that such a success and As project ideas So we've got the EDA for coverage adapters that fits well under platform sig We've got Several others if I recall correctly Yeah, plug in installation manager. Yeah, there it was. Thank you I guess the service custom distributions also under the platform see Yes, custom Jenkins distributions now in the platform seek, right And the Jenkins window services in the bottom is also platform seek, right Now Oleg are there things that you're seeing that that we should review here in the meeting on those project ideas Feels like there's lots of conversation around custom Jenkins distribution build service Really encouraging Yeah, so I discussed having to discussion on the platform seek use rick. So it's in the developer market rick is unable to participate in the current Time frame so I proposed to select the new time slot. There was a doodle meeting Doodle for that, but the meeting time hasn't been announced yet Okay, so so I will pink rick It might happen tomorrow or might happen later But yeah, I will host it And yeah, I will get the video posted for those who cannot participate Great so that and that was for For gsock And rick's rick specific project. Was there one specific one that he was custom Jenkins build service? Yes Okay So for the projects I'm also happy to host Discussions if somebody wants to cover them The window service rapid rate was discussed multiple times before the seek meetings Basically nothing changed significantly since the last year there Except well window service rapid changed, but the project idea is still there um Not too sure if I'm asking it in the right as sick, but uh, does the docker Uh proposal come under the platform seek? Technically yes I mean Could could we get a call for that because I'm not sure justin or a falco is Pretty active on the guitar chart. I mean Oleg is that possible? Uh, yeah, it's possible, but yeah, you could just self organize So the recommendation is to use my dentist that is a threat about this project idea Okay Ask Andrey and justin to organize such meeting Okay, they're open to that. Okay. I'll be happy to host the session Yes Well, I assume andre and justin are probably european time zone or us time zone Very easy in states. Justin is in europe. Okay, that's if I recall correct Wrong, excuse my fat fingers. There we go. Ah I would like the hyperlink not Yeah All right excellent and slightly that was am I pronouncing your name correctly? Yeah. Yeah mark your absolutely spot on All right, so and and that was one that you're interested in and Want further schedule the session to meet with them good I was also wondering if if at all I could get more involved with maybe, um, any of the docker, uh I mean issues or so because apart from the new Newbie friendly ones. Are there any specific issues that the team is currently working on? Are there any issues that I could tackle There is this docker build rework pull request right here But certainly we would love to have additional review on it and additional additional eyes looking at it So it's a great excuse for you to take a look at how does the docker build process works? How does the current process work and what is jim proposing as the changes to make it work differently? and those those proposed changes are significant and and Worth invest worth evaluation So that's I'm not sure I would call this a newbie friendly This is major and significant rework, but it's a good place to learn how things work today. All right. Thanks a lot Anything else on google summer of code? nothing specific from me, so Usual proposal if anyone has any other ideas in mind Then please let us know And you will appreciate any contributions any project ideas Because it's always great to have more All right now and I I'm I guess I should report Is I'm somewhat behind schedule is let's just call it what it is. I am behind schedule reviewing The newbie friendly pull requests But we've we we've made progress. They are in progress Happening and the students that had submitted some newbie friendly pull requests as part of their getting Familiar have made significant contributions to the Jenkins plug get plug-in and the Jenkins get client plug-in It has j unit four or j unit three tests in it that they're converting to j unit four And reworking in various interesting ways and using that to learn how to write tests And learning how to deal with my comments about how they write tests. So it's a real positive Okay Next item then was rework the docker slave And the docker slave j and lp slave and the ssh projects to support alpine And this is a an open pull request that hasn't seen a lot of attention in the last two weeks It was a it's a proposal to I think Do something similar in terms of its build structure? On the same kinds of ideas is what jim is proposing for the docker images themselves This though needs more investigation and more more visiting jim I wonder if this would be one where it might be good for us to have you take a look at it as well Just knowing what you know what you've learned On the on your rework on the docker image for s390 Yeah, is it is it is the pull request Changing from Open jdk to adopt I don't think it's is at this point It's just proposing to adopt to use newer open jdk images But I think that It's a good place for us to make you aware of of this Contributors work and him aware of your work or him or her aware of your work Yeah, I'd love to hop in and look at it Oh, like the the did you propose the polling for docker registries I just took look I looked at the proposal and that's actually pretty cool to Do some webhooks or to basically look at any updates of those base images and update ours Accordingly It wasn't me. I'm pretty familiar with this project idea If you have any questions But yeah, originally it was andrey who proposed it idea Great All right Yeah one comment about agent packaging What would be ideal to have is to have unified packaging scripts Which would work for all our master and agent images At some point I've taken scripts created by IBM And edited them to the remote in project Just a second. I will find a link and I kind of got the agent packaging talk also working for multiple platforms but at that point Unfortunately, I abandoned to this request because the priority is changed but Yeah, I'll find a link and post it in the agenda but From what I have seen at that point it was possible to create scripts, which could be for example hosted in a separate repository Uh, and which would be packaging multiple images and architectures using the same repository structure So that we can reuse the same packaging scripts Okay Oh look for the the packaging. Are you talking about like the manifests? Like scripts to generate manifests Firstly scripts to generate Configuration images and then of course manifest because manifests would be still interesting for us So, yeah manifest packaging approach may change if we stop using co-emo For generation, but still we can do a lot of things But yeah in this repository, I just decoupled the scripts which were related to multi-architecture packaging And they kind of worked. Obviously this pull request thing would have to be just closed and reworked But in principle we could have something like that Yeah, sweet all the folk around Mm-hmm Yeah, so when we create scripts for packaging of docket Of jinx masters if you could just You know document destruction etc. You could create a mon generic work in the next situation Thank you Anything else on the agent agent packaging techniques and supportive platforms? I guess I have I have a personal interest in this one I'll let me offer it as a question I've got a challenge that The git client plugin needs to be and the git plugin needs to be able to test Many different platform versions In their automation, but I don't really want to encode to test every platform combination every time. I've been using randomization Uh and allowing the tests to run on different variants of linux, you know open suzi Debian red hats or let sentos etc Is there is there a better way to handle that? in terms of using docker agent images or olag, I think I had heard of a A docker image Form some kind of a docker image. I could embed inside an automated test Yes, maybe I'll take it up with you separately olag Yeah, I might be missing something. Oh, yeah, let I'll take it up with you separately All right Next topic then jim docker image publishing improvements um That's still uh, mostly the review in terms of the to juice for people to look at the pr um One thing I wanted to add Count on the previous one. Has you any of you guys? Looked at dish or less containers um, I know google has some of Examples out there like just running java in a container or it basically dish or less From what I understand is just the runtime. So just the python compiler or not compiler, but python runtime java runtime um And it you know, it kind of helps with uh security vulnerabilities because there's no shell There's no thing that anyone can escape out of into Um, I'm star is pretty interesting. I started looking at it for the adopt repository as another I guess base image we can support even though it doesn't even have a distro So how would how would that in our case because we do integration I would assume that's a real struggle. We've got We've got to have git. We got to have git lfs. We've got to have a shell um, typically we we install some tool java or java plus c-compilers java plus What about the this while I was mentioning and thinking about the agents because Are the agents just running Uh pure java in terms of doing stuff or they also call down and utilize git and You know other things on those subsystems They they they do So it already depends on the pipelines, but they may Oh, right fair Yeah, so for example, uh, what we use on c-agent can say, uh, we didn't want to manage a git versions there So c-agent say, for example, use jgit So it doesn't rely on Git executables at least on the majority of containers But yeah theoretically There might be use cases where users of images rely on whatever tools embedded Even if not gil git it's likely to be shell or something like that. Uh, so Yeah, all containers are not fully isolated Yeah, I don't it was just very much like a stretch and like an edge case But I saw it Uh, especially for adopt organization where there are, you know, people who pull from that image And might just want to run a complete java and not touch the cell if they're using like open liberty or something like that Um But just a suggestion I just want to throw it out there. Um, because I'm playing around with it So and also if any of you guys knew anything about it, it would be great to pick your brain as brains on that but Anyways, sorry to go off track. Um, back back to the Dr. Imaging, uh, it's really just the pull request needs to be reviewed Uh, have people look over it And give some feedback and then as we go along and get access to the s3 90 and get access to power pc Uh, we can pull those in and start doing Um Tests with the new pipeline anything else jim Not for that. Um, and then the get lfs um I linked actually I think on the last to do this Uh to the pr where I actually talked or the issue where I talked with the developer of the lfs Uh, he mentioned for the installed We want to make sure we're doing that on the jankins user not up on the root user A lot of the jankins images that we you guys are producing Get the get uh get lfs initialize or install. I forget where the exact command is and he says if It's per user. There is a way to do I think passing like a system flag Which will initialize it for the whole system Uh, but then you need to figure your get lfs like, um commands or config to utilize that system I haven't really used get lfs. So I don't know much about that but in my pr I addressed the changes needed to uh, basically move I I still do the install up as root because I need the permissions But then when it gets to the jankins user when we switch over I issue the command uh run, you know get lfs initial or I think install Yeah, and in my experiment yesterday with power pc 64 le There were specifically so I I installed get lfs Yesterday by using the download download the binary from the get lfs distribution and unpack it I assume that's the technique you're using in the In the Good, okay, so I use that technique And was able to install it and confirm that it minimally worked for me as a user It completed my test as it as a user then when I ran the jankins agent Some of my automated tests were failing Until I did some I until until I wiped out the workspace and did a retry and I'm not That needs more investigation from me on why I don't know that that's a power pc Specific thing it may may be an indicator that my test automation For get lfs is weak and needs to become more reputable Uh and additionally forget lfs Um, the if you look at the pr or when you guys are reviewing it You'll see that uh from the most of install methods. Uh, I basically pulling from the binaries that the uh Uh maintainer produces I got him adding s390 and and power pc binaries to the releases Uh, so I basically just check the check sums and they didn't install like that The one thing that uh, you'll see in the different docker containers is alpine alpine Of course uses muscle As it's a c compiler. So the binaries weren't compiled with muscle in mind So he fixed it, uh, but it's all have to wait until the next release when the binaries for alpine are posted So once those alpine What binaries are posted I'll update the container to pull from binaries instead of pulling from the package manager Uh for alpine Because the yeah the package mean the alpine has a get a get lfs package That someone else maintains the author does not retain it And they're usually always a couple versions behind and I I guess you know going with you know containers We always want to stay in sync and what versions we're offering of different tools and stuff like that So, you know, that's why I'm busy pulling from the binaries. So we always get you know 1.10 or something like that Thanks. All right anything else on That one it feels like will mostly resolve itself as as a result of this the pull request as we continue working through the pull request That'll naturally resolve Any other anything else on the lfs topic? No, there was it Okay, any other topics that we need to discuss today? No, thanks specific from me I think we should spend some time in the next months to discuss what are we going to do with java behind java 12 but It's too early for that So you say java 12 I missed that beyond 12. Oh, okay. I see java beyond right. Okay, because so oh leg on that one I had understood that they they have now determined or I've heard talk of a java lts possibly being 17 Do you have use it might be but So there is no final decision as far as I know they wanted to have java 14 as lts But it looks like they kick it down the road for a while for us it's Definitely fine because even java 11 adoption is still tremendous to low But at some point we should start preparing to newer versions Because yeah, they are breaking changes which will impact us Right. So so for instance, I know that we had the The uh, what was it as access there was an access violation warning that they that they'd initially with java 11 plan to make it a hard error and then relax that I assume by java 14. That's become a hard error now. No, it's not. It's not. Okay. All right Well, uh, the problem that is used everywhere So right now they are not sure whether it's going to ever happen Oh, okay. All right That's the if I remember right, that's the access reflection reflective access to Fields and methods Yeah, right Yeah So yeah reflective access is the least of our programs we have issues with model removal Because yeah, it was one of the major issues with java 11. They removed more models and more methods Yeah, while we stay on java 8 support, uh, we cannot adopt a new syntax, but we still need to somehow resolve all the compatibility issues And we know that we have limited support for multiple these jobs in our tooling So Yeah, eventually we will need to start resolving these problems The nice thing about java 14 also is The release of early access builds for alpine That with the muscle compiler So that'll be a nice improvement in case people need that or have tools built around that Because that's always been a pain for I guess a doc to get some of the alpine images up into The official docker images Because they were installing glib gc on top of alpine, which uh, they weren't overly enthusiastic about Yeah Great. All right. Well, so I will include that in up in future agendas. It's it's noted any other topics Yeah, maybe one real one java 11 because with java 11 we now have several recently reported revisions So one issued highlights that we still have limited test coverage for java 11 So one of issues we had historically over past releases a bit, uh, third ci wasn't testing with java 11 at all It's now fixed. Uh, so thanks a lot to wadiq for long year So now security fixes at least get some test coverage with java 11 But still there are some reports for example, uh, Web socket support Doesn't work with java 11 As opposed to covers Well, it's not a problem because it's an experimental feature and it's experimental for reason but Still, uh, there are some Issues, uh, which might impact users So i'm just a distance of what is what pops up in our labels in terms of distance issues So now oleg It looks this this indicates you've clearly been doing triage of java 11 issues. Thank you very much Do we need to do we need to have a discussion about triage on those on platform specific issues? I've not been doing any triage of platform specific issues But it seems like this might be a place for that kind of a conversation at least About the bugs that would be a subset related to platform Oh It would be reasonable to regular take a look. Let's see it doctor images because right now the most of the agent is around to do kept For java 11 to be honest. I'm not sure because Support period for java 211 is over It means that now plug-in maintainers or java score maintainers are generally responsible to fix issues It's still nice to do triage if we get into architectural issues But right now we don't have such problems. Well, maybe except test coverage, but it's a completely separate topic But yeah, so so for example, I saw that You know, I think as as one example of a possible platform issue I think I saw that there are some cases where the the platform or the Plug-in library the shared library for pipeline that's used on ci dot jenkins.io Sometimes intentionally skips windows tests. I don't is that I thought it was on on the jenkins core itself It's not sometimes not just Yeah, windows built are not enabled on core Yeah, it's not enabled because we still need to fix it We broke windows compatibility with summer And we temporarily disabled windows builds So, yeah Almost one year later. It's still disabled. So probably a time to do something about it Yeah, all our tools including build plugins support option code from windows Many plugins do skip it Because windows infrastructure is much less stable in jenkins Hopefully with the ci changes, but yeah, right now windows. First, it's a resource For which you have to wait more for virtual machines and also it's less stable So Understand plugin maintainers who disable it. Okay For some plugins it really doesn't make sense to test on windows because If they you don't do anything platform specific Most likely you have no problem with windows Right, but yeah, some plugins like platform label or whatever. Of course they are supposed to have some windows coverage right, okay Any other topics we need to be sure that we include or consider for future agendas All right road maps Oh road maps. Okay, go ahead. Yeah, so this is a follow-up to a contributor summit we had in russles I still have an action item to communicate The results hopefully I will do this week But one of the items we discussed today is setting up Top-level road maps for the jenkins project so that uh, a special interest groups subprojects Actively develop plugins and other things that could can somehow declare the intent to deliver some features So it's road map not in terms of future commitment and time But road map in terms of where we want to get as a project And yeah, I plan to start working on that to have aggregated In a road map and your platform seek is also invited to share their vision And their goals so we can just put them Now at least Great. Thank you. Yes. I I just realized to my horror That I have not been recording this session. So I apologize everybody there will be no recording of this I cannot go back in time 45 minutes and turn on There is a recording button Yeah, it shows the recording. Oh good. I feel better. I thought I didn't record excellent. Maybe I'm lucky very very good Thank you. Okay. So something is being recorded somewhere. That's good. We'll take it. All right any other topics though Oh like back to your road maps topic It feels like that is a good place where for instance the docker image Publishing rework and the series 390 and the power pc It would be those would be reasonable to put on the road map. Am I as understanding correct? That's for sure Okay, so what is my current vision actually to apply a road map which was used Blows before and which we use Uh, you know several other projects So instead of using quarters or whatever just have three categories Now later maybe Or something similar to that and put All projects in these three buckets. Maybe use some particularization. For example platform support infrastructure Uh, so we'll have such two dimension matrix I do not want to go too far with formalizing the things. So for me it's just Uh These three columns categories and hyper need to location somewhere. Whether it's jira Genki sayo or something else which provides more documentation Great yeah So we could spend some time at the next meetings to write down what we would like to see Is a platform see That's a good one. Well, and and that's one where contributors here could certainly We could we begin that discussion By email and then be reviewing so there's I'm sure there are places where alex has Ideas of evolution that should happen with regard to windows windows infrastructure I know alex you you're certainly doing the evolution on aci and If we go to if we as we go to aws I assume there will be something similar where you use the aws equivalent to get us a faster to start windows image On ci.jankins.io. So yeah, definitely One thing I could forget I resubmitted my pr for Windows master image So if people could review that that'd be good Okay And when you say windows master image meaning a windows An image of windows a docker image for windows that runs a jankins master Yes, that's correct Okay Hey alex, um, I guess anyone else on the call. How Is those builds uh currently handled or are they not being handled right now? Uh, the windows ones. Yes Uh, they're being built in on ci.jankins.io Okay They're built on trusted Okay, because the the whole build pipeline rework I did It basically just accounted for You know all the linux distributions obviously, you know, I don't I don't think windows Don't quote me. I don't think windows works on as 390 But uh, I don't know if there is a need for multi-arch windows Maybe there's like an alpine thing, but I don't know the base windows thing that even works on That's not an outline arm so They do have some art stuff But it's difficult right now Yeah I mean Props are used to take it over the windows things. I mean that that's difficult in itself I don't even know where to start when it comes to windows on docker, so It's not too bad. All right. So I guess alex while we're here. I do have that question I I had hit a bump where I needed command line get Is that Is that still going to be a while? I assume Before the ACI image gets changed I should be able to look at it this week. I just haven't had a chance yet with work stuff Yeah, and no no press because I've got plenty plenty into spare on my list of other things that need my attention So don't don't feel pressed to do that. I just was asking in case it was on your list and if it's not no big deal I've I've got plenty of other things. I need to be doing All right any other topics Okay, then I am going. Thanks everyone. Let's call that the end of the meeting. Thanks very very much