 Hello, welcome to the platform special interest group meeting. Today we have a first regular SIMCAP about ongoing projects. We will talk about docket multi-architecture images today. And we will also talk about the Java 11 preview availability announcements. So let me share my screen. OK, do you see my screen? Yep. Yes. Today we have several people on the call. So there is Adrian. He will be presenting a demo of Java 11 on the server green. There is David Curry, Baptiste Matos, Mark White, and Olivier Bernin. We asked Oliver to join the meeting because last meeting when we had a discussion, each in each projects we had questions about architecture and specifically about trusted CI. So it would be great to have a discussion how we could make it better. So today's agenda, I want to make some announcements about the platform security leadership. So I will be off for the next few months. I mean, I will have a really limited availability due to personal reasons. And I had some discussions with Mark and with Baptiste. And they agreed to be also leaders of a platform special interest group. So Mark and Baptiste, they are long-time Jenkins contributors. I believe you both started contributing much earlier than me, so I do not know for how long you have a project. And I believe that it's well deserved. So Mark did a lot of activities around platform support, including various evangelism activities. And Baptiste is an active contributor in Jenkins project, including Docker images and also Java 11 support now. So I think it's well deserved. And thanks a lot, guys, for doing that. Regarding, so I will be sharing all permissions with Baptiste and Mark. And if I'm not available, somebody else will be able to drive the meetings. And speaking of meetings, since we have regular meetings now, I tried to improve the meeting scheduling process a bit before that it was just maybe increased and discussions we will start in between before each meeting. I decided to actually create a Google Doc, which looks like Jenkins Governance Meeting document, but in Google Doc. So here we have some dates. And if you want to propose a specific topic for the meeting, you can just propose a change. And then, yeah, you may see that Mark just does the change right now for January 17. And yeah, for each meeting, you are welcome to just propose agenda. If you want to put something, just propose change there. And we will be monitoring changes in this document. And here are meeting notes for the today's document. Just in case you want to join this call, we also have a Gitter Chat. And in the Gitter Chat, you can find a link, a participant link to the presentation. It's here, part from Sieg. Yeah, there is a link. And if you have any questions, feel free to just to join the call. What else have you done about that? I have created a public Google document, sorry, public folder where we will also keep all the meeting agendas and meeting notes. OK, MS up at the link here. OK, so the actual link is here. I will fix it. Yeah, just fix it on the call. OK, now it should be fine. Yeah, so if you go here, you may see that there is this meeting agenda document. And also all a platform Sieg meeting notes from the previous meetings. And I hope you will be using this folder just to put all kinds of public documents so that we can keep them in the single place. Unfortunately, Jenkins Project has no JCUt or similar applications. So we cannot store it somewhere on the project account. But at least we can share something in a single place, which is already a good step forward. OK, regarding other topics, maybe we could place it with the agenda. Or since we have Olivier on the call, maybe we could briefly talk about access to trusted CI. What would be your preference? I'd go with the trusted CI topic if there's a hope that Olivier can then excuse himself later if needed. Yeah, let's do that. So the bookstore, we have three projects. One is multi-architecture docker images. Then it's Java 11 with all the docker images. And then this is Windows installer with changes to packaging. And surprisingly, all of these three projects are related to Jenkins release flows, which are hosted on trusted CI now. And last time, we modified docker repository for Java 11 releases. There were some regressions. There were some unexpected behaviors in the publishing process. And our main problem that we have no access to trusted CI, we have no way to diagnose what's going on there. And effectively, each change we do is you leave only once, even if you're a maintainer of the Jenkins docker official image. You have no traceability of what actually happens during the release. And it would be interesting to resolve this issue somehow or at least get notifications if something goes wrong. So Olivier, if you have any ideas of how it could be done, it would be nice to know. So from the notification part, I was just thinking to do that from a Jenkins file to just say that when a job is failing, just send an email or just send a short message, for example, in the GitHub channel to just say, oh, by the way, that job is broken. For me, that was the first step. The thing is that I doesn't feel comfortable to share access to trusted CI because there is a lot of critical information there. So something that we may discuss is, for example, provide access, for example, to seek leader or something like that, maybe we can find an agreement on that. One person can have access to trusted CI, but more globally because we never reach the time to really design who has access to trusted CI. The default behavior is just nobody has access to it. So that's the first things. I don't know how many people should have access to that. I don't know if you have any suggestions there. I'm not sure we can consider any first access, but there was a ticket to try and at least notify indeed things. So if we add something like a function in the pipeline library to send its IRC messages, we could then use it pretty easily from many critical pipelines, even evergreen or those because that's kind of the same issue with evergreen. I've always regularly issues with the pipeline being broken on trusted CI, but I have no idea until I look for it basically. So what do you think about first having something on IRC like a pipeline library so that it's easy to send messages and maybe? I think anyway, that's the first iteration to get notified when a job is broken. As you mentioned, it's not only an issue for the platform stick, it's also something that happened for evergreen and all the other decor images. The main concern that I have here is how to share access to all the different people managing, maintaining the different decor images without compromising that main Jenkins instance because the purpose of that instance is really to stay hidden. So, yeah. Yeah, one of the ways would we actually be able to deploy a local instance for evaluation? For example, for CI Jenkins, there are some unofficial projects. For example, I have a demo configuration as code which allows to develop pipeline libraries. And if we had something like that for trusted CI, maybe with some documentation, it would help us to do at least some local development and minimize the risk of issues. Because now we are just blind when we deploy something. And if we had at least the basic instance which can reproduce common steps and common behaviors on the instance, it would be already really helpful to us. Something that I've been investigating for other SEGI is to deploy a bunch of service on Kubernetes. So the idea, for example, to have a custom decor registry so we can just publish the decor image on a sandbox decor registry so we can use exactly the same scripts and do exactly the same process that we would do on the Hub but just on private services. And this is something that can be quite easily done with Kubernetes. So I don't know if this can be a solution as well. It could be a solution. Another solution which could be even more simple is actually using another Docker Hub organization for deployment and maybe using standard CI Jenkins Ion. So in addition to local instances, what we have created before, if you don't mind, I'll just show. We have Jenkins for VAL organization. This organization has been created in order to enable Durgadas and other contributors to prototype Jenkins images. So for example, there is Jenkins for IBM platform. There was also ARM image before. There is repo for COIMO, for Blowsh, et cetera. So this is a separate organization. And one of the things we could possibly do is to just grant to create credentials for CI Jenkins Ion with full disclaimers that it's not secure or whatever but at least grant people opportunity to push to this particular organization so that they can push experimental images and assuming that CI Jenkins Ion and trust CI are relatively similar. It could be also a way to debug and to prototype something. And we could also deploy pull requests, builders, and other such things. And maybe some continuous delivery flows for untrusted images. Could it be an option from your point of view? Yeah, I just have another suggestion that I just forgot that we have a private registry on the infrastructure that we give. I mean, we have a private registry in Azure. So we could also use that registry if we want to do sandboxing. OK. Yeah, if we could do that, it would be really nice. Because it's already there. And in that case, we don't have to share it. I mean, it's not official. Yeah, but will the users be able to access it? Because one of the advantages of Jenkins for Eval is that you just publish it there. And then it's a standard Docker image. And if you're there enough to use it, you're more than welcome to do that. If you use some Docker registry, it may be a bit more complicated. Yeah, sure. Because you have to put in place. You have to configure your user name and password. It's not the defaults, but in that kind of case. I don't think it's too really complicated. But yeah, it's more offensive. But technically, if we wanted to deploy one of these approaches on CI Jenkins, how much difference are instances between each other? Nothing, just in the case, anything that we change is the user name and the password used to push the Docker image. And of course, the image tag, which is another image tag, the image name, is changed. The image tag will be a responsibility of repository maintainers, because it's their responsibility to create Jenkins files properly. Otherwise, they will be just getting failing built due to lack of permissions. And they will ensure that they do not get go and grant permissions to these temporary users. But yeah, for Jenkins for Eval, that could be all for private registry. This approach could really simplify the things, because we will be able to set up CI flows. We will be able to set up CD flows for trusted images. And it doesn't seem to be a major effort if we have capacity on our infrastructure. I think I filed something about that for Evergreen, but that's kind of the same issue in general for trusted CI things. We didn't make sense to also kind of introduce some data.dog-related monitoring for a thing that's supposed to be successful on trusted CI side and so that it's kind of triggering alerts or... Yeah, so this is something that I've been thinking that I've been thinking several weeks ago. So there is two things. The first thing is we have to configure Jenkins with the data.dog API key. So we have an integration between Jenkins and data.dog, and then we can put in place some monitoring checks. The only thing that I will need to do is just to install the data.dog plugin on CI.dog Jenkins.io, and then someone else can just create the monitoring check since those are public repositories now. Very quickly, data.dog provides a sponsored account to the Jenkins project nowadays, right? Yes, so we are using data.dog to monitoring on our infrastructure, and we have... So there is a docker. There is a docker of the GitHub repository, which is taking Infra slash data.dog where we have all the monitoring configuration. So everybody can write specific monitoring check as long as we have the data, of course, on the data.dog account, but it's not a big deal, especially on CI.dog Jenkins. Yeah, that's cool. So just, yeah, if we use that data.dog, we don't have to modify the Jenkins file, or we can just go with the Jenkins file and notify a specific RCA account. Okay. Yeah, I think it would be really a good addition for... So yeah, probably there is no questions. Probably we should note the action items and move to other topics, because yeah, there is another topic for Olivier about integrating the pendant patches. So I propose to actually create a ticket, maybe, Olivier, if you're fine with Jenkins for evaluation, maybe it's something we could do. And maybe another ticket for notifications and monitoring. Yep. I don't know if I'll recreate the one for the data.gif. Mm-hmm. Mm-hmm, let me check. Mm-hmm. Let's dok. May exist in another one for your notifications, right? The thing is, we don't have to write, yep, data.gif. Mm-hmm. Yeah, right. And regarding Docker Hub user for this, temporary deployments, do you prefer to set it up on your own, or do you want somebody to configure it? I can't consider it. Yeah, it would be additional level of security so that we do not know credentials. Yep, I'll do it. Okay. That's cool. So anything else we're missing here? No, I think it's fine for me. Any questions? Yeah, then let's move forward with the agenda. So one of the items we had here is about multi-architecture Docker images. Unfortunately, the docker is not on the call, but there is a pull request and there were pinks about getting integrated in the mailing list. There was pink in mailing. Yeah, there is a mail thread with many questions. I was about this one multi-architecture support from Alie from Alinar or if I recall correctly. So there are some questions, but generally one of the questions is when is it going to be landed? If I understand the status correctly, Olivier, you are fine with this pull request. So we're ready to land it once. Yeah, Josh, is it ready to integrate it? Is it right? Yeah, the only thing is I would like to coordinate. So I'm around where we need to run that on trust.ci. So to be sure that if something is broken, we don't leave a broken job during this. But if you have Jenkins for you all, we can run it on Jenkins.io. Sure. So then in that case, you have to wait that the Jenkins.io is configured. I see Jenkins.io for... Oh, yeah, I messed it up for Jenkins for you all. Yeah, I'll just make a note so that we don't miss a bit. Okay, and then we can just move forward to the next topic. So yeah, you're totally right. If we use this approach, we can just update this pull request. And then we can do many things, actually, because we have the same design in Jenkins Agents, which are now being released from Docker Hub. And actually, we could start slowly moving them towards the new flow as well. Okay, thanks a lot, Olivier. Actually, it answers questions about Docker packaging. I also have one question to you for Java 11 packaging. So if you could stay online maybe for five minutes, it would be nice. Okay, let's go to Java 11 then. So with Java 11, actually what I wanted to discuss with you is about... is announcing preview availability for Java 11. We had a discussion two weeks ago about the status readiness. Actually, at this discussion, we reviewed the current status and there were three major blockers. One is for pipeline support, another one for test automation, and the last one for Docker packaging. But currently, all these three major blockers are actually addressed. And yesterday I sent an email to the developer mailing list summarizing the status. It was a bit late for the platform seeking discussions. Sorry about that, but it summarizes the status. So for Docker packaging, everything is done. Thanks a lot to Baptiste, Olivier, and Carlos. So we have weekly releases with JDK 11 talk. We also have all agent images. We still need to deploy bloat, but we will stick to experimental repository for now, which is already available. For testing and automation, actually, it's also good. So we have automated builds on Jenkins Core. We have acceptance test harness, which is kind of passing. So Oliver Gonza and Lucie, they did a lot of updates around acceptance test harness. And if you open the pull request, actually it has been integrated. So now we can just probably take a look at the master branch. But here, what we achieved there is that now all automated tests actually run for Java 11 and Java 8. They are paralyzed. There were some questions about Olivier, about the increased workload from acceptance test harness on CI Jenkins. So sorry about that, but what we actually achieved. Okay, some bits of interface are in Russian. But I think that we have tests running for both Java 8 and Java 11. And there is only a few differences between these releases. So after the discussion with Oliver, we already said that acceptance test harness is actually okay for Java 11. So this is one of the new updates. And what else? There were a lot of updates for plug-in forms in pipeline libraries. So for example, now if you want, you can run tests on both Java 8 and Java 11. And there is an example of the new configuration. So you can define different configurations, Linux, Windows, which JDK use, which Jenkins use. So here you may see that there are multiple combinations running. And this request is also green. So now as a plug-in developer, you can deliver all the plug-ins. And last but not least, we also did a lot of manual testing and exploratory testing. So in June, then in September, then in October, we had hackathons. We were doing a lot of testing between them. And we know that the most critical functionality is actually working fine, except some bits of pipeline. I'll talk about that later, but that's good. And now there is a work on running plug-in compatibility tester with Java 11. Actually, it was planned for GA availability, but we want to do some work in advance. And thanks a lot to Adrian. He did a lot of patches for PCT and for pipeline libraries so that we can actually deliver this component. So it's also good. And the last blocker, which is not fully resolved, is about pipeline support plug-in. So the story behind it that there is a breaking change we need to apply if you go to this pull request. By the way, am I still screen sharing? Yes. Okay. Just in case I messed up. Okay. So there is a pull request and in this pull request, you need to update a J-BOSS Marshall to the new version. And unfortunately, it causes some compatibility issues for runtime context. So if you restart your instance with running Jenkins pipeline, then Jenkins cannot load the instance after that. And technically, we need a better patch in pipeline support and Sam Van Oort, who is actually on the call, he's working on this patch. But what we... Spent most of the yesterday digging into issues here that are posing problems. Okay. So if you want to provide more detailed update, feel free to go for it. Okay, begin. Our company is not going to be terribly helpful unless you're deep down in the dark, scary innards of workflow support plug-in right now. It's basically this is causing some subtle and nasty changes in the way the pipeline program is serialized. Those are not compatible. They also seem to be causing some side effects that are breaking several of our tests, which are indicators we may have other problems at play here. Yeah, right. I can give a list of tests, but I don't know how meaningful it will be for the other people on the call. I probably will need to revert that so it will be explicit. But yeah, one of the approaches we were discussing is actually to deploy a temporary experimental version so we can unlock preview availability. And this is the approach actually we would like to go forth with the preview availability next week. While Sam is working on the full patch, we have a pull request here. This pull request is totally not a solution for the issue, but we did a lot of exploratory testing and actually it seems to be working okay in real life. And what we would like to do is to actually deploy it somehow. And there was a discussion about creating a new experimental update center so that we deploy an experimental update center only for Java 11 updates so that it doesn't impact users of Java 8 even if they use an experimental update center. And we spend some time implementing these patches. So here it's an actual implementation from the plugin developer standpoint. There is a new release of plugin POM and what this release does, actually it starts injecting minimum Java version and the plugin metadata. And if you overwrite it, you can say that you want this plugin to be working on Java 11. So we integrate these patches. Here's an example of what is being projected. Effectively, we just inject the minimum Java version in the manifest entry. And then our update center. So there is update center repository and there is also a full request against this repository. Actually, it extends work by Daniel Beck. He was working on some Java version provisioning maybe one year ago and effectively this entire work including maybe an HPE patches under the hood is based on Daniel's work. Here we provide a new update center which will be hosting on the Java 11 patches. So it means that we will provide this patch only for users of Java 11. Of course, when this patch is deployed because we need to also update the generation of Java 8 update centers so that it doesn't get Java 11 patches there. But this is our way to address this issue and actually I wanted to sync up with Olivia is how would we deploy that? We had a meeting last week to discuss what needs to be done and we addressed changes requested by the Jenkins infrastructure team. So the question would be what would be the next steps for this deployment? That's a very good question and I don't think that I can answer that right now because I don't know that application enough so I prefer to avoid saying things. So I'm fine to help to merge this and to see how it goes but right now I can't read. Usually this repository is managed by Daniel Beck and of course we will try to get his approval. But yeah, after that are there any specific tutorials needed to deploy this experimental update center? The thing is, no, I don't have enough context to speak about it, sorry. Yeah, no worries. So we will just talk to Daniel once we are ready. So it's under review and I think that it should unblock the deployment and actually the major changes here and there are some other changes unrelated to blockers. So for example, Mark Wake has delivered the fix for Java Web Start so now we do not offer the Java Web Start button in the web UI when you run with Java 11. Then there is a purchase from Adrian and Baptiste for Java 11 support in Jenkins Evergreen. We will have a short demo today and there was also an exploration of Java 11 support in other companies like Jenkins Faldranar but they are out of the scope for Java 211 so now it's rather catching additional defects that we haven't covered yet. And yeah, my question about this topic would be whether you are fine with doing preview availability announcement next week if you go with this plan. We still know that there will be a lot of regressions and actually this is one of the reasons why we want to deploy the preview availability and why we want to announce that. So there is some adoption while Java 11 availability is protected by the feature flag. So now you cannot just run Jenkins on Java 11 because you need to set the flag otherwise it refuses to start. Of course in Docker it's a bit more easy but still and yeah, we want to make it more public so that people can try it out, report the defects and we are going to have a Java 11 support team which will be trading these defects and processing them and sometimes fixing them. So it's pretty much the same process as we used for JEP 200 last winter and spring and yeah, we will be maintaining Java 11 at least till April 2019 so there will be a pretty good chunk of time where we can fix the most of the reported defects or at least we can prioritize them and write some documentation how to avoid them or whatever and we will also be creating developer documentation so people know how to verify and fix their plugins if they discover issues. So this is our steps we are going to also take before the preview availability but generally we are ready to go and yeah, I wanted to ask you guys whether you think that it's ready to go or whether we need to do some additional steps before we go forward. I think it's ready. There are plenty of things still to do and we will identify more by going into preview mode. I think we are at the point now where we are stable enough let's go preview and learn more as we hear from other users. Would you like to deploy that version on the Jenkins Infra project or would you like to build some project on it? It's actually discussable so if we could create an instance for exploratory testing it would be nice but generally we have docker packages so whomever wants they can just run docker package there are some demos like demo jinx configuration as code which have been all updated for Java 11 so I don't think that it's specifically needed to have an instance. What could be cool is to have Evergreen instance running on Java 11 because Evergreen submits telemetry so it means that if something goes wrong we may get some information from there but yeah, I'm not sure how Evergreen is ready to that so maybe Baptiste and Andre have some insights. That's kind of a good point I think. We are not very far so as I've explained in other areas basically Evergreen is close to... Right now it's kind of a preview too we say it's not prediction ready but I think it wouldn't require a lot of work to actually make it usable and start using it for some kind of subset because that's kind of the stage we are right now and that would make sense and deploy it somewhere and see by the way what needs to be fixed in Evergreen itself to be able to run the workload if we misconfigure something that's right now not been worked on enough so that it's actually working out of the box as per our definition of Evergreen and yeah, so it would make sense and I think it could work and then we would just, you know, it would discover along the way so it would both be small work depending on Evergreen's side but it would definitely also be interesting from the Java 11 side so I guess it's kind of a joint effort interesting joint effort and what if we have a demo of Java 11 on Jenkins Evergreen? Oh my god, that transition, you're a superstar. Andre, yeah. No, it was intended but it works. Good job. So let me share my screen first. I have a small demo so I guess that every one year knows about the current situation of Jenkins that it's... we can start Jenkins with Java 11 but we have some issues. Maybe we can start the re-answer with interrupt maybe we can start like I should have said that just before maybe like 30 seconds before trying to re-contextualize maybe Evergreen or something. Yes, please. Quick summary of Evergreen concepts, other channels, etc. I'm going to kind of do that because I'm the one who's I guess so yeah, if you feel like you want to do it I'm really happy if you want to. I might not be the best person to do it. Sorry, that's my fault I should have thought about that before so Evergreen I'm going to try to do that in like one minute tagline. Evergreen is a new project we started roughly at the beginning of this year so 2018 which we want to use to kind of propose a way for our end users to use Jenkins in a very easy way. So the tagline is that we want people to be able to actually start building their own projects in less than 5 minutes. So there are a few differences for people's flavor. So one of the first ones we had were the Docker Cloud Flavor which makes it easy to use it on your machine to try it and to actually see how it works and everything. There's already another existing one which is called the AWS EC2-1 which is something that's going to be automatically configured your whole instance and right away you will be able to build a pipeline using automatically configured and provisioned dynamic and transient instances in EC2 VMs and so we leveraged that nice flavor feature so that we can actually create a new flavor that's based right now on the Docker Cloud one to see how we could like propose an Evergreen flavor running on Java 11. The nice thing with Evergreen is that we also because we don't want users to do that we want them to concentrate on delivering software, we preconfigure the plugins that are installed, the versions we can push versions and we can also get automatically automated logs, telemetry we get automatically the logs and errors for running the errors that happen. So we worked on that and Adrien worked on delivering Java 11 flavor of the Docker Cloud one so that's what Adrien is going to do right now basically we are leveraging the infrastructure and the feature of Evergreen so that we actually can propose a version of Evergreen that's going to be running on Java 11 without you having to do like anything but running just a single command like you would usually do to run any other Evergreen flavor basically I guess that's really enough Yeah, so really the idea is to have always up to date Jenkins and with plugins that's offered by Evergreen so I have here a normal Jenkins, the latest LTS running on Java 11 and just to show you that I'm not lying I'm really using Java 11 and I have a small pipeline job that's just the hello world and we can see that it's failing because of known issues with Java 11 on the latest release of workflow but we are on Evergreen we have we fixed that by the version of plugins offered by Evergreen server so basically to start an Evergreen instance you can run a docker command that is shown on the websites you have all the details the only tweak here is to make sure to use the correct label so the Java 11-docker-cloud and that will start the instance locally it will load all the plugins that we that the Evergreen project recommend I guess we can present that this way so you have let's say normally the log you don't look at it and so you have all the details of Evergreen and when it will be started we will have an instance with the correct set of plugins for the soon enough there we go I shouldn't even have to rebuild my screen there we go and so to get the admin in this world again because it's not that I will go again on the website so everything is shown here the only thing that is not shown on the project pages the tag to use so again here I can show you that again running on Java 11 when I start a part time job doing exactly the same thing so the classic hello world and normally that should run perfectly on the first try so just some time to start up but we'll see that the build is actually running and on Java 11 when it's starting you can probably explain why it would do what it would do without our pre-configuration of the right version of the plugin yeah but basically it's what I shown at the beginning you would have that kind of issue and the build would fail and you could you cannot use a workflow on a classical Jenkins instance because there is an incremental version of workflow that was deployed to enable the support of pipeline running on Java 11 and that is something that incremental version is consumed by Evergreen so maybe we can insist on what is an incremental version of the plugin yeah I guess well I would say let's take that for later because it's I think it's not an Evergreen presentation per se it's more to focus on the Java 11 side and the nice also side for users and contributors to spend like 5 minutes to test Java 11 and see if something works yeah exactly so in a command line in a simple docker run you have a Java 11 Jenkins and you can test your classical pipelines to see if we need something we'll discover some plugins that we need to be fixed and so on but basically it helps you to run Jenkins with a set of plugins that can support Java 11 great any question anything I should share with you but I'm not Adrian who is monitoring the telemetry that's coming back from these Evergreen instances does that go to Baptis to others in the Evergreen world should I be monitoring are there it sounds like a great opportunity we can get more people just using Evergreen to do this and we will automatically collect their experiences so it's not coming to me Baptis maybe you can speak a bit more about that yeah I can explain quickly but again I think it's kind of well it's partly your right mark it's partly in the context but yeah so we are using leveraging the Sentry software as a service tool that's a user log error logging I would say aggregator and on which the Jenkins project has a free license like for Datadog the Datadog that we talked about earlier and so there we can actually monitor I can show you that quickly like again I don't think you need to do the details I was more worried about the human beings who are thinking about what the messages say so here you see the dashboard so we see what was sent and we see how many events we saw for a given event like this one for instance and we saw how many users how many different instances was impacted by this basically so we have so many Evergreen instances already no as I was saying I think