 Okay. Hi all, welcome to the platform special interest group meeting. Today we will have a discussion about Java 11 support, Windows installers and also custom docker packaging. So if you participate in platform seek, you may find a link to the meeting notes document in the email announcement. And I also put the meeting notes to the data chat. So it's here. And if you want to join in this meeting notes document, there is also a participant link. So if you want to join the call instead of watching PT on YouTube, you can just join this session. Okay. So we have three topics. Should we just start from Java 11? Oh, probably I should be sharing my screen. Okay. Yeah, it always happens. Okay. Do you see my screen? Yes. Okay. So just to repeat, here's our Gitter chat. And in Gitter chat, you can find a link to the status document. So all the agenda is here. We'll be putting the meeting notes and here's a participant link. So you can join the discussion. Okay. Three topics. Let's start from Java 11 update. Yeah. There was an announcement on September 25th that Java 11 is now GE. So it means that everybody can install it and it's possible to start using it. Actually, maybe we had some testing cover last week on the early access build. And yeah, we will talk about the status of Java 11 support and Jenkins. Speaking of the status, there is a JAP link which describes what is our plan regarding Java 11 support. And I propose to talk about this plan and also talk about the changes we had over the last two weeks. So we had a hackathon at the DevOps World Jenkins World Conference. There was something, how many, maybe six or seven contributors to Java 11 there. So we can just summarize the status and then deep dive into detail. So Mark, would you be interested to summarize the status? Sure, you bet. So as we gathered together as a group at the HackFest and a number of significant improvements came in the packaging, in the development flow, especially the way developers interact with Java 11-based development. We now have the Java 11 early access package on ci.jenkins.io. We'll submit a new ticket to ask for the final release but we verified that we can build on ci.jenkins.io with both Java 8 and Java 11 in the same call to build plugin. It works. Now it does require changes, things like update to the latest Jenkins version, those kinds of things. However, really solid work. So the development build flow is there. We performed interactive testing as well. Mac OS, Windows and Linux using the development build and had good results with all three of them. We've still got plenty of work to do that's described in the tickets. Oleg, did that cover what you wanted or are there more things you'd like to summarize? Yeah, I could put some additional summary if you don't mind. Okay, so back to the document. So yes, we've got a build flow for plugins running. We also got a good progress for Jenkins Core itself. So there was one full request which meet the common build flow working on JDK 11. So currently you can build Jenkins Core or Warfile. You can run unit tests and the other rows, find bugs, stop running Java doc with some exceptions. So yeah, generally it's a good progress and we integrated this change towards the master branch. I mean, Java 11 support branch. So it's a master for Java 11 and currently it's available in our Docker images. And actually during the testing we discovered several issues we've seen for Java 11. So first one is about update center signature failures on the recent Java 11 release. So we worked around it in our images and there is a patch from Daniel which we need to integrate. Then there were some regressions in our tooling which we fixed and also regressions in process management which we didn't catch during the testing in June. So I have a staged pull request which should fix a bit. Yeah, Unix Reflection class compatible with Java 11. So currently it shouldn't be working on Java 11 at all but apparently it was somehow this fix it should be working. Regarding the status in general. So in JEP we have three pools of tasks. So it's general availability and weekly release which is what we target now. Then there is J in LCS at some point and a lot of other issues which we don't consider as non-blocker ones. So regarding other issues actually after the hackathon I believe it's not linked correctly. Yeah, so other issues are here. So we have a pretty big epic about it. Many tasks are already resolved. Some tasks still need some polishing but yeah, still Java 11 works and one of the critical issues we had was about Jenkins pipeline. So we need to update Groovy in a Jenkins core in order to support declarative pipeline but reported the update of Groovy caused the meta space leaks on Java 8. So we spent some time at the hackathon in order to reproduce that and to identify and actually we were unable to reproduce it so fast. All performance tests were pretty good. So we decided that we learned Groovy update in the weekly release. So the current weekly release already includes the Groovy update we needed. Yeah, it's here. And it means that this patch we are getting closer to get the stuff running in weekly release. So currently guidelines for running Java 11 mostly work for weekly release as well. Just second, running Jenkins with Java 10 and Java 11. So this is our guidelines and these guidelines should now work for weekly releases as well. There are still some patches we haven't upstreamed but many patches are already there. And yeah, you may see that there is a bloat in GDK 10 image and actually we produced another one for GDK 11. It's located in another place because we needed to stop and use the data over there. But now if you want you can try Java 11 with vanilla packages and with bloat and pipeline which includes everything just by using Docker images. So Oleg, I think you said that I may even be able to test with 2.1343 as my base Jenkins version. Yes. Cool. So let me show you the difference. So we have an ongoing patch branch. It's called integration branch. It includes a number of commits just because we merge things here and there. So yeah, it was read for a while but now it's back to green because we fixed the unit test. And if you scroll through the patch actually there was not so many differences in code base. So it's for CI. Yeah, so there is annotation indexer patch which we haven't upstreamed yet. It's just needed for the build. Then there is JNR politics which only prevents illegal reflective access warnings. So this patch has been originally created by Devon in June and probably we should integrate it but it's not a time tracing vision. But yeah, I think it would be great if somebody creates a pull request against master branch. Then there are meta in services. It's something with indexing of extensions if I recall correctly. That's why we needed to upgrade that. And in addition to that, there is a lot of refactoring to make the code compelling. I'm waiting for reviews so that it also gets upstreamed. And yeah, then there is process management. And the rest of the code is actually already integrated into the master branch. So if you want to try it, you can just try master now. You just follow the guidelines here and set up enable future Java flag. And it should be running. For Java 11, yeah, it's a bit more tricky because we still have these components. We haven't resolved Jaxby issue so far. But we are getting closer to that because Made in Compiler plugin was updated. Now we can use multi-release jars if needed. So yeah, probably we will be able to resolve this issue. But with this command line, you should be able to run a weekly release on Java 11 now. Excellent, thank you. Mm-hmm. Yeah, so disclaimer, it's not an official support in whatever meaning, but it's available as preview mode as described in our policies. Okay, so this is a positive part. Less positive part is about test automation because in order to get it in the weekly release, we need test automation. And here, yeah, we have a lot of gaps. So I spent the most of the time of the hackathon in order to get plugin and core build stuff running. Core build is still in progress. So we have unit tests, but we still don't have integration tests ready. So there is a pull request. I reverted the flag for running communist project builds. And yeah, so it was running only a number of Jenkins test harness test before. And yeah, there is some issues now, which I need to fix, but yeah, I'm pretty optimistic about that. So let's go here. Okay, so JDK 11 build process. JDK 11 builds will fail, I believe. Yeah, so there is still some timeouts due to whatever reason. In some components still failed to build. Yeah, here we have a full log. So yeah, this is a finding bugs issue because finding bugs doesn't really support Java 11. And currently our plan would be to rather move to spot bugs in order to fix that, but effectively it's harmless. So we just pollutes the build log and doesn't break anything. Then yeah, there is maybe dependency plugin issue. So there is a pending update for it. And yeah, it will be fixed. And yeah, there is also some test issues, which I don't see here, but yeah, we are doing progress. So currently the only issue I see in my local build is this test related to running a build itself. So Jenkins test, you can close tests like build trigger test, which effectively woke maybe my build as a part of the test. And this test apparently not compatible with Java 11. So yeah, they retrieve Java version. So it becomes 1.1 instead of Java 11 and then it crashes. So yeah, we will need to do something about that. But yeah, generally it looks good. There is not so many test failures. It's about integration tests. But yeah, even when we get integration tests running, we still need to do something about acceptance tests, harness plugin compatibility tester. Then we will likely need to update build plugin in order to make it a part of standard flow. So what Mark did for platform labeler plugin? Mark, you generally used the explicit dependency on the incremental series, right? I'm not sure I caught that. So the platform label or dash, sorry, the sing the dash, that one. Yeah, so this pull request uses a, yeah, you can just show the diffs because they're actually quite simple. Okay, so yeah, Java version. Yeah, here you just picked the recent core version. Right. Well, nice to see that it works in such way because what I did in the label verifier, in label verifier, I switched to the incremental version from the upstream. So here I use, yeah, this guy. So it's a development branch, Java 11 support. But yeah, and so yours was pre, your change was pre the release of 2.143 probably. Yes. Right, okay. So even if it works, it's actually really good. Maybe in compiler plug-in. Yeah, that will be deleted when the plug-in palm updates won't it, I think. Yeah, right. So for plug-in palm, we have updates originally created by some, I guess, Java 10 parking. Thanks, sir. It's got my name on it anyway. Oh yeah. Okay, Sam. So yeah, I believe we need to get this pull request over the fence. So yeah, we need Maven compiler and actually you'd rather need to go after 3.10 because it includes support of multi-release jars. Oh, that's good. Yeah, so they finally did that and it means that we can actually start building our own tooling about that. So but all like I didn't see a 3.10 release from the Maven project, has it released or is it still snapshot-based? I use it. Oh, you do? Yeah, I use it in a code parent form. Or maybe I messed it up just a second. Maybe I meant another version. Oh yeah, it seems so. So what I meant, actually I should have updated this one as well. Yeah, it's Maven compiler, but it's not 3.10, it's 3.8. So this one includes support of... I think I have that in that build actually already. And you do, yeah, that is there, Sam. Okay. Yeah, right. So I'm still not sure that all our tooling supports that because we had a lot of issues with multi-release jobs, but yeah, it's definitely a step forward. It really doesn't support it properly yet. That's what to do. Yeah, so Nikola did some experiments with that. We also have some patches created during the hackathon in June. For example, Leap process utl said this. Yeah, so this one converts the process utl into a multi-release job. So he and Nikola just built it on his own. But yeah, probably now it can be converted to the common Maven compiler flow. Yeah. So let's see how it evolves. But yeah, I think it's really useful, especially for cleanup of legal reflective access. Because currently, if you want to clean it up, for example, I have an issue with the code. Yeah, this one, Unix reflection. So our Unix reflection was dependent on a class which was actually removed from Java 11. So here's some crappy things. So I have to retrieve things differently for Java 11 and Java 8. It kind of works, but either they would like to move this component to a multi-release jar. So the code is clean and we do not misuse these APIs for Java 11. Yeah, because I retrieve process handle then I will destroy all methods are public. So we don't get reflection issues, but yeah, it's a bit weird now. So like where should I go to learn more about multi-release jars? Is that a common thing? I can just Google search for it and it will get me an education. There is a good to jump about that. Okay, I can read that then. Yeah, I can find it. Yeah, so actually a JEP 238 is a good starting point. And yeah, we'll put the link to the status doc. I sent you one of the better resources I found, Mark. Thanks, thanks very much Sam. Something I was using during the HackFest. Yeah, we will have a set of links somewhere in HackFest documents, that's slides. So if you have any links, please feel free to put them to the document. So yeah, I think we should review some of the status because we spent too much time on that already. So we did a good progress. Yes, we still don't support the Java 11 in weekly releases. It doesn't seem to be impossible, but in order to do that, we need to invest a significant time into test tooling. So it's still an open question. But yeah, if you want to do that, there is an opportunity that we deliver it by the next LCS baseline. So we have something like one month and a half to get all patches in the master branch. And once it's there, we can probably say that it's available in weekly releases. And then we can say that it's available in LCS. Maybe this plan is too optimistic. I do not know. What do you feel guys? It feels very optimistic to me, but let's keep working on it. I've got some stuff to sort out in Pipeline land here that I'm looking on, but I think that progress is progressive. We're getting close. I believe we still need to workflow some support plugin release. Yeah, that's blocked due to something, some test failures right now. I'm fairly sure that something in the serialization format that JBLS Marshalling is emitting is not backwards compatible. And the question is if that's a blocker or something we just call out a big flashing marquee neon letters when we cut the release to say, hey, don't run any builds across these two. So when you say across those two, Sam, could you go a little further on that? What would that mean? Yeah. So basically, so what that would mean is that if you're running a Pipeline before you upgrade, you won't be running it after. I see. Okay, meaning Pipeline execution is not one of those things that you get to carry forward across the change from Java 8 to Java 11. If a Pipeline is in running, sorry, you're going to have to restart. You're going to have to run again. Okay. Yeah. That's actually not as bad as it sounds because you can simply wait for Pipelines to complete before doing that upgrade, but it will be a surprising change for users. And I expect there to be some hard work if we do that. Worst case, if we get close to EOL and I can't figure out how to bridge that gap without massive amounts of effort, then we say, well, here's a big giant caveat. Be careful with this upgrade. Here's why. And then, basically, once you do the upgrade, there shouldn't be any issues as far as I can tell, but it's only in that transitional period that it's a problem. Okay. So, yeah, I believe that it's something important and we need to figure out which other blockers do we have? So, Pipeline support. It was out that we don't have Pipeline available, which doesn't make sense. Then, yeah, in a gap to 211, we have some other bits, but they are effectively there effective here. Java 11 compatibility in weekly releases. So, what else do we have on the table? So, there is a lot of test automation. In the JEP, I said that we need to have ATH passing, to have PCT passing to release that. And I believe we still should stick to that. So, it's two test frameworks, which will likely require significant updates. The tricky thing there that we do not need these frameworks to build on Java 11. We need them to run on Java 11. But, yeah, our current steps do not support this distinction. So, probably making them to build on Java 11 would be a nice step. And, yeah, once we do that, probably we'll have the most of test automation in place to say that it's ready to release. So, there was another on the build plugin, Infra 1692, sort of at the bottom of your screen there. Yeah. So, plugin, do we want something that says if the plugin is asking to run a test against a version that doesn't support Java 11, exclude Java 11 from that matrix of tests. So, for instance, I won't wanna update the get client plugin to mandate that it must require Jenkins 1.2 or 2.143. But, I would like it to matrix test eventually against 2.1 whatever the next LTS is. Yeah, right. It's actually mentioned. It is. Okay. So, that's already described. I think it's exactly about that. Yeah, it's, or maybe not. I can put it in there. Yeah, it's just like Java 11 for test. Yeah, it's different. But, yeah, we have a task to support matrix exclusion or whatever because, yeah, it's a big issue for Java 11 testing now. Great. That was, thank you for that answer to that question. Yeah, I'll find this issue but it's somewhere in the list and you're most likely it's in the blocker list. Okay. So, I'll explicitly put it here as a blocker. I have no idea how to suppress geter notifications here. I always fail with that. Okay. What other blockers do we have? Documentation is probably okay. You also need to do agent packaging, which is not there. But, yeah, I hope it's a quick win because we tested remoting and it works fine on Java 11. But, yeah, I believe these are all blocking bits and then we have some minor things like big ability documentation for Java Web Start because there is no longer Java Web Start in JDK 11. And it means that general P agents will require web UI updates because, yeah, general P will not work at all. So, just to be clear on that one, and so you're saying that JNLP as a protocol won't operate or just that the convenient user interface that lets me use it for my web browser. Yeah, it's sort of a confusion. So, Java Web Start won't work. It means that, yeah, when you start the agent, when you create the agent, that is a cool launch agent button in the web interface. So, this button won't work. It does pretty much nothing right now. So, it allows you to download the general P file but it doesn't start on your machine because there is no executable, which is fine. It doesn't impact remoting protocols. So, remoting protocols are named general P due to historical reasons. They have nothing to do with general P at all on their own, so they won't be affected. Okay, so I don't lose the ability to execute a Windows agent using my old JNLP style, my old command line that would invoke slave.jar. That continues to work. It's just the user interface convenience of that button. Thank you. Yeah, right. So, all CLI things won't be affected. I'm a bit concerned about Windows agent installer because it injects some Java stuff but according to preliminary testing, it works well. So, I wouldn't bother much. And yeah, anyway, the current JEP presumes that when we landed it in the weekly, it has only experimental support of Java 11. So, we don't immediately say that first release with Java 11 support is fully jailed. We firstly start from something like preview but in a weekly and then we expand to GA once all bits are in place. Yeah. Okay, so this is the current plan. Yeah, one month ago, I was presenting to this JEP document at the SIG meeting. So, I wonder whether there is a feedback about the plan in general. It's still a draft. So, this JEP hasn't been accepted yet and we can make any edits you prefer. Yeah, and I'll submit some proposed minor edits but I reviewed it this morning. Oleg and the plan looked really solid to me. I had no complaints about it. There are some places where I may have some questions but they were minor compared to the overall. I especially like the Java 11 maintainers concepts on your screen right now that we will need a triage team. We need folks who are able to help when this goes out because we can expect a raft of a collection of reports that we hadn't detected ourselves. Yeah, I lost it. So, we did this maintenance team for JEP 200. It actually worked really well. I mean, taking a JEP 200 fallout scale, something like 100 plugins, having a maintenance team helped a lot. We've got good communit ratings and weekly release after weeks, I guess. So, yeah, I believe it's also something we should do, Kim. Any other comments about the JEP? This is great. Okay, were you expecting to say this is all terrible and horrible, Oleg? Because if so, you're talking the reality. I do not see Daniel on the call. Usually he's the one who says that. He's listening in. He's on the YouTube look, I think. Well, yeah, generally it's something risky, but yeah, it's something we need to do anyway. So, we should have Java 11 support at some point. And yeah, our main problem with this plan is that we shouldn't break Java 8 support, which is much more important than Java 11 and so on. So, yeah, any feedback from Daniel and all other folks will be appreciated. Now we just upstream smaller patches, so that changes get gradually into weekly release. But yeah, I think we really need to be careful. You'll see, for example, this patch, apparently we still had Java 7 support in our code base for reflection. Not sure why, so I removed that. I keep my fingers crossed that nobody really uses it somehow. Now I am curious, are there techniques that could help us other than read every line of code that could help us detect things like this? Because I found some Java 6 references and some code I was looking at once. Are there ways that we could detect things like this automatically? So, one of the useful things is just IDE. So, in Village IDE there is a number of Migration Helper scanners. So, for example, Jenkins Core, you can just click Analyze, inspect code and there is actually a number of detectors for Java 11 compatibility. And other compatibility layers, so you just run Analyzer and you get a number of issues. So, they start from Java 5 best practices and up to Java 11 best practices and some issues or something we could pick. So, it's one of the ways to start from that. At least it captures all the reflection and other things. And yeah, I believe it's something we should start first, but yeah, from what I see there is no static Analyzer tools which verify reflection and say whether it's correct or not in Java 11. At least IDE doesn't do it automatically. So, this stuff just failed in the tests. And actually, before it failed in the tests, I wasn't able to see anything wrong in manual tests if you're running. So, yeah, let's see. Okay, should we move on? Yes, yeah, we've got two other topics I think that we need to touch. Yeah, right, so we are sorry for taking so much time with Java 11, but yeah, I hope it summarizes the current status. Okay, I'll stop sharing the screen. So, we don't have Alex on the call actually. So, we cannot talk about Windows installers stuff, but yeah, probably we could switch to Docker packaging now, do with this. Are you ready? Okay, yeah, hi, Alex, yes. Okay, hi. Okay, so for the Docker hub, the Docker sync hub, right? So, we had one full request that was raised by J.M. Pritchard, right? So, where he used KMO for cross compiling the binaries and then pushing it to the Docker hub. So, we did see certain issues there. But the KMO released a new version for which fixed it for Alpine images, right, for the Alpine platform. So, with that, at least the cross compile version works well. I cross checked it with the S390X, I cross checked with Power Platform and they have acknowledged that it's working fine. I think it's the full request 719 against the Docker. Yeah, just a second, I'm looking for it. Yeah, do you see my string now? Ah, yes, the first one. Yeah, the first one, so multi-architecture stuff. Okay, so yeah. I did have a look at the changes and that looked fine. Though he's using the manifest tool, which was the previous version, but I think that's okay for now, but it's doing what it's supposed to do. It's cross compiling all the binaries for all the platform that's mentioned and basically solving a problem. Now, the only concern that I see with this approach is that if tomorrow there is some changes in the platform. So what happens is, and if the KMO team is not following that up pretty fast, then we might end up with breaking the functionality. So that's the reason I would still incline towards having a native support because that's more a safer approach than going for a cross-compiled version. Though cross-compiled works at this moment, but there are high chances that once we go with this and we have customers using, and then suddenly with the newer version, if KMO doesn't follow up with that and the functionality breaks then that could lead up to an embarrassment. So, and we cannot do anything unless the KMO team fixes it, right? So we could start with the native approach with whichever platform we want to start. So S390X and maybe I can also get the PPC64 that is the power team on board it. So that at least we have two platforms who can get their CI up and build those image and push it to the Docker Hub in the organization or the repo that you have created. So now the only question that I have here is how do the corresponding teams, right? Like Power and S390X team, whoever wants to build a Docker image and push it to the repo, how do they get notified, right? On what triggers, maybe a new release is created. So how will those notifications happen? Okay, so what, so Jenkins for, well, so here we have this repository. Here we have S390X and also multi-architecture images. So you should have access to both repositories. So you can just push the stuff here and then if you want to subscribe for events, the plan is to use webhooks. So for example, when Jenkins build passes, we can configure a webhook so that it gets propagated to whatever other destination and it triggers the event. And then you can run your build and deploy the build here. So it would allow to do Jenkins for evaluation or whatever. But the problem is this approach that it won't allow to do automatic weekly releases, for example. And it may be especially important in the case of security things. So it's fine as long as it's experimental support, but when it goes to the weekly, you will need to have something better. Right, so I like the question, right? So when you mentioned that we can have a webhook configured. So does the webhook move towards the GitHub or can it also point to some of the Jenkins job which is publicly available? So I would rather propose to have webhooks from Booker Hub for now. So yeah, we could send webhooks from trusted CI for example. So when the Docker build finishes, we can send an notification. But with the same result, we can configure it from Docker Hub. And from Docker Hub, you will get much more visibility on the Docker site. Okay, so this is good. So since we have now two repositories created one specific for S3.9.tech which we had initially created. And the other one of wherein we want to store images belonging to all platforms, right? So then the new one. Yeah, right. So we would go with the native approach, right? We would rather not push the K-Move based images onto this repositories. Is that the understanding with the group? Or do we want to have a repository where we push a K-Move based image as well so that whoever wants to try can experiment with them? I'm happy to create a repository for QoEmo. Right, yeah. What I can do is I can start with the approach for the native one. So that it can build for various platforms. With whatever hopes they have. But the problem here is that if we have multiple platforms, right? Say our and S3.9.tech, right? Say this are the two teams who wish to contribute and get their Docker images for GenK on the Docker Hub. Then at this moment, the way the approach that we are following is that we are setting a web book, right? And the web book will be specific to one of the platform. So S3.9.tech in this case. But if you want a common solution, then you would want a single CI or single some solution which could contact various agents for those platforms and then build the image. So how would we get to this solution, right? Especially if you're working on an experimental approach. Yeah, so for experimental, actually it can be set up on pretty much any Jenkins stuff. So you can just use the flow on your own. So you take a Jenkins instance, you build, you create a single web hook which triggers your pipeline and the way you build all architectures you want and then you build multi-architecture. But when we move it to official repository, we will need to connect a decent infrastructure to official instances. And maybe we have had a discussion with Olivia. So once it's configured, we could start thinking about it. Okay, perfect. So this is what I plan to do and maybe the team here can let me know if this might be the right approach. So what I'll do is I'll have one Jenkins instance on our side, okay? We could receive the notification from Docker Hub. Okay, and I will ask the power team to provide me an agent, right, which I'll configure on our Jenkins say as a slave. That whenever a notification comes in from the Docker Hub to our instance, I will build it on SC-98 as well as the power. And then the resultant images, I will create a manifest and push it to the Jenkins for eval repository. Yeah, and it also makes sense to rebuild. Yeah, then you create, I mean the... Manifest. Okay. Approach is similar to the KMU based, only thing is that the build is actually happening, the resultant images are actually natively built rather than using cross compilation. Yeah, right. And yeah, we can still use a cross compilation for other approaches. So I mean, if you want to support them multiple platforms, we just can use cross compilation steps here. Correct, yes. If tomorrow if we say that we want to build on a ARM platform, then any agent will just configure as a slave on the CI, which receives the notification and we should be all set. Yeah, it's something we can do. Yeah, com approach. Yeah, so if you prefer native builds for your platforms, it's something we should really do. For KMU, yeah, I believe that this pull request is still something we really need to get at least for evaluation. So I mean, a lot of work in here. I have another version of the same one which I have within our repository where I have just segregated certain methods so that it's more readable. And I have added also power support there, which is missing in that pull request 719. Okay. It's something I can help you with. Yes, sir. Yeah, we also need information from you about where to center webhooks. Correct, so I will work on getting the machine that is publicly available and I will get back to you like on this so that you can configure the system. Yeah, right, so there is no, so you want to create a public machine as a builder for that, right? Correct, yes, yes. Okay. So I'll have a public machine that can be notification from Docker Hub. It could be a Jenkins job as well, right? Which can be a receiving point. Yeah, it could be a Jenkins job. So we sent webhooks, you receive it, for example, on your site, for example, Docker Hub notification plugin. And then using this plugin, you trigger your meals and do whatever is needed in your pipeline. Okay, okay. So, yeah, so I'll just create one for KMO, as you mentioned on top. We can have all the approaches in front of the customer, right? So till we get on this solution. Mm-hmm. Okay, yeah, I can remove these ones. You mean that, right? Yeah, the third point that you have in your notification. I got confused. Okay, so does it look good to you or do you want me to do something else in the plan? Yeah, just create one KMO repository. Mm-hmm, yeah, it's actually, yeah. And what I will do is, I will send another version of the pull request, 719. I don't have access to his pull request, but I can definitely create one more, which includes the Jenkins file as well, which you can work with your infrastructure team to get that triggered, just as an experimental purpose. Mm-hmm, okay. So yeah, then I'll create, yeah, so it's Jenkins multi-arc. I think, yeah. Yeah, something like multi-arc KMO, and we define it there. Perfect, yeah. Okay, we'll still need to remember my credentials, I guess. Okay, so yeah, anything else we need to do now to help you with this topic? Ah, no, I think I'm good. And I'll start working on the need to one, and I'll pick up with you wherever you need to. That's cool, okay, thank you. So any other comments on this topic? If no, so we don't have Alex on the call, so I believe Windows installers sync up, it will stay till the next meeting. So I believe for now we mostly focus on Java 11, and yeah, good progress there. Any other unexpected topics to discuss? None for me, thanks very much Oleg for doing it. And congrats everybody, thanks for your contributions on Java 11. It really looks great. Lots to do, really proud of how far we've come though. Yeah, this hackathon approach apparently works really well, you need to do it more often, but after two hackathons, you've got much closer to Java 11 support than we were in May. But yeah, not quick enough to get it on the 10 by 25th is an announcement, but yeah, we never targeted it. Yeah, okay. So any other topics to discuss today? By the way, for the hackathon, I'm still writing a blog post, but eventually it will be there. Okay, then thanks everybody who participated in the call. Yeah, I'll stop the broadcast, and I believe that the next meeting will be maybe right before the VopSwall, JankSwall in Nice, maybe after, but it depends on whether we have topics to discuss. Thank you Oleg. Yeah, thanks everybody. Thank you very much Oleg, thank you, thank you.