 Hello everyone, welcome to the platform sync weekly meeting. We're going to make the usual tour over the last events in the project. So today we want probably to cover at least a Java 11 status report because if you didn't follow, we announced the Java 11 preview availability in late December. So we had a check, there's no outstanding report I would say, one from Mark which we are likely going to cover a bit I think. Anything else you have in mind, Mark? So I put a topic on there for Devin to talk about the workflow support plugin. It's in the workflow support plugin next release item, the one that Devin's here and he can discuss that. And the Docker build topics there. We also had ATH and PCT on the list that I wasn't sure if they should stay in. Baptiste, do you want to talk to those or is there nothing changed since two weeks ago? Either is fine. There's a few smallish things like there's one I remember because I was the one working on that but to a really small one PCT now displaced the JVM version when it's being reported. So it's easy to actually double check which JVM version was in use for even test. But it's small one. I think, yeah, there was the world already a few improvements here and there are not yet. There's a one that is actually now in review or so from Ramon about making things work better. So I'm not going to open it anyway. So that would be the one. And so yeah, there's a nice announcement for the workflow support plugin release next release. I mean, because there's something was fixed by Devin. So I'm going to, by the way, probably give you just a stage Devin to discuss that. But basically one of the outstanding issue and an MPE that was surfacing got fixed by a version basically. Devin, can you explain a bit maybe? Yeah, sure. So I guess the high level overview is that pipelines were failing after restart 100% of the time for any pipeline whatsoever. Even the most absolutely basic pipeline was unable to restart. And the root cause of it turned out to be that J Boss Marshalling somewhere along the line had a regression that made them no longer compliant with the Java serialization specification. And so in certain circumstances with certain class and object hierarchies parent class constructors wouldn't be called at deserialization time. But it turned out that they had actually fixed this later on. So we just needed to bump the version to pick up the fix that they had already done for it. And so once we picked up that fix, things looked good. And so we should be ready to release that probably today and push it out for everybody. I guess the kind of the tricky piece of this is that it also means that because restarts were never working, that functionality has never been tested. So we've never tested the long tail of functionality that is somewhat specific to the after restart behavior. So for example, like Mark raised an issue that probably doesn't have anything to do with workflow support itself, but might be some other issue that was just, you know, we never could have seen before because this would have always broke first. And so just to clarify, make sure it was, you mean it was never working on Java 11 or is this larger because it's actually related to marshaling to the door or something. Yeah, this had nothing to do specifically with Java 11 or Java 8. JBus marshaling to the door would have been broken for Java 8 and Java 11. Right, but given we don't use JBus marshaling to the decks on Java 8 that that's an issue we did that didn't surface then right. Yeah, I mean, so we never released this to non experimental Java 11 users. Yeah. Yeah. For people watching, indeed, we are using JBus marshaling 1.x right now, still because it's not JBus. The workflow support plugin still uses it in non experimental versions right now. It's very likely though that will release it for regular users as well rather than having a split branch for Java 11 and Java 8 because there's nothing in particular here that makes sense for Java 8. But then even for Java 8 that I understand because for for again for people watching and for even anyone watching it afterwards or right now in the call. I think this bump to from from JBus marshaling from 1.x to 2.x makes it impossible to restart a live pipeline even on Java 8 right. Yeah, I mean, so the in general updating JBus marshaling they make no guarantees that there's binary compatibility in the serialization format from version to version. So every time you upgrade JBus marshaling it generally means that you cannot live restart pipelines across that upgrade. So when we release this we'll have to add a warning that says basically, you know, this time you need to make sure every pipeline you're running is completely finished and stopped. Yeah. For you continue. That's a reasonable one shot issue and that's a good also very good news because then it means that we can just indeed as you were saying as I understand push that out as soon as possible in the main line so that people would kind of hit that possible issue as soon as possible before Java 11 becomes a wide list GA general availability and then that would be an issue that people that have already gone through that update wouldn't see just because of Java 11. Yeah. Okay, I'm not sure about East I thought you were advocating a more aggressive approach than I would have wanted. It feels like this is one where the j the upgrade to JBus marshaling to potentially could break many people and I'm I'm hesitant to do that on the production code until we had significant testing to verify it. Could you clarify further what I think this is not really what I was saying I was trying to rephrase and I think that's Dave that's what what David was saying isn't it. Yeah, so I mean, basically 100% of the time guaranteed there are no workarounds when you update JBus marshaling, you will not be able to run pipelines across that restart without I mean, maybe we could try some real heroics and like reverse engineer the binary format but at the end of the day we're just going to have to accept that that's not ever going to be a compatible upgrade. And so we are not aware like this change in isolation for Java eight users is not particularly interesting other than that things aren't going to work across the upgrade for Java 11 users it's absolutely necessary. So from pipeline teams perspective, we'd like to do this update anyway, so we'll probably just push it out for everybody at once. And then by these points kind of once we move this past us we're not going to we don't foresee updating JBus marshaling again anytime soon, except in the case of regressions we basically never touch it. Okay, so I've captured in the note when in the notes when we update update JBus marshaling from one to two pipeline restart won't work in the first restart after the plugin upgrade did I say that correctly. Yeah, yeah, specifically any pipelines running across that restart will fail at the restart pipeline running across the restart. And it's not necessarily specific to the from to the one to two jump even minor updates of JBus marshaling would cause this. So this is kind of like an update that we never really do because of that problem. Okay, so it's not tied to a specific version. It's that they don't promise binary compatibility from any version to any version. Yeah, I mean in some cases it does work. Okay, we've skipped like seven or eight releases with the sub bump. All right, so they now work in the first restart and we know that they don't work in the upgrade from upgrade to the pre-release version that's used for Java 11 right now. Yep. And so as you say, given this JBus marshaling 2.x bump is required for supporting Java 11 in the future, you're saying the pipeline team is strongly considering doing that bump. I mean, like right now even for Java 8 users, because that is something we won't kind of have a look at or like deploy as soon as possible, even the fact that well, that's what one shot issue should be likely rather sooner than later behind us or something like this. Yeah, and I mean, you know, we don't want to be maintaining multiple long live branches with significant differences between them. Yeah, there's also that indeed. Yeah, right. Because as we are approaching Java 11 GA, yeah, let's let's try to avoid multiple branching, right. Sorry, Mark, I think I screwed up your type. Great. No problem. Thanks for helping. Well, I was thanks for trying. In the end, I didn't. I created more issues than helping anyway. Yeah, thanks. Sorry. So what was the next thing on the... So that that thing, I think we covered a pretty, pretty lengthy. So that's probably okay. Anyone has any question about that issue? Any concern or whatever? Yeah, there's one question. I'm wondering, is there any way how to get the pipelines that didn't survive the restore to die in sort of an easy to identify way? Not necessarily trivial. They all fail with some kind of error. One of the causes will be failed to load pipeline, failed to deserialize pipeline. But the root causes will be generally weird things like stream corrupted or things like that. I think indeed some was hinted about some things about that some time ago. Maybe I'm wrong, but couldn't you, Devin, like, catch the fact that it's being restarted? Try to identify the version of marshalling and, you know, at least issue warning or something. I'm not sure it's feasible or it's really worth the effort, but yeah. Yeah, I mean, we certainly could try something like that, catching the specific exception, logging it. I don't know. I guess I'm going to talk to the team and see if they think my inclination is just to add an incompatibility warning for the next release and explain it in the release notes. Right. You mean, yeah, like in the, in community warning only, only the release notes, you mean, or like in the plug-in metadata so that it actually triggers a warning while updating? Yeah, in the metadata. I mean, that way at least people will have a chance. You know, I assume anybody who is clicking through those warnings is going to be okay if things break. Yeah, right. If there's a big fact, big fat red warning and like, okay, please make sure there's nothing running or you're treating that one shot of grade, then the issue will be gone. Makes sense. Yeah, I mean, that's a good point. We can think about what it would take to... So what would be your, what would you request, David, or something? Maybe, could maybe Oliver file a Gira or something about that? Or how do you would like maybe to try to be able to track that? Or is this enough like this? Yeah, I mean, if you want to file a Gira, we can. Yeah, the main thing is just that, from my point of view, add some complexity that, you know, only matters one time, so I don't know. Yeah, and not even in old cases because not everyone is going to push an upgrade while pipelines are running, maybe? I don't know. But yeah, certainly if you want to open a Gira, by all means, I mean, I'll mention it to the team as well, see if anybody has opinions. Great. Oliver, are you okay with that action item? Why should you open a Gira to help us track that? Yeah, I will take it. Thank you Oliver. Yeah, nice point. Thank you for writing it. Anything else? Not for me, thank you. Great, thank you. So I think we could move, just feel anyone free to interrupt, but I think we are going to move to the next item. So it was one about Mark fixing the Docker bill, which is still something that's open for review. And by the way, I think I screwed up because it's not where that thing should be. It's more the notes, so here, right? So that's the one here, sorry, this way maybe. So that was the PR. So basically, Mark, maybe you want to explain, I'm not going to steal your PR. Super, yeah, so the Docker image that will be open JDK, 11 project chooses to deliver, delivers an operating system that's based on Debian SID. SID is the Debian distribution that is always unstable, always. And it never releases. I think they made it a disastrously poor choice. I'm clear, not clear why they made it, and I didn't find any indications why, but I think we should switch and base ourselves on a stable version of an operating system, not an unstable one. SID has no security team. I love Debian, but Debian SID is not a place to do production work. And they tell us it's not a place to do production work. So the proposal is switch to the upcoming release of Buster. That's one alternative. Right now, if we don't make this switch, we have a broken build on the master branch of the Docker images because the JDK 11 build is failing. So we've got to do something about it, whether this pull request or another pull request. Okay, that's cool because now you're explaining that that's what you've written, but now you're saying that by voice. I had missed the fact that I thought it was all open JDK that was based on SID. So you're saying only the JDK 11 branch is based on SID. Correct. Only JDK 11, they only made that choice. And during JDK 11 development, that's not a bad choice. But now that JDK 11 is production, having a Docker image based on an unstable operating system is at stake. Yeah, it makes sense. So definitely, and I would say probably there's an action we should do on our side to kind of close the loop is to actually look for or file an issue on the associated tracker somewhere. I'm not sure which one, probably Docker one, Docker Inc one, probably to report that issue or make sure there's one existing, explaining that that's probably not something desired or something. You see what I mean? Like making sure because open JDK, given the fact it's an official image, probably as a tracker, you know, it's maintained and followed by the Docker Inc team, I think. And most likely there is a tracker somewhere to report issues. I think we should actually have an action to make sure this is known about so that we actually know if there's some rationale behind that or something. And I think we could we should just at the same time move away from it because indeed you're right. We should move to Buster or whatever. But yeah. Thanks. I've recorded the action item for me to raise an issue on that image. I'm passionate enough about it that I'll I'll fight that. Thank you. Thank you. Yeah, I think it's kind of making sure to close the loop, making sure because it's always in, I think, useful to understand the Russian or whatever. If there's none, maybe this we are going to be the first one to raise the issue, maybe, but yeah, we kind of contribute upstream or so. But there was one more action from that I in view, which I'm not sure who has it, which is who decides if the pull request is allowed into the Docker image or not. I'm not sure who has permission to say yes, that's OK. I have or like has and we are a few people. Probably you mean it's it's it's as as any kind of thing we have on the Jenkins project. It's like maintainer and time based. So if you're interesting, probably you could become also a maintainer basically and about the image choice and everything. I think I think it's kind of I think we discussed that some time ago or so about the fact that there was that issue about supporting or not Alpine, you remember. So basically right now for for people watching, there's an issue, an ongoing issue with Alpine because Alpine didn't provide yet a JDK 11 version. And it's unlikely to be fixed soon because building JDK 11 is a mess and it depends on pretty big things in the leapsie. And you know, leapsie is not what's used in Alpine. It's new leapsie or something. So it's a Musil, another library, which is supposedly mimicking, but actually not everything. So it's it's kind of complicated. And given the fact that open JDK release cadence is accelerating. It's not really likely that it that is issue is going to be fixed for good because then we will have JDK 12, 13 and everything. So there's nothing decided yet, but indeed there discussion to maybe unsupported pile or whatever from the Jenkins project standpoint, because it's maybe not really sustainable. Even the open JDK itself, they can't provide an Alpine version. We are not going to do that better than they do on Java itself. So my point is that maybe at some point we should actually have a JEP that describes the rationale. The thing I'm talking about right now basically plus many, many things about, okay, the supported platforms, the reason why and you know, that we could then iterate on an explanation and everything about what we support. And probably there. So that's kind of my point. That's why that long explanations come from is that then there we would explain why we chose, for instance, Buster or why we didn't chose chose something specific for whatever reason. See what I mean? Does it make at least a bit of sense? I think so. So is the action item then, is the action item then that we should do a JEP for a discussion about which Docker images we support? Help me clarify that. I'm happy to do that. Yeah. You're right. So to create, to make something concrete out of the library, I just put out basically, yeah. This kind of still thinking right now, but I don't know what others think. But yeah, probably we should at some point have a JEP trying to describe the platforms we support and we put out. Because right now it's kind of, you know, a de facto support and de facto de support, I would say, because for instance, we cannot. So should we just stop doing it? And it's not great. So yeah, I guess doing something long term, you know, more. Explained or whatever would make much sense in a JEP. Okay. So, so one of the topics that was raised there and so a JEP is a great place for a conversation. Oliver, this is one I think that that you may be a very interested party in one of the suggestions that you have. To me was, should we consider switching to or adding a centos or red hat based docker image into this for as part of the Java 11 exercise. So I think we would discuss that in the context of the JEP. Absolutely. So Debian. So Debian. So Debian. So Debian. So Debian. So Debian. So Debian. So Debian. So Debian. So Debian. So Debian. So Debian. So Debian. So Debian. And that that's definitely a JEP that could be handled overall. I mean, is there some, if someone needed to write the, the core of it, but it's kind of a, definitely, I think a platform sick concern to try and define platforms we do support, actually. So, yeah. and then Durgobus would want probably some some indication on S390. And also changing the default recommended platform for instance right now we've been discussing a lot that you know DBM or Ubuntu might not be the right place nowadays to do Java development given the crap right recently you know about the DBM back porting that that happened to have forgotten some part of the upstream patches or something that was that Ubuntu that you discovered recently mark about the fact that on Ubuntu if you installed JDK 10 I think or something no it's the contrary if you installed Java 11 on Ubuntu right now it installs Java 10 right correct so I'm wondering if really there was some some explanations on on the internet recently people were saying well seems like unfortunately DBM based platform seems to lack the necessary empathy and like expertise maybe with regard to make actually make sure the platform is you know nice to JVM based applications. Yeah so it feels like a JEP is a good place to consider that I'm happy to open that JEP. Yeah I think and anyway the JEP process is a good thing because anyway as I remember it we would need to create a thread on the dev list to discuss you know to try and trigger discussions trigger thoughts so as part of the JEP process so anyway I mean having feedbacks and trying to make the ball move forward kind of make some sense. Okay I think what do you others think what do you think Oliver for instance? Well I have no no strong opinion there's probably one thing I do not understand are we as a platform sick motivated to actually support different containers based on different Linux distributions I mean I would understand having some container supporting Java either Java 11 but is there any you know consumers that actually care for distribution we based on? Well that's a good point I think we should probably support not a lot of different platforms if that is what you are asking but what I'm saying is that maybe for instance we should think about making I don't know CentOS or whatever the default instead of DBM given the recent issues strong issues that happened recently in the DBM based platforms for instance and so there's that other thing about Alpine which makes us consider okay should we actually because right now we do deliver an Alpine based flavor of Jenkins Docker images so can we should we keep doing that for instance and that that's uncertain I think. Right yeah sorry I probably misunderstood the thing so in I understand the situation as I see it in long-term it is convenient for us to switch from distribution to distribution to work around the you know limitations in one world and that's perfectly fine but in order for us to keep doing this in the future it would actually be beneficial if we do not commit in using a particular distribution right so we just say okay here's some Linux with Jenkins in it but nobody should really depend whether this is Alpine or DBM or water vials we would have to switch to in next year. Yeah I mean that that's I think that that's actually I mean I'm not going to because I think I've done that too much already but I think I shouldn't be trying to solutionizing actually but I think your points are really important and would definitely be interestingly covered I think by a discussion about a JEP that would try to define why and what we support actually because that's what we kind of the goal of it in it so anyway I think there's there's what you're saying Oliver shows I think shows that there actually needs to be discussions about how why what we do support do we support something I mean is this just a best effort is this some kind of some kind of you know minimum support and so yeah yeah yeah interesting there's a lot of to be had I think for let's not solve it let's say something for the for the JEP exactly but yeah it's interesting because it shows that there's a lot of questions maybe more questions right now than answers what time is it okay right we have supposedly at least 14 15 minutes left maximum not say we should consume it all if need as always but yeah so are we good in terms of images I think we are probably going to move forward soonish on the java 11 front because indeed that's not great that we support something and anyway right now it's it's broken so we need to fix it but that's kind of fine to move quicker because that's an experimental image for now but yeah indeed it creates probably the what we just discussed the discussion and this is the need for digging into the long-term support when this JEP comes and everything to actually document and discuss with the community what we are going to be selecting as a base image or whatever for the future okay let's let's move over to next point given we have a few things to cover okay this is just a copy paste of the last time so not sure we're going to cover everything but Mark you think we should cover the other items I don't have anything to add on ATH or on PCT if you do that's great the exploratory testing is just to invite anybody who has capacity please help with the exploratory testing of java 11 Devin's highlight on that there is a large area of unexplored territory with the j-boss jboss marshalling to upgrade is an important thing to note it'll help us all a bunch if we get more exploratory testing and bug reports that come from it yeah I've been monitoring uh since the last two weeks or something the new issues that got creating under that that by the way there are that issues filter is public don't java 11 non-triaged so I didn't see anything filed by by by some external contributor by external I would say someone different like that mark or people already well known from the community side so if there's anyone finding issues outside or you know we are really really welcome to file issues but right now I didn't see anything surface maybe I miss it or maybe there are these there these missies the right labels but yeah those for the main the main issue right now we know about is the fact is the one filed by mark and it's already looked being looked into by Devin which is a slightly different one as I understand it than the what about the NP that got fixed recently which is a more random one but we don't know about anything else that that's really so that really surfaced since the preview announcement basically right that's that's correct yeah it's we've got we've got one in one example this one looks like it's relatively challenging it's not the java 11 experience I've had has been overall positive this is the only exception this safe restart but all the other things it's behaving quite well I'm really pleased with how my exploratory testing is running I'm prepping all my git plug-in 4.0 beta prep release work is being done on a java 11 instance and it it feels surprisingly smooth great I'm very pleased with it but I think we need more bodies than just just the one or two of us testing yeah yeah but that's really great because you're one of the rare contributors to also run their workload and your tests on many platforms like Windows which is not that common I think so that's great because if you're saying that you don't see issues even on you know Windows Linux and many other flavors that that's already really cool given given the the high level of usage of the git plugins so yeah that's great great so let's see what we would be covering I think we covered everything the rest is just a copy paste of last time so I'm not going to go through it if nobody asked something specific to cover I think I was actually going to just delete those the blocks correct is there anything that you wanted to report to us as new information on PCT about this not really I think as I said there was a really trivial improvement so I don't think that's really something people should be bothered about during that meeting anyone can have a look at for PCT for instance there was a few I think also on ATH we fixed since last time I think there was a lot of effort to actually fix GT score but that was something about ATH and so we ended up fixing it so now the core is better but that's not really related to the platform thing partly because of ATH now being used you know is all the time for Jenkins core PRs so that's something that that now gets normally better I think I didn't have a look like in the last two days or something now in contrary to two weeks ago or something everything is almost everything is green now sometimes there's the random failure because it's much much rarer rarer so that that's great it before it was like this so now it's great so that's an improvement that we achieved some time ago thanks mainly to Raman and yeah and that's it I think let's just cut the meeting short if we have covered everything I'm not going to contribute the time just for the sake of it anything else anyone before we go we had an action item for Oleg on jep 217 I assume nothing to report there with him being unavailable jep 21 I don't remember what was that one he was oh yeah that was the one about the developers list about making progress on jep 217 yeah right so that's that's so that we can explain maybe quickly so that one is pretty easy basically we are going to have an automated publishing of images from the because right now images official images I publish from what we call the trusted infrastructure which very very few people have access to for for obvious security reasons but for making it easier to test experimental releases we are going to we have already started to publish things to the experimental organization on docker hub and it's actually publishable from ci.genkins.io contrary to official builds but I mean yeah that that's not something I'm going to dig into me too much right now because I don't think that that's something really needed as a sync up call and anyone interested could dig into it I guess and it's explained well enough in in the in that jep by the way I think we are good we covered that so yeah thanks for raising it Mark anything else anyone nothing else from me great so thanks for coming thanks for watching and see you is this just checking next week same time I think or is this every two weeks yeah right right right right indeed so that's the next time would be just checking yeah right the 17th right thank you everyone and see you next time years