 All right, welcome to the Jenkins Essentials Open Planning meeting. I apologize for my lack of video and the noise Unfortunately, there's no better place for me to do a meeting right now So Batiste, I'm gonna ask you to drive this a little bit and I'll type in Yes So welcome everybody, so I'm gonna just share my screen right away so that we have The place to look at By the way, welcome Mandy. So it's she's showing up by Amanda on the left She is working going to work with us and as a back-end service developer And so right away switching to what I was working on and I will Give the mic to Mandy slightly later on I think so last week Was mostly at the Eclipse conference where I gave a talk about essentials, so I hope the recording will be posted soon Because I think it's kind of a product vision Higher-level talk not really technical so it kind of summarizes the intent of the project and so on so It would be valuable for for people to wanting to discover what that product project is about That video is posted. I think it would be great. I can do it or you can do it But to write a blog post with an update on our status. Yeah, sure And I actually pinged Through Twitter yesterday the eclipse con foundation and eclipse foundation and so on for now and no answer yet I hope it will be posted soon so that I can then indeed find something of luck and tree and reference it from there, right? So what I've switched to Since mostly and also trying to help Amanda Why should call Mandy on the project is the LWS support So it's kind of the sibling issue of the docker Cloud flavor we support since roughly two weeks So now we are trying to see you how we are going to set up a class Agents right away from LWS so The status about that is that it's just getting started for tomorrow morning. I'm going to have a Work with someone who knows ECS much more than me so that we optimize the timing and See if we're able to spawn agents from there and see then what are the possible caveats Basically, we were just discussing that earlier with Tyler. There are likely two three maybe four possibilities Which would be so the legacy original EC to plug in Then ECS, which I thought might be slightly better for the cluster aspect and the I think out of the box of the scaling and those kind of things, you know providing Docker containers, I never tried it, but I suspect it should be providing Provisioning agents quicker than you know, proving a brand a full-blown VM on EC to and so as Taylor was saying There's also EKS solution, which I think for now is probably Too much. So yeah, that's bottom line tomorrow. We're trying ECS and see how it behaves Then even if I separated it from that ticket We are going to very soon just once it's done to look at auto configuring or so when it's using the AWS flavor of Essentials, you know auto configuring to use the external artifact storage to store things in F3 as was developed some weeks ago by the by the architecture team. So Oleg, JC and Carlos And I guess that's it for me for for the last week and it was a problem Some background for for Mandy and for anybody else that watches the video One of the reasons that we're looking at doing this work on AWS Well, there's two reasons for the work on AWS. One is we want to take advantage of the stuff But Jesse and the rest of them have built but we also want to configure Jenkins out of the box a bit more opinionated and that includes disabling all executors on the master instance, which means we have to provide some means of doing Doing work on the Jenkins instance. So we have to configure something to provision agent Ciber as containers or vms And that's not something that Jenkins has really done out of the box before So this is a little bit Uncharted territory as far as starting up the Jenkins with The very very beginning with zero masters and automatically having some other agent provider configured So I think it's gonna be cool work Yeah, so There's a system built into Jenkins core that allows you to define a cloud which is sort of at this point I would consider a kind of the legacy support for Actual clouds, but basically it says if you're if you're please go ahead and start allocating some new agents out of Some pool that you have available. I think in the longer term We would like to take advantage of stuff that Nikola for example is experimenting with whereby we can Directly make a connection To a fresh cloud machine using whatever service we have available In the course of a build without it having to go through the Jenkins queue, but that's all the plugins that Patissas for the services in the plugins that are supporting those services that Matisse are using are using that cloud API Exactly, so I'm using for the Docker cloud what we call flavor the Docker plugin Which basically indeed it laments the cloud API for Docker and I'm using kind of the same for ECS and there's another plug-in called easy to plug-in Which is doing exactly the same for provisioning From easy to and so yes, and so Jesse just maybe for others and so for myself I think the main issue as I understand what you were saying about the cloud API for instance is that It's necessarily because of the fact it's going through the queue and then waits for it when it goes to fast and so on some kind of Latency which we can really reduce, you know to zero Before some some agents starts being provisioned, right? That would be kind of the original design issues with this Yeah, I mean the the cloud API was sort of transitional system that sort of took the existing model of static agents and tried to Make it work with an elastic system But basically a response to increasing pressure by allocating more agents and then after a while it can The cloud implement case the implementation can drop some of them but I mean what we really want to be moving towards is that You know, there's no There's no general pool of agents attached to Jenkins floating there. We you know, we want a particular build to Go and as part of the build process Attached to a fresh machine do whatever build steps it needs to do and then throw out that machine The moment that build completes or even in the case of pipeline the moment that node block completes really Yeah, and I suppose have some ways of pre-provisioning things so that the business starts right away and so on Perhaps I mean that there is some work done. I know by Google to Create a warmed up jar cache so that you can start up a Remoting agent with all of the plug-in jars that needs already present Statically in the image before it launches. You don't have to transfer code In the normal case over the channel. So there's various optimizations you can make Okay, I did I did want to mention one thing that's pretty critical for adding AWS agents, which is that you need to think about I am roles or whatever your security scheme is so in the case of the S3 plug-in we made a decision that We expect the the master to have Some sort of AWS credentials that could those could be implicitly via I am Or we're also working on a GUI that would let you configure particular credentials and Jenkins if you prefer But either way what's important is that whatever agent computers you provision should not have any Special permissions at least S3 Yeah, okay, they should be considered untrusted Yeah, I remember the JEP about that Yeah, so it's it was supposed to be pushed from the master so that it just have some some temporary Thing I remember something I used to download and maybe not to upload or something and go through the master Yeah, I remember something about that in the in the JEP about artifact storage, right? Yeah, so it's using pre-signed URLs So the master is solely responsible the master has the only keys to actually access S3 The agent is just given a Provisional lease to push a particular artifact or download a particular artifact. Okay Yeah, indeed the IAM is going to be slightly more tricky than than just using An access an access key and its password as I already somehow discovered a bit earlier today, but Anyway, so let's just go on Enough speaking already more than 10 minutes and let's switch to others So let's keep the better for the end. I guess so let's reach to Tyler and will Have switched to Mandy. I think afterwards. So Tyler. What's the status today? You want to hear your voice again with the noise behind? the This poor advice is killing me The The work that I've been doing is just getting the computed update manifest to the clients and As far as I can tell at this point This morning everything looks correct from a system standpoint, but the smoke tests are still failing Though I'm very happy Patisse that he did the work to put those smoke tests into place Well, it takes me forever to run those Actually quite helpful Integration of client server and Jenkins running So this this before request that's related to this computing update manifest It's gotten way too long, but there's no way that I can use this apart and it includes Generation of ingest.json, and that's just sort of the Puted list from an essential.yaml of all of the versions of plugins that we would need in their dependencies That's enough to get us moving right now. There's some hard-coded overrides for Stuff that's in the incremental repository But we'll need to find it at all later points some way to say get this in the essentials.yaml and Incrementals repository out, you know a little bit closer together to generate that ingest.yaml But from that the server is only giving the client what updates it needs so what that means from a Dataflow standpoint is every time we create a new update level and this is defined in the JEP 307 All of the dependencies are listed there every single You know artifact that's part of that update level is listed But when the client contacts us We're going to have the latest versions of software on that client And then we do a signature comparison and we only give the client what it needs So we're not redownloading the entire world every time we have an update level so I I'm reluctant to say that this works because the tests are still filling but it looks like it's working And I'm not sure why the tests are going right now Which is why we've been sort of locked up on getting that that production environment under Evergreen dot Jenkins IO Deployed it's because this needs I need to land this first and at least like that at least you made a huge step like 30 minutes ago or less Because now that we can see on the logs of that our client in DCI that Jenkins does start So now there's something wrong with the client handling which is receiving non-json response from the update service as it says But we can see that Jenkins now starts when it wasn't before. That's something I've got to figure out a better way to handle this when We're turning a 304 not modified Either express or feathers is not allowing you to define the body And I think it's just inserting its own body So it's the client is requesting a constant type of application JSON and the center is saying Now I'm going to give you an error message instead And then the client just turns to parse that So I reduced that to a warning because it's it's non-fatal And it's it's something that doesn't need to be fixed, but it's not preventing the client Correctly Yeah, that's that's the Once that is done, then then I'm on to deploying Evergreen Yeah, let's table that for later. I had a question about that, but let's just do that later So groups that's great So, yeah, I will have a look and so I'm not sure by the way, I can really help on that part But seems like now what Jenkins is starting So great Let's just keep that for the chat and later discussions that now switch to Mandy And I'm not forgetting anyone else. So as I was saying keeping the better for the end So Mandy welcome and thanks So we discussed a bit earlier that you were going to Got getting started on that ticket, which is about aligning the code about The JEP 308 making sure things are aligned with the spec. So how are things are going for you, Mandy? I'm digging my way to the code base right now and I've already got a test written that is currently failing So I am looking at fixing the code to make sure it actually returns the 413 Great and don't hesitate by the way I mean, we generally try to push PR that are ready, but you can definitely or so in that very case I guess push, you know work in progress PR so that we can provide feedback and get a get you more, you know help even more in a synchronous way sometimes I mean pushing things in a in a explicitly Tagged label as such PR so that we can you know provide what we think are failing and you know, you can use the Yourself the github Comments feature to say, okay, it's failing at that line and I was and why and so and so forth. So yeah Great and so I was saying for others I just told earlier Mandy that it would I think it would be easier and anyway That's kind of what we're trying to do in general to just file a specific PR for only the first case About you know making sure when we're setting too large data It's which is that it's to be something instead of just you know making a big PR addressing everything with regard to aligning to the CHEP It would be making the review easier and overall anyway. I mean it makes sense. It's not it's those things are convenient or merged separately Yeah, don't don't make four requests like me Somehow Every time I turn over a new rock whenever I'm working on a full request. There's more stuff that's sort of deeply related This um, you'll find this on the back of Mandy that there's a lot of There's a lot of stuff that was kind of stubbed out originally and Unstubbing some of those things out once you get out of error telemetry Becomes a little bit more involved than this small little ticket that it might look like But that's what I that's kind of what I expect from a new ish system anyways Yes, so you have a template to to put that that line on each of your PR, right? This this poor request got a big bigger than what you did too, but Anyway, kidding. So I think we are most I mean, I think we are that's a good thing I think we're all aligned to the fact that we should keep PRs easier and smaller Because we all know that it makes reviews and everything Orders of magnitude easier and quicker But we also know that sometimes it's just impossible. So that's that's right. I mean So I think we are mostly done anyone Has anything to add I think Yeah, we don't have roll Jesse. I'm kind of curious We had that issue with with incrementals the publisher last week that I recall fixing Has incrementals started to get more broadly adopted across the plug-in base Being used a bit more from pipeline code. I also Seen that it's being used some from the Java 10 hackathon so that people can Have a I think all I was working on setting up Docker images using incremental builds of plugins that we could have a Ready-to-run image containing a complete snapshot of stuff That had various known fixes in it So I have certain changes I need to make the most important one is that I want to change the calculation of the integer At the front of the number But that's I think that should be a Barely limited technical change. So I'll propose it as soon as I get a moment Now I they would say probably the reverse if there's Yeah, I don't have time to work full-time on anything related to the what do you call it the Ingest manifest or something But if I can advise or help out on How that could be formatted or how that could process Inputs and tie-in can tie-in to incrementals or there's some other Technical stuff with maven and it's over. I've just reached out. I honestly think I might be able to just give them the model That I've got in this pull request right now I'm just grabbing the update center as the latest source of truth, and I'm not actually caring about any versions I can see a very clear Sorry Yes We can hear you or maybe lower the Quality or I don't know so that's it. It reduces depend with available needed anyway, I think Tyler you were saying that what you already put in the PR is already reflecting Reflecting what you think should be done something like this Pretty big picture of what you think should be should be done Parsing the update center and so on. Anyway, let's just stop guessing Yeah, so I think it's writing so it's Okay, exactly. So thanks everybody. I think we are done and we're in time five minutes left That's also awesome So, thank you everybody and so see you next week same time and If you need anything or you're interested to join the project, so I just remind that it's happening Mostly this could most discussions are happening here on on Gitter so associated to the repo and That's it Thank you everybody and see you later. Bye. Bye