 All right, welcome to the Jenkins Evergreen open planning meeting I'm going to go ahead and share my screen so we can look through some some tickets and stuff Yeah, we're so close So close to something that's really really Really really cool and interesting. I'm looking forward to getting this mouse on one wrapped up so over the past and A week or two there's been a lot of testing that's already been done and But by myself Bobby Sandel Michael Neil there's a few folks from Cloudbees who Were kind enough to try this out fairly early on Which means we've got some good some good tickets and bugs that have all come in The first big thing that I actually wanted to talk about and this is more with them Something I've been waiting for you to get back from vacation to discuss Batiste is the concept of user installation of plugins Which I know is something that you were you were certain was gonna be a problem Right away As luck turns out It was When I saw your comment I was about to you know run run around in the room shouting to everybody, you know They run around saying I was right No, anyway the point my main point was not that we should not ever do that but that I think it was maybe a bit Yeah early to do everything, you know to open to open the pipes Yes, yeah, I agree there there there may be So we've got these two tickets me pull open the other one too These two tickets that Bobby had had filed after after given this this a try and Pretty much the behavior that Batiste was worried about turned out to be true that You know adding plugins that we haven't tested that are outside of this Package of plugins that we distribute will destabilize the system Which is exactly what happened So I assigned these two tickets to me and I think for now for it for you know this milestone one I think it's It's reasonable to say that we're gonna disable the installation of plugins and stick with the The included list for now, which means we probably need to open up that that inclusion list to include some Other useful plugins. Yeah, right Yeah, my point is that we should try and close down the list But yeah have some kind of process even if it might be still be a bit early for that But you know some some core things that we do know that they are going to be really useful even for core use cases But for you know things that are not clearly Core then have some kind of way to say, okay, we are going to do that later or maybe much much later or unlikely a letter But yes closing things in the mid in the yeah kind of a beta beta I don't know to find the right word, but you know beta stage Period To have this because right now for for people watching We could say that even probably with a handful of instances It could already become very very easily and quickly overwhelming the number of Warnings we receive over a century which by the way works fine We already started receiving a lot of things so I think if we open to more people and less, you know Experts and and friends which are going to Report things the right way It might yeah backfire So yeah, I wanted to I don't know if you've looked at this Mandy I Wanted to to bring up the century it looks like we've gotten Large number of warnings, which is fun These I've been seeing in my In my system logs, I didn't realize that they actually went up to century But we do have the the feedback loop here and as Batiste was doing earlier today There's some improvements that we can add to century to make it a little bit easier to triage and to with these tickets Yes You've you've used that you've sent three before Tyler right, right? So if I understand correctly on the right the column users should show I guess the number of people different people who hit that kind of warning Is that this is somehow the the meaning of that column? That is assuming that we're sending that information over We're not because I reported something this morning and already filed the PR to to enrich that It's already been merged. Oh, right. I didn't see that for that. Oh my god Wait, so but but then the issue is that we got back to your earlier point I think this week or last week about the fact that we are going to need pretty soon ish more than soon ish Be able to update the client, which we do not right now. So that brings me to the second topic Of my god traditions Yeah, so let me segue here so with the with the plugins that there's some behavior that I think I can easily restrict and I plan on restricting from this This engine X work that I've been doing So this is a pattern that I experimented with a project called code valet last year which worked out really well But basically because our container, you know, as soon as I finish up this work Our container will include an engine X proxy in front of Jenkins then instead of needing to hack that Evergreen plugin to restrict some of this behavior, which is perhaps the put in quote right way to do this We can easily black hole some routes like the tool installers or the plugin pages And just use engine X to redirect those to some built-in documentation that says here's Here's why you can't access this page and here's what you should do about it. If you really if you really care So I'm gonna take on that as part of as a follow-on to this work this work being the providing feedback on the booting process to users pull requests that I have open right now that I am anticipating fixing today and then Once that's done then the next sort of project is something Mandy and I chatted a little bit about last week Which is upgrading not just evergreen client, but upgrading what I've been referring to is the evergreen support system so if we look at go to the distribution code so we've got client and config Which we definitely are going to need to be iterating upon At least in some fashion and with the changes that I am incorporating now We will also have some static pages and documentation which we need to upgrade Or distribute in the same way in the same sort of pattern that we distribute the Jenkins distribution right now And Absolutely because actually last week, I think as I was Adding the sheen somehow in the scripts directory. I were thinking kind of the same I was thinking okay, right because well I was fixing fixing some issue that was reported by the way by by Bobby You know the fact that we changed the admin password each time and I was thinking okay, right next We can't update that because it's So we are going to have people to you know ask people to Just to a container their image they pull a new one and so on so right now We basically would need to be able to get anything in there in the thing or as I commented I think this morning in some issue Maybe for some cases for for simply simple this reasons and so on We should ask people, you know on a regular basis or something to actually get out of the normal update process and Talk a pool and occur run a new version of the container or of the image What do you do you have some thoughts about that? So I do I think We're going to have to do the docker pull the docker container update anyways Because there's going to be You know Java package updates that have to happen or node security updates That have to happen exactly Those are going to be very very infrequent though. I would expect those to only happen maybe once every few months if that But the the jingan support stuff we'll definitely want to upgrade more Frequently than that. So I think we have to do both Yeah, I think Thank you the the simple case for now is You know, we're going to be improving evergreen clients and some of the configuration is code stuff as Configuration is code matures to where we just we will want to push that in the same way as we're pushing today the plugins and other parts I Think if we if we if we view I was having a conversation with Daniel yesterday He categorized my my viewpoint as slightly patronizing towards users Which I agree it is a little bit patronizing, but I don't think we can expect all users to be updating on a regular basis and So if we assume that most users will not update with any frequency Then the things that are really important to make Jenkins evergreen work I think we'll want to put that into our update distribution system to make sure that it's to our users. Yeah and there so we somehow similar to that I Think at some point and probably rather through sooner than later. We need to have some more involved testing Framework because right now we don't test a number of cases like for instance updating the Vending the container for instance the the image, you know that has been built in the middle and so you would you know restart over the volume and so on because I and I am talking about that because I I saw something from Jesse somewhere saying that he lost he's built built records and I did see on my instance doing exactly that pool and so on that I lost my first four spills So I didn't take time until now to dig into it But well there would be some cases we definitely do not test right now and we need something bigger and something more involved to think bigger than the just you know the door thing I set up in the in the beginning and now You Yeah, if you haven't filed a ticket for that we file just a task to implement that Upgrade test I Think yeah, we've got enough in in Docker hub to where if we just pulled the latest Container and then ran that with a volume and then tested our thing our new container with the volume That could be that would be definitely hopeful but I wanted to I wanted to Chat a little bit with Mandy So she can come off mute about some of the Updating of the clients or the pattern that we had discussed a little bit yesterday I'm wondering if you've given that any more Sorry yesterday. We discussed that Friday. There's your Friday Yeah, I'm wondering if you've given any more thought to how the upgrade process for the Distribution should work The support distribution rather right. I mean yeah, it's because basically I'm at the point with the script where we need to Pan it down specifically before I can finish it okay, um The short answer is okay, so we were planning I think a new section in the Status response, right the update level. Yeah, and then with that my plan for that was to We would download the file The client itself would have to handle the file renaming which will need to come up with a consistent scheme for that Probably just renaming with the version numbers would be the simplest And then it'll be up to the startup script that the supervisor D um The client will have to kill itself Which will be fun and then the startup script will handle Assuming that the files are already in place It will handle starting up and if it doesn't work then I will look for a rollback file Replace this file with the rollback file and try to start up again Now if that still doesn't work, I'm not sure how to handle that Do we want it to keep rollback or at some point is it going to go? I don't know what to do until it's hands up Hmm I think what we can what we can probably do is Like we'll have we'll have a status page after my work is done here in this pull request and So we'll have some means of saying to the user like look things are really really screwed up right now You should just download the container anew and try again What I'm thinking like as I'm as I'm talking through this right now what I'm realizing is that The update level process that we have in place right now Assumes that the client is going to start Or sorry that the you know the system is going to start with Only with the latest update level right like the the evergreen client comes online and then it reaches out to You know evergreen dot Jenkins IO and gets the latest if it doesn't have anything locally But we would have something baked in It's gonna be interesting I mean that was one of the points that I brought up in our conversation where if the containers kind of old And you're starting it up. You could be jumping ahead quite a few versions for the initial update hmm, I Think with the support system that might be okay Batiste do you have any thoughts on that? Not ready to be honest. Um, okay That's not something I gave enough thoughts up to now So the big reason Mandy for background the big reason we're very conservative with the Jenkins plugin updates is that and and we want to walk update level by update level is because the As well stop sharing my screen for now um, is because the Jenkins plugins by and large can have Data incompatibilities between big version jumps to where they might write data to disk for version one And then if you go directly to version two you may miss a An update which actually does it data migration That has happened before but I think for the evergreen support distribution it might be We don't really write a lot of state to disk We only write the keys and the latest update level to disk We might be able to get away with a You know a big jump ahead for the support distribution Yeah, I think we had agreed to just kind of go with it for now And if we ever hit that we're in the future we can revisit retooling it Yeah Okay, well, I think Since Batiste probably needs some time to a to mentally process Some of this since we hadn't really discussed this before and he's still shaking off that vacation I think What I'll do today Is my meeting load is looking quick promising. I'll get a pull request that creates the Tarball package that should have everything everything that we need to apply those updates For the support distribution, I'll create a pull request with that and then maybe that'll give you something you can start from Mandy and we can tweak it as we move along The stuff that we were anxiously waiting for you to get back from vacation for is all of this AWS stuff and some of the configuration is code stuff Yes Think I Don't I don't recall how far along the AWS Cloud formation stuff is but for me that was one of the big blockers for a blog post on Jenkins I owe saying that milestone one was ready is having a You know a launch stack button. You're right that a user could like to launch it in AWS Well, there is There's quite a few things in the in the pipe right now which needs fixing I guess before be able to jump on to that but I like for instance securing the instance that like we Those of the of that with off-skip and so and I think some some of those things for instance are probably more urgent than than that button But then I Or maybe in parallel I could be probably able to ask some friends and colleagues because I think we already did that in the past I mean we as company or Or Relatives Because yes, so as for people watching we do have already the cloud formation Templates, but we do not really I do not really know how to make it, you know Seemlessly usable through some click or something which I never tried and so my initial thought about that was that we would need to set up some kind of Marketplace thing thingy and you were saying that you thought it would be You thought it would be feasible without needing that and that was a state of discussion and The middle with it's a few other other things but yes to Make it clickable and I guess it would in your mind. It was something that would love you in Into your, you know WS account and blah blah blah Well, let me Wonder if I'm logged in to Azure right now. I'm not Logging to Azure real quick the the behavior that We have on Azure is that you can Oops if I could spell my username correctly that would help Very important Actually an interesting Interesting security problem Microsoft. You're not supposed to disclose whether an account exists or not So let me get this open And I'll show you how this works with Azure which I think cloud formation does an identical version Okay Yo, and share my screen And That is not in here Know that but he's I seem to remember that thing So what we have right now, this is a This is a button similar to the launch stack button that I've seen for cloud formation and what this does and this deploy to Azure button is actually references like it passes a URL parameter to The Microsoft portal With a URL to the JSON Arm template, which is the Azure resource manager template, which is the logical equivalent of cloud formation And when actually say they used to do that. Let me go to the quick start. I forgot they changed that It's This will make it more clear than extra templates so Let's just Take a simple one. Let's do ACS Kubernetes. Of course that doesn't have a button. There we go Mm-hmm. Yeah kind of lug you in and then it's going I would expect it was going to be Asking you the parameters and yeah, that's right And I'm almost certain that AWS has a similar behavior with cloud formation templates likely I guess To where I was thinking if our cloud formation template was in our GitHub repository that the launch stack button Would be able to point to that Possibly I can and at least have a look at that. I Just never had the time to dig into it and I thought just kind of yeah thought that It would get a require to be marketplace and it would offer some some kind of thing like this, which I Because so for Azure it's really different and there's also a marketplace and those are two different things, right? Correct, right. Okay, so I guess I guess it should be the same I don't think I don't think we'll need anything in the marketplace. Which would be good Um Yeah, as you mentioned, I actually just went in ahead and reordered some of these like that some of these bugs are definitely a priority what I'm For me the prerequisite to the Jenkins IO blog post The prerequisites sorry this plural We need the secure defaults configured like these three tickets or sorry these two two bugs should be Should be resolved Mandy and I should have the support distribution update resolved and Then I should have the status page the built-in status page and information also resolved Shouldn't you file Jenkins at IO blog ticket and marked, you know those tickets as It's just a good guy something like it will help everyone track what is blocking there the the noise announcements Yeah, those are those are to me the sort of like mission critical Really important tickets to address before we can sure sort of open the floodgates if you will by the way Yeah, I wonder if we should make a pass. I think now many things I generally put in my stone one And if I'm not sure I'm going to tackle it myself because I'm sure it's important And so on I'd really assign it to you so that you can just you know decide as the PM Decide if we should just defer it to my stone to or You you have too many caps that's that's why Yeah, so last week I did go through and I actually pruned a lot of the stuff in milestone one I have to move Raul's tickets out. I don't know if Raul is going to come back and help or not But I think that we will have Most of the stuff I've been moving into milestone two that are sort of nice to house and I've started to triage what's in milestone two and what's redundant and Sort of where we would go from there Yeah, in my previous position we had some kind of special milestone not a really nice one But we call one was called the fridge and one was called the freezer and When things were in the freezer for long generally we would close then I won't fix because you know Unlikely to come ever to ever come back and anyway, we can just we open it really comes back, but yeah Try and free your try and free your mind. I'm guessing that you were using pivotal tracker No, we were using that thing that were acquired was acquired by CA Technologies Which in turn got acquired recently, but I don't remember the name. It was kind of a one of a major agile tool I can't contri again Competing to competing with many G rise both you know in different area anyway diverging so with them With the tasks that we have open are there anything like for you Mandy Are there any things that you need other than the the support distribution layout that we discussed already? No, I'm working through the tickets that that teeth sent to me. So I will reach out if I have any questions on those Okay embattis same for you No, I think that's good. That's okay. And anyway, I'd generally use a Synchronism and I ping you over the different giras. I need your feedback on or I think you should be aware of so well the The thing with asynchronous is it only works if we're both around the same time but if you're a If you're sending me stuff early early in the morning when I'm asleep, then of course I won't see it No, but I mean if if it's a rich era you are if if you're saying that You get too many notifications from Jira to get ever able to to process them then I can proceed Proceed differently. No, that's what I mean Okay, fair enough. Then yeah, I get plenty of I get all the stuff for the evergreen project in Jira So yeah, keep filing stuff that should work Okay All right, so let's get any questions before we move forward I guess that's that's fine Thank you and see you soon. Yeah. Bye. Bye. See you both back in Gitter