 Hi everybody, welcome to the Jenkins inframitting. We have few things that we have to discuss now. So let's start with the infrastructure and infrastructure sponsoring. So few things up in there. First, from an Azure point of view, we try to reduce the build from, we try to be under 10K per month. Right now we should be working fine, so we were able to be below 8K for this month, so I hope we'll be able to stabilize the cost. But basically what we did is we deleted few non-critical services. We reduced the number of machines that we can provision and see at the Jenkins.io at the same time. So it should not really impact the Jenkins.io, but it will just allow us to better control how many machines we deploy. And so we just spread the loads. From an Azure point of view, at least one service that we would like to validate is the Evergreen database. Do we need to keep it? I mean, do we need to keep a backup basically, or can we just delete it? Maybe a leg or some opinions on this. Yeah, so what was our consensus? We can delete all services and resources, which we can recreate if needed. Yeah, it's a history of updates, basically essential plug-in sets. I think we can just delete it. Yeah, I just wanted to be sure because I was working today in the Terraform codes and now everything is ready to remove. So that's regarding Evergreen, otherwise regarding the Amazon sponsoring. So I had few meetings with Amazon last week and we are on good progress. So the content that we have at Amazon sent the requests. So I should receive a response within 10 days, but normally everything should be approved. Normally we should be sponsored up to 10K per month and for one year. So the contract will end in one year. So that's regarding the budgets. From a management point of view, the Amazon account that we are using is one account provided by ClubBees, which was a temporary account provided by ClubBees and it becomes a non-permanent temporary account. So we have to migrate from ClubBees to the CDF in order to allow more people to access it. So we discussed with Amazon and we realized that it would be faster to just approve the budget and then work on the migration of that account from ClubBees to CDF. So until then, it's mainly ClubBees folks who can access it as a first time. But yeah. Considering that we will have some money that we'll be able to use on Amazon, the plan will be, at least the plan will be to move as much as possible from Seattle Jenkins that I workload from Azure to Amazon. Yeah, I have to send an email about that in the coming days. I'm just waiting to have a confirmation that the budgets is approved and that I have that budget on the account before I start in working on the Amazon part. Do you have any questions until now? Nope. So let's continue. Yeah, I'm just looking at the Azure dashboard. And what just surprises me that the forecast, now it's about 7K, almost 8K. So I'm not sure what exactly happened to now infrastructure, but two times down being compared to the previous year. So basically we deleted few services. So the deprecated community cluster had been cost less than 2K per month. We had a testing, another testing community cluster, which also cost us 2K. So I deleted that cluster beginning of the week or end of last week. I don't remember. But yeah, that's one of the things. So basically I just review all the services and try to delete all the things that we don't use. Yep. So yeah, what I just mentioned that if our target for CDF is 10K, right now we are going to hit this target while doing nothing. Yes, so yes and no. The thing is right now we have to be below 10K, but you have the Amazon account that cost us 5K. We have the Azure account that cost us 8K right now, and we still have a machine in Rackspace. So we still have some work to do in order to reduce the infrastructure. But yeah, we are making progress. Basically, we have an agreement with the CDF. So the CDF approved to reimburse the budget up to 20K per month until April. So the goal is to reuse the cost by the end of April 2019. So while we are still discussing about the sponsoring, I'm still in contact with Fastly. So ideally I would like to be able to see if we can use Fastly to distribute packages. We discovered last week that apparently Tyler forgot that he signed a contract with Fastly sorry years ago. And so now we have to revoke that contract and create a new one. So yeah, I mean the procedure always falls at Fastly to find a good solution there. Normally the contract for there is one year and then it's renewed every month until. So yeah, that's regarding the infrastructure. Regarding the work that there was something in CI. There are a few things that were team help a lot. So he started working on creating virtual machine for Azure. So basically using Packer to create small virtual machine so we don't have to reinstall all the packages when we create a new machine. So almost everything is there. We have now to configure CI to use that and to generate those images. Considering that we will have some widget from Amazon. I'm not planning to spend a lot of time on CI to improve it. I want to be sure that first we have the widgets. How much money we have? Sorry, sorry. That's sorry about that. So yeah, so I'm not planning to spend a lot of time on CI to Jenkins that I own. The only work that is useful and team also started working on that, which is writing a configure config as codes email file is CI to Jenkins that I go configuration. The only thing is it did not configure Azure resources because right now we are not sure if we will keep using Azure and for how long it will use Azure. So that's the thing that you have to figure out. Otherwise, any question regarding CI to Jenkins that I own? Nope. So I have one, two, two last topic, which is package Jenkins that I own. This is the way we distribute packages. As you may notice, we had several issues in the coming the past days. Why normally as far as I can tell, there are some Apache workers that are leaking. So they are just there and do nothing. And then when someone tried to establish a connection with package Jenkins that I own, that person need to wait quite a lot of time before receiving the packages. And so we had to restart the Apache in order to everything to be back to normal. Why this happened every few months now it's thought to be happening every few days. So if we don't have, if we have, if we have the fastly account and then we can just distribute the packages from fastly, it would be perfect. Otherwise, we will have, we may have to reprioritize the work that I did with more bits to change the tools that we use to distribute packages. So that's something that you have to keep in mind. And otherwise, the last topic that I want to mention, we had a discussion regarding we use JIRA or GitHub issues for the Jenkins project and for the Jenkins hosting. So basically started with Tim just asking if we can use it for the hosting projects and it evolved to the, what should we do with JIRA. So there is nice threads about that on the mailing list. So I really invite you to have a look at tweets and to contribute. While I do not expect to have a clear final decision now. I'm just really happy that we started the discussion. So we have few months before we have to find an issues. But meanwhile, I ask Atlassian to renew the JIRA license. So this is something that I have to do in the coming days as well. I'm just waiting for KK to approve the access. That's pretty old for me. I don't want to have to want to talk about something specific. I see that Alex is around so. Yeah, so I just wanted to. Oh, Mark, go ahead. No, go ahead Alex, you have the floor. I was just gonna say, so I've been testing a Windows based ACI agent. So it requires less resources, requires less money on Azure. So it's a Windows Nano server image with, I've tested the JDK8 version. I haven't tested the JDK11 version, but it's up and running. It spins up fairly quickly, not as fast as the Linux Docker or the Linux ACI agents. But it's something that we could hopefully have people start testing to reduce the number of VMs that we need for the ci.jenkins.io. So Alex, can you describe how the, what the transformation would be there? The ACI transformation in the build plugin Jenkins file seemed pretty straightforward. Are you envisioning this would be similar, a similar kind of thing? Yeah, so right now you can actually enable this feature already. If you change your Jenkins file to pass in force ACI true as a parameter. It will pick up and try and use the, or it will use the both the Linux ACI agent and the Windows ACI agent for the builds. So people can start testing this now. I just did it in a branch for one of my plugins just to kind of test it and see how it was going. So obviously that was only one plugin and it was kind of a more simple plugin. So it'd be very helpful to get people to test it with some more intense, maybe to get series of plugins, make sure that the agent has all the necessary components for, or the Docker image has all the necessary components for building plugins. Is it something that we can enforce? So we can, for example, switch every jobs to using that. Yes. Yeah, we can change the build plugin groovy. So right now we're bypassing usage of ACI for plugin builds on Windows. We could go and change that so that ACI was the default on both Linux and Windows in the future. So it is something that we could enforce and kind of, you know, it wouldn't require everybody to go change their Jenkins file. Because, because, because to me, because to me, if it's working for you, let's, I mean, let's say, let's, let's wait one weeks and then just enforce it and see if people start complaining about that. There is one issue with it. So we have two frameworks we use across Jinx plugin ecosystem. One is Docker fixtures, which is part of Jenkins this harness. We have a container since some plugins. And we will need to provide opt out a way for those who use this components, or maybe we need to design build plugin a bit to run on the Docker specific integration tests on such agents. So we've been discussing some of these topics with Mark before, but I think that we could redesign the flow a bit to use ACI by default. But if you just make it dropping, you'll let it break a lot of plugins. So what, what, what, what, so basically what would be the next steps now regarding this? So what would be the best which one? I would definitely like some additional testing by some other plugin owners. Basically, I just created a branch in my plugin and changed that my Jenkins file to have force ACI that true. And I can send out the plug and I did that on it's not hard to figure out how to do, but I can send out that information. And then I just ran some builds, you know, making changes in that branch and make sure things were running and it launched the agent correctly and things like that. Thank you. So how's it, how's it different to the Linux force ACI with Docker? Is there a way to opt out? Right now you're opting to ACI and we will need to implement opt out. Right. So I'm ready to spend some time next week to implement it because I already started drafting a few pull requests for the plugin redesign. So I'm ready to spend some time next week to create a prototype and to test the windows ACI and parallel. Is there any other topic that people want to bring to discuss during this meeting? Jim and Jordan had topics related to additional agents. So Jim, you want to go ahead? Yeah, so I think a couple of you guys looked at my PR so far. But in my PR I made against the Jenkins Docker repository. It was kind of a rebuild of the build pipeline. So instead of having one script that kind of drives the building of these images, we switched to a kind of parallel builds, which is going to save time. And also one of the things I put in the mailing list I talked about is when security patches happen, you can just build that specific variant. So if Debian has a security issue, then through this pipeline I could just patch Debian. And there's a couple other improvements I made, but one of the big changes in terms of infrastructure is getting off the use of QMU, the emulator, and building on prem, but building on architecture or platform, whatever you want to call it. And I mentioned this to Mark a couple of times. IBM is willing to offer up some S390sZ resources and also power resources to you guys. There's a couple of different options to acquire those and pull into you guys' kind of infrastructure. One of the easiest probably for you, depending on how everything's set up, is we can just give you naked VMs where you guys can install agents or pull it in however you want. So that's basically one of the things needed for this PR to kind of go forwards. And obviously you guys have X86 resources. I think Mark mentioned, I think AWS has arm support for platform, I think now, to build. That's us. Okay, yeah. The problem that Armory is just one arm and if we need something specific, then it may not be enough because AWS offers on the Arm64 and if we need 32-bit platforms or whatever the bigger problem, who cares about 32-bits nowadays. Yeah, yeah, I guess Arm64 is the more like the newer Raspberry Pi's, right? The Pi4 and I think 32. Yeah, I know that is a major change for you guys. So I don't really know much about the infrastructure when diving kind of deep into the whole, you know, Jenkins repository and working with Alex and Mark and even you Oleg. But I don't really know anything about your infrastructure and how it's set up. So this is a big change and I love people to at least give me some feedback or to work with me on getting it going. And one thing to add really important is I didn't touch any of the main build pipeline scripts. So, you know, pushing to the official Jenkins repository you have. I didn't really touch any of the unofficial ones either. All these additions I made are basically just that additions. So it wouldn't affect anything you guys have currently. I think the path might change slightly for some of the experimental builds, but that's it. So first from the... So first regarding the infrastructure that you are willing to offer, I can definitely add that. As you saw, there is a release environment that I've been working on for the past few years and it's almost released. So I would be really happy to test that on that testing environment. So before... So I mean, we have a way to learn a little bit more about that. If we can work on a way to just the credentials there we can move forward. That would be really nice. And regarding the process, I think that the thing is I would prefer to work with you in order to bring that in the new release environment, but otherwise we can also configure the current. So basically what we have right now is we use the other Jenkins.io to test as much as we can. And then we have a second machine called truss.ci which is hidden inside Amazon networks. And so that machine is reconfigured with all the secrets. So if we... And that's the place where we also generate the Docker images. So either I work with you to configure that machine to use your resources. Yeah, but something that you have to... Maybe we can discuss together after the meeting. That would be great. It might take a week or two to get the resources, but it should be fairly easy to get power and S390 for you guys. We have a couple of public offerings in terms of, I think, Oregon State University hosts their Jenkins-filled pipeline. But since you guys' pipeline is kind of complex in terms of what you guys are doing, probably just a native VM would probably be better in everything for you guys. At least, my understanding, right? Is that what you guys want? Yeah, to me, that's a pro-SIMS webinar so we can already learn more about SIMS in practice. Awesome. Yeah, I can work on that with you and getting that kind of going, at least giving you the resources and then we can take it from there. Perfect. Any other topic? I have a question about mailing list. So, I'm trying to finalize the Jenkins-ary Meetup cleanup story. Plus, today we had a discussion about documentation, SIG mailing list. Plus, before, we had discussions about whatever board-related mailing list. So, I had a question, what would be the best way to proceed with all these stories? Because they're quite isolated from the rest of the infrastructure. But we really have insights about who owns mailing list, who can manage them, who can... So, let me check one thing. I have a few passwords for the mailing list. I'm just checking which one I have. No, but Olivier, I thought Oleg's question was not about the mailman list, it was about the Google groups, right? Oh. Yeah, I still have mailman lists in my kill list. Just to clarify context for others, we still run our own mail server. And since we moved infrastructure mailing list and all other mailing lists, we could get rid of the service quickly. So, just one thing. So, we do not manage the mailman service. That service is provided by OCSL. So, we have a few mailing lists there with one admin password. And so, we started moving away from mailman in order to use Google groups. So, for all mailman, basically, the only thing that I need to do is just switch them in read-only. The only one that I did not switch in read-only immediately, which is the Jenkins here for our mailing list, because I just want to be sure that people can finish the threads that we started there. So, I'm going to switch it in read-only. Regarding the Google group mailing list, it's a little bit more complicated because everybody can create a Google groups. There is no central identity right now. So, for example, I created the Jenkins in front mailing list, but I'm an owner, I think, on a dev mailing list as well. But the thing is, it's always who created the mailing list first. And then we have to find the right person to do the work there. So, for example, the doc mailing list, the doc mailing list, I have no access to that one. And the same for Jenkins, seven meetups, right? More than probably, yes. Let me verify this. So, for a special interest group mailing list, we have a policy as a part of JEP4 that the intra-officer must be added to the admin list. Okay. So, we can at least prevent it from being in the admin list. But fault once is the problem. My guess would be, my guess would be just to look at who sent the first email on the mailing list, who replied the first email on that mailing list and then contact those person. That's a good theory. Now, essentially, my permissions don't allow to really check who's owner in some mailing list. One other question I had. As I know, the platform SIG is a glitter chat. And I know you guys recently were just talking about the mailing list moved over to Google Groups. You guys have a glitter chat for the infrastructure team? Or are you guys just using IRC? We are using IRC. Okay, sweet. And we have no plan to move to Gitter or something else. So, why IRC is not perfect? Yeah. At the moment, that's what you are using. So, Joakim, regarding the mailing list, if you can try to find that and who sent the first email, otherwise, I can also have a look. To me, I think I only have access, so that's what I'm looking at. I only have access to the mailing list and the inframing list. Okay. Yeah, I figured out the owner for a few mailing lists. So, I'm going to call it for documentation through this DPI. And apparently, it's Taler who owns the mailing list. I'll call it the same approach to mailing lists I know about. Okay. I guess we will still have an issue with company CLI and for donations mailing lists, but it's something for me and Alex and other board members to figure out there. Okay. The way we manage the Google groups, it's quite confusing because everybody can create a Google group's mailing list. It's not really who has access to what and be sure. Maybe it's something we need to figure out. So, if we just agree on a policy that we want to have an infra-officer as an owner at least as a corner, it might help. I think having an infra-officer is a good idea, but we also have to be sure that the person, I mean, the closest officer to a specific mailing list also have access. So, for example, on the dog mailing list, I don't want to have privilege access on everything, especially if I'm not using those, but yeah, it makes sense to me to have access to those mailing lists. So. Okay. I'll try to deliver on that. If we don't have any other topic, I propose to stop the meeting here and continue offline on RSE or whatever. Jim, I will contact you for the resources right after the meeting. Thanks everybody for your time. Thanks. Thank you. Have a good day. Thank you.