 Okay, so to share a concert or maybe mark you can share the notes. Well, you say you can't share you should be able to. Huh. Okay, I can certainly share them know if that'll help. Wait, weird. Here comes the shared screen. Oh, I can't share. Okay, we'll let you drive that. Do you see my screen? No. Yes. We can see the GitHub. GitHub.com in dark mode. So you only see GitHub.com. Now we see the notes. Change seconds ago. So first few announcements. Is it recording? Yes. Okay, so I think we have to cut that first part of the video. So welcome everybody to this new Jenkins infrastructure meeting. We have a few announcements on the sponsoring detail. The first one is Amazon is going to renew the sponsoring. So last year they gave us $60,000 to cover the CIA infrastructure. And we got to recently got confirmation that they will renew that sponsoring which is really great. We just have some technical. We have some documentation that we have to solve because what happened last year is sort of current situation for the Amazon account that we are using. It's officially owned by cloud bees. And we have to transfer that account to the CDF. That's that was a longstanding projects. And what happened last year is when when the sponsoring sponsoring arrived, it was it appears in cloud bees. It was a longstanding information. And this is something that we want to avoid this time. So we are currently investigating if it's faster to transfer the account to the CDF. If it's faster to transfer the account to an independent organization, or any other solution. But yeah, at least, at least the good thing is the cost will be covered from a second sponsoring standpoint. Anyway, we'll offer some money. If my if, if I remember correctly, it will be 600 Europe months. So this is also some, some compute that will be able to use for the CI environment. So in specific areas we may have some some some work to do because the that that region would be a little bit further than east to west coast. So we may probably use that money for a specific CI environment. We still have to to identify which one. So if you want to help with scale wave, feel free to reach out. And we can see how to use that money. The call is obviously to reduce, it's always to reduce the cost of dashed accounts, and to reduce the dependency on one specific cloud vendor. That's something that we try to do. So for a long term that's really nice. Any inputs, Damien, you want to bring here. No, I just want to check if it's monthly. I understand the same as you, but I just want to be sure it's not the one shot 600 euros because then that would be really cheap. Yeah, definitely. But anyway, I'm looking forward to work on that that specific area. I'm before the next points. So if I go back to the agenda, we the first thing that I want to mention as well was the puppet agent issues we had so we recently discovered a week ago that the certificates and the puppet infrastructure part two months ago. And so the puppet agent stopped working several months two months ago at the same time. So that certificate was configured to work for five years. And we had not no monitoring in place. So what we had to do, which is a little bit cumbersome was to first renew the certificate on the puppet master. And now we have to go on every agent to renew the certificate so the agent can connect with the master. We did it for few machine trusted CI archives that CI, and we now have to continue bringing back everything on track. In the process we identified that some changes were not correctly apply this mainly this mainly affected Damien SSH keys that were not deployed on some machines. So he needed me to do to run the agent. So that's still a work in progress any question on this. No sounds perfect so that then let's continue. The next topic so we covered I was sponsoring we covered the scale was sponsoring. So we have now the next topic to the agenda is agent on our CI environments. So Damien I think you want to bring this as you as you're the one who been working on that. So the initial goal, and it's still the scope of the initial chance it's to update regularly the operating system packages and tools inside the agents. Right now see agent inside you and all other instances that we use are using images from December, which mean we need to upgrade the contents the operating system mainly. The main challenge we have, and we discovered is that by updating we have the latest tools, specifics per cloud on AWS, it's named SSM or easy to launcher they provide services that you instance used to connect to the web services gets the instance data gets the private key prints to the console etc. And at least for the easy to plug in it create a big issue. It's that you have to wait for the instance, not only to be started started and ready to be connected to better to start all the easy to services sometimes it reboots. It, it's resize the file system it does a bunch of stuff before being able to print the fingerprint of the machine on the console, which mean a machine that start that is ready to use in one, eventually two minutes might need eight to 10 minutes before being able to launch an agent. The main reason is because we use, we try to use as much as possible a strong security that say, even the initial SSH connection is checked against the fingerprint that come from the cloud console, just to be sure there is no man in the middle attack during the initial connection. After the initial connection there is no risk because we always checked afterwards but that's the first one. The question is, oh, is it a real threat, the man in the middle for an initial SSH connection in our context, because we build the API, these are not outside API. And most of the time it's inside a private network however that's still a risk. I'm not good enough to evaluate the potential of this one, but we have that challenge of do we want fast instances, or do we want really safe instances during the connection. Right now the idea is to try to keep the slowest instances, that's the proposal, I prefer staying safe, but that create challenges depending on the combination of operating system and cloud provider. So I'm not sure that's enough details for this, but if you're interested or have an advice on that, don't hesitate to ping me on the IRC. The second point is for the Linux instances, IRM and AMD, the goal is to use Docker as an unprivileged process, which means the route inside the container won't be the root of the virtual machine. The goal is to strengthen the bits on that part, which means we can totally target on Linux to use an unprivileged user for the Jenkins agent. Which means Jenkins agent is able to run container, but the Jenkins agent even with container is not able to own or to be a root. That's the goal. Windows for Windows though we don't have the unprivileged Docker container engine, it's not possible. So that's why the initial reason is to keep the administrator user. The second reason is also because I haven't found a way to create a user on Windows, that's as simple as that. You can create a user, but you don't have a home, you don't have a set of predefined settings, predefined keys. So on the end, the user must be unprivileged. So that means no need to create an unprivileged user and add too much code to maintain or prefer removing the code and get started on that. So with these experiences that has been documented on the repository, the goal is to have the same set of images that should be updated at least monthly and deployed automatically on all the instances. So we will have the same behavior between for releases, for external contributor on CI and for us in the info. So that should be end of the month for that set of tasks. Thanks, I mean, I would just just add something quite important while you're looking at the windows. Agents, we still have an issue and the release environment where we had to use a specific version of the SSH plugin. So yeah, if you have enough time and feel confident, I mean that's if it's remain in your. The thing that you're working at the moment, maybe you could also look at it, maybe that's something easy to fix. Good pointer. And you have a question. So, yeah, Mark. Sorry, Damien, you said that the goal is to have a similar configuration between agents in those various environments that I capture that correctly in the notes. CI and trusted and release and info now cert is certainly one that Daniel cares about intensely and release is one that the release team cares about intensely. I don't see a problem with what you're proposing but I assume we've got conversations with others. So that they know hey where can we would like to unify our handling of agents across all the all the CI servers. So that's something to specify here that a release environment rely totally on communities. So we use Windows pods for the Windows parts. So Windows containers and we use Linux containers running on communities. That's one thing. The other thing is the work that Damien is doing on CI touching kids that are you that's where we spend most of the money. So having fast windows agents and stable windows agent is pretty important for CI for the CI touching kids that are you because that's our biggest instance. Obviously, once we have a good process we can definitely apply the same configuration to serve that CI to trust that CI as well that's what we did in the past, the previous situation that what we had before Damien started working. Even I would say one or two years ago, we were just configuring everything from Jenkins so when Jenkins would start an agent it would install Docker will install the user, and we would just run a script for Windows agents. And then we started creating custom Windows machines using Packer. This is something that Alex started working on a year ago, maybe two years ago. And we realized that for instance on trusted CI we were still using the previous configuration and that's the challenge of having configuration that we modify manually is that sometime we have a fix happening on one instance and it's not replicated on the other instances. And so the work that Damien is doing is to have small agent a small I mean templates that we can reuse across different CI instances. And this bring to the next topic which is having configuration as code for CI touching kids that I owe and the idea would be to start using to cast to configure the cloud agents. But not changing everything in one time but just working on one area at a time. And I think it would be really good to have that because the cloud configuration is quite big and when we have to go to the web interface it's quite difficult to know when we change something and we just having a configuration that we can commit and that people can review would be definitely easier. Obviously once we have, I mean that's the same that happened on release that CI and infrared CI once you have a configuration as code file, you can just copy paste the same configuration for a different instance and that's really easy to use. And the main difference is CI the Jenkins that you enter so that CI are using puppets to be deployed. And we're not planning to change that in. And I mean, in a closed I mean, sooner. So we will still use puppets, even for me probably for coming here at least. Any question on that topic. I proposed to move to the last one. So I had discussion about using discourse in the Jenkins project so for those who are not familiar with this course. It's a tool that you it's similar to a forum that's a tool where you can have people discussing we can group the peak regroup topics. It's a very nice way to organize the various areas in the Jenkins project. Oh leg initially came to me asking if we could use this course for the user for the Jenkins user. So we could use it on top of the user mailing list. I think it would also be great to use it for dev and for the infrastructure because this idea of having yet to split the information into topic would be really helpful. I've been looking in the past to discourse to deploy it manually. That was not a great experience. I mean, I don't want to go down that path. So I would like to look at either if we can have some sponsoring from discourse or maybe pay for the service. Right now, when we start the trial we have 14 days to test it. So if there are people interested to look at them and look at that with me once we have enough people to test this course, I would like to just run a small experiments. If you need any help, don't hesitate to ask me because I've did exactly the same for the traffic community. I was in charge of testing, evaluating, deploying and maintaining it during the first year of existence. So what was your experience there? I think that's a subject outside this but it was overall pretty good. I liked it and you have a way to personalize, customize and that the conclusion was the same. Maintaining it ourselves is complicated. So if you have to do it, just be prepared to have way more things. And it was working really smoothly on their web service and never had any incident on one and a half. That's quite nice and you can tune a lot of things. There are a lot of plugins, integrations, customization on the look and feel, so that's a nice service, really overall positive experience. I think that would be also nice to look at the sponsoring options because I know that the CO is sponsoring the Jenkins project. And the LFX community bonding project. Yeah, so we had sponsorship from this course. Whatever the name of the company, it's a bit complicated. But yeah, it may be good, quite significant sponsorship. I believe that we could get sponsorship for managed service. So I've been working with a few other communities recently, and for example, one of the communities open source design. They use sponsored version of this course from the vendor company, so they don't manage this course on their own, they just get it as a managed service. They don't have such kinds of sponsorships. Yeah, they imply potential risks. Sponsorship continues, etc, like we experienced these other services, but at the same time you wouldn't need to manage it on our own. So for experiment, it seems to be like a good approach. And then we can discuss whether you would like to host this course on our own or presently managed service. So as a board member, I offer my help if you need some whatever collaboration with the vendor company. But yeah, technically it's totally possible and I believe that if we ask, there's good probability that you would be able to get sponsorship. Otherwise, the cost is 1200 per year. So it's definitely not one. Yeah, definitely. So yeah, we can potentially even pay for this discourse from even our current infrastructure budget. I wouldn't say that it's the desired approach. So this court vendor donated $2,500 to us. So potentially covers the cost for two years of managed service. Just by that, and we haven't allocated this donation anywhere. We are pretty good candidates to them. This should be a pretty good candidate to the open source program for the record with traffic. They evaluated that traffic even though the source code is open source traffic is only maintained by one company, which is continuous. So then we were not candidate. They refused and we had to pay for that. But two years ago at least they said Jenkins was an example of an open source project. So they said that's very well known. There are company different people from different interests. They considered us at least three years ago now as a candidate. So that should be a good, still positive thing. I don't know with the experience, but don't get me wrong. Maybe they will say the same and they will ask us to pay because the business behind or something. They can always find this. Let's ask. Thanks for the inputs. So we cover all the points that we had that we have for the agenda today. So if you have any last topic you want to bring in discuss here. Otherwise, we can close the meeting. Well, so more FYI than anything but the weekly release has been temporarily delayed and intentionally delayed by a day so that we can get a little bit more time to evaluate a possible change. If it was in the release channel, it's nothing it in for needs to do about it just be aware, don't enable the weekly build process at the moment, let Tim Jacob continue managing that. I've been especially, especially for that I've been considering disabling the management of the release instance. We're experimenting the file. Because the risk that we have here is if for some reason we restart Jenkins instance, we, we, we remove the luck and the job. Well, the product is ready over so. In this case, in this case that's fine because that's a crunch up and so we are over the date anyway, but I've been thinking doing the same for trusted CI so that would be pretty simple if my mouse want to go back. So it would just be trusted CI. No, no, sorry, I didn't want to say trusted. Okay. I'm not sure if an authenticated. Just in case you're still sharing. Yeah, I know that I know that's what I want to do. So if we go to cluster. Here, if we just comments, let's say in the release environment where is that Jenkins release. If we just comment this line. The job will not be managed by infrared CI. So the, the, the, the, that application won't be managed so we can temporarily disable any modification done to that that service. And I think this is something that we should do in the future. Yes, Marcus seems a bit. I don't object it just didn't this particular condition I think is handled well enough by the job being disabled and as Daniel noted the cron job. But if you think that would be a good safeguard that's great as well I just would hate to have lost the management of that because we forgot to enable it again after this period is done. We have two jobs that need to be disabled here, we have the release job on release the CI, but we also have the infra management job that manage the communities cluster. So if for some reason we update Jenkins if for some reason. Yeah, if there is an update on Jenkins, and we accidentally merged that change. The communities will be updated and obviously we restart the Jenkins instance so we restart really the CI so that's why I was suggesting to just comment this nine so we are sure that we don't we don't update the service accidentally. So in this case for this weekly that's fine I'm just thinking for future security release or stable release. I just wanted to mention this is mostly relevant for security updates, because we skip the weekly in the week of the security update. And that has happened I think once or twice now that I disabled a job. Free for five days later someone merges something Jenkins restarts and the job is enabled again. Exactly that that's why that that's why I was thinking to this. And yeah, if we just put a comment in that email file. I think that should be part of the process. Okay. Any other topic. Then I propose to finish the meeting here. Thank you for your time. We already put hang on Olivier. Quick update. Okay, so we did some progress on AWS sponsorship. So we still to be communicating to public. But we we shared we showed that to pick at the beginning of the meeting. Okay, I was late. Okay, but no other. I just shared that we have agreement from Amazon to renew the sponsor the sponsoring, we are just trying to figure out the best way to the best location for the Amazon accounts. Yes, we have written so currently the account is owned by called these we had some issues related to that in the previous years. So the idea that we would start a new account and we move for Jenkins resources there and then we would connect sponsorships so that sponsorship is dedicated on the to the Jenkins account. And after that, the company will be looking for putting transfer ownership of this account to the continued labor foundation so that it's officially under the roof of CF like Microsoft accounts. So that's the plan. No timeline for that. So that's the initiative the first steps on both the sponsorship. One of the major benefits to transfer that accounts to the CDF would be to invite non clubbies employees there, because right now only clubbies employees can have access to the Amazon account, which is a blocker. So that key contributors who had access to Azure to the account really help to unblock situation. And right now, we have strong dependencies on on. I mean, the number of people who have access to that account increased, because we have Mark we have a leg we have Damien we have me, but I mean a year ago I was the only one to have access to the accounts. So, there was, there was not a big factor because sorry. Yeah, strategically it should be transferred the continuous labor foundation. We had a discussion about that more than a year ago that time, but nobody had time to do that we had much more serious issues with Asia accounts. Now Asia problem is more settled via thanks to Olivia for pushing that because finally as a board member received the invoice first time. So at least the process against the streamlined. So, you know, we can focus on the AWS. And why we talk about Azure accounts. We still have to reduce the cost of that accounts. So with the recent change that we did to the mirroring infrastructure we already reduce. But it's too early to want to know how much we reduce the cost, but we should definitely reduce I hope to reduce by 1000 dollar based on the change we did two weeks ago. So even just one thousand dollar reduction you'll be above the target, right. Yeah, exactly. So the challenge is every services keep growing. And so while we had that issue with the mirroring infrastructure, we still have to optimize, especially in the CI agents. So I hope that the work that Damian is doing to increase at the time to start an agent will also affect the building. But we have to work on that. But we are running over time. So thanks everybody. Thanks. Thank you for your time and see you on RC. Before you leave, I already put the notes to the next meeting agenda. So if you already have ideas that you know you want to discuss next week, feel free to add them. Thanks. Bye bye.