 Okay, Olivia at yours. Hi everybody, welcome for the new Jenkins infrastructure meeting. Today's we are going to talk about the releases, few change coming for the Jenkins releases. And we also have topics regarding JIRA fixes, JIRA configuration, the account app and many other. So what I propose is we start directly for the Jenkins release. So today we did Jenkins weekly release. Most of the things were correct, accepted that we are having issues to build a darker image for the latest version we are currently investigating why. So very brief, very quickly, we are running bats to run some tests on the darker images, and for some reasons, the tests are not passing. It's kind of the, it works on my machine, but in this case it doesn't work on the CI environment so we are still investigating specifically for that. Any question. Anything to add. Hello. Thank you so much for that team. The next topic also that I want to mention with the Jenkins release so basically it's specific it's regarding the Azure file storage issue we had several weeks ago. So Microsoft came back to me over the weekend to say, basically nothing really new. They explained that there is a limit on the number of files we can open at the same time. That limit is 2000. I cleaned up that limit because they have instructions about cleaning that limits. So I have the feeling that they weren't able to identify the root cause of the issue right now. The thing is, at least for the moment, the Azure file storage has been working for two weeks or three weeks. I'm not sure. So what I'm going to do in short term, I'm going to roll back to that, to that systems, but I'm probably try to improve that in the future as well. I'll probably do that tomorrow. I'll have some time to finish this. I didn't want to much the PRs related to that because I wanted to be sure that the release last week was okay and weekly today as well. But now we should be ready to roll back everything to the previous situation. Olivier, good question regarding the Docker image issue. I mean the Windows images are published and the problem is just the Linux images and that says command not found. Is that not straightforward? It is straightforward. But there is a PR open to resolve that. It's a typo by me. Unfortunately, you can't. We don't test any of that part as part of the PR. So when a PR was merged to debug something else, it broke the release part. But we're struggling to get the PR to fix it through the build system because all of the bad things are failing. I see. Okay. I just misunderstood what was the problem then. Thank you. I mean, yeah, we could force merge. I mean the PR isn't actually going to be tested because it's a change to the published script and we don't run the published script as part of the PR. So you could just merge that. You're wasting your time then if it's not testing it. We were just trying to understand whether something else had been introduced in sort of inadvertently because everything was failing. That explanation. Also this remind me, we are now monitoring each time we do release so each time we published a new version to the Maven repository we have metering in place that it takes. So we're not monitoring if they've been read that's used in Windows packages are available, but we are not monitoring if the Docker image are available. So that was the first step to be able to measure the fight that when we do release we try to have that per that moment very short between the moment we publish the release and the problem, the moment we generate a different artifacts. And that's ongoing work and that's which will come in the coming weeks. If not a question, I'm going to move to the next question to the next topic. So we noticed that when we may create the jira to the Linux foundation for some reason the account app stop creating users in jira. So, instead of relying on the account app to create the user, I made few changes this morning. Basically now, when someone tried to look in on jira, if the user does not exist, it imports the user directly from that. So it created a new usual locally on jira from based on that. And then we can modify the display name and the email from Jerome. I mean, this is not ideal, obviously, but we just realized that people were modifying the display name and the email so for now that's a current situation. Everything is back to normal now so the good thing is, we should be able to remove the account app to replace it by key clock. We also have to review the account app code to be sure that there are not other specificities done on the account app. So one thing I just noticed, I realized that we discussed this yesterday when there was a problem. This basically now means that the security team is in the situation where I always was against a user account migration, because now there can be plugin maintainers that do not have a jira account. Because the account only gets created on login. And before they log in, if they track issues via GitHub and only ever log into our factory to have upload permissions, I will not be able to ping them in jira, which is a problem for me. Daniel, I'm not sure I'm capturing the concept correctly. So there is a user account, but it only exists, you use the jira user account to ping someone with a security issue, but that jira user account does not exist until the user has logged into jira. Did I say it correctly? Yes. Yeah, so that's the difference. So previously on the account app, when we created a user, we automatically inject that user in the jira database, which is not the case anymore in the current situation. So in this case, that's the jira who import the user, but only when the user tried to login. So, yeah, it's an invitation for Daniel. I don't have a good answer right now for that. The hosting user will have logged in. So whoever hosted it will have logged into jira. Well, the one who's asking for the hosting request, but if you take over a dormant plugin, you may have never accessed jira, especially if you track issues through GitHub. So, yeah, I acknowledge that it's not the default flow, but there is a valid ownership change and all of that is something we do and we support it. And it's what would allow someone to get into the situation to not be reachable via jira. So ideally we can, there's webhook support, perhaps in the beta account app or something like that, where we can keep operating a minimal sort of account app that would create those accounts or periodic accounts or something along those lines. In theory, I could create those accounts in jira manually, but that's so incredibly annoying that I would really prefer we had a different solution here. So I'm checking right now. I think I don't think we need to go into more details here. I only brought it up because you were saying, yeah, we don't need the account app anymore and I didn't want to leave that uncommented for the for the minutes. Okay, none of that. That's a good remark. I'm going to check if that configuration doesn't allow me to synchronize user on a regular basis. I want to once a day or something like that. Those are definitely okay. Good to know. Okay, thanks. Thanks Daniel for your input. Thanks. I put to the agenda the next topic which is Windows Windows Docker image but I don't think anything changed on that topic. Garrett, would you confirm? Yeah, that's that PR has merged now. So we should. Okay, and they are very available for the new release that just came out. Those ones, those ones worked. Perfect. Before we talk about Seattle Jenkins radio I also want to remind people that I deploy a service name status to Jenkins radio. So the idea of the service is just to provide feedback to the people. So it worked into ways. The first one is we just write a markdown that we published in the kitchen repository, and that markdown provide information about a coming maintenance, a current issue or whatever. So the idea just to provide a central place where people could be notified about ongoing work. Then also, I started creating basic data. So I insert data. So I just want to provide basic information if the service is really running or not mean it does not mean that the service is running correctly. It just means that just basic information. It doesn't take a lot of time on it. So that's a really good time to provide feedback on that service. The only thing that I would like to work on it if we decide to stick to it is I would like to provide more basic monitoring so so provide monitoring for the other services so right now we only provide for get the Jenkins radio updates the Jenkins radio package to Jenkins radio. But yeah, just embedding more I frames and suggestions here. A quick question. I may have asked this before. Isn't the usual approach to put a status page on a separate domain so that even DNS issues. Would not affect the status page. I know nothing about this but that's seemed what the motivation for some of like, I don't know cloud copy status GitHub status, all of these. Yeah, that could be a good practice but especially in our case we only have Jenkins radio engine can see it at work. And we have a natural DNS for those two domains. I think we used to have more Jenkins domains but I don't have access to them. So I think it's, it's kind of okay. I think it's an acceptable solution to have it running on the same domain then Jenkins radio. Normally you have like a different domain register and everything is like a super thing because then your domain register grows out and you still got it but I don't think we need it we just we're just starting it. Okay, so so it would basically would be overkill and I mean we always have the Twitter account in the worst case. If everything else goes on. No, your DNS is that huge issues. I did go out early to So I was, I was expecting less actually data on the page the status pages I'd seen had typically not included performance information Olivier can you share some insights, what prompted you to want to put the status I frames inside there. Was it so the reason why I put the frame is because we don't have public monitoring. And the thing is, I really rarely have requests from people saying, is it me or that website is down. And so that that that was the motivation so yeah, in this case I'm just providing the information over the last week so you can easily guess if something is wrong. If you don't have any data since one day let's say you know that something wrong is happening since one day. And if the status page say says that everything is up and running because nobody provided for that that information. Everybody can open a PR and say there is something wrong. Let's say with a link to the to the mailing lists are to add into any information that could be useful to the other people. So I did I did some quick experiment based on the Atlassian status page and this one CC estates the lessons that you speech. We could have it part of the open source sponsoring program. It's more advanced. We have better notification and so on, but it will remain an application where you have to log in, provide a password and only privileged people can modify the information that we see a state is that everybody can open a PR everybody can say there is something wrong with that service everybody can provide link to let's say Twitter, Reddit or whatever the information you want to provide. And then other people can catch up. So it's not only the infrastructure team providing information if the services are not and providing the default monitoring to say okay that services responding. That's really a good information for the person who want to open a PR because yeah, if that if we don't have any response time then then something wrong is definitely happening. And if nobody is talking about that on the mailing list or on our seat there is definitely something wrong here. So that was a motivation. I think it could be really useful if we also plan to notice in advance when we do maintenance. So, if you look right now in the status page you see that I sent, I sent a notification saying that we're going to update a community cluster. This was a test, I think, for instance, the next time we do a community cluster great. Even if we are not expecting any downtime, it could be useful to just notify people one week in advance to know that they may have issues. So is this, I've typically on ci.jankins.io just noted to the IRC channel jankins.info, is that a place where I'm upgrading plugins, I should publish something there would you envision or is that a likely lifetime of that outage is maybe a minute. Yeah, the thing is, there are two things. The first one is you can, if you know that the maintenance window will be let's say one hour, you can open a ticket you can publish that markdown fine and you can say there is a windows maintenance and the service will be done for one hour. You don't have to eject more information once your maintenance windows is done. I'm not sure if we have to create an incident when the service is done for a few minutes, but if we know that the service will be done for let's say 20 or 30 minutes. Maybe that would be nice to add information there. And I think when we do, let's say ci.jankins.io version update that would be useful to put information there. Because I think last week I saw people asking if the service was done or just maintenance. It was just the maintenance. But yeah, I think it could be useful for the people. I just wanted to add I really like the graphs because an experience that I had several times over the last few months with GitHub is I'm basically on GitHub all day, and something starts feeling broken, slow, or not right. And I look at on GitHub status and there's nothing. And I look at GitHub status 10 minutes later and I noticed 10 minutes before they did. Yeah, so the graphs giving me a life inside without someone having to create an incident might be useful, especially for a for our environment which is not quite as monitored as I'm sure GitHub.com is. Thanks. And I propose to move to another topic. So I want to say that we are starting to investigate how we can improve the stability of ci.jankins.io, and we are looking at ways to run ci.jankins.io to Kubernetes so in order to move most of the agents on Kubernetes. We do quite major refactoring, because right now we both use containers and virtual machine running Docker, Docker again, and the Jenkins file would not be compatible. So we are still at the beginning of the process to see how we can improve ci.jankins.io to be sure that it's reliable. And yeah, we are trying to identify the different areas that need to be improved. We really would like to see like a best case about the best way to run ci.jankins on Kubernetes, which is something that interests a lot of people today. So yeah, if you're interested to provide feedback, this is a project that we are starting now. Any other? Yeah, so yeah, that was just some information. If it's okay for you, I propose to move to the latest topic, which is sponsoring, so not only Oracle. So before we talk about Oracle sponsoring, I had a look to the state of the infrastructure sponsoring from iOS, and we are almost running out of credits. I think we still have one month in front of us, so I would like to re-engage with Amazon folks to see if they are interested to continue the sponsoring or yeah, basically what's the current state there? Does it correlate to the previous topic? I mean, if we start moving some workload of ci.jankins.io on EKS? This could be related to the previous topic, yeah. That's why I want to bring this here. Oh, but actually isn't a move to Kubernetes a good thing, because if we needed to move to some other hosting, to Google, or to IBM Cloud or Oracle, whoever donates to us, Kubernetes is likely much more portable than our current Azure-based infrastructure, right? So for me, it feels like Kubernetes is a good thing, no matter whether we stay with AWS or not. So yeah, so you're right, you're right, but what I want to add to Damian's concern, the reason why we decided also to move to Kubernetes was to have a better portability. Right now, each time, I mean, we moved from Azure to Amazon, we moved from Amazon to Azure to Amazon, and so we had compatibility issues and it forced us to fix small issues. The idea here is would be to use Kubernetes agents, so wherever we run the service, it's fine. The main difference is one of the limitations until today's to use Amazon to run communities was the Amazon is still owned by CloudBees, so only CloudBees employees can work on it. So Damian started deploying in Kubernetes cluster on Amazon, so we could provide the Amazon cluster and then we could work on Seattle Jenkins.io the same way we're managing the other services. So using ham charts, if we don't have sponsoring on Amazon, we may have to look for other solutions for the infrastructure, I mean for wherever we run the Kubernetes cluster, but the work to run Jenkins on Kubernetes, the work to identify how we monitor efficiently Seattle Jenkins.io, how we run, how we write a Jenkins file to run on Kubernetes.io, the Jenkins file on Kubernetes for Kubernetes is still relevant anyway, because if tomorrow we have to move to another cloud provider, I mean everybody's providing Kubernetes clusters. So we just, yeah, that would be a good way to be portable. In a kind of related note, has anyone looked at the kind of Docker deprecation from Kubernetes and the potential impact that I have for us. Yes, to be quite honest, there is nothing to worry about, at least for the use case of agents running on Kubernetes. The reason is, it's because the deprecation is not about Docker, but about a wrapper between the Kubernetes agent named the Kubelet and the Docker engine, which is container D. And on the middle, there were a wrapper named Docker shim created to ensure that it's Krio. Sorry? We lost you. Oh, no, I heard you just fine. So, okay. David, you said low risk because because in fact, it's, it's only, let's say black box or Kubernetes interact with the container engine. The container engine by default is container D, which is the same as Docker. The thing is that Docker is kind of model it with a lot of features and Kubernetes is only using the container departs. And what has been deprecated is that they don't use the Docker shim wrapper that were used until this latest release. So by default, if we use pod and we deploy Kubernetes as we used to work, there should be no concern on that part. However, the risk could lie on how do we use Docker in Docker or Docker on Docker. So as soon as we have, we need the Docker engine to run a Docker run command or Docker build command as part of whatever build. In that case, depending on what we want to do, it might be an issue. However, I will say this will be tackled on by the security part, because Kubernetes is theoretically able to run Docker engine as rootless container. So that will be basically part of defining a Jenkins file that say on that part, I have a container with a Docker engine, which is only mine, no share dependency. And if we can ensure that this run with the security concern we could have on such a cluster, then we don't have to care about the outer container engine which is managed by Kubernetes without us knowing anything. I am confident I did not capture it. So it makes CI more complicated where you're using it for CI. So it's things like services that use things like test containers and want to spin up like integration test databases and that sort of thing. So I don't think it would affect us directly. I don't know of any places right now, isn't in the current state where we mount the Docker socket. It certainly affects people who use Jenkins as a CI service and runs their jobs in pods because it's fairly common to mount the Docker socket so that you can run at least test containers if not other things. Yeah, the thing is that it's really risky in terms of mounting the Docker socket is something risky. Especially in a Kubernetes cluster. Basically, you have access to the Docker socket, you are super user of the underlying host. Unless that Docker socket is restricted with containerized and isolated Docker engine which run as rootless. But this depends on the kernel of the host you are running in order to have Docker inside Docker nested containerization with the right level of isolation. That's why most of the time the CI pattern is when you need such case like running test container that will run Docker run stuff, you need a virtual machine for your workload to be sure that you have a one site agent. This might be something we're going to hit. But we need to investigate on that part, but no, no concern for the current status of Kubernetes that we're using related to the Docker deprecation. We still have to release of Kubernetes before anything will be changed by defaults. The next one dot 20 will take three months to be available on both AKS or GKE or AKS and it will only print warnings on the logs deprecation warning. AKS is moving one dot 19. We're not using one 19 yet, but it is to a I think I believe it's to a on AKS never already moved to using container D and not using Docker. So while that's the same with GKE as well. Yeah. And I guess, and I guess we're still we're still running on one that's 17. So, yeah, I think IBM did it a year or two ago as well. So it's, I mean, I'm just other streams kind of behind the cloud. I mean upstream is just killing anyone if over I guess most of the cloud vendors already moved away from that. I just proposed to we stay focused to them, because we are almost running late so we are, I mean, we are running late now. So what I propose is to quickly cover the last topic which is about sponsoring by a record mark do you have some inputs here. Just to note that Oracle offers a two year, two year 75% discount for their for their resources as part of a thing called Oracle for startups, and they've offered it to the Jenkins project. At least they've expressed willingness to consider it. I've sent a message to the information mailing list. I think it just is a place to discuss it further there. I'm going to do some experiments with Oracle cloud right now on my personal time, trying to see what would it mean. How would it interact etc. It's a cost savings. And one more thing about sponsoring. There's something that I forgot. We are paying for the Jenkins infrared projects, get up accounts. So it's something like $300 per year. And now get up changed several months ago to provide private history. So we could switch to the free plan now. I don't know if you have any input there. Maybe I should contact support. I know that we'll have to pay by the end of the month. So I just want to be sure that there is that we're on a legacy as well and you don't have access to everything. I think we don't have things like triage commission and other things because those are only available in the plans. Okay. It says unlimited. So free is unlimited public or private repositories. I think we have. Oh, yes, free. I thought this was done. Sorry. Yeah. So the only thing so on free we miss out on codon as which we do use. I do have an information regarding the Jenkins CI organization. I don't know if you are limited to one organization because I'm pretty sure that the Jenkins our organization is not sort by itself. The Jenkins the Jenkins CI organization is a free organization. We have no private repository. It's completely open source. However, the Jenkins CI search organization has a sponsorship that Tyler organized. We currently have like 300 private repositories there. And it we basically have a 100% off forever coupon, but I'm a bit hesitant to change the plan and everything in case that vanishes. Okay. Yeah, I understand. It's just Jenkins infer is the one I guess we're paying some money for. We don't use codon is very much that we use it on things like the Jenkins IO repo for the governance stuff. And like we need to use it. It's mostly for automatically requesting reviews. Oh, but but I'm thinking, oh, it really is used to control to control access right you can't approve a change to the governance documents unless you're a codon or governance. Yeah, it's just saying that if you're on a free plan doesn't look like you have codon is. Okay. I'll investigate. So, your dolls a year or whatever we'll always ask about sponsoring for it. Okay, thanks. I propose to close the meeting here. We already four minutes after the line. So I propose to continue the discussion on our team and on the meetings. Thank you for your time. And see you later. Bye bye.