 Okay, move your faces. Okay. Here and here. You can start the recording whenever you want. Go ahead. Hello everyone. Welcome to the Jenkins Weekly Infrastructure Meeting. We are the 8th February 2022. So today, as attendees, we have myself, Hervé, Marc, Tim and Stefan. Okay. Let's get started with the announcement. So last week, we didn't have a weekly meeting, but last week, the weekly 2003-03 was released successfully, as far as I remember. At least it's running in production for us on Infrascii. So thanks for the release. This week, there isn't any weekly expected, at least not today and not tomorrow, as far as I understand, because there will be a security release on weekly and LTS. I don't know if we should trigger a weekly Tuesday or Friday. I don't know, Tim, what do you think about that? I know. So there will be a weekly tomorrow. It will be 2.334, but it will be a security release. So it's 2.333 plus just the security upgrades or security changes. So no normal weekly expected. I don't need a security one this week. Yeah, no form modernize. No form modernize. Okay. Yeah, form modernization first arrives in next Tuesday's release, right? Okay. A huge PR that was made from Monday. And Tim, any opinion, just for my while we're here, any opinion on should we include system D, transition in that, or should we wait a week after that? I think we're probably okay to include system D in the form modernization one. Okay, great. Thanks. Yeah, I marked it as on hold as I didn't want Bezel to merge it to master. Merging it before the security release would be disastrous, right? So that's one. Yeah, good. Okay. About the next week related also, there will be a set of security release for plugins that should happen 15 of February. So I took the responsibility to say we will delay the weekly to the 16th of February to Wednesday. There shouldn't be any problem releasing weekly on plugins the same day. However, given that it's sensitive because security, because we messed up the automatic weekly last time, I would prefer to have one thing per day just to be sure it's clean to avoid any rush, because if security is late, or if we have mirrors that happen at the mirror synchronization at the same time, there might be what be of yours, resulting on security plugin release before the announcement. So there should be no problem, but better safe than sorry. So the weekly with system D and form modernization should happen next for the Wednesday, 16 of February. Tim, that okay for you since as release officer, that's probably your domain to make the ultimate decision. It's just weekly, isn't it? Yes, right. No LTS is this week, so nothing but LTS. Don't care about weekly really, just whenever it comes. Great. Okay. Just to say it's system D, not system V. System V is the old unit, or it's referred to something. Transitions. Is that not right? It's maybe okay to include system V in it to system D. Ah, okay, sorry. I misunderstood. Sorry, I was skipping my bad. Just poor phrasing from the person doing the typing. I fixed the phrasing. Hopefully, it's more readable now. I thought that's why you were writing initially, but then I saw the end of it. Yeah, it's the non-English native era. That's misunderstood. Also just poor phrasing. So there's nothing to do other than bad phrasing on my part. No problem. So these are the three main announcements. Do you have other announcements to make? Okay. So let's proceed on the tasks and notes. First, top priority about the two security releases. So this week, tomorrow, we will have cores only. Remember, don't merge anything this afternoon or tomorrow until the security release is done. At least nothing that could have any impact on release.ci or ci Jenkins.io, please. This morning, we took the opportunity to update release ci infra ci ci and trusted kernel plugins, everything that was available this morning. So tomorrow, the security team will have a clean state to work with that would avoid them having a zillion of plugins to update and they have to manually select. Most of the time, they don't need to, but yeah, at least we have a clean state for them. Once again, the weekly automatic release triggered automatically. This time, it's no one's fault. Compared to last time, it's because we had a short notice from the security team that they were willing to have a core release tomorrow and they were going to stage it Monday. At least I received the notification really, really early, really late. So yeah, but just to be sure, thanks for the review. We disabled the management of release ci on Kubernetes. So the proposal is that we don't put it back on automatically managed until the security release is finished and we will have to do the same thing next week. Just to be sure there isn't any unplanned reload of the configuration on Kubernetes that will enable again the job. So is there a way we could change the Kubernetes definition such that the job starts is defined disabled during these periods or that's not not really workable? There is a contribution to be made on job DSL so that the attribute disabled is exposed on the job DSL configuration and used. Okay, so right now you can't create with job DSL a disabled job. Yeah, exactly. Part of the code is already there, but it has to be exposed as an attribute from the configuration. Yeah, we get to do that. But in the day, we don't really want any change to release.ci. We don't want anyone updating plugins or the image or anything while we're anything that could take it down while we're about to do it. However, I also had a smart idea that could be complementary to that, which would be changing the pipeline for the releases to check for a log file or not. So that would mean if anyone from infratim or security team commits the log file on the repository, when the automatic weekly trigger starts, if it sees the log file, it will stop immediately with a message saying, I got a log file, I should I'm not expected to trigger weekly. That's a simple one that will be manipulated as code and we won't have to care about any Kubernetes reload or job DSL. So say that again, which log file? That sounds interesting. Okay, sorry. Perfect. No, so it was it was a file that explicitly says this system is locked. And because this system is locked, do not allow the job I see. Thank you. Okay, which will be full as code only restricted to the release pipeline. So releases.ci.genkins.io could still continue working and being updated without any risk. Right. So that should be a complementary process, but we still need job DSL contribution because disabling jobs is something we need on numerous places. Okay. So that's why I wanted to mention that part. There is an issue. I've put all the links. And Danielle yesterday did the part of the staged the release part, not the packaging part, only the reason generating the war and signing it for the weekly, it worked perfectly. I mean, last week, last week, last week worked perfectly. So that was the same code. So that's fine. About LTS, still the same issue, which time I make a change to the master branch. I forgot about, I don't know which branch I have. I don't understand that it's really painful for me. So we ended up on the same issue as we had two weeks ago. The jar verified missing from the bin because using the doc image older than the one weekly. Just to be sure, Danielle did the manual, the verification manually, like we did two weeks ago, I just ready to continue the packaging. But just in case, thanks team for reviewing that, by the way, I've merged these changes from the main branch to both stable 2.319. So the LTS branch and the security new branch. Because as I understand, there is one branch for each patch of each LTS on security releases. Most of the time, they build it from the stable branch. That's too much for my brain. So I've, I did that these two places just to be safe. If I forgot any target branch, don't hesitate to open the pull request and mention. I suspect we already have a target branch for stable dash 2.332.1. So maybe worth checking to see if. Is it for the patch on the weekly? No, that's for next LTS for the March LTS. Okay, so that means the error also happened on that one for sure. And I assume that Daniel, oh, it was using an older version of the pipeline set that is still working by any luck. It has that that release hasn't been done yet. So we wouldn't we wouldn't have detected the failure. It just I suspect there's a stable dash 2.332 branch out there. Okay. Who is going to do that part that staging release? Is it us or security team? And sorry for clarity, there won't be a security release of 2.332.1 because that's next LTS. That's the March LTS. Okay, okay. This is just me for warning that I think the branch already exists and needs the cherry pick. No, nothing more than needing the cherry pick. Okay. We have to take care of that. Can I defer that to someone else than me just to be sure that that's knowledge is shared? Sure. So I'm happy to take it or I could I could pair with Stefan and he and I could do it together. It might be a good experience for the two of us to pair. With pleasure. Okay. So how about if Stefan and I take it? Okay. So let me write this down. Yep. Cool. There are stills. So yeah, that one was important to have in mind for tomorrow's release. For tomorrow, I don't have any other elements. Do you have any other points for tomorrow's core releases? Okay. Just a word for next week that will happen before our team meeting. That's why I wanted to mention it. Trusted will be needed at all costs because it's only plugins. Release.ci However, let's keep it safe and CI Jenkins.io, of course, because they will need to update the plugins if we have some. I haven't seen the list of plugins impacted by the CVS stuff. So let's assume CI Jenkins.io will be, especially since Vadek asked for the status Jenkins.io. For both releases, I need someone. I can take it. Someone can be me, but someone need to update status Jenkins.io to say that CI Jenkins.io might have some restart and unavailability tomorrow and next week, the 15th. Is there someone volunteering to updating status Jenkins.io? For the whole day? No, no, no, just for it's usually it's a our history says it's about a it can be anywhere from a 15 minute to a 45 minute downtime. And Stefan, if since Stefan, I are going to be cherry picking on a branch, why don't we also take that one? Because that way he and I can do a how do you do a submission to status? Looks good. Thanks, folks. So let's so that will be for tomorrow's release and next week. So you can do this and pull a request. Good. Yes. Okay. Is there anything else about the security releases this week or next week that you have in mind? No, okay. Thanks, folks. Next priority topic, Digital Ocean, work in progress by RV. Irvade seems that locally you successfully created the resources. You were able to run the terraform process and fix it on numerous levels. It seems that the read me that I originally created for AWS terraform management needs some a bit of revamp or update at least and the file structure. We had good exchanges about how to build a stronger shot pipeline library for terraform management and testing and stuff based on the issue that they had locally due to, let's say it's the youth of that terraform stuff. I understand, Irvade, can you confirm that you are the CI steps, meaning making the CI jobs working? It's created, but you have to make it work so it managed Digital Ocean cluster. For now, my big issue is the credential for the backend config file, which are turned correctly interpreted. It puts the environment variable name inside the backend config file instead of the content referenced by the environment name. So I have to manually upload a new version of them each time the team or mark does it ring the bell when specifically saying specifying a credential of type file from jikask? That should be a base 64 from environment variable. And what happens is that the environment variable is put as content of the credential, so it's correctly encrypted and decrypted when pipeline use and load the credential. But we have to set the multi-line content by ourself inside Jenkins through the UI because it doesn't work from jikask. It should work in jikask, but it's easier if you just avoid this and just use string variables. I'm not sure exactly what's in this file, but normally if you just do something like an AWS or Azure, you just do tf-r-m-i-d equals and then have the key as an environment variable. So like with the Azure Terra from provider, you can provide files or you can just provide environment variables and it's not simple. Not for the backend, you must use a file for the backend sports because most of the key are or you have to define all the you could define the keys inside the backend.tf file, but the Terraform in it needs a file in order to have a lock. Yes, it's to have the lock on the backend. Is this for the like the S3 bucket or something else? Because looking at the digitalization talk, so it's very sturdy. And I'm surprised it's the same mechanism as AWS as far as I can see. You can supply it on the command line. Yeah, I can. I think I can. But I was thinking I can inject the variable via a little bit of script in the beginning of the pipeline if needed. It also says that you can do environment variables too. You don't have to use a file. We still have from 1.1 maybe, but not with 1.0. We haven't been ready to latest minor. I don't think the stuff doesn't tend to change very often. I mean we also even if you can't use environment variables at the very minimum, you can supply it on the command line. You can go Terraform in it's dash backend dash config and then you can supply a set of those. Also another clue is that since the content of the backend config for the actual Terraform and the digitalization are almost the same, there is only one change, one key. The idea would have been to reuse the existing credential and then only define the key on the backend.tf of digitalization because it's not a sensitive data. Only the key is just a namespace for the Terraform state inside the bucket. That was the thing I had in mind. However, we still have an issue on our setup and I don't know if it's gcask, if it's sops, if it's the way we inject the contents, but we have exactly the same sops data for both credentials and here you see the configuration inside the cask configuration passed to the elm values and for both we use the base 64. In the zoom set, I have put the error log I've got in the Terraform pipeline. Where we can see the contents and the file contains only the variable instead of its contents. So that one line 154 works as expected, that environment variable has a multiline string content coming from sops decrypt it and it's correctly base 64 hashed and it's set as a credential that works as expected. But the digital option one doesn't work. Did you say it's just not been substituted as if it's just got the variable there? Yeah, it's got the content of backend config in the pipeline. It contains the variable name, literally what I have on the line of my screen share. Okay, I mean it doesn't sound, it's probably not jkask related. I'm pretty sure. And I've checked the variable name, in case it does a typo, but it doesn't seem to. I think jkask will substitute it, it's been in the string if I can't find it. So it's probably, I'm not, I think it does that. So it's probably... My understanding is that that variable is not defined. Yes, so it's literally jkask just take it because it has different three ways of using the base 64 method inside the environment. It can use read the content of a file and base 64 it or a string or it will have a reference variable. So there is something weird. Sorry, we sidetracked, just in case it was a well-known behavior, I don't know. So we have something weird and we need to change or to fix that because it's slowing down our way. I feel like it'll be a type or somewhere. Yeah, I'm too bad at that. For sure, but yeah, we can find it. But yeah, outside that credential issue, almost there. So good job, RV. Next topic, unless you have a question or something else. So update CLI, so great work, Stefan, on adding more update CLI tracking on more dependencies. We have HashiCorp tools, which has been a bit too recently and the most interesting part is Kubernetes management is now tracking the Amei for the EC2 agents that we recently added on release.ci and infra.ci. That should allow us to build using Docker engine on virtual machines and that should allow us to use the same labels for both CI Jenkins IO and our Kubernetes Jenkins Controller. So in the future, we should be able to shrink the pipelines to use only Jenkins file and migrate trust it to release. That will open this road. So thanks a lot for that work, Stefan. You're welcome. Private AKS cluster. I haven't done anything. I was in day off, but that's my top priority for this week. Focusing on the Groovy pipeline library for terraform management and then re-initialize the Azure repository like the digital ocean that is working on. So we can create a private cluster and move at least in FraCI and release CI with behind the VPN. We had a word error on service mirror Jenkins IO. The error was said a bad certificate except that we never used mirrors with HTTPS. That was the whole reason for migrating from mirrors Jenkins IO in virtual machine in Amazon to get Jenkins IO with mirror bits inside the Azure to have TLS because the tooling used for that legacy mirror Jenkins IO web service wasn't supporting HTTPS. So we don't know how the probe, the Datadog probe could have worked before that day, but it was eroding. We removed the probe, but that starts the question of decommissioning the service mirror Jenkins IO. There is an issue to be created. Thanks, Tim. That's a good point that you're on the line. That's worth communication, at least a blog post on the Twitter before really starting to decommission. And as I've been underlined also, we might want to search for the references of mirrors that Jenkins IO inside our own code base in Jenkins Infra, at least test RNS and pipeline libraries and change them to get Jenkins IO. So at least they will use the correct service. Not sure. There are several references to mirror the Jenkins IO on Jenkins IO organization too. I don't know if we should put a CNAME alias to keep mirror the Jenkins IO or to change this URL there too. That's a discussion that would have need to happen inside the issue for that decommissioning, but yes, you're correct. Should we change on client side, server side, but another word on rating the Jenkins IO. We have Jeremy Playout from Cloudbees that started to work on to take again the work that gave in on maybe others started one year ago, at least the goal is to migrate the service rating Jenkins IO from the tiny virtual machine on Amazon where it's running today into Kubernetes. So it took the work that gave in pointed at us to have an Elm chart that he was able to run locally. There is a pull request open that we have to review. The next step will be to prepare the installation of that Elm chart on our pod public cluster. Attention point for the infra people. Rating need a database. It's a PostgreSQL database. I don't remember where it hosted. I assume it's on AWS as for today. Yes. Need to be checked. Okay. Yes. So we might want to create a new PostgreSQL database inside Azure. At least manually and I will import it on Terraform later when it will work, but we need a PostgreSQL and there will be a migration phase. We'll have to dump the SQL from AWS and insert it inside the new database. The reason is because accessing a web service on Azure, accessing a PostgreSQL database on AWS, not only it will be slow with latency, but also we'll have to tweak finally the IP allow to connect to the database. Given the tiny amount of data inside the database, better to migrate it directly all on Azure and remove all these resources on AWS. What do you think? So thanks Jeremy. Jeremy is out, so we will continue. It's not a priority topic, but on background, that's interesting. An external contributor doing that for us. That's really nice. It's four, so I propose unless you want to bring another topic that we delay the other topics to next week. Is there any prior topic that we forgot about or do you want to speak about right now today? Okay. So thanks a lot everyone for your work. I'm going to update and prepare the note for next week. Move everything we didn't treat it today and publish everything on this course once the recording will be available. Thank you. Thank you everyone. Have a good day. Take care.