 uh account okay so everyone welcome to the weekly Jenkins infrastructure meeting today we have Mark, Stefan and myself Damien team and RV are off today let's start with the usual announcements so the ISO notification that the package and the docker image for the weekly release are available I assume that as usual we have the weekly release check to be finished until end of the day is that correct Mark? Yes I saw that I saw that the tag has been applied and I started the initial looking I haven't haven't run the checklist yet but the the changelog is there and I'm preparing the revised changelog to correct some flaws in the automatically generated changelog some release checks to be done I'm sure for the docker image because I triggered the builds on trusted CI to see what the logs looks like related to the IRM issue so I saw the image being built oh good excellent thank you so I don't that's that's very good okay um on other announcement I don't I have the digital ocean credential exposure so we published an an issue so it's public knowledge now however I prefer mentioning it as an announcement because last week we said more information to come um so you have details on the issue that I'm adding right now so um nothing serious happened no sensitive information were accessed but due to a digital ocean technical token uh that has been stolen or leaked we are not sure about that part some virtual machine had been created 10 days ago they consumed some infrastructure credits for 10 minutes except one that we kept just for analysis so there were basically mining bitcoins or whatever crappy virtual money um we confirm with digital ocean and github security teams that no sensitive information were tampered or accessed which is a good thing all the analysis of details are on the issue and if it's missing don't hesitate to contact us or contact security at genkins.io we have synced and we had a lot of help from Daniel and Vadek thanks also Mark for helping me and reviewing these elements um the we have some tasks that has been done or are being currently being done to improve the security because we have some gut feeling where the leak could have come from but we cannot be sure that's the conclusion so let's try to improve the security layer by layer and see in the future how it happened special huge thanks to thanks to Hervé he's a half today because he took on him to call me on a sunday he took on his personal time so really thank you Hervé for that because um yeah I understand that calling someone on a sunday can be a hard decision to make but he reacted really quickly and we were able to immediately stop the suspicious activity so thanks for that don't hesitate to contact me on the future on that area that's a call for everyone no more announcement for me is there is a real announcement for you folks oh thanks very much for what you did on the on the that thanks for and thanks for the ongoing work no problem that does a very fast reaction great job yes that's really positive team yeah so happy happy with that outcome um just to not we had some a lot of tiny tasks done um we had a lot of ongoing issues some access requests to the virtue to the vpn mainly to different people um Hervé because we removed in from everything and we did it but our future release officer as well that was opportunity to improve the documentation um we helped uh we helped a bit gave in on the plugin site issue so thanks for your work gave in on that particular application which is around the plugin dot genkins.io website it's done that area so the goal was to improve the automatic update release of the mcharts so each time there is a new docker image now it should be picked by update cli making gave in on any contributor autonomous to review it if they can approve the update cli pull requests uh so there are some initial e-cups on the initial deployment we had to correct some kubernetes subjects ills check probe like all kubernetes application as usual um what did we have we had some issues around genkins permission but most of the time it was fixed team helped a lot i haven't tracked all their there were a lot of requests from plugin maintainer that were fixed about archiving some repositories on genkins ci so that's an area which is the ldesk is used by the ldesk but most of the time it's not on the area of the infrastructure team so thanks for our team here come for helping on that most of the time or jc these were the main sorry these were the main topic uh we have some work in progress subjects main one is the digital loss and credential exposure the main consequence is that we have completely disabled by default the github checks uh from infra ci our private instance so it's another step further more security we only publish the status exception of the kubernetes management job which has explicit github check instruction on its pipeline because we know the content which is exported to github in that case is highly highly specific we need it for uh being efficient and reviewing pull requests and the sensitive data has been removed we can still think about disabling it and keeping everything on blue ocean a lot of time was spent on rotating all credentials so in that area haven't tracked everything because it's on a private repository but i'm working on a run book listing all the credentials to rotate we have reached the point of half of the credential can be rotated almost automatically you go to a repository type some commands that will be documented and then you only have to copy past the encrypted subs file and then it just work so thanks uh on the on behalf of the team to olivia verna because he put he made it available with the work on subs so that's really efficient um um one or two main major elements before i go to the open issue today first one upgrade to kubernetes 1.21 so thanks stefan for taking care of starting that task the first part was updating the kubernetes command on our environment the thing is that since we deleted all the digital ocean cluster as a safety measure we need to create a new one uh the thing is now kubernetes 1.20 is not supported anymore by digital ocean so we cannot create a 1.20 cluster so we need the tooling to be able to support 1.21 so ongoing tasks that should be fixed in one or two days so expecting the next milestone the digital cluster uh back on that area mark did you start were you able with rv to start the blog post for digital ocean have not i still have that that action item i did have a question how's our how's our expense profile on digital ocean or maybe that's where you were leading with the question on the blog post because before we ask them for more money we've got to be sure we have a blog post to highlight their expectations so far that that's the point um we shall run out of credit uh in one month on digital ocean based on the previous month's consumption that's uh that's what we anticipated so um don't uh don't scream if end of the month when we will reach the end i'm since it's my credit card i might go myself and delete the cluster end of month still it's better to recreate it because we need it's recreated for the sake of the automated process so i will take care of my credit card and please take care of the efficiency of the bills but yeah i need you if you and or ever don't have time i can jump to help but the goal is on the we need a blog post on the upcoming days or two weeks at least yes and we want blog posts anyway so let me work with out of a on that thanks for the reminder um there were some minor exchanges with the linux foundation uh gira reaching end of life on october this year so uh we have uh requested to linux foundation if they can update the next lts that should be end of life next year there are they should come back to us with a proposed the date for the upgrade because we nothing to do on our side except putting a message on statues.jenkins.io the three past upgrades last year were five minutes shutting down issues that jenkins.io so we have to let the user know and that's okay now there was another linux foundation topic that is a new topic you're okay if i bring a new topic here yes so it may need infrastructure team help the linux foundation has asked for permission to send a survey to jank to active jankins maintainers about specific areas of interest to the linux foundation and as our sort of sponsoring organization they're the parent of cdf we're open to consider it but i've raised the question to the jankins governance board because i'm not sure what the policy is in terms of with whom who who is allowed to know the email address of jankins active maintainers and so it's a it's a question for the board the next board meeting is i believe actually later this week or maybe it's early yes it's tomorrow so it's a topic on the board meeting agenda but be aware that if the board approves it i may ask for infra help to identify active maintainers and their email addresses and active maintainer is a an arbitrary arbitrary call i haven't don't yet know what criteria i'd even use to decide active or not okay to be decided so don't hesitate to raise thanks for sharing raise the eldest issue one if we have to do this i'm not sure about the local what's the name of that european laws about the email address and stored somewhere i'm not sure right and and that's that's a piece that i've thankfully olag as a member of the board can can help a little bit on that he's had some experience in that area so i i'll look forward to that and we we we need a decision from the board and then some conversation about okay what are we allowed to share what are we not allowed to share and etc if required to have the infra team gather it's okay thanks for letting us know mark on the linux foundation area as well uh around the email hosting so no answer from mail gun at all no answer from kk i've raised the topic to ask the linux foundation to us the mail server for genkins.io they closed the issue already directing me to the cdf because they say there should be a pmo as i understand that to be the project product manager officer or i'm not sure about the acronym but we should have a pmo a person from cdf that should be the person contacting the lf for that request not us directly um so i've opened an issue on the cdf foundation repository but i haven't had a answer but it was five days ago with an Easter long weekend so i propose that we wait at least one week before booking someone at the cdf i'm not sure who our cdf contact will be i'm not sure maybe i don't know if it's oleg directly if we have a pmo associated yeah we'll probably have to ask i'm used to working with andrew grimberg of the linux foundation uh he helped us with the transition from from uh genkins self-hosted jira to linux foundation hosted jira and if you want i can i actually i can send you his email address you could start the question with him because i think he probably is part of the project management office their pmo okay and he's not at cdf he's actually at linux foundation but i think he's the right he's probably the right person to ask okay so i propose i will ping you uh in let's say i propose eight days uh time out we are at five days so in three days if we didn't hear from the cdf then we try contacting that person directly sounds good for your mark yes so right now the issue about email press will be on hold uh and we'll see the next step uh gave in the has been talked about that topic so if he doesn't answer then we will wait to contact lf and cdf but sounds like administrative decision on who might do what uh we'll see on the next step so let's let's let the cdf decide for the for us in that area uh also contacted docker i haven't had any answer around the open source program but i've been shared the latest contact from someone i know at docker who is working on the technical part of the open source program uh he told me to contact them and don't hesitate to ask and what i shared that with them um we described our use cases the issues we had right now we'll continue stefan and i for this iteration this milestone to continue working with our accounts considering it should be non open source uh on that area we're gonna fix the current issue now and then we'll see based on docker answer one when we will have one um on that area i've exchanged with olivier and he shared with me all the credential and information he had so runbook has been written about the policy all doing manage docker organization which one are we using there is a pull request and that will be part of stefan and hi walk to finish the that pull request on the runbook that means adding what stefan and hi already built so we need to complete the missing information but it gives the main direction especially the decision that was made around free seats per organization which means two owners that are human the infrastructure officer and its backup and the technical account and the technical account is used by us for pulling or pushing um so that has been documented so if you think uh if you have been blocked by that please review the runbook team uh i need help on that area i have to drop off damien i'm going to i let's see i think i need to transition you to host so i'm going to do that make host okay you have the hosting now i'll have to drop off thanks a lot mark see ya so the rest of the topic won't belong we have some issues that i'm going to transition to the new milestone mainly replacing blue ocean as default URL for ci jen kinsayo so it's not removing blue ocean the discussion happened on the issue the goal is on all the links that you see on each kit up check sometimes you have a green icon here red for ci jen kinsayo only these links are generated by your setting on jen kins instance that creates the link directly when sending the github check back to github so the request is to change that link to point to the jen kins classic ui instead of blue ocean because blue ocean is a dead project preparing the the exit of blue ocean okay exactly uh there is no blue ocean removal based on the discussion that will break too much use case but at least it will start a transition the introduce artifact caching proxy is on hold for now because of the digitalism consequences and for the rest that these are minor issues to fix the main one is about the lts image the latest lts published the 6th of april wasn't published for irm and cpoz so we have to diagnose and fix that as soon as possible uh one last note about the mile the new incoming topic before we stop that call so we're gonna have this one um one information so we are going to speak about that next week in details but um we have the topic of migrating updates to another dot jenkins dotio to another cloud which is blocked by when first one to sunset the old mirrored system which means that the mirrors dot jenkins dotio or jenkin ci uh domain name which are accessed in plain htp today will be moved to the actual gates dot jenkins dotio so it doesn't remove pkg and exactly exactly htps will be forced so if you are using these services with htp only you will be forced to use htps the redirection will be done automatically but your htp client must support that so a blog post on public communication will be done we are going to start updating the usages we have on jenkins ci and jenkins info organization there have been lists anyone having issues with that please mention uh on that issue start now we're going to expand the audience so in the upcoming days and weeks and i think that's all for today see you next week bye mr. precolding