 So, starting out, this is the, um, okay, working group meeting for November 9th of 2021. Don't forget to put your name in the attendees list. So we know here and, uh, we'll do a quick agenda reviews, everything in the agenda, anything that people want to change your ad. While we've got it up before we get started. Going once. Going twice. All right, and let's jump into it with release updates with the dean. Um, first of all, can you hear me? Is my mic is kind of weird? Yes, we can hear you quite clearly or I can. Um, so this weekend, we were supposed to have a 40 to release, but it's kind of postponed because. Uh, the rebase to 121.5 of Kubernetes and Kubelet and things didn't land just yet. So it's going to land today. Uh, so we kind of skip this release because first of all, we didn't have much to fix and for it. Uh, second issue on the table is, uh, for nine. For nine, at least already, uh, for at least the testing and we would need to switch to for nine stable eventually. Um, I think the safest way would be releasing for eight. This weekend. And releasing for nine is stable the weekend after. That gives us quite enough time. To test for nine at least that also approximately at this time. OCP upgrades would be allowed from for eight to 49. I think they are allowed now, but probably blocked by something at least. Um, we don't see a huge volume of that. When it starts happening, we would have quite a good results of the health of for nine. And, uh, we'll get you safe to move. Uh, next on, uh, Mustafa has fixed the storage test. In, uh, communities, all cherry picks have landed in. Master 122 and 121 that the branches we care about basically. They didn't land in. Micro releases yet, but once they do. Um, open shift with rebates on top of them, we'll get these fixes and we would finally be able to. And then those Linux policy. Uh, package, which has been causing us quite a lot of problems. Rather, it's been way to secure. And, um, another problem has been found in for 10. Rather, it affects. Technically it affects all the installations, but it's most visible on for 10 because, um, AWS M six families using. Uh, Nitro, uh, hypervisor and it's crashing on latest kernel. Five 14. So you won't be able to launch latest door across. Uh, once you launch, uh, for nine or for eight. Using. This it would crash and for M six, but, uh, and five would work. And unfortunately for 10 has moved to M six by default, because these machines are actually cheaper. And so far the mitigation is that we rolled back the kernel to five 14. You'd still have to use an older for the request due to this issue and some more previous ones, but in the end it starts correctly. So that's what we're testing right now. Um, it's, uh, it appears to be an actual either AWS issue or upstream. We're doing most likely upstream kernel issues. So not much we can do until we wait for the fixed land. Um, I believe the, the interested parties and notified identified all of the upstream bug report. That's what we'll do. Um, if this issue won't get resolved. Um, GCV is known to be working fine, but it appears to be affected. At least AWS and VCR. I don't think it would be, um, would be affected. Um, yeah, that's pretty much, that's pretty much it. I don't think there's been a new changes. But again, um, the door across testing develop has moved to 35 and would really be interested to try this out. But the kernel problem is, uh, at this point looking at once we would push it away and would be able to fix the saloon. Policy pin. We would release for 10 to 35 and check out the results. Yeah, I think that's all I've got. Any questions or comments for Vadim and the. Okay, the updates video release updates. All right, seeing nothing we'll move on to the, uh, F cause updates with Timothy. Yes, so hi everyone. Um, so I've got two items today. The first one is that, uh, with federal certified released, we are now moving to federal certified for federal chorus. So I've put the link into the document with a link to the tracker of where we are right now. And so next has been on federal certified for a while now. Uh, testing is either just right now or just going to be in federal certified in the next one. I don't remember. And then we move on to next to stable in two weeks. So testing in is not right now. So it should be in. Yes, right in the next release, which so with the. With the base with the major releases we've reached to a new way of doing the release. This is the way so we kind of accelerate the pace. So the documentation somewhere link in the ticket if I can find again. So that we do the updates a little bit faster during the move to federal certified. But yeah, so essentially it's coming really, really soon to a testing release near you and then two weeks next weeks after to a stable release. There are three that will help. Okay. D a lot because a lot of the hacks you currently have with. You will be able to drop them. So that should have things. Um, yeah. That's on the front of the, on this point. And then we have a second one, which is kind of like an update of what we discussed next previous time. About the previous time was two times ago. Where we're working on creating the way to quickly use for chorus systems. So, like for federal chorus, you could have your base system and then overlay on top much more in a much more easier way all the packages you need. For example, for OKD and the instruments is posted there. I have put the link. The progress. We're progressing quite rapidly with that and support so should should be available shortly, at least for testing it out until it's fully integrated into the MCO and everything. And we get fully it, we get full full the full thing available. And yeah, essentially right now I think we've got like the ability to do local. Tools from containers and clean layers and everything so like the basics to work and hopefully this will like really help for OKD. To with with issues and everything avoid the rebuilds and enable. Overriding packages when needed, overriding things, changing config defaults and everything that essentially you just use the base for the first image and then do the tweak some stuff. And yeah, and that's the two main items for me today. For this week. Any questions. The Fedora cordless side of the house questions or comments. Excellent. Moving on to the docs updates with Brian and 3t. Actually, she's on so it might just be you. Okay. And yeah, so the, the new documentation system is now live. So if you go look it looks a little bit different. And it's now based on MK docs or it's pure mockdown. So pretty easy to update if you want to. I've now put the instructions for if you want to go and update the docs and now they do it within the container. And if you've got a chair or already workspaces on a cluster, you can edit it in the cluster or you can install local tooling. So that's all set up. And let me know if you need any help or any issues, but that's all good to go. And there is one issue that I think might be worth talking in this group. And, and it's an issue raised by Sandra, I know Sandra was on so I don't know whether you want to actually pick this up and talk about it. I stumbled upon a CVE report for OpenShift virtualization last week. And I was wondering if there is any anyone tracking CVE that are also affecting OKD from coming from F course or coming from OpenShift or any other layer that we are building upon. So I looked for the website searching for a security contact and couldn't find one. I saw there is a security policy for OKD itself in the GitHub code repo of OKD pointing to Reddit security, but I'm wondering if it's someone is actively looking at that. Don't just raising the topic. Yeah, no, that's a great, great question. We don't currently have anyone volunteering to scan the security related security issues. I think that's a great idea. Is there anyone in current anyone currently in attendance that would be interested in doing that legwork like every two weeks or so basically looking for stuff that would impact OKD so that we can let users know that we are aware of these things. And if it's something upstream, then we can say, hey, we're waiting for this fix upstream. If it's something that we can mitigate in the few cases where we can, we can talk about that. So I don't want to be our security liaison. Maybe something you want to put out on the Google. Yeah, I think we need someone. I think that's a great, it would be a great thing to have someone doing that legwork for us. If nothing else, you know, as far as the open shift code base itself goes, because it's the same, it's the same images, same code base. To CVE really could track from the red hat site, Fedora, Cora OS on the other, that's a different matter. There are likely being reported to Fedora, but we don't have that. We don't have anything that's tying those together. There's one exception to that, Cora. Occasionally, we are not occasionally like, you know, semi regularly, we have CVEs come in that we don't disclose through bugzilla. So they're private bugs that only red headers could see, but they might affect pieces of open shift that those CVEs would not be discoverable by people outside of red hat. And I think even for red headers, there's probably some document we signed that says we won't expose those things. So I think that's probably the one area where this might be difficult. Yeah, that is true. Yeah. So on the point of contact, I'm looking for the exact URLs and things, but essentially for all Cora 3 pose. We have this one security policy on GitHub, which I think we can apply to OKD, which by default says contact red hat security team. And we'll take care of that. And yeah, I'm looking for the exact link where this is stored and all this is set up, but I don't remember and I'll have to find that again. Yeah, that seems like a good default like defaulting to giving the red hat, you know, address this one. That seems like a good place to push people towards. Is that something red hat is okay receiving in that way. So speaking for federal chorus at least we perfectly fine receiving that this way and I think it would should be perfectly fine for OKD too. I mean, yeah, I concur with what Timothy saying pretty much. Considering that OKD and OCP are actually identical. I hope they're okay with it because if they're not, then there's a problem. A bigger problem, right? All right, well, let's what we'll do is as a task will add the docs group to actually come up with a security page. That sort of outlines our security posture. And then we'll provide those 2 links for people to be able to submit and get info in terms of F costs and in terms of OKD. And we'll go from there and I'll send something out over the working group for security liaison. Anything else on the topic of security, Sandra does that does that get us closer to what you're thinking? Yeah, that makes sense. So on the topic too, we have this one. So this is like the row list of all CVs or all adversaries for that OCP. So this one probably also applies to OKD in a similar manner. Considering that you will have to remove all the real specific ones and add all the federal chorus ones in there. But essentially, it might be a good place to start. And this one should be full as far as I know, like fully accurate. Thank you. Anyone else have any thoughts on this in terms of getting some of the volunteer security liaison or any of the documentation, anything else we would we could add to round it out. Okay, then, Brian, you want to continue on with. I guess we'd be a code of conduct and then Twitter and then the survey. Yes, so we're looking at adding the code of conduct. We're taking from the Ansible repo. Michael is adapting that. And we're going to actually use that as a code of conduct. And I think the idea would be that we could refer to it when we start a community event or a meeting. We can actually refer to that and it'll be actually on the dock side as well. And I think as far as Twitter is going, we're looking to set up a Twitter account and really was going to actually set it up. So we didn't get a we didn't come up with a good background image yet. Like I think we're using just something kind of she's been handling all this. She's not here. But yeah, it's it's up. We just need to start deciding on content. And so that we've got the one background image that was sort of pulled from like the website. I think it was like just grabbing the website. But we might be able to come up with something that looks a little bit nicer. It's a little bit fuzzy. Yeah, so she did come up with that. And I guess the docs group or this group, if you have any thoughts on what type of things we should be putting out over this. Like do we want to put every release that comes out, you know, Vadim, does that make sense for every single one? That's an official release that we, you know, announce that through the Twitter, announce the meetings, things like that. Is there anything else that we would want to put through the Twitter and the docs group? Of course, we'll deal with this as well. All right, we'll make sure you follow it. And I think we'll probably have it start following other people to sort of get that, you know, read going. And we will start using it and then folks can forward it. That's the other thing is for social media like this to be successful. So folks need to forward and reshare social media. So in this case, if you could retweet the OKD tweets when they start coming out, that would be super helpful because then that builds up, you know, that social media grid and makes those connections and stuff like that. And survey, Drity hasn't gotten much farther on the survey, but it's, but she's taken that on and we'll know more about it. I'll touch base right now. It's like 11 PM or getting close to 11 PM there, but I'll reach out to her and see where things are. And CRC stuff, Charles, take it away. A couple of free moments. So I built a 4.8 version and stood up. That's about all I know that's happening with CRC. Are you getting any feedback from it? Has anyone else played with it? Somebody tried to use the an earlier version of the Linux one that had a the upstream group made some changes. I had to I had to make a patch and push a new version. But other than that, I don't know how much use it gets guys using or not. I don't. I actually don't use code for any containers. Yeah, I see it mentioned occasionally the OKD variant. But it's like maybe once every couple of weeks, I'll see someone talk about it on a mailing list. That's about it. Do we have any any sort of stats in terms of visitors or downloads? Does anyone have access to the account that could pull that off? It's it's just a URL hanging off of the OKD.io. And we're using some borrowed space on one of the Fedora servers. So that's speak of. Yeah, doesn't seem like it's easily. Neil, do you happen to know who in Fedora gets logs for that infrastructure? Do you even know what infrastructure that is? So dl.fedoraproject.org, which is where it is is the download master server DL IB 01, which is in the ID data center in Virginia. That is managed by Fedora infrastructure specifically by the release engineering team. So the access to that is controlled exclusively by Red Haters. So someone on the Red Hat side will have to talk to the CPE team. For those who don't know, that's community platform engineering to talk to the Fedora infrastructure subgroup of CPE about if we if there are any stats, if not, if we can start collecting any. Justy might know who to talk to. I think he's the one that got me access to it. Yeah, probably. I can do a Fedora project. Can we put that as a to do for either one of you to hunt that down so to see if we can get some stats on on downloads or anything. Even remotely letting us know, I mean, we'll have the survey, obviously. Sorry, we're the two people. So, well, three of you any of the three of you, yes, absolutely. Are any of the three of us Red Haters. Okay, cool. So let's start with you, Charles, since you might be the best at figuring out from the Red Hat side, who has access to that. And if you already have access to it, you may have log access to. Or if you want to decide to delegate that to one of the rest of us, and you can get us access to do that, that's fine too. You know, I got access to this machine before I went to work for Red Hat. Yeah, this was back. Oh, goodness, when did we I guess it was last summer. It was before that when we when we pushed the first. Maybe it was, I guess it was 20 early 2020. Yeah, because the actual okay before release was late summer of last year. No, it's been out longer than that. Nope. You're thinking about the COVID year that everybody lost. I've been doing that all the I've been I've been conflating 2019 and 2020. Yeah, I'm talking about the COVID year. So, does it make sense to spin off the CRC stuff into a separate meeting? I noticed that's been added as an item. Can someone take that on so that maybe the CRC folks can. I had volunteered to organize the meeting last time and I was actually waiting on you, Jamie for instructions on organize said meeting that didn't actually make it into the task list last time. So that's that's my bad. I put it in the task list this time. Okay, great. Yes. Okay. If I if I don't hear from you in like a week. Should I email you slack you? What is the best way you can slack me. You can smack me. You can smack me. You can smoke a machine, whatever, whatever it is to get my to be honest. I was busy with enough other things that I wasn't like itching to do this. So that's why I'm up, but okay, cool. It's on the task list now. So hopefully we'll close that loop. And I apologize at my day job. I'm pushing a project to production. The past 2 weeks and it's just been fun and excitement. So yes, I totally understand the information and then we will go from there. Okay, anything else on CRC that the that the whole group needs to hear about. Um, so I think the main 2 things that we'd want from a from a CRC. Subgroup to start with based on the conversations we've had here are. Um, discussions about automated builds and the info for them. And then the larger existential question of does anyone use this. And if not, is it worth. Should it even exist? Are there other are there other agenda items that people think are important for a CRC subgroup to go off and figure out. And if not, that's fine. I just wanted to ask the larger group before we broke off to make sure that we weren't missing key things. Yeah, Daniel, whether or not people are using it. And whether or not people would use it more with different configurations or sort of 2 separate questions. Sure. Um, yeah, does anyone use this and is it is it worth doing or 2 separate questions. Thank you. Because like, if you if you look at say mini cube or something like that. As a, an alternative, which doesn't have the same usability. And you look at the resources that that takes. Then that sort of gives you a useful baseline for comparison. So much more compact. Well, exactly. It was a lot easier to cluster up. It was definitely a lot easier to run mini shift. Okay, so, yeah, so that's that's that's a good question to answer. Besides a metric for anyone using it. Um, the, the, is there a configuration of it that would make it useful for people to kind of figuring out what the direction of it should be. Okay, cool. Um, that's enough of an agenda to get started. I just wanted to make sure that. That the things people wanted that committee or that some committee to do were actually going to be done. Yeah, yeah, and Mike, your point is valid. It really is. It runs better when it's running on it. I was just thinking like, this is like, it would be really cool if you could take a CRC. And like run it in such a way where it actually installed the stuff on the host machine instead of trying to create like, you know, a virtual machine to do it in and then like. And then you could have a situation where you could just take a bare metal host or just a bear host, run some sort of CRC installer and have it actually deploy to fill the resources on that machine. Like, I think that would be really interesting use case kind of going back to the days of OC cluster up kind of behaviors. Yeah, yeah, absolutely. Cool. Okay. So that that's, um, that's a bunch of good stuff that we can that we can cover in that in that meeting and I'm just saying if you're looking for big fish to fry, you know, I don't know. I don't know what I'm looking for. I don't know what I'm getting myself into but that that looks like a nice big fish there. Yeah. Maybe you'll boy get around on one of these. That's what you want. All right, let's move on to any are there any issues in the repo that are something that came to our attention that we need to pay any attention to. There are those kernel crashes. Vadim mentioned that most like all of the other issues are very old. So not a lot of issues. Of course, Kai. Ramell posted something that came up that looks like there is a must gather that he posted about 7 hours ago. That's someone that we may want to court actually for the working group. He's always he's actually running OKD in production and is always posting stuff where he's finding bugs and whatnot. Maybe I'll start a conversation with him and see if he's interested because it'd be cool to have him sort of contributing or at least coming on every once in a while. We definitely want to hit him up with the survey. Anything else in the issues section of the repo that you want to bring to our attention or talk about? Okay, is there anything in the discussion section that folks want to talk about? Someone mentioned the CRC OKD mismatch. Packaging OKD binaries for Fedora. I did not see that one come in. Oh, yeah, I did. Yeah, yeah, yeah. Mike Rocheford is not here and he's yeah, I just noticed that. Yeah, that was Mike's line. All right. Well, we will shelve that one to until the next meeting. Anything else in the discussion items there? Generic certificates for old OpenShift origin. Yeah, I don't see anything. All right, let's move on. A lot of the discussion stuff it seems to be like just a user question to resolve here. Location of the main repo Diane is going to check with legal. I need to actually create a discussion item. So for tasks moving forward, and if folks could remind me or help me with this, the goal is to create discussion items for every task. And then this way we can actually track that people are getting their tasks done and folks can talk about tasks and things like that and follow up on them. So my goal is starting in the next week or so to start creating discussion items for things that don't already have discussion items related to tasks. So Diane never responded to me. Charo, did she respond to your internal? Yeah, in fact, she's not, she's not online right now. Okay. Yeah, so we'll find out about moving the main repo. The other thing I wanted to talk about, which is kind of a loaded question, and it may lead to a long discussion or short discussion. How do folks feel about set weeks for the OKD working group? In other words, the first and third week of the month, or the second and fourth week of the month. This came up and I'm starting to sort of ease into participating in the GitOps working group. And I asked the question there and it apparently was something they were talking about. And I think it's something we could talk about here. It would be a lot easier to know which meeting between the docs meeting and this meeting and be easier to invite people because they'll always know it'd be easier for scheduling. Is there any reason why we wouldn't want to go or anyone have any general thoughts on going to like a first and third week for the main meeting? Are we already effectively on that? It ends every two weeks. Yeah, but you end up eventually it ends up being you get out of sync with that because if there's a fifth week on that Tuesday, like if the month ends, do you see what I'm saying? I don't know. For me, it's easier just being biweekly. If I have to go in and figure out how to schedule it for the first three years. It's kind of a pain to fix that in the calendar. Like I rely pretty heavily on my Google calendar telling me I'm about to miss a meeting to actually join and doing those kinds of things, those kinds of gymnastics are quite hard. I move mine into exchange calendar so that all the other people at B-CIT can see it. And I don't think that that's as flexible as Chrome jobs so that you can do like the first Thursday of whatever. So like every other, like a repeat every other week is something that is easily manageable in all calendars. Jamie, was there a reason giving for moving to a 1st and 3rd rather than just the current biweekly cadence? So some people have scheduling in their day jobs that are the 1st and 2nd week or the, you know, or the 1st and 3rd or the 2nd and 4th week. Some people have voiced that as something of a concern. And then the other one is that in those weeks where you don't fall on the 1st and 3rd or 2nd and 4th on that 5th week, there would essentially be a break. And so people could breathe a little bit. But it's, I mean, is anyone else have any opinions? It sounds like most folks are into the just every other week thing. That's, and that was the other thing. By the way, the other, that was the impetus is the initial impetus was me actually asking the get-ups working group, well, what week is this? Because it says every other week for their meetings. And I was like, well, what does that translate to? When is the next one? So for inviting people, it's problematic or for posting it in such a way that people could just happen upon knowing when the next meeting is. So we would have to, and Mike, I think you just posted something. Yeah, so Mike just posted something, right, putting something on the website and also via the new Twitter and stuff like that. So, either way, I think, yeah, we should do that, like put it on. I'm kind of curious. Couldn't we just have like, okd.io slash meetings where we have like the iCal files like easily downloadable and maybe like just a display that says this week's meetings are like x, y and z or whatever. Yeah, well, so we have the FedoCal entry. Yeah, the FedoCal calendar, you could just like import it. Yeah. If FedoCal can be configured, you know, the FedoCal setup for this event can be configured with whatever way you want. Then it's fine because it doesn't actually matter what restrictions everybody else has, because if they just subscribe to the live iCal feed, they'll just get the events as the calendar system pushes them. But like, if I have to make calendar events for this is going to be very, there's going to be some gymnastics involved, and I don't know if I'm going to get it right. Mike, what were you going to say? I just, I've had a poor user experience with the FedoCal stuff like it doesn't import nicely to my Google calendar for some reason. Yeah, I noticed that as well. I actually have, on my personal Google account, I have like 20 different FedoCal things imported in. You have, and it works fine. You have to tell it when you're pulling in a calendar event from there, you have to grab the iCal link. You know, so you go to that and say copy iCal link, and then you tell Google Calendar to subscribe from a URL. That means that's not the same as importing. Importing from a URL does a one time thing, and then that'll be wrong. But if you tell it to subscribe to it from a URL, then it will actually work correctly. Why these are two different things? Hell, if I know. Yeah, I mean, yeah, maybe my Google food is just weak here. Thanks, thanks, Neil. Alright, so so is the temperature of the room then that folks want to just continue with bi-weekly? Alternating bi-weekly? That's the temperature? Okay. That's what we will do then. But don't be surprised if we do cancel one of the meetings on either of them just so that people do have a break. Because Diana and I do burnout. Okay, we have about 15 minutes, but we've made it through our meeting items. Is there anything else that folks want to talk about before we break? Do we want to just pre-cancel all the December ones? Good. Good. It's like last year we ran up right up to the line and then we decided we were going to cancel all the December ones. Well, so what do we, let's see, we have December, there is... Maybe we could the first two weeks and then we cancel. Sounds like a plan that we come out with a new F-class version, major version, a new OCD major version, and then we cancel all the meetings at the same time, right? That sounds like a great plan. Yes. And shut down the website for maintenance. Right. So, are we saying the 14th? Through the 4th? No meetings? I'm fine with that. Or do we want to do that the 14th and then go 21st? That's kind of late though into December. I mean, if folks are going to be really busy. I mean, my PTO basically starts on the week of the 12th. So, so after that it's just... And my PTO doesn't end until after New Year's. So, I expect at least from my perspective, I'm personally canceling everything that I'm attending in those three last weeks of December. I would say, yeah, the 21st nobody's going to show up and the 4th nobody's going to show up. All right. So, straw poll vote for canceling meetings from December 21st through January 4th. Raise your hand or say aye. Do it. Even if you don't, I'm not going to be here. You're welcome to meet. Okay. Does anyone have an opposing opinion on this or a different perspective? Let me say different opposing frames it has. Different perspective on this than what we've just... Anyone? Okay. Now, where are we in terms of actually a little bit closer? November, are folks okay doing the 23rd? Because that would be our next one for November. I'm fine with it. Yeah. All right. So, I will see you all in two weeks. Expect your task list to be created as discussion items. And of course, if anything comes up that you need to talk about the group, you can use the Slack. I've been successful at making the dev Slack channel mostly for dev questions or dev discussions. So, feel free to pop in there or use the Google Working Group Google group. Awesome. Thanks, folks. Thank you. Bye.