 Welcome to the working group meeting for April 26th of 2022 and we have a pretty full agenda here. So we're going to jump right into it. We'll start out with agenda review. Take about 30 seconds to take a look at the agenda. Please put your name in the attendees sections that we know that you were here and we can sort of keep an eye on if we need to reach out to someone who wasn't able to make it. That maybe has tasks or maybe will be affected by some of the things that we talk about. So we'll get started with some release updates. So Christian, why don't you go ahead and take it away. Yeah, sure. So I wasn't actually involved in making this weekend's release that was Vadim doing his great work again. So but I saw there's a new release out so yeah everybody please test it and then get feedback back to us. Regarding the question that came up that just came up earlier before the recording started with regards to the at CD corruption issue. I am not actually sure this solves it so the question somebody brought to me was whether we could do another 4.9 release to fix the issue that is in the latest on the last 4.9 release that we have. We can't actually do that since we have this rolling release model we are now at 4.10 so we can't go back and release another 4.9 version. If the version of at CD that that we that we released this weekend though is bigger than I forgot the specific version numbers here. But if it's if it's bigger than than one of the effective versions, it should be safe to operate but I am not sure whether that's actually the case. So if there's anybody with more knowledge, I'd be very thankful for them to chime in now. I actually came up as a question in the channel and I couldn't answer it directly off the top of my head. How can we check the at CD version, either from the command line of the. We could log into a node and do at CD control right to get the version that's running. Is there a way to do it just by looking at the manifests or the containers or any aspect outside of an existing installs anyone know. It might be on the cluster operator resource. I don't know. I'm not set up to check right now though. Okay. I think something we should figure out that way we can. I think we should post something about this, the documents, documentation subgroup can take this up. We can post something about it and actually say, hey, if you want to know where you're at. These are the versions and you can always test yourself by looking here. See which version you're running because that email today. I mean, this has been out actually as an issue for. What, like 3 weeks now, I think, or almost a month. And like that email today, just everyone just went, it was like the don't panic scene from airplane. Well, isn't this blocking lcp for 9 to 410 or something. Yes, it is, but if you do 28, I think it is or whatever just came out, it actually is the edge. So take a look actually. I don't have it in front of me, but yeah, so actually it is the releases do open it up for that. I've just posted the, I've just posted the knowledge base article in the chat. We can also put that in the notes, but it's at CD 350 to 352 are the affected versions. Yeah, and we're talking about data corruption and essentially. Not all of the members are getting the updates under heavy load. So yeah, I think the docs group should should take this up as something to let folks know about. Just because we're going to I'm sure we're going to get questions about it. Moving forward. And Christian anything else you want to let us know about in terms of. Yeah, one, one noteworthy thing is that we that the latest release now fixes or moves away from the cryo version that is affected by the TV. I don't have the number right right now, but there was a cryo CVE and unfortunately we were stuck on that version for a couple of weeks as you know, and that is fixed in the new release. So if, yeah, that is a good reason to upgrade definitely and I think previous versions were also affected 4.9 possibly. So definitely. Yeah, keep that in mind. That is now fixed in the newest release. Very cool. Now in terms of overall process. You, you were missing access. Is that right Christian, you were missing access to something to cut a release. Is that resolved now. So yeah, essentially, there's a few things I wasn't able to mirror out the images, because only but the has a mirror of the images to quay because only buddy has access to that that is fixable from our side. So I can, I should probably just request that access it wasn't given to me back in the day. There's the the other issue that the whole build system is essentially owned by Vadim so that it runs kind of in his name space using his, his secrets. I can only say right now there are going to be changes in in the build system and we are going to pull that into prowl it back when we set up the system it wasn't possible to do those are PM West tree composes within prowl. That is entirely possible now and we also have something we're investigating which is the core West layering effort, where we would reuse an actual F cost release and just layer our things on top and thereby preserve the actual F cross versions because we've always had had this package version mismatch mismatches for all the packages because we only pulled in the Fedora products manifest and then did a new compose you are PM West tree compose for for KD and with this new concept we will be reusing an existing compose. And we can now do both things within RCI within prowl are completely new composers and also these layer layered composers or just layering of an existing compose and we're going to be changing towards that in the near term. Okay, so near term. Um, three to nine weeks. Okay. 123 spreads. Okay, excellent. All right. Does anyone have any questions or comments for Christian in terms of okay the releases. There were a couple of small but important fixes in this release. One, there was a kernel issue. So you actually pin the new kernel or pin a previous kernel, the new kernel was was breaking on VMware. And we've also hopefully permanently until we have a real solution fixed. Kiways issue with the search dot issue. We put that into a spot where that will always get resolved properly. It was a time issue before we had a five second timer in there and with 410. It looked like it was acting between four and six seconds. So sometimes it would work. Sometimes it won't. Well, we're able to take that sleep out and put it into a proper spot. So now it happens after networking is up. And it just does this thing have not been able to make it break yet. John, can you drop that issue in the meeting notes for folks that are sure what we're talking about. Basically, there's a DNS search. Issue because of the three different competing systems. For name lookups on F cost around fedora. And so there's things that get written to resolve. And that needed to happen after other things. And so. Information was not found in terms of external domains. Yeah, go ahead. There is one outstanding thing that will be fixed probably with the next release. We, we found the third and issue with the installer. And that will be the installer with creating a bootstrap node and not being able to resolve API int that fixes and will probably be in the next release. It's already in the 410 and it's been approved and all that just missed it by a day for this one. And there is one still outstanding issue. Again, this is a Kaiways issue where the master controllers don't seem to sync properly at the beginning. That's, we don't know what's causing that. I mean unless Christian, you have an idea. I only have a time. So this is the the MCO and the machine conflict that that gets rendered right so the two, the two machine conflicts that aren't picked up some sometimes are not picked up and not included in the in the rendered machine configuration. Those are exactly the two machine conflicts we are adding in our custom okay D build process. Yep, though it. Yeah, and I don't know why they sometimes get picked up and sometimes they don't but yet also seems like a timing issue they seem to be created to at two later stage where the MCO, or the MCC has already begun, which is the machine that the controller has already begun rendering that that configuration and therefore doesn't include those two. Yeah, not sure how are we. Yeah, I don't really know the way forward or what the fix would be. Maybe it just goes away with our new build system I'm kind of hoping, hoping that but yeah, I don't really know what causes it. It's it's definitely specific to our builds because it's it only really only happens with our to wrestle. It's gotta be in the bootstrap process because this never happens on the worker side and the worker side is identical. Right. Yeah, it has to be some kind of timing that we don't create them early enough or something. Those resources. Yeah. So that'll be on somebody's list at some point to look at. I did look at that issue again and I wasn't sure there's no bugzilla link to it. I think it would be a good idea to open a bugzilla and put that on on the machine config operator team to. I know that they've already been tagged on the GitHub issue, but if they have a bugzilla, they really have something to to look at and they have to look at that. So that might be I'll reach out to him. Yeah, if he can't do it, I'll do it. I haven't looked at it yet. There is an issue that's going to hit us with 411 unless the fix comes again for the installer. There's something they've gone to a new a non for version of some of the installer code that is causing issues with bootstrap deleted or some issues in there so that's out there also. But just about that issue, a colleague of mine on my team is looking is looking at that, as we speak, and I think you'll probably be fixed before 411 is released because that will also affect, affect open ship compute from the product. Probably go away. Yeah, he reached out to me earlier today, so I gave him some information. So hopefully that's just in the other thing that I know Bruce is interested in this. They may have a fix for the set that stuff. I guess we're testing it in development. Hopefully they'll push it to stable. But what I don't know is how long it takes for that to get to. But they seem to have found what the issue is. Dusty can probably speak to that. Let's let I know Dusty came in sort of. Dusty is on a limited time schedule. So let's actually transition quickly to. Updates and then bounce back to anything else just so that we can let Dusty take care of other things. Dusty any updates? Hey, I don't have anything in particular. We're getting pretty close to the fedora 36 release. So, at that point, our testing stream will get moved over to fedora 36. And then shortly after that stable. So, Christian, I don't know if we have any like advanced testing of like next, but that would be useful if we find issues there to report those and see if we can figure them out early on. Rather than you guys having a freeze. Yeah, we do have that one branch that builds off of the. The dev out of custom also will probably get a build from that. Yeah. Cool. Yeah. And just to mention one of one of the changes there was IP tables moving to NF tables by default, which shouldn't be an issue because Red Hat Core OS. Or really made that change sometime ago. And then there was. What else was there? We have pod man 4.0. In fedora 36, which had some breaking changes to the pod man API. I don't remember if O. K. D. slash OCP still uses pod man for anything at all these days. I know early on there was some. You know, pulling from a registry type stuff that was happening, but I don't know if we do that anymore, but that's just something else to consider. I think it's still potman is still used as a fallback in case OC isn't there, but I'm not sure in which cases that would actually happen. And I hope that API hasn't changed too much cool images. Yeah, if it's if it's just a scripted change, like if it's executing pod man in a script, it should be fine. The only thing I think that really changed in a breaking way is if you're using the pod man API. So not the Docker API, which pod man also supports but the pod man specific API. There were some breaking changes there. Yeah, as far as I know, we don't use that anywhere. Cool. Yeah, the CLI itself should have no breaking changes if I understand correctly. Excellent. Any questions or comments for dusty in regards to stuff. I guess this is to Christian and dusty. I mean, I know that we have the kernel pinned right now. I know we don't want to keep that pin. How do we go forward and and testing new kernels to see if we can pin this issue. So there's a bugzilla for this and we should definitely follow what what happens there maybe they'll patch the kernel in Fedora maybe and I think there was already an upstream fix proposed. So maybe that gets back parted by the kernel maintainers or even upstream. We'll just have to wait for now. I wonder if we wait till 1111 what 4.11. Oh, that might be a few months out still. So yeah, I'm not sure maybe we can if there's no CBEs or other big things popping up in the meantime, maybe we can do that. Yeah. Once there's a fix, we'll fast track it in Fedora Core OS. But like it's not it's not currently pinned in Fedora Core OS just because we didn't catch it before it hit stable. And because people are working on fixing it upstream, we're not necessarily chasing the Fedora kernel maintainer to revert that commit. So we're hoping that basically a fix comes in and we can just get it back ported. I can as soon as it's ready, I can test it. I mean, cause I was able to recreate it 100%. So that'll be perfect. That's what's important reproducibility. Yeah. All right, any other questions or comments for dusty. All right, thank you dusty for your time. Appreciate you popping in to give us the latest. No problem. I might hang around for a minute, but you might see me drop off too. Okay. So when to bounce back to speaking of Kai. He brought up an issue with the registry. In four nine pointing to the catalog in four nine samples catalog pointing to sent us images that were that there. It's pointing to registry red hat registry dot red hat.io instead of quay or key, as Brian would say. So we know there aren't any for nine updates coming out. Does anyone know if 410 if that properly points in the correct place. I haven't checked yet. But Kai pointed out that the sent us based images, all of the sent us based images point to non existent places and registry dot red hat. Anyone has any insight on that. If not, I'll dig and take a look. Okay. All right. Man, we did buy him a beer sometime or whatever the beverage of his choices that guy finds so many bugs issues like everything. It's pretty amazing. Brian. Yeah, go ahead. So I was just going to ask is that specifically in the samples. So I know we have quite a few operators in the operator catalog that are also in the same place. I think it's just in the samples I'm pretty sure it's just. Okay. All right, let's move on now to actually Brian with documents updates. Okay, so we had our meeting last week and a couple of things. So, we're going to start planning to move to the new it organization the okd project get organization. What we want to do is you want to plan it and do a single move so we're not left with people spanning multiple repos and we get all confused. So we're going to start that work. We've shut down the community repo already removed everything out of this so that hopefully is gone now we requested it be deleted. So we're now just the. Sorry, Dan. I believe it was Christian I think put in the request for me. Okay, that's great. I put in the request to archive it, but I'm not sure they've done it already but yeah it's in the works. Okay, so that's going to me leave the main okd repo and then obviously the okd.io repo that will move across to the new home at some point. And we'll keep you updated as we sort of make that plan and when we're going to do that move and we are working on the technical documentation. So we want to add a new section to okd.io, which is covering technical documentation and the tasks and I did put a link to the discussion forum. So we have this conversation last meeting. If you have any best practices tips, if you know of any sort of obscure places within the must gather logs, where it's useful to go check any little sort of magic scripts that you've got to help work out when things go wrong. If you can just dump them in that discussion thread. Again, the links in the tasks at the bottom of this meeting. And then the docs groups probably me will render that into something worthwhile on the okd site. And also I'm still working through a work sample of how to compile modules. John gave me some very useful information. So I'm working through that we'll write that up and we'll also get that pushed there. At some point, Christian, as we sort of move the build, we do want to sort of write up some things like the machine config operator and some of the other sort of bits of magic that form up a release just to help the community understand how it all works and when they're looking at the repos and help them do that. So that is sort of a long term project to actually get some technical documentation there and hopefully keep it updated as as the project moves on. And regarding okd.io, we're going to reorganize the working group site. And we're going to make this main group and more prominent, and we're going to rename all the subgroups as subgroups. And they're going to be in a separate section as subgroups, just so it's very clear that this is the primary group. And then there are subgroups underneath. So just be tidying that out. And I'm going to pass it on to Jamie to talk about the subchairs because we need to do one thing to get our becoming in line with what we put in our sort of code of conduct. Right, so in terms of the subgroup chairs, I wanted to be able to make it a public. Like, hey, the okd working group is doing this and I wanted to be able to have post something on Twitter that says the okd working group is doing this. It turns out I don't have everything I need to get into the Twitter so I'm going to work with Diane on that tomorrow. It turns out I need an email address for verification. So, but once we're able to communicate out that we are actually having a process of defining chairs, then we'll move forward with that. It hasn't been a big rush until now. I didn't want to rush it because what I want to do is make sure that we let the community know that we are a as much as possible open and democratic ish community. And let people know that that their participation is appreciated and that there's a path for them to get involved. And this factors into the larger discussion of like outlining a path for involvement, which we've talked about before. So once the Twitter stuff is cleared away and we can actively like promote that we're following this process, then and that will be in the next couple days, then we will actually do that. And the other thing is I'm still behind in posting videos it. I have two more to do. And then I'll be, but then there's this video today. I just, I needed to get more hard drive space because right now we are at dozens and dozens of gigabytes of meeting videos, which is fantastic, but I'm getting my own storage actually for this. So then meeting the meeting videos will be up and also the meeting minutes have started to get posted to the OCD IO site and the link to the minutes will start getting included into the video description. So people can actually go into the video description watch and go to the meeting minutes as well. Okay, that's passing back to Brian now. Anything else for docs. No, I think that was it. Oh, yes, there was a still a call for a quest for anybody that wants to go and love, see or see. We are still looking for somebody to go love that project and create the updates. Yes. And I think it might be something that we look at to see if we can get a copy. Oh, yeah, go ahead. I'm trying to find that raise your hand button, but I can't find it on my screen. I'm trying there's a group inside of red hat. I think I mentioned this on the docs called called operate first and they have a cloud. I think it's based on mass open cloud. It's in Massachusetts somewhere and they have some cloud computing resource and some bandwidth and they like building the ICD stuff. So I've asked them and shared with them. Cheros build upstream without a paddle log posts and ask them to take a look at trying to recreate that on operate first and so I'm going to try and get them to come to the working group and talk about their adventures there. But what my goal is is kind of using this as a POC with this group to see if we can do if we can get a community build process for CRC on this cloud that we can help manage and and automate and that I was hesitating at first because it's at Boston University. Thanks like that that I was hesitating at first because I didn't think it was that much uptake on CRC. I didn't want to spend bandwidth on that but I've been told that there are people actually using it and who want it. So we'll see we'll use that and then maybe if it all goes well, we might even be able to build a community build process for OKD on that cloud too. So I'm just just so I'm being transparent about what I'm doing in the background here. I'm trying to find some additional resources and the compute resources to do the CRC and to do a community build OKD on F cost sometime in the not too distant future if that works out. I'm not positive that the OK the operate first cloud is optimal for building it. But I thought the CRC would be a first good test. So there's your quick update and I will now go back and hunt them down and see if they got any any further than I in the past week since I last talked to them. Fantastic. Thank you Diane. What does this group think about tasking the documentation subgroup with maybe moving Charo's document into the OKD space so that it's actually an OKD document and we can build and update it without having to rely on Charo. Charo's incredibly busy these days and I think the more that we can make it institutional knowledge versus an individual's knowledge, it would benefit. Does anyone see any downsides to that as long as Charo's happy that we take his content and move it pilfer from him. Sure. Okay, so that's chat but I'll say it again like the maintenance of that. I think if we bring it into OKD we should build the notion of like someone like trying it and if it fails updating it because like just bringing Charo stuff in and letting it bit rot under the OKD title is probably not not nearly as cool as letting it bit rot under Charo's title. Yeah, one of the things that I talked about as we were going through the guides and those folks aren't here today, but the folks that are going through and helping get the guides updated. The idea was I suggested the idea of let's create a calendar that every year cycles through our documentation or every quarter. However often we can do it cycles for our documentation and we assign someone or someone volunteers to look at that document and verify its updatedness. You know, it's, it's freshness and verify that it's not stale and that we do that on a regular process. So a lot of groups I'm involved with will do that with their policies, their guidelines or whatever. I think we could do that with our documentation is just create a calendar and get volunteers. Right. And if we, yeah, Brian. So I think another part of that is a few meetings ago we talked about automation. If we can automate a lot of the processes that are within our documentation, then we get kicked by a failure notice if our documentation is out of date. I think that's also a very useful way of keeping documentation up to date. And just make sure that we link the if you fix the build, you got to fix the documentation. So the to stay in sync. I think that's also and sort of reviving your project. I know that you gave us the link to some scripts that you'd written Jamie is reviving that project and actually try and automate as much of the, the instructions that we put on okay. So we know when it's our date. Yeah. All right. I think that's it for documentation. Let's keep moving on. So discussion items are there is there anything in the discussion of the repo discussion section repo that folks want to pull out and talk about or think needs special attention. I didn't notice anything I don't think that. So there's one about the upgrading the 410 the new warnings. They're there on purpose. Those were specifically added in 410. Right. Future API stuff is coming out and they got to get off of it before they upgrade on a certain point. Right. And specifically for it was like internal resources that were that were being flagged. And he was wondering if there's anything you should do to fix those I don't think it was anything that any like user workloads. Yeah, I think we just have to hope that upstream gets to them, which they will. I'm sure. Yeah, I'm sure they will. I think we like 1 of the things I've listed in this is like, does it make sense in the meeting notes? 1 of the things I put in the meeting notes is does it make sense to kick this to the documentation working for us to come up with our own little explanation of this. And maybe help folks understand what is an internal workload. And what is an external workload because the red hat documentation itself doesn't do a very good job. I notice folks are running the command. And not really clear. It's not clear what they're looking at. Right. And so I've seen several comments that are like, yeah, I got the warning messages. I ran the was it mcp or whatever. And I don't know what I'm looking at. Does it make sense for okay D to write something up that sort of helps our users sort of understand what they're looking at. I mean, you might take someone 20 minutes. If we can, if we can make the documentation a bit more clear for somebody, I mean, kind of in the FAQ. Sure. Christian, you've got your hand raised. Yeah, I found the button somewhere in a menu. We do actually because it's a separate build for the OKD docs, we can actually put some of those comments we don't need to put them on our OKD IO specific docs we can actually put them in the main docs and just put them in like a specific statement that says this only display this for or okay D. There's actually we have this one docs person from rat at Michael Burke who has been helping a lot with with our issues. He's now, I think, planning another PR to add a list for the open stack variant that just, yeah, essentially the compatibility matrix. Because in current 4.10 we only I believe support open. Is that ret at open stack 16 which is open stack train upstream and he'll he'll put that that list for for the upstream version names in there too. We can actually use that more heavily and just put OKD specific comments in the actual docs. And then shows up to our work our documentation subgroup meeting. Awesome. Yeah, we can definitely kick that there if if folks feel that that's the appropriate place versus something on the site. Would have been. Oh, sorry, go ahead, Christian. Can we repeat the questions so I don't talk and talk. Well, you know, the question might be is, are there any downsides to putting it into the product document so Christian in the in the documentation working group we separated into product documentation versus sort of community documentation. Are there any downsides that anyone can think of of putting it in the product documentation versus the community document. I don't want to belabor this but I'm just curious if anyone's using it. If you have like specific workarounds or commands for things you might have to do in that in that instance. We have these things as kind of warnings or notes on the on the product docs and if there's something you need to do specifically for Fedora. In that case, I think it would be perfectly fine to put them in the docs as well and then later on with that version of whatever software you have to use. It comes into the product that people can just move that that note. So I think it's, yeah, and obviously it'll get reviewed by the product folks, you know, the docs folks that maintain that repository. So, trying to move as much valuable information into the docs I think is is a good thing. And since we can just kind of exclude them from the product docs by by putting them into the the if statement. I think, yeah, we should try to to use that as heavily as possible for concise obviously concise notes and tips and warnings and whatnot. If, yeah, if you think it's a good place, I think you should try to put them in. Any other thoughts on this particular topic folks. All right, any other discussion items worth mentioning here. I don't see anything here in the past couple weeks that stands out. All right, moving on to issues what we've we talked about most of the issues earlier on but are there any other issues in the issues section that we should bring to folks attention. What's the I've seen a couple of these float by about f costs and our costs getting mixed up in the installer for bare metal. Does anyone know about that and what what's happening there. We did receive an issue about that. And someone mentioned in the channel about this, but I don't know anything about it. Has anyone seen this where a bare metal okd install pulls up our costs notes. It seems weird. Yeah, but he forget to change. I'm not quite sure how that would happen because the installer only has f cost images in it. But it's bare metal. So, I mean, it's a bare metal IPI. Exactly. I'm in a loss, but they were asking in the channel and I think it's the same person, but I can't be sure. Because I think slightly. That's me actually. Yeah. Yeah, I don't, I don't really know what's happening other than I extract the bare metal IPI installer from the okd release that the previous one not really recent one. And. Yeah, then. Oh, I have an idea. Hey Christian, you still on. Yeah, I'm here. So, you know that other PR that I have out there about making the installer artifacts. Yes, I think I remember. So, when it when we're building the installer artifact, which is maybe the where this is coming from the okd tag doesn't get propagated down so me. Well, no, but that's only for that's only for. That doesn't make sense that because that's. I think it might make. I think it might make sense though because I think we do use different containers for the different release artifacts and you're trying to to pull the open ship bare metal install binary specifically and it's it's a command that like this I've actually never used it myself. I've only just pulled the open shift installer from it, but not bare metal install and I think you might be right that we aren't actually propagating the okd. environment tag environment variable into the other artifacts builds containers and just the main one. And I think that is the issue. Yeah, changing it in the Docker file might actually be might actually be sufficient here. Which is the PR you open. Yeah, right, right. That would be. Yeah, if you could put that into the meeting notes that would be great. And is it Eric thank you for bringing that to our attention. Very helpful to know. Yeah, because the bootstrap know that gets set up as a virtual machine on the provisioner thing that installed with fedora chorus but the fedora chorus. They're getting served read that chorus. Also, since I have you here. Do you know how I can modify the. Like send something in that would allow me to get serial output while I'm installing the bare metal nodes. That would be a good question for dusty and I think you might want to pass a kernel argument. Yeah, but I don't know how to get that from the manifest if I can do it by the manifest or how that gets passed into the ironic or however the. Yeah, that would be you can do it with a with a K are probably passed in for UPI. Or I don't know, tell you what, we will look it up and find out and we'll put it. Did you open a discussion item on this one on the site? Or did you just mention it? I saw your question in the channel, but can you open up a discussion item on the. Okay, D repo. And if you put that discussion item there, we'll respond to you with a response. We'll reach out to dusty. And also find out if this is something that that's doable for IPI or probably just UPI, but we'll check and see. Yeah, thanks so much for what it's worth on an IPI install. You can also break out the manifest and modify stuff. So if it is a matter of doing it from the manifest. There's probably a way you can do it with IPI. You know, if you need to. Okay. Okay, cool. If it's not in the manifest already you can do it via ignition and it should, at least for the second boot. Yeah, put into the actual real route than have that kernel hard to find. Which, which still is a bit special in okay because we first boot a standard Fedora core OS and then we pivot into our machine OS. So, yeah, your experience may vary here. I think but we'll definitely find that out. I think that's interesting. Christian, it sounds like this would just be a custom machine config right that did something different with the kernel. Yeah, you can, you can specify it through machine config, but that actually isn't. That's not a pure ignition. The MCO currently consumes it's an MCO API so the MC. Yeah, the machine config operator will consume that and then the MCD will apply it and reboot the machine so you don't get it from the from the get go like you would with native. There is there is a jure card for actually making use of the native ignition API to have that come up right away with the kernel hard to find. But it's not currently the case. Slightly more complicated than we expect. I put that pull in the chat. I'm not sure where you wanted it in the documentation. So, or in the minutes. And nothing's happened with it in quite a while. So, okay, let's hold on. All right, I'll just pop that there. All right. Well, thank you so much for bringing this to our attention. And we will do our best to get these addressed and get the information to you. Post haste and thanks to your patience while we figure this out to very good things to our sound. Much appreciated. And thanks for coming to the meeting too. It's nice to have some folks shown up here that isn't the usual crowd. All right, moving on to the survey. So I did have a chat discussion with 3 T. Right before last. She has had other things going on and it's going to send me a copy of the survey for folks that don't remember this was something we started working on last fall, but things got sidetracked. And so we're going to get a copy of that survey revamp it and then send it out to our users to get a sense of what they're doing. And then we're going to be using okay D and how they'd like us to what they might look for an okay D in the future and how the community can grow and change to meet their needs as a community open source community. Any questions or comments on that. Any feedback on that. All right. We did the project updates. We're pretty close to the end. We've got task items in there so that we remember all of our tasks that would be helpful. I'll try to put in a couple too, but if folks who put in task items that will help us remember all the things that we said that we were going to do here. And I was just going to double check with Christian Christian, I have both you and Vadim down for the coupon gathering, giving an update is that is the team still coming or is it just going to be you. I think Vadim is going to be there. Yeah. Okay, cool. Then we can take him to dinner. Oh yeah, absolutely. And send him to a spa. And Brian in this, I will, I do still owe you the Dublin and the London dates for those and if anybody else is in the UK area. I think July 6 and June 20 something to gatherings there that I'm looking for okay the collaborators with Brian to give the community update otherwise it's going to be me doing the song and dance with Brian. By the way, I'm looking at that installer. There are quick segue since we're running out of time. That is a that isn't a co since dollar. I don't see anything in it that references route or our RHS whatever everything in there is is all Fedora F costs. Let's give this a shot if someone else can can can test this and maybe we can get a couple folks testing it and then narrow this down. That would be awesome. So we need some bare metal testing. I might be able to throw it on a machine and test, but if anyone else can that be awesome. Yeah, I don't have any way of doing bare metal at this point. Nobody but there's all do bare metal. On the other side of this. I'll try to give a review on that installer PR John just for what it's worth are the installer team has been under like a lot of pressure. They have they have not necessarily had a lot of spare cycles just to look at pull requests, but I'll try. Yeah, and it was a minor one for me. I was like, you know, I just happened to notice that because going through burning had to do the build and I happen to see that it wasn't passing the tags through. Because I didn't know how to build an OKD installer, you know, till the dean put that information out. So just happened to notice that that wasn't being passed down. Not a huge deal. You got the important without. Right, right. I just want to try and give a little, I mean, like, obviously you can't talk too much about it, but like just try to be a little transparent about like some of the statuses of our team. Excellent. All right, we've got a few minutes left. Is there anything else. That the folks would like us to talk about. Take a look at or anything else before we sign off. Excellent meeting. All right. A lot of good positives going for this meeting because the last couple of you know, we've had issues with releases and everything. So there's a lot of. Yeah, like today was today was good. We had a lot of good things going forward. Yeah. And I think we're a large part, I think due to Brian's documentation work, John, your work and Christian's work and everyone shipping and we're starting to understand the nature of the beast better. I think. We're actually like understanding O. K. D. and what's underneath this in terms of the build process in terms of the installer, et cetera. And I think that that's fundamental. Well, testing to I mean, looking at looking at the power results and digging into when they do the tests and seeing, you know, what in the world is proud doing and the results that they're getting. Digging to that is almost as bad as digging into a log into a log bundle. But it would a few times it gets easier to find out where that, you know, okay, this broken proud because of this. Oh, this is a known issue. We can ignore it or this is a new issue. That's, you know, let's investigate it more. Right. We should do a deep dive into the prowl artifacts. Yeah, sometimes. And I hope that with the with the change in build system that will will soon have with pulling it into prowl natively not shelling it out to service CI. This will be especially the compose or layering process will be even easier to to follow and debug and just take a look at in the easiest sense because yeah right now well and generally following those logs is and debugging them. Is is a hard task and takes a lot of time. Christian, who do you think is the best person if we were to have like a, like a session meeting dedicated just to exploring prowl. Who would be the best person to lead that. I think we have any any of your red hat employees here have have knowledge of most of them. I would say system and they would bring a piece to that meeting and the experience you look at different things for different components obviously and I don't want to like pull in a member from from the test platform team right now. Who isn't here who could probably explain it best. But yeah, because they are they also have a lot of things to do. Yeah, so we'll put it on your shoulders. There is a there is a actually have great documentation about the system and I think it's docs.ci.openshift.org. And I haven't checked right now but yeah that should be it and that is actually canonical documentation for that system and it's very helpful. I think instead of, you know, instead of trying to like look at the process to understand it like in abstract through like one presentation or a series of presentations more likely. I would say it's probably if we can come up with with tasks that we could do like a deep dive around that might be the most beneficial like so take something John just said, you know if we took the notion of like, I'm a community member and I'm putting a PR up. This is what I see happen. Right. How do I then investigate. What are these tests that failed which ones do I need to pay attention to. How do I find out where it failed how do I debug it you know maybe if we focused on it like that. We could probably easily record like an hour session where someone kind of leads it from the perspective of like, I've just put up a PR. And here's what happens next. I like it give a context. Yeah, for sure. And Christian would be great at like showing how to do all that stuff too. Oh yeah, in their time. Let's Christian and I can talk to some of the powers that be to and see if we can find someone between Christian and I, as opposed to just putting on Christian shoulders. So let me see what we can do with maybe to talk to Michelle creaky and others. And find some folks to help because I think training up the community on this is a huge benefit. So, let me see what I've got a couple calls coming up. I'll see if I can force them with some coaching to find somebody for us. Excellent. Thank you, Diane. All right, folks, I'm going to hit stop on the recording and thank you very much for attending the meeting. Thank you Eric and all of the other guests that contributed as well. And so next week at the same time is the documentation subgroup meeting. And then this meeting happens again in two weeks. Alright, thanks folks.