 you know, especially for developers, managers to understand that it's, it is in fact a valued service because this third party CIs, you know, occasionally crash and have to be fixed. And people always need to prioritize that work. So we want to talk about real quickly is just why we have it, what it does for you as an open stack user, and then encourage you to take action. And there are a bunch of references at the bottom of the page. Just on some general stuff. So we'll be having the PTG next week where we do the planning for the upcoming release. So that planning is the pad. I mean the PTG is open. So if you want to attend that's good, or if you have something you'd like to see the sender team discuss that would be great as well. I need to point out that I can't hear anything. So if you want to get my attention, you can just go like this, or use the raise hand function in zoom, which I don't know if that's working for me or not, either. Okay. But anyway, and the references so the PTG planning if there's something you'd like the sender team to address or something, you know, some kind of use case you've got that's weird or some kind of work around you've had to do that's unpleasant. It would be good for us to know about that. So if you give us some information on the etherpad, that would be fine. And also, if you could attend, that would be even better. So here's the link to general project information. It's the new contributors page but it's actually turned out to be a pretty useful thing because it's one place stop where you can see all the deliverables that sender project has. And also the communication methods we use if you want to get in touch with us and just general procedures. So it's kind of a useful place and then focusing on what we're talking about today. There's some links to the to driver information. So there's a general overall page about sender drivers. And then there are a couple of wiki pages for people who are setting up the third party CI. And then the how to contribute a driver page goes into. So if you're curious about what people have to go through to get a driver and that's a good place to look. So before setting one up, we have links in the somewhere in the how to contribute a driver page to Luigi, what's that thing called the audio host, I'm trying to remember software factory. So we were encouraging people to use soft the software factory distribution for CI I don't know that anyone's actually done that. There was a quick tutorial yesterday on getting started with Zool. So you can take a look at the recording from that. And that's another way to get some kind of third party CI setup. Okay, think we've got like a forum here so let's get started. Alright, so the main thing is, as you know, sender provides an API interface to allow you to use block storage in open stack. And all that stuff has to actually go somewhere. And that's in the storage back ends. And if you look at the sender logo, which I don't have available at the moment, you'll see that it's a horse, but it's the back end of the horse, because the back ends are extremely important because that's where all the stuff actually goes. Now the problem we have as upstream developers is that we don't have access to all the possible hardware that people might use with sender. And so that's where the third party CI comes in. So we've gone back and forth on this over the years, all the projects have that rely on different types of hardware is, do you keep the, the code for the drivers that interact between the back ends and sender itself in tree, or does it have to be stored somewhere else. And then when somebody wants to use it they go and get it and then they put everything together themselves. We've tried to come up with a compromise so to make it easier for operators. We do keep drivers in tree. But in order to have them tested, we ask that the contributors keep a third party CI system, so that the tests that we run in the gate with the community back end. So like Ceph or the LVM one, which is really only for tests, we don't recommend that you actually use it. All right, those same tests can be run with the individual vendor back ends. So what that does is it enables us to know that when we make a change to sender, we haven't done something that breaks the back end. And the vendors can know that, right, that's the case also. So it gives you a bunch, it gives you some reliability so the vendors are asked. One of the conditions to having your driver in tree is that you agree to run the third party CI. And on each change that submitted, right, this third party CI's will report back. And so that way, we have some kind of feedback about what exactly is going on in the whole sender ecosystem. So it's important for our project because we don't have access to the hardware. So the vendors do. So they host this. And it's important for operators because then they have some kind of confidence that the code we're working on upstream isn't going to break the kind of hardware they're using as a back end. So this is where we should introduce ourselves, which somebody might have reminded me, except I can't hear anything. Okay, so leading discussion today got me. I'm the current sender PTL Luigi Toscano who's, I'm at Red Hat Luigi's Red Hat. And he's a sender tempest plug-in core. And we've got Lucio Seki who's who works with NetApp. And he's the most recent sender core. So we have several perspectives. But the point of the forum is to try to get feedback from people who are open stack users. So what we'd like to hear is my pitch is I want operators to communicate with the vendors of the hardware that they use that this is a valued service and that they are glad that third party CI is running. So that they have, so there's a guarantee to the greatest extent possible, right that you can do with software that the code works with the back end that they have purchased. Okay, so my problem is if I stop talking I can't hear anything, but that's probably what I should do. So it looks like so from the etherpad anyway it looks like we have no operators here, which kind of defeats the whole purpose. So if I'm late, please take a look at the etherpad and add your name to it. Otherwise, we could have our sender weekly meeting right now. If that's all. So can we get Brian dialed in? Use the call in for zoom. All right, well, let me see. That's so there's theoretically I can use the call. All right, I'm going to shut up. I need so I need somebody else to like fill in time while I try to figure out what the call number is. It's in the zoom chat. Okay. Gotham put the number in the chat. Everybody pasted again. How hard it is to like get his attention to like say where it is. You can't hear you. Yeah. No, yeah, like taking him on IRC would help. Oh, I think it's something on his end. I'm not sure exactly he was having issues when he was trying to connect originally. We suggested he restart his computer, but he was like, I don't think that would help. Nice to meet you yesterday too. Can you hear us now. I can hear, I can at least hear Jay. Yeah, let's check, check, check. All right. Yeah. Hearing. You want to get real zoom fancy, you can link your phone number and your video together so that it's, it doesn't also have your phone number in the list, but like, that's, I do not want to get fancy. Thank you. So what I'm going to do is as soon as this is over, I've got to remove Zoom and regroup my system. Because I had a really bad problem, anyway, it's a long story, and it has nothing to do with third-party CI. So let's do some forum action here, people. So this is much more like a typical forum session that I'm used to. So on Monday we had like 127 people show up. So I think a lot of people came to it just because it was the first thing listed, which was fine. But this is a big letdown. We're 100 participants short, but we can make up for it in quality. All right, there's some action on the Etherpad. I can make you the host, Brian, so you can share the Etherpad if you want. Oh, okay. I always find it easier to share the Etherpad separately because it's hard to read on the Zoom, I always think. So I like to have the Zoom where I look at all the people and then the Etherpad is working. Okay, so I should at least... Can you put the link to the Etherpad in the chat? Somebody? Sure. Sorry. Thank you. Let me beat me to it. Okay. Okay. All right. So while we're waiting for everyone to get up, courage to participate, what can attendees expect to learn, supported versus unsupported drivers? All right. So here's the deal with that. So as I said earlier, we have the drivers under our entry. So they're part of the code repository. And so when Cinder gets packaged, they're included. So part of including that is running the third-party CI. So for a driver to be supported, it's supposed to have an identifiable maintainer, but it's supposed to also have a CI that's reporting on all the patches that are being proposed to Cinder so that we get feedback. Now occasionally, the third-party CI crashes, people are having trouble getting it started back up again, hasn't responded in a while, so then the driver can be marked unsupported. So what this means is that in order to use the driver, you have to set a flag in the Cinder configuration file. And so that notifies you as an operator that you're working with an unsupported driver. Unsupported drivers are eligible to be removed in the next development cycle. So if you're using an unsupported driver, you do want to communicate with your vendor as soon as possible to find out what their intentions are. We've been carrying some unsupported drivers as long as they don't break our gates. But the problem with an unsupported driver is that you're not getting the third-party CI running, so you don't know for sure that changes that are being made to Cinder would not, in fact, affect that driver. So it's really important to keep the third-party CI running. And in fact, I mean, you know, it's like, Cinder's a good community. I mean, most of the vendors whose drivers have gone unsupported have made, you know, changes within the next cycle to get the CI running again. It's usually a CI problem. And so with that, recently, we've had a case with the fiber channel zone manager driver where Brocade decided to stop supporting it with the train release because they didn't want to upgrade to Python 3. And as you know, Python 2 has gone to end of life, and OpenStack is now all Python 3. Gorka recently fixed the bugs that were preventing the driver from running with Python 3. The driver's unsupported, but we're carrying it along. If somebody's using that in production, it would be nice to talk to them to find that out. We'd like to get third-party CI. So what we're doing right now is on a best effort, we're carrying the Brocade fiber channel zone manager driver, but we've only run CI with the first release candidate for Victoria. So at that point, we know it's working, but we don't have the ongoing CI that the other drivers have to keep us, you know, to keep it safe. So just so you're aware of that. I mean, I think everybody understands why the third-party CI is important, right? We're working with software, and there's no telling what changes we make, might have consequences elsewhere. And the way to know that is to actually do the testing. And so that's what the third-party CI does. And just one thing about the third-party CI, it's called that, I mean, the third party is the vendor who's supporting that driver. It's not like there's some kind of third-party, you know, neutral board that does the certification. It's just a service that's run by the vendor of the driver to make sure everything continues to work, right? So I'm obviously just filling time here. So I'm going to be quiet and let other people and my dog wants to get in here. So did you do an audit recently, Brian, of the CI's? What is the health at the moment? We be quiet. My dog is entering in here. I'm sorry. OK, so yeah. So the third-party CI situation is not that good. So it's kind of mixed. We have a few CI's that all the time. And then we have the rest of them that don't. So I didn't go ahead and push in a lot of. So the two things with the worldwide pandemic situation, a lot of people have been having trouble getting into data centers, get the CI fixed. So I thought that was so anyway. So I didn't actually do the audit and start kicking people out. But I'm going to have to do that before M1. Or we can talk about it at the PTG next week. It has. If you look at the well, when I looked at the there's a link on here for the. I think. Yeah, for the current CI results, yesterday they were all messed up because Garrett was crashed. So I don't know whether that's stabilized or not. But if you scroll through, right, the success rate is kind of miserable, actually, I don't know if that answers your question, Jay. Yeah, you know, I so. There are a couple of different ways this discussion can go. It's not really going anywhere at the moment. So so let's let's just spitball some ideas. There, you know, you have the people who have said that third party CI was an idea. It was a good idea. It hasn't worked. Do we. Give up. So there's a double edge sword. If we give up on enforcing. CIs, nobody's going to do them. And I think there's at least some value in still having test. That shows that those vendors are involved in our testing their code. It looks like from the ether pad, there are at least a few people here who are users and users. So is there anybody who who feels better that we have the third party CIs and that their drivers are being tested by the vendor? Because that's one question is, does it mean anything to the consumers? Any of the consumers want to speak up, that would be awesome. Jay, if you're talking about the three people in the end user or someone who doesn't fit into the above. Yeah. So they are all from that up, but they are working for Manila. So yeah, they shouldn't be users either, right? OK, I was hoping maybe they were users, but but no, OK. I have a feeling end users right now probably wouldn't care too much until their storage of choice stops working right. Yeah, I think I kind of agree with that. That was part of why I wanted to just surface that we have this third party CIs and that people should worry about it. But I wanted to check about a couple of problems that you mentioned, Brian, one is, yeah, I mean, right now, you know, getting access to their infrastructure is really hard, etc. So I think that is I think the biggest concern even I've heard in Manila. So technically, we, you know, as long as they're able to post results that are off their DevStack that they have, you know, that they've run against their storage system and stuff, we've been fine. Although, I mean, we've not had the policy that we remove back and sorry, mark them unsupported in Manila at all. But there have been times when they've when vendors have tried to push fixes, but their CI systems not working. So we've been OK with them, you know, running tempest tests on a, you know, on a test system somewhere else. That's not the CI system and just posting it on the on the review and such. Do we know of other projects? I was going to say that the people who. I mean, I think from all the patches, I've approved the cycle. I mean, that the CI. Well, it wasn't responding all the time, but we at least got a CI result on the actual patch. So that is kind of a plus. So it really seems like it's it's been more of a, you know, this situation, keeping the CI running. Just in general, as like a preventive thing. Yeah, and we discussed things in the past, at least in the similar context, right? That we instead of having per commit testing, we'd, you know, put up a patch and have them all vote on that patch periodically and such. So have have those been, you know, shelved? Have those ideas been shelved? Because I remember that there were patches by Sean that said, yeah. So there were patches by Sean, I guess, a while ago that, you know, all CIS was supposed to pass on this because this is a no-op change. And then there was a patch that broke of send a volume manager and just said all CIS are supposed to fail on this. And and there were no, I think so. Yeah, I haven't done any checks like that recently. It's been more just, you know, following the per commit testing and such, yeah. Yeah, so it's mostly, I mean, so we've been able to pretty much make sure that for any of the changes that have gone through the CI has run for that particular change. But yeah, I haven't done it general. I mean, you can see from the results if we did just a general patch, you know, patch that should pass or patch that should fail. We have mostly not get responses. I think right now it's we're at the situation where we haven't merged code without knowing that there's been a CI run, but we don't have the preventative nature of, you know, ongoing CI for all changes. But the wider question, so that that's one thing, is just getting the current tests that we have running. There's also the question of, you know, how thorough the test coverage is. And it's always better to have more tests. But there have been some changes. We'll talk about it, the PTG that haven't broken anything. Well, that's not true. They have broken stuff, but it hasn't been detected. So the third party CI, it does give us some confidence, but unless it's ongoing, right, but if it's not ongoing, it gives you less confidence. And then with the Cinder Tempest plugin, as long as those jobs are being run, we can add tests and then increase the test coverage across the board vendors. And it doesn't really change much on their end, as long as they're pulling down the latest version of the plugin and running those tests. So, I mean, my my feeling is that it's an overall nice system to have. But, you know, it entails cooperation on the part of everybody participating. As you mentioned, Cinder Tempest plugins. So we discussed this many multiple times, if I remember correctly. But, for example, I think that the number of third party CIs, which are also running tests from Cinder Tempest plugin, is not so high. And, I mean, I don't want to get to in the technical details, but we also have the problems that many of the CIs, at least looking at the jobs that are running. So when you can get to the logs, because in some cases, you cannot easily. I mean, the logs are not easily accessible. This seems to be the old legacy jobs, not native sole jobs and that shows. And sometimes you can even easily find the tempest results or whatever other tests they are running. So it's not clear that the logs are not as complete as the even the legacy jobs that we have, we had on the main CI. And I'd like also to stress the fact that highlight the fact that the legacy jobs are probably going to break at some point in the not far future, because we are trying to deprecate them in general. So and that means it's not the problem with legacy jobs. The problem is that we're going to try to deprecate DevStack gate and all the legacy jobs depend on DevStack gate. So this is probably even though most of the jobs have been ported for the main CI in the during this cycle, we just have a few leftovers, probably for still right now, DevStack gate is going to stay. But at Wallaby, it's probably going to disappear. So that's another call for the third party CI, CI's maintainers to take a look and try to fix some of the issues. Now, I know we discussed about this many times in the past and I don't really have a solution, but it's really time to, I don't know, try to have them to contact them for real in a way that that we can have some kind of results and try and that may be the time to see what, which, which problem are they encountering, apart from setting up Zool itself, but that's really the time to check again the general status. While we had a testing group, this was like before I was working at Cinder. So in the wiki, there's traces of there was like a CI group that had meetings occasionally. Maybe we should resurrect that and just because you're right. I mean, as the infrastructures are aging, right, they're getting less reliable and pretty soon they're not going to work at all because the stuff they depend on isn't going to exist. So we do need to spread that around. Are any of the vendors you're interested in doing something like that? Or sharing their notes on how they've, you know, upgraded from legacy jobs to Zool V3 or they don't use Zool, they use something else. But it would be nice to have some vendor publish what they do. I think that was something in the past from the Cinder community. Yeah, there's some stuff in the wiki, but it's starting to get kind of old. So yeah, it would be good to revisit that. All right, well, I think that's getting close to time. I think there's a PTG. Let's discuss this and see how much interest there is among the current Cinder team to host some type of meeting and then get the word out to vendors and see what happens. So I think the kind of general consensus we're coming to here is that we don't want to stop doing third party CI. We know they're not reliable, but it's at least a litmus test as to whether that vendor is participating and that the driver is cared and fed and that, you know, let's maybe just put a little time into seeing if we can rally the troops. I think one of the biggest problems is the fact that we've never had really. Well, it's hard to do and we've never had great documentation explaining how to do it. So maybe we can talk about that more at the PTG. I mean, one question I have is what is the alternative? You know, what I'm getting from my dev team is that they would prefer to have something they could do in their own infrastructure rather than out. But if it's not as good as the public infrastructure, then that's a factor to take into account. Well, for the third party CI, it's actually is running in the vendor infrastructure. It's just reporting the results back up to the community. Right. No, I understand that. But but the alternative is to take your stuff and go run it in the publicly hosted infrastructure, right? Or do I misunderstand? Well, no, because this actually runs against the vendor hardware, not against some sort of simulation. Oh, well, then, yeah, we absolutely need this. Yeah. So the alternative is Wild West and we don't have anybody test. So but, you know, I think I believe ironic still looks at their results. I would assume Nova is using it. I'm not sure, though. Yeah. I mean, just because you don't enforce that everybody runs these tests doesn't mean they don't do it. Hopefully they do it. But to have visible results is more of a discussion, right? And enforcing the visible visibility of those results and success of them. Yeah. And by the way, I haven't met anybody just want to say hi. I'm with Kyoksia and we're in a weird position because we're we don't make servers, we make the storage devices. So when we think hardware, we're thinking about, you know, CM6, CM5, you know, individual. NVMe devices. So there's a subtlety there that even goes deeper, actually. Gotcha. And interacts with what PCI-Gen and, you know, what your network bandwidth is and yada, yada, so it's. Yeah, testing to the factor for us. Well, since nobody's screaming, I think we're at least doing the right things. And the fact that, you know, we watch the results where we don't come out with a baseball bat, like we did at one point, we at least, you know, come with a pool noodle and kind of tap people and say, hey, are you paying attention? And it seems like things have resolved themselves in the past. So that's good. So, yeah. Sorry. So one thing that I believe that is that even if the third party is not enforced anymore in some in near future, it's interest from NetApp to keep running those tests because it's very useful to avoid issues in production, which is much worse than having to fix the CI. And one thing that I noted when answering questions from new vendors joining Cinder project is that software factory seems not obvious to install and configure. And talking about ourselves at NetApp, we still run legacy jobs with Zulu V2 instead of Zulu V3 and we will be impacted when DevStack gate is deprecated. And we need to prioritize the tasks necessary to migrating it to Zulu V3. So, yeah, it would be very helpful if someone starts deploying software factory and tell what are the main issues on doing that. The guy who talked to me last time was Nomura from Itachi. So he started trying to deploy software factory, but it was too complicated for him. And then he used it a bunch of scripts, I think, as well as playbooks called SOS CI. It's an old script from John Griffith. Yeah. And yeah. And it's working so far. But if DevStack gate is deprecated, he will need a lot of time to migrate his recently deployed CI to Zulu V3. So we, unfortunately, he didn't join this meeting today. I think it is Japan. So it's complicated. So when is the DevStack gate being deprecated? Now, I mean, the original plan was to deprecate it now at the cutting of Victoria. But that's we still have a few legacy jobs left. So it's still working right now. But for Wallaby, that should be the cut date. Of course, if all the third party CIs are going to be broken by this, maybe the TC will say, no, let's wait it for a bit while. But let's not take it. I would say let's not take it as an excuse to delay the work. Because it's I mean, if it's going to take sometimes, it's not going to take just one cycle because it's going to take more than that. So sooner it starts, the better it is. Just a question about those things about with software factory, but did the people who tried to set it up, did they try also to contact the software factory developers? Because so far, whenever I, I mean, I didn't have to set it up myself. But whenever I asked them for details and information, they always said, yes, please contact us. We are ready to answer questions. We they I think they still have a pending patch to improve the document. There is already some documentation on how to set it up and integrate with the OpenStack to create a third party CI. But they also have the pending review to extend the documentation. So it's kind of like they are waiting for feedback, but if the feedback doesn't come, they cannot improve and so on. So it's yeah, when I talk to Kai from Hitachi, he already had deployed the the Zuv2, so yeah, it was too late. So Gorkas, Zuhens on series a tutorial of how to deploy it. So yeah, yeah, it's basically you have a look and it didn't help. It will certainly be very helpful for me and I can talk to him again and advise him to start looking at it as well. Because yeah, his CI will stop working sooner or later. So yeah, thanks for this. Yeah, maybe we can after you try it and if you encounter problems, maybe we can get in touch with the developers and maybe have a session with them to to see what's going on or something like that. Sure. Yeah, and we'll have Chuck's team setting up a brand new CI. So we can hopefully get some feedback from them. Whose team is that? It's the Keoxia team. Oh, he's right next to you in the Yeah, you're right next to each other. Yeah, we don't know anything. Forgive our ignorance. I'm trying desperately to catch up. OK. It's the stinky Zerilla question. And as usual in developer land, this is a very deep question that led me to this whole world of the new icons. Oh, my God. OK. So would would you be so at one point, Sean had started a and we may be over time and we could cover this in the PTG. I don't know. Yeah, I think so. So something to maybe discuss. I know at one point in the past, Sean had started a process to try to spin up his own CI system just to see if we have the documentation and the information to do it. And I don't know where he ever got with that. But, you know, it might be an opportunity here if you're bringing a system up, Chuck, and you're looking to get involved and help that we could work with you to update the documentation accordingly that we have out there and kind of come up. We've been in the need of coming up with a repeatable documented process for bringing CIs up for a while. Well, it's the one thing I can do is right. So. OK. And then if we can, as a community, I know you, Lucio and Luigi have done work around this recently, can help support him to get him there. Then I think that would be a good step in the right direction. I mean, I mean, what we're bringing next week is actually an architectural question. And we don't know OpenStack well enough and really will rely on your guys's experience and wisdom to guide us on how to do what we're trying to do. But we do, you know, the essential issue is we do resilience from the client side. You know, we use a little agent that configures MD raid on the host computers to write to multiple targets in parallel on the back end over NVMeOF. And we're just trying to figure out how we bring that into the OpenStack setup. Cool. Yeah. So we'll we'll just bring our best guests and have you guys guide us on what we should do. Yeah, that's that's what we're here for. Yeah, cool. All right, should be good. We're working our deck. We're going to run it with Brian tomorrow and, you know, hopefully we'll be ready. But yeah, the CI will be the next thing. Cool. OK, I think we're over time, so I guess we should stop. Thanks, everyone, for attending, although you just been us. Thank you. Yeah, so put notes on the iPad if you have questions to follow up. That'd be fine. And thanks, everyone, for attending. Enjoy the rest of the week. And I think I'm going to see everybody next week at the PTG. And I love the donkey. I just think that is so cool. All right, we're proud of it. Cool. All right. Talk to you guys later this week or next week. Thanks, Brian. Oh, thanks, everyone. Thanks, Brian.