 Hi, everybody. Welcome. You may have noticed that this week we have a bit of a theme around highlighting participation and learning how to participate more effectively. So one of the things that we wanted to do this week is bring in some organizations and contributors at those organizations who have been highly effective at participating in open source projects and talk them a little bit about how they do that. How does it work out well for them? How does it work out well for their organizations? And what are their motivations? What drives them? And what makes it successful? So we have a few panelists here with us today. I'm Allison Randall. I serve on the board of directors of Open Infrastructure Foundation, as well as Open Usage Commons and the Software Freedom Conservancy. First, why don't you introduce yourselves? My name is Kansiyamman. I work full-time in upstream, especially in NOAA, API site stability, QA, then helping technical committee. I'm also involved from NEC. Hi, I'm Mohamed. I've been involved in OpenStack for quite a few years now across several different roles. And I lead a company that kind of focuses on providing infrastructure solutions with the open infra projects, mostly. OK. I'm Luis, and I'm an OpenStack operator at CERN. My experience with OpenStack started around 11 years ago. Here I am, so it's really nice to be here. Well, to get started, tell us a little bit about how you got started in open source and what motivates you to stay involved. Yeah, so I started around 10 years back. So one thing I really liked in open source, like Azure Developer also, you grow a lot. So you get a chance to work with a lot of people from different countries, different culture, and obviously the unlimited technical knowledge so you can gain, and obviously you can help the new people to gain the same. And overall, if you are contributing in open source and you are building the relationship that you can convert in your organization to help in existing business or like attracting the new business also, so that's what motivated me and that's from like I started and continuing in the open source since then. Yeah, for me, my organization, Next Source, we were kind of looking at finding a way to provide cloud. At the time, it was very early on and we're looking at this OpenStack thing and we realized that everything we were doing is exactly what OpenStack was doing at the time. So we said, okay, why don't we just join them and work together with them? And so that's what we started doing maybe 11 years ago by now. And it's a very interesting journey because once we started, we realized that it's such a fun collaborative community in general. So for me, I've never had open source exposure before OpenStack, but it's very interesting because you first of all get to get access to a lot of very knowledgeable individuals so you get to learn a lot from them and it's really fun because sometimes when you're doing something inside of a silo, it's hard to find help outside of it. Let's say if you're working on, let's say a closed source project or the internal team, you just have you and your team whereas with OpenStack, in my case, there's so many people around us and you can bring it up and somebody else would have similar issues. And then it's really interesting because you gather this knowledge and you learn more. And so it's funny, I always say this, my first contribution to open source was actually a typo fix in Nova. And back then, Nova was so untested that it was like the function argument name did not even match the variable name used in the function. Like that never had passed any tests at all. But it was like that kind of, you get that and then you kind of work with someone that helps you land it and then eventually it kind of motivates you to do more and more inside that. In my case, at the professional level with open source, I started as well when I joined CERN in 2012. Basically, it was hard to work in the adding support or deploying hyper-recomputed nodes in our cloud. And this was an amazing experience because I got that the personnel you get to collaborate with back in the day with Peter from Microsoft or Alessandro from Cloudbase, you meet them, you create new and you give your feedback, you design how the tool is going to work, et cetera. So that is very, very rewarding. And then I was lucky as well because back in the day it was when this agile project at CERN started. So it was this project where we were adopting many open source solutions like OpenStack, Puppet, Form, and CERN. So basically we were moving all the IT department from in-house software to more open source communities and being able to experience that transition how the culture of the department towards open source change and adopted all these projects is something that over time you see the benefit. And this is obviously at a personal level where I think this was useful because as they were saying you've rewarded something that you like, you contribute, you feel fulfilled in your profession and that is very positive. But then you can see the other side, the side of the organization that you trust a technology like OpenStack to be a core tool in your infrastructure. So you obviously want to make sure that that community is healthy. So you want to contribute with code, things like that. But also with doing outreach. So if it's something that works for you, you show others how they can adopt it. So they can join the community if it's something useful. Then you think for the organization is positive to leverage all the community because instead of doing all the things downstream you move all your modifications to the community and then you have this critical mass to move your things over time to new releases. So it's something that's beneficial for you. And same thing with, I don't know, debugging fixes or getting input because something that, in this transformations of the Sanity department with all these tools, something I think that it works is all the feedback that the different communities or OpenStack, Puppet people and Chef were getting while going to conference, et cetera. It's like, oh yeah, these people that are doing this here and there, maybe we should change the way we do things. So you get all this knowledge from other places and I think this is something very positive. And the last point, because I think there are so many points that we could be talking for half an hour or just about them, it's like selling some public organization. So it's, we are funded by public money. So one of our, let's say, goals or missions to give back to society. So one of the ways of giving back to society is contributing to this open source project that then can be adopted on the different countries that they are funding us. And also, part of our mission is to train engineers and professionals. So we see, we get all these people coming from the different countries to share in the case of IT with maybe a basic kind of Linux profile. And then they get to start interacting with these communities. They get skills over time. And then when they leave CERN, they leave CERN as a train engineers in these different fields. So they have more chances of getting maybe a more high-skilled job in the community with maybe knowing already some people. So this I think is very valuable. That's great. You talked about contributing upstream. So let's dive in a little more to some of the ways that your organizations participate in open source projects. And starting with the simple one, bug fixes. Kind of how do you view bug fixes contributing them upstream? And the value that your organizations get out of doing those that work upstream? So that's a really important thing. Because if you see software, you see it has been developed initially. They do the basic things and features obviously keep going. But bugs are the important part. One thing, we cannot test everything in upstream. We don't know every use case in upstream. So filing the bug is very important. And then how you fix the bug in upstream is a very small, small help, but it contributes a lot. And in my company, I have seen a few of the team like we're facing customer. And they know some of the bug, as per their use case and all. So they keep coming to our upstream team and they're saying, in my company, upstream team, we have a group of people. So they say, OK, this is the bug and we want to fix it. And sometimes we directly go and fix them. But sometimes we tell them you go and file the bug. You talk to the community. If you need help, what is the process of fixing the bug and getting the review, then we will help you. So that's something we think is really important and spread that people facing either a customer team or you're from a different team within the company, you start contributing with the bug fix and bug filing. Yeah. And I agree with that. And I think what's really interesting in doing open source is if you've worked with, let's say, a close source commercial product, it can get very frustrating if you have a more technical background, because you'll see an issue in front of you, but you're just kind of helpless in how you can fix it. You can't do anything about it. You can't change it. You can just file an issue and hope that somebody gets to it. The nice thing with open source is that when you're running to small issues or bugs, you can go in and look into it yourself. You can go in. And what's great is it's not just code, right? There's history behind the code. You can go and look into it, get blame, and see exactly why that line changed or when it changed. If you've upgraded OpenStack and saw behavior change, you can go in and see the commit and the exact time that it was done. And then you can go and look at the review and see why the discussion was done and why it was done the way that it was done. And so I think that's really powerful when you're trying to solve bugs as well. So that's a really, really important thing. And by being part of the upstream community, you get to be part of the team and maybe you see a change. Like I might, for example, John Schoen works a lot with the RBAC things. And if I see an RBAC related thing, I can reach out to him and say, hey, we were looking at this thing and we think this is a problem here and what are your thoughts are? So you kind of get this extended team that you can also talk about bugs. And I also think it's interesting at the OpenStack PTGs where the project teams gather, I've had so many scenarios where you'd be sitting in a room and someone would come in and bring up a bug and then the team would say, actually, we've designed it in that way and you need to do ABC instead. And it's really fun and interesting to see kind of that moment where people are like, wow, okay, no. I understand why you do that now. And so you get a lot of that knowledge. And yeah, so fixing bugs is easy. Filing bugs is just as important. And a lot of us, you know, when we're troubleshooting, we're going through the code and sometimes you can pinpoint exactly where the fix is. And some folks just make that change inside their environment. But I assure you the delta of extra work that you might need to do to contribute it upstream on that time that you fix that is way less than the delta of maintaining that thing forever because you just start to get, you know, you don't just edit the code once, you have to keep maintaining it. And so by pushing it up, first of all, you get two things, right? You don't have to maintain it in no sense. But secondly, you might actually get an approval telling you that you're doing the right thing because you've got core members of the project that of years of experience that'll say, oh, good catch. Or they might say, it's nice that you caught this, but in two years, this is gonna break horribly when we do this other thing. So it's always really good to bring those things upstream because you get that feedback that you wouldn't be able to get if you're doing it in a local or kind of on your own. So really important to bring that knowledge up and kind of consider the upstream teams as like an extension of your own team and your own work. I think I couldn't add more than that. But I think the important is that in this case, because sometimes I think people that have this idea that contributing is just writing code. I think it's important that something as simple as proposing or reporting a backup screen can be valuable because yeah, a race of awareness that something is going on. Also even if sometimes you can feel frustrated because maybe you don't have the time or the experience to maybe tackle it and implement it. But maybe sometimes you can just, as it's just given feedback while it's implemented because in my case, as an operator, maybe I know have some knowledge of the implications of the bug. So this is something that happened recently with something that we reported in Manila, the Manila developers they fixed. And I was keeping a look to the review and I see the way they were implemented could affect the behavior in this case. We've saved behind the scenes some default behaviors so if your input, you didn't implement the fix but you put your small feedback there that will help this to be a more solution that yeah, it will work in the future as well. So I think this is something very valuable to do. And just to add a little bit to that, I think documentation also is super valuable, right? Like I bet everyone here in the room has a confluence or an internal wiki with a whole bunch of stuff pasted in there of like when this happens, do this, when this happens, do that. And it's almost like if you take that knowledge and kind of bring it up, you can start sharing it with other people. And then you don't have to maintain that page. You can just say when this happens, go to the documentation that we've helped, you know, push some changes into that. And how about code reviews? Do any of your organizations participate in doing code reviews for other people's patches? And how does that help you and how does that help the projects? Yeah, I think the same thing when you do the code reviews, you get involved into that discussion, right? So many people come and they tell like, okay, you might see like, okay, at first stage this code looks good, but when you have that discussion start going, then you realize, okay, there might be that use case. And what I am thinking like, okay, this code can just work fine in this requirement, but there are a lot of other things also. And that code review, that getting the discussion going on and all, that also help when you do any bug fix or feature implementation or any code thing, right? So because you get to know the complete knowledge of that surrounding of that code change impact or any like regression it's going to cause. So code review, I feel like, I mean, when I started like in Tempest and NOAA, more than that, just pushing the code, my main target was doing more and more code review. So that helps you to understand the code and more than that, like the discussion and the reasoning like why it has been done that way, why it has not been done that way. So that gives you more like knowledge and understanding how you can use it in a better way or how you can fix and implement the feature in better ways. So I feel like that's as important as like fixing any code or implementing any feature. I agree with that. And I think code review is nice and important when you're like getting told to learn early on, you get to see how people are implementing things. For example, I've learned a lot about how to, for example, introduce tests that are purposely failing in one patch and then actually implement the fix in the next patch. And I would have never kind of thought about that but being that exposure and seeing how that happens is really, really helpful. But I also feel from another sense, in a weird way, I kind of see it as some sort of karma, right? If you're gonna be pushing up code all the time, you need reviews to get your code in. Well, you also need to do a little bit of your part to do your part of reviews so other people can get their code in. We need the back and forth because if we don't have that, we can't get things through eventually. And I think also the fact of participating or even being exposed to the workflow of reviews, et cetera, is something that at least at CERN, you see that it has become also a learning exercise because you end up learning for all these experienced people from the community. It's something that then you bring home to your organization and you see internally that this culture of reviews, how it works upstream, is something that then you can implement on all the internet workflows for all the tools in your organization. And not only that, also learning from the experienced developers in the way they structure code, they propose the different features. We have some tools internally that are very specific to CERN but you see that there's a lot of inspiration of all the experience gained contributing upstream of all the different OpenSTAD projects and I think it's valuable to get this experience from all the community and then you bring it home and you use it for all different purposes. So I think that's a good thing as well. And one important part in that is like, don't think always you have to do complete code review of that change. So it's always like, because nobody knows everything. So even like as a operator also, you don't need to validate that code with 100% guarantee. If you can just think, okay, in upgrade, this thing can cause the issue. So you just comment that, review that part only. So that is also helpful. The partial review, the something that you can verify and say, okay, this thing has been changed. So this is all good from my side. Other part I have not reviewed. So those kinds of things are always helpful to do like partial review also. Because sometimes we think, okay, this has been code done, a lot of things. I don't know these things. So let's not review that. Don't think that part. And I think the nice important thing to talk to that is that review, we're not necessarily in code review. Sometimes it's review of the thought or the idea or the process. And as maybe a non-technical contributor or mostly an operator, for example, you mentioned the Manila case. You might have, I don't know the details, but maybe you didn't actually look at the specifics of the code, but you were more thinking about the overall concept of how it's gonna affect all of this and bringing your feedback and saying, I don't think that this is gonna be good. And I think the nice thing with doing that is because of the open source communities in general, value that and what they do is you can't possibly review everything. But what happens is when you are an active reviewer, they will reach out to you and say, hey, we know that you do this and we're thinking of doing that. So could you give us your two cents about that? And so by being a more active participant, you get a chance to kind of get called out, let's say, and said, hey, can you give us some insights or thoughts about if this is a good idea or not a good idea, would you use this? Would you not use this? Things like that. That kind of leads into design discussions. So that's sort of on the spot, but do you find it valuable to participate in the advanced design discussions, like sharing your use cases, looking ahead to what the project should be over the coming years? Yeah, and that's where the important part, even the R back, I can give the example. So as a developer, we think this can be done in that way. This we can implement this way. We can give these the option in this feature, right? But like when there are a lot of use case and a lot of requirement, right? And it can be used in very different, different ways. So getting that design, like the feedback in early stages saves a lot of time. And our back example, I can give a very specific, like initially we thought of like giving you the kind of a isolation between system level access, versus project level access, versus domain level access, then reader, admin, a lot of things. And we started the implementation, right? And we had at one stage, we were saying, okay, the system admin, system level isolation, all is ready and you can use it. Then we got a setback that, no, we don't need that. They break us, so what we needed exact this particular project reader and all only. So then we realize, okay, we have to go back and change the design, change our implementation to make it more useful and not to break the operator. So that's really a good example and experience for us, like having the earlier design reviews, pack reviews, the idea reviews is very, very important. And that's where like, from the upstream community side, also I request operators to keep engaged with us, even on OpenStack Discuss Mail English, IRC, even you don't want to review the pack or something, but you still can give feedback. And that's where like we have the mix, one example is Mohammed, like he's always on IRC there and something we are doing, he keep stopping us, okay, this can break that, this is, should be done that way. And that's a really good, earlier feedback we receive from any operator and that's like as important as like we spend time and save time and it will be like more useful if operator, it is more close to operator requirement. That's how it should be and that's where the spec review is important. Yeah, I think that I kind of have like two approaches to this, I think that if you feel like you want to be involved in a design discussion, be there, the community is open, there's no registration or anything as long as you're in the event, you can open the door, come in and sit and there's no obligation or anything. But I do find that sometimes it could be very intimidating because you feel like, for example, you're sitting in like a, I don't want to call out Nova, right, but Nova is a very complex project, you come in and sit in a Nova design session and you might feel very like, we're overwhelmed. But I always think that by going and still just sitting there and just listening, that could be very, very useful because you'll just, you know, you'll start to see how the thought process goes and some of the things that the different members are paying attention to and there will be something that you can bring to the table. Like I really believe that no matter how deep you know all of the depths of Nova code or how little you just use it as just a package you install, I think there's something you can bring to the table to that discussion because nobody knows the whole thing. Everybody has different components that they can bring to the table. And so, you know, rather than just saying this is gonna be too advanced of a topic for me to be around, if the topic looks like it, you know, it could be relevant to you, just sit and listen. If you don't, if you aren't able to contribute anything, if anything, you'll get a lot of very useful information out of it. And then potentially the next time around, you know, you'll probably say, okay, well, I feel like a bit more comfortable with this subject and maybe I can jump into that. I think this is a good point because in particular these days, if you yam today into OpenStack, that at the end is a project that has been running for many years. So, yeah, you might feel overwhelmed with all these projects and history and people. So, I think it's good for people to have this mindset that when you join the project, it will be some time that you have to get to know all the bits and pieces, how things are evolving, what is the direction of the different projects, and then over time you will be confident enough to give feedback, to contribute, to find your area of comfort among all that landscape of projects. But coming also back to the question of participating in design and features. I think this back to one of the points I mentioned in the beginning. So, at the end, you have this project, you trust it to be used as a core component of your infrastructure. So, I think it makes sense that you wanted this to make your needs visible to the community. So, maybe this way, maybe you will get feedback from others, raise interest on A or B. And I think we have plenty of mechanisms to discuss all these different ideas from PTGs, IRCs, blueprints. And yeah, I think it's something positive that people contribute. For instance, at CERN, we have something like this sales between NOVA that I remember, Vermeer was participating and giving a lot of feedback and we implemented many features. It was something that at the end of the day, it allowed us to scale to the size we are running these days. So, I think this can be very positive. So, we're pretty close to the end of the time. But the last two things I want to cover, one is participating in things like contributing CI resources and developing tests and things like that. And then I'll come back around the last question around leadership roles, because I know some of you have actually served as project leaders and board members as well. So, testing, releases, QA, that sort of thing. Yeah, that's a really important part for community because they are not like the things which people directly use in their production or get the money from customer. But you have to realize like how important they are for the software development in community. So, myself like being a QA person, what I feel like if we don't do the QA in upstream, how unstable that code can be. And you can see how much effort every say vendor or distributor or operator, they have to spend in their downstream testing, like everyone duplicating the effort and all. So, QA is very, very essential thing for open source, especially where you have like a lot of people calibrating with different ideas and all. So, you have to verify things at the central place and where like that code is being consumed in like say private cloud, public cloud and distributor and all. So, you have a lot of use case. So, you have to test with the certain like stability. Okay, these are the things passing all these set of standard tests. So, we are making sure there is like the part of a basic, I can say the stability. I will not say like it's 100% stability it gives, but definitely that part and then release is obviously the always the background heroic effort you can say releasing like, we have around more than 600, 700 deliverables in open stack, release team do the release and all. Then requirement infrastructure CI CD that is not easy to build and test. So, all these we can say the support team are the pillar for this software development. And in community like we used to have a lot of people contributing in that and helping, but nowadays things has been strong. So, we also like if more and more people help in those. So, that thing help indirectly to you only instead of like you might not be using them directly, but that's the base like of the community. I feel it's very important. Yeah. So, I think I want to add up a little bit to that as I vex those part of one of the many organizations that provide infrastructure to the open and for community. So, one of the things that maybe you could very much do is if you have a cloud and accessible and you have some extra spare resources and that's something you can look into because obviously these resources are being used to test open stack and that could be a very easy way to be able to support the community without having to let's say do too much man hours or something like that. Essentially, the team just needs the username and a password and an endpoint and they can make use of that with Zool and everything. So, perhaps that could be a way that you can think of contributing by just providing some cloud resources for all the QA that Gansham just talked about to work on. I think it will be important to stress the point that I think all these infrastructure or what is there in place to do QA, to do testing. I think it's part of the success of the project because I don't know. I remember at the beginning when we were, when I joined CERN and we were participating back in the day in all these Fedora open stack test days and you were testing their releases, et cetera and how things have improved over time. I think it's thanks to all these infrastructure that is there to do this and so on. And also a little bit connected to this, it's like contributing, it's also contributing to the project but also from operators and many other people the fact is that you want also to run this product in production and running in production implies many other things that you can contribute there as well like packaging or testing of these deployments for the different distributors. So I think this opens many areas that people can feel welcome to contribute or more comfortable contributing when they are running open stack in production. So last one, leadership roles. It's kind of keeping the community functioning and the development moving forward also involves a lot of things that aren't necessarily development. So what motivated you to serve on the TC and on the board? Yeah, initially like honestly saying when I was doing all this coding and development I hated all these leadership roles and I was like why we require that, right? But when like you manage that large community and like the diversity and cultural thing and you need this big community to go into the same direction and most important the transparency and all these like you hear everyone voice. So for all these things you need a leadership, good leadership who provide the platform for everyone who makes sure like things are going in right direction, right? Because when it comes to the culture, different culture working together you need some kind of a really good standards and the leadership also like where we make sure things are working fine and other part leadership I think define the process because as an individual you can do like lot of good thing at certain level, group you can do lot of good thing at certain level but when it comes to the large community and all so it has to be driven by the process and process with like transparent process, openness because we are in open community. So all these process you have to define the process, make sure process are running fine in a openness and transparent way so that leadership is very, very important and that's where I also started like if I can help in leadership because that is as important as like pushing a bug fix or code or a review. And from like TC and also like from board I feel like another important aspect is that you get connect the communities developer all with the high level of strategical discussion like where the community need is. So you understand that and you try to fill that gap at leadership level in term of the like process defining or technical help or like strategic discussing at the board level and see how things can go on but I feel like this is like the main important part for this openness of the community. We need. I agree. I think you're making really good points and I pretty much echo most of that but what I wanna share is like a small story of how I got started with the leadership side of things. Actually, it was Emilien Mackey who used to be on the TC and he had reached out one day on a message and said, hey, the TC is discussing a topic that might be interesting to you as an operator. And to be honest, I'd really never even worked or ever talked to the TC before. So I engaged in that conversation and I was like, this is an interesting technical discussion to be around. I started attending the weekly meetings or the biweekly meetings at the time that were being held and then as I started attending that I was like, okay, I start to see it understand the point of this and then slowly but surely I made my way into the TC and then I eventually had support with Doug Hellman who helped me into getting to be the chair of the TC as well. So the thing is, I wanna, I hope I can be the Emilien for some of the people here to ask and say, just show up, sit in the meeting, look at what's going on and you might find that there's some interest in there for you to participate in that conversation and eventually just being around enough will kind of motivate you to start and saying, well, I wanna take a more active role in this exact thing. I might talk to you. There you go. I haven't had a management role in the inaugural stack yet but from people at CERN like team that was in the foundation board from El Miro that was as well and technical user committee. I think the stress the part that Gantzian was mentioning of, there is a lot about the process of how it's all works together and being there, I think it helps to oil that process and yeah, reduce the frictions and move towards this openness that he was mentioning. And I think that is something that should be a core value of the project. I think we're about out of time. It's gonna be lunch break. Thank you to our panelists and for all your organizations due for open source. We'd like to hear your story too. There's a link up there. We're gathering those stories to kind of inform and help teach other organizations the value of participation and what they can get out of it. So thank you all for coming. Thank you for hosting. Okay, thank you guys.