 Hello everybody. We're still waiting on Justin, but he should be on his way over here. So let's get started here Again, welcome to flock it's really exciting to see people in person There are a lot more people here than there were 10 minutes ago. So I think that's a it's a good sign Hopefully some people on the stream as well possibly people here at the conference on the stream in their hotel rooms because they didn't want to actually get out of bed after the Pub trivia last night. We'll we'll see Hopefully more people be around soon But a couple housekeeping things first Do we have evening events coming up? There is a scavenger hunt and What else tonight and dinner Wait, I'll get that part on the mic, but yet So yeah, there's dinner at six and then a scavenger hunt There's a bus or two buses leaving from the front of this hotel at 5 30 sharp So be in front of the hotel at 5 15 or so to get everybody organized onto those buses We will leave people behind so take the city bus if that happens But also for the scavenger hunts. They wanted to have pre-arranged groups So go to the front desk and sign up for a group there We will put you into a group if you don't have you know Don't worry. We'll make everything. Okay. There's no no no pressure about finding the cool kids or anything like that We're all we're all cool kids here. So that'll be good There's a ghost tour tomorrow. You have to sign up for that as well And there's an early and a late one I assume that the ghosts come out more in the late tour than the early tour So depending on your level of ghost comfort you can decide whether you want to do that anyways Hi there I Am happy to present our keynote speaker who is also one of the main organizers of this conference Jen Madriaga who I first met at one of the I think one X foundation open source things a while ago and I Was very surprised to find somebody from marketing who was actually at one of our events and understood how Communities worked and open source and was just excited to talk about you know Fedora and actually had Linux the rel on her laptop amazing and So here she is Jen Madriaga who will be telling us very well prepared intro because I didn't prepare at all but Jen is very prepared That's that's that's assuming a lot Matthew. Hey everybody. Good morning Man, it was late night last night. So I'm so glad you're all here. So Anyway, I know we're running a little bit late the keynote. I'm gonna try to be on time as much as possible even if we're Having a bit of a light start. I was gonna stand at podium, but apparently you can't see me if I'm Behind the laptop, so I'm gonna but I'm gonna be peeking in and out so I can Cheat and look at my notes. So Anyway, I'm here to talk to you about framing DEI globally It's a conversation that people have been having I think for a while for a variety of reasons and I Wanted to provide Some frameworks through which to view DEI As well as relate some of my personal Experiences around it because I think it's super important to have storytelling as a part of furthering DEI efforts So a little bit who I am. I think I know most people in the room, but So I've been working in open source for nearly nine years and Before working in tech industry. I was working in higher education administration nonprofits and arts organizations So I've got quite a bit of experience across the board and very different industries I'm someone who's got an international background. I've lived in numerous places Including abroad in the Philippines and France and in different regions of the United States Including Hawaii, California, Virginia, North Carolina, Rhode Island, and so I was born Hawaii I was in the Bay Area for a little bit Bebe Optiver to Virginia meets Virginia and in Providence, Rhode Island And now I am in Durham, North Carolina Which is 30 minutes north of Red Hat Tower if people know where that is And so it's part of my job. I travel pretty extensively. I've been to pretty much every region And I get to interact with a lot of different people, which is honestly one of the best parts of this role like it actually really satisfies my desire to learn new things and Meet new people because I just have a natural curiosity about the world And back in the day, I also did graduate work around issues of diversity and equity and inclusion and so DEI has evolved a lot in the time I did that kind of work and so I did a certificate in women's studies. It's actually not called women's studies anymore It's called gender sexuality and feminist studies and that was from Duke University in Durham, which is where I still live and then I also co-founded the Red Hat Asian Network and I currently serve as its chair But even though I'm chair the Asian Network, I am deeply interested in intersectionality because we don't just have one thing about us We have lots of different things about us, right? All of us are very multifaceted and that includes myself So before we get started I wanted to talk about the definitions over diversity equity inclusion and I also threw in justice in there And part of the reason I put this up is that I thought this was a really cool framing because it asks the series of questions And that's the way that I want people to approach DEI. There are no Absolutes. There's a lot of questioning. There's a lot of conversation. It is not a linear process, right? And I thought this was a really really nice framing And then, you know, most people probably have not seen the justice part, right? I feel like the justice part provides additional nuance to the diversity equity and inclusion piece, right? Because, you know, people think of diversity, like, okay, we just need a lot of different people in the room and it's solved But then, you know, you have to ask, well, who's not in the room and Then when you're in the room, are you allowed to be heard? Like, are your ideas allowed to be heard? And then If your idea is heard, is it going to be taken as seriously? Like, are people actually going to think that you have any credibility even if you choose to speak up? So so there's a series of steps in it and, you know, part of what I want to say about this journey, because it is a journey, is all of us are learning together So maybe you're learning, but you know what? I'm learning just as much as you are So I would never ever style myself as an expert. I can only say that here are my experiences. Here's the research Let's have a conversation and evaluate everything together because I was trained to be a researcher and a questioner and You know, I want to make sure that I have an understanding of where you are And I hope that I can express an understanding of where I'm at So I want to talk a little bit about diversity because people are like, well, what does diversity entail? And actually there was some research done by a person named Emy Edmondson out of Harvard And she actually is One of the very first people to write about something called psychological safety Has everyone heard of psychological safety? Raise your hand if you've heard of psychological safety Like we've been talking a lot about it. I'm going to talk a bit about what that means But before we go into psychological safety I want to talk about team diversity and so she identified four different types of team diversity along with Another research there Catherine roll off and a bunch of people are working in this field now So we're going to go through the different parts and then I want people to raise their hand if they think it applies to them So the first one is demographic Diversity So it means you've got a variety of people in your team. Maybe there's some disparity So maybe you've got a bunch of identity groups, you know, maybe they're from another country another part of the country Another race another culture different sexual orientation Stuff you know stuff that maybe this is your typical definition of diversity So does anyone belong to a team like that where you have people who kind of fit that definition? All right, so so a lot of you The second Type of diversity is expertise, so you've got a bunch of different skill sets You've got some people who may be senior you may have people who are junior You may have people who are really great at this one thing and other people are great at this another thing so Who has been in a group or is in a group where there are different levels of expertise where you have junior and senior Members or have been a part of team like that Okay, great All right, so this third one. I think this may actually apply to a lot of people geographically dispersed team members Who has to work with teams and other regions that are not where you live? Okay a bunch of you All right, and then the fourth one is disparity and so At Red Hat we like to think that we have a flat reporting line, but I work in marketing and so we're More of a traditional corporate structure where there are social norms of deference to authorities, so you know basically Usually in open source we say titles don't matter, but you know sometimes they do And so if you've got some sort of reporting structure where you've got a manager and people under them That would be basically that kind of framework So who is on a team where you've got someone above you and maybe people below you in terms of hierarchy Okay. All right, so it looks like the majority of you fit into One of these categories of diversity when it relates to your team or maybe all four I actually have all four so So yeah all sorts of diversity and so I think it's really important to frame what we mean by diversity And how it can apply in different situations So I have been doing this work diversity equity and inclusion And justice and other subsets within this world for a very very long time I don't want to tell you how long because I'll give away my age, but I've been doing this for decades and There's something that comes up in every situation that I've been in And I think this is universal And one of the things that I want people to take away is that DEI is not separate from your everyday experiences It's a part of how we experience the world even if you don't think You're experiencing the world in a way that involves that but some of my observations is That people have a very deep need to belong People are always asking What's what's my place in the world and can others understand my experience and Another thing that I've noticed is people need to be seen and validated You know people need to feel their contributions matter and not only do they matter, but I'd like to be recognized for the contributions that I make The other thing That's come up is can I bring my whole self into situations without fear or self-consciousness You know so an example would be like if people knew something about me, you know that I'm different in some way You're automatically operating from a place of fear and hypervigilance and that is not a fun place to be And so I think it's very very important for people to be able to be in an environment Where they can be their whole self without judgment and without fear And then another really fundamental thing that comes up time and time again, and it doesn't matter where your opinion is Around Dia is that people are really concerned about fairness People really focus on what's fair and how do we get to fairness right and to me at the way that I define fairness is You know, I want the same access to resources and support You know like if there's one group getting Access to support and resources I I'd like to think that I too would have the same access to those resources and support and then I also want access to opportunities I Want an opportunity to grow and to evolve Into something better than what I know Evolve into circumstances better than what I know So let's go back to psychological safety, right? So I talked about bringing the whole self to work or even to your everyday life. Maybe the grocery store You know, maybe while you're walking around your neighborhood So Psychological safety, you know, maybe you think of it in terms of being an individual But it relates to a setting or role Where people can express themselves, right? They can And that and in this particular instance work relevant thoughts and feelings, right? And psychological safety also includes in its definition Permission to make mistakes If you've made a mistake, can you raise your hand who's who's ever made a mistake if you haven't I want to know who you are I'd like I'd like to know who you are if you've never made a mistake, right? And so not only is there permission to make mistakes But there's permission to learn from those mistakes You know back in the day We used to have this sticker at Red Hat that said failed better. Does anyone remember that? I don't know if anyone ever saw that sticker. I actually have that in my laptop I loved it because that was like such a mind-blowing thing to see something like that in the workplace And I took it away because people are misinterpreting it and saying, okay, yeah that People are understanding what it means, but for me, I was like, this is great You know why because it means I'm allowed to make mistakes and learn from them and be better after it. That's how I interpreted it and so I Think everyone should have permission to fail better. I give you permission to fail better, you know so Don't be self-conscious if you make mistakes I think the key to is if you make a mistake What do you learn from it? How can you grow from it? But you know what we can't define psychological safety without talking about teams Because really psychological safety is Related and dependent on teams and why is that? Why would teams be important in Psychological safety hmm You know Part of it is a setting right like think about Maybe a past work experience Or a past school experience think about a teacher or a leader Think about the ones that made you feel afraid and judged and then think about the places where you felt safe Or you could laugh Where you could share things that were uncomfortable and know that you would be supported Have people been in both those environments? Have you been in an environment where it's like? You know, I'm afraid to speak up because I'm not sure what that means for me and have you been in an environment where it's Hey, this is great and with people that I really trust and I know that I can rely on them to have my back And actually, that's how I feel like when I'm at flock. So thank you community I always feel super supported in this community and the fedora community is one of my favorite communities for that reason Like I can nerd out and be my weird self Without judgment. I don't know that jackhammer sounds like it has judgment, but it's okay All right Another thing I want to talk about right if is if we talk about diversity by itself You know people assume falsely That if you create a diverse team Everything's solved Yay Hey, all of our issues are shot because we put a bunch of different people together And they're gonna work on a team and they're gonna solve everything because they're all different well Studies have shown that diverse teams often underperform relative to homogenous teams What you mean putting a bunch of different people together actually doesn't work To create an environment that's more productive Why is that? Part of the reason why it doesn't work is that there is no framework for communication You have to reimagine the world When you are working with people that are different from you So just think about it if you have a bunch of senior members on your team that have been doing Open source for 10 15 years and you have a newbie that comes in that doesn't know the terminology That knows how to access command line, but they're not sure where to start or where to go You have to interact with that person differently You can't interact with that person like they have 10 or 15 years of experience and if you did What does that bring out right It brings out a lot of fear and smallness because this person doesn't want to ask questions because They know that they're assumed to have a certain knowledge and skill set that they don't possess So we have to set expectations. We have to set frameworks for when we deal with team members that come in and as I said, there are different forms of diversity Right, so there are different things that you may have to do to address those types of diversity but the key thing is creating a communication framework into which all groups can work and understand right so Here are some things that you can do To ensure that your diverse team excels right So framing is super super important We have to be able to frame things as a way To provide opportunities for information sharing So if you've got that junior person on your team That's never been An open source and they but they want to be an open source You have to create a framing with meetings where it's opportunity to share information You know Information sharing is really really key right and then we want to frame differences as a source of value right, you know One of the things I loved when I came to red hat is the idea that the best idea wins and it Doesn't matter where it comes from doesn't matter if it comes from your intern or the CEO the best idea wins Right, and that's part of you know Like I think about like even in terms of my height because I'm really short and there are things I don't think about because I'm short so for instance when I'm on the airplane and I stand up on my seat I don't hit This part like I just stand there Versus like someone like Adam probably can't do that, but it's not part of my everyday experience So I don't think about stuff like that And then another thing that I think about is that like my refrigerators very high at home and my husband likes put things on the top Shelf and I don't ever see them. I can only see I level Right, and so we have to consider What are people experiencing right and you have to frame those differences, right? So one of the things I've asked my husband ideas that can you put stuff that I actually use on the lower shelf So that I can reach it and actually see it and not buy like five other items that are identical because I didn't realize it All that stuff is on the top shelf So anyway, so that says very important, you know, you want to you want to have conversations up frame those differences And then increase like I told people I have a natural curiosity About the world and I ask a lot of questions. I ask a lot open-ended questions and Sometimes listening is really really hard, you know, especially if we're already attached to an outcome or Attached to a certain expectation or attached to a belief system, you know You know, I may be at the mindset because I don't ever hit my head when I stand up on a plane That like I don't know what the big deal is why where people complain about not having enough space. Come on Suck it up. I can stand perfectly fine. You know, I've got clearance too You know But you know what I know That's not the experience of other people You know, I'm sure that folks who are Vertically challenged this way Have their own sets of issues, you know, maybe you can't be in a hotel room without having your feet hang over the bed You know, and it's impossible for you not to ever have your knees hit the seat in front of you Which is probably pretty excruciating On a long flight, right and so I want to ask questions like hey Adam, what's that like? Do you find that comfortable or do you wish that on other people? Yeah, he wishes on other people, okay, that's good to know. All right But part of these questions, right is you also are trying to find like shared ownership and causality And what do we mean by shared ownership and causality part of it is we're trying to Establish understanding of different things, right? So we're we're putting things in the pilots I I call it like recipe making so like if you're trying to create something As a group if you're in a project of some sort I always like to use the recipe analogy where like if you're wanting to bake a cake You can't bake a cake just without Would just flour or with just eggs or would just butter or with just sugar you need all those things To come together with a certain magic So that you can have a baked good at the end, and I love baked goods So who doesn't like it baked goods, right? And then we have to talk about bridging boundaries, right so People have a natural threshold for comfort because of their experiences, you know They may be only comfortable with You know this much space like personal space is another thing that I think is always really intriguing when I travel I think in the United States. We like to have an arms length As comfortable personal space But I've been in cultures where people are right on top of me and almost touching me And it's kind of like whoa, okay. There's something I have to understand here because this is their norm. It's not my norm and So I need to figure out where people's boundaries are right And so part of how you bridge boundaries again is without open-ended questions Like if we're working in a project everyone's got hopes and goals for that project, right? So if we're in a work setting a lot of those are assigned things that we're needing to do and accomplish And it may be that the various people on your team Have very different hopes and goals and you know what that's actually very important to know that So your senior person may have very different goals from your junior person and You need to have a recognition of what they are so those gaps can be bridged, right? And then you want to do an inventory of your resources and skills, right? So one thing like With a newer person is they may see things that you won't see because you're so used to doing things a certain way You may not see something that actually needs to be seen and someone less experienced or less familiar Can see those things because they're still learning And so those are the things you want to think about is that you know what just because this person does not have the same experience as you do They're still bringing something and then This part about concerns and obstacles. I think is super super important particularly if you are leading a team But this is a conversation that you also want to have as a team So people have an understanding of what's going on like what are people up against? What are you worried about? You know sometimes these are one-on-one conversations and sometimes these are conversations that you Want to have as a team, but it's very very important what people's obstacles are Like an actuality maybe would process but also You know psychologically, you know, maybe people are held back by fear They don't want to make a mistake because if they make a mistake then oh no the world ends, right? But the world really end So part of it is you have to put people's fears and perspectives and maybe you have to put your own fears and perspectives because usually The world won't end maybe your servers will go down for two months and that's terrible But you know the world's not really ending Or for some people might so I want to talk a little bit about inclusion and how you evaluate teams So this is I'm not going to read through all these questions, but I found this article by Chelsea Troy about how you evaluate team members contributions for our inclusive culture and I've already kind of gone through a version of this with my last slide But I thought this was really great, you know so part of what you want to look at in your team there Are they able to moderate? Their meetings are they able to make sure that people are Expressing their opinions and then if they're not, you know find a way to solicit them I mean there are people who are both extroverts introverts. There's some people that like to give verbal feedback or Written feedback and so it's really really important to provide different forms to To do meetings for feedback So same with the soliciting opinions attribution. Oh attribution is super important So one thing that I think That I already spoke about was the need for people to feel recognized, right? And in another really common experience is people Expressioning an ID and then someone taking credit for it, right? So one thing that I think that's really really important to realize is that it's important to Recognize a contribution given by someone And not appropriating it and if you're appropriating it, you know ask yourself why you're doing that Usually it's related to wanting to have power over someone and You know that actually doesn't really create a great team dynamic and what it does is it shuts people down So, you know, let's go ahead and get recognition to people because you know what it lifts you up to Because you're seen as someone who is a supporter as someone who is here to uplift and make people's lives better And honestly, you will get so much in return by doing that instead of being small and taking credit for something that you didn't do So, all right, I'm just gonna go through that. So because I want to be mindful of time So I want to talk about the power of the individual So we talk a lot about individuals particularly within American culture And I was talking to somebody about this yesterday, you know, because people always ask about like what can my company do? What can my organization do? But sometimes it's up to the individual. It's up to an individual exercising their own power Even when they're fearful. So one of my favorite quotes is Speak to truth even if your voice shakes and my voice Shakes on many occasions because I'm not a person That is naturally comfortable with putting myself out there. This is something that I've cultivated Over many years. This is not something that's comfortable for me. In fact, I was conditioned not to speak up My dad was always saying don't rock the boat. You don't want to draw attention to yourself But there are times we do have to to speak up And I'll talk a little bit about my work and creating the Asian Network at Red Hat in just a bit But what's important to realize that sometimes the greatest changes don't come from organizations themselves it comes from individuals Within organizations or me outside organizations. They're willing to put themselves out there to do the work without an expectation of compensation and Then it also involves relationship building because allies are such an important part of Making change happen like I can't make change happen By myself. I have to make sure that other people have buy-in to making things change and so Individuals can start those processes and then they can catalyze Things happening But then we also have to talk about leadership Because leadership does matter One thing about individuals creating great things is that if that individual leaves your organization, there's usually a void and so Your leadership It's necessary to implement structural changes. So as an example Within the United States We had to do a voting rights act for people to have equal access to voting and that's actually being challenged right now but in theory everyone always had the right to vote but in order to ensure legislation had to be enacted and Other words structures had to be changed because if you're looking to individuals or existing structures to make change Happen it's not going to happen you were going to have to enact change on a structural level and in this case you would do it through legislation also You need leadership because remember when I was talking about when you have diverse teams you need to have a structure for communication It's your leaders are going to make sure that that infrastructure gets created and So even the individuals are greatly important if you want lasting change in your organization You are going to have to have the buy-in of your leadership to make those things happen So let's talk about creating inclusive leaders, right? you know, so Again, I'm a fan of the Harvard Business Review. So you'll see that I get a lot of citations from articles And if you want a copy of my slide deck, I'm glad to give it to you because I have all these links embedded But when they quizzed leaders about being inclusive Inclusive leaders and we'll go through what makes it an inclusively or just a bit. They're 17% percent more likely to report that they're high-performing 20% more likely to say they make high-quality decisions and 29% more likely to report behaving collaboratively and Then it also Increases work attendance. So basically, you know, if you're really hyper focused on a productivity Aspect of DEI maybe not so much the human element There is a correlation that if you practice inclusion inclusivity that you will thrive so what are some examples of Strong inclusive leadership So when they surveyed people when they defined a strong inclusive leader or they said that strong inclusive leader is not afraid to share their personal weakness and An example of personal weakness may be You know, I know I'm good at this, but I am not good at this and I need your help I Need your help in making sure that I'm understanding things because this is not my area of strength So that would be an example of sharing a personal weakness The other is just learning about cultural differences and realizing that there are different styles and so Like I said cultural could even extend as someone being introvert or extrovert As well as people being from different regions different cultures or different countries But understanding, okay, here's a preferred way to communicate Here's something that needs to be highlighted You know and so that's super important that this leader leans into learning about those things and doesn't assume Things they actually want to learn about those different things and then also Acknowledging team members as individuals so, you know, one of my favorite stories I like to tell about when I came to Red Hat was The afternoon of the first day of new hire orientation we had a reception where your teams could come and Welcome you and I remember There is this gentleman with blue jeans a belt and a white button-down shirt. I'm like, who is that person? Why is this person here? And he was he's like hey, I'm here because I want to welcome you to Red Hat and it was Jim Whitehurst our CEO And I'll tell you I was like what kind of place am I working at Where the CEO comes to shake my hand and to welcome me and tell me That he's looking forward to me helping me the organization better I Mean, I still remember that story and I also remember the times that he would come to the booth Completely uninvited because he wanted to say hi to Hatter's and he would take selfies with all of us Man, those were really great days. So Jim Whitehurst wherever you are. We love you And then least inclusive behaviors Overpowering others, you know like getting your way like my way or the highway Displaying favoritism like I you know, I think this person can do it best I don't care about the rest of you and then discounting alternative views like my views the only one that matters because it's the right one You're like sure that there's nothing else that can be introduced to change your viewpoint And one thing also that's really important to know is that you know, we're a lot organizations or fans of surveys You know feedback forms But it's really important that feedback are distributed results rather than average results What do you think that is? Because if you average it out you hide Where the differences and disparities are in terms of perception it hides your problems And so be very suspicious. It's like hey 80% of people are really happy, you know You need to kind of dig down a little bit deeper. Is it is it really 80% or is it, you know, how is this calculated? Data can be manipulated any way you want So how can you be an inclusive leader? So it says inclusive leadership shadow I don't know if anyone is familiar with The psychologist young but he talks about the shadow side and All of us have that shadow side all of us are really convinced that we're really good at what we do But sometimes we're not and we have to ask people for their feedback And we have to be visible and vocal about all those things So one thing to remember is that inclusive leadership needs to be cultivated and conscious and When exists everyone does benefit from that and I know that we're running out of time So I'm gonna just blow through these last slides really quickly One thing I want to talk about is contact theory and this goes back to you know when people think oh if I have a diverse team Everything's solved if I put a bunch of different people together. It's solved like my DEI problems You know are gone and it's not true So one thing about like putting people together in a room There's a thing called contact theory and this one is specific to racial prejudice But if racial prejudice can only diminish when you're in a group with people is if you're under certain conditions And I think this also relates to Psychological safety that I talked about earlier, which is where everyone is sharing roughly equal status and belonging and working in a common purpose so if people are very different and they're in the same room just because They're together it does not mean that things will change the conditions have to be equitable The conditions have to be fair feel fair to the people who are in that room together And they also need to feel like they're part of a larger purpose So one of the things that I have to do on a regular basis is work with people with very different personalities different backgrounds For myself different philosophies of how to get things done But one thing that always remind folks when I'm doing a work project with them is like you know what in the end We want the same thing we want to accomplish something together and make it great What do you think needs to happen? for that to be the end result and That's Sometimes hard, but finding that larger purpose that you can agree on makes a huge difference so I'm gonna come back to the slide and so hopefully maybe you understand the justice part a little bit better You know the question that needs to happen for all of these things to be a reality in your organization And then I'm gonna just talk really quickly about Red Hat Asia Network And then I'm gonna let everyone go to their next session, but as I said I was a Co-founder for Red Hat Asia Network, and we were actually primarily concerned with Asian Americans Just wanted to shout out to our founding committee two of these folks are no longer at Red Hat One was laid off. He was a chairman's award winner Was working in the office of CTO and then another just left to go to another organization So there's me and there's Robin Chan my coach here and in Tesh Patel They're the only ones that left Joe to sigh and Laura Fu are no longer at Red Hat and actually executive sponsor just left three weeks ago So it's just a few of us are left who have done this effort So I just want to talk a little bit about why I did this organization Because I never talked about DEI at work. It's been a very long time and now I'm talking about it a lot But there are different things that happened one was COVID-19 and the racism that it brought out and Not only the racism But it also bought up a lot of old societal wounds for a lot Asian Americans about being othered and Not to be blamed for something was pretty painful And then another thing was that there were a lot of associates who are not being promoted into management Asian Americans are least likely group in the US be promoted to management within tech and then within tech It's Asian American women. Oh, that's me who are least likely to be promoted into management roles So one of the things that was really important to me was to challenge people's assumptions about belonging to a group So within the United States we have something called the model minority myth Which is that Asians are super successful and they don't need anybody's help And That's not true for a variety reasons first of all the term Asian American is a socio-political term That evolved in the 1960s It's an umbrella term for a wide variety of folks and the reason why it was created was to create a Way to ally together to help each other To find a more equitable place To find opportunities For equal access like other groups So I Just want to leave you with people being intersectional beings. So as I said, I'm intersectional I Identify as a woman of Filipino descent. I Am also neurodiverse. I have ADHD And I've lived in particular places and from the way that I am so Anyway, hopefully this is helpful Like I said, this conversation is not over and if you want to talk about a little bit more. I'm happy to talk with you But hopefully this provides some additional Lenses and other things for you to consider that maybe you haven't considered before So thank you for coming to my TED talk karaoke later Yeah, and stay in touch find me and Master on our Twitter or X or whatever it's called and then there's my LinkedIn as well as my email Okay, I'm really done now. Thank you. I think we can go to break now by the way Our session I So let's do this Well, we represent the four ninths of fesco so even we don't have quorum today And And normally we would cancel the meeting but let's let's I know Let's try for a bit So my name is Bishak. I work on system D and I have been in Fesco for Six years now I think Yeah My name is Neil. I've been in I do lots of stuff in Fedora and other places and things like that And I think I've been in Fesco. I think I first got a fesco at the start of the pandemic. So I've only lived in pandemic era I'm Kevin Finzi or Narek and I've been in Fesco since it was the Fedora extras steering committee That's a name I haven't heard in forever and I am David Cantrell. I have been on Fesco I guess also during the entirety of the pandemic. This is actually the first time We've met in person So that's that's kind of interesting to think about I've been in Fedora for a Long time I work I work on software. Yeah We do things we've been around So, yeah, I mean meet your fesco the way these normally go is we just kind of open it up to questions And you know if however technical you want to get or non technical we can talk about anything really Check if there are any questions online. I guess that's a really good idea So one thing that I That we tried is moving the discussion of Fedora changes to discourse discussion Fedora project org and I Think that the results were mixed We started with some very tough ones, so we are overloaded by the number of comments Personally For me email is just more comfortable easier to get around in and more familiar more efficient but On the other hand We had more comments from People from new people and people from outside of the community or fringes of the community Which was well Changing in some ways, but also good in other ways so I Don't think that it's at least for myself. I cannot say that did this move was Clearly good or clearly bad. I can see Yeah See it both ways I hope to That with other proposals which are less contentious we will have a smoother workflow and then maybe It will work better and And all that's that's that's how I view it. I was wondering how other people see it My brain melted during that discussion Yeah, well, no, I have I have some thoughts on it like the the I'm always in favor of making it easier for people to To provide constructive and valuable feedback and and input into the thing that they're using and building upon and and In being invested in right like being Enabling more participation is always great the flip side of it though has been The this particular this particular discussion wound up feeding itself into a very deconstructive destructive frenzy is I think the the easiest way I could describe it and It got amplified and re-amplified and fed on itself and Because The It's a it's the trade-off of you know, how far you want to go and what you know how much how much you want to have input from and I don't know if we figured out a good balance for this with putting it into discussion FPO I my my Personal opinion about this is I have some reservations about continuing to do this, but I'm also willing to see how some of the others go before making a Full judgment on it. I mean, I'm glad we're trying the experiment and seeing how that's going to work out and seeing What kind of participatory effects and what kind of quality of discussion we get? it gives us a good idea of Whether this is something we want to to continue down or not, but You know, I guess we'll see Yeah, my my observation has been that You know, we we wanted a way to get more input on change proposals That's the core item that we work by on fesco. You've probably seen them before There's a process we discuss it. We vote on it. That's how changes get into fedora On the mailing list side, which I will say I'm more comfortable with that only because I've been doing this for a long time The mailing list side it felt like or at least over the past couple of years Change proposals would only get comments by roughly the same five to ten people I don't know if that's because they were not as visible or that they People just felt like they were not You know welcome to comment on it or it was kind of a thing They couldn't participate in so Expanding that getting that outreach through discourse has been good because at least it makes our discussion on fesco easier because we Have input from more of the community, you know, what do people actually want to do? Is this something that people want? so just having more data points is useful, but I See the the the discourse side is having an opposite effect So the people that were comfortable on on the mailing list are now uncomfortable on discourse We can pick one set of people we want to hear from or or the other and and we can't get both but the answer isn't to Post it in both places. We need to come up with a better solution I do think that when we we tried the discourse method We did not really do a good job of explaining what the change proposal process is to people who had Largely ignored it. So I I observed a lot of Community members thinking that these were more of Announcements of decisions that had been made as opposed to discussions of changes So I think we could have done better at explaining what this was going to be and what we expected everyone to do So that was I think kind of new for everyone. I don't know if you guys like There was definitely like a lot of people said, you know, this is a fork on conclusion And then and I think there's there's a couple of reasons for that that at least I observe is like one of them was Are the way that that changes have landed in fedora over the last like, you know, decade That I've been observing this is largely been Uncontroversial super easy they go through there's the feet the feedback is discussed and things are resolved before they get to us So it starts looking like whenever they get to the Fesca level that they're quote unquote rubber stamp But the reason that they get to the point when they reach us that we're all comfortable with this because we're also participating in these discussions and people are resolving our concerns way before we get to that first meeting and so people have this Conception of we rubber stamp things when we actually don't And then on top of that the way that the news reports on fedora's changes process Like so if you look at some of the pharaonics and others There is a very very Important reason a very big reason why we now have this banner that says this is a proposal in the wiki pages Is because there was a number of incidents where? Volatile changes were being proposed that people assumed were going to just happen Some of them didn't actually happen But they didn't know that and they reported on it and misreported and then there's this like View that a change proposal isn't in fact a proposal But like the equivalent of like what a proprietary vendor does it says hey, we've done this thing your You got to just deal with it, right? Like that's that's the I think that's the two big parts of it Yeah, I think there's some some really good lessons learned from this this discussion that we had on that controversial change and I think that's one of the big Outtakes from it is that people don't understand the process and we need some way to tell them This is a proposal. You're supposed to give feedback. It's not done yet. It's still under process Also, I think there's a lot of Associated things around that like people didn't realize a lot of people thought with this particular pose proposal Oh, it's a team in red hat So red hat proposes things and then people just approve them Well, no anybody can propose something if you have something that you want to do and you willing to do the work And you're willing to write it up and get it through the process. That's fine. Anyone can do it but I think a lot of people didn't know the process and Perhaps were pointed at the discussion from a context like oh here You should say no to this and they just would go and say no and not actually read through the the proposal So I think there's lessons learned from it. I think I'm also on the mixed case I think we got a lot more input from people on the outside in the community and that's great I think expectations were bad. We learned some lessons Hopefully those will feed into this and make it actually better. I think it's still worth doing I think it's a good experiment and we don't have enough data points really yet. This is kind of a Crazy outlier this I hope it's a crazy outlier. I don't want I don't want discussions to be like this all the time How do we think the nano default editor proposal would have gone? So I think it would have actually gone the opposite of the mailing list you bring that up in and in all seriousness I think if we had let in in in theory crafting land that we had the nano discussion on Discourse rather than on the mailing list It would have been completely the opposite way Because the people that are on discourse are the complete different set of people Then what we've been interacting with on the mailing list a lot of them are either newbies or first timers or their Thinskin thin their first into Linux and and their their points of accessibility into the terminal and all this other stuff They would have appreciated the change more than a lot of the people that have been old school have been battered by the You know the old unixes and whatever's and they've they've lived by the scars of their heads and vies And so I think it would have actually gone the other way and it would have been contentious for a different reason Because you'd have some of those old-timers complaining that we just did something for you know Well, I think we've actually had some of those complaints during the thing about like we're we're Simplifying or or making it like we're not making it less. Yes. We were making it too easy and stuff like that Oh, yeah, that definitely comes up too often as people people think because it was hard for them We should keep it hard for everyone like I want everything to be easier than what I had it. Well, yeah I mean, that's kind of what we want to do so Comments from the audience please So we've robbed people of the learning experience of how to get out of them that I mean that's a good point and you know people people brought that up and you know, I It's kind of funny all the points that came up specifically on the the nano change proposal and I don't use nano I used them for like 25 years and and then changed to to emacs and Actually became way more productive, but that doesn't matter. I mean the editor is everyone's choice, right? And you didn't just slip that in just because right yeah, yeah But I I think I am one conversation I brought up like you know guys like gentu which is you know if you really just have a lot of time to burn and you want to Set up a Linux system you can't but their default editor. I think was was nano. Yeah, so you know It's it's viewed as kind of a high-tech, you know end user Sysad men distro and it's using nano. So, you know, whatever and another counter another counterpoint to this particular like you want to keep it hard so everybody learns the hard stuff is like I have skills for how to Screw around with DOS systems like I started with MS-DOS and I use DOS editor I screwed around with config dot sys and auto exec dot bat How many of y'all after have to touch one of those these days? I promise you almost none of you do the unless you're like doing retro stuff for the fun of it like I do sometimes so Like those kind like you don't need to keep things to be hard to keep it fun and to keep people learning You give them new interesting things to learn and grow from and there's new hard things for them too also Yeah, the hard things don't ever completely go away. They just change This thing is like nice to hear but Is I actually don't mind about either of them nano in whatever, but I would love to hear Your inside in the recent decision about the NF5. So That's the controversy. So put it on the stage and I especially curious Like yeah, it happened to revert But why it was reverted in in row hide especially why not to foster in in row hide and continue Testing development there. Yeah, so I think I Said this multiple times. I think that I'm very proud of the DNF team for like looking at this and deciding that it wasn't ready You know, most of the time people who invest so much time and energy and work into something And there's a deadline and they're not quite meeting the deadline they'll try and get it in anyway, but I think they did a really smart thing and a really a really good thing to Decide to pull back from there and I think the the question as to whether leaving it on in rawhide to get more testing I mean, it's still there people can use it. It's just not going to be default But we don't want rawhide to be used as a just a test bed where It's not actually intended to go into the next release We want it to be a integration point where people are actually testing the thing that will you know go to the next release So I I think it's good to revert it there until it's ready to to actually go in I don't know how much more you really much more testing would go on there It's hard to say but there's a lot of the stuff like the API wasn't stable until fairly recently as well And so there's going to be a lot of stuff that's porting over to it, but it still can be tested. It's just not default so Yeah, I'll add that the DNF team is the team that I work on as well and you know, it was I guess it was Disappointing but really the the correct decision it Just the feedback in general on DNF 5 and the current state leading up to Fedora 39 There was too much that was unfinished in my opinion I DNF is is one of those components in Fedora that we absolutely need to work and Since we already have one that that is implemented to move to one that that loses core Functionality that we have documented that we tell users they can rely on Just it it just felt incomplete. So it it needs a little more runway to to sort of Complete that and just this morning. I was talking about test days Implementing more DNF 5 test days We I think we came up with three that we need to do But the Festco ticket where we were talking about implementing the contingency plan to revert it There there were a lot of things that were pointed out there and to me I feel like that was a kind of a rare situation for us to get that much Feedback which which to me shows that people were trying it But a lot of stuff just didn't work yet and and it just I you know I like like Kevin said the the DNF team just saying like yeah, it's not ready We should we should pull it back and and even still Not just bouncing into the next release Which is another common thing too like we need to rethink our approach here and maybe it won't be ready for for two releases Yeah, and as someone who's you know I have a connection with the DNF team as well because I've been a community contributor For almost eight years now in in the various, you know RPM systems a software management group projects the The the DNF 5 work You know when we started when it started getting rolled out and people started using it like it was very very clear that The the use case prioritization There was a mismatch between what Fedora needed and what community needed it and what the team thought that needed to to be able to make it happen and We were encountering pain points and issues and the community was bringing them actually fairly promptly And that was that was also unusual usually We don't get anything for until like the last minute and things were just you know And they were being very very responsive in addressing the concerns and working through them But it's just when you get a huge pile of them and there's just only so much work You can do it once and you've got to qualify all that you got to make it work And and like David said it's the core of our system We have to make sure that it's actually solid to the point of like not breaking it like the one of the Reasons we went from yum to DNF was to massively increase the reliability of Doing the core thing that we need the distribution to do which is managed software and we don't want to screw that up in this transition And so I'm really happy that we're taking a step back to take that next step forward Justin go if you remember during the the yum to DNF switch You know that was actually rolled out it not not it wasn't the same state the DNF 5 is but it was a rolled out a Little prematurely as well, and there was core functionality of yum That was now missing from DNF and that caused a lot of problems now We got over them. It was fixed and you know DNF is great now, but We don't want to repeat those mistakes exactly Yeah, that I Was on the anaconda team at that that change point and one thing I remember is During that switch over we would we would have a bug report for yum. We would file a bug report That team would would fix it but fix it in DNF and Not yum so it was like we were starting to have the two drift apart And we just had eventually had to make that cut over and yeah We got we got past that but it was it was bumpy it was you know They wanted to say like look the time for DNF is now It is fundamentally different. I remember one of the the the big difficult things with yum and DNF At that time was Installation if you had yum as the back end in the anaconda you would get a different Resolved set of packages to install versus DNF because there were different depth solvers And that it's like well Okay, I guess we're just gonna have that now maybe and yeah So that's long since past but it's it is important to remember that so that we don't end up in the same kind of situation one that I I Was disappointed that is not fully implemented yet as the system upgrade capability that we rely on now in DNF That's not in DNF 5 and I was like well, I mean I use that Like that's got to be there, you know, so little things like that, you know, we definitely need to get Is my question is so do we reverting this stuff like do we have somewhere the list of things Issues which needs to be solved and not just in DNF, but with the integration with other tools Fedora review Discover other stuff like which needs to be solved. Oh my mark is done. That's that's my project So Which needs to be done as so we track it and that surprised one year later again Because I ask yeah, I reckon when I check was the status of the other plugins as well Not just copper plug in. Oh, it's tracking somewhere inside our GitHub like but it's not visible outside for the people who are not willing to dive into the bunch of Issues, etc. We have Yes, we have that information. It's incomplete right now because it's mostly from the first DNF 5 test day that we did but it does have a lot of those results What what did not work? What were we expecting bug reports were filed? They've been prioritized in my view They've been prioritized differently than what Fedora was expecting. So we're at a point now where we since we've reverted this in in Rawhide We need to go back as the DNF team and look at all this and say okay This is how we need to prioritize this work and yeah, so that information existed. It's coming together So there was some additional information on that in the Fesco ticket to people were saying these things are blockers And then you know some of them weren't some of them are this is implemented or it's different or whatever But some of them definitely were like system upgrade is not there and you know things like that Well, and you mentioned Fedora review, but also all the integration with Fedora infrastructure And like I did a lot of the early porting work for moving from young to DNF for all for a lot of the relinch scripts and things like that and I literally couldn't start figuring out how to port things from DNF 4 to 5 until like two weeks two or three weeks ago And that's really not enough lead time to be able to make sure everything works And the API wasn't stable until recently so right and it's not like I wasn't aware that they were working on it and think about it But like going through the effort of making all these changes, especially as a volunteer and spending that time and and not having documentation and not having Understanding of the expectations with the API interfaces and stuff like that's it's a real chore to go through all that stuff And like I'm not going to ask any volunteer to to do all that work when they don't know what they are supposed to know and You know once we like that's that piece of information is something we Kind of have started to get and I'm kind of and I'm hoping that that gets fleshed out more in the intervening months so that we can Actually start, you know giving that feedback from the API side because I know that the test days have given a lot of user feedback But we haven't done a whole lot of producer feedback Which is also important for being able to switch over to it because I don't want to repeat of where it took us Five years before we could start using rich dependencies because nobody could figure out how to port like the Bodey and other stuff to use DNF instead like that was I want us to be able to move like that properly to everything But that was still good because that Before that we had like what ten years of speaking about is also I'm pretty proud that it actually happened Yeah, but that was also like the most painful porting projects So When the DNF five discussion in fesco started we made an effort to Well try to build a list of requirements and this list was I mean there was some list creator was very incomplete And I think it's kind of the same Problem that appears in this kind of switches and I I think it's I mean one lesson for me would be that we need to build this kind of list and Building of this list is of such a list is hard because you need good insight into Not just the thing that is changing but all the surrounding tools and processes and that's hard And I think it's particularly hard in the case of of Of we are more DNF because there's just so many so many points which need to be touched and So many of the external tools are like Well dead or semi-dead somebody wrote them five years ago and they live in in length scripts or somewhere else and as long as they work Nobody will start them again. So nobody knows and there's no clear I mean Like I know we have federal review and Obsolution scripts and this and that and I mean in half of those things have no clear owner or no ongoing maintenance and nobody Will think about fixing them until we switch and then we realize that they're not working Yeah, so if we could do this again, we should probably try to figure out how to how to Find about those things earlier I want to add to that. Yes. I agree, but on the general topic of requirements. This comes up from time to time and Fesco to me Fesco should not be the body that is dictating Hard requirements for projects like that. I mean really when it comes down to to DNF like we defer to the change proposal owners the the engineers who are working on that but Generally speaking our job is to to vet these changes and say, okay Well, the requirement from our point of view is don't break things for users, you know Like so have you done that we need we need information to determine if you've you've really set things up that way And I think in this case the DNF case They they're needed there were such a broad impact that that team needed help to to really understand Who all needed to be a requirement so in that respect it was a little unusual But you know really changes or changes and you know that might mean that requirements change for software So we shouldn't say you know in a change proposal this team wants to do something We we shouldn't go in and say well. No don't change that because we like the old way, you know I mean that's just not really what we're supposed to do Another thing that actually that change proposals really do and I think do well is I agree with you that the team needs to kind of figure out what they're doing and propose something But like in this case DNF is so pervasive across the entire project that they Probably very likely do not know all the things so once the change proposal is out there people can come and say well You know my thing uses this and have you talked to me? Well, we didn't even know you were using it So I think that the visibility has helped a lot there and this is where I say this is the change process working as designed Because if you think about what a change is actual proposals about it's a description of An implementation of something that is going to affect the distribution and the community right so the community Responds so that provides feedback in kind and we as fesco collate that feedback and turn it into something In bite-sized chunks that they can that they can turn around and and provide something actionable like we go through and we Understand and like when it goes to a meeting we say like these are the things that the community Highlighted to us that are important, you know, please. Can you can you find some way to address these? so for making this acceptable and that's essentially What our job is at least the way that's how I've yet and we're doing and and the dean of change is and Actually both changes. We've talked about today how our excellent Examples of how we Arbitrate those changes and make sure that you know, we try to have the community's best interests At heart when we're when we try to make when we implement things in fedora Linux I have command and I concur with you and disagree with David I don't feel but fesco is just about vetting stuff, but like foster and then a Lot of people things that fesco has executive power and we Know that you have no executive power You can do anything. I would love to know where this executive power comes from. Yeah, but I think you have a huge power And that's the poking And we actually seen that when this be shake start like what's the status of the DNF 5 and that was the moment when this starts Happening to move and that was just just a poking so so like poking about the other stuff like pushing Our job is cat herders, so I think I actually we don't even have poking power We have mostly stopping power, right? I mean fesco unfortunately can say no, but it's very hard to Force any kind of action. Maybe as individual members we can poke stuff But as a body we can only say not this cycle Yeah, but they could also just ignore us and do it anyway. Yeah, they can ignore you but but but We mentioned here like we are waiting till the DNF 5 will become default And then the semi-dead projects will only fix the stuff and I think It can be Benefit your body group to like poke before like yeah, it's going to happen Please poke poke like change it change it, please. Yeah, you can if they if they will ignore Yeah, that's just a life and it can change from semi-dead to debt project and Well, I actually I think that's where change proposals can be proposed with Like more structure around them like like have have deadlines within it Like you know we intend to land this change initially give us some data points that we can do that poking by But I mean really Stopping power is kind of the the only thing we have and that that's why we have the contingency plan Requirement and too many change proposals, you know, we'll come in and say not applicable You know self-contained change or something and it's like well, maybe that's true But we still need a contingency plan, you know, we need the ability to back this out What do we do when everything breaks, you know, basically yeah Like I've seen a few change proposals that had been before they even reached the mailing list like where You know Ben when he was you know doing the thing is like yeah He would kick back ones that like had the section completely empty You would be surprised how many of them don't have one at all Before they get to the point that they get posted, you know, that's actually We maybe at another event we should do a presentation or a Workshop hackfest kind of thing on how to write a good change proposal. Yeah, so We've all written some yeah, and I I help people when when they have questions on how to put it together and it's it appears trivial, but It you know, there's there's key information you need to think about and put in there Although the hardest part of it is actually doing the work Well depending on the change proposal, but yeah, yeah, but see you can you can write the change proposal And then have someone else do the work. That's great. If you could get it to happen. I Mean one change proposal was literally one change proposal I did was literally in including one extra package Into a cons group. So it was a one-line change But the change proposal document itself was like I think six seven eight pages long and it's just Yeah, so that also reminds me of it was one that I think was a banded some years ago that was proposing There were some spec file macros the really complicated Sort of all the forge macros Not it was it was something related to that Yeah, it was just the change proposal was just so long and so detailed But whoever wrote it sort of abandoned it. Yeah, and and I don't I'm glad that happened because if we had to discuss it and vote on it I would have had like a crazy long meeting one day and and then never voted But yeah, actually if you look at the wiki for change page incomplete, which is the initial status There's a whole bunch of interesting things out there, you know You look at a proposal and you're like, huh, that's actually kind of interesting But it was somebody who started doing it and you know for whatever reason didn't have the time to complete it Or whatever and that's actually interesting to look through back through the history there And sometimes it's it right and sometimes you look at one of those things. You know, wow That's a really good idea. Let's see if we can revive that Yeah, I mean a couple of the changes I did were actually Me reading through the old features list when they remember when they were all called features way back in the day There's some old abandoned ones there and it's like oh this this is interesting Let's see if we can like pick it back up now that things have changed enough to make it better Like it's also if you if you're thinking of doing something cool or interesting or you had you want to do something cool And interesting and you don't know what you might want to do you could go look at the old graveyard of changes and And see if there's something that catches your eye and maybe you want to spruce it up and try it Just I'm gonna ask. Please don't revive the fedora k free BSD No, don't do the idea that that came up at some point. I don't know if it still exists on the wiki, but yeah someone There was a joke proposal several years ago. We hope it was a joke proposal I hope it's a joke proposal to to to add the k free BSD and then like add the Resulting infrastructure to be able to dual test everything on Linux and BSD. It's like no Somebody did that somebody actually did do that as a derivative distribution. I don't ever want to see it again Justin wait, wait, I can't hear you Sorry, I'll start building that currently coper we can we can get started now Yeah, and and if you do that can you also build the Darwin kernel too? Oh, yeah, I know there's a lot of people that would love to Darwin. Yeah Fedora Darwin, I don't like having hardware working. So, you know, that seems like an ideal kernel I actually IBM did some experimental kernels, too We should maybe throw in I believe they use Linux drivers to make it easier like the K13 kernel and things like that Oh, that's perfect. Yeah, yeah, then we're good to go. Let's go. Yeah So joke or not we needed on the recording and also nothing ever gets deleted from the wiki. So I'm sure it's still there So We had a question in the chat what all things Fesco is thinking to do to explain how change proposals work and how Fesco works to the larger crowd And we have been touching on this in some aspects, but maybe so There were a few things that actually that Matthew came up with that I think are pretty good, especially for the discussion side There's actually a video out there that describes the change process and being having a link to that for people who like video Explanations of things, you know, right there in the top also some more Explanatory stuff, you know in the change proposal itself Like maybe going to the process or a short summary of what the process is or something like that I think could help but yeah, it's It's difficult to to provide that context with somebody who's just coming into new to the project It doesn't know what's going on and we've added the disclaimer and stuff like that to the top But it's it's difficult to convey like the whole process But yeah, do we have a section in the documentation about the like writing a change proposal? You know, I think so yes. Yeah, we do we could probably expand on that Maybe do some blog posts just raise awareness as well. I think it's a topic that we should probably Be continually Revisiting just because people come and go like it's one thing to do a one-off, you know, and and that's good But the audience changes. So yeah, the video is a good idea and you know just increasing the Marketing and communication the heart the hard things to do The other thing that's really interesting and I thought of this earlier And then it kind of faded from my mind and now we're talking about this again So it came back is the way people talk about as if there's a fedora team that does stuff I would love to know what this fedora team is because I've never seen them. Aren't we all a team here? I guess I don't know would like but the it's really important to know the really important thing to convey is that a Change proposal can be done by anyone who is involved in in the fedora project at all, right? And there is no like team of people that can make them and do the things and whatever It's just any member of the community who is interested enough to drive an Improvement to the project. I mean we've we've seen the changes process be abused for things other than the actually changing things in the distribution so clearly it has gone a little bit further than that but Like it's intended for technical changes, but we've used it for project changes. We've used it for culture changes We've used it for a lot of things Anybody who's interested in driving change in the project It is the method that the whole community is able to synchronize on and understand what's going on and Get consensus on so I think it's one of those things that It's hard to do from the inside because we have been living this process for so long that we don't see how it looks for people from the outside and maybe some newcomer should take on this Job and and figure out how to explain it to people who haven't been familiar with it for a long time Well, we all actually stopped talking That happens every once in a while Any anyone have questions? More any in the chat? Nope. How are we doing in time? We can Okay Okay, so some fling so you would have executive power. Oh, no, what do you would change in Fedora? Whoa So I have I have one thing that I was I was recently looking at fail to build Tickets in my packages and other packages And I there were a few they were like there was a package that was preventing 50 other packages from building and I start looking at it and I see oh Actually, somebody filed a poor request to fix this package a year ago and it's been hanging there in This gift ever since and I know if we could have a way to to just have poor requests out to apply after a week if nobody Protests then that would like to see that Are you sure you want that because like that's a be careful what you add what you wish for kind of kind of a of a wish and also, I mean it's so so we have the package a dashboard which makes it easy to see all the poor requests but It's also Apparently very very easy to miss poor requests if you I mean you get a notification But I also get there may be a hundred other notifications every day. So if it flies by that's easy to miss and maybe there should be some Mechanism to escalate this. That's one thing. I Have one like it's it's on a lighter notes. I if if we add executive power. I Would like to bring back the fedora release naming and Yes, the reason I say that is that And I'm not saying it has to be implemented the same way But what I liked about that is it was a participation point for everyone in the community Regardless of their level of contribution or their technical skill Everyone could vote on a name and then we had like each time we made a release It felt like we were this community around that release and we would have that branding now We did make it extremely difficult the process and stuff like that sure I get that maybe we could fix it but I liked the I liked the release naming thing that we would do it's always in the back of my mind is that I wish we kept the code name stuff because It was it was a fun point like one of the earliest entry points for me was you know helping pick The name for I think was one of the names that was on the voting roll for fedora fedora 12 It was a long long time ago and like I I Think that that is like literally one of the easiest entry points that someone could have to have Some ownership of what we what we make as a project. Yeah, exactly like people You know you if you've just joined where we're about to do a release You know you get to vote on this name and you you see it written up in articles and online and stuff You're like, oh, hey, I had a hand in that and it's just it was just a ongoing Entry point for anyone to really join Maybe we should bring it back for that alone. Yeah, you know wouldn't that be easier with the discurses Where you can vote just with plus one and I don't think it was the voting that was it was the names that were on the role The problem was of course as it often gets down to lawyers. So yeah, um it well the spot can Share some some backstory with that But it doesn't it doesn't look like you you really want to right now, but oh Yeah, so so maybe I don't know maybe we can get out of that business now, but I Mean just just from a community participation standpoint. I think it was really useful That said Of the release names that we did have What do you guys think was the the best name? Oh? Ever I'm gonna I'm gonna say Zod Shoulding there's a cat was shorter. Here's cat is mine as well. I mean we also tested the UTF supporting the distribution. So Yeah, I'm gonna go with that. Yeah Although werewolf was a close second because of all the kinds of combinatorial like the community artwork around it was phenomenal That's true. I I do like the Schrodinger's cat one because it did create the release blocker So on an apostrophe. Yeah, so the the act of participating in choosing the name actually led to Actual technical bug find, you know, which was which was neat And well Sometimes have the feeling that it's too hard to do things in federal like that Things that should happen and in general people agree that they should happen but they don't happen because stuff is slow and so sorry, I mean, I There's many places where I think we should as a community keep in mind that Just delaying things indefinitely is not a good Outcome and so it's so if I had executive power, I would just use this power to generally makes makings happen if possible I'm gonna go with that be careful be careful what you wish for for that one again because That can go very very badly. I Have another one on on on Exactive power. I'm gonna assume that we have an infinite budget. Oh, yes Let's go with infinite budget on the condition. So I would like to hold a flock and fund everyone in fedora to come to it Yeah, regardless of distance and all that stuff. So that I with the infinite budget now with the leftover money What would you guys do? with the leftover money of an infinite budget Next-generation koji with being able to do reverse step tracking all rebuilding because then Then I don't need to hear of proven proven package or request simply because they were the unfortunate souls of a library But they can't actually upgrade anymore because they can't fix everything I would like that kind of grunt work to just go away. Yeah, I was gonna say I could spend an infinite budget on Infrastructure, there are so many things if I could hire 500 people to redo this thing in a sane way and Fix the tooling make make things so easy for package It would it would be amazing. Yeah What do you see is the future of disk it in light of things like Packers Current maintenance status and people moving things to get lab and things like packet leading more people to Not really treat this kid as the canonical source for spec files anymore. All right Let's leave time for it. All right. I will keep I will keep my statements on this short. I promise Having pulling away from the canonicity of disk it is a recipe for disaster I have been in distributions where so some of you may or may not be aware that I actually am involved in much more than Fedora Over the years have been involved in over a dozen different Linux distributions Who have done a variety of different ways of shipping software? One of them, you know from a lot of those experiences one of the things I personally feel is a colossal mistake is Orphan in your agency of the method in which you ship software And so one of the things that we have to be really careful about With stuff like packet and what we are calling source get others called merge source trees or Or get package trees or whatever you want to call it, right is We cannot orphan our agency of What we ship because we are still ultimately responsible for it the distribution get or disk it is Our packaging get reposed that maintain our agency for that the canonicity of that hosting it inside Central Fedora infrastructure is how we preserve the integrity of our systems Projects like Debian which do not have that requirement Have to work around that problem by doing things like their reproducible builds efforts which Give say that for packages that are going into Debian We need to try to find a way that you can do a Reproduction of it from the output artifacts to recreate the output are in from the inputs produced by the output artifacts To create the same outputs again We have much more sophisticated ways of being able to Verify builds Then they do because of these other guarantees that we have that to be frank We don't talk about we don't talk about like why we've made a lot of those early decisions and what what benefits they offer The Packer versus whatever thing the maintenance status of Packer is In flux because we've got new contributors coming in who are actively working on the project earlier like late last year as one of the I was able to with the Fedora infrastructure folks finally get the CI Infrastructure working again, which means I could start accepting contributions and that has led to a Slow revitalization of the project and in a way that we're actually heading towards a new major version Where we're doing a lot right now. We're doing a lot of cleanup killing out a lot of cruft code There was a lot of weird experiments that we've done that are not needed. So We'll see how that goes. We have three minutes left or two. No two minutes left Well, I will be actually very brief also I Think we need to have another discussion about this. I know people maybe don't want to hear that Oh, no another discussion, but I think things have changed and I think this shouldn't be a top-down dictate You know, we shouldn't say oh, we're moving to get lab You know, that's that's what we're doing. It should be all right Here's our options. We could stay with Peg error. We could revitalize it. We could use it We could go to something like Faustodon or Gidea or you know those sort of products now That are out there and have the possibility of federated repose we could We could do get lab We could do get lab in different ways. We could run our own get lab We could let them run it to get lab for us, etc. Etc. All these things are a choice that our community should You know have in front of it and look at all the facts and figure out and maybe it goes to a decision from pesco Or something like that or maybe it there's constraints around, you know costs or whatever that that way into it also But I think we probably need to have that discussion again I don't think that it you know, we're going to get lab is a Conclusion from several years ago that we should stick with said that's my take Yeah, I I just want to add one one thing because you mentioned packet as well as where disc it lives So I separate sort of thinking about the workflow of package maintenance and and where source lives and and that sort of stuff Versus the the system you choose to put it on like that that is a there related problems with different what I would like to see is a discussion and decision in fedora about the Direction that we do so packet like if a project is going to adopt packet Building from upstream then I don't want to also see work happening in Disk it like I don't want to see patches and PRs there and have it be Bidirectional because that's really not like we should we should define what the flow looks like from upstream to downstream integration in fedora and I think I think that's important because it gets confusing with some projects So I think it needs to be handled the other way like I would all say that this kit is the canonical place And you are welcome to use any workflow you you want to put things there And I think packet is a very an option that this certainly growing in its usefulness But you it must take into account the The proven package of workflow and poor requests from the community. They just must be integrated Into I mean like any changes that you do must Take into account that things happening this get independently keep in mind that if you like let's say we go with the idea of like if you're using packet Then you don't have you don't people don't do into this kit that means for example proven packages are completely cut out That also means that other community contributors if they are not accepted members of the upstream project that's using packet They can't get changes in that also means if we do modernizations or infrastructure changes They can't do anything either like it's actually like an extremely painful Problem if we block people from being able to work in this case. So I'm not advocating blocking I'm saying that Fedora needs to define what those workflows are Because right now that's too ambiguous and and anything can happen and what I see is stuff gets lost Well, it's actually not that ambiguous the packaging guidelines pretty clearly say that this get is the canonical source for Spec files and patches and whatever but some people Think of all these reasons why they don't have to follow that rule or whatever it is Yeah, that and and that kind of goes to Communication and marketing and making sure everyone understands and is on board with with what what that says Speaking with my packet head on I get the problem is more like like not As Neil mentioned like not allowing people who has the right to do something in Fedora this get not allowing them the upstream but like when people use packet or other tools and maintain the spec file somewhere else and The this is just technical way where they put the stuff. So Koji has something to build and if people submit the Patches or poor request there. It may hang there for weeks slash months slash years We don't need to make that for it for us to happen So so so so like it's more about like letting people know that the majors actually do the stuff somewhere else and if they Want to submit some contribution. They should probably submit it somewhere else and it may speed up things So for me the canonical point of view like I Don't know what is canonical Think in in this context. So I think that this was a major design problem in packet. I Mean it was I Know it's it's sorry, but things will not work well this way and we'll dig a hole I mean like We have what 8,000 projects and with 8,000 upstream so we'll dig 8,000 holes for ourselves to to fall into Hey, I don't know that it's necessarily a hole. I mean I understand the problem, but like we're doing example Anaconda They I don't know if they use packet But they they do releases upstream and then they maintain the spec file there and just sink it to disk it but there are occasions where a compose will fail and Adam will figure out what the problem is and he'll talk to the anaconda folks He'll be like here's this problem, you know file bug a file pull request But I'm fixing it and just get right now and building a new anaconda so we can have a compose And that's fine because when the next thing comes along and it overwrites There's already a pull request because he knows that you know in the next version that they release He wants to have that fix already there. So I think it's to your point. It's more of a Expectation thing if you don't know that and you come along and you're like, oh, I'm gonna file a porcus or the kernel as Justin Will no doubt point out. It's the same way It's you know, if you're if you're filing a pull request there or changing something there, that's great But you should realize where you need to make that that's what I'm talking about. Is that that? So we do actually have well So we had documentation on the page when you went to the page saying with a link to come on our saying This is where it's maintained But we still allowed pull request I did actually turn off pull request because whether there was being ignored But I don't think that getting rid of disk it as much as I know a lot of people in red hat would say source It should be the canonical source. This is where we're working But I think this gets serves a good purpose there in that I do yeah, I overwrite this get now I check the diff every time so if somebody's done something I'll at least see it and Hopefully they've they've put in a merge request and and the source get but Having just get there. It does mean somebody can go fix it without waiting for everything else You know yeah, yeah exactly and and so that does serve that purpose But you also what we've just all said the expectation that you know, you're gonna fix it there But I know it's an upstream project that's gonna feed back in so I I understand how that works And I'm not gonna have that change loss, but right in the interim I need to fix it now. So the other problem is that we have not defined an expectation of upstream projects that wind up doing this because Here's the deal when you're integrating into Fedora and you're doing this stuff Ultimately the Fedora contributor has to win in a conversation Where something needs to be done to fix something and if that is not happening if there's no responsiveness in the upstream project There's no responsiveness downstream in the Fedora disk it something has to win something has to give and right now what happens is if somebody has gone to go Packety source kitty, whatever and and someone so contributes to submit to change upstream like they're doing it and No one responds this is a even worse situation than we have now because There is no agency. We have lost a way to fit to to realize a solution because we have deferred our agency to someone else who doesn't have to Care about what we're doing and that is the crux of the problem. I have with upstream upstream packaging But what do you just said is that you describe this kid as a Last resort. No, I'm saying that if you're following that workflow you consider disk it as a last resort It has to exist as one. Yeah, it has exist But but yeah, it's I think we are slightly heading to situation where we have to clear clearly say what what is canonical Source and differed from the last resort or some something to do about it and I probably With packaging we will probably get off in near future And we will see where we'll Had it us the important thing is that if somebody is going to use upstream project packaging in Fedora and Then ship it into Fedora They also need to understand that there's an expectation that they've got to respond to Fedora needs or otherwise We've going to have to do something right well I think the point there is that things can change Yeah, like say you have a project and you're you're maintaining it upstream and you're just putting that into Fedora Through disk it and everything's great and Rosie and whatever and then maybe somebody takes over the other the upstream project But you're still maintaining the downstream thing and they make some change And they're not responsive at that point you may want to break that Relationship and you may want to say okay now we're just going to maintain this here Here is the canonical source because the upstream is doing things that I don't understand or be not being responsive So I think there needs to be a way to tell people what the canonical source is and It can change it could be that it's not the same tomorrow that it was today Yeah, I think it's a big problem if we're saying that this kid isn't the canonical source anymore I mean one from the agency aspect and for the ability for other people to Collaborate on the distribution together, but also for the contributor experience like if one package is using get lab And one package is using code for another one's using one is using get that free desktop that whatever It just creates a very bad experience for contributors and we kind of lose that collaborative aspect And that anyone can go in and leave a review And I don't think that we should be you know trying to communicate to people that the canonical source is somewhere else I feel very strongly that the canonical source should be this to get Yeah, so this is you know when I look at the contribution experiences between Debbie and and Fedora This is the core difference because Debian doesn't have a canonical source requirement Which means that we do they do not have a singular path to contribution for any package It could be anyway Actually, you're not even required of a VCS for Debian packages You can just upload things and they will do that and then you're just gonna have to figure out how if you need to do Something how how it's gonna get done and that's not something I want to have in fedora at all on the other hand Who has more packages Debian or Fedora? We're How many of them are alive? Is it a Debian because and I think like yeah, it's make hard thing contributing to other packages But on the other hand like this situation make easier other contributors Taking care about new packages because it's more easier for them. So so so Very very good problems I think number of packages is less meaningful than it perhaps used to be because now we have rust and go and yeah A lot of that stuff is just sort of auto-generated, but you know, yeah Also, like you have to think about the quality and the usefulness and the liveliness of those packages There are a lot of packages in Debian that do not have a functioning Upstream or development or whatever they're just carried forward forever And they don't fail out of the distribution because of things like oh, they don't compile because They don't they don't do those kinds of things like we do where we fail out packages For compiling for not compiling on a year after year right like those are the way we handle our distribution Tilts towards active maintenance, which means we will always have less packages Than some other distributions that don't have that preference Yeah, and I personally would rather have more high quality packages than less low quality packages because Fedora is you know a collection of high quality packages that has guidelines and standards Not just a kitchen sink of every single possible open-source project in existence so We are 11 minutes past the I'm initially planned end of life of this session. So I think that I Mean this is a topic we could continue on for for an hour another hour. So let's just wrap it up now and Thank you all for coming. Yeah. Thank you guys. Yeah. Thank you. I think we can we can get started Yeah, hello folks My name is Neil Gump. I was here on the stage It's a little bit ago, but I have a new companion this time David Duncan. Come on introduce yourself Hey everybody, I'm David Duncan. I'm a well in the floor team I work on the cloud edition and I've spent a lot of time Working on that in my professional life. I'm a partner solutions architect and I work on cloud Cloud images and cloud solutions for Amazon Yeah, and I actually also wind up occasionally working with him professionally too because I work at Red Hat as a Senior black belt on managed OpenShift solutions and that means that I get to interact with him when working on Red Hat OpenShift service on AWS. So or Rosa as everyone loves to call it So we're here kind of to talk about, you know You know with you all of you. This is a participatory panel thing about You know cloud images and getting you know people to be able to make their own what kind of process type stuff and You know what we hope to achieve and what you would like to see From Fedora cloud to help make your lives better running Fedora in the cloud. I Think I think one of the things to you that we're we're curious about is Where in your position, you know, where in your workflow might you have a you know a good space for something like a bespoke image We spend a lot of time looking at what we can do to create images that are small minimalized and and provide Sort of a generic base, but That's not quite I mean, that's that's just the tip of the iceberg. You know when it comes to Come comes to how the cloud Cloud experience works, you know, you want to be able to Generate manifest and create Application space that is specific to your workloads, I mean there are many things that we we We do in our daily experience that That we don't want to do twice Effectively and so just to give you a great example the the core OS images that are created for building out the the OpenShift services for all of the public cloud providers is Is in fact Not just core OS it is core OS and series of tools and technologies that are necessary for deployment So that we can get that you know get that deployment model down or or decrease the amount of time to To ready to to a ready state Right, and you know you know like with things like reddit OpenShift service on AWS or if you're running, you know Maybe as we were talking about yesterday in our talk about Fedora cloud KDE, right like these are Scaffolds of workloads that we build to provide something useful and differentiated on On the cloud that that you will be enriched with and we want to know like we what we want to do is You know we want to help you be successful in the cloud doing these kinds of things and doing cool stuff with it and We want to know what you guys are doing in the cloud with the stuff to help us or to help them Yeah, just just as important is is what documentation do you think is is? significant to make the next generation of your of your Experience with the cloud edition better, right it doesn't have to obviously doesn't have to run on a public cloud Or it doesn't have to run on some managed cloud it can run on your desktop and and We expect that the experience to be just as important just as just as critical For for others and with some of the things that you can do now with cloud in it and in vert install and yeah There's a lot of ways to like there Like there are some of the things you may not be aware that the the cloud working Fedora cloud working group actually Maintains is you know the vagrant images are ours, right? Like we we take we take care of the vagrant work stuff I think we recently started doing what cube vert images or something like that and We're also doing things around Adding it to additional cloud providers we just added Azure recently and We're looking at Oracle cloud and a couple of other Providers just to kind of round out the table of things and we've always had the network of You know the common VPS is like your linodes and digital oceans and things like that that we've also been Having our offering on for many years So anyone got something that you want to? Yeah Hey, how are you doing? And my name is Brian and I work quite a bit with upstream Qvert So I actually came here today to ask you about if the clouds it would have an interest in Publishing images for Qvert VMs basically. Yeah, yeah, I think we have the definitions for making them But we don't know where to put them Yeah, so I think at the moment we're building our own kind of container disk images Which are basically just kind of wrapping we say for our cloud sent us cloud Images, so they rely on cloud in it and then we store them in quay as container images sure So Yeah, it's great to hear that you're interested in actually developing some of these images because that's fantastic Like our goal here is to provide all The deliverables that are needed for people to have an end-to-end positive experience using Fedora technologies in In the cloud development and production process So whether that's on your computer or actually in your infrastructure or somewhere in between like we want to we want to make that experience good and Help help you succeed in that front Yeah, so what you know one of the things that made us excited to talk about this today was that we wanted to talk about what tools are available, right and so a A couple of the things that are available right now is is Is just being able to create the build definitions inside of Koji and then build those in Koji And we were worried that you know, there are a lot of people who don't know or in our community, right? who don't know that they in fact have that up that that Ability ability. Yeah, just today right to generate those images in a way that is consistent with their expectations I know Adam knows about it because he does it. Yeah a lot We appreciate you Adam yeah and well and but he was also very vocal about learning and you know learning how to do it and and There's probably if you were to if you were to just wander back in in the infra lists You'd probably find some great great conversation around around that Yeah from him But we want you know, we wanted to make sure that everyone knows that these the tools are there the There are some other you know, there are tools for uploading to Cloud providers and and whatnot that are there and we while we don't expect that we'll be doing that for everyone Individually we expect that you know if there's an image that we need to have in a place that makes it available for Specific cloud providers that that's something that we would manage and maintain for you And then the you know, obviously OS build is out there to help you do that in your own independent Like in your own individual accounts. What? Yeah, so if you're using Image builder composer whatever for you know, it's gone by a bunch of different names all All the things that OS build is the back end of right you have some way of producing these things In your own environment. We also I mean the tools that we're using in fedora cloud day You know, we're in the middle of a transition moving away from some of our Legacy tools that are kind of specific that can only kind of run effectively in fedora infrastructure To stuff that people can run anywhere. So whether it's OS build or In some images will be look we're looking at using Kiwi for and the idea is that we want to move To a model where everything that we're making is something you can make too Because like we know that as much as we want to serve and offer all these wide variety of things The key power of the cloud is being able to build for yourself tailoring it to your experience And we want to make that a core tenant of what we are doing We want everything that we're making to be something you can take and make your own Yeah, there's a lot of there's a lot of times where we get feedback from the cloud providers about you know security or Identify some some sort of vulnerability that they don't want to see in the machine image, you know as such They want to see it. They want to see a new image deployed and then that you know that that image has The governance around that is really is really complicated because we really want to just put out one image and then put out an updates You know associated with it But their process doesn't fit that You know fast forward to the customer experience, right? Then you start they start to have a deeper impact on their own customers their own customers think that oh, okay Well now we know that our process is to update our machine images and then and then or or just images depending on who it is and then and then The updates themselves are there for us to be able to make something that is more consistent, but then to make it like You know once once every two weeks or so, you know So we know that the people who are people who are working on Cloud solutions are also being you know have a high amount of pressure to issue New images instead of just creating a patch management model for for the instances that they create from those, right? Yeah, because if you're in the more ephemeral world and you have the ability to subscribe to a feed to get the latest image every time you You provision then you probably want to take advantage of that But on the flip side if you're one that's starting from this and you have a machine environment that's Sticking around for some time and you're going to keep it and lifecycle it then you you would favor You know being able to do patch management and things like that So being able to handle both types of workflows is something that we try very hard to also do but and that kind of comes back to you know, we were trying to make our tools easier and simpler and More flexible to be able to handle these divergent use models because they're they're both valid And we want to make sure that people are happy using these things in the cloud locally wherever Yeah, yeah, the Kiwi decision was one that that came about because we were looking at how to create Composable image definitions and it was just a fantastic way to do it we could create any basically any kind of image we wanted from the WSL all the way out and So where we were We've been pushing in that direction On the other hand, you know, the uploads and configuration and like post post config We started off trying to determine how we could build our own tools, but then it looked like Ansible actually does most of what we needed to do so So we turned our we turned the the cloud team has kind of turned towards Maintaining those The software development kits in you know in fedora so that we can have full support for the For the the Ansible collections that support them and ideally like these are things people can take and use in their own environments one of the you know, I spent eight years as a DevOps and doing DevOps type stuff and You know, one of the things that made my job challenging was that I saw all these interesting things and ways that people were doing stuff and Things that I could have loved to use in my own environment But they didn't make it accessible or available for me to be able to easily take it Adapt it and extend it for my own use cases. Well, it's still not easy to like Integrate with the infrastructure. Yeah, I know but I'm speaking aspirationally. Yeah So like this is something that we like David and I have been like working really hard on trying to like Make that a guiding a guiding point for us when we're when we're making our new decisions about how things are Supposed to work and what we're producing but it also allows us to put limits on what we are going to do and For example, we're probably not going to make a cloud variant of literally everything that that exists in fedora There we don't need to that but we want to but we definitely do want to make it, you know cloud variants that are that are going to do things that we think are Good a segue into more of a fedora experience So there are lots of people who come to those of one of the things that I talked about yesterday actually in the cloud cloud discussion was was the Was how we looked at the like cloud nine right cloud nine and inside of Amazon doesn't have a Fedora-based image, but it could and so, you know, it's for us That's that's something that we want. We we are looking at how we can produce It's I mean obviously obviously backlogged on on other things like getting rid of Python 2 7 but But but we're you know, it's it's a The you know, our our goal is to see how those things can fit into the process that you know processes that The users are already dealing with and say hey, I you know, I I couldn't use fedora as a as a foundation for this instead of just using You know some other distribution Yeah, and like it kind of riffing off of that a lot of this was well, you know Sys admins that are like in an emergency scenario. They're away from their main computer station You know being able to spin up like say a cloud Katie desktop that's got like, you know They're open-shift and Rosa command line utilities their cube definitions their access to their their version control systems All that push button provision so that they can do emergency work In the environment that they can actually be comfortable with I mean god I don't know if any of you have been assisted been in the mobile phone on-demand era like like I've been and But that sucks typing into a terminal and having to like do things from your from a from a crappy phone console That's in Android or iOS compared to like being able to access like a remote desktop from a web browser is Yeah, I would take the remote desktop in a web browser because then I'd have access to all the tools that I actually need to Function and and that's the kind of stuff that you know We're looking at to like enable cloud experiences, but also We want to have a framework where like let's say for example You know the Python classroom the Python say who maintains the plot Python classroom Lab for Fedora wants to also have a cloud variant They are perfectly capable of reusing our framework to be able to provide that on their own as part of their lab We don't have to do it for them. No, we would give them the tools and the capability just do it and make it part of their own deliverables we want to be able to provide that kind of flexibility to people in The same way so that they can actually have their own tools for success. Yeah, I mean I think we think other a lot of other groups are feeling that They they see that segue and we want to help them or not. Oh, sorry. Yeah, we want to we want to hear You know, we want we we think that there are other members of other teams that are looking at things that they want to be able to Stand up quickly. They want to verify You know, they may be working on windbind or they may be working on on, you know key cloak But they're but there's there's a lot of things out there that you want to just be able to to bring up look at and Drop and and that process of building those machine images and making sure that you have the updates and that you have sort of a, you know, basically a step You know kind of a step function style approach to ensuring that you have what you need We want to help make that make that a reality for lots lots of other groups Maybe you want to work on open QA on and on a virtual machine, right? Perhaps you want to be able to make your own just Environment to do all this stuff just with pushing a button in the cloud Any other questions? Yeah, just keep it So you mentioned that members can build their own images. Is there a good starting point for that? Is there a cloud sig docs page or is there a good starting point for that? I don't think we're quite there yet. We're trying to I mean, that's what we're doing That's part of our goals with the rework of our image build stuff In fact, that was a little bit of what I was hoping to to get us a springboard for today is to see where Where the interest is and how how we want to how we want to frame frame that documentation Yeah, so like part of the reason we did the cloud Katie you talked yesterday and why we're doing this panel today We wanted to have some interest and learn from y'all like what kind of stuff would you be? That you would find compelling and interesting to do With fedora cloud stuff and because we have some ideas of what we want to do And I and I think we're on a good path, but we also want to hear from y'all To see like what are you interested in? What are you looking for? What kind of questions? Do you have about like doing? Stuff in with fedora in the cloud and that sort of thing Yeah, I mean, I'll be honest the Kuvert thing is super exciting to me because You know, there's there's two aspects of that one Just generally Kuvert, you know, it's a great way to to create a great modernization strategy and to It segues into some of the work around lightweight virtual machine Environments that we have not gotten an opportunity to work on just yet But you know porting to firecracker and things like that is a big deal Also, congratulations, Kuvert for reaching 1.0. You finally got you finally did it. Yeah, we finally got there. Yeah And you also mentioned Rosa on the site. I was just wondering is is container native virtualization included in Rosa? Um, I don't think so. Can we I don't yeah, I don't think so thing Yeah, we so it's it's a him. It's a him question for whether we'll ever do it though so so Whether or not it'll be supported. That's a that's kind of a question. Yeah questionable, but then but like There are flags in the metal instances where we're still working on the metal. Yeah, like most the reason why we can't do it is because The instance types we support literally don't let us and also it just To some level it feels weird, right? I mean, I think there are reasons where I think it would make sense but like you could also just easy to and And so like but I also can get the idea of like you want to have The abstraction and the sim the same API everywhere to be able to do all the things But we want to do your testing and yeah nested vert and then there's that and I mean like part of it So this is going to step back into peeling back a little curtains about fedora cloud stuff Part of the issue that makes fedora cloud images currently today so kind of painful is because the tools Currently require us to boot up into a virtual machine to produce the cloud images Which and the cloud and the virtual machine boot up process is kind of special and That makes it difficult for people to be able to replicate our processes on their own And I want to move away from processes that are so complicated That like someone who's like just getting started fresh into the stuff and is excited Doesn't get so discouraged that they want to flame right out of the whole thing I mean you can you know you can flatten a kickstart now and just figure out what's what's in the You know what's in the process, but that's like that's just one way to make a virtual machine I mean you could literally make it in the same way that you make an open VZ You know or made a note made an open VZ image. I hope you're not making open. Yeah In the same way that you know we did a gazillion years ago and that's that's not That's not unusual. I Know the Amazon Linux team when they when they build in it when they build an image It's basically just you know, it's a tar file And I don't think that You know You know, this is this is also one of those places where there's more than one way to skin skin You know skin this cat and and I think that that's that's a an important part of the Of what I think is interesting for us is that we don't necessarily have to do this in a way that is consistent with the expectations of The the one tool, you know, if we decide or determine that there's something that we want to do here That's different. We can we can we can be versatile and as long as it's maintainable Because I what we don't want to wind up is in a situation where we have a litany of things where we don't Actually know how we can keep them going Agreed, but you know if we're helping somebody sure work on their process I don't want I don't want it to be constrained to like oh, well, they're using kiwi. So no That's not that's not what I mean, right? Like I if a tool is is kind of central to Building the thing that they that they want or need and we need to and as long as it's Straightforward enough to integrate and and and support. I don't have problem with it Yeah Yeah, so there's a lot of ways there's a lot of different directions you can go with how you make your you know, how you create your or how your process Whether or not that process is consistent with the one that we have and I think I don't think that that's that should be something that would concern you know concern you The most I mean to me the most helpful thing is knowing that you can pull this you can you can pull together this Configuration you can push that into the utility the tools that we have already in place you can get your build alerts in the same way that you would get them with any other Fedora projects and You know you can throw it you can throw them over to Adam for testing Or Adam might just do it anyway and then tell you he did it. That's right any other questions any other Comments comments. Yeah, there we go This is more looking farther into the future But one of the things that I'd like to do eventually is more or working with like GPU instances in Fedora and that's Predicated on getting all of the stuff packaged in Fedora so that it works. Yep But is that something Assuming we can get everything to work is that something you think would be reasonable? Yes actually so like We briefly mentioned this yesterday during our cloud KDE talk like GPU accelerated instances and being able to support them properly in Fedora cloud is absolutely on a roadmap of requirements Most most of the most of the public cloud providers also have the ability to distribute the NVIDIA controller drivers so we can we can we can cheat piggyback off of that. Yeah I guess the the one of the problems I'm seeing is where we're starting is with AMD not with NVIDIA that's fine too. I know of one cloud provider who has one instance that has an AMD GPU in it And it's not current. It's a couple generations old. It's better than nothing, but yeah Like I was just saying I'm just all the cloud providers do have NVIDIA There's Amazon has one instance that has an AMD GPU in it And I have not been able to find a single other cloud provider that has an AMD Well then props to AWS for having AMD Radeon GPUs. It's pretty good Yeah newer ones would be awesome. Yeah more new ones because it's not technically supported by Rock-M There's questions. I mean it's not on AMD's official list Well, whether it works or not is a different question. It's just it's not on their current official These are the GPUs we support and then the one that's in AWS is different. Well, I suspect you know I'm not gonna speak for David, but at least from the Fedora cloud perspective I think if we start having those toolkits in place and being able to have instances Have images and in with that stuff preloaded that they that instance that when they're provisioned on those instances That they can just take advantage of them. I think they're gonna see a different dynamic start building up for it But you know, what do you want to talk about on the Amazon side? Well, yeah, I mean so from from the Amazon side we we have our own Crafted drivers for those so Excuse me the The way that works for them from the Amazon perspective is the drivers are produced by AMD for For the AWS environment and then they're maintained in an S3 location for for it for the instances But they're available and they're easy for us to distribute and there is a rail workstation That's that was created for specifically that reason to have the To have two things that incorporated into the instance one was the GPU drivers for the Jeep for for both Standard GPU workloads in general purpose GPU workloads and then to have on top of that the the nice DCV VDI solution in in place so the server is already there you just connect to it with a client and it's basically like running Running it where you are and the reason we wanted to do the KDE Workstation is because we don't have an upstream kind of of experience for people to have have that same You know to iterate on for this for some of the same reasons so We're really super interested in making sure that you know customers have our users have that available to them Because the user I'm the user experience around machine learning on Fedora is a story that we very much need to tell Yeah, I know I did it. That was part of one of my talks yesterday. I was You know what it is and hopefully how we're going to fix it. I am for for my perspective. I am super excited about seeing The ability to use GPU compute in Fedora out of the gate Seeing the Rockham stack it integrated in and being able to start doing that kind of thing I hope that it what it will do is encourage more GPU compute workloads, whether it's a IML or something else I mean render farms are another example of GPU compute workloads that Will use AMD GPUs with open drivers and an open Software stack that you know everyone it gets to exercise The flexibility and freedom that they need to be able to do what they want and not what somebody else prescribes of them Well, and just like you're doing you know Our goal is to build this narrative so that people don't forget or don't think that Just because there is another narrative, right that this one is is absent We want to make sure that this one is is very clear and you know in there Within you know within their space So for me, you know the machine learning. I mean if If you'll let me soapbox for just a minute The machine learning conversation is one that I think is really hard because first off, you know most of the applications that make this space Or work in this space are impossible to package, right? I mean we've had we've had that conversation several times and then The way that others have approached this Has been to take all of that Stuff that they can't really figure out how to package throw it into slash opt create an image and then just just make that available And that's how kind of you It's a scary, you know to me. That's not it's not a helpful model, right? Like there's no way to do updates. There's no There's a consistency there That maybe I I might like as a as a scientist But the but the real But this goes back to our like the whole idea of developing images for yourself We want to have those images in you know like in my project in my account, right? I want to have those those Those images but that doesn't necessarily mean that I want to find them out there in the world, you know In in an incredibly vulnerable state for you know for super long or I want to you know I want to find out that they are they disappear at some, you know, amazingly fast rate That makes it impossible for me to have a steady workflow so I mean I would love for us to to to To try and figure out how we can do that, you know Akash deep and his work on on the neuro neuro fedora. I've thought was a great a great entry way into this space and But it's but it's not, you know, I mean open data hub right was was a really important part of that process around the open shift experience and and you know, we we have I mean I have built out machine sets, you know that were that were GPU enabled for for doing machine learning workloads and in in the open shift space but Really a lot of that work can be done on a single machine in a very, you know I've sort of a limited space and so you can bring up one one instance Create, you know, do whatever it is that you're doing against what whatever data lake it is that you're accessing and then turn that off and Helping people to understand how to do that Means giving them the tools that they need to do the work Just I want to follow quick on something that you had said a while back to make sure I understood So it is is it is not a requirement for for Amazon to have the driver supplied by AMD It's just something that was enabled So if we had something in fedora that was completely upstream and didn't have AMD's binaries That is an option for yeah, that's only an option. You just wanted to make sure I understood correctly Yeah, you can use the open the open driver stack in there if it's pre-loaded Amazon just happens to have a bundle that you can just source it and install if you don't have anything, okay? Yeah, without getting without getting into trouble No, that's an important part Anybody else then I'll just ask I'll ask Isaac's question. No, we don't have the risk five images Just yet. Oh I You know I have expected someone gonna ask it like and I was so prepared He's leading the witness here. So I'm gonna I will I'm gonna ask two questions You just asked one of the questions, but the real question is when will Amazon? support risk five in Their cloud public cloud. I'll defer to people who are you know more more capable of answering that question That's the chicken and egg. So when Amazon declare that they're gonna do that then we can say okay, so when are we gonna have fedora? Images to sit on the risk five boxes. I Mean, why would we have to wait for Amazon to that because we fedora cloud does way more than that So fedora cloud is also, you know, we can you can run these images on Personal hypervisors you just boot them up with vert install and pass in the cloud the cloud in it user data or you use You know, you're using cockpit like in there and then that actually will boot fedora cloud images as as VMs Or you run an open stack on risk five or you're running Cube for like there's lots of different ways that this could happen and that's part of the fedora cloud thing There is but he's mr. Amazon. You're mr. Fedora, right? so That was a blunt question for Amazon and from a fedora Perspective then you still have the hardware enablement conundrum, right? Yeah, well, so yeah Yes, we could do all the things you said, but we have to have the hardware enable to begin with it's customer driven and For us it's somebody's got to give us the stuff to be able to start doing it because we're going to help triangulate the giving of the stuff to help you do it and We're also going to Politely encourage all the cloud vendors not just Amazon to support. I mean you can It's on public record Ali Baba actually have a tea head where they're developing risk five servers already And and their hook line and sync bought into this equation I would love to see Amazon do the same. I have nothing to say I mean, you can't you can't rule it out, but you can say the roadmap is definitely driven by based on What we're what we're being asked to do And all I'll say is that risk five is an interesting architecture, and I hope that it's bring up goes That the machines bring up go better than previous machine bring ups have Probably a small question, but do customers not want risk five on Amazon? I mean You're asking the wrong guy, but yeah I mean, there's you know, there's some other people who are more deeply aligned with the you know Peter DeSantis would be the one day and so to ask that question not not David Duncan and For speaking of people in the third person Neil Compa is just like well, I I would like to have As many architecture supported in Fedora and and offered in platforms and be usable in all the different ways But I suspect but I suspect that really it's just most people don't know that they exist right like you don't ask for something you don't know exists and and once you know it exists and You find a reason to want it then things the dynamics start changing And that's true even in what like the Fedora open source case Like we started having this interest and development into all these you know These architectures because people come in and they're like we really want this and they start talking about it And other people get excited about it. It's that flywheel of of success So I just like that to that that's yeah, it's a bit premature and early yet for Enterprise servers that are risk-based But they are coming and if anyone's familiar with the guru or the demigod or from Mount Olympus called David Patterson David Patterson maintains that risk life is actually going to surplant all CPU architectures over the next ten years And he's a you know, he's a former x86 guru etc and he's ridden the wave at arm and Risk 5 is something that's it just gives so much choice flexibility To the HPs of the world the startup chip vendors of the world's customers countries like India, China, etc That it's an inevitable Equation, but we're not quite there yet. So at some point, yes the Goldman Sachs and the Bloombergs are going to start consuming this and Amazon gonna is gonna spring a surprise on us and Hopefully we'll evolve on the groundwork to to enable all the hardware on donations then we can be off to the races. I mean That's very lofty my comment on that is that I remember when James Hamilton said that he thought that arm was the was the future and Then it was like three years before yeah Yeah but three, you know three years for him to make it make it work and it's very interesting to see from the you know from the perspective of of a An engineer Because the questions the question, you know the questions that That happened as a result of that were like More economic than they were than they were anything else so it's it's very interesting and then and then of course like a series of really amazing engineers started to solve really big problems really fast and You know, it's just like that seeing that the way that hardware can evolve in such a very short amount of time always makes me it always surprises me and and I Feel, you know fortunate to have worked with a lot of these people I Will kind of add to this a little bit You know your comment about how it'll be everywhere and surpass all the other architectures and whatever The the concern I have right now is that when you look at how risk 5 is evolving and how Development is going and manufacturer interest is growing and how CPUs and all this other stuff I Find some degree of credibility that there'll be more chips risk 5 than anything else by the end of the decade I Don't know if I could put a pin and say that it means that risk 5 will be the dominant Computing architecture because of this before the fragmentation between the risk 5 variants and instructions Sets and stuff. We don't know how that's going to settle out yet because it's still changing before this day But before we get on to the onto that question Let me just say that one of my favorite places to watch what's going on That's going to be relevant to the way that cloud images are being built is the work that's being done in the vert team that Karen runs and And a lot of the things that start start to come back to like what's what's being? added to the QEMU KVM Code code bases makes a huge difference in terms of like what we can support so I mean I would like my my My my goal would be to make sure you know to ensure that that KVM when you Support support is there and it's there in a way that we can we can all take advantage of and importantly You know as I was saying with the fragmentation of the instruction sets Whatever gets implemented in QEMU and that the distributions and all of us rely on The hardware vendors have to do can't really be the other way around if the hardware vendors start stratifying the the the the because risk 5 Ties a bi to instruction combinations so if your instruction combination isn't correct you cannot run the application and so we can't have a situation where Something calls itself 64-bit risk 5 and you can't run it because you don't know Like when it tries to come up. It's like oh all the instructions are missing So we can't run like there has to be a guaranteed baseline that everybody's going to support for that end Like even today now with fedora risk 5 being bootstrapped There are two other separate efforts going on right now that are doing the same bootstrap For different instruction combinations and that's the part that scares me because If we wind up being in a situation where we have to deal with the different instruction subsets that are not compatible risk 5 is going to fail Because well can't figure out how to work with it. I want to make sure that we all know we're on a tangent Yes, I mean we are we definitely have right hold but let me go back and say the tenure comment was a Patterson quote Yeah, so put that in context. Yeah, and also that You know new architectures require a lot of work under the hood from all the distro vendors upstream kernel enablement and That in itself is what actually causes a standardization Yeah, and helps prevent some of the potential fragmentation. Sure. So that's the beauty of open source when you're working things and and it's open hardware definition and Because it's in the open that means that the definitions will evolve with a little bit of common sense from the community sprinkled all over it So I'm not really too afraid of it going off on a potentiary on 50 different tangents Which it could it could for all the edge devices, but rel is never going to run on those devices Sure, so it and Amazon's never going to have to worry about supporting them So for the enterprise server cited the equation, I feel really competent Confidence that things will Consolidate and come into focus because of the community effort I mean it's everyone here and everyone beyond here that helps Create risk 5 and helps create Linux and and that's the beauty of open source So to tie it back to fedora cloud as we're talking about for this to bring this tangent back to the nose I don't care just about the enterprise servers or in just the cloud but I have to care about I care about the whole cloud experience down to the person's computer and so You know whether it's risk 5 or whether it's arm or whether it's you know running a desktop Or if it's doing kubernetes or whatever or s390 or as Mainframes are special. I'm gonna that one's getting put aside But like when you when we're talking about a fedora cloud experience for that I Don't talk about just Amazon I talk about just I also talk about like the desktop that someone's working on to develop the cloud workload and I talk about the the the Experience of going between the two and I and all of those other things So there has to be an underlying level of consistency at all those levels You can't have an experience that only works in the server side It has to work locally too and that's the piece that You know with arm it took a really really really long time It took twice as long as it did to get the server parts the server parts were easy getting everything else was harder and and I am very happy and willing and optimistic about supporting risk 5 for fedora cloud But we have to have all those pieces in place to make it work And again, it'll be because we have all those component parts in the KVM when you code base and we'll be looking for that that's what we work on and and And that means, you know that means dealing with a lot of special problems like like, you know, how do you get I mean? On cloud instances one of the things that we worry about is like how do you get a dump, right? I mean how do you get a memory dump? What happens to the program crashes? Yeah, where do you where do you get your? Yeah, where do you get your debug? Those are and that's actually a surprisingly difficult problem it is but but we want to be there to help you and anyone who wants to You know to build specific specifically cloud images Our cloud-like images the the opportunity to To have our help and to be a part of the part of our you know part of our experience and part of our Really part of our team We're open come join us if you're interested in the stuff actually we need you to come and join us Anything else I think we can call it Thanks for being here. Hello. Hello, everyone. My name is Lenka. This is David And we work for Red Hat in the community platform Engineering team our colleague Patrick was supposed to be here with us, but unfortunately couldn't make it Today would like to talk to you about How we authorize open ship hosted projects to the community members in Fedora Why did we need to write a solution for authorizing open ship hosted project Well a couple of years ago. We had a community face in open ships cluster called community shift most of you probably remember it Please up your hand or remember a community many many people That was supposed to serve for people to deploy test apps learn experiment get their hands dirty and This cluster was retired as part of the data move of the data center move in 2020 and Then in Q3 2022 Community platform engineering was tasked with solving a number of blockers holding up the Re-release of the community shift cluster This talk is not about community shift as the cluster is not ready yet to be presented to the community We're still trying to work through some legal blockers Which we as developers cannot influence But while working on the community shift There were a couple of areas to solve and I would like to focus on the on the one on the Authorizing one we needed the administration of projects to be self-service and with no or as little added work For the Fedora infra team as we had a lot of tickets being opened Regularly request and access the systems We also needed a way to sing Fedora account system and the cluster And we needed this to happen automatically Apart of that we also needed the for the community shift purposes We needed to solve the storage and the quota, but that's not the aim this work of this talk And the Fedora account system API requires Kerberos authentication. So we needed a keyed up We decided to use the Ansible operator SDK Which provided us the structure of a Kubernetes controller then That we then completed with Ansible and Python that provided us the functions we needed So operator SDK is a Kubernetes native application that consists of two parts first is a container that handles the interface between OpenShift and operator and Reacts to received events the second one is the Ansible role and playbooks code which is run as required These operators can then be managed by an open source toolkit called operator framework We named our operator Community shift authorization It connects the fast json is all of you familiar with what fast json is all right, so for those who don't know I'll Resume it. It's a Jason gateway the query free IPA built for the Fedora account system so the operator Connects there the through a key tap file and Retrieves all the community shift dash something named groups and Group members Then it connects to the OpenShift API and ensures that all the data is synchronized So it's a one-way sync from the fuss to the OpenShift API Well, then we set it to reconcile every 20 minutes So it's looping every 20 minutes that can be changed if we need it later Well, why did we go with the operator SDK? What advantages does create an operator offer? Well several things SDK is a framework that does a lot of things for us all the structure the loop logic All the when we run the operator SDK in it. It just provides us all the three of the files and the molecule tests and we only set the reconcile period and Also, immediately, we have access to any module that Ansible can run Including custom modules that we can write ourselves which is what we did as part of the initiative and It's also reusable once we created the community shift authorization operator We realized that we could reuse the solution for any other project and that project emerged soon after It was called fast to discourse Faster discourse synchronizes The Fedora account system group membership the Fedora discussion, which is a discourse instance I saw that Matthew Miller will have a talk about discourse after after the lunch and In the beginning of the fast to discourse there was a ticket from Matthew who got inspired by David solution of authorization of community shift authorization operator to synchronize the fast groups and their members with the community shift cluster and He was thinking about using the same technical solution to sync fast groups with the discourse groups and David will now describe the technical details So sorry, it's gonna probably be a little bit of a piece of what we already said So the the fast to discourse operator. It's very similar to the community shift authorization operator So, I mean we still have the Fedora account system. It's our single source of truth It allows us to provide fine-grained access to various systems in the Fedora community So for example in this far as discourse is concerned. We can do things like members that are people that are in a particular group might have particular privileges for posting in a Certain area on discussion And again, it's a one-way sync. So basically from now on if You want to add people to a particular area in discourse. You need to do it through noggin and basically IPA and Basically, that's a count stuff Fedora project at org He searched for your particular group you add or remove users there and then this operator then will automatically synchronize the changes over Don't do it the other way. Like if you make changes to discourse, it'll be overwritten by the operator Yeah, and to activate this then You add you activate it on a new group in discussion by basically creating the group So if you create the group in in discourse the operator then will begin synchronizing providing that Yeah, the group exists in discourse and it's identically named to a group and fast Yeah, and basically this approach then we've taken could be used for other services in the future Yeah, so on the on the last slide It's a little demo basically it just shows you on the left you have the the Fedora account system fast and Here we're just adding a user to it and in the background. Sorry on the right hand side, then you can just see the So this is the community shift authorization operator You can see that that user then will be added to the particular group in in open shift. So in the future Every project in community shift will have its own group in fast and basically the sponsors can Decide in a self-service manner who who gets access to it themselves and it doesn't involve anything related to Fedora Infra you don't have to open tickets completely self-service Yeah, so basically that's all in theory as we said because the community shift cluster is not officially released and Perhaps in flock 2024 or something. We'll be able to give a workshop on how to get access and get on board it into community shift Yep, so just a quick lightning talk and just brief overview of what we did But if you're interested in digging deeper, there's a couple of resources there I'm just thinking yeah, the links to the two operators if you want to look in so it's all them It's entirely Ansible and Python, so it's pretty pretty readable to most developers That's pretty much all from us So if you have any questions, please free feel free to ask but we've only got a 15 minute slot So we don't want to take up any any time. Yeah, please enjoy the rest of flock So if you have any questions I can encourage anyone to look into the operator SDK and especially the Ansible operators that you can produce because You can pretty much access anything in the open shift API and it's a really nice framework Potentially rather than creating or rolling your own API. You could just use and extend open shifts or kubernetes Yeah, and the operator SDK is a really nice way of doing it. So That's it Thank you. Okay well Before then that and everything Thanks, thanks a lot for be here Well, we have a nice presentation From Mexico City and Mexico the country we are Fedora Mexico a local community From Mexico and this is our presentation growing together and how we Rebuild this community I'm Alex Callejas Let me present my team Alex Acosta Hasim Anaya and Ivan Chabelo So let's start with this Okay In Mexico, we have a lot of communities a lot of developers communities and open source communities, but All communities very very special. We have a Lot of technical people very very very important in Mexico and In the first years This don't happen. So Let's Share with you how how we can do this thing Okay This is an old graphic, but it represents all the active members by country in Fedora so Mexico have 45 active Fedora conti contributors but in our community in this moment, we have 1068 members on meetup and 516 members on the telegram group is very very huge so in the main groups We have Mexican contributors like ambassadors advocates packages and Okay, this time from story time we have to share with you or journey and or Learned lessons All right Well, as you have my already guessed I am the older guy in the group Probably in the conference, but well at least in the group and I'm the oldest and I'm also the the ambassador I'm not Thank you Also my my Membership or my fast account is pretty old. I joined Fedora in 2008 And well at that time it was a very small group. We have Like eight registered ambassadors, but only three of us where I were actively contributing to the project From those three or from those eight. I'm the only one who remains from that time What they were interest interesting times then because All of the events were managed and driven by the Fedora latam community which was bright and Extended at that time, but that also represented some challenges like difficulties in in the logistic for budget and for shipping. I mean Concentrating the swag in Latin America was Asian challenge to chip to Mexico. I personally live 250 miles out of the North American border and and I was Getting a hard time Having a hard time getting the swag from Latin America, right? This is Something that not necessarily a nature right now, but in that time the Mexico Red Hat office was just a commercial and support office and They they they was nearly to serial involvement with the community and We also have mostly talks at university and colleges that photo is from Food Contempe and you might recognize some of them Some of the of the people who participated in that time Tatica, Nucho, Neville. It's in the photo Good times In 2014 one of our founder members go and boss and ride to Managua, Nicaragua to the food con and there's where Our story start, but how we do it how we do it to get this community well Sorry Okay for the next part one of the main points to grow the community in Mexico was the Communication across all the country across all the people and one of the main points is the telegram group It's an official federal channel and we have more than 500 516 today Registering the channel One of the points that we grown with the channel is the moderation. We have a console We have every one is on a check-in area All today the code of conduct the community and Providing hub providing support providing advice Everyone can join Participate with the community and from this point we Manage all the me all the meet-ups. We have a monthly meet-up one of the most active in Latin America All the communication is Born there in the channel. Okay the next step that we have is Create a meeting of group Fedora Mexico In this time of the On the history we have support from Red Hat they Be led or Meetings and in the office so we can do Very very large meetings and We find the best speakers and topics of general interest different different people all to share His knowledge Like a set three we participate in community events Generate or own events like a release party a federal women's day and others community events like open source contributor summit Cost and Latin American flea software installation festival known as please all In this community events we have a huge participation of a community and we have a Very very nice events in the Red Hat Mexico office That the participation in the community events we go to Guadalajara to talk about how the people can contribute to To the Fedora project We do a workshop where learn to the people how to Contribute to the project since Translation and report books and packaging There's a lot of people there was a nice nice nice event And the Fedora women's day from 2019 we have a huge participation from the university the biggest university in Mexico and participation internationally because Tática Participate with remote with us and this girl Susanna he she's in in German in Germany, so was a Very nice event too. So We have participated in 12 out of 20 edition of the Southern California Linux Expo Expo, which is one of the it is the largest community Run open source and free software a conference in North America It's not it's not a completely Fedora Mexico credit for this but we have participated and it is important because as you know There is a huge Spanish speaking population in that area actually it is the second one Next to Mexico City. It is the second speaking Largest Spanish speaking population in in in the whole whole world. So This last edition we we earned the most memorable boot Fedora earned it. Yeah Because It is awarded to to to the voted by the by the attendees For the most I Don't want to use the wrong term, but it was like a more memorable experience for them So this is something that I I would like to it's because we're memorable There is something that I would like to brag about and this is a photo that that was English Ivan and myself were interviewed for one of the digital News news site Hasim Ivan myself Ruben if you recall him he's he's living in the Netherlands, but he was an active Ambassador we have all participated in in this conference as well Okay, but not always we have a happy place of course We have a lot of issues because you know the people have different ideas and the first issue that we found was the modulation and we talk about all these People like the Chrisman we are our friends Yeah, we've had Some trolling cases in the telegram channel There's some I don't know why people that use arch Linux or our clinics like to I don't know give us a hard time but we try to Well, I try to be Like count the tree Don't get mad and ask the other guys to calm the waters, you know Because I'm a little explosive and I'm supposed to be a moderator. So I always say okay Hey guys, there's someone out here saying that blah blah blah blah, you know stuff and We try we try to focus on the technical we try to focus on the social we try to focus on being friends and I think that the most important part is that we try to help each other if you enter the this telegram channel and you're a newbie and you're asking like the the Most basic question, you're not you're not gonna get the RTFM answer either a seasoned expert is gonna answer you or Also a newbie is gonna answer you and there's gonna start a conversation There's times I'm really happy about that because there's times that a new guy Helps a new guy and they end up helping each other and I just read it, you know I didn't need to be in that at that time. You just say well, it's cool. They they're wrong but they are They are finding the truth together and then someone comes in and helps him or I help but we try to to be very open for for the new guys and and Because most of the time they come really really excited and Have you I don't know if you if you have been on on more more technically hardcore oriented communities in which You feel like you you you shouldn't answer you shouldn't ask stuff Because they're gonna they're gonna think you're dumb or stuff like that We try we try to create the environment for the newbies to to ask them everything and And we are there to help also Oh The code of conduct I don't even I don't even haven't read it, but it's easy. Just don't be an a-hole, right? It's just like that and well At what point at some point, I don't know how or why I was talking to my friend Renich and he's a musician and he's also a musician and We started saying hey, we should he lives in in some area in the Jalisco state. I don't remember the place and We were talking about doing music in Linux and we all know is not like the most pleasant experience of experiences, right? It's still an area of opportunity and But he he was using a software He's not free software, but it runs on Linux Mac and and and Windows called Bitwig I think it comes from from some guys that got off of Ableton or something like that and And he said hey, let's let's do a song and I don't remember if I or or he he agreed to to To do the free software something Spanish and So he he he is a guitar player and a keyboard player so he wrote the melodies and and the guitars and And and and the drum beats and he sent me everything and I recorded the some vocals And I send them to him and then he sent them back and so back and forth It's not today. It's I like to say that it's not the Ideal but It's a it's a very nice community experience to make music remotely Only using Linux That's that's the main point and that that that's why that's why we we chose the free software song because which we were trying to prove that you can do music in Linux and I Shouldn't say it, but it came out pretty well Would you guys like to listen to it? Yeah, yeah this this effort a how about a lot of All the community join to perform this video some guys do some things This this girl Cassiopeia She Do the the musical video for this song in Blender Yeah, she's she's a leader of the Blender community in Mexico and and we send them with it We sent her the the song and she did I don't know what but it came out pretty well and and an animation around the sound of and the frequencies of the of the song and The thing and and just before you you press play I want to I want to I want to be very very emphatic about this We were we were asking which in which community shall we share this stuff and Like I don't know at unison. We all agreed We should we should share this in the in the Fedora Telegram channel because that's the place where where where we hang out and We are sure that that's the place in which this is going to be welcomed So that that was it like the premiere or something like that We did it on the Fedora Mexico Telegram channel. So here at DJ you guys want to sing You said now John Everything is in Spanish so Feel free to translate in your heads Let's have those pounds again. I was going to sing it But we couldn't afford to the band for the band to travel here and perform Yeah Well Where we are and where we are going in this moment. We have improved packaging skills in our community. We have some people do want To pass to to get as packages. So we are Perform a packaging workshop, but we found that the documentation needs Yeah, some some members of of we always we're always trying to to to we're always asking our members to try to package or to enter the experience of being a packageer on the Fedora ecosystem and But we found some some And I like to call them areas of opportunity not problems Are the the documentation for new is in this area is a little confused and it's it changed tends to change over time in English and Might be our fault. Oh, it's not we don't make it Pair to pair with Spanish. So sometimes Your browser shows you my browser shows me my the web page in Spanish and the web page in English is different More more. It's a newer web page. So might be that we have to coordinate and help with these translations but It's a it in in general, it's a little hard for new packages to learn the whole process like Creating your fast account and then going and download the tools and learning the workflow Because it's a weird workflow, but it's nice. I like it. I am a packageer but if you're used to Just sending the patch via email. You're gonna find like that is kind of bureaucratic, which is not but I'm saying a perspective of one of the guys that that is trying to become a packageer Actually helping for for LibreOffice We all let me let me let me just tell tell this as part of the communities. We are also Helping it's not this is not only Fedora Mexico, but members of Fedora Mexico are helping On the development of LibreOffice mostly for Mexican stuff, but there's now some expertise around writing the code for that huge project and We're also creating packages around the stuff we're we're creating because they It takes time for patches to to get accepted so we created our own RPMs and and Trying to we're finding stuff that we can contribute to Fedora and and that's when one of my friends had a little problem With the onboarding to be a packageer and just he was she was saying hey, man I'm just a five line patch Why do I have to do all this? Oh, you know why I told them you know why because you're gonna contribute more and If you know the workflow, it's gonna be easy like water for you. Well, at least for me it's easy I Confess that I have to open the manual, but Still it's it's easy So then we have implemented adoption of package program You know command training the package between members of the community looking for I impact package as mentioned by by Ivan and other Challenges we have is supporting the growth of federal atom for us is very very important collaboration across borders because Federal atom was all our inspiration to create the federal group. So now we want to Get back the favor Help now that we Our brothers need need us. So we are part of this great family of Latin America countries we need a support and to consolidation of Federal atom with technical support in federal Spanish channels and maybe sharing knowledge in remote face-to-face talks That's it that is We that we do it and that we want to do it so I Don't know if you have some questions or comments No questions. Oh In Spanish or English? So given the power, I don't know if you've played with chat GPT It's actually quite good in translating From one language to another and I'm wondering if more more Companies should use it leverage it for Translating documentation into Spanish. Yeah Chinese etc. It's actually quite good I was surprised how good it is in Spanish and Italian and Irish It does a really good job Great, thank you. Thanks Yeah, yeah, we have a small discussion one day in the telegram group because Someone someone asked to check it GPT how we can install some Package tool on Arch Linux and the process is correct But that package don't exist. So we don't have a lot of confidence with Yeah, yeah We know but We don't trust It is it needs validation, but yeah, it is it is a good idea Yeah, because most companies their Complaint is and the world revolves around English and there's this Assumption that you have to speak English to work in IT and to use products Yeah, which is really bad. Most of the planet does not speak English Yeah, and we should have documentation in all languages. Yeah in our talks we always talk with the young people Don't worry about the English. You know English your operating system is in English You all all that you are practice if you have a Linux machine, so And there's there's something but in in Mexico for example, we have We just don't speak Spanish. We speak a lot of languages a lot of native languages and And Everything in computers is either in English or in Spanish and there's there's a barrier for for native communities to to use technology because it's not being Shown in their language or at least their mother tongue They have to like in head in head translate or stuff like that the guys at Mozilla They are doing translations of the browser to now at and and that's that's pretty cool And and for a week when you said that I just remember for a while I have been trying to ask those guys to join the Fedora community and start At least checking out if how hard is to translate the operating system and documentation to to native languages Another question comment. Oh We have a If you wanna this you need to ask you just came for that. Yeah, you need to sing that the free software song. Yeah I'll I'll ask one so On the earlier side you talked around some of your future steps and trying to build a stronger Latam community Yeah, or across across borders. My question is From the Fedora project side, are there resources? Infrastructure tools that would help you do these things that Fedora could provide Just wondering how could we help you? Yeah, we do a lot of things We love the idea for example the Spanish channel in matrix. This was our first step maybe have a Bridge between the telegram channel because it's Very very very interesting the talk all day Technical technical talk all all day in this channel and maybe if you see in matrix you can All the work and see what we are doing so, yeah, I believe we believe that the this first step to provide us a Spanish channel was will be will be very very good for for start and then with Luis and Jose we can coordinate to do some remote talks or maybe they can Go to Mexico or some of all people go to Panama or another countries and I don't know They win we need to Restablish that friendship that all we have all this time We were discussing with Luis previously that we also are First of all, we need to identify the communities and the individuals that are active and contributing and are willing to reactivate What in the past? It was a huge community all over Latam. It is not the case in this these times, but Yeah, I think as we need to resolve all the issues, but we can do that Right. Yeah Step by step one by day. Yeah, there's there's something beer money, please of course and if we Could we have a food con in Cancun or something like that? Why not? Yeah, maybe celebrate these events in Mexico or you know someplace well in Mexico. Yeah Yeah, maybe the next block of I don't know who knows Yeah, we can pull it off. We are looking at that and another question or comment I'm gonna do double duty of asking a question and operating the camera So my question for you is Do you see any other open-source communities that are very active in Latin America that yeah? Yeah, we are we we see a lot of communities in Mexico Python communities C-serving communities Blender community Ross community they have a Lot of of of events and we don't go with them our first step is try to get this Communication and participating in dead in their events and everything, but yeah In Mexico are a lot of communities working and they they inspired us But there are also companies that are Increasingly looking this integration for with communities like the one that that organized the event that that Alex showed that Open source contribution Comet that that that is a that was organized this that one was organized by a but a company but looking to integrate and and to activate the communities This event is growing and taking it. It's a very very huge event. Maybe One hundred or five hundred people. Yeah Yeah, a lot of people in Mexico and for everything Google genome Python Everything you want is there, but Fedora know That's why we want to participate in all kinds of events in everything Yeah, I also have given talks to the goal-length community in front with the Ross community and I always do the Like I come come join the Fedora. Yeah. Yeah I know from my perspective I work for a university and doing open science and open scholarship work and Much of the world actually looks towards Latin American Universities and institutions as like leaders in open science because there's been so many really successful and like well-built initiatives Throughout Latin America. So I know it's like now, you know Fedora and open science There's some projects, but I thought it was a cool connection looking at the work Yeah, a lot of people in the universities use use some distribution in Linux When we go there and say, you know who creates a genome What's created in Mexico, of course Of course Yeah, obviously for example Yeah Very common and Miguel Miguel de Casa. Yeah Yeah, and also not only universities, but research centers They they try to use and I don't know why but Might be CERN uses CentOS I think and might they might be influencing other there's work If for example in Chihuahua a very prestigious research center and those guys use Fedora and and and I From time to time I bumped into into the those PhD guys and in the bar and they start giving me hard times of This package I will help Be like that But they they are enthusiasts and and but they're doing science hard science with with Fedora Yeah, even the the government have a lot of projects with open source and Linux. Yeah any other questions or Comments not really a question, but yeah Thanks for your presentation. It was amazing like you're doing some really cool stuff and I love the energy I think like you can see all the passion and energy you're putting into that and We'll talk after I would really be happy to have to have like maybe Fedora Coro's workshop or like Joshua Levins and have someone from the team come and talk about Do a workshop or stuff like that? Great for sure. Thank you. Thank you. Thank you anything Well, gracias amigos. All right Yeah, we have some some stickers from Mexico and We have two coffee mugs to coffee mugs with a nice logo to My didn't do it for us. So maybe the first person who My name is Peter boy Yes, you may see and just about myself and my relation to Fedora server. I I'm a scientist at the University of Bremen and Graduated in sociology far away from server, but as minor I had applied mathematics, which was a academic home for computer science those time Well and once the time be managed with our Institute to apply For a so-called accident center. It's a special funding in Germany the effect was you got a lot of money and You got a lot of independence from our central university IT But the obligation to do it of our And because I had that dual Qualification our colleagues things. Okay, it's job for you as a side job. Of course if you need everyone to do such Fortunately, I got some compensation in teaching but I had a lot of money and Nice side job It was we started with visit AI exit the IBM version But soon the interboxes got more powerful and we started with Linux and I had to Check which she looks At which which she looks distribution. Okay, it's There was somehow redhead Looked for me with all these advantages. I know I knew from IBM It's a good interface a good support everything like that. So I decided for redhead unfortunately Short after we did that ref redhead dropped the academic license program So we had despite a lot of money, but not so much Fortunately, we got scientific in looks and we use that in we'll use the combination of scientific in looks and fedora server the classical case of usage Scientific in looks for the stable version and in looks and fedora to get the latest software sometimes ago scientific in looks died as well and I Had the option sent to us or fedora and I decided for Fedora and put everything to Fedora server edition It was in that time so I had a small bunch of 20 about 20 servers boxes in Fedora server edition and some Virtual machines on top Well, it's my special Experience you had no any problem with Fedora. Everything works fine. Just one case We upgraded the release and the colleague of mine saw a strange message something as long But he was saying okay. It worked forever Don't worry be happy But afterwards you were not happy. This is something to do. So I Think we at Fedora server should build in some quirks. So Admins keep watching and not to be sure everything works fine. Well, that's my personal experience with Fedora and I started then end of 2020 when Matthew was started to reboot the server working group and And I had this stranger idea to spend our internal Fedora documentation to Fedora instead to build up a long website and started with documentation Well, and somehow I ended up as a coordinator for all the business we have No, I do okay So what is Is it Sorry Glasses I can't but probably first the power some Remarks about the current state of Fedora server edition and some information what's going on and At the end what we are planning to do for the next time and Well, the first thing is a working group we had a reboot in 2020 and Currently we have officially 70 members according to the working group relations for Fesco We have two weekly meetings and we have it constantly about the seven the last two years So I think the reboot if it's But when for when fine Well, we have nine members who in the soon had two and a half years have regularly regularly Contributed and discussed specifically contributed some work and Two members who some sometimes Discussing well and six members who constantly do nothing over the last one and a half year and Well, it's probably not no problem But anyhow we have just to see how to handle these cases I think and we have the big advantage that we are a mixture of old hands so to speak members who Were part of the working group from the beginning and a lot of newbies So as me who try to catch up and to carry it forward All in at the end I think the reboot was a success and we have now a quite active server Working group, but the other thing is the addition We update our technical our technical specifications And we are shifting from the previous base server role as a UDF base server role to Ansible Support or specifically supported services. There are services which should be Well, what is relevant so in currently is not but Maybe we have to introduce some of them as blocking or some kind of blocking Services Well, we discussed our goals We discussed the issue if you are all goals are to old now over Come to something between but we stayed with them. So it's not these are still Stick with the technology with the technology technology which supports various options not just one Which does not just one option to this specific specifically hyphen hyped in the discussion And we are stick with a server which is easily to can easily be can easily be adapted to different needs different Not very different needs different structures different contexts So we will stay to be a multi-purpose server And that means we will stay at the near future with package-based distributions installations, I think Nevertheless, of course We are looking for what's what's the development is and we are looking for new new New or nearly new use cases. So we may adopt server or some distribution media is to new UK new use cases Some things I have to look at and And I think it's not so bad with our goals If you look at the development past over the last year, we have a constantly grow of Update as the graph is the number of DNF updates per month. It may be a better Figure to indicate the usage of server as is near download Numbers so we have been a constant growth. I think we are on the right path. So to speak Yes, we have a growing demand and if you look at the Numbers which Matthew presented yesterday was it. Yes with the fedora server, we have still a good amount of users In relation to other traditions in relation to other parts of the fedora Distribution, so I think I meant at the moment over the time being we can do what we are doing Well, what's going on? A constant task is dog's improvement and If you looked at the con commit numbers, it was the last it's the last year. Yes Compared to other additions, we are not that bad We have What is it? We have the second most. No the third most. Okay The commitments and actual updates of our documentation and I think we are a good way to improve that further But of course if someone of you would like to write some documentation everyone is welcome And We have a lot of things to do we are not We have no lack of tasks to do we have a long list here as you see but the main task for the next future is to Bring up or to get up our unsupported support project because it's the main the main issue of our product The product demand and if yes, what is it's product requirement document? Yes And it is the main section in our technical specification, but unfortunately we didn't we didn't Get around it in the last year And we have to review our installation media. I think that's the last time the installation media Reviewed is probably eight years or so ago. There was there were some minor changes But about but after eight years, maybe it's a good idea to Do to review with the issue? There is no actual no problem. I think nobody's complaining about it, but nevertheless, there are some strange issues on it which may be Deleated and there are some new Ideas which you can or shouldn't do it into it. I know Server Admin says there's strange people including me when someone comes near to a server it is Don't touch don't do anything as long as it works And that's the way we didn't touch the content the installation media until now Well, but sometimes you have to touch but we are Dependent on our own hands to make this work. I say because I'm I Try hard. I had a lot of hard times to produce our new Virtual machine images and that beast called what is it in the factory? It was not so easy to get up and running and I think this is a installation media. It's much more complicated Well, then you have some upcoming projects The first is in Fedora server the virtualized environment Until now we were too much for cues answer Fedora on hard hardware and probably virtual machines on our own heart So Fedora based hardware But nowadays there are not nowadays since some years There are a lot of providers which offer not only cloud in cloud system but also offers virtual private servers or dedicated private servers how you will name it and At the moment, we don't have anything to offer for those But I got a lot of questions about how to install Fedora server on one of these Offering so The idea is to produce adjusted install images depending of the providers Own system and to create documentation how to do it Anyway, I made an initial initial post on the mailing list and I got a lot of response and questions and Demand to make it as fast as possible So I think there is a field where we can gain new new users. I hope so and The other thing is server on single board computers. It's a Well discussed item at the last years well We had a lot of inhibitions, but should we do with such a do-it-yourself thing in a In this Fedora server for production meant for production but we started to discuss the item and Find options to find ways how to use it in a meaningful way So we started to discuss criteria how to select one of the single board computers and There is some simple things as a solid case. So nothing It's me now fire extinguished Well, we tried to determine some selection criterion and we tried to determine some useful applications that and we find maybe applications with need of 24 7 operation and are less power consumptive, so it's maybe a way to offer these services in a economical useful way Like backup software backup server or server monitor or cluster monitor software or Cockpit gateway such things like that, which I don't need much computer power But need to have an uninterrupted operation. So it may be a useful. Maybe you have maybe a useful Use case for these kind of combination our valuable Fedora server is such a do-it-yourself device and the long term we are Looking to be a to get to a bit set up an economical energy efficient home server which offers a lot of services and Compensating what can use to be to compensate software limitations of commercial Nuts of network attached storage with their web interface which limits you a lot of When it limits you not to be used to be able to use a lot of Faith copy abilities which operating system of the network attached storage provides, but you can't access it Because of the web interface and you can't override it because the web interface overrides you overrides as soon as Start the box again. So that's So in short The plannings we are discussed or I have discussed in our server Double server working group. Yeah, that's it. Thank you Okay Just one question I I Used to run Fedora server on my email machine and what I have all my stuff and I had the problem that every six months. I had to to what not every but There's a there's a point in which the the distribution goes and end of life and It was a little it was hard to being Reinstalling in a remote environment and and eventually went went and Installed CentOS, but my my question is it do you guys have like a path of upgrading and avoiding this these types of problems Just as an example when you upgrade open LDAP The thing will not start if until you do the checks of the database and all that stuff. So Do you have a path for upgrade? Yes, it's a standard usually it's a standard upgrade path using the DNF system upgrade option and if you wait two or three weeks, I think we have a documentation how to set up an email server on Fedora server, so It may help you to set it up in a way. It doesn't interfere with update operations. I Can speak to that one a bit to Peter. I'm Adam Williamson the Fedora QA team lead I used to do the same thing running a mail server and I ran into a lot of the same issues probably you ran into So there tend to be just things when you upgrade a mail server where you know a server Thing has bumped to a new version and you need to change the config file So it won't run properly as something is getting denied by SE Linux that always happened There is a potential path there, which is that so for free IPA and Postgres we actually have test Those are blocking for server So we have release blocking tests for it and we actually have automation the tests that you can deploy a free IPA server And upgrade it to the next version and it will keep working as one So in theory if we make the new mail role release blocking if we decided that was an important enough requirement We could then also do the same for that, right? I only have the resources to devote to doing this work for things That are release blocking. I can't really do it for all the roles that aren't release blocking But that would probably be the best Way to improve that experience because yeah, I do know all the things you're talking about and it is painful Yeah Thanks You mentioned you did the answer I just must what did you have to do on the other side for answerable? I mean, what are they did it involve? I'm just a coordinator Well, we started with a with a model project so to speak to install Right right fly in a way that we can do it without an rpm using and Ansible to to to install it to install the system D infrastructure and okay and to maintain it and Well, the guy who was working on that project was a problem. He could he had to Lay it for some time, but he has two issues of wisely to To get while fly up and running with the best certificates. There's a current wild fly option version Which causes some troubles and here Problems to select from the many answer will skip to support while fly to support the one He needs so Yeah, we are If you could give us some advice some support, it's very helpful, I think But I'm not the person who do the work. It's a moment. That's the case we have and If you look at the fedora issue Get in park park you are Pagu however Is pronounced I think it's up to that We have an issue for that it is 64 I think you are 60 and I had it here it is Yes Yes, it's issue number 60. So just in case I'll admit to not having kept up with the the discussions around the Fedora server working group But can you help me understand the difference between the work that's being done by the door IOT folks? And the single-board arm computer work that the server group wishes to do I could be a little louder. I couldn't understand. Oh, sorry. I heard myself through the speakers Can you help me understand the difference between the the server working groups single-board arm computer thoughts and the work done by something like Fedora IOT Well, we are cooperating with the arm sick of course and Maybe our use case is different from IOP IOT. Sorry We want to assist or replace a network assess Network attached storage Whereas IOT Ames to a different Goal, I can't describe it exactly. I must say the only thing I Decided for me it's different and the same is true with the with our now with our Virtual environment project. We are not aiming at the cloud and not aiming at cloud services, but aiming at virtual virtual service which I run independently from any Administration service on well on on your hardware instead of mine. So I think it's different It's a different use case Okay, and if someone has some time to spend please join our working group We are friendly Been busy and good work people so And we need we need soft we need we need sense if you look at our project Thanks Good afternoon, how's everyone doing on day two of flock Whoo So we're going to go ahead and get started up here on the main stage. We've got the I should have had my my notes up here from the full name of the panel But we have the upstream collaboration and cooperation in the enterprise Linux ecosystem We have our four panelists and our moderator up here who will Matthew Miller who will do some introductions in a moment That's our our facilitator I just wanted to acknowledge first, you know generally with flock We really try to prioritize the diversity equate inclusion side of things and I realized that we try to avoid the Manals but due to a number of circumstances this year with travel and the logistics not the ideal but you know something I just wanted to acknowledge and Point out that it's something that we were aware of and I hope that we can continue to Bring more diversity to some of our panels that we have at flock But I'm really excited for the session and for what we have planned And I'm going to go ahead and hand it over to our Fedora project leader to kick us off. Thanks, man Hi, everybody. So Everybody in the room knows me. I'm Matthew. I am the Fedora project leader for the benefit of anybody who watching this for drama online I think we're planning to not have drama So you're going to be disappointed. Sorry But I hope people are walking out of the room now. Yeah I Thought I'd start a little bit by just talking about myself and why I'm here how I got to this because it's actually a little bit relevant I've been the project leader for 10 years But I've been involved in Fedora for a long before that and I got involved there by you know working in Building something called BU Linux, which was a downstream distribution of first Red Hat Linux and then of Fedora and of CentOS for some of our server systems. So I've been a rebuilder. So I kind of have say, you know I know I know what goes into this and I also really care a lot about Fedora and This seemed like an important conversation at the time. We're at right now This conference is a community conference. It's not a big corporate event so this is in some ways a little bit of a different thing because People here on stage are kind of speaking for their projects and normally we are all individuals so I wanted to start with Asking everybody to introduce themselves as who they are who they gotten how they got involved in open source separate from the project that they're working on and then a second round of you know kind of what hat you're wearing on stage and you what what What project you know how you're involved in the project that you're involved with so Go ahead and start. Oh wait. I also want to say thank you Neil for proposing this as a session here Yeah, absolutely, and I'm super happy that we're able to have these conversations and this while a little bit off you know Kilter I guess for for flock as a usual subject. I think it's an important conversation to have you So anyways, I'm Neil Hanlon. I got into open source Actually with with Greg with the form and project many moons ago writing a DHCP plugin for the company I was working for and trying to open source that and I think that's part of the process of Trying to open source things in the company. I was working for at the time was new to them and it was something that I was trying to pave the way for and Eventually that led to my involvement with Rocky and eventual You know being hired by CIQ to work on Rocky full-time and so that's the kind of journey I've taken in the the open source world and it's been you know more of a dream more or less to be able to do this in a full-time capacity to enable the community and also With myself be able to contribute back to the communities that I've benefited from for so long Hi, I'm Brian Exelbeard thinking about open source I first encountered open source in a role that I had as a developer and The idea that I could get access to everything in that operating system and extend it was was new to me I wound up writing some drivers for Pam that's what it was Pam. We were Long story short. I extended Pam. Y'all don't want to know the sword at the time my management was Opposed to us releasing this code because they thought that somehow this would mystically cause all of the Competition to be better, which was not true because this was a terrible idea But that being said after that I wondered for a while in the desert of academia and wound up at Red Hat and While at Red Hat, I got to work on documentation in the Fedora project I got to work in contributions around calendaring and some other Stuff that Red Hat was doing in the Centos project and then became the f-cake as I mentioned yesterday And after that wound up in the the rail business unit, which is why I'm here on behalf of rel And I'm like Neil. I'm grateful that you proposed this I'm actually really excited by this and even though it's a little quirky. I I hope this sets the basis for something We can do more regularly. I've been excited about the co-location of Centos connect here, and I think this is pretty awesome Hi, I'm David a cabalca. I started using Linux in like 98 or something I I think my first like significant for an open source was I was I started using it and contributing it and eventually Helping maintain a small embedded Linux distribution for media center systems like back then where you'd like have a like a six megabyte I saw you burn alongside your movie to watch movies a cult Geeks boss. That doesn't really exist anymore And then I was fortunate enough I kind of was able to spend most of my career working on open source and around the open source base So I I used to do small business consulting back when I lived in Italy and like setting unveil servers Those kind of things and then these days I work in meta on a little sister space team where one of the things I get to do is Engage with upstream projects and upstream communities to try and like take the work we do internally and met and bring it Alongside the world and there And that's also what eventually brought me to working closer with Fedora was sent us with all these communities Hey, my name is Jonathan, right? I got into Linux Early 2000s my dad had a box set of red hat 9 and I believe it was Sousa 7 and I wanted to host websites and team speak servers for some video games I played so that's how I got my start in it and Then you know from there it just continued on I got into CentOS probably 2005 2006 ish I've been in web hosting for about 15 years now and Being in web hosting It made it a pretty easy A pretty natural fit to get involved with Alma Linux as it started up because you know web hosting is very heavily Or has been traditionally very heavily reliant on CentOS So I got involved with Alma Linux and that was my first opportunity to kind of contribute back to open source From there I met Carl George at open source summit a couple years ago and he encouraged me to become a packager and later sponsored me So I got into that working on Apple and then naturally ended up on the Fedora side and Yeah Cool. Yeah, so now let's do the what hat are you wearing here today thing a little bit So what yeah, talk about your role in your project and how you're representing the project here today Just very briefly. I'm not wearing a hat. No hats I will be representing Rocky Linux today. I'm the infrastructure team lead. I head up all of our Operations basically for building and running and releasing Rocky on wherever it is I'm representing well today I just didn't get the memo about shirts and I only packed one red hat shirt for this trip And I wore it yesterday, and it's not clean. So I spared you all by wearing a better shirt And I am on the stage today because I am a technical business strategist in the red hat in a proselonics business unit And my area focus includes our upstream communities. I Represent the Centos project. I serve on the board of directors for Centos as of a year and change With Amy who's somewhere there and Sean who's the community architect and other folks, of course I also co-founded the Centos Hyperscale SIG We didn't send those and generally contribute to the project where I can I'm representing Alma Linux I'm like Neil the infrastructure lead. So mirrors internet presence stuff like that Pretty pretty good fit for web hosting Cool so I don't my days are it was yesterday I am backs gave a speech gave a talk. It's not a speech. I'm out of weird a talk It's what it what does red hat want which is a traditional talk We've had it flock for a while and so people who might not be familiar with fedora and flock might think oh, yeah Right had to big sponsor of course to get a sponsored talk. That's actually not where this talk comes from This is something that so fedora has been sponsored by red hat since the beginning and sometimes communications between people in the community outside of red hat and red hat the company and people actually Working on fedora inside red hat and red hats like business once and whatever have been really confusing and unclear So this is actually an invited talk the fedora community Together asked red hat. Hey, could you just kind of tell us what you want instead of doing weird hints? there's a thing inside red hat where the business unit complained about how they always get pizza that they did in order and From a fedora perspective. It's kind of like so there's these weird people in suits like outside our pizza shop And they look kind of hungry and they're sniffing around and like I guess so yeah We sometimes feel we feel sorry for them and give them some pizza and then they complain about it So instead of that I thought why don't you come tell us what you want from from the community? So that's kind of the origin of that talk so Having had that though that seemed like it'd be In good spirit to ask all the other projects, you know, what does your project want? So this is very much a with a hat of your project on like what are you trying to do? What are you looking for and? Kind of it. I want to kind of maybe do two rounds because first one is kind of like a general like here So the project is all about and then I want to talk more about fedora and specific and kind of a second round of things I Think that for that it's sort of Two pieces and one of side of it is what the rocky Linux community and our users are Wanting and or expecting of us and also what we want as a project of you know The rocky enterprise software foundation which has the rocky Linux project underneath it which has a board and We ultimately are set up to act with the best interest of our community So what we've heard from our community is that they are still wanting What sent to us was and what rocky Linux and all the Linux are you know sent to us? Downstream rebuild basically of red hat that can be used for free what we want as a team a group a Board is to collaborate with all of our upstreams sent to a stream Fedora, etc. And the broader enterprise Linux community which involves more than just the the red hat derivatives But Susa and and everything else that comes in that van. I mean Troy Dawson and the the apple State of Apple yesterday I was talking about all these weird Linux distributions that end up showing up in mirror stats and such That may or may not be real things or or may or may not be real at all So I think from that perspective we very much want to collaborate and figure out What the future looks like and not try to define it by ourselves because that's not what our project is about it's founded in a way right to Not be beholden to any corporation and we're still as an organization figuring out what that means and We obviously can't do that in a bubble either So we want to collaborate for not only the good of our project But for the good of all of the projects that have been spawned in the past few years or even before that I want to also point out that Troy Dawson not only works on Apple, but also made this amazing shirt that I'm wearing here And you cannot get the Fedora logo one on Etsy, but you can get the a bunch of other ones there So it's good plug plug his store. That's great Bex you want to do a abbreviated version or yeah We want to double pepperoni pizza and we want you to remember we haven't worn a suit in basically eight years No, I won't repeat my whole talk from yesterday Well the way I would summarize it is our ultimate goal is vibrant communities that are advancing open source that are a place where we can meet Everyone and contribute and collaborate together Our upstream first philosophy is one where everybody benefits if we all go as far upstream as we can with these Collaborations and and I think that you know this stage represents an opportunity for us to continue that collaboration I think for centos the main thing that We want that we think would be important to develop is a community where people can feel comfortable Contributing and advancing the state of the project and the ecosystem as they see fit Within within the boundaries that are available. I think for center specifically There are various entry points that one can use to that end like their centers trying proper But there's also the six and I think the six are a great example of a space where if people have a specific niche That they would like to focus on their specific thing that isn't currently well-adressed by the project by the ecosystem where they can work on develop that and then use that as kind of a launching pad to see if their idea is viable see how that will can work out the develop it deployed Get see if there's interest and see their customers customers in the broader sense not necessarily paying customers But people that have any interest and want to contribute But not just be users but also be contributors and be able to be able to work on it together I think that by far that's that's the main thing we would like we would like to see in this environment So the the mission of Alma Linux is the same now as it was when we originally started that The users that came to Alma Linux want a downstream rebuild of red hat. They want Rel like features, but what's been changing over the past month is how is the value to the entire ecosystem defined? And I'm sure we'll get into it more later in the talk But we're we're shifting a little bit with the community as as we announced a few weeks ago and I look forward to you know at first it's easy to get hung up in Negativity and only looking at the downsides of things But as we've had time to think about it more We're very excited for the future of how we can collaborate and contribute upstream In different ways that add diversified value to the whole ecosystem Cool, thank you everyone Yeah, so part of the fedora mission, which I cannot say word-for-word offhand, which is terrible but is To make a platform that others can build on to do cool things for their own users so We want to make nice things within the fedora project solutions for our own users and for ourselves things we care about We also want to make something that you know can be built on that's one of the things with the asahi linux Announcement yesterday. We want to be a great place where people can come and do really interesting projects like that and Downstreams both rel and everybody else here like this is part of what fedora. We're here for that really so I guess this is kind of a two-part question of you know What can fedora do for you and you know conversely what can you do for fedora in addition to your generous donations to this conference? Thank you So for what fedora can do for us You've already done a lot of it we've built rocky and used a lot of the open source code and infrastructure as code and Standard operating procedures and a lot of the really great stuff from fedora. There's much great stuff in fedora So we've used a lot of that and taken it and modified it slightly for our purposes, but not in a way that I don't want to say requires but Facilitates upstream collaboration for that But we want to continue building off of that with the tooling and everything that we build upon because there is such a great history and Tight integration with all of the fedora tooling and Even more than that a community of developers that already know all of the fedora tooling and specifically for packaging and such but we also as A organization for our special interest groups where we are trying to add our own diversify Whatever on top of rocky we always encourage and have from the beginning collaboration with upstream Special interest groups and parties are our wiki has said and I mean I think on our main page there for how you can get started with Rocky It's it's pretty much from Writing that page. We're like if you want to collaborate with the overall enterprise Linux ecosystem You can also just go and collaborate with the open the enterprise Linux ecosystem in CentOS stream and Apple and in the existing places there so For as for what we can do for fedora and what we plan to and are doing for store is you know I'm personally have been involved with Apple and fedora for the past few months now similar kind of story from Jonathan getting some mentorship from people like Carl and and some some other folks and I'm aiming and working to Bring more of the rocky Linux community into that ecosystem and help collaborate and build those relationships with the packages and all The different things that we're trying to do I don't mean to repeat you Neil, but it's great But yeah fedora is actually doing a lot of what we need you you have Sad at that intersection as I was talking about yesterday between integration and innovation and I am grateful for the way that you push Things forward in that space Candidly I this is partly me But I really like it when fedora pushes back on red hat a bit I love the fact that butterfs is in parts of the fedora editions. This is good We want to see that kind of stuff proven out We want to see that demand shown to us and sometimes we just literally need to see it in front of our faces And I want to continue to encourage that that's where system D came from Yeah, everybody is like red hats forcing system D on us conspiracy theory. It was fedora forced it on red hat Which is great like I that is that's how this is supposed to work And you've been a great home for ELN where we do the next major release of rel's development And you know, I'd love to see more like that kind of collaboration Obviously we provide a lot of sponsorship to the project Lots of this event candidly and and other things and I don't see any of that changing and going away in a direct sense We were certainly going to shift around how we do it at times based on what's going on in the universe But yeah, you're like you're doing it keep on doing amazing things and and maybe one last echo point continue to fill that white space Like maybe that's that the place that I should mention this. I had the privilege of sitting in on the fedora server Conversation with Peter boy earlier and he mentioned the server groups desire to work on single-board arm nas Devices that's not a red hat objective, but that's really cool, and I want to see that come to fruition I'm also happy that butterfaces in Fedora But kidding aside the Fedora is the ultimate downstream for everybody on the stage and the thing that I think is important to To stress is that when people are looking at where to contribute it can be difficult sometime to figure out what at what level do you want to contribute an able stage and what you're interested in and Contributing at the level of Fedora versus at the level of center stream versus at the level Of well contributed to rail meaning like filing box to rail unless you're a customer or contribute to rockier Alma They're all different layers of kind of the same cake here and like That's something that I have personally been struggling with like communicating to people and making sure people understand where where they want to sit here because And that is something where I think we could all do probably a better job in making it clear how this works because generally speaking if you if What you are really interested in shaping the future of the ecosystem and distribution? I would say the work you want to do is in Fedora Fedora is where the leading edge is is where this work happens Is where all the new technologies get integrated first that is by far where you want to be If what you're interested is in working on things that you want to see landing the next minor release of rail are in Stream you probably want to work in stream if what you want to see is Working at the level of the rebits are that all our projects. That's probably where you want to work And I think figuring out Where sit there is something that is important But yeah Fedora has been absolutely fantastic And I think when you look at things things like a pill provide tremendous value for the entire ecosystem like in You're trying to say something this panel is great. I would like to have another round of people saying nice things about Fedora after this one That's that's what I mean I'm serious like is Personally think generally speaking where you're speaking we're setting up a system with with any of our distributions on the stage There are very few instances where it makes sense to use it without a pill enabled And that is something that provides tremendous value both both in the immediate and in the potential in that it makes it possible To get any package that is in Fedora now Potentially available for the wider enterprise ecosystem if they need arises So that's a great example an ELN that you mentioned is another great example of Something that in the past used to be nebulos and invisible which was how does the new rail major release? appear from the ether Then now now is a thing where you can actually sit on a meeting on IRC every Friday and discuss how this is working and look at it and Get your packages in a workload so you can get notified when they break and actually fix them ahead of time Like all of this is fantastic I Think you took all my talking points No, I was I Was gonna lead with Apple. I mean, it's Especially over the years as Red Hat has gotten more behind it. Officially. It's a huge resource to anybody running anything in the EL ecosystem, and I mean that's largely in part. Thanks to Fedora contributors who otherwise Don't have a direct hand in the development of EL and prior to CentOS stream you know changing and becoming a little bit more open to Contributions getting into the EL product sooner. Apple was the best way to for the community to contribute and in turn one of I feel like the most valuable ways that Rebuilds can contribute back is to contribute to Apple. I So full disclosure Mike McGrath is my boss's boss so but he's also a longtime Fedora contributor and actually is The primary person or at least one of the main people who started the Apple project back in a long time ago So we have I know Mike spent the villain of the open source Internet for a little while So let's let's let's give him some credit as well for like really starting that project And actually this was actually another thing that Red Hat was like, okay, I don't I don't think you should do that And Fedora did it anyways and look how it's been a lot of benefit to everybody here obviously for people who don't know Rel and the downstream distributions are generally much smaller than the package set in Fedora Linux Which is in the tens of thousands of some amount But it turns out that so you know when you want to commercially support a bunch of software That amount of that's it's pretty untenable to do a really good job across all of that So it's a more restricted set, but all of that software is actually useful That's why someone put it into Fedora in the first place So Apple is rebuilds of that Fedora software against Enterprise Linux base so it can be used in Much wider audience in fact if you some of my previous charts and graphs talks I haven't done it this time, but the number of systems running Apple is Order of magnitude higher than those running Fedora Linux But you know it isn't all isn't all a numbers game, but it's also one of the things where People working on things in Fedora have a really big impact beyond just the things they care about in Fedora Linux itself Which is sorry, I'm speechifying a little bit, but I think it's And it fits into all this and since everybody mentioned it seems fair So those are all the questions that I had might I guess do people in the audience here have questions And is somebody else a mic runner or am I the mic runner here? awesome There's one back there actually cool Yeah, that's fair. Let's even have questions specifically for Jonathan Was there a question No questions at all everything is settled mic drop They don't have to just be for Jonathan Hi, I'm Shaza. I'm an intern at red the hut and I work in the stormy project I'm new to the open source contribution and projects and all of this So I just have a quick question about if I would like to contribute to any of these projects after my internship As a student, I'm a computer science student by the way Would there be any level of difficulty in between these I'd say four projects or five projects Or you know all the projects that we listen to throughout these two days in your opinion So like you know more like I'd say beginner Stage that I can start from is there any of these project that is you know easier than the other or are they all in the same level? I'm gonna let someone else answer that and hope they all save the door I was gonna say for Dora Docs, so I Think I'll go first I guess but All the projects have varying degrees of Where you can enter into them based on your commitment level your experience level and I think The most powerful thing about all of the community is being you know facilitated or part of or Adjacent to them watching them go on like they're all Very very inclusive and welcoming to to new people Whatever that level is and none of them push you away or anything like that. So I would say something like low There are parts of the projects that don't get a lot of attention like documentation That can be a tremendous leave valuable to Participate and and there's there's tons of other ones besides the documentation. I just can't think of them right now. I Was just gonna very front say I think your best bet is to consider starting in either the fedora project or the centos project primarily because rel is developed in centos stream and then major releases in fedora and directly contributing code to the rel product is Basically impossible without the internship that you already have right now But on the other hand these projects are the entry points for that and and I like the way that you expanded the envelope of Contribution there like contribution is a lot of things and so I think it it really comes back to what are you looking to contribute to? Is as a computer science student I could see you wanting to take a specific subsystem apart because it has some area of interest that you have and that may actually be better in centos stream because it's a More slowly moving code base where you could look at very specific kinds of tweaks and changes and then get advice from The existing contributors and engineers in that project of where upstream that really needs to be socialized because it may not be directly Appropriate, but it's actually a great entry point for direction on the other hand You may also be interested in some new language stack that's being used and and bringing that into the fedora project is going to Also bring it into the Apple project, which is going to get it wide range of of impact and visibility And of course like to give you to give you a small assist there There's also like art and and other design and and I will tell you right now I think all of the projects would really love some help with promotion and those kinds of things But those may or may not be your interest and you shouldn't feel pushed to them Yeah, thank you so much I was gonna add to that just a little bit From a technical standpoint if you want to get involved the I mean you got to think fedora has been doing this with community involvement for What over 20 years now? the documentation is You know almost unmatched so to get in from a packaging standpoint for example Fedora is absolutely the place to start Recently I sent some of my first merge request into Senua stream and while the documentation there is lacking a bit I was able to do that with a little bit of outside help from Some friends, but it the process is so similar to fedora because that's where it all starts So if you want to contribute packages as a you know be a packager or something like that Absolutely start in fedora because everything is really structured around how fedora has been doing it for so long anyway My personal recommendation is to focus on something you find personally interesting that you that you would like to see to see in the distribution And then yeah, I think I agree that Fedora probably has the widest breadth of well-developed opportunities for newcomers Fedora also has things like the joint SIG that is explicitly designed to help newcomers get started on the project find I Believe you still have a thing that I find get there with a mentor or like something like that That that's definitely helpful for people new to the project new to the space new to the environment a lot of these a Lot of distributions like that from distributions here But distributions in general in addition to the code itself There's a lot of process around things that there's a lot of ways that people do things within the space Where people have strong opinions on the way to do things and I find I think it's always fantastic when there's new cameras that point out No, this is stupid. There are much better ways to do this. We should think about it And as I'm very much radish when that happens No, this is stupid isn't necessarily the best way to introduce yourself, but yes It's a fairly it's a fairly common one And some of the people who started out that way, you know actually come around and become Valuable parts of the community and then later have somebody tell them that what they're doing is stupid in a nice virtuous cycle But yeah, like finding something that is frustrating and you think it ought to be better is actually a really good place to start because you Know what your pain is and getting rid of it. Is it? You know Satisfying I guess David Okay, so that's a great way to start with I Want to I want to accept Fedora and rel from this. I'm familiar with those products. I Have a personal server that used to run CentOS. I Like to run an EL distribution on it, but I don't want to deal with rel on this personal server because it doesn't matter Why would I choose Alma CentOS stream or Rocky or? How do you differ from each other and which one would make the best choice for that? You're welcome. There's there's the fighting. Let's go Well CentOS streams always broke It depends I Hate it, but that's it's the answer in everything. It's when I was doing network engineering. It's the answer to there It's the answer in computer science. It's ultimately, you know at this point CentOS stream is Just as fine as anything else to deploy it it's It really ends up just being a matter of preference I think right and if you if you exclude rel like let's say rel for some reason somehow didn't exist and Source RPM and it's just a period of the ether I Think it's really just a toss-up of like what name you like or what community you like and I think the The silver lining of this entire situation that's going over the past month or so is that The enterprise Linux community as a whole has been brought together like forcibly smushed maybe but it's a silver lining that I am personally really grateful for because it's enabled us to have conversations between projects like Rocky and Alma and Sousa and CentOS stream and and all of these different projects that are part of the enterprise Linux space the the EL community IRC channel has grown massively and and there's Conversations good and bad that happened there, but ultimately I think it's a you know Net benefit for the overall community and that's why we're all sitting on the stage right now I would say from the point of view of stream so food disclosure I ran straight met home on the router in my home, which is just a Linux box and it's fine I would say running stream is roughly comparable to if you're used to running CentOS Linux is running CentOS Linux with the proposed updates or candidate updates repo enabled with the difference that on stream all the all the updates They go out to go through the the gating and the testing they go through so assuming the gating at the testing on the right side is set Appropriately there. They should be higher quality when they go out I Would say stream gives you slightly faster updates because you don't have to wait For things to go to point releases and it means you don't have to deal with point releases Which is something I personally like a lot because I don't care about point releases on the other end if you were running software that is specifically tested against Well 9.1 or whatever and you want to be sure you're running Well, if you're running software that needs to be certified you should probably be running rail But if you're running software that for whatever reason you don't care about running rail And you still want to be pinned to a specific point release then I would recommend not running stream But running one of the rebuilds or running rail The other thing I would say is with stream You probably have a slightly faster Cycle if you want to get changes that you eventually want to get into rail as well Because if you contribute to stream directly that's that's where the development for a lot happens So that's where the changes will eventually go into for as well and again assuming the process works the way supposed to It is reasonably reasonably quick like the single stream is that every package has different maintainers like in federal So that the response you would get depends on what the maintainer does what the priorities are and all of that So whenever I do this quite a lot because for my day job I I end up doing quite a bit of like by reposting contributions to stream. So you'll have You'll have cases where you point out Hey, this package should really be your based and it gets re-based Within a week and you have cases where you need to have a conversation and a bit more a bit more of justification Why you care about that? Just a little bit further. You said you want to run an enterprise distribution Enterprise Linux distribution my question back to you would be how do you define what enterprise Linux is? Yeah, I realized I was thinking about that while you were talking so to me for a personal server I want to run an EL distribution because I I want the slow-moving Pace of the system. I don't want it to change out from under me. I want it to run the web server Or whatever I'm running on it. I don't want to have to change it at the rate. We change things in fedora So basically I want to avoid home IT day like twice a month Seriously, and and that that's my definition of it for a personal server Not for my laptop I Want to sneak in another possibility, which is you can run for our coro s on that and Not worry about it and have your workloads in a container running one of these running or running UBI Which is the rel based door magically turns into rel if you buy a license. I think that's the way it works I don't know. Yeah, no, I know that's an option. I'm but I'm trying to pick between these three You have four choices sorry so far we've had rocky Linux recommends into a stream and we've had CentOS stream recommends into a stream So what is all my record? Oh? Okay You haven't said anything that rules out anybody up here Okay, so the things that start looking at then is going to be support lifespan. Do you need? Yeah, I mean, I don't guess you need support, you know, you're doing it at home You're messing with it yourself. I'm on the supply side of bugs. So, you know, which community like the best which community do you like the best? David said he is on the supply side of bugs. I just wanted to yes Pick a community get involved go with it. Okay. Thank you It is also fairly trivial to build your own Coro s like spin with whatever distribution you want or mix them all together Yeah, and if you package for Apple by chance You can use any of the distributions to to build your your Apple packages against using the Fed package utility and They should all work. I will say to add on to David's point there is movement in CentOS stream and Stuff happens like there is breakages that occur They don't happen all the time But it's also a very different situation than it was three years ago when if something was broken when it came out in Well, you couldn't do anything about it except open a bug now if there's something broken there's recourse and there's gating and there's policy so that you as a consumer of that package can also go in and Contribute directly to help make those tests better in some way to prevent that from happening in the future Which I know was a frustration of me as a consumer of CentOS in the past just because of inheriting bugs downstream, right? and the ability to Look into that and kind of help contribute is New and fascinating Yeah, I would say depending on what your level of The kind of SLA you expect to have for your for your home server I would probably not put a cron job that runs DNF update every six hours On any of these systems, but on streaming particular I probably wouldn't do that just because sometimes updates would go out Not necessarily in exactly the right order so you might notice some breakage if you run it at exactly the wrong time that that said So the way I mitigate this at work for what is worse is that we just take snapshots of the repos every Every week or every so often and then we're all out this snapshot of the repo as a unit So that's one way to mitigate that I don't think this is a concern for a home deployment to be honest And I've I don't think I've ever hit this at home and at home I just run the NF update at random whenever I remember it was a station to the machine So it's definitely not an ideal well-managed scenario. I Definitely agree. You need to be running DNF a lot on a lot of different systems to be able to see the signal of Possible breakages that occur because that they don't happen that often you see them in CI more than you see them anywhere else I think or maybe in your deployment of systems. You have enough to see who if things break Yeah, the other thing is stream doesn't have body So it's not like in Fedora where you can look at you can look at body And you can a group updates in bodies It's a it's designed in a very different way from that point of view and it has benefited in a Starbucks in that regard So if your expectation is something like Fedora, it definitely works a bit differently there Yet something I was only gonna pitch Fedora IoT at you because I use it personally And then you took it off the table and then I was gonna admit that I lied a little yesterday because it turns out I run another OS called has OS because I run a home assistant server and I was gonna tell you that you really need to think about your use case because Like while I love all these folks up here. These may not be the OS as you need And then I was reminded of the fact that you're on the supply side of bugs and like David You don't write you don't call you won't run well yet. You work for us. Where do these bugs come from, sir? There's another thing that I want to bring up because I think it's not obvious But it's it's a defining difference in that if you if you have Any of these systems generally speaking updates between major releases can be tricky or at least aren't necessarily Officially well supported That doesn't mean they don't work and you can generally make it work in a home case This may not be a problem On the other end if you use Fedora generally speaking if you use grants like Fedora server You can do system upgrade from release to another easily the downside a lot of the flip side Shall we say is that Fedora releases a lot more often? You may not necessarily like forklift upgrade your whole system every every time Fedora releases Also in the in the shorter window Fedora Releases way more often seems like there's kernel updates a lot more often You may not want to reboot your server to get a kernel update whenever there's a kernel update there So that's another consideration to keep in mind One more thing I think is a little bit unique to Alma up here is we a couple weeks ago We've changed we haven't changed directions, but we've changed our contribution path To where sometimes we can do builds of our own do bug fixes of our own and release those that aren't necessarily a direct clone of what rel is so with rel you're getting rel with Rocking I mean correct me if I'm wrong no with with Rocky you're getting a one-to-one Sorry, if that's the wrong terminology you're getting a clone of rel and with Suno a stream you're getting what's coming up next I'm really excited to see with the changes that we are making and Alma where how we might can differentiate in the future what extra we might could bring to the table and You know that might be a value to you you know depending on on the direction that basically our community takes it and Where people choose to contribute and where that goes And just quick. Yeah, we've got a queue of questions here, too Yeah, so I'm a bit of an enterprise Linux noob, so apologies for any ignorance in this question But and it's a two-part question. So the first part is for everyone up on stage You know, what do you would you say is your major differentiating factor that you're providing with with with your distribution? my second question is You know, we have many sort of very similar distributions up on stage Do we ever feel like we're kind of over differentiating from each other by creating these separate distributions? I'm curious about your thoughts in a Kind and respectful way. Thank you Ah shucks So I think that as Jonathan was saying my our rocky is differentiator right now is that We are Continuing to do what we promise to do for our community in terms of rebuilding one-to-one with red hat I will add on that we recently proposed a fast track SIG kind of a callback to the the Cento S fast track SIG We're we're looking to be able to Bring in more contributors and also collaborate more with the enterprise Linux community like with all the Linux and the patches that they're Planning on adding to be able to bring those as well as other bug fixes and and packages That our community finds helpful into the operating system as an optional thing to bring on but at its core We'll always stay that kind of one-to-one For for the promises that we've made to our community in terms of the the over diversification, I guess Yes, and I think that's why we're sitting at this panel right now to Begin, you know figuring out what that all means for the future With rel we have built very carefully an ecosystem of vendors and partners Where we try to provide an OS platform for them to help their our shared customers Execute workloads in a safe and reliable way with certifications and all of the things that those kinds of rel spaced holes exist to fill And so that's like that's a significant differentiator for us from say for Dora Linux or Centos stream I mean, you know Alma is going with a Mixed approach as I understand it and and our friends at Rocky are going to attempt to copy us And that's that's fair. I think that's that's a reasonable statement. It is not menace a slight The thing that I think is is interesting in your over differentiation question is this and I think that everyone in this room needs to acknowledge it at its core all of our operating systems including folks who aren't on stage our friends at canonical our friends at Oracle our friends at Sousa They're all the same thing. They're all made from the same upstream components You can sit down and do basically all of the same stuff on all of them. Sure. Some of the commands are different DNF versus apt I know which one I like to type But you know sure whatever some of the files are in different locations But they are at the core that and I really appreciate you bringing out this differentiation point Because it also calls back to David's question around that of you have a specific need and one of us needs to fill that and So like I said yesterday, I'm really excited about the white space in enterprise Linux being filled by other projects and other people Because not every problem is a well-shaped problem. I would say on the on the center side There's a few There's a few size to these on one end is what I what I mentioned earlier that because training is continuously continuously deliver distribution You effectively only have a continuous stream of updates You don't have to pick minor releases or think about that at all and that that has its benefit and has it all back So it's something to to consider The other thing is the sentence isn't just rain, but it is the ecosystem of the six So if there is something that you want to do that fits into what one of the six is trying to do That that can be very that can be very useful that can be very beneficial for what you're trying to do And conversely, there is something you would like to do in the project that isn't there yet Starting to say and running support around it could be a valuable way to go ahead On the differentiation point, I would say I don't think differentiations per se is a problem I think where it can become a problem is if we end up with silos that don't communicate to each other And I think the fact that we're all sitting here is a good sign that that isn't happening right now But what I would personally like to see is a way for all of us here to contribute Back to the same part, which ultimately is Fedora at the end of the day But in a way that we can all benefit and that if there are specific issues that One of our projects is solving and addressing that isn't well addressed by the others We can figure out what is the root cause there together and get it addressed where it makes sense and where it makes sense Might be in Fedora. It might be in Santos. It might be a process around How these distributions are composed together. It might be something that has to be rethought on the right side But I think that is something where the most value can come out of the differentiation Around the players on the stage and outside of the stage. I wouldn't say that What was the term you what was the term you use for you know, all of us? Yeah, I wouldn't say that over differentiation Is bad as long as everybody at the table is bringing their own value That's kind of the beauty of open source when you have projects that don't agree on every tiny little thing you end up with two projects And going back to what David said those different projects meet different needs So it brings more to the table overall because in the end that all goes back upstream and if somebody You know wants a combination of both of those they can go take that and you know run with it as well So it's it adds more more building blocks for everybody to build with Great, so I just want to let you know people have been raising their hands. So I've got Justin here I've got the gentleman beside David there, and I've got you next. All right Maybe we could skip this because I feel like it was kind of covered and it came I was came up when you're talking about the contributor pathway with Alma So this might just be for Jonathan But I was just curious what you think about not having to be one-to-one bug for bug compatible with rail anymore I guess you did kind of just answer that but I don't think else you want to add to that I was just curious like what that looks like for Alma and There if there's anything else you wanted to add to what you yeah, yeah, I'm be it was that a online question Or is that from you? Okay? I'm beyond excited because it Allows us to do things that we previously couldn't even entertain You know things of everything from butter FS support in a kernel to you know Putting raid card modules back in the kernel and everything in between are things that have been mentioned and prior to now it was things that we would simply have to dismiss and say well, you know rel's not doing that and We couldn't you know that that was what we were doing We were following rel now that's things that we can entertain and talk to the community and say hey You know is this are these things that you want? Do you want to help contribute them and kind of see where it goes from there? And and going back to earlier some of the the value that fedora provides to us You know this is all new this this whole path has happened within the past What six eight weeks so one of the things that we're strongly looking towards is how fedora handles? you know making decisions and changes within the operating system and The way that fedora has it all set up is is wonderful So we're able to look at fedora and take take a lot of the ideas and and you know turn that into something positive You know as a differentiation for Alma That that can meet different needs So my question is kind of in the same vein Because if I want to contribute to center stream I contribute to center stream If I want to contribute to change something in row I contribute to center stream But what happens if I want to change something in rocky or in Alma? Where do I go? I guess I'll go first if you want to I'll let Neil cover rocky. I don't want to speak for him If you want to change something in Alma it could go either way We're still using Senua stream as our primary upstream So anything that does get in stream you will see an Alma Linux, but we can do things in addition to that So it could be something you know say you contribute it to stream or talk to the maintainer of a package in stream or want to add another package and You know there might not be a pathway to do that in stream and subsequently rail That's something that we can still look at doing And that that's kind of the difference that we can take with the rail compatibility, but not real clone model Basically same thing for rocky Outside of having the core what you get when you install rocky and update your system every time being what you get from rel we're adding Improving and facilitating collaboration in our SIGs with the upstreams as well as just internally to help folks that are using rocky to add To change what they want to and and add the value to rocky in that way without having to necessarily Go through the process of contributing in different upstreams It's I can speak for myself that getting involved in the fedora and sento a stream communities and Collaborating in those personally from my background and coming from the rocky side wasn't a super Anx-free experience for me And I know that that's also true for other contributors from rocky and from Alma and other distributions who may feel disenfranchised in some way and I think that's something that rocky and Alma and sento a stream and everything else can help to Build back in and bring those people back into the fold under the fold into the ecosystem I would I would disagree with your comment about feeling disenfranchised. I feel like I was When I wanted to get involved with the door I was I was welcomed with open arms Yeah, no, I let me rephrase I I was definitely welcomed into both communities with open arms I just felt personally, you know from where I was coming from that there was Resentment or something coming from that side which may or may not have been there and probably wasn't right But I know that that's not an uncommon theme for folks that are coming from that background coming from the maybe More emotional side right and I think that as humans we are all very emotional It can be hard for them to feel accepted into those communities even when those communities are literally saying we're welcoming you Okay, I see The the thing I would I would start again on the on the stream size specifically is Because trainees was used to effectively develop rail There's gonna be a set of changes that aren't going to be Acceptable for stream, but that doesn't mean there is no space for those changes in the project So the example is for example the one of the things we do in the center cyber skill seek is maintain the latest version of system D As something that can be used on center stream and that's something that obviously wouldn't be acceptable for string proper or for Well, because they're not necessarily gonna re-based system D every time there's a release of system Although they have re-based system D a few times now in in the in the nine cycle, which I was personally very pleasantly surprised So that's an example of something that can be developed within within the ecosystem It's also something that then is available to other projects in the area that want to leverage it if they so desire Thank you. I saw my question mainly for for red hat and sensor stream this point Regarding the contribution model between red hat and sensor stream For example, if red hat now finds a bug in a code that they are pushing upstream What will happen if the developer upstream didn't want to accept that code? Would they still be available in rail or not and what will happen in a few months time? When sent us a stream goes end of life and red hat still continue with rail had it There is no way to push upstream unless you will backport the patches from rail 8 to send us a stream 9 as far as I understand So if somebody can shed some light on this, sorry if I can ask a clarifying question Because you you set upstream multiple times and it's a loaded word Do you mean upstream as center stream or upstream as the upstream project? If I understood the question correctly, and please correct me here if I wander off course Now I'm a little worried with this clarification But let me try let me let me give it a college. I can split it like two parts if it yeah split it into two parts Okay, so at the moment Howard hat is contributing city sent us a stream our bugs fix it in sent us a stream first and then merge it back to rail or Well fixes the bug and then submit it to and release it to their customer and subscription and developers And then push the patch to census stream. This is part one So if you look around the room carefully while I'm answering this you will see the people that you want to talk On the hallway because they're all about to blanche But as someone who does not regularly push mrs through either process I'm gonna give this the college track the general process for almost everything that red hat does in rel Except for rel 7 because there is no sent all stream 7 is to do that work in sent all string That work goes through an mr. Process. It is discussed and it is merged Period there are some exceptions and the primary case for those exceptions is like an embargoed CVE where that work cannot be done publicly and In that case we must do it to the side We then ship it to our customers in rel and then we put it into sent all stream there is no artificial delay in this process and My understanding is that depending on the engineering team involved. They even have the the m are ready They're just literally waiting for everything to come out of the embargo the ship to go and then they're pushing it in stream Like it's not even a now go start over problem They've been keeping it up because if you think about doing that kind of work like you're already in there with a needle Picking up drop stitches to begin with you're not gonna want to set this up to be something you do twice This actually also happens with fedora packages with red at maintainers where there is Embargo to issue that red hat is aware of that Fedora me the community might not be is often the case that the maintainers you build it internally and have That ready to go when they can but it asked to wait for the embargo to be done Justin's got a clarification here Like I said look for the people who are blanching right now that's that's the person you really want to talk Yeah, no to clarify that was special with embargo TVs build it internally Yes, we can build it internally. I build embargo kernel CVs on my home system all the time I test it locally as much as I can make sure it works But there is no system in fedora for me to build and stage something that would be an official build So I can't start the build Process until I can commit to disk it and then get it through Koji and that's all once the embargo is lifted That's the same in stream. So yeah, that's that's basically the cause of the delay There is not having secret places to do things and I've gotten yelled at when I proposed secret places So just so you know the community is working The second question that I believe you had was what is going to happen with Centaur stream 8 at the end of its five-year Life that the centaur's project has announced and that red had obviously supports because I didn't deny consensus We don't know yet We're in the process of figuring out what our development strategy is going to look like and then We're going to move forward with letting the world know what our answer to that is we will of course do that in consultation with the centaur's board But the project board has also embraced the idea that centaur stream from their perspective is a five-year thing And and the USS Enterprise will launch on its new mission. So I had to work that in there, right? So so I think that that's that's the perspective and I do want to leave room for David I if he wants from a board perspective or project perspective to add something Yeah, I would say from a in a general in a general standpoint if you go on good love good love calm Right I send a stream and you look at MRs on the group you can see very very clearly How the activity flows you can also see that oftentimes the people making the MRs are Still learning how to work with the process. So you you might see a variety levels of quality in the comments, shall we say? That I think is proof among other things that this is really where the development is happening And there isn't like a secret cabal on the side doing development somewhere else But it is understandable that for things like embargoed CDs and things like that there has to be a separate process That just no there's no way around it and like if you've I've been in environments where you get the email that oh Yeah, tomorrow at 4 a.m. Expect something to happen to open SSL I'm sure you might have been in environments similar to that. There's really not much you can do in those cases He the good thing though is that least in every example. I can think of what these happened As soon as the vulnerability was public the the commit was published almost immediately on get love And then the build went out within a reasonable time I think there's only a couple of example. I can think of where these took Slightly longer because people were still learning the process effectively And I apologize. I don't like chaining, but I want your question reminded me of something And I want to publicly acknowledge it when Centaur stream was first announced it flowed backwards. Oh Yes, we fixed that we fixed that a long time ago and Non-red hat employees were not the only people upset about everybody trying to swim in the wrong direction So we did at the very beginning of time have this weird system where things went around the curve And then kind of pretended they flowed down that has long been gone And I really encourage people to do as Davada has suggested and go take a look at get lab That is your proof like you don't need to believe me If you if you find it useful I brought Like a short block post for block the census the dog a year or maybe more I don't know it's weird we covered but like some time ago I wrote a blog post that kind of tried to explain how this process was because it was something that I wanted to understand I found that fighting it was a lot easier and that people at work needed to understand So maybe that helps is that from Alexandra? No, no, that was something I wrote Alexandra also gives a talk and I think had also right up that was very helpful We should probably write up part two of these that was the two together because I think that would be valuable She couldn't make it because of visa things, but would be in the audience correcting us on everything we say here Correctly to So just wondering what the collaboration workflow processes for in Bargaret CBE's which regard to rel sent us stream Alma and Rocky Because you know, I imagine that if we get another specter meltdown type situation It kind of behooves us in the sense of partner and openness to Include the distros as early as possible And of course as early as possible is when they get read in and when is that but I'd like to know What is that process today or does it need to be hammered out? I will speak at least from my perspective and from what I know and obviously I can't speak to what happens internally at Red Hat But I do know that a lot of the CBEs are circulated on a private Openwall.org mailing list pre-release in the embargo period where those patches are discussed and Red Hat canonical a number of other organizations. I think meta might be one of them I don't remember. I don't know if I can talk about this. Okay. Well. Anyways, there's a list. It's public but That's kind of where that process happens and there are Requirements to be part of that group and so I would personally love for more collaborate for us Distributions like Alma and Rocky to be able to participate that And I think that these sorts of changes that are being made specifically with Alma being able to vary and and build off of rel in a diversified way that there's definitely Reason for more organizations. I think to be part of that and to collaborate I am not an expert in this space And so I am only going to say that Red Hat fully cooperates within this space where it where it exists And and I honestly don't know the status of Alma Lennox and Rocky Lennox with regards to participation in this process I Encourage everybody who is building code to pay attention to where that code is coming from and to be able to have the Enduring wherewithal to manage and control all of the pieces of code that matter to them Beyond that my understanding is that we do all of the CVE work on behalf of Centos Project and on behalf of Fedora project my suspicion is with a focus on Fedora Lennox and a focus on Centos stream However within the Fedora project my understanding is that it is still true that everything that Fedora produces is derived from Fedora Lennox and That everything that is currently being built by a SIG in the Centos project is derived from Centos stream So indirectly all of those other pieces will benefit from that work. Yeah, I think the only caveat here is There is no formal CV process for content coming from Centos SIGs So it's up to the need to a SIGs how to handle that. I would say most of the SIGs end up building Content as derived as you mentioned from either Fedora or Centos stream So it's pretty immediate to do that But if you're looking for a process that involves certified metadata and things like that you are probably not gonna get that there Maybe you can start a SIG to look at adding that Mirroring everybody else up here. I'm not the best one to talk about that But internally some of the questions we have at Alma right now like when we created our Arata for example That was done in a way to mirror rel and recently when we were able to push a CVE fix Before the fixes made it into stream. It basically calls us to ask ourselves a question of you know We've got to rethink how we do Arata now Because we pushed that first and and at the time we had no idea if that was gonna make it into to stream And rel or not it turns out it did so you know now we'll pull that same Arata But we've got we've still got some rethinking to do and you know It's worth saying that none of this is a hundred percent figured out. You know, it's our plan we're figuring out the exact structure and what it looks like as we go and Trying to get involved in the development more and contributing to stream is helping to force our hand on figuring that out It's forcing us to stop and ask ourselves these questions I think we're running out of what we're way over time way over time right so I guess If there's more questions, well, this will be a continuing conversation. I think I'm really glad for everybody came up here You know, it is a little bit of a scary thing. So it's good to I think everybody who wanted drama should have come to my talk Where we talked about discourse and the mailing list there you could have had it I think we did a good job of disappointing everybody who was looking for it here. So thank you everybody and Think we're gonna go on a scavenger hunt and dinner. So Justin's got some stuff to talk about so first Thank you. Have a pause for our panelists and our moderator so Quick housekeeping for tonight. We are going to go to a community dinner. Everyone is invited tonight We will have shuttles leaving at 5 30. So I would suggest being in the lobby at 5 15 So take some time to refresh or we also have three workshops that I have another half hour. Well, we ran over a little bit So they're probably gonna 20 minutes But meet us in the lobby at 5 15 We're gonna go to dinner and we're going to do the scavenger hunt through Cork City from the restaurant so I hope to see you all at the dinner and We do have to leave this room because there is another event in 20 they're gonna set up for the event and take everything down here now So we will have to do we are gonna have to do a quick exit here But thank you all for coming for day two of flock and hope to see you at the social tonight