 Do I have mic and I speak with the check check check good. Okay. I can't I can't hear myself at all or not We have engineering things that need to go the other direction Josh is allegedly the conduit for that. That's right Hi there everybody My name is Remy to Cosmaker, and I'm the fedora community action and impact lead So mostly what I do in council land is the stuff that is not engineering Generally speaking so ambassadors budget budget budget ambassadors Education as well. I'm also the fedora council's education objective leads So figuring out how to do more stuff on universities and campuses around the world not just in the US and I serve on the Red Hat OSAS team So I helped to liaise between Red Hat upstream fedora community. So when things are Broken or not happy or whatever these guys are all doing the good work and The productive things and I'm here to help take community input and provide it back up to Red Hat land. So Come talk to me So welcome everybody. My name is young kujik and I'm program manager for Fedora. So I'm responsible for scheduling I also have the role of change Wrangler and I'm also running elections. So I guess that's like this three main areas. I'm responsible for So if you have any anything Please send me an email because if you ask me here, I will probably forgot it once I leave this room. So You can ask whatever we want But if you want something just send me an email and I will probably or hopefully I will not forgot it I'm Dennis Goma. I'm one of the release engineers and Don't send me email because I get too much. So if you want to get a hold of me ping me on ISE. It's much much better and I You know my job is try and make sure that we get the bits out the door and can do it better and faster and more My name is Adam Miller I work on the floor engineering team. I also recently joined Fesco I'm also kind of I do what Dennis Needs done doesn't feel like doing To the best of my ability I try to float around also I do Docker Atomic things in Fedora space. Well, I work on them. I don't do them I will not even remotely attempt take down that much credit. I also participate in the foot or cloud sig I'm Langdon White Originally, I was on the council with no responsibility and then I which I enjoyed But then I foolishly got led into taking over Fedora modularization So my role is tricking people into doing that, right? The things to say first of all first of all Adam snuck in here He's the community monkey. I think is for a Q&A. Am I in this now? I just yeah, we're like we're pulling like you're You can stay sitting if you want or you can come up on the stage. Okay. I'm oh man, okay Oh Gee, thanks. Okay, I'll go and stand in the corner. I'm Adam Williamson. I'm the Fedora QA community monkey officially I'm like the QA guy. I guess People call me the QA guy. Thanks. Yeah, you're welcome Also, we have on the Fedora council, but not here. We have listed up here. I'm Christoph Wicker Who is been appointed by Famsco to sort of represent that ambassadors and community side of things Then Roby duck who's one of our elected representatives as well And he is also from the websites team is the leader of the websites team and he's generally awesome and then Maria Who we a lot of us know as Tatica is Going to be our new diversity advisor and that's just starting up as well. Okay, so that's that's the introductions Now it's the questions time Hey, Brendan. Do you have a question? Where's the mic runner you're Yeah, sorry, I almost appointed you but look we got one Does it work it does work. Okay first. Thank you for like living and breathing Fedora It's what I use every day and it makes me happy. So two questions So RCM question about secondary architectures First I've heard two slightly different stories about the secondary arch In moving to the new Koji will build failures initially Block the success of an overall build or is the notion still that the secondary arches if they fail the build succeeds So if we get to a world where we are Building all of the RPMs. So in the RPM case building You know, it has to succeed across all the architectures and if it fails on one It will fail the RPM build in the case of composing the distribution and putting together Fedora If for some reason the you know, the cloud images fail or you know building of anaconda fails because building the install Environment and you know, we don't get install media if that fails on a secondary architecture that will not fail the compose Because it is a non-blocking deliverable and it will not block stuff going out But in the RPM build sense it will we're looking at putting in ways to make sure We're looking at putting in ways so that if You know if it fails particularly leaf node stuff if you just add an exclude if you add an exclude arch The second the secondary arch teams will get an automatic notification that someone's committed something that has Excluded an arch and you can do that and move on the core stuff generally just works And it's we don't perceive that there's much risk of holding up anything in the distribution because of that So so hang on hang on slow down slow down hang on wait arm. Yes Arm is definitely the slowest, but we got some new build hardware, right? And so they should speed up I'll repeat the comment while it's running the comment was that it'll be won't be really slower much slower to build everything If we block and all the art So we have new ar at 64 build hardware. There's 30 nodes The testing I've done on it act when running an arm v7 VM on it it takes say for example a good two hours off the current Arm v7 kernel build and in some cases Possibly even faster than the current x86 64 hardware in the case of Say the power So power 64 big end in little end in the hardware is actually significantly faster than x86 64 hardware that we currently have So in the vast majority of cases moving forward The hardware will be at least as fast as the x86 64 stuff So you shouldn't see any speed regressions and as soon as I get networked to the hardware And we can start to put it in to production The builds will go up significantly faster. So the arm v7 hardware we have at the moment is about 1.3 gigahertz 4 gig of Ram The VMs that we're running are like eight cores 12 gig of RAM or something like that with SSDs and various other bits and pieces. So They are fast and a lot faster than the current generation of arm v7 hardware The age we're just gonna be awkward and do this a lot Okay, switch in switching from I get also hardware to security So we have Fedora packages get the certificates generated every half a year Currently they are generated on the server. What is the plan to? Basically switch the Being generated at the client side where actually the private key should be not at the server I don't think there is any such plan at the time to do this But if you want to help contribute to doing that, that sounds all right The question Dennis was right now when we read we regenerator certs every six months He asked when we're going to start generating the certificates on the client. So the private keys there So we do we the CIA certificate that is used to create the user certificates When I created it, I'm like, oh 10 years will give us plenty of time So it actually expires in two years. We have to do something either renew the CA or something else What I'd really really like to do but need to have the cycles to do it is input dog tag in Infrastructure and then have the user would create the key on the private key on their machine and we could then use certmonga to automatically renew the certificate and We could set this other certificates to expire like every two weeks or four weeks They'll get automatically renewed in the background and that way if somebody steps away There's it reduces the risk that you know, someone could use that get a hold of their key and you know use their machine to you know build things and So you know from it from a general Perspective I think would be better overall, but you know, we need to have time and resources and what not to do that And we have two years Or wait, what is the plan to make that susceptible for other people to help do it? I heard somebody was mentioned Kerberos in the audience are you was that you volunteering to do it because wouldn't that be awesome? Okay Okay Okay, well, perhaps we need to talk because I've never heard of all right, so that is Yep, and I think we've had historically some communication problems about how to get those kind of things into stuff That's happening and in the last year or so if you went to tennis's talk, which I skipped We've got a much better. I know right. We've got a much better process for Making sure that those things don't get dropped on the floor when they come in so Yeah, I mean there has been some talk and moving fads to be based on free IPA and to enable Kerberos as well as SSL certificate authentication for Koji and different pieces of infrastructure but it takes people You know doing the work and you know, we're getting better at enabling people to do stuff But we're still not perfect We have the fastest basically been rewritten recently so I mean that's that was a discussion that went on for a while I don't want to rehash the whole thing here But I think eventually it's one of those things that you know Maybe we want to move away from from NIH in it and and doing something different and I don't know what that thing might be Hopefully, you know something like IPA would work, you know, maybe there are other things out there I I don't know the answer to it because I'm not the technical dude who would suss that out But I'm certainly interested in finding things that would reduce Cases where the Fedora engineering team ends up being like their own upstream Because that just means more work for a small number of people and as awesome as they are there They don't have an infinite amount of time. So Now agreed to make that up the question or the comment is We want to avoid writing custom software and they want us to avoid writing custom software And it would be awesome if we all work together and we'll be done in two years Problem-solve let's go home Next someone down here then someone up there By the way, I don't hear that snuck in late. He's Paul Frills. He's the Fedora engineering manager He's also my boss, which is another reason why I'm standing up here So in case you didn't know former FPL yeah, Paul Just He came in late and didn't want to be awkward and sit down. So he was like, yeah Paul's also on the Fedora workstation working group Yeah, hi guys, so I actually wanted to ask a question to make this poor guy run up the stairs I Have two questions. So there's nine of you on stage. Are you the nine power users that might Matt referred to this morning? Yeah, Dennis is one of them and Adam is one of them Okay, so now do you know who they are Matt? Yes, I'm not going to name all the names though I was just curious My my question actually is is and I know you guys like to dump drive Right into the technical details and have these technical conversations But I would really love to know how somebody who has no coding skills whatsoever can get involved in fedora and it would be really great if you could Talk more about that So That's a great question and there are a couple different places where people can get involved So there's a new team within fedora that we started this year called community operations Which is sort of a generalist group of people that are ninjas that are supposed to go between the different sub projects And within that team there's an onboarding process so that you can get acquainted with each part of fedora So we have an onboarding series of badges which are part of fedora badges So that there's easy on ramps to sort of get introduced the project and do tasks like Write blog posts interview team members help with the release schedule tasks that sort of don't fall into one particular place Like helping yarn with elections and helping to craft the release announcements and a lot of non technical But very vital community infrastructure once folks can start on ramping there and with also fedora join and What can I do for fedora org? Which are great sort of sorting hats to help people figure out where they want to get in Then they can get introduced to different parts of the project and figure out who the upstream mentors are within fedora And then get plugged into those teams So we're working on that a lot and with the diversity and inclusion advisor tattica I think we're going to do a much better job of engaging with other international communities as well in other underrepresented communities You know, there's a lot of it's not just about culture. It's about atypical contributors So having non technical ways to contribute to the project is huge and outreach He is a a program that helps us with that as well where we can get interns who are not necessarily technical to work on this Kind of stuff. So this is a conversation. I'm very excited about I would love to talk more about it and there are lots of other places too to help and I saw other people who Wanted to answer so i'm going to pass the mic Awesome So I just want to say qa. We obviously have a lot of stuff for non technical people to do We have Entries in what can I do for fedora and we have a wiki joining page Which is not the best thing ever we would love help with cleaning it up But it is there and it gives you a list of various things you can do that are not technical And helps you do them. Who's next? Our dumbest guy on the stage gets the mic. That's awesome. Um, so yeah So my my entry into fedora was non technical and you could argue I'm a little more technical now Like I actually wrote some code in the last few years, but it's terrible. So I think that counts in my favor as being non technical I trained for that after I came to red hat like it was it was like I was almost forced to so anyway, but um There's you know Documentation group is out there. That's how I got started writing docs I have very very strong strong love for docs. I know steven's grinning up and steven gillson's a content strategist at red hat and he is awesome and um, and one of the things that I feel very passionately about is making sure that people can contribute in places like that Without having to learn a whole bunch of tools and and right now the docs group is kind of re-energizing and sort of thinking about how that works. I do not want to speak for the docs leadership Which includes p travis. I don't think he's here. He's an awesome dude And I think he also feels that the tools and things they're using are have become way out of step with what the rest of the World prefers to use and there are so much better things available nowadays So for example having somebody who does has no affiliation with fedora Able to actually drop in edit a document live and have that change actually come to the docs team As what you know a pull what we call you know a pull request in the Coding world right and somebody in the back end is going to see that and look at it and take it in or not right And when you know when people get a little taste of being able to do that and they get a thank you back Or you know if they sign into fedora later they get a badge All those things can be used to sort of keep the project sticky, but it's all about Making possibilities for for people to do things in the long tail because as matthew showed You know we we we actually have you know 10 like like I said, I think 10 percent of the 10 percent or was it 10 percent of The population was doing a third of the work. I can't get I keep I keep forgetting what it was 10 10 percent are doing two-thirds of the work. That's that's not terrible But it's not great But the good thing about it is is that it means there's a very long tail of 90 percent that are doing the other Third of the work and that is a lot of work So if we can make more ways for that tail to get bigger Like more of the mass to be in the tail like the brontosaurus to have a bigger tail That will be awesome. Although. I know we don't like brontosaurus, but they make a really good visual for that They're okay again. The brontosaurus burgers So the only thing I would also add great Right now this minute The best way you could help to contribute in a non-technical way to is to actually work with remi Working on the how do you contribute in non-technical ways? Oh and design. Yes designs a big part Moe can't do all of the design is basically Well, hello Hello, I I'd like to add something from my from me. In fact, I unfortunately we have issues Not only with non-technical people but also with technical people who want to join fedora For example who want to work on the infrastructure Like I'd like to remind you what's going on for just to went here when a guy Who knows a little about cavernous wants to participate and all all he got in return is just You come come and do something. Unfortunately, it doesn't work like this in the open source world I met you. I already told you about a year ago and probably previous year It's almost impasse it almost impossible to contribute to fedora infrastructure Because in order to contribute One could Try to implement a part of infrastructure to want to contribute and this is impossible because we don't have a configuration We don't have as schematics. How everything built That's why we don't see a lot of people Starting contributing on fedora infrastructure. What's compared to for example other areas Yeah, that's that's cool, but I didn't finish yet what we need is the The things Is a plan how to contribute actually you have to download this and this is a let's say koji We have to know that and that's that's a body body our body and if you want to improve it This is our repository and this is how you can you download how can you start it in this configuration our virtual environments We don't have it yet So before we start to Well All right, so Last tuesday, I guess it was at config management camp, which was in gent in belgium We gave a talk on open infrastructure in fedora So it may not be as widely known, but we have a wide open infrastructure the fedora infrastructure apprentices program All you have to do basically is sign up for a mailing list Introduce yourself respond to an email once a month and join the weekly meetings We recently migrated in the last two years all of our ansible playbooks are absolutely public So every part of our infrastructure is public and open and if you join them Wait, if you join the meetings We talk about all of the things on the agenda and the fedora infrastructure team is actually one of the most open parts I mean, so that's like if you want to get involved in fedora and for specifically There's a slide deck. It talks about we just presented on this in gent And I agree like we can absolutely do better with this stuff always and we need to do a good job Now So It's not it's not just promise to do better we've got So we've got developers at fedora project org, which is not infrastructure specific, but it is stuff for fedora developers We're working on hubs, which is Not specific to your question, but it will show you exactly what's going on in the fedora project kind of in a live state Right, so I agree. We have these problems where people That want to contribute to fedora or fedora infrastructure specifically don't even know where to start, right? That's kind of the problem So we're This is like the one time I get my boss to be like So I think that over the years we have realized that this is hard to do and we're working on it And there are small things that will eventually bring them together and now i'm gonna let paul talk just because he pays me I do not So one thing the the one thing that we like to say is we actually there are a whole list of tasks that are called issues And the fedora infrastructure, right? We've got a track that's full of them And those are things that people can work on so if you are showing up not knowing what you can work on That would be a way to target what you are working on and then you can work with one of the people in the infrastructure team To say specifically here's the thing. I feel like I can do what do I need to do in order to be able to fix that thing But I will also tell you there is a barrier to get over and it's called trust, right? When you show up We're not just going to let you you know traipse around our infrastructure Yeah You're what you're saying is you would like an environment that you could replicate parts of our infrastructure and do that kind of thing Right. Yep. So so yeah, yeah, that's what so your rem is getting to the exact thing. I was going to say so our We have an answer That's Right, we they they they don't even hand me Like like supposedly i'm these guys boss and they don't let me on these systems either It's really easy to join the fedora infrastructure apprentices and get basic access to things. It is I did it Yeah, yeah, we we but but I will say Okay, let me get to the last word So the last thing is that We do give read only access to all of the infrastructure when you join the apprentices group So even if you don't have right access, we are absolutely trusting you when you join the infrastructure team to see exactly how everything works There's we're trying our best to lower the barrier to entry for trust and this is something we need to talk Now task identification is something we can do better with absolutely. We have easy fixed bugs But that is everywhere. So we can talk more about this. We got to go on another question I have to say one more thing. Oh, then we really are going And and the last thing was that um, I definitely hear what you're saying about being able to replicate Infrastructure and I think that is a worthy goal So I am going to take that under advisement and see if that's something that we might be able to pursue somehow Because I like the idea of being able to like Shoot down like a at least a microcosm of what we do so that you can actually practice at home or try something And see how it works. I think that's a great idea Hi You've been to like 12 years ago made linux really simple. So codex drivers. You can simply install them there I know why it's hard in fedora, but will we finally catch up there like Sooner or later and when how Matthias, would you like to take this question? You would like me to take Paul would you like to take this question? At the moment we're working In getting the final steps put in place so that we can have a fedora built and Cisco shipped open h264 codec And there's lots of hardware now that is implementing In the hardware like media engines to decode all the codecs And as long as all of that everything that's patented is in the hardware There's no reason we can't have the glue in fedora that can you know street You know demux video and audio and send the video to you know the hardware to do the rendering and I think it It's gonna be a less of a problem over time, but but patents are continuing to be a problem This is a government issue that we cannot fix. We are bound by the law So if the longest the law is hostile to us, so it sucks That's that has nothing to do with codecs though, right It doesn't I mean it simply works because they don't rev their kernel Yes, but quickly because this is a big and it was Really hard And it was way more disastrous than you think it was So just take my advice We're really not in as bad a position as you think we might be Where the where the projects originate and where they're bound to by law like we we are we have to comply with different sets of patents and things in export law compared to Something that's founded in south africa I know I know but like I feel like it's it's a circular question Like there's certain problems. We can't solve because we're legally bound Yes, he asked about the invidia driver so To answer your question. Will we ever include the invidia driver? No, but but But um I believe there is work that is either planned or already done Where if a user were to go out and get the invidia driver For example and install it on their system and somehow magically make it work We will try not to break their system for them Okay So the problem with the invidia driver aside from it being proprietary and closed source and everybody knows this Is that you install it you finally get it working You run dnf update and you get a new kernel and you reboot and now you don't see anything on your screen Or you know Other bad things happen. Um So I believe there was some work planned I'm looking at matthews because he knows of this as well. I'm not sure it's done, but there is work planned Yeah, so so that if you do that Um, and it gets working and you update your kernel It'll try really hard to not set the default kernel to the new one And then you'll have to like manually go in and say, oh, there's a new kernel I need to select that and figure it out now It's not going to fix it if it is broken, but at least it'll save you the trouble of like Having to reboot seven times trying to figure out why your new kernel doesn't work when really it's your invidia driver So we're not going to include it, but we're going to make it slightly more user friendly so because I mean There are people who need faster drivers and that's cool I'm not going to weigh in on the the morality of the issue But we can't do things to make it better for users who are stuck in that situation We have another Well, I didn't check but can't you just include the dkms it will so why don't lip gel But alternative system is I mean just copy that what debion did the alternative system is still in fedora But it doesn't work well I would let you do that, but you got to be quick because we're like going down a rabbit hole here All right Yeah, this is a question that comes up every time we had this panel, but we we had a slightly new answer Yeah, I mean it took 10 years, but we're kind of starting to listen All right next question five minutes make it quick There's one back there. All right, so matt has the Popularity of fedora over time graphing with a giant spike at the end of it Um Have we mapped that to anything because like if you look at distro watch or anything And there's no massive spike in fedora in recent I'm saying can you prove in any way that the stats you have are Relevant like Yeah, you just had it. It was like I believe that linux mint is the top of that graph because people are searching for other kinds of mint and you know Like for their drinks and whatever and there they go on that page. Um, so We have some correlating evidence. I actually don't even have that slide in this deck anymore Yeah, so there's some other other things that show With the correlating evidence that shows the popularity is going up for these releases There's a new feature a network manager which hits a File on our web server to help determine whether you're trapped behind a captive portal like at this university or not So that it can tell you that you need to log in to their terrible page and all the terrible things that that entails Uh, and so that feature was added in fedora 21 And the hits on that are going up at the same rate that the that the adaptation on the update server of 21 and on are going up Pretty pretty closely. So there's correlation there So I am confident that the line is definitely going up. I still have no idea what the axis means But I'm confident and they're in their relative amount It's a happy graph and also start dot fedora project dot org, which is the page that people get by default as their initial install Sorry, so there's a start dot fedora project dot org page that mazilla fire fox features When you first install fedora A lot of people we believe leave that on their leave that as a default I mean just because they just never get around to changing it and it's not that big a deal anymore And uh, you know, I know laying to raise his hand. I do too. I don't think I've ever touched it But um, but that number also is going up And so that would also be a good indicator that there are more people picking up So so that's like is that there's there's several correlating factors here Like we didn't look at this and go great. We got the numbers we want. Don't look any further Well, I did but But then you didn't so that was important. All right. Do we have two minutes for one more? And you got a better question better question That was a good question I sometimes browse the fedora users list archives and Um, I know matt answers a lot and so does some of the members on this stage So any comment on the tone and how The user happiness or unhappiness Um, you observe, uh, I would observe that email mailing lists are for old people as I said before And yeah, and in fact, I think that for a while we had a problem and I'm I'm an old people in this context So I got to put that out there We've had had a problem on the mailing list where the tone was really negative for a while And that drove away all of the people who didn't want to deal with that So the default tone on the mailing list is set at a grumpy-ness level that is higher than I would like it to be Um, I hope that as we move to hyper kitty Then we will have a sort of a new uh, the influx of people and I intend to be very Um aggressive aggressive. I'm not an aggressive person aggressive Enthusiastic about encouraging people who take a negative tone on the mailing list to not do so Um, so that new people feel more welcome on the lists there including enforcing our code of conduct well as well as The other It would help if more people went to hyper kitty and use the hyper kitty features to vote things out I was going to say to other communication mechanisms as well. It's such a lot of time. Is anybody stopping us or just keep So there are there's still work to be done here. So we are getting there though All right, thank you Red switch Dude do we hear me