 hit record. All right, so thank you everyone for coming to the usual Fedora Colonel talk. For those who don't know me, I'm Laura Abbott. I'm one of the members of the Fedora Colonel team. The Fedora Colonel team is part of Fedora Engineering managed by Paul Freels at Red Hat. It is my full-time job to make the Fedora Colonel to be the best it can be. So general overview about what's going to be talked about here. We're going to talk about who's actually working on Fedora Colonel. Things we've worked on last year, things we're going to work on last year. Pretty simple. Several things you're going to hear about are pretty similar to what you heard in the Fedora Project Leader talk this morning. A lot of the same things. I wrote mine independently just having to come out the way. You'll also hear several themes come up a lot, especially in terms of getting new contributors. So who actually makes the Colonel happen? LWN, the Linux news site, likes to put out a report each release about who is actually contributing to the upstream Colonel project. These days, most of the contributors to the upstream Colonel project are paid contributors by companies. There are still a handful of non-paid contributors or just hobbyist contributors, but by and large, most contributors to the upstream project are paid maintainers. I'd say the Fedora Colonel follows a similar path. We don't get a whole lot of general community contributions. We want to change this, but the type of contributions we do tend to see from the community tends to be things like my laptop isn't working for this specific bank. Problems that are very specifically focused on individual hardware. This is value. We always appreciate these contributions, especially when they come with a specific problem and a specific patch. Anytime I don't have to go looking for a patch or some provides that or even just brings a problem to my attention that helps to overall make the process better. So again, we encourage you to make contributions. I can't guarantee that every contribution will be accepted, but bringing a problem or learning to us is a good way to get it to our attention. So we in fact know the problem exists. So thank you to everyone who has contributed to the Colonel as a community over the past year. Okay, so who else is making the Colonel happen? We work with a lot of engineers who have specific focus areas. The Colonel is a big piece of software. The number of people who are familiar with all of it can be counted on one hand, if that's especially in the Colonel, you end up with people who end up with a particular area of expertise. They work on a lot. So we work with a handful of groups on graphics. Graphics is a big area where we get help. Graphics are one of the most common places where we see problems. Graphics is broken down by teams. We have different teams working on the I-915, UFO, etc. The same people who work on the user space drivers also work on the Colonel drivers. This is why if you report a Colonel graphics bug, you'll often see the component change is something that looks like it goes for user space components. This isn't us trying to get rid of the bug. We just automatically are not care about it. It's because it helps us do the tracking. So, yeah, for the graphics team to help us make things better. Another area we get help with is input drivers. This is especially true for things like laptops. Laptops seem to have different quirks each time, different inputs, keyboard mappings, something that seems to be breaking. And one of the reasons why we rely on other people to have who have the special knowledge to help with us is because they know exactly where to go looking for things as opposed to me, for example, trying to guess where things are supposed to go. So, especially with input drivers, you generally get people who know exactly what file of change to match on a specific PCID generally helps make things better. Architecture specific fixes. The Federal Kernel Team tends to work mostly on X86. It's just on hard support models. There are a handful of other teams who help to support other Fidora architectures. ARM, ARM64, PowerPC, S390. The people who work on those, they do a great job of pulling fixes and making sure those are validated. So, thank you to everyone from those who are also working with the Fidora Kernel Team. So, I basically talked about everyone except the actual Fidora Kernel Team. So, who exactly is on the team? So, there's three of us this past year. There's been Josh Boyer, Justin Forbes, and me. So, I talked a lot about what everyone else does. So, probably I think what the rest of us tend to do with our time is releases. This includes your daily raw height updates, stable updates, rebases. We take the release kernel that comes from upstream and make sure it actually gets out to Fidora. When I was working on my goals for this year, I spent a long time trying to figure out exactly how to describe the process of doing the updates. When I describe it doing the updates, it sometimes sounds like you could replace me with a CI box and just get rid of me completely. I think we have concluded you can't quite do that yet. And I think a lot of what we do with releases and what we try and do is really we're tracking everything, both what's in the kernel and upstream. The three of us on the team, we try and stay active on reading LKML knowing what's out there. So, the hope is that we'll familiar with what's out there when bugs are reported that we'll be able to have an idea about, hey, I think I saw a patch to make this work, or I think I saw a change that might have grown something like this. So, really, we're trying to help set the direction the Fidora kernel is moving. And we're always trying to listen to everyone as well to find out exactly what everyone else wants Fidora kernel to go. The listening is important because we don't just want to make decisions in a vacuum. We really want to make sure is that the kernel is moving and solving the same important problems as the overall Fidora project. Okay, so that's everyone who does things. So, what exactly are we putting together? The kernel is still just a single project. A single does not mean simple, though. There are about 10,000 lines of configuration, kernel configuration options, sitting in the package get directory. The kernel spec file itself is about 2,000 lines when you take out the change log. So, quite a behemoth that'll have been intimidating. We sometimes get questions about why is there only one kernel? Why is there not something like kernel cloud, kernel server, kernel IoT, whatever. And I think a lot of the answer is that the kernel is complex enough as is. We have enough time dealing with the support we have without trying to add complexity. For some architectures like ARM and X86 that have 32 bit variants, we do offer PAE variants. PowerPC has a couple of architecture options as well. And those architecture options that we have already add to the complexity. So, the small amount that we do have for actions, packages, add stuff as well. I would probably say that if the Fidora council came to us tomorrow and said, we really want to drive this idea of different kernel packages, here's all the resources you need to make it happen. We could do that. But until that happens, we're just going to be working with a single unified kernel. But even if we can't have different packages, what we are dedicated to doing is trying to enable as many options as possible. If a driver can be enabled as a module, we want to enable it. We want to have it available for people to use. Obviously, we can't enable everything all at once. We are eventually going to have to make some trade-offs. When that happens, we're ultimately going to be looking for what direction is Fidora moving and who is this overall going to be benefiting. So, again, this is where we appreciate hearing input without knowing what people want. We won't have any idea how to make these decisions. Okay, let's talk about some numbers. Sometimes we've talked about bugzilla numbers. For various reasons, I don't like talking about bugzilla numbers. I don't think they're a great reflection of what we do. So, what I'm going to talk about is some stats related to actual commits to the package get repository. I ran some numbers for one year from July 1st, 2015 to June 30th, 2016. This range was kind of chosen somewhat accidentally. This was about the time period when I started working on the presentation. But it turns out I think this was a nice snapshot that shows the eboflow of Fidora releases and where we end up spending most of our time. We had F-21 who went EOL, part way through the period. F-22 who went EOL towards the end of this period. F-23 who was released during the middle of the period. F-24 towards the end. We're all high given the usual rolling release. So, a wide variety of releases. So, the basic stats, this is the number of actual individual get commits we made to the various branches. This is everything, bug fixes, updates. Oops, I screwed that up, commit. And I say this does a nice job of showing that the activity really falls out towards EOL. You can definitely tell that F-21 has fewer commits than anything else. If you notice that F-24 there has a large, pretty large number of commits as well. That's because some of the time period included a time before a raw hide came out. So, this just serves to reemphasize that today's raw hide is tomorrow's stable release. So, testing and reporting from raw hide is very important for those who do test raw hide and feed bugs. Thank you once again. So, I'll add a little bit. I'm going to look at some bugzilla numbers but in a slightly different way. This is the number of commits that were actually tagged with bugzilla numbers. If you notice it has a slightly different curve. F-23 ended up with the most bugzilla tags over the entire course period. I think this kind of matches with what we saw in the state of Fedora this morning is that there's an adoption curve. So, F-23 was stable for the longest during this period. So, therefore it had the most users and therefore you're going to get the most bugs. If you notice some, there's definitely a fall-off though of bugs. As release is going to be of all, we definitely don't see a lot of activity there. I was on vacation the week before F-22 meant EOL. When it came back there were no bug reports for F-22, which was a nice change from the usual flood of stuff in my inbox. You'll also note that raw hide has a fewer, has a less number of bugs than anything. I attribute this to the fact that raw hide has a smaller user base and therefore you just get fewer people reporting bugs. So, this is the number of actual updates we did. This includes major updates going from example 4.5 to 4.6, also the stable updates so 4.5.1 to 4.5.2. This again shows the curve of fall-off. You have almost half of each of them. Again, the number for F-24 includes some of the work for raw hide. As you can see, raw hide definitely keeps us pretty busy and active for what we've interviewed. The EOL fall-off, I'd like to talk a little bit about what we did for F-22. Part of this release is going towards EOL. We have to figure out the right balance of pushing a new kernel update versus staying on an older release. So, about the time that F-22 was headed towards EOL, a long-term stable kernel was released at 4.4. This is a kernel that gets dedicated support and updates for a couple of years. We decided to keep F-22 on this long-term update. Honestly, I think it worked out pretty well. What we were finding is that it was taking longer and longer for the updates to actually get enough karma and go need to actually go out. So, instead of actually having to try and wait for karma for big update, we could just wait for a smaller update and therefore F-22 was able to remain the stability. So, I think evaluating when to do the updates as far as when things go EOL, this is something we're going to continue to do to look at when it's appropriate to rebase versus just keep it. So, this is the kernel versions that were covered during this time period. So, if we take these numbers to be typical for a Fedora release, then this means during the rough life cycle of a Fedora release, you'll probably get about four or so major kernel updates of or about a year. This is not too surprising if you look at things go. The upstream kernel gets released on a very, very regular basis about every 80 days. We aim to get new kernel releases out roughly by the second stable update that comes out, which means roughly two-ish weeks. So, for example, 4.7 was released a week or two ago. And I expect as I when I get back from Flock, one of the things I'm going to be working on is moving that into an update for F-24, F-23, et cetera, et cetera. So, that's what we want to try and do. And then we'll hide the numbers still up there. So, okay, summary what I just talked about. Some of this I already mentioned is that older releases have a fall off. Not surprisingly, there are fewer people reporting bugs and therefore we're going to spend less time trying to get them updates. It's also worth noting is that we are pulling in bug fixes in addition to what's coming to stable. And I think this is a positive sign for us is that we are trying to be proactive and not just relying on stable updates, but also bring in to whatever people come in. Oftentimes, this may mean is that the patches we bring in may have a short life. They may just get replaced by the next day of all update. But the point is that we're trying to get the fixes in and new features in as soon as we can and out to people. Rawhide also keeps us pretty busy. I did the math and it really is getting about one committed working day on average. This is, I think it tends to be heaviest during the two week merge window for kernel, but that Rawhide really is a big portion of our work. And finally, your release is going to get several different kernel versions during this life cycle. Fedor is a fast-moving distribution and I think this is a real opportunity to get a chance to make sure Fedor is working with new hardware and also get a chance to report more bugs. So it has pluses and minuses, but I think it's great to work with. So what have we learned over this past year? I think we've come to internalize some lessons. First, people don't like bugs. This really sounds obvious. People want their hardware to work and when it doesn't, they get grumpy. I'm the same way when my hardware doesn't work. I get pretty grumpy. But I think we've discovered the only thing people hate more than bugs is surprise bugs. When something used to work and it didn't, then it stops working, it gets, it's not a great experience. I think the situation we ran into with some of the Dell XPS sound hardware is a good example of this. So the software support for the Dell XPS sound hardware was on the bleeding edge and the most reliable ways we found to be able to work involved kind of a hack of the kernel config option that had some conflicts with secure boot. We turned it on because we wanted to try and get the hardware working. But then eventually I decided I wanted to try and turn it off so that we could see if we can get it working properly and make sure that we could actually support it properly. People really were unhappy when I pushed the update and things broke. And I think the biggest mistake here was not realizing that people would not be happy by this. And even if it's possible to later fix it with another way, the experience of having something break is really not great. So planning is key and I'll be talking a little bit more in the future about what we're going to try and do to prevent things like this. So as far as where bugs are coming from, most of them are coming from the upstream project, not from Fedora itself. I say this is good because, well, we're not creating any bugs, but it's bad because it also means that it's a lot harder to figure out where the bugs are trying to come from. Several million lines of code and through people plus whoever else, all the other people I talked about who are contributing, it's a lot to manage. We really try and report the bugs upstream where we can. We encourage you to report bugs upstream as well. The process isn't perfect though. Sometimes we get bug reports that for various reasons we can't do anything with. Sometimes we report bugs to maintainers upstream and maintainers don't respond. And it really is frustrating just because I think one thing at least I've realized is that I like fixing bugs. I like being able to close bugs with a patch because I know it's helping to overall my pain is better. So the kernel is a big project that we really try and do the best we can in terms of fixing bugs and getting things out there. Finally, we really want to try and get more people involved. We love to see people who try and propose fixes that they either try and create themselves or have found. The kernel is a project for everyone. So there seems to be this myth with the kernel that you have to be X amount of awesome before you can contribute to the kernel. This is really not true. If you want to contribute to the kernel, you have argument the awesomeness level. Again, not every contribution will be accepted, but we want this to be a place where you can learn and have a discussion about what you try and submit. If you have any feedback about ways we can help make this better for you to contribute, I'd love to hear it. Goals. So this is some of the stuff that we had for goals for last year. So last year we talked about doing some work on power management. I said I did some work related to power management. I produced some graphs, but I think ultimately we concluded that just saying make power management better in the kernel, it's not a great goal. It doesn't have a clear deadline. And I think what we decided is that we certainly care about power management, but what we're looking for is problems that have been identified as being specific to the kernel and not say caused by this behaving use-based processes and also ones that can actually be fixed without support from the hardware vendor just because sometimes power management problems can be like to hardware vendors having a specific magic bed or making choices in their firmware that we can't actually fix. So again, we care about power management, but this isn't going to be one of our top priorities this year unless we come up with a specific goal. Automation and testing. So this has really I think been a high point for us. A lot of this work has been done by Justin Forbes, one of the members of our team here. He's done a lot of work to make sure we're getting regular builds and every time we push a build, an automated set of kernel tests is running. And he's also working on a regression test framework. So I talked about how people don't like surprise bugs. One of the areas where we tend to have a lot of surprise bugs is third party modules, NVIDIA modules, VMware. These are modules we can't officially support, but people still want to run them. And inevitably, when they break, people come to us and then we have to have a back and forth about saying, that's a third party module, we don't support it and then close the bug. And a lot of times people are even if we can't support it, people are still not happy that something they were using broke. So the hope is is that we can create a dashboard and do preemptive testing about the third party modules so that people will at least be aware of what's going on. And for example, if they see that the next kernel update is going to break their virtual box drivers, they may decide to hold off on updating the kernel until that can be fixed. And perhaps even if they see that something's broken, they can report it to the actual third party maintainers and maybe even get it fixed faster. So the hope is that overall, this gives more information out to everyone to be able to improve their experience and improve our experience with booksella. And again, that's all on Justin. And upstream. So we always have a goal about working with the upstream kernel community. The kernel community is a very individual relationship based community. You have much more success in the kernel if people know who you are because you're doing good work. The supplies even to people who are working at companies like red hat, the maintainers say repeatedly, we don't care what company you work for, we don't care who you are. So it's really the Fedora project's best interest to have people who are actively involved in the kernel community contributing. And I'd say on each of us on the team has our own areas we like to work at. So I have a lot of areas. I work on a few things related to arm and mobile, sometimes pocket memory management, sometimes do a little bit of security work. If you're interested in any of these areas and would like to know more, I'm happy to talk to you about it afterwards or over a year. And the kernel community also really likes it when we can submit bug fixes and by section results upstream. So this is why it's also in our interest to make sure everyone is reporting bugs where they can upstream. It helps to overall grow everything up there and the connections there. This is a goal that's never actually going to be finished. I don't think we're going to be say we're done with upstream for this year. Yay. But it's important to describe why we think this is important to keep saying doing it every year. And it's overall because we want to make sure that by working upstream, we're giving Fedora the best experience possible. Um, so this is some stuff that may have been hotter last year and is maybe less so this year. Last year, there was a lot of talk about 32 bit being in the decline. And say this year, this is still the case. And even the general community, there's been a little bit more discussion about 32 bit bugs and how to find and test them because fewer people are actually testing and finding them. An example to highlight this is that earlier in the year, there was an issue where the 32 bit live CD wouldn't actually boot up. It was bouncing around between the kernel boot loaders. It was sort of sold out for a long time. And eventually, we had a few members of the community step in to help to do some testing and bring it to resolution. And I think this sort of goes this continues to go with our strategy in the kernel of keeping 32 bit issues low priority and letting the community help out with it. So if you care about 32 bit issues, please make sure you're testing and reporting them directly upstream and letting us know when you find the bugs. Again, if you find it, if you have if you if you found patches that fix your problem, those are really easy for us to turn in. So again, thank you to those who do try and keep 32 bit alive for doing the hard work there. Butterfists. So butterfists seem to be a really hot topic last year. And this year, not so much. We haven't got a lot of questions about it. I'd say your opinions are still the same as last year. Butterfists is is a solid file system that is not the right default choice for Fedora. Those who have done their own research can decide to use it for their file systems, but we don't think it's the right choice right now. We continue to monitor the situation. And if something changes, we'd be happy to let you know. Generally, so for the question is, why do we think it's not right choice? It's, they're still finding bugs that can possibly affect production servers, I think. And the real thing is that for most Fedora users are individual users who don't have a lot of data duplication, where it's being used as in big companies like say Facebook, where they have a lot more support to be able to deal with, I'd say, possible file system issues, whereas an individual user, you may just end up being out of lock and you've lost your data. So it's, it's, it's certainly an option. But I think for the community, Fedora is going for it's not the right choice right now. It's the default yet. So that's he's not gonna do that anymore. This past time last year, he said he was telling us no, and he's not interested in this conversation anymore. So please stop asking. For the recording or camera, basically, the butterfist maintainers have actually come back to us and they have actually said, you know, please don't make it the default as well. So that's another contribute big contributing factor. So for why it's not the default. So kd bus. Okay, so this is actually had some relatively new updates. But kd bus was an internal IPC mechanism, trying to bring the bus, the bus protocol into the kernel. The developers pushed it. And it was not very well received. There was a lot of back and forth. People concluded that there were some security and performance issues. And ultimately, the kd bus developers decided to withdraw the patches and we dropped them from our hide. And then we basically didn't hear anything about it for a long time. And then this past week, there was some planning for kernel summits in November, where all the kernel developers get together. And then one of the kd bus developers has announced they want to have a discussion about their new project called bus one. So they're clearly out there and being active. But I think they're trying to do things a little bit more transparently and actually work with the community to get things out there. As far as what it means for fedora. I think the plan is just to take a wait and see approach. If they come to us and say we'd like you to bring this into a high like you did for kd bus for testing, we'll certainly be willing to have a discussion about whether it's right. But until now, we're just going to keep them be active and review exactly what's out there. Okay, so that was last year. So then what are we going to do this year? As mentioned before, we want to make sure we're having good surprises, not bad surprises. We want to make sure that it's great when your hardware work and less when your hardware doesn't work. So again, I talked about the third party module, regression framework that's definitely on our goals. We also want to try and increase the variety of hardware. We're testing out to try and find bugs. Hopefully we want to try and find bugs before all human community find things. We also want to try and increase our communication. Being kernel developers, our work is very focused and a little bit more insular in nature. Everyone depends on the kernel and that the kernel is needed to run. But because of our way things work is that we don't end up communicating with a lot of people. It's very easy for us to get absorbed in our own world and not actually share what we're doing. So we want to fix this. We want to find new ways to be able to share what we're working on and why we're excited to be kernel developers. All of us have blogs, sometimes we write about things. On my blog I write about some things related to Fedora, sometimes I just complain. But the real thing is that sometimes it's hard to know exactly what to write about. So if you have things in the kernel you'd like to see, please let us know and we can possibly turn them into something for that. We're also going to try and work to improve that tooling. So we're a small team of full-time kernel developers and a small team doesn't scale to a growing community. But what does scale is better tools to be able to make dealing with kernel easier. A good example is my bisection script. So I've had a project about having a set of Python scripts to be able to make bisecting the kernel easier. And the people that have used it have found it really useful. I got another one today with some feedback I need to look at. But the point is that it's tools that make it so that the community can solve their problems easier without having to do as much back and forth as much through bugzilla. So this summer we've had an intern through the outreach program, Miguel, who's been working on trying to help improve some of the scripting that we've had floating around the script. He's been doing a great job and my hope is that by the end of the summer there'll be some scripts that will be useful for things like building the kernel and applying patches. None of this is saying is that we don't want you to interact with us, but it's really just to help reduce the time to solve your problems and make it easier to reduce the back and forth and get to actually solving the issues. And again, if you have suggestions for toolings you could make your life easier for the kernel, I'd love to hear it. Finally, once again, we really do want to get more people involved. You've heard me mention this several times by now, but I really want to emphasize it just because the kernel gets a bad reputation for being difficult to work with and we really want to work to change that. Generally when you hear about the kernel is that you generally hear upstream first, upstream first, and it's true we do want people to interact with the upstream, but realistically the first experience most users want to have with the kernel is probably going to be their distro, so therefore we want to make sure we're having the right environment. So we want to expand on what we've done for outreach to help make it easier for new contributors. So come try things out with us, ask us questions. We really want this to be a positive environment where you can learn and be able to grow your own knowledge and help us as well. Again, we are here for for the community. We are here to help the community, so participation and helping you as part of our job. Let's see. Finally, that's a link on the Fedora Project wiki for some more of our yearly goals and details if you have any more questions. And I think that's it. So questions? Yes? So back in your site, when you went through the number of commits that RAHI has, I think over the course of that year you had like 500, right? Yeah. But I just wanted to point out that, yes, that's 500 commits to the Fedora package repository, but it actually each one of those commits represents a multitude of commits. So like when we do rebases in RAHI, like right now the 48 parts with those open, every single one of those commits can map to like 2,000 commits. It gives it a little bit more impact when you talk exactly what those commits represent. And it goes down from there, but even some of the stable kernel updates, the numbers seem small. Like, why was the panel, my panel seems really small, but actually in reality they're very large teams. That's a good point. Other questions? Yes? Yeah, I was just confused because I've been to these talks before and I've been playing out convention. It goes on in kernel and one of the things was we were always told upstream, upstream, upstream, in fact that nothing could even get into the Fedora kernel unless it had made it into at least been pulled into a point release. Upstream. And so I sort of took that as don't bother us go upstream and work with your issue and then wait till it comes downstream so that or wait till we know it's at least going upstream and then talk to the Fedora kernel. There was a long time ago. So yeah, that's a good point and I think that's true, that's what we want to happen, but at the same time it can be, the kernel community can be intimidating to work with. So what I'd like to basically is that if there are problems, if there are ways that we can make it easier for you to contribute upstream then we'd like to help with that and if that means occasionally looking at a patch then we can see if that. So basically we want to try and be the resource to let you be able to get stuff upstream. So it's basically we want to be the guiding hand, so. It's the kinder, gentler kernel. And then you said that if you wanted to be more communicative, what is the proper way, because I don't know if it's a core kernel, unless there's really get much discussion. We can. There's the IRC channel that it's very little, unless you're occasional, not often. So I mean either of those, there's no reason why there can't be more discussion on those mailings I think or IRC. I think it's they've been quiet just because people haven't been participating. There's no reason to say is that we can't have discussions there. I think we might, I think probably what I'd like to see happen is have some discussions there and then perhaps is that if it's to your point we can say okay this has been a good discussion, now can we perhaps propose this upstream. So the idea is that we can be an incubator for ideas maybe for upstream and then, but eventually be able to get them to the kernel community. So if you do our kernel reports on a monthly basis and it's fine, it took like a day to pull out a new together and then send it out, but there'd be no feedback. So like am I talking to myself on this mailing list? So the mailing list itself is really hard to know whether people are even reading it. That's true. And I'm not asking people to reply to things. This is awesome things every single time, but it's just good to know that if people found that valuable then that's always something you can get. I mean everybody loves the kernel until they go problem with it. So well that's what I'm saying, nobody reads the change log on the kernel until something has gone wrong and then they try to say ah maybe it's this item, the change log. I used to write those messages, but I did not notice when they stopped. Exactly. Yeah. And I mean I'd also add is that we say we want people to contribute. I don't think we're still working on figuring out ways to do this. So if you have suggestions again we'd like to hear this. But the point is that we want to try and be we want to try and be interacting with our not just be in our own little kernel world huddled up in a blackhead pack in a laptop. So that's a that's a really good point. And this is I think one of the valuable things I've gotten out of mentoring with outreach is that I figured out I have a better idea about how exactly to help new people get started and everything there. And I think part of this is also figuring out what are tasks that are good for new people as well. So but that's a good suggestion. I'll I think I'll think if I can find some candidates for that. So yes. I'm just talking about the fact that one from each two is a lot of people can do things around it. So obviously you pick up individually every corner to an upstream. Yeah. So for a while we do a snapshot which is where we take whatever is of the master at that time and then pull it in as a patch and then rebase whatever else we're carrying on on top of that. For stable updates again we're pulling in what comes in as a patch and bring that in is that. And then the idea is is that each foot or commit is going to be correspond to a patch or get I'm sorry. Does that answer your question or it's like every rawhide every day rawhide is a little like stable releases. We wait much longer right. So like Fedora 24 is on four six five right now. I don't know four six probably six before it before it goes to four seven. And then when we go to four seven it's the whole four seven at one time. It's like one big jump. But rawhide is like a little time and time repeat. So at least for what we have at least in our sources there's generally the last stable update then an RC patch and then a get patch on top of that for rawhide. And then for stable releases we have the release base four dot whatever kernel and then the patch the dot x on top of that. So you have you're having a patches coming in in different parts. So yeah. Is there any documentation on how you guys deal with work with the kernel? Because here's my issue. Say I want a patch. What do I normally do? Well I can inspect through the patch inspect build it in Koji. Wait hours and and my mill machine is fast. Then install the package which I know is dumb because there's a far easier way to do it. But I don't want to get to the point where I don't you know anything broke. Not confident to get into you know boot. I know there must be a simpler way to build the install because otherwise you guys couldn't wait for three hours for Koji to build it in change. But yeah I don't want to go down to the raw kernel source you know. I like dealing with the package you guys produce with spec file. So as far as what you're saying. Yeah we do actually wait for at least on my machine when I'm doing local builds. I will be very patient and wait for it to build. There's also there's no a fast build script which makes things a little bit better if you want to test things locally. But but I think to your that's your specific question the more general question about what we do. We've tried to keep the wiki up to date with steps and things like that. So if you take around the through our project Colonel there's the main page if you realize it was up to date. At least a lot of times you learned to sort of like okay there's a wiki page but because of certain 2008 helping so if it's up to date. So when Laura joined us a little over a year ago one of her first tasks was to update all that stuff that we developed over eight years. It was mostly because I was the first really new person in a long time. So therefore we were discovering exactly what actually was not documented. So I tried to do that to keep that up to date. I think most of the steps right now are recently up to date. If you find things that aren't you're welcome to create some discussion on the mailing list. So if you know the right plugins for media wiki I have a dream of having one in our media wiki which is older looking with like curling and stains and cobwebs on them as they are untied. See the next thing is a terrible thing though. It is awesome for this purpose. That's a bad motivation though because what if you think like the older stuff looks cooler like you've got spiderwebs. That's awesome. It's like God documentation. So back to your point there is no magic bullet. If you like the packaging and have management aspects of it then you have to wait for there at the end of it. And like when I was doing raw factory bases what I would do was just a local mock build and it would take roughly 45 minutes and then I would use the ansible to deploy it after a bunch. There are faster ways like you don't actually have to edit the grub file like if you do fake install it will do that or you if you have to write back and just install which is the result that you follow from the raw. But then get rid of the spread. Exactly. The problem is you have to go in a manually and clean them up. Right. So. Because when you do in a bisect it's just not so. Yeah, bisect you have to do a manual. Obviously. I mean Fedora's or Laura's scripts make it so that you don't but you still have to wait and build in a bisect from right to the front. Yes. So generally when people could be with bisection I generally try and encourage them to figure out what are the last two built in Koji that actually worked and then say okay once you've had those two tags then you can start going down to the nitty gritty and trying to figure things out. There actually is a I wrote a script a point two years ago called Koji bisect to keep meaning in kernels and it will download and install them. Oh. The problem with that script is that nobody uses it so I don't like it so much. No. It's there though. I mean if it doesn't work with you fix it. Sure. You mentioned file systems you said BTRFS is still not right. Yeah. There are other file systems. Does anybody look at them to see if they're getting them to be in fedora not obviously somebody looks at them like to be cashed God. I don't know what the heck he's doing but he keeps going up with this thing and his benchmark numbers are neat but you know I don't even know how to test that kind of thing. Yeah. I think it's part of it as well is that I don't think any of us really have done a lot of work with file systems so therefore I think all of this was on our radar but we don't know exactly we haven't really thought to try and push for it or bring it to our attention. So again this may be a place where we to start discussion on the mailing lists about exactly what we have defined. I mean yeah. BTRFS and Facebook Facebook does not use BTRFS for data they only use the word file systems so yeah. So if you actually talk to some of the red hat file systems developers right and the guys actually do that well get a channel and yeah those guys if you talk to them they will tell you that in order to become production ready file system takes 10 years of development right. So BTRFS is nowhere near that yet. It's about half of their name. Like you said Facebook is using for different purposes yeah. What Fedora would use and Facebook employees all almost all of the BTRFS developers and there's some guys from SUE said that actually would be quite a bit to it and they've done some fantastic things because they use it as less for certain estimates but they also cheated. In a smart way I mean I'm not that's not derogatory but they cheated they turned off a lot of the interesting features right. Okay. So you can you can run it and you can do things like snapshots but you can't do like on-flight compression and you can't do like data migration could do otherwise if Fedora. Now you can do it Fedora but you run the rest including your data so it's it's a little bit different world. The interesting thing in my opinion is that whenever somebody asks you why we are using BTRFS I always ask them why do you want and they never actually talk about like the file systems itself they talk about wanting snapshots they talk about wanting moving stuff around from machine to machine that ease the new speech from trade. And when you talk to the real kernel guys that they have an answer for it but it's not as useful right. It's always take XFS and put it on top of device mapper and LVN I don't know what to do with that and that's fantastic for enterprise users the door users don't like those things not all the time anyway so it's it's mostly coming up with maybe a tool that mirrors what BTRFS product does and makes it easy to use most of the technology doesn't improve right. Because people don't really care about which files isn't they're running they care about the uses of it. I think we're just about actually I think so. Can I strongly suggest that you ready to go on magazine or career to do our goal about ways to interact with the federal which makes you say it's a bit of a surprise. Yeah so I think that's a good suggestion I was planning on possibly turning some of this into a thorough magazine article so yeah. Yeah. I have no idea I have to take a look at the people who are the bads on the top of my list. Yeah. Yeah. You know at the end of the last block I tried to do Colonel Buggsville and Riyosh for one day. I got three Buggsville so it's it's impossible. Yeah. You know there's nobody gets enough information and they don't know what information will be. Yeah I think we generally kind of accepted that Buggsville is going to exist as Buggsville. Yeah. There's just you know I don't know how to deal with that. So you don't know we're going to Nice talk. Thank you. I posted a picture let me know if it's like is it okay? That's fine. Thank you. All right. See you later. Thank you. No I don't know why you would but Oh really? This is the only guy here. No. No I don't know why I should put it in. The other room I was just there was just kind of the scissor ball. Okay. Is this the is this the council or right. You're going to build a computer, you're going to build a computer, you're going to build a computer, you're going to build a computer, you're going to build a computer. You're going to build a computer, you're going to build a computer. You're going to build a computer, you're going to build a computer. You're going to build a computer, you're going to build a computer, you're going to build a computer. You're going to build a computer, you're going to build a computer. You're going to build a computer, you're going to build a computer.