 Can everybody see this? I don't know if the angle is too good or the print. Okay, I'm going to talk to you guys today about rawhide. How many folks here know what rawhide is? Pretty much everyone, okay. How many people are running rawhide every day? Quite a few folks, good. So a little background about me. I've been running rawhide full time on my laptop, which I use for everything, day to day. For about three years now, there have of course been issues, but nothing particularly bad. There have been, in the past, in the distant past of rawhide, there have been like the problems that everybody thinks about for development releases. You know, like G-Lib C getting messed up and your machine will go under boot or those sort of things. But none of that has really happened in the last few years. Things have been a lot more stable due to a number of reasons which I'll get into. I'm assisted in by trade and I've been involved in Fedora for a really long time. I maintain a bunch of packages in package collection as well. A few little notes about rawhide. It's the always rolling book release. New versions of things can appear at any time. So when you expect them or when you don't expect them, suddenly, boom, there's a new version of something. Major versions can appear at any time. Minor versions appear all the time. So occasionally, or I'd say the vast majority of the time, things update and you don't actually see that much user-visible change. You know, the package is updated, but it updated from 2.0 to 2.01 and it was just a doc file change or something like that. Unless you're looking at it closely, you don't really even notice as much what's going on. It's not a 100% sign, but we're working on that and actually I'll have some slides on that later, what the progress on that is. It's now a full compose of everything. It used to be just the packages tree and a few other things, but now it is a full compose. So it's like all the images, all the trees, the server, the workstation, the net install, cloud images, you name it. Everything that a release of Fedora would have gets built every single day for rawhide. OpenQA is now testing all of those things and so there's a lot more visibility when things are broken. OpenQA, I believe that talk was earlier today, but there was a talk on OpenQA here and it's basically an automated system where it takes the images and it does the things that a human would do. So it boots it up, it chooses install, it runs the install, it clicks on certain things, it looks for certain screens and you could see the results of that testing day by day. That's been really good. So I pulled this off of the wiki. There's a number of goals for rawhide. Basically it's a place for people to build the new upstream packages and integrate them into Fedora so they can see that it builds with our compiler or the compiler version there, the libraries kind of all match up, things like that. It's also a good way for advanced users to get the newest usable packages. And it's also a place to do incremental change that are too minor for stable releases. Like as I was saying before, if you have a new version of a package that changes some documentation or it's some very minor bug fix or something like that, it may not be worth pushing out to the stable Fedora releases, but you can push it into rawhide and people can see that and test it and make sure that bug fix worked if they're interested in that particular thing. And it's another goal is to fix issues with packages before they get to the stable releases. So those minor releases or even the major releases, that gives them time to be actually used and tested. So a little bit about how rawhide is made. There is a Cron script that runs every day at 5.15 UTC. If you look on pagair.io, Fedora Pungi is the project. And that actually has a script that goes through the whole process, calling all the things that it calls, creates all the trees, creates all the images. And it usually takes about four or five hours to do all that. It could probably be optimized into deal. There's a lot of stuff where it doesn't do things in parallel where perhaps it could do things in parallel. But there's also a lot of steps that it has to do in a certain order, like it has to build the trees before it builds the images from those trees and that sort of thing. But you can see all the nitty gritty details in that script if you're interested. There's also, I didn't put it up here, but there's a mailing list called Relinch Cron, which just contains the output of these sort of jobs. And if you're interested in it for whatever reason, you can go and look at those. A lot of the Relinch people are subscribed, but don't look at it unless there's some problem. If you want to go look and see what happened, you can see the output from everything and you can see where problems are and so forth. And some of the output is large, so be aware. So why run in raw hide? You like getting new software every day? That's one reason. You like troubleshooting problems? Seeing problems get fixed? There's a really good short feedback loop a lot of times with raw hide stuff, I found at least. So you run into some problem, you'll be like, where is this problem? I'll troubleshoot it down. Oh, it's in this component. I'll file a bug. I'll talk to that maintainer and they'll like, oh, yes, I see the bug. I fixed the bug. And then the next day you have a fixed version of this thing. So the release cycle, feedback cycle there is a lot nicer than, say, a Fedora release where it might be a lot longer before they are able to reproduce the bug or it might be a lot longer before a package gets to a point where you can test it or that sort of thing. Of course, if you have a challenge, everyone does. And of course it helps out people that are using stable releases. So you find bugs before those people do. A lot of packages kind of update in stable releases and raw hide all at once. If it's say it's a security release for some package, they provide updates for everything. But the raw hide one almost always lands before everything else in the stable releases. So I've seen numerous times where maintainer will push updates for all the releases. The raw hide one will come out. I'll find a problem with that one, file a bug, and then they'll redo all the stable ones. So it's definitely a good way to find bugs before everybody else does. We are missing one point. I hate updating updates between stable versions. I'm using raw hide and I don't have to update. You know, I will not agree with you. Basically, update from one stable release to another stable release looks the same as update to raw hide from a week ago to carry. You flashed 3 gigabytes of packages and replaced the whole system. I've got some slides here in a little while that I'll talk about the rate of change. It's interesting. Yes, here we go. So I looked at some stats from the last month and there was one day where there were only 18 packages updated. That was the slowest day. It was a Saturday, I think during a holiday weekend. It was like right before a holiday weekend. And then there was another day when there was about 2,500 packages that changed. No, that was actually, this was a Python got rebuilt. Right, an inside tag and that's when it was folded back into the main raw hide tag. RPM got a new feature to handle dependencies on Python packages. And in order to take advantage of that, the Python maintainers wanted all the packages that are Python to be rebuilt. So the way that's usually done is, you know, they build a new RPM and they send it in a side tag in Koji. And then all the things build against that and then they're merged back in all at once. Now, that seems like a lot, but that actually isn't a lot. The biggest amount that you'll see a raw hide change is right after a mass rebuild of everything. And, you know, those days that you're talking, you know, 22 to 23,000 packages, basically every package on your machine is replaced virtually. So the amount of change is high sometimes. I checked also into the manager requests, which I thought was interesting. That's the number of people hitting mirror manager and asking for a raw hide mirror. And it varies quite a lot. It's between 12 and 25,000 a day, which actually seems more than I was expecting. But again, that's going to just tell you how many machines hit that were requested that mirror link. You know, that could be some machines hitting it very often, or a bunch of machines behind a single IP address hitting it a bunch of times. Or developers requesting information. Right, exactly. Anything that goes to that metal link. This isn't like unique, unified or anything like that. Mr. Bill, raw hide was first announced in August of 1998, which was about 18 years ago now. Yes, it can. Or those using it can, if need be. So getting on the raw hide trail, this actually turns out to be the more difficult part. Once you actually have a raw hide install and you're just updating day to day, really in my experience, there are very seldom are there showstopper problems where your machine is just toast or not functional. But getting that initial install is sometimes very challenging because the install path is very fragile. So we have open QA testing it now, but if anyone has looked at this recently, and I guess I could maybe try and pull it up. I'm not sure if I'm on the net. But it's been broken for the last three weeks or something like that due to installer changes and then broken dependencies in LibreOffice and things like that. And if you actually had a raw hide install already and you were just updating, DNF would not give you those broken dependencies. You'd just say, okay, and keep going on. But if you're trying to get an actual install, that can be a difficult problem at some point. Adam Williamson has created a page that uses the open QA results. If you go to this nightly.fudoreproject.org, it redirects to Adam's page right now. And it will actually tell you the last raw hide images that passed open QA and are installable. And they're quite a ways, quite a while ago at this point. So it may be more productive to just use DNF and install a stable release and then use a DNF update. You may run into broken dependencies on that, but... Yeah, exactly. Mixing raw hide and stable, the questions come up a lot where people are like, well, I'm running Fedora 23, but I want LibreOffice 5.1. Can I just install it from raw hide and go on my way? No, I think it's a pretty bad idea. In most cases, the exception being the kernel, since you can have multiple versions of the kernel installed, you can install the raw hide kernel, boot on it, see if it works for you. If it doesn't, you can just boot on a previous kernel and you're fine. If you're installing some large application from raw hide, you're going to pull in glibc, rpm, and basically you're essentially moving to raw hide whether you want to or not. It's very difficult to untangle the dependencies if you want to try and do that, so I'd recommend against that. Since there's full images, when they are working, when the install path is working, there's all of the spins, all of the labs, all of the net install, so whatever method you normally would use to install should function. If you are foolish enough to want off of the raw hide trail after you've been on it for a while, some good points or good places to do that are the branching points. We just did that for Fedora 25 a couple of weeks ago, and when that happens, when Fedora 25 branches off of raw hide, there's new Fedora repos packages that go out with the new definition for Fedora 25, and if you want off raw hide at that point, you just tell it, you enable the Fedora repo and the Fedora updates repo and the Fedora updates testing repo and disable raw hide and then you go on down that path. There's a number of folks I think that do that sort of thing. After a release, they go to raw hide, they stay on raw hide until the branch and then they follow the branch until release and then repeat because they want to be involved in testing the next release. Particularly, I think the desktop folks do that, some of the QA folks also do that. So it's certainly an option. Distrosync also works, DNF Distrosync, if you want to downgrade to a different version or sidegrade as the case may be. So the raw hide repo definitions are in a sub package of Fedora repos called Fedora repos raw hide and it is not installed by default. So if you want to update or sidegrade or distrosync or whatnot over to raw hide, you'll have to install that package. That was changed a while back because inadvertently users were, it used to be installed by default and users were going in and saying, I want all the software so I'm just going to enable all of these things, Fedora, source, raw hide, debug info, I'll just turn that all on and update and of course they would get raw hide and they'd get confused and so that got changed a while back. Obviously reinstall is another option if you want to get off the raw hide trail, just reinstall with a stable release. Nice. A note about hardware. In general, things that work in a stable Fedora release are going to work fine in raw hide. Sometimes raw hide has support that you need. I don't know how many of you here have Skylake laptops? That's the more recent Intel chipset. There were a number of problems because Intel released the laptops with that chipset and the support for it was not available in Fedora 23. And this was before Fedora 24 came out. So in that sort of window, raw hide actually turned out to be a lot better than the stable release just simply because it had the Skylake support in the kernel and the X drivers and things like that. So sometimes you'll see that kind of situation. If you use any of the non-free video drivers like NVIDIA or ATI Catalyst or any of those sort of things, raw hide is probably not a good choice. Usually there's a kernel every day, just about, which means you would be rebuilding your modules for those about every day. If they rebuild. If they rebuild. And they're often very laggy behind versions even in stable releases. So much less raw hide, they're often broken and they're not going to come out with a new release for a while. I don't know if it's still true, but it used to be that the NVIDIA driver wouldn't work at all because raw hide kernels have debug, a bunch of debug options enabled. And it turned out there was a problem with their driver and certain debug options where it just wouldn't work at all. So you couldn't even run it with the raw hide kernel at all. So then we are using the default kernel or with the debug information? I am using the default kernel. And it does have debugging, but it doesn't seem to really bother me that much. I guess I should have thrown in a thing about the kernel versions. So the default version in raw hide has a bunch of debug info. It used to have a thing called slub debug enabled and that was that really slowed things down bad. So they actually got rid of that. It wasn't actually that helpful. So they turned that back off. Also, the first release candidate of every kernel has no debug enabled. So like if 4.70 RC1 comes out, the first kernel build in raw hide will have no debugging. I mean, I hope it's the stable version. If there is like 4.70 more, there is no debugging. Right, correct. It's the same way. So it's like the first RC1 one will have no debugging, but they'll turn it on again for successive RC1s, RC2 turn it off, etc. That gives you, if you want to run without the debugging, you can just stick to the first RC or the final. There is also another repository, kernel raw hide no debug, which does every version as no debugging. The problem with that is if you use secure boot, they're not signed. So you would have to turn off secure boot and they tend to lag behind. The builds are done in Koji for raw hide and then later in the day they're done as a scratch build for that no debug repository. So they tend to be like a day behind if that matters. But I don't find the debugging to be that big a deal anymore. It is a little noticeable like when you're booting you can see it takes a little longer, but day to day it doesn't seem to matter that much. Alright, run free. Ah yes, if you have old hardware you may have problems just because as things are disabled upstream or whatnot they will be disabled first in raw hide. So if you're depending on a really, well we're still building 32 bit Intel stuff at least for the moment, but what's a good example of this? You still use Matrox graphic card, it's time to boot. And you probably don't want the slowness from old hardware when you're dealing with updating lots of packages and rebooting a lot and so forth. Alright, so a few tools everyone here hopefully is familiar with DNF now. This is of course your first go to tool for handling raw hide day to day. It does not, but you can sort of do it yourself manually. So basically what I do and this will be mentioned in a later slide for workflow, but I usually update once a day and just pull in all the new stuff. And usually it's pretty quick when you notice a problem in any kind of application that you use a lot. Like an email client or a browser or something like that. And you know one of the first things you can go to there is, oh well Firefox isn't starting or something like that. And there was a Firefox update and the ones that I just did, I'll just downgrade it. Does that one work? Oh it does, okay well now I know that it's that particular update. If you don't update every day then you run into problems where you have to go further back because there may have been several versions between the one you had and the one you updated to. There may have been several versions. You may have to use DNF and say install this Firefox version. Okay does it work? Yep, how about this one and bisect it that way. Usually you don't have to worry about broken depths if you don't care. You can just do your update. It'll tell you oh I'm not installing these for broken dependencies. You could report a bug on it if you want or talk to the maintainer. But it's not going to install those packages of broken dependencies. It's just going to ignore them. Every once in a while you run into a problem where there's some package that you have that maybe you didn't get from Rohide. You got from a third party or something like that and it's holding back a huge bunch of other updates. For those cases you can use DNF dash dash best and it'll tell you more information. It'll tell you what thing this particular package needs to point to where your problem is. And then you can remove that package and update things and rebuild it or get a newer version of that package or whatnot. Can it happen usually when you use a VM version? Right, they tend to lag behind. So if you have a good example is FFmpeg which every time it updates it changes its library SO name. So everything has to be built against it every time. So if there's something that FFmpeg depends on or if it updated, you'll have the slew of packages and if all of them didn't get rebuilt, just most of them it'll not. You'll have the broken dependency and the old one will be depending on the old FFmpeg. Usually that stuff sorts itself out pretty quickly so you can just either ignore it for a few days or try and track it down with a desk. Kevin, since you were speaking about downgrading, I remember that Jan was behaving better in this case because we've got the previous version in cache and DNF does not do that. I don't know if it's... You can... There's a... I mean, this is something that this was disabled but maybe it would be helpful if there can be the previous composites for one week somewhere around. How do you get the previous version somehow easier? I've actually... I had a request for this a long time ago and I've talked to a lot of the release engineers and I've not been able to convince them, mainly Dennis, but it would be nice if we kept the previous version always in the repository. Because then you could just DNF downgrade and it would always work because you'd have the two versions. But the problem with that is, say there's a horrible security bug in version X, you update to version Y and version X is still in the repository, so... I mean, it's still a question which I should decide. I know. If I know the security bug, I should be able to decide to use the insecure version because typically it... For me... It may not matter, right? Yeah. And another problem with it is that it increases the amount of space. It makes everything twice as big. But, you know, we may be able to implement something like that. Actually, I believe the next slide I've got something that mentions that. There's also basically DNF repo query unsatisfied. If anyone remembers Yum Check, that is the DNF equivalent of Yum Check. So it tells you of the packages you have installed, do any of them have broken dependencies? But you shouldn't run into that if you just follow the UNEX suggestions. So while the FFMPEG has, you know, not all dependencies have been refilled, it doesn't work? No, it still works. You still have the old version. It just doesn't upgrade you. So, you know, if you have FFMPEG, they updated that and package A that needs it. If you have package B and it hasn't updated, then you try a DNF update. It'll tell you FFMPEG is not updated because package B needs the old version. I mean, would you use the desktop best? Right. If that doesn't change anything, that just gets you better reporting. Oh, right. So if you do DNF update, it tells you FFMPEG was not updated or something very generic. If you use best, it'll say FFMPEG was not updated because package B needs blah, blah, blah version, which is not provided. So it gives you more indication of where the actual problem is. Should the best be defaulted then? They don't want to do that because they think it's too confusing, I guess. It would be a really bad thing if we made it like this before now. No, it's not. And I think it's good. Yeah, I mean, you can still see for a long time it actually, for a number of years, DNF didn't report any of the broken dependency stuff at all. So you would do a DNF update, it would say, I'm updating these five packages, done. And you'd be like, cool, I'm up to date. But it resolved that FFMPEG thing and just silently didn't tell you anything about it. So at least now it does give you like a little blurb. DNF is a perfect package to report one bar per week. And there is a long time open bar to even make the reporting of the broken dependency. Do you still have lots of issues when you're actually trying to find those radicals? Oh yes, I'm familiar with that bug, yeah. When you said like the mixing of stable and all that, which is kind of going back down, doesn't it resolve that? No, because, well, to some extent it might. The problem is any package that changes its other stuff that's not packaged. So say you update to the raw high version of PostgreSQL or something like that, and it changes the database format, you can't easily go back, or you update to Firefox and it changes your user preferences or whatever. But if it's not something like that, then yeah, you could just do a history undo, blah, blah, blah. But provided you had the repository enabled, you need to have all the packages still available to do that. But yeah, that is an option. And you can exclude or set particular versions of the DNF.conf. And I've had to do this a few times when there's some broken package. I file the bug, the maintainer knows, but it's not updated yet. I don't want to get it every time I update. So I'll just stick an exclude package version, blah, blah, blah, and it'll ignore it. Or yeah. And usually if you're doing that, it's very good to have the full version in there, because otherwise you'll forget that you have it in there and you'll never get any updates from that package again if you just put the package name. So if you put package name, dash, 1.0, whatever, it's only going to exclude the broken version and then it'll get you back on track after that. Okay, Koji, the command line. I don't know how many of you folks are package maintainers, but Koji, the command line client is actually very useful for non-package maintainers also, especially those folks that use Rawhide. So Rawhide is not 100% signed, but it's mostly signed. We have an auto sign process that sits there and tries to sign everything. But the problem is the compose starts when it starts and the auto signer is just signing as it goes. You can run into a case where builds land and are not signed and then the compose starts and then those unsigned packages go out. You can use the Koji command line tool to download signed packages if they're available for whatever it is that you're trying to get. I should have put a command line note in there on how to do that. But there's a Koji option to get a package signed by whatever key you wish if it is signed. Koji download build is very handy. If you've updated, Firefox broke, you want to go back to the previous day's version but obviously it's not in the repository anymore, you can use download build to get the previous version out of Koji easily instead of going to the web page or newer if you like it. And this one is interesting. It's something I do which maybe other people don't want to but I find it handy. I tend to update once a day after in the morning, get all the rawhide stuff from the previous day but if I'm going to update all this stuff and reboot I would like to get the day's kernel since they're usually about daily. But a lot of times in the mornings when I'm doing my update the kernel is still building so I could wait until the next day and get that kernel. Or you can go into Koji and actually grab it as long as you're not using ARM v7 because that's usually the last architecture to finish. You can grab the... This would change. It will, very soon actually. But in any case you can grab things. Again the Firefox example, Firefox is broken. You downgrade it at works and you go okay, well is there a newer version in Koji? Well there might be a newer version that's already fixed. You just need to grab that version instead of downgrading. So upgrade. So do you have like an automated strict grids? No, I just go in there and look. You really want the kernel right? Well, if I'm going to reboot anyway I might as well have the newest one. Very handy for getting older working packages. Sometimes there's scratch builds available to test things. Maintainers will tell you to test the scratch build and see if it fixes whatever. Sometimes I will do a scratch build. If I see a problem and I know what it is and I know how to fix it or I think I do I'll try and do a little scratch build to fix it and if it works then I'll tell the maintainer that's the right fix. So Koji is fairly handy. One thing about Koji. If you work with any secondary architecture then there is ARM Koji, PPC Koji and S390 Koji. Right. Which just do the same but coming to secondary architecture instances. Yep. So you know about SC Linux. I know a lot of folks look, if they run into a problem or whatnot they would look at SC Linux but I've noticed a lot of Rahe using folks don't tend to notice this stuff and there are SC Linux bugs that happen from time to time and so one of the first things you can do if you're debugging a problem, especially a boot problem or a desktop login problem or something like that is go to permissive mode and see if the problem persists. I can't count the number of times people pop up on IRC and say oh this is, I can't log in and Rahe has broken and I'm like what? It was working for me, alright, let's look at this. Oh yeah, you know, try permissive mode. These things tend to get fixed fairly quickly. The SC Linux policy maintainers are actually pretty on the ball but it's a really good debugging thing to try. The don't audit stuff, how many people here are familiar with don't audit rules? Yeah, they're kind of sneaky. So SC Linux logs when there are denials except not if it's a don't audit rule. So there are cases where you can say oh I don't think it's SC Linux because I didn't see any denials in my logs. Well I'll try it in permissive mode anyway just to see and there are those cases where it won't log anything but permissive mode will fix it. How do the don't audit rules get there? There's a list of them in the policy and you can turn it off. There's an SC module command that turns it off. Unfortunately then you get flooded with all kinds of irrelevant messages which is why they're don't audit rules. They're like things usually that shouldn't affect anything and just generate a lot of log noise. But there are definitely occasions I fit where things don't work. And another thing to note here is if you're having some problem and you set in force zero and you're like oh well the problem's still there well try rebooting also because someone ran into this just a week or two ago there was an SC Linux problem in the login process. SC Linux was a system D moved something SC Linux wasn't updated for that yet and when you would log in it would not give your user an a seat it wouldn't show your user as logged in and then you couldn't log in at a graphical thing. Just setting that permissive after the fact didn't fix it you would have to set it permissive and reboot impermissive in order to get that process to actually function. So lots of fun debugging. Of course there's abort and SC troubleshoot D good for simple stuff and if you run into something that crashes and you have an abort report where it can file the bug for you easily and then you can just downgrade to the previous working version well that's fixed. Gathering news about changes there's obviously all packages or most packages have a change log or a news file or some sort of get information as to what changed if you're trying to track down what package is responsible for some particular bug or breakage DNF logs are very important you can see what packages were updated sometimes it's you know you think oh Firefox isn't running so it's a bug in Firefox but then you downgrade it and it doesn't change anything so then you look at what was upgraded well this this this this well maybe that's related and it gives you a place to start the raw hide reports on the mailing lists are sometimes useful for tracking down win a particular change went out in raw hide when a particular version went out all those are good for looking for more information one thing that I use a lot I have a Firefox alias for it we have this packages application and as part of that it can give you a page with all the bugs for a particular package so I have a Firefox alias for this so I can just enter the name of the package and it will give me that bug page and I find that a really easy place to be able to go and just search for the terms of whatever bug you know Firefox crashes on start or you know GNOME high DPI support or whatever it is that I'm looking for it's it's kind of easy to see if there's a bug already filed by going there and a lot of times in raw hide if you're searching for whatever is causing your problem package maintainers will put bug numbers in the in the package changelog as they rebuild things and even if it sounds unrelated I found it to be very useful say again Firefox breaks and the Firefox changelog says fixed bug 1 2 3 4 5 okay so you go look at that bug even if it's unrelated it might be still and a lot of times it is related and you can actually just comment in that bug rather than opening a new one hey this broke Firefox it doesn't launch anymore because the previous version works so that's kind of a nice shortcut and that gets the maintainers attention more quickly I think mailing lists it's good to be subscribed to the QA list because there's a lot of discussion there if something big is broken you're going to see somebody mention it there there's also Fidora Development QA on IRC which folks are happy to just look as concussion below yeah no they don't like that right so here's the update cadence I do daily I redo it every time there's a kernel I do have a rescue media handy I haven't had to use it in probably a year and a half something like that or maybe even more than that it just doesn't happen that there's a problem that is that bad I'm running low on time I think so alternatives it's a good idea to have alternatives I being a crazy person switch between XFC and known desktops so I know how to use either one I can get my work done in either one and it's very handy to have some knowledge of another alternative there say GDM breaks and you can't log into known or XFC terminal breaks and you can't use that or whatever if you have these alternatives you can switch to that and debug whatever the problem is browser XFC basically is a good argument because it's rather a change it's true absolutely I have Firefox as a browser breaks every once in a while so it's good to have Epiphany installed or Midori or something like that that you can actually still browse if you need to alternatives also help you troubleshoot so you can see if a problem and Firefox happens in Epiphany or whatnot sometimes testing with a new user is a good idea you can create a new user account login is that see if the problem still persists if it does then it's system wide if it doesn't then it's something in your user's config just a quick on backups backups are as important as ever I do backups on my laptop nightly and I almost never restore anything so back up the system partition I do and occasionally it's useful to go look at old config file or something like that but it's just you want your data to be preserved a few notes on branch which we just branched Fedora 25 off raw hide and for the first part of its life it kind of lives like raw hide there's no updates every day and there's no update system we then enable BODY you can use BODY-D to download updates so if there's a particular update you're interested in it will download all the packages in it for you branch has the advantage of all the new stuff goes into updates testing and then only goes into the base repo so there you have actually a way to do a downgrade because the base repo has the old version and updates testing has the new version Fedora Easy Karma is good for providing updates oh yes a quick note even further on the edge if you want to be even further than raw hide on the edge Koji has a build route that it uses to build packages you can in fact enable this as a repository the problem with it is it's not multi-lib so if you use any 32-bit stuff it's not going to have that it's not signed and there could be problems where a maintainer builds a package you update to it and then they decide oh that package is broken and untag it and it goes away and you still have the newer version but that can be very handy to pull things from if you know that there's some fix I tend to use that for when things are tagged into to raw hide use that to update and see if there's any big problems and occasionally I've used it to look at important things like a new system d and see if it breaks anything that kind of stuff you can also grab stuff from Koji so in the future we're looking at gating raw hide and by gating we mean some steps so that we can know that it's all signed and be able to advertise that the install path we're talking about gating on openqa so openqa says hey this all passed then we can push that out if it says this didn't pass then we just don't advertise those images we keep the last working images that are available and maybe update the images once a week something like that we're automated testing taskatron hooked into it the multiple versions thing that we were talking about earlier and we're at the end so questions not that I know of but Koji has them all so I would say it would be perhaps feasible for a d and f plugin that did that that talked to Koji but yeah I haven't there are many same dependencies there's a large chain of dependencies you would have to find the whole tree yeah actually Koji doesn't have this arpeggio sign on Koji they are if they are well if you download just from Koji go to the web page and click download you get the unsigned version but if you look at the command line you can tell it Koji download this build signed by this key and if it is if it has been signed by the key you'll get that signed version so there is a way to download them signed from Koji if they're signed anything that's been released has been signed so if it's an end of life relief it's not something that's still tagged in yeah Do you have experience with the standard ruling list or like I don't know Arche Linux in comparison? I don't I have those sort of appeared after my my use of raw hide but yeah I don't know I haven't used Arche I've heard quite a bit about SUSE's tumbleweed and that's a very interesting thing they sort of do a raw hide version where they do raw hide and then they test it a bunch and then release it like a snapshot every week or two something like that so it's it's slower than raw hide but more stable perhaps because it's gone through the QA and we want to try and gate raw hide with open QA and I think that'll help that as well I think raw hide is a rolling distribution some people argue with me about that and I think some people like that sort of thing but most people don't because you do not have any control over when major things land so like say you're a document developer you depend on LibreOffice you use it you know full-time if you're on a true rolling release you get the new version of LibreOffice when your distribution rolls it to you and so you know maybe you're working on a big contract or something like that you don't want it you have to have a way to downgrade when our trim is downgraded it just does everything for you okay so in the Fedora world I mean those people would want to stick to a stable release and then oh I need to upgrade to Fedora 25 I'll schedule that you know next weekend when I have time so you could but you don't know you don't know you don't know his problem well you don't know it's there until it's on there it can be not LibreOffice sure absolutely and Chromium and Chrom are crashing right now it's not dayfall it's because Glyphs was updated yeah sometimes it's hard to track those things down yesterday I did Fedora 24 PM installation just to get working in press when I run raw hide in press it worked fine but try to press anything that gets in the reaction impossible yeah I actually started doing this presentation on in press in raw hide and it was really slow like you would type a sentence and then 30 seconds later it would start to appear so that may have been the same thing you saw so I switched to hovercraft as much as you how long have you been running raw hide about 3 years full time since my well actually before I had a raw hide install that was using butterFS and that didn't go too well I had crashes and data loss and then I reinstalled and I had that same install since then was it the fault of BDFS was the special specific I don't know what caused it but I worked with one of the developers and he gave me a tool that got a lot of data back but not all of it it was bad this was 3 years ago though so that's a long time in file system because I think snapshots on BDFS are the best and it's very good especially with rolling distro that you can go back before the update but if it destroys your data it's not that good anything else anyone alright well thank you on the butterFS point I use butterFS on my laptop I don't actually run raw hide but I use butterFS on my laptop and snapshot the whole thing including a boot so like that includes all your kernels and you snapshot it in the past and that's a pretty good model I think it would help raw hide as long as butterFS stays stable so that's the dependent factor you don't know when that's going to break if it ever does you could do the same thing with LVM right do you have a boot like this stuff would you use? yeah so that makes it simple but I have some a blog about it if anybody wants to go read it justymang.com ok alright well thank you guys very much