 Hi everybody. I am Matthew Miller. I am the Fedora Project Leader. This is a Fedora Council video meeting. So in the Fedora Council, we try to conduct most of our business so that doesn't need to be done in meetings at all, but we found that having regular meetings on, you know, IRC, chat-based meetings help us make sure that our regular business gets pushed forward because that kind of cadence is important and also that having these kind of high bandwidth, literally high bandwidth video meetings let us keep in sync with some of the other important parts of the project. So we've been using these to focus on a sub-area of Fedora that we have there's something interesting going in on or might need attention or needs more visibility or something like that. So this week we are for this month, this week today, we're focusing on the Fedora re-spins pig, which is largely Ben Williams here. And it's one of the things that's been going on in Fedora probably one of our longest-standing sub-projects that's still active, I think. And so Ben has a presentation for us and then we'll have questions for him afterwards. Welcome Ben, take it away. Now my question is how do I share this presentation? Aha, there should be a share link up at the top of blue jeans there. There's a share screen thing, it looks kind of like a monitor icon. All right. Should have done a tech check beforehand. Okay, loading presentation, awesome. It's working. You see the Fedora re-spins? Yeah, I see the whole window there, it's not in present mode but I see it there. I go and present mode and then it takes up both my screens so I can't do that so we'll do it this way. All right. Hello everybody, I'm Ben Williams. I'm JB Willa in the Fedora world and also Southern gentlemen and KK4EWT. I'd like to thank you for inviting me to talk about the re-spins and hopefully I can answer any questions that you may have. Today's my topics for this presentation are the history of the re-spins and some information about re-spins sake, how and when we create the re-spins. The history of re-spins, first Fedora re-spin was done by the, was done as Fedora Core 5 by the Fedora Unity project. Fedora Unity project was Fedora community members, me and about five other people were looking at the documentation in Fedora and it was very lacking and so we had sites that we offered how to fix issues and stuff like that and we also did the first set of re-spins. SC5 when it was released had a kernel that had a lot of issues and so we re-spun it for the community and that's how we got started. The Fedora Unity project also released the first Fedora live, it was not an installable live, it was just a live instance for SC6. The first installable live was released by the Fedora project for SC7 or F7. Got an extra C in there. For the Unity produced updated ISOs through F12. Valley and Anaconda issues left and right and then the Fedora Unity project basically dissolved. I personally built re-spins for my own use from F12 to F15. F15 we had shell shock and I was talking to Kevin from Release Engineering and I said hey if you give me a place to store this I'll release updated ISOs so people will not have the susceptible shell shock at the install. With that Release Engineering provides alternate re-spins for a location for us to distribute our ISOs. The Fedora community has seen that this was a one-man operation and step forward to help us with the re-spins and thus the re-spins save was born. According to the past Fedora re-spins has 15 members. We probably have about seven or eight that are active. Re-spins are created either by using the lab media creator or live CD creator. There's a link there if you want to see how to use the lab media creator. Re-spins were being created on each kernel. Each even kernel example would be 5.10.12 but due to the fast release from the kernel upstream that has been cut back to the first on the 15th of the month. There are one there are a couple weeks we were getting four kernels a week. This is a volunteer situation so that was a little bit much for the test and try to get two releases out the door in a week so we backed it off to one release every two weeks. Any questions? Thank you that's a great overview. I guess wow was shell shock really that long ago? I guess that's my main question in my mind. Wow that makes me feel old I guess. Yes it was 19 versions ago. That's amazing. I have a question if you don't mind. I would like to try to be joined and try to help the community on the rest in live image and I checked the documentation it was kind of you know I lost in it I didn't fully understand how do I going to start it I know how to create a live so image on the update ISO files sorry updated RPM files to create an updated ISO but I couldn't actually find how to submit this results I only saw a big chunk of global documentation as far as I know and that's about it I didn't find any other like like tangible information for myself maybe I couldn't understand it maybe the documentation is missing but I'm kind of lost between that process so yeah thank you. Come see us in Fedora respins on free note. IRC channel Fedora respins is the place. That's where we hang out we we have a builder which is a VM at the moment I'm working on a cloud instance at the moment trying to build for F 34 and we run into some issues and I have filed bug Zilla zone and I'm in communication with Brian Lane who is the Lorax developer or maintainer in communications him issues we're having I've got scripts we've got a script that we run it it builds all the ISO's for all the labs. Right now the door community is not set up for individual people to submit ISO updated live ISO's. When I said submit means I would like to submit my test results like there's a taste matrix as I saw so how to taste it like how to contribute that part that part is I didn't find that I didn't understand it so that was the problem I had. Okay we do have a matrix we have a testing matrix that we use let's see if I can get that URL. Well you look for the URL does that use the same test matrix like the same test criteria that the release criteria uses or is it a separate test framework? It's a test separate test framework our framework is manually driven usually the workstation and KDE ISO's are when we submit to alt Adam Williams is pulling that information in the open QA and they usually test those ISO's as well. They get automatically tested once once they're uploaded basically. Yes sir. And so you're building these on a is it a VM in Fedora infrastructure or a VM outside of Fedora infrastructure that this stuff is? It's a VM on my infrastructure. On your infrastructure. Okay. We we've got a cloud instance that is sitting on Fedora infrastructure at the moment that we're trying to get to work. Okay. If we get it to work we'll I'd rather build there than on my machine. So I know so Mark is here from Lenovo he has some interest in this because sometimes we want to have newer newer versions of Fedora workstation to put on the Lenovo thinkpads that are shipping with pre installed. And so we recently did a thing where we basically asked release engineering for an official or semi official re spin from there. I'm wondering if we could bring these processes closer together if we could make it so it's easier for you as the re spin SIG to do it. So you know I I think Adam Williamson and Mohan both volunteered and Peter Robinson for that matter had a lot of people volunteering to say hey I could do this no problem as a one-off. But if you're doing something twice a month it seems like having a SIG responsible for it rather than adding more work to release engineering would be nice. What do you think about that and is there a way we could move this to enabling you to do it? Well if I had Koji access where I could do it yes that would work out great but QA is covered up trying to test and get things going for the next release. So that would take a council vote that the re spins are legitimate if they are done on Prodors Koji and testing done by the re spin SIG is adequate and if we say it's not worth releasing we don't release it. At the moment there are last build mate did not did not it built but it had a boot loop. So it was just bringing up the login screen and just keep looping to it. We will not release that. So that's good. So when that happens you basically just skip it for that release. Skip it for that release. And if any of workstation and KDE don't build we don't release anything we just leave what's out there till we can get something built. Do we have a way to give do the spins maintainers get feedback that their re spin didn't build? Well usually it's not the spin maintainers problem. The kickstart can be working and there can be some Fedora update that day that things are just not working and we can give it a couple days and things get fixed. And if it doesn't get fixed in like a week time I start filing bugzillas. Alright that makes. So how how different is like the framework used for testing from like the framework used for like a Fedora test day or something like that? From what I've seen in the helping with the test days not very far off at all. There's something people who are involved in the test days could get involved in this kind of testing pretty. Oh yeah. Yes sir. Yes. So that leads me to one of the questions actually. I run most of the test days around Fedora. So it would actually be good if we can collaborate sometime and we can have respin test day. We have a set of test days going around anyway. So and test days is kind of not tied to the release cadence. We can pretty much have test days on any topic. So that opens up for something like a respin test day or a test week because it's it's it involves testing a lot of images. So most likely a test week would be good. I would this is definitely something I would like to see become more official because it's useful to a lot of people for you know hardware enablement and also just you know not applying you know two gigabytes of updates which time you start your system. I know once long ago I don't know if he's still mad at us but it was it was on Google plus Google plus Linus Torvalds claimed us for not having having official respins with New York kernels. We've outlasted Google plus so maybe that's not an issue anymore but I think there it is definitely a thing that people would like and you know we'd like to keep Linus happy as a Fedora user so we could we could do that that would be cool. How much trouble did we cause with that sort of the official we low image that we did for the X1 carbon a couple of weeks ago? It seemed to be no trouble from the side that I saw but I think I think what I'm hearing from people is that doing that a couple times a year is no trouble but if we'd like to have something that's kind of on the cadence of you know doing it twice a month that starts becoming an extra job for somebody and we'd like to be able to empower the people who have been doing it and want to do it rather than adding more work for people who are there just because they have to be the only people to do it. From your side Mark is there things that could have gone better with that? No actually it was amazing and genuinely thank you so and yeah I mean that image is going through well I think I have one bug related to the fingerprint that we're still trying to reproduce but otherwise it's all looking very good so thank you for that and it's good so your question about testing obviously you know we're not going to release an image that hasn't been through our test process but that's not the same as getting the Fedora thumbs up right? I want both In the chat I'll post a link to our test matrix. We'll see if we can make sure of those. Edward can you copy these links and make sure they get to the YouTube when we post this? Yeah sure. Awesome thank you. Alright so so far it's been a pretty short meeting I know there's a lot of complexity here. Ben other than getting like more direct access to things what else could Fedora as a project do to make re-spin SIG more successful? I asked that question of the SIG members and they said we don't know okay I other than what we said is if we could well on the websites if we had it could have a link to the downloads for updated ISOs but I understand why it's not because it's not official we can have a link on the website oh yeah there's updated ISOs done by the community they are located here and a link for it besides that that would one more thing I would like to add is the only respon link when I learned in the first time it was only in IRC I only figured this out that link the respon links it has been when I learned long ago it was only has been showed in IRC title on the federal IRC channel other than that it is like it is unknown a bit of it and when I know I try to share as much people as I can because it's actually updated images they do you don't have to download tool gig something if you start Fedora at the moment right now try to download the respon image and I show that thing he doesn't even have any idea where it is when I show him now he knows it other than that it is impossible to find it so actually we need to show that more maybe in the magazine maybe in the website yeah I would like to see that as well but it needs to be more open and more feeable because it's only in IRC and it's very bad so it's on the planet when I release I put a blog post up and it hits the planet yeah and there are people who are picking up my blog post and the street yeah I do that so one thing that you get a little more official visibility is maybe putting in the community blog yeah basically it's copy the same blog post there would be a thing to do I think for putting it on the official website I think basically we need two things we need it to be built in Koji which I think you know there's some technical limitations but I think we can probably overcome those and then we also it's the testing level that I'm concerned about because it's it's those things you said that are like you know like there might be some little bug there which we might not get we do pretty extensive level of testing on the official you know release cadence releases and I'm a little bit worried that a light the lighter testing that the respins go through might give some people a bad experience and we don't want we don't want that obviously so far all the years I've been doing this everybody is saying it works yeah I and usually when somebody has a problem I point it to the re point them to the respin some and pound for Dora and it solves the problem yeah so I think yeah you're right that it there's also the there's probably more problems fixed than ink than introduced each time so so it it might actually be nice to get to the goal of having the official downloads and this is this is not a next year goal but a a aspirational goal having the official downloads like the first download you get always be an updated respin the life cycle of the release but I see Shimano dropped off video I scaring away QA completely by suggesting things like no I think that would be nice and at the moment if somebody installs workstation the release workstation there's 1.1 gigs of updates yeah so at that point you might as well install download so anyway so and it went when we had the install the default installer was had the option of network sources and apply updates that made it easy to avoid that but the default installer from the live right now installs from the live so you don't get that option so yeah I I don't know I personally think that it would be a good goal for us to move towards that and so I think the steps of making the making the respin more official is probably a first step towards that and then you can look at getting it more publicity yeah that would be amazing I think we can we can create a trigger to I think maybe Twitter for a post and in the telegram project news post maybe triggered by the same planet post you can go to the community blog to the planet and to Twitter and to Telegram that is the the things I can manage and I cannot manage more than that but I think I can help with that I think it's very useful I remember it was like two or three years ago and the last free soul that is a installation fest that is here in Latin America we can we have a specific ISO updated for free so that was awesome because the bandwidth in normally in install phases very very low and having an updated image to offer to people was amazing we were the only these two offering an updated image everyone else have to install and update Tom I noticed you say my bad a couple times in the chat here do you want to elaborate on what you're saying my bad about we keep bringing up that we're pushing these giant piles of updates thank you for doing those no spot if we didn't have those updates we wouldn't need the respin so no that's part of it we love it Ben do you have any metrics on how many people are downloading the updated respins no sir that might be interesting to do to again it would be interesting to see what the audience is currently even with the difficulties of finding it if you've got a reliable user base there that's an interesting data point at the moment I've asked for numbers each span release engineering cannot give me the numbers I can tell you the numbers from our points it's right about 150 gig per release I would say the torrents are probably more reliable because I've talked to smooch about getting stats from various things from the website and the problem is the internet is so full of nonsense these days that it is basically 90% bots doing various nefarious things and five percent mistakes and maybe five percent signal in download written out the count so you can't really get any useful useful numbers from webs like the FTP or web stats it's not because they don't want to it's just because we don't have a really good way of measuring those things and also we're using mirrors right although if we only had one site we could have the stats but since you got some mirrors out there you can't do it right which you know saves us billions of dollars in bandwidth so it's hard to argue with that yeah and again you know this is part of the trade-off we make you know fedora does not do any tracking there's no paying backs when you go to do an install if we were if we were basically anybody else this would be an easy problem to solve because we would just make the install or send all of your personal information and your social security number to our data collection service and then we would know but yeah we don't have a real there is a question I would like to point it out is there when you release the recipe image is it shown in fetch message yeah because if it is it will be nice to make a bot to track them all because I would know there's a filter and title to easily be handled because the fed message actually handle that so much nicer on that point that was that is beyond the scope of the skew stick so I think that if it were produced in Koji there would and there would be fed messages along with that but the trick is is there a fed message that is associated with an approved official release rather than with that will be nice because we can actually make a bot what my friend said in the telegram that will be much more viable solution so that's why that's one thing that might if you got somebody who's interested in making bots making a command that the respin sig could run that would say this release is blessed and just put that on the fed message would be actually possibly a useful useful thing yeah give any any any information can be tangible I can also utilize that and put it in my community in my language show that that's fine that has been it's not more publicity more people can access it that would be nice yes yeah I also had a thought on another way to help publicize this we've got a fedora quick doc section on installation and writing a quick doc that just says they are explaining what the re-spins are explaining you know the trade-offs of testing versus newer things you know giving the community built status and then linking to them right there would probably increase the visibility by I don't know a thousand percent so you think people read the docs so right people do go read the docs people and people it's something that people can point it at as well I know I know but we are also very people also do searching for things and it will come up in search results that the fedora docs has pretty good Google gravity and we're also hopefully going to have actual search for those docs themselves sometimes I'm going to say one thing everybody likes magazine just please and make a post on magazine 100% better than a documentation yeah so all right a post on the magazine would also be a good idea although I think probably don't want like you know every time there's a re-spin but just like a what are the re-spins no no just show this in the description this is how you access that would be even so much better yeah I'm sure the magazine editors would be happy for I guess the difference is the magazine article kind of makes a splash at one point and it's always there later but you know two years from now it will it'll be unclear if it's still current information because it's kind of blog you know time-formatted so having it you can put the same content in the magazine on the docs I'd say put it both places I guess the problem is is a year down the road people will be trying to pull the old ISOs and the new stuff's gonna be there so that's the only thing I've got about the magazine yeah you wouldn't want it to link to the specific ISOs you just link to the you know this is this is the tree where there's a general ISO for every time it shows the latest ISO that link enough because it's always show the latest ISO in the in the link you can always redirect to current and then point the current to whatever the new ISO is that's why the Qweta is basically face the current testing matrices right off I will let people who care about the redirects and current and so on on the distribution network worry about the details there that's else that is they don't stream for me so that's not my issue right so and you also said these are available for the a torrent where what's let's let's make sure we put the torrent link in here as well do you the torrent links are on the alt respents okay and they are not stay safe what okay when you when you update the image it also updates the torrent magnet as you're right but it doesn't stay same right the torrents this do not stay the same they make it a new yeah all the links go in there at the same time I see this over there right there as well okay cool I can just look on a dot turn that's fun just this let me post it please this is the only link is the show always pointing the latest one but the inside of the ISO files is always changing than the the list in your ISO so yeah the ISO is changing so is the checksum the hash um yeah and the torrent file all that'll be the only links is the general link we can counter on so yeah and there is a tiny URL link it points to the same location and that's usually what I put in all the documentation only I didn't find it yes you did cuz you found it in the pound for door channel yeah I mean I just showed it okay so this is the link I have but I know the tiny one in the I'm downloading from zero zero peers here so I got a question I don't know if it makes sense but do you for the testing that you do on the images is any of it automatable if I was wanting to run my own open QA yeah okay but I like that I know it's always case of if I can make a bunch of platforms available it's that it's always hard to get the hands to go with them and I'm just wondering how automatable would be to help on that side of things so I think that if again a side effect of doing this in the official Koji was that it would be pretty easy to plum into our official open QA as well with automatically flow so I don't know about how you open QA works but is that literally just something it would pick up an image and and I could deploy it to a whole bunch of platforms and find out if there are any issues we maybe we need to have Adam on here to talk about open QA so I think that it is all virtual systems in open QA yeah all of them is just right I would be easier we also Shimantro can you talk more about this we do do automated testing at Red Hat on things but I'm not sure how all that goes so usually Adam has this tool called Fed Find which will basically go find the candidate image that would basically just flow into the whenever it's built it's basically sends out a compose message and then that is plugged into the open QA it's basically in short it would basically capture the screen the needle would match whatever it is supposed to match if it doesn't match then it would call it a failure it can it is basically something which can click on buttons and basically move forward records a video of if it fails it pause on that record that screen send a message via a VRT back to Buxelizer Chris report and then that's how it basically goes so technically how I understand it is if we have it built in Koji it would be like it would just flow into Adam's do we have any way do we use beaker or any test framework to test things on actual hardware at all that would be very interesting to mark yeah that is something Adam can be more precise on because for all I know when we had this power PC le got added there was a bunch of hardware that we had to add to open support so I think that is something Adam would be more happier to discuss okay yeah no I don't have an infrastructure that I could drop this on to yet but I just think it ahead I was okay if I can get to the stage where I do have something I'd be nice if it worked so that we could be part of what triggers and test it on the hardware like if you want to provide me some infrastructure I'll test it on the I don't make the same request I don't get it then you should listen down because they know how to do it yeah no it's absolutely better yeah but my imagination is I think about like a hundreds of hundreds laptop and Lenovo showing how to install anaconda and federal images all over the place that would be nice pictures to see it that would be awesome yeah I'm gonna drop the link to beaker in the chat here as well we're gonna have a lot of I have looked at beaker and and this is I don't I don't have a an infrastructure that I can use that with yet but it's it's something which I'm looking at okay and I I don't know if it's possible that open QA can also do things like that but we just haven't set it up to do that I don't actually know the answer yeah let's go ahead question trying to think ahead of what would be useful good thank you are there other questions anybody and yes Matt I'm building on Fedora boxes and Fedora VMs using only the Fedora repos that was what we were told back in FC5 when can't remember the project all the way down it's everything is run on Fedora using Fedora tools and in the like so FC5 that is either either Max or Greg because he came back to the Fedora unity project because release engineering was you but to a release a live ISO and release engineering told him it couldn't be done so he asked us can y'all do it we said we'll try and so the day that FC6 was released we had early access to a meter mirror and we created the ISOs nice cool and then release engineering come talk to us and the next time for F7 they released installable lives that's great well evidently something must be going on because I'm getting a flood of people trying to get the respins at the moment so excellent quick question I know the also I'm not a cat I'm David Cantrell so the question I have is around the images available in in cloud environments like Amazon EC2 things like that are you doing respins for any of those now is there any interest in that from people oh there's interest in it I do not have I don't have the koji instance to correct those okay so alright I'm saying some normally more interesting in the in marketing and this kind of stuff maybe my question is are you interesting in having a maybe a more appealing website with the press with a rest pin log or something that we can make it more than just an Apache directory listing I'm sorry I didn't understand the last part of that a suggestion was to make an updated a website a respins website rather than just the directory listing we mentioned earlier talking putting it on like the main get fedora website but also maybe in between just having a respin stuff nor a project on our web site or as always things come down to man power well that's what I'm offering I'm offering the mom power yeah you can make it yeah it is a start it will be an aesthetic side that just have it beautiful directory listing at the end so it won't be just one do all right that that sounds reasonable too and after you make it who's gonna keep it up I think it will be in the Fedora infrastructure just we need the the DNA is going to be the maintainer after you put it up who's gonna be responsible that every time I release a set of respins it's gonna update that page every time you have a release you have the this directory listing that is in the chat again in the tenure URL form so I just can create a site that have this director listing about and everything is dated it will be automatically dated that's the idea I have so it's just the aesthetic page it points to the the download the download the doors that's it all lab respins yes that that's the idea something like that seems seems doable I I know Marie is working on getting the website team up and going again as a viable ongoing thing and it seems like this would be something that we'd want the website steam to ultimately own in collaboration with the respins sig for actual content but I can work on it I can take it as a marketing test I see that we're not using respin stuff at our project at arc so it's available awesome all right any other questions or comments all right thank you very much Ben this has been very interesting and helpful thanks for everybody for showing up thank you mark particularly for your comments as a consumer of this that's very helpful for us to know about and invite yep and see you all you know on the Internet said back in video call I think we're talking about the new logo next month so that'll that'll be exciting see y'all later