 by Steve McIntyre. Steve is a former DPL project leader and also a developer who is a part of multiple teams in the project, which are key teams, very important teams in our project. So Steve, the stage is yours now. Hey, how is everybody this morning? So yes, it's awesome to be here in Prizren again. I've trailed this session several times this week, as have other people. I hope it's not a disappointment. There we go. So I'm going to say my usual spiel, this is a boff. I like running boffs because it means I don't have to do all the talking. I want this to be a conversation. I don't have enough material to keep us busy for the full time. I'm going to talk or lay out what I see as the problems we have, options for what we might do to fix them, and then I want you lot to help me work out the right answer. I have a gobby doc, as always. Please, if there are any great insights during the session, if somebody could make sure those are jotted down, that will be great. I will do my usual and promise to, at some point after the conference, I will collect everything together and send out a summary email to various mailing lists. It's something I find very useful and important. So what is the problem with firmware? Well, background. Most hardware needs firmware to work. Hardware typically is very, very difficult to drive direct in software. It may only provide very basic functionality. If you want to have a common class of device, you typically, you will have some firmware on it which will expose the features. It might aggregate some. It might do all kinds of other things. It will be there to provide basic functionality. Some devices have firmware built in and you don't even know about it. Your keyboard, your mouse, disk drives, they all have software built in that we call firmware because then it's not software. You typically don't know about it. You don't have to know about it, but it's there. Some of the firmware is loaded at runtime. So this tends to be for the more complicated devices. Wi-Fi is absolutely the most common. It's common in graphics, some wired network cards. Audio is becoming more common. It could be disk controllers. It could be all sorts. The vast majority of this firmware is non-free. We don't like non-free things. That is a problem for Debian. Hopefully most of the people in this room recognise the thing about priorities. Our priorities are users and free software. That's not ordered. We don't pretend that one is more important than the other. We care about both deeply. We don't want to be shipping non-free firmware. However, our users need firmware to make their hardware devices work. I see that as an issue. So for a long time we've been shipping a non-free section alongside the main Debian archive. We have been shipping as part or to go for installation, you either have a separate firmware bundle that you can have on a USB stick or something alongside your installation media, or we have what we've been calling for many years unofficial installer and live images. That includes firmware. I say unofficial. They are in a separate path on our CD image server. We call them unofficial because we would love people not to have to use them, but they're prepared by the same team. They're prepared using the same software. They're released on the same day. They're signed with the same key. It's a very vague meaning of unofficial here. Jonathan. The unofficial part comes from it doesn't meet our standards. That's why it's unofficial. It doesn't matter who prepared it. It doesn't meet the Debian standards. Therefore, it's not an official Debian image. Sure. I was a person who added the unofficial piece here because I didn't want them to be in the same place, but it's not great. So this is all a bit of a mess really and it's only getting worse. As a quick guide of the people in the room here, how many of you have a laptop that you can use without loading non-free firmware? One, two. Well, I have lots of, all my old laptops become servers at homes and they don't need any firmware to function like that. So as a laptop I need firmware. Well, exactly. So this is the issue. And of course, again, if you go back and think about it, there is non-free firmware on your system. It's whether or not you have to load it at runtime. It's perfectly possible or it has been until very recently to install at least a basic system with wired ethernet on a lot of common laptops, but that's getting harder and harder. If you want to get wireless, you're basically out of luck. There are a few free chipsets, but they're becoming less and less common in modern machines. This is becoming more of a problem that for a typical end user, the official images that we produce and distribute are not actually useful. I don't think that's great. One of the things that we have, I started a discussion about this. I blogged about all of this about what I'd like to do. Ansgar has been awesome and has already done some work for me. For me, for the project, but for me obviously, because he loves me. If I do that, we actually have, right now, supported as part of DAC, we have a firmware non-free section. For those people who haven't seen why that matters, at the moment, if you install firmware as part of your installation, so if you add the non-free section to get at firmware, what you end up with, of course, is that firmware. You may also end up for the rest of time on your machine installing some non-free software without realising it. That may not seem important to you. It does seem important to me. We as a free software project do not want to be encouraging people to install non-free. We would like to be showcasing the best that the free software world has and that's not it. By adding a new section for non-free firmware only, this is a step on the road to making that better. There is no magic with the new component. What people will need to do, and I've checked, no one's done it yet, is you will need to add non-free firmware slash section in your control file for the binary packages. I appreciate that that doesn't actually apply to probably anyone who's here who's been around. I guess not. Probably nobody here actually maintains a package that needs to worry about this. Gunna does, of course. We did briefly consider should there be extra magic somewhere on FTP master to do filtering here, and then Ansgar, I did have a good point of, we don't like special cases. Let's not do a special case. Let's rely on maintainers. There's only a handful of packages that need this anyway. We will need to add support elsewhere. We will need to add it in, or anything that knows about the layout of a Debian archive will need to understand that there is a new section, so a new component. It shouldn't take too long, I hope, to it would. So, what else do we want? So, I want, so, Helmut. Brief question, will there be a transition period for users like being able to pull firmware from plain non-free for a while as well as pulling it from non-free firmware? Like duplicating the packages and two distributions. Is that planned? That was a question. It has been suggested we should do that. I personally don't like the idea too much, but I don't feel too strongly. That's something that we will need to work out before we release Bokwyrm. Gynna. Given it's a few packages that are affected, well, my plan on this was to replace the package for a transition one for a dependency on a package on this different suit. Of course, users will still have to add the non-free firmware component instead of just non-free, but they will be, like, I mean, I haven't really implemented anything yet, but they will be at least prompted that their firmware now depends on something that's not there. And, well, we will see how to communicate these two users. Yeah. Cool. Thank you. So, now we have step one. One... Sorry, I have a quick question. Because non-free packages are not automatically built by the buildee, so I want to know whether non-free firmware will be built on buildee. It's slightly more complicated than that. Some non-free packages are auto built, but we have to set things up so that the buildees will build a certain subset. We absolutely expect that the non-free firmware packages will be auto built for the same reasons. Sorry? Yes, they get built. We expect that they are architectural. We don't actually build any software for most of these. They are binary blobs that we are given by upstream, so long as we can distribute them. So, I have enumerated the possible ways forward. I actually had five options, but since I first suggested this, we now have the non-free firmware section, so I have amalgamated two of them. There's no point arguing on that one anymore. As I said, we currently have two different sets of installation and live images. I can see that we can carry on like that. We could make it easier to find the non-free images. That is something that we haven't been good at. That started off as a deliberate policy. We shouldn't be doing that anymore because we just make it harder for our users. So, we could basically make no major technical change here, or we could stop providing non-free images altogether and go full-on, Debian is free software, free software is all we do. That's an option. It's not one I prefer because, again, that doesn't help our users who need this. We could make the non-free images official and at that point actually push them alongside our normal existing images that don't include firmware. We would need to name them appropriately, label them, give users good guidance as to which one they might need, or do we just declare that actually firmware is a necessary evil, it's part of what we have to do, and do we just stick it into our official images and just move to having one set? Or is it something else that I haven't thought of? If you think there's a better answer, please shout out me because I'm an idiot, I must be. Jonathan. Do you think it's a problem that what we call firmware isn't firmware anymore, if you consider the AMD graphics firmware or Nvidia firmware, it's a whole operating system complete with not only firmware but drivers and memory management and file systems and almost anything you'd find in a complete modern operating system these days. Do you still consider that just firmware and that it's just an issue of where it runs because obviously it's got a lot more complicated than that? Well, of course it has, I mean, but then again, firmware has always been a potentially large blob of all kinds of things that we just didn't know what was in there. We now may have a better idea of what might be in there because people have reverse engineered bits or people have told us what's in it. I don't think that necessarily that makes that much of a difference. Whether it was a tiny 8K blob that you might upload onto your network card to let it do pixie or whether or not it is a 100 megabyte lump of shaders and God knows what that you put onto your graphics card, it is stuff that we have no control over. The hardware vendor provides it, it is necessary in their eyes to make the hardware work and it doesn't work fully without it. I appreciate it, it's a difficult line to draw. The typical line that most people would draw is does this stuff run on your main CPU or is it running in some other capacity? Is it running on a separate controller? Is it running entirely outside of the operating systems control? I think the difference comes in where many people still consider it as just like a few kilobytes of initialisation string and it's just an issue of where it gets loaded but it's clearly not that simple anymore and I think people should just keep that in mind too in making decisions. Sure, it's a thing. Christof, regarding the choices you are listening there, I think I would kind of prefer option number four just include stuff but the installer can still ask the question if people want to use it so they still have to make their choice just not at download time where it's way too early and that would even work in the case where they can try without or maybe the installer can just try without and if it doesn't work go back and try with. Thank you for asking exactly the question I was going to move to next. So these are broad descriptions and that is a key point here. One of the things that we will do if we embed firmware more firmly sorry bad pun is that we will make DI more explicit about what it's using firmware for. We will were possible ask the user do you want to use this firmware or not and list the devices that we think need the firmware explain and maybe give links to wiki pages or whatever with more information about them so people can read about afterwards. We would love to encourage people to write free firmware replacements for these things because of course we want more free software that's what we do. It gets slightly more complicated. If you remember I mentioned earlier that one of the latest non free firmware blobs that we have to worry about is audio on the most recent couple of generations of Intel hardware. That is an absolute nightmare for logistical reasons. Imagine you're a blind person and you're trying to drive the installer through audio through the text of speech that we have. If you don't load that firmware immediately on first boot you get no audio. It's difficult to ask people do you want to load the audio firmware without loading the audio firmware for example and I know that's a contrived example but it's one we already have. The other one that is coming is I expect we will have to load graphics firmware on devices that need it soon. If you want to drive for example a high DPI display it's this is not going away if anything it's only going to become more of a problem. We do want to at least however as I said make it clear to the user which firmware that we have detected and we have started to use so they can learn more about it. I was thinking and I'm sorry for bringing this into a probably corner cases that are not that common but I think we should be explicit and draw line towards developers on the use of firmware itself because for example we may find abandoned games from the 8 bit consoles and they are technically firmware that doesn't run on your main processor but you can install an arcade machine emulator and then download from Debian hundreds of old games which are not free software and would be run via an emulator so I mean I think that's a case we we would not like to distribute that. So my argument is that that's not firmware it is that is software in it that is designed to run on a on a machine even if it's not the machine you're currently on. What we're talking about here and I want to be focused on this is we're looking at just the bits of non-free firmware that enable the hardware you have in front of you. Anything else I explicitly want to ignore as a distraction. Thank you for bringing it up it helps. Yeah in the last mailing list discussion Gunnar there was a discussion of what would that firmware component contain and the initial suggestion was all the packages that start with firmware dash and packages that install in libfirmware so that seems like a very reasonable first step to start off with I think. Yeah I mean we don't necessarily have everything nailed down on this but definitely let's not get let that get in the way of useful progress. I really like the idea to include the non-free firmware on the official images and then add a question to the installer do you want and having the default yeah use it or not but since we discuss a lot of corner cases and technical things we should still keep a focus on the end user and I would like to have this question do you want to run the installer without non-free firmware that the end user does not see this question at all because so we have this possibility to enter the expert mode and I'm pretty sure if you ask the end user do you want to run the installer without non-free firmware and still have the information your wi-fi card or your audio may not be working then we will get the end user get scary because the end user then has to decide and he cannot decide because he does not know do I need the firmware or not so it's in my opinion it would be very good to have the default yes run the installer with non-free firmware and don't ask the end user about this. Yeah thank you that's a very fair point so where I want to go with this and you know and I've made no no secret of this is I want the project as a whole to make this decision I am going to take this to a GR apologies I should have had this GR out already life got in the way sorry that was me but I mean I'm the person leading the images team I could have just done this but I don't think that's right because this is a real change to some of our core of our core principles so I want all of Debian all of the Debian developers to make this decision and then equally it's also we have a shared responsibility for it I've already had I won't call it abuse I've had some strong feedback from people who don't like the idea of us including firmware at all and I get that totally you know we are a free software group I've been living and breathing free software for you know nearly 30 years now this really matters to me too it's but however it all comes back to the our users and free software and that's a you know that's a really difficult thing to split. Christof you had a point yeah I totally don't like the idea of not scaring the users but I think that this is the only place or the single place where we have a chance to tell people that non-free software or non-free firmware is going to be run on the systems maybe it's just a matter of wording that message in a way that it's not scary the installer might just display a single pop-up your system is running this graphics card you will need this firmware we will tell you where you can read after it afterwards later so sure or we could maybe we could maybe have us have a screen that says I have automatically installed this firmware um okay and an option to say tell me more yeah I think it's just a matter of making it not not scary there because this is the only chance we have to talk to people later it's Thomas wants to say something you're all right but my impression is yeah we care a lot about non-free firmware but the end user doesn't care at all if he uses non-free firmware or not he just wants to run Debian on his hardware so to be fair some of our of our users do care some of them don't yeah but if we have the focus on the end users most of them will not care about the technical details and maybe not about freedom or whatever they want to run linux but I think it's a very good idea to to keep an eye on the wording and not to scurry the end user so we do need to make sure that we spend time on the messaging and how we label images we should make it clear so people can find afterwards after they've installed exactly which non-free things we have included again I mean there's a yet another example here I'll pick on Gunna again the Raspberry Pi images you know for the older versions of the hardware that rely on non-free firmware to boot so of course we can't ask before we before we run it do you want non-free firmware we can't even start an installer without it but we should definitely make it clear in the description of the image what has been included and once something has been installed it should be easy to find that information too so we can help people to learn about this so currently we say like our official images are without firmware if you happen to need it you can like install it but you're kind of on your own and this comes with a lack of support kind of and if we go include firmware and images it kind of comes with an implicit promise on supporting that I figure it kind of changes the acuation over there so my question here is how many regressions do we experience in firmware that actually affect users and how do we do about that supporting because we can't just fix it is there plans for like letting them downgrade easily or do you have any ideas on the support side of that picture sure so Enrico just behind you I think has an answer that I don't know if it's an answer but it kind of ties together with giving feedback and the feedback can include how much Debian can support it so it even I mean I personally would hate having to click enter once more to acknowledge a pop-up about non-free firmware that always shows up in 99% of the hardware I'll end up touching but there could be a differently coloured background or something where if it's green Debian will fully support running your machine and if it's orange then Debian is one of the components running in your machine and can only do what it can by itself and the other bits in the bus network are outside Debian's control so if you want something completely supported you want a thing that is green and you work to get the Debian thing green and maybe yes so going back to Helmut's question of course there is an option for people to downgrade these packages just like any other package in Debian but it's not something we can really support well or widely I mean there are known regressions that happen as you upgrade firmware if you upgrade your Intel microcode to fit to take on the workarounds for spectre and meltdown for example your performance can go through the floor however it's a security fix which do you want and yeah and okay that's again it's not a contrived example it's one that does affect people every day um but it's a it's a choice and it's something you might there's not there's very little we can do about it um what we can do is point people at the support pages for that for those firmware blobs I hope the packages already have the information that included as part of the packaging anyway if we don't then that's an obvious improvement um one of the things that we want to do and again there was some discussion on the mailing list about do we want to include the non-free firmware component if you've installed things and absolutely we must do because of course once you've installed one of these blobs you want to get security updates for it just like any other package that you have um so the default should be we will always pull in newer versions yes so looks like we've got several reasons for not doing this questioning and display in the debian installer actually the installer could get useless without the firmware most people won't even see the installer because they will get their machines pre-installed even if they did see this in the installer they will probably forget it like in a week so maybe it's better to put this question into the message of the day or into the gnome software banner or somewhere where it is constantly visible and it even could provide some information about the security or supportability status of your system yep definitely we could add this for example add a message about this into the initial you know the first user startup with gnome we could put it we could drop something onto the first user's desktop or whatever with a weed me you know here is more information about your system there's a whole bunch of things we can do here we'll be looking for help to do whatever we think is best uh i just got the idea i think at the very end of the debian installer we say the installer has finished now we reboot the computer and maybe we could put a message there we installed your computer using non-free firmware just as an information and maybe some more words that would be better than asking in the very beginning yeah possibly yeah i second that on having the post factum saying okay we've installed it with the non-free firmware and then write the information about this in something like libfirm where debian installer text file with everything links and everything then you can have it as a reference later and you can also have it in the standard file file format on the pre-installed systems as well so you can always say yeah just look at this file it will tell you what non-free firmware you're running now that's it oh yeah cool thanks so i'm going to ask to do something whiskey and could i ask for a quick show of hands in the room and i know this is not representative which of you here would like option one the most that's conclusive who would like option two best equally zero who would like option three in a make the non-free images official we have a yeah a reasonable number of hands how about option four i've just pulled it all in okay option five should we do something else now jonathan what else should we do you know you're not going to get away without that i've mentioned this on a list i've mentioned it in my talk last month in germany on my dpl talk i just like the idea of having another iso available that has only the non-free firmware installed and on the download page we make it clear that this is the debian image that meets all our all our criteria for what a free operating system should look like and the second image is one that you probably want to use if you have a laptop or a cell phone or a whatever and then a user can easily make a decision on what iso they want without it being too complicated for them and without too many hacks in installers or boot process or things like that i think that's an easy fix it doesn't need too many work before bookworm is released i think it's easily implementable i think it fixes the vast majority of our problems that we have without having to compromise any of our values so i think that's the best of these five options in my opinion but we'll fight it out in the grs so it's fine yes in in the in the least aggressive of ways obviously mike yes i believe graham ask isn't that option number three no the the images with the non-free images wouldn't be marked as official they'd be next to the official images and you can choose which one you want to download no the first one is official the second one is one with some crap on it that you can download if you want but if we are advertising them directly side by side i think the unofficial tag is is kind of a joke then maybe we should move away from that term because it's completely misleading to our users we should make it a the debian and debian plus non-free firmware or something more explicit because if everyone has a different definition of what official means then it means nothing yeah agreed so joey hi um i just wanted to mention you mentioned it very briefly but one example of these firmware packages we're talking about is the intel microcode and the amd64 microcode um and that's and that's security updates for the processors um that's used by by almost every system out there even even servers and things so it's it's not like we have some things that's just for laptops that if you're not a laptop then you wouldn't use this at all so yes thank you and that is a really important one it's also one that actually straddles the divide um because that is um firmware that that is code that runs directly on your main cpu by definition however it is outside of the control of the os you know it is part of the setup does the kernel provide an interface for querying all loaded firmware i'm sorry can you repeat that slightly louder there's a fair bit of noise sorry does the kernel provide an interface for querying all loaded firmware now or will it do so Ben is right behind you and can answer that no but it does keep a list internally it's just not uh it's not just not visible probably so there was one other suggestion which i i don't like and i'll admit i haven't given much airtime to here i wish people would stop making so much noise during a talk um which is to ship free images but also have an extra piece of software that we can get everybody to run afterwards which could tweak and add the firmware as well i think that's a terrible idea because it's a it just it's another step that makes it harder for a non-technical user to even start using debian so i i'm not going to nix it i'm not going to veto it i just don't really don't like the idea i'm sure somebody might speak up for it but i can't see it getting much traction yeah so please no please no extra steps but um what i think we are lacking is some tool that is actually able to tell the user what's going on on their system i have absolutely no idea which of my systems are using which firmware and just having some utility that would print out a list would actually be very very helpful um even just for this discussion or whatever yeah agreed thank you we're kind of running out of time so last couple of couple of points uh just on the point on the same point uh if we are being hopeful and we're hoping that they're going to be free software firmware in the future the same tool could allow you to change it so that that kind of thinking forward as well would be nice definitely um of course that's an important feature here we want we don't just want to be documenting the bad things we want to be helping to to encourage people to develop and use better things since we're designing a new tool i'll add the use case of checking um if the firmware that is loaded matches what is on disk or if i need to somehow reboot the machine to is to to run the new firmware of something like when you install a new kernel but until you reboot you don't get the security fixes i don't know if the fir we the firmware we have the same problem that's a good point yes it's we should we should be able to do the same thing yeah Jeremy i think you'll have to be the last one hi um i'm a bit concerned about having a scary warning about non fully firmware installed in the system um because when people get scary warnings then they want to try to fix the problem and i'm not so sure that there's a problem that needs to be fixed it reminds me a bit of of some lintian warnings that may be maybe badly designed and so some maintainers um not fully understanding the situation try to fix all of those and end up causing more problems to their system yes totally that's a good point we're not trying to scare people we're trying to inform people the wording and the style of what of how we describe things really matters here um oh go on last one right at the back why does everybody sit right at the back is that the cool kids on the bus again sorry about that uh one experience i had been in lots of groups of newcomers to debon mostly notebooks nowadays don't have internet parts so people only rely on wifi anyway they go to download the image for installation they always go to the net install and that's a problem because if you don't include firmware and you do a net install you can connect to internet and then they get only a shell and it they get a lot of problems so maybe we can say there oh if you don't have an internet part please download the image with uh interface with GNOME with KDE so that could be an alternative maybe sure thank you so i declare we are about done thank you for coming along this these have all been useful points i know there's going to be a lot more there will be a gr about this apologies we know grs are not something everybody likes but this is genuinely a technical gr that match it that that i think the project needs to make a proper decision about what we do going forwards um let's have a useful discussion and a useful outcome like i say you can probably tell where my feelings lie but i want to go with what the project wants regardless i will implement or implement in my pieces of the solution whatever the project decides i can promise that today um thank you all i hope you have enjoyed the rest of the day