 So I think we're ready to start. So next talk is for Martin about Debian or ARM devices. So it's Debian on ARM devices. So it's not about the Debian ARM port itself, or one of the various ARM ports. I would say it's sort of a mix of different topics. It's basically, I mean basically, I often get the question from users. Well, Debian works on this particular ARM device. I have a device which is really, really similar. So surely Debian should work on it, right? And so far, the answer, unfortunately, usually was no. And it's really frustrating for users because they don't see why. The device is pretty much the same, so why doesn't it just work? And so basically what I'm trying to do is to show like a little bit behind the scenes of how Debian on ARM works. And there has been a lot of progress. So I was gonna show how did it work, what are the improvements, and how is it gonna work in the future. And it's sort of more from a developer perspective, like how do we support ARM in the Debian install as well, so it's like a mix of stuff. And I should say, I'm really looking at it from someone who basically add support for ARM devices in the installer, but I'm not like a kernel guy or anything. So there are a lot of people in this room who know like way more about the tech details, but I'm not gonna really drill down very deep anyway. But if I'm incorrect, I'm sure there are plenty of people who will correct me. So where do you find ARM? And pretty much it's everywhere these days. I mean, pretty much every phone has an ARM chip in it. All those internet of things, gadgets, a lot of them are ARM based. You have NAS devices, and that's really the area that I used to focus on. Those small boxes where you basically add a hard drive and most people just use it for storage, but for us it's actually a full PC where you can install Debian on it. So I always found that to be very interesting. And nowadays you have a lot of development boards, and I'm actually not sure if development board is the right word, because when I hear development board, I think of these like really expensive, like five K for boards or something. But nowadays you have those, maybe a better word would be like bareboards, like Raspberry Pi, where you basically you just buy the board, it doesn't come with your case, even though you can buy those as well. And it's like $30 or something you can get. And I think that's really where most of the excitement is going on for Debian users at the moment. And also a lot of the frustration I should add. And everyone says that ARM is gonna be big on servers. And I think that's something we're gonna see. But right now again, it's a little bit frustrating. So how does it work? So Debian installer, there are basically two big different ways to install on those ARM type devices. So historically the normal way would be to install on like a screen. So that can be a serial console, or it could be a real monitor with mouse and keyboard. And then you just download why the network or from a CD image or whatever. But the other method which we actually used for those NAS devices was a network console where you basically SSH into the installer and you perform the installation via SSH. And that was really the only way because those NAS devices don't have any input output. So even though you can attach a serial console to most of them, that's something we didn't want to require from the user. And the other thing that's coming sooner, I think it was actually committed today is screen support. So basically that you can have multiple sessions within DI which can be useful if you want to open a channel to debug. But so how, and so this is really about looking back at like running Debian on NAS devices, which is something that used to be really popular. And the way it worked was basically that we provided a installer image which was sort of a firmware image. So a lot of those NAS devices have like a firmware upgrade mechanism. And so we would basically fake, we would create a firmware image which the software would accept, but instead of a firmware update, it would actually contain Debian installer. So you would install that upgrade, you would reboot and you could SSH to the installer. And obviously in order to SSH to the installer, it needs to bring up the network. So there is a tool called OUTSUS pre-seed which basically reads the network configuration from the device and sets up the networking. And it can obviously can also do DHCP. And then the user connects via SSH again. There would be some indication, maybe there would be a beeper, maybe change the LED to indicate that the installer is ready. And then the user basically performs just a regular installation, just normal DI. They don't have to do anything differently. And at the end, Flash Kernel runs to make the system bootable. So Flash Kernel is called that way because it used to support like the initial device it supported booted from Flash, but it also generates bootable images on disk for devices that need that. So it really, Flash Kernel really requires an understanding of each device. And our philosophy in those cases with those Nest devices was really, we don't touch anything in the firmware, like in the configuration. So sometimes we have to hack around stuff. So instead of changing the root device in the U-boot config, we would actually hard code that in the RAM disk. Just because we wanted people to go back to the original firmware if they had to. For example, if they had to send it in for repair or something. And that kind of approach really worked well. So I think we really had a lot of people, a lot of users running Debian on those kinds of Nest devices. And it was really easy to do. You get a firmware image, you connect to the installer via SSH and it just works. It's a normal Debian installer the way everyone knows it. Because some of the other distros, they basically provide like tab balls and instructions. So you need to partition the disk, you need to untar it, you need to change those files. And even though that sounds simple, it's so many steps that you often get something wrong. And then you put the drive into the Nest device and it doesn't boot and you don't know why. So where did you make that mistake? Which step? And you basically have to start from scratch. So I think Debian really provided something unique by having that Debian installer support. But anyway, so that's the way it used to work. Nowadays, with a lot of those spare boards, it's much easier, I would get to that. So at the moment, there are three different ARM ports. There is the old ARM EL, which used to be the new ARM EL. But now it's old. And one of the discussions we're probably gonna have later today in the both is about, should we remove that after the stretch? There is ARM H, F, and the ARM 64. And so that's basically the question I'm hoping to answer. So if device A works, I have a device which is really similar, but it doesn't work. Why is that? So there have been a lot of changes in various upstream projects, especially the kernel and U-boot that really made things easier. So in the past, we basically had a kernel image for each platform, where a platform is basically like an SOC family, and they would need a different image. And because ARM takes a long time to compile, there was always some debate about adding a new platform because that would be a new image flavor, which takes some time. And it was just really like you couldn't just have one ARM kernel which works everywhere. And a lot of people didn't understand that. So why do you have those different kernels, those different platforms? But there has been a lot of progress upstream. And basically nowadays with ARM HF and with ARM 64, you really just have one kernel. And upstream basically, so maybe some of you remember the rant by Linus about the ARM people doing everything in different ways. And there has been a lot of standardization over the years. And basically the other thing was is, so they used to be for each device, there was a board file, it was like a C file to initialize the different components. And the boot loader would pass a machine ID to the kernel and then it would load that boot file. And nowadays, there is basically a device tree in the kernel which is a description of the hardware. And then you basically compile that to like a binary plot, the DTP, B. And so basically you just need the kernel image and the DTP which is hardware specific and then it boots. So, and obviously for us in Debian, that makes it much, much easier to support a lot of devices. The other thing that changed in Uboot, when you install the Debian kernel image, it creates the VM Linux file in boot and also the RAM disk. But with Uboot, you couldn't actually load those files directly. You basically had to wrap them in a Uboot image and it's not really hard. It's just a command you need to run. But all of the different devices had different load addresses. So again, that's hardware specific knowledge that Flash Kernel needed to know. And nowadays, there is a command which can just directly load the kernel. So that again was a step to make things easier. And the last thing which really made things easier is distro support in Uboot. So basically in the past, every devices in Uboot would boot in a different way. Just in terms of where would it load the file from or what kind of variables it would use. And nowadays, there is something called distro support which is basically a standardized way to boot a Linux distro with Uboot. And there are basically two ways. Either it can read a conflict file or it can basically run a boot script. And that's what we use in Debian. So we basically have a generic boot script which loads the kernel, loads the RAM disk, the DTP and boots Debian. And we can use that generic boot script in almost all of the modern devices. So nowadays, because of that, it's much more standard and it's much easier to support those devices. So here are some examples from Flash Kernel. So basically Flash Kernel has like a database of devices which it supports. So it's like the machine entry from broke CPU info. And then the kernel flavors, it can run. And so this one is an old device. So it still uses a board file and it needs that Uboot wrapper. So you're basically setting the machine ID. Those are the Flash petitions where the kernel and the RAM disk are stored. And that's that load address for the Uboot wrapper. And now, and that's a device which used to have a boot file and which then migrated to a DTP. And so basically, so that was really, really, really painful for Debian. So basically the kernel people said, oh, we're gonna move to a device tree, but don't worry, we're not gonna remove those old board files, you can still use them. And then a few years later, they realized, it's really hard to keep both alive and they got rid of the board files. And that was really painful for us because we had to migrate. And so you can see here, like a kernel version. And so from that kernel on, you need to use the new way. And one of the reasons it was painful, especially on the QNAP devices, is because they actually have two different CPUs. There are different variants. And with the board files, the same board file worked regardless of the CPU. But because of the way a device tree works, you actually needed two different device trees, depending on the CPU. And so now we basically, something which just worked, it used to work fine. You just had one kernel with that machine ID, things would work. And suddenly with the device tree, we need to figure out which one do you need. And so Ian Campbell, fortunately, did all of that work. So that's actually a script that runs to figure out which device tree that the device needs. And so now, so that's actually the reason I wanna show that is how simple things are these days. So this is an example of a modern platform which uses distro support. So the only thing you really need is a machine entry with the name. And then like the kernel flavor, but that's the same for, there is only one kernel flavor. And then the device tree ID. And again, all of that stuff is generic. So it just uses the generic boot script in the generic boot path. So basically, all you pretty much need for a new device now is like these two entries. So it's really, really simple. So here, I just wanted to talk about the different ARM ports. And so for me, it was really hard to structure this because those changes in the kernel and you put happened independent of our ARM ports. But at the same time, because the ARM HF one is much newer, it works in a different way. So basically with ARM EL, so we still have different flavors, but we were able to combine the Orion and the Kirkwood into one mobile flavor. And we have the versatile one. And so one of the problems we have on ARM EL is that a lot of those Nest devices put from Flash and some of them only have two megabytes from the kernel which used to be a lot of space, but nowadays it's not. So that really puts a lot of restrictions. So we basically disabled some stuff on ARM EL. But ARM EL I think is really, really widely used because of those Nest devices. So I think they are slowly getting old, but there are still a lot of people who use them. Like I said, it requires that you put image. And originally we used both fires and the device tree, but now most of them have switched over to device tree, even though some of them still use both fires. And adding a new device requires a number of changes in the installer. So basically you needed to map the device to the boot image. And it was just a couple of things. So basically adding one device, you had to change like five different places in installer. And it was quite confusing for people who wanted to edit new devices because there wasn't really documented very well. And so some examples are listed here and people who are involved in that part. And so Roger is actually someone who is quite new and he really got involved in supporting those old devices. And so ARM-HF is much nicer. So the majority of devices support that distro support. And the other thing we do is for some of the devices we provide SD card images, which contain U-boot and the Debian installer. So basically Vagrant maintains U-boot in Debian and a lot of those devices are supported in U-boot in Debian. And so we just provide an SD image so you can just store it to the SD card, you put it in and because of the distro support, it just loads Debian installer. You do the installation, you reboot and things just work. And so I think it really has come a long way. Yeah, and so nowadays adding a new device requires much fewer changes than it used to be because everything uses the same like one kernel flavor. You don't need to touch all those different places anymore. And because of distro support, it just works. You can pretty much use the generic boot script. So ARM64, so the problem until recently is that there simply wasn't any hardware that people could buy and it's changing rapidly now. Even though it's still not ideal. So you have, for example, the Raspberry Pi 3, which uses a 64-bit CPU, but the software they ship is only 32-bit. But most of the work is now upstream to run 64-bit on it. And yeah, there are a lot of devices with sort of work, but not quite. But anyway, so the idea for ARM64 is that a lot of the new hardware, especially on the server side, would use UEFI. And so you basically just, it just works like a PC. So you have Grub, you can run them in install form Grub, and then you get Grub afterwards and it just boots like a PC and Steve has done a lot of work in that area. And in theory, it should just work out of the box. So if it uses UEFI, we don't need to add anything. It just works, there is nothing to do. At least that's the theory. So what I found is, so I made the disclaimer, assuming the kernel support and stuff. And to be honest, right now, that's a big assumption. So I've been playing with a few 64-bit ARM boards and basically what I found is there is some support upstream, so I can boot a kernel, but oh, there is no USB support, there is no Wi-Fi support, so I can't actually do anything with it. But I think that's just something that's, because ARM64 is still fairly new and there is a lot of work going on upstream to support the various platforms. So I think over time, that's really gonna be much better. But even though that UEFI idea is there, in reality, we're gonna see different solutions on ARM64. So we are gonna see UEFI in particular on servers, but we also see U-boot. So a lot of those bareboards, they have U-boot. And so right now, we can use that distro support and I think it works really well. But one thing that SUSE has done, so they basically have the same issue, they wanna support all those devices, but they just want to use the same mechanism everywhere. So they've actually implemented UEFI on top of U-boot. So you can basically use U-boot to load GRUB and then you have GRUB. So I think that's something we need to figure out. Do we wanna stay with distro support? Do we wanna use UEFI on top of U-boot? Is that something we wanna give users the option? And then fastboot is something used in the Android world and a lot of those, like I see a lot of both bareboards, but also game consoles and stuff which are sort of Android oriented, but you can also run normal Linux and they would use like fastboot or something and there are like tons of other bootloaders out there. But I would definitely say like the big ones UEFI and U-boot. So I gave a similar talk a few weeks ago and someone asked, so what about non-free firmware? Like can I run that stuff purely with free software? And so the thing with ARM is there are a lot of different platforms. So I'm not sure about all of them, but some of the platforms I looked at, yeah, you do need some, there is always something proprietary. So in a lot of cases, I see where you have at least a proprietary, like a first stage bootloader. So Raspberry Pi is an example. So I don't actually have a Raspberry Pi, but the way I understand it is you basically put some boot files on an SD card and they are proprietary and then they load the kernel or in case of when we support it in Debian, that first stage bootloader would basically be used to run U-boot and then we can use U-boot and the U-boot is U-boot support. All of that is free software, but that first stage bootloader isn't. I've heard that there are some people working on a free replacement for the Raspberry Pi though. With NVIDIA Tegra, which is something I'm working on at the moment, you also have a tiny first stage bootloader, but then again you have U-boot, which is free, and in that case you also have some firmware images for the GPU and for some other stuff. For the Dragonboard, so the Dragonboard sounded really interesting because it actually has a graphics chip which can be used with free software. So that sounded pretty cool, but again it has a proprietary first stage bootloader and then there is a second stage bootloader which is actually open source and then there is actually U-boot as well. And then you have some binary blobs which also need to be stored in Flash to work properly. On the Marvel side, I'm not really aware of anything. I've never needed to flash anything proprietary, but I'm not sure how U-boot gets started on Marvel, so maybe there is something. So the future, so Nest devices, like I said, that's been something that's really been popular in Debian, especially on the QNAP devices. So the problem is, so QNAP, so the devices we currently support are pretty old nowadays and they have some newer devices, but they are not properly supported in the upstream kernel, so I have no plans to support Debian on those. I recently did some work on some Seagate Nest devices which actually pretty interesting, but again they're a little bit out of date already. And then there is the whole 64-bit, so I think people are really waiting for proper 64 servers. I think there's going to be a lot of work in that area. And then there are all those development boards, Raspberry Pi, Pyn64, Allwinder. Basically all of them sound exciting, but if you look at them, all of them have some upstream issues, so it is really quite frustrating at the moment but I think things are really moving quite rapidly. And then there is this 96 board initiative which is actually done by Linaro, and so Linaro supports Linux on ARM, so you would think, wow, there are some really nice boards coming out, and they differentiate between the consumer edition and the enterprise edition, and so I bought some consumer edition boards and it's really horrible. So first of all, I had to spend like half a day just getting the components because it uses like a non-standard power supply, a non-standard serial console, so finally I found all the pieces I needed and then I was expecting, well, it's from Linaro, so surely everything just works upstream, but it doesn't. It was like, yeah, I can boot the Linux kernel but there is no USB, there is no Wi-Fi, no nothing. So yeah, that's a little bit frustrating. On the enterprise edition, I think that looks more interesting, I think that's more standardized, but again, there have been some delays. The clue is all upstream except for PCI. Yeah, already. Okay. Yeah, so that would be interesting to see. So the questions that I actually have nowadays is, so because of those changes, because of having a modern U-boot on most of those new devices, having a U-boot with distro support, having a kernel which, one kernel image which works on all those devices which are supported, it's really easy to support a new device as long as it's supported by the kernel, but at the same time, I think that's a big challenge for Debian because so right now, if you look at ARM-HF, we basically say, well, take the installer and if there's a device tree, it's probably gonna work, but what does that mean, right? And on the one hand, we have some devices which really work, which means, so we have vagrants having U-boot support in Debian, we have people testing Debian installer, testing the Debian kernel, so we have devices which are really well supported and then we have some devices on the other hand where well, yes, there is a device tree, but no one has ever tried it and at the moment, we have no way for users to differentiate those use cases, like those support levels. So I'm wondering if we need like a table, somewhere where and maybe some support levels like green, yellow, red, like green is, yeah, we have Debian people who testing that stuff, yellow would be where we have heard some reports that it might work and green is either it doesn't work or we haven't heard that it works, maybe we need something like that, but right now, I see from users, I wanna run Debian on my ARM device and they don't know if it's gonna work. Yeah, Ben? Yeah, so Ben is saying we also need to track what's the earliest and the latest kernel version that has been tested. Yeah, so I think we really need to come up with some criteria to have, you know, define those kind of support levels and indicate that. And yeah, the other question is related, so with all those boards, how do we actually test them? So I keep, and it's really one of the things which is great, but also annoying with those boards, they're so cheap. So it's like, oh, there's a new board, it's like 30 bucks, oh, I just get it. And then you have like those pile of boards and you realize, well, I don't have time to test all of that stuff. So I think that's gonna be a real challenge. Like they're like 100, I mean, it's really hundreds of boards coming out and how do we support all of that? So I think that's the question I wanted to raise. And maybe that's something we can talk about in the both. But yeah, I'm obviously open for questions now. Hello. Yeah, so I don't quote me exactly. So I believe Red Hat has this problem as well, obviously. And I mean, they're obviously more interested in the ARM64 only server stuff, but I believe the way they're going about it and maybe it's something you could collaborate on is they're trying to make sure and push all the vendors to be standardized because as you pointed out so elegantly, it's just not, it's crazy with all the different device tree differences and so on. So I believe their strategy is to work with the vendors and require everything to be upstream and no device tree, specifically to push everything to just boot with one kernel and so on. So maybe that could be, you know, we sacrifice a few of these shitty boards but work with the vendors that make the funds that are upstream. Well, so I might be mistaken, but as far as I know, Red Hat basically says we only support UEFI and ACPI. And that's fair enough. And I think that works for them because that's like the server world which they care about. But I think in case of Debbie and there are so many boards out there which don't meet those specs and we sort of live in like the real world. So I don't think we can, well, I mean, it's different. It's, I mean, if Red Hat wants to target, you know, the server people, like the people who actually have money, I mean, that's fair enough. That makes sense for them. But Debbie and we run everywhere. So I think we need to support all those weird cases as well. And maybe if it gets too weird or if, I mean, I put in some work on the Dragonboard and then I realized, why am I actually spending that time? Because no one, like I've heard from no one that they have that board. I mean, like a lot of people have the Raspberry Pi but I've heard like pretty much no one has the Dragonboard. So maybe I'm, you know, it's not worth my why. But if there is a board where people, you know, want to use it, I think we should support it in Debbie and we do have the infrastructure. So we, you know, it doesn't have to be UEFI and ACPI. We can support other devices because we've done it before. But I definitely agree with like the point about getting more standardization and that stuff. So I agree, yeah. And the whole, I mean, the whole distro support in UBOOT that has really made things easier for us. I might be gonna say some of the team things Steve was anyway. So yeah, on the IP versus DT thing, it's more a question of quality of implementation of the firmware and of the upstream support than it is. I mean, DT isn't fundamentally worse for upstream support than ACPI is. Red Hat have some reasons for this of varying degrees of sensibility but it's not fundamentally about me. It doesn't fundamentally stop people supporting it, especially in a community distro like Debian. I was also gonna ask Martin, have you talked to people like Kernel CI about the testing and coverage stuff? Yeah, not really, but I think there are some things we should look into. Yeah, because the whole, obviously the whole making sure the kernel boots on random hardware stuff is exactly what that's doing. And there's at least infrastructure there that might be nice to play with. Yeah, that's a good idea. So on the whole 96 boards thing, sorry. Most of the engineers involved are totally aware of how awful it really is. We've tried to tell management they don't want to listen. Yeah. So there's been a huge amount of pushback saying, you know, Will and Noah, we should be making sure this is all upstream. Yeah. No, I'm aware. It's just, Linaro has a really good brand. So people expect, you know, when I saw 96 boards, it's Linaro. So obviously that stuff is gonna work and that's why I was really disappointed. One positive thing there is that the first EE board, the SOC does basically work upstream already with current, it's not like there are some patches that don't need to go in, apart from the PCI, it should basically work. So hopefully that's gonna be a much less spectacular story than the consumer edition boards have been, because they're unfortunate. Yeah, the first EE board, the cello board is due to start shipping really, really soon now. There are examples already out there with engineers for validation. Literally, we're talking in the next couple of weeks is what I've been told. That's the first 96th board I would actually spend any time on at all personally. The CE boards, I would joke. I'd forgot what else I was gonna say. Oh, yes, and then, I guess we'll be going through quite a lot of this again in the on-buff later on in terms of which things we support. We'll carry on the discussion. Yeah. Is this working? So, we've gotten all this great support for distro boot command and U-boot, and now they're starting to get to UEFI emulation in U-boot, so even for boards that don't have UEFI emulation or UFI support, if they support U-boot, it might work. Although that means all the standardization we've worked towards, we throw it away, but yeah. Yes, I think that's something we need to figure out. Which one do we want to standardize on, or is that something we want to leave up to the user? But yeah, I mean, I guess it was sort of confusing because there are like those different ways, but what I really want to show is that things are getting much more standardized and much better. So it used to be really pretty horrible, and nowadays, if a platform uses a modern U-boot, if it has upstream support, then it's actually pretty trivial to add Debian support, and I think that wasn't the case in the past. I think Ben had another one. Sure. So I'm interested to know how much interest there is among ARM portors in graphic support, GPU support, rather than selling sort of dumb frame buffers, and how many of these boards, because quite a lot of these boards do have HDMI ports, and I don't know to what degree they're actually supported. I mean, I have the current product at work, which definitely is using the GPU on our OMAP system, and the what's in Debian, it's a Debian based system, but the Debian, what's in Debian does not support GPU. Well, so I can't speak for everyone. So I've personally done some work on the Tegra recently, and NVIDIA is actually working upstream on new boards before, so I'm really interested in it, but I think a lot of the boards use those Mali GPUs, and I'm not sure what they, maybe you can give an update. Can we get a microphone for them, team? So on the Mali front, there has been a lot of history of OM being scared of releasing any of this in any way, shape, or form, without huge EULAs and all that kind of stuff, so it's not been redistributable even at all. We're a long way from it being releasable free. There are packages coming that should be shippable and non-free at least, if you want to get full acceleration on your Mali stuff. Hopefully that will solve some of the problems. There are a lot of people inside ARM who are very, very keen to support it properly, free and upstream. It's a long fight, it'll take a while. I mean, most of the Mali driver developers themselves I've spoken to would love to make it free. It just comes down to legal people being scared. Do we have some questions left? Yeah, thanks for coming. There is going to be the ampouf later today and Vagrant is also going to talk about his experience about running a built network on ARM HF-Born. Thanks.