 Okay. Yeah. So, hi. Hello. My name is Andre, and I want to talk to you about how to make off-the-shelf distributions better work on single-board computers. Who am I? First name is Andre. This is French. The last name is Chivara. This is Polish. I'm actually German and live in the UK, but I'm proud to be here today in Brussels. So, this is kind of two talks, question to one. So, apologies in advance if there will be a tough right in the parts. If you don't get everything, treat it like Star Trek, like you don't understand what we are lying in the warp coils really means, but you will enjoy the show anyway. So, the first part is about booting. Basically, what is the situation today? What are the problems and how to solve them? And the second part is about how to improve Linux kernel support and make it easier. And, yeah, also what is in there, why it takes so long, what can we do about it? And therefore, feel dauntless to give a demo, maybe. Yeah, so I'm with ARM, limited in Cambridge, and during day-times work on serious kernel stuff. But this is not an ARM limited story, so this is all the stuff I do when the sun has set, right? So, this is about SBC single-board computers, think recipe pie, but not particularly the recipe pie, more the other stuff like orange pie and someone said fruit pie is a funny name. Yeah, so everything that has this form factor, maybe a bit bigger, has an ARM core. It's not about service. The SOCs we usually see are from all-winner, a Rockchip, Mlogic, Marvel, Realtek. Those small, cheap SOCs which enable basically those cheap development boards. Also, this is about DT, in particular. DT Device Free, which is a hardware description, aiming at generic voice support. It's not about ACPI. Yeah, also when I talk about firmware here, I mean board-specific, low-level software, possibly including the bootloader, but not more. So, firmware sometimes, if you look at rooting phones or something where firmware is the whole thing, including the kernel and including user-land, this is not so firmware. It's just maybe the bootloader, but not the kernel and specifically not user-land. And this is about mainline, not PSP. Don't get me started on that. It's just nothing here. So, that's the current situation. That's kind of table I tried to make up how to find out what distribution supports certain boards. I found it pretty tedious to actually fill this table. And then, so, actually it was probably wrong because the thing you find is outdated and stuff, and then everybody says, ah, but we are supported anyway, but the sheer existence of this table is a shame, actually. So, the point is that we have, you see, basically, what Stable also showed, you have some distributions specialized like Ambien, which actually support quite a lot of boards, in this case. Some distributions don't, and that's not bad. But we'll show later, they actually do the right thing. And there are some boards, which are more popular than others, and better supported. We're missing the kernel versions. So, one thing, the one thing, this and that thing. So, the point is that this table is only a short excerpt, actually. So, it should be not five columns, but 300, which is the number of distributions that DistalWatch gave me. And it should not, like, four boards, like, I think, 50 to 100. That's what we aim about, those SBCs these days, if you look at Ubud support. So, the problem is, this table should not be about distributions supporting boards. The actual technical dependency is that the kernel that distribution uses supports the SOC for that board. And once you have that settled, you don't need special distribution support for a separate board. Why is that? So, what are the problems? Three things. Traditionally, there are no well-recognized standard ways of putting the ARM board, especially on 32-bit. Yeah. First thing. Second thing, many boards come without onboard storage, which is really annoying. So, there's, by definition, there's no firmware on the board. Can't be. And the third thing is that with the current model, how we support those things, the distribution has to ship the board DT. So, the distribution has to explicitly kind of support this board somehow. Yeah. So, what can we do about it? The first thing about booting. I know that many people have seen the logo, wow, UV, let's grab and laugh. Yes, we could do some Ubud magic properly to make this work better. But if you remember the XKCD strip where we have 14 standards and let's invent something that covers everything and then now we have 50. So, we just use EFI. It's there. You don't need your asbestos trousers for EDK2 anymore. You can use Ubud these days because it implements it. It implements it well enough. UFI is widely recognized and supported. That does not mean it's pretty, but here it is. It's there. Most distributions support it anyway, especially on M64, because it's the way the servers work. So, you get it basically for free. Once you have UFI in a firmware, you can boot the distribution like this. No questions asked. Mostly distributions use grab EFI and then load the kernel wireless, which shows a good news because grab is well supported in every distribution for the X86 site. So, all the stuff where you boot menu and kind of in-head IDs and comatline options that's already covered, just finished. And the good news is it actually works already with Ubud. And you can boot default ARM64, EFI, and store images. Okay, there's that card. Second part is about spy flash. Many socks that I listed actually can support boot from spy flash. They are cheap and they are not too big from footprint. The idea is that if you have spy flash on the board, you can put the firmware on it. And that's something that changes the game totally because it allows to keep the firmware A separated from the mass storage where some partitioning tool may accidentally mess up the thing because it installs a GPT partition table and overwrites your firmware, which sits in the front or something. It gives you easily network boot support without anything just because you can put Ubud and Ubud support PXE and everything works. And that's a cool feature. It actually allows to ship the device tree with a board. Just you put it on there and then it's like it should be. Also good news, Ubud supports mostly for most SSCs supports booting from there, loading from there. And even for all winners, it's the same image so that the same image can boot from MMC and SD and from spy flash as well. Yeah, so it's a small step for a board vendor, giant leap for the users, right? So now the DT story. Who actually brings the DT? That's some kind of question. These days it's the standard way of doing things is it comes from the particular kernel repository you are building. So a building, I don't know, 414 stable kernel and all the DTs that are in this repository then are shipped with a kernel. And the good thing is that by doing so, those DTs, you're sure that matches the kernel and that works and that they get reviewed and everything. But the problem is A that prevents supports for new boards. So everything, basically to get a board supported, you have to have the DT in mainline. Sometimes that just don't happen because there's nobody taking care about it because they just do whatever they want. Sometimes that happens but it's too late. So for instance, I mean the kernel is shipped by the distribution, right? So if you just happen to miss that window where to get it upstream, then your distribution will not ship the board. Although it could be perfectly supported because just the DT snippet that's needed and the SoCs actually support it. Also that requires every OS to copy those files. I always feel a bit pity for all those BSD people that do a really good job but they always eat the breadcrumbs that get left over by Linux, right? So they have to take the DTs and kind of match it and in this case they kind of copy the DTs from Linux so it's a really nasty situation. Actually, the DT describes the hardware so it should come with the hardware, right? It's kind of a logical thing. That's how it was actually conceived in the 90s with PowerPC and everything. So what does that mean? The idea is that the DB now sits in the actual firmware image that how this is actually implemented is a detail, I think. It can be easily part of U-Boot. U-Boot itself has its own DT that's usually appended to the end of the image and U-Boot is quite relaxed about what DT is so it can easily change the implementation of U-Boot to make it work with the normal DT that Linux uses as well and we do this already for many platforms or it could be provided as a separate file in some container so if you have a fit image which gets loaded by SPL you can put it there the point is that this DT sits in the firmware and boots up. The good news is that you get immediate support for new boards so once you get a new board that comes out so for instance this week came out the RockPro64 which is in Rockchip 3399 which is already supported, the ISOC is already supported Mainline but of course the board is not in 414 and not in 450. But if you could just create a DT, put it on there it could boot 414 kernels basically, right? Also it scales much better because now we don't we just have to, someone has to do it once for this board and that's done and that covers every distribution it covers BSD, it covers everything you get immediate support for other kernels. The kind of disadvantage is that it requires stable DT bindings so you can't just change the way Linux interprets DT or you can change it but you have to be sure to make it forwards and backwards compatible so that older DTs work with newer kernels and the other way around and that is something, I believe it's possible some people say nah, don't even try this but I think it's worth it. And something that needs to be, have a look at this if kind of board vendors start to ship the DTs they need to be reviewed somehow so I mean they can take the route to the kernel I believe in the kernel repository or some extra repository reviewed by the kernel community at least, but they should not be shipped by the kernel instead shipped by the board vendors. Okay so here's the things I propose so the first thing I ask for distributions to do less basically stop shipping board specific images I know that you want to certify your customers and make it as easy as possible and you bake an image which contains everything contains the boot loader, contains the kernel and contains your actual distribution but let's better combine the efforts into creating generic firmware images firmware, remember I meant possibly only the boot loader so in this case would be like U-boot something and then we can all work together and it's not like SUSE and Fedora and Debian also they all do their own stuff in a moment so this could be combined and those images should be distribution and operating system agnostic so think about it should boot FreeBSD as well those firmware images contain the device tree so actually there are many boards it's possible to actually ship multiple device trees with it so you could have generic firmware image which supports wide range of boards and then you have some way to select which board you actually have those firmware images implement EFI boot services that could be UDK2 if you like but you could just use U-boot because U-boot is already supported by those boards and there needs to be some kind of update mechanism ideally especially for the DTs because the DTs they change I said they change in color because new stuff gets supported and you want to benefit from this so it should be relatively straightforward to update the DT and board vendors if they haven't done so already they should add spy flash to their boards possibly even preload it so some board vendors do this already that's great the others should do this as well because it makes a whole difference I think I will cover this with a demo at the end so the second part just some things some ideas about how to improve the mainlining of SoCs of new SoCs in a moment the process is like if you have some new SoC then there's probably some first board using it and that gets shipped to some developer probably if the people are smart enough some kernel developer and then the developer starts looking at what's needed to support this new SoC and chances are that new SoC is not that different from the last generation SoC so you can probably go ahead and copy and paste drivers that most effects clock drivers, pin control drivers because they are the core of it and things so then they add also initial DTs as eyes so the core DT and the board DTS for this first board to the kernel and then it's the usual testing and discussion phase and then eventually if everything goes well it gets merged into the platform tree first that's about RC4, RC5 usually depending on the platform and then if that goes well it gets merged into Linux tree so the next RC1 then it takes the whole testing cycle let's get released into a vanilla kernel and then this kernel hopefully gets picked up by your distribution which may take a while so if you have bad luck and you just missed the window and your distribution releases every half year then you have to wait like five and a half months or something so it takes a decent amount of time I think kind of edit it up ideally if everything goes well it's 20 weeks to reach mainline plus the thing why is that a problem? so what can we do about it? one thing would be we make those boards a set of mutation available earlier so that's how proper computers work so if you look at Intel they really start upstream very early at a point where no one publicly has access to the board so they have time for about a year or even more to do the proper upstreaming and that's great to be honest I'm not too hopeful that this will happen for many of those small SoC vendors so that's something that should be aimed for and I think that worked quite well last year for Rockchip they started to upstream the 3399 really early and from a guy who was in the company which is basically what we need to do and that worked out pretty well so by the time the first board was released the SoC support was mainline that's pretty good but in case we can't do this we can look at other options one thing is that IP blocks are similar in SoCs and by having more flexible device refinings we could exploit this and also we can try to abstract something via firmware interfaces so let's look at this first more flexible device refinings in a moment there's this kind of habit that you have a compatible string which tells you which device it is and the driver hardcodes many things by looking at the compatible string so there's some kind of matching and then you say oh that means that there's many pins and the pins are mapped this way which is comparably easy to maintain and you can make sure that it always fits and it has some advantages but I think it would be better if you try to design more forward-looking bindings or try to work out what is actually really SoC specific and what is more like an implementation detail as I would say so for instance we had this thing that we fixed the last time is the DMA controller works out that the DMA controller from the old winner H3 is actually the same as the A64 and also the R40 the only difference is the actual numbers so which lines connect to which devices but it is covered by a device screen anyway and also the DMA channels are different so what we had in the table so far is oh if this is in H3 then it means 12 channels that was kind of written in a driver so it turns out there's an actual generic property for DMA controller bindings which is called DMA channels so by basically saying I put the number of DMA channels in the device tree and kind of am able to support with one driver basically the H3 driver can support this and we don't need to support the A64 specifically in the driver we just say it's compatible to the H3 but it has number of channels which is 8 in this case for the A64 and that for instance allows the R40 that falls into place because it has 16 channels but it's the same so I sent a proposal for pin controller which was a bit more involved that wasn't received too well yeah I hope that this will be solved sometimes so I have an idea to kind of bootstrap this wire U-boot and see if U-boot can use the new binding and then Linux can piggyback on that the second thing is using abstract firmware interfaces so for some less performing devices we might be able to just drive some wire firmware so firmware means we do some have some one-time component which is not really running all the time but you just call into like PSCI for instance and the cool thing is if there's some interface that requires only one generic kernel driver and you can hide all the sock details and firmware and remember you ship the firmware now with the device so you can basically hack it up in a week and ship it you can of course improve and update it but it would be much quicker and for instance there's STPI interface and SCMI is really great and then making up abbreviations and mixing the letters so if you look at STPI it's the same letters as PSCI right it's amazing so anyway this is a firmware interface which gives you support generic support for clocks gives you supports for regulators for device power planes which are something like you can easily just say please turn this device on and off you don't care about the regulator particular which voltage it has just say on and off also it's DVFS so CPU frag support all this kind of stuff where you don't need a table which tells you which certain voltage and frequency match together they're just exported by the firmware and you just pick one and the cool thing is that STPI is up mainline for a few releases already so by just providing something so I did a proof of concept implementation just hacked it up in ATF in an arm trusted firmware and that just worked and it gave me CPU frag support out of the box you can have sensors support temperature voltage current so you can easily read out even how much the battery is left or something that's also covered by that and yeah and new devices just fall into place you just implement the firmware part it's probably not much different and it's much easier than in a super generic and abstracted linux way of doing things because it can be very specific to this SOC yeah and then new support for that just shows up and that's especially attempting for the clock support which is in the moment very so specific and we had this problem with the H3 and the H5 which are pin compatible and very similar internally except for 32 and 64 bit cores but that's not so much an issue in linux but actually there was one clock more because you need support the msc controller got upgraded to support high speed modes so you needed special clock and one pin was more so basically we needed new drivers which then were in copy and pasted or kind of abstracted but it was just a shame right for this single thing a single pin and a single clock you need new driver support and older kernels couldn't support it although they got almost there yeah so what could you do you could do testing spread the word and engage in running this discussion so sometimes when I propose something there are a lot of people posing me it would be helpful people say yeah it's a good thing and it's something like we should do and we should not be to fearful and say I guess that will not work anyway so I guess in the end even if we try this and that fails we are not worse than we are today but there's a chance that we are better yeah so if you could help out to make sbcs behave more like real computers that would be great thank you let me try the demo because things do we have I hope so this is the pine 64 LTS that's a new version of the pine 64 the changes it has spy flash which is why I demoed this and it has an emmc slot which is not populated as you can see and feel like a magician can confirm there's no SD card in here so that's ideally how the board arrives you take it out of the box and it has ideally something on the spy flash it depends on the board when of course but for pine 64 they are very very happy to do this so actually that should even work with HDMI but that's a bit too much for the demo I guess so I do it by the old developers way of serial so so this is this is my hacking pre-boot firmware which I put on it ok so this is something that doesn't work yet but I do this so basically it boots you boot directly from the spy flash and you can then do stuff and the cool stuff that you can do actually you can download your distribution from the internet generic arm64 efi and install a ISO didiered on your USB stick and you put the stick on here and then you say ah let's you have to reset the USB because your board doesn't detect it let's just reset the board so this typing you don't have to do so normally you just put it up you get this and you let it time out and then it starts looking it's all the magic you boot scripts you see a boot AA64 efi so it picks up the efi grub this is an old Ubuntu stuff and you just say install and then the grub loads basically the installer and you get some error messages and voila you get the installer and then you can go installing yeah so that works already thanks actually you can download the image for the p64 for my kit up repository and I will populate with more images but it's basically a upstream mainline new boot there's no magic involved in it all right questions? okay cool would be nice to have to pre-populate the flash on all the boards there would be a way to go if we can manage the board manufacturers to actually do that so we can so for some we can some do it provide proper new boot, proper mainline based new boot instead of 10 years old Android port so the good thing is that the spy flash support for the older new boots is not so good so probably it will not work really with the old stuff so actually the only working stuff is the mainline new boot but yes that would be something that would be required but in the end you can always kind of reflash and those boards easily boot from SD cards and there are boards without spy flash I see that so chances you can provide generic images from SD cards which act as the same and then you can kind of go there yeah so what I have yes well that's the ideal situation you have one firmware image for one sock if it doesn't work well then you have multiple but I try to keep it as small as possible so what I did for A64 is F2 in a moment because of what you said about DRAM so one for LPDDR and one for DDR but the rest is kind of the same and for instance we can work out so what happens on the old Windows stuff the actual board name is stored in the SPL header so you can kind of while flashing this generic firmware image to your board you actually tell it which board it is which once you have to know say it's a banana pie generic images and it looks out and I have some wrote some small tool which kind of helps you with that so you get a list of this firmware image supports multiple boards namely those and those and then you pick one from the list and say ok I have this board please flash this to the firmware and then it can the SPL even can so what we do already in parts in the ATF we look at this name and then it's a bit hackish but we say oh it's a pine book please enable the following power lines on the power management controller to enable LCD for instance and you can do this before you actually have proper device support very early in the board because it's easy to do you had a question any easily added spy flashes oh yes so far so far that depends well it depends very much on the way they boot so on pine 64 for instance all the pins are exposed on headers so that's how I started this I put some board some spy flash for two pounds on ebay and put it on the headers with some cables and then it worked as well that depends very much on the board I think many boards don't expose the proper pins but sometimes you can do this yeah so hopefully it's not when it could so the whole idea is about do this with mainline things so base u-boot and the kernel basically or the mainline dts and so what I said it's kind of it's pretty easy if a sock is supported already to add board support and yes review is nice to have but if you do it properly it's not much risk to actually boot it up and if you screw it up it won't work and you will see and you can fix it and there's nobody accepts to be the first board the chip maybe to work fully out of the box but the experience could be much better and you avoid this like half a year latency which is annoying but this is something that needs to be worked out how we still make sure that the quality is right and there probably needs to be some review but it doesn't necessarily need to fall into the whole linux of okay we have RC1 and it's cut off and then we wait 8 weeks and then better like next time and stuff so that is okay yeah find me okay thank you