 Hi, I'm Vagarin Cascadian. I do a small consulting business called Aikidov LLC. And I'm going to be talking a bit about how to use Debian on embedded systems and best practices to maximize your long-term viability with using Debian. I'll probably talk a little bit about some of the history of Debian. I'll be getting into some of the norms in Debian and some of the infrastructure in Debian to help kind of give you some pointers. I'll talk about some devices that I've kind of worked with getting Debian working on and kind of the successes, failings, and challenges there. And I'll be talking about ways in which Debian can be the ideal view of how you should be doing this. And then I'll talk about some compromises that you might want to make along the way that might make your ideal view fit a little better with some of the practical realities when people are making some hardware decisions. So when I started writing this talk, I realized I've been doing this stuff for 20 years, which was kind of a surprise to me. I've been involved in Debian in some semi-official capacity for about 14 years. Somewhere along the way, I got stuck with UBoot just because I asked to enable a board. And I've been doing all of the uploads for Debian ever since. And I've been working a lot on the reproducible builds, farm running a bunch of arm boards to rebuild all of Debian. And just a couple of years ago, I got my black belt and aikido. That may not seem relevant now, but I'll work it in. So Debian started almost 25 years ago from counting it. I think it was later in the year. So it has a huge repository of packages available from which you can just basically, if you can run Debian on your system, you have access to a huge amount of available software in the free software world. Not everything, but a lot of things. It also gets regular security updates for the stable releases. And it supports quite a few architectures, including many of the architectures typically used for embedded systems. So as embedded hardware is kind of edged beyond the microcontroller era and on to general purpose computers, Debian becomes a lot more viable for systems for a lot of these strong reasons. One thing that kind of sets Debian apart. Oh, you can't see the top. All right. Well, there's a Debian social contract. It kind of guarantees some things to the users. And these things, typically, it's going to remain free. You can use it, modify it, redistribute it, and we'll get to that a little bit more later. Debian tries to give back to the community, tries to do so in a transparent way. And we love our users. And there was a pragmatic, somewhat controversial compromise in Debian where, officially, everything in Debian is free software. But there's this little section we tend to shunt off to the side and kind of support it that includes some components that wouldn't fully meet Debian's typical freedom guidelines. So the social contract, not every distribution has a contract kind of like that. So they basically just say to you, this is what we're doing. This is why we're doing it. The Debian free software guidelines kind of spell out what we mean by free software. It includes things like using, modifying, redistributing the software. And this stuff can really help avoid vendor lock-in, which to clients, sometimes, that's really a valuable, valuable feature. And it gets into some more details. And the Debian free software guidelines were used as the basis for the open source definition. So they're very similar with a slightly different angle. But one of the things that really sets Debian apart from a lot of distributions is they have a very strong policy. So it has the Debian policy, which really describes mandatory requirements in order for a package to be in Debian, and also best practices, which are kind of optional, but strongly recommended for it's suggested practices you should do. So what this gives you that not every distribution has is when you install a package from Debian, you know how it's going to behave. And if it's not behaving that way, it's a bug. It's not some feature, or it's not some, well, I wanted it to do that way. So things like packages can't overwrite the configuration of other packages. You install a package, and it shouldn't break some other random package on the system. That's a bug. Not every distribution really sees things that way, and it has some pluses and downsides to that kind of an approach. But the policy really gives you a somewhat of a guarantee of how the system is supposed to operate as a whole, which you don't necessarily see in a lot of other distributions. The release cycle has a number of kind of stable and stabilizing phases. So packages typically get uploaded to the unstable distribution, which is kind of a staging area where they migrate to testing. And then testing is basically a work in progress towards the next stable release. And once all of the release critical bugs are sorted out in a testing release, it'll typically be released as a stable release. And then it gets roughly a two-year support cycle and their efforts to extend that for I believe up to five years and maybe even beyond that. And then there's the old stable release, which was the previous stable release. They typically have code names based on Toy Story characters and they have numbers, which within the Debian community aren't really used a lot. So having to learn all of these Toy Story names may be a little odd, but that's how it works. There are a huge number of derivatives and or projects working within Debian as well, such as the Freedom Box. But they're basically, Debian has formed this basis that can be reused for other projects that maybe have a slightly different goal than Debian, but the guarantees that Debian provides allow it to be a really useful framework for other projects of note, since this is the Embedded Linux conference. Armbian is a really interesting one. For the most part, it's just straight Debian with possibly a different boot loader and a different kernel. So that's basically some of the approaches we'll probably talk about in a little bit. Reproducible builds is something I've been really involved in. This kind of is recently added to the Debian policy and this can ensure that a given package will actually, it gives you a trust path to figure out when you're building this source code, it results in this binary, and it does that by having arbitrary independent people able to rebuild it and come up with a bit-for-bit identical package. So that's something that more and more distros are working on and that's really exciting, but they're not really there yet. And neither is Debian, but we're working on it and we've come a long way. So in Debian, related to embedded systems, a really important package is the Linux package. So Linux, there's a tracker URL, tracker.debian.org, and that will take you to a webpage which kind of gives you an overview of the status of Linux in Debian. Typically, the versions of Linux in Debian are for the most part taken straight from upstream and then they remove some binary blobs and there are a few other Debian-specific patches, but for the most part, it's straight from upstream and if you want to get any new features into it first, you have to get them into an existing branch of the Linux kernel. I'm pretty much the only current UBoot maintainer in Debian. So similarly, we usually backport patches from upstream, but I have taken the liberty of occasionally testing patches out in Debian first and then submitting them upstream, but typically we really like to only deal with backports and not implement new features. Arm-trusted firmware has started to become a bit of a thorn in our side in Debian. Not because of anything inherent, the licensing actually looks pretty reasonable, but I haven't found many boards that actually work with the mainline Arm-trusted firmware. And there are countless vendor forks of this project and ironically, the first Arm-trusted firmware that actually got into Debian is one of the all-winner forks of it. And this is really needed for just about all Arm64 systems. So that's a work in progress. Would really love to see more work towards getting boards supported on mainline Arm-trusted firmware that would really make the world an easier place for Debian. So another thing that, because Debian strips out a lot of the driver firmware, binary blobs without source code, there may be incompatible licensing. Some of these will be integrated into the non-free components of Debian, but some of them aren't or aren't yet or they're not readily available. So these things, how you install Debian is basically like many distributions, you can network boot it or boot it off an ISO in the Arm world, in the embedded world, maybe USB boot or off of a microSD. We've got images for netbooting, netbooting or SD card images for the Arm-HF architecture. And really, we get down to the key things of it needs to be supported in mainline Linux. It needs to have a free bootloader and it needs to have system or driver firmware available. So that's kind of the ideal world. We want all of these things merged upstream. So I'm going to talk about a few kind of case examples. In fact, this device I really enjoyed working with a long time, it's got a great keyboard, it's got these little mouse controllers and it really everything just fits at least my fingers quite nicely, but I kind of got stuck in this cycle for a number of years where it was based on a Linux 3.2 kernel and at the time that was the current Debian stable kernel. So I thought, okay, it's got a few patches but they're working on doing the right thing. So a security update would come out. I would rebase the patches against the security update. I would build it, I would test it, maybe something wouldn't work and then I'd have to do it again maybe just two days later and this started to wear on me but eventually the SD card on which I was running Debian, it just failed and I said, okay, I see that there's some stuff in mainline for the Open Pandora. I'm not gonna use this on this crazy old kernel anymore. It ceased to be a recent kernel. So the upstream support on the Open Pandora, well, because it is a small keyboard, it's got a function key to be able to get to a lot of keys and that didn't work. The mouse knobs, I couldn't get those to work. The Wi-Fi didn't work, that's kind of hard to live with because it's not like there's an ethernet jack on this thing and USB still didn't work. I've never really had USB working on this thing even with the image that it came with and the U-Boot support, all you really had was serial console output which you have to plug in an adapter so if I need to, if I do the kernel update, I need to make sure, oh, I've got another computer on hand so I can do a serial update but I've taken this thing traveling without having to deal with a laptop and I've done development work. I've uploaded packages to Debian that while doing development on this thing, I mean, it's really a nice piece of hardware. I was really sad to give up on it but that's basically what I did. Another board I started getting involved with was the Beaglebone Black. A number of years ago, it used to run an Anxtrum-based distribution and for whatever reason, the main person maintaining that disappeared and they're like, we should give Demi a try and so somehow I got involved in this and they said, well, we've got patches based on the 3.2 kernel and I'm thinking, oh, I've been doing that for my Open Pandora, that should be easy and turned out they were for a different variant of the Beaglebone. I think it was the Beaglebone White. They had their own Benderfork of Uboot. Then I saw that Uboot was supported in Mainline Uboot but the version of Uboot in Debian was several years old so I was like, hey, can you enable this? And that's how I ended up maintaining Uboot for the next few years. But now the support from Beaglebone Black is pretty good in Debian. They've got most of the drivers working. Uboot support is reasonably well supported but the Beagleboard.org foundation is still shipping their own image with a custom kernel and a custom Uboot with a bunch of patches which they've just recently tried to push some of upstream again. So the Beaglebone Black is actually turning out to be reasonably well supported in the end. Didn't look great at first but they were doing the right thing and that's great. I also got a few of the little chip computers which I don't know if you ever tried to do a search online for chip but wow, that's hard to find. So that was kind of unfortunate although pocket chip was a little easier to search for but on the early models which are the ones I got because I got all excited and got on whatever crowdfunding platform it was and supported it but the NAND is not really supported on Mainline Linux and it doesn't look like it ever will be. I would love to be proven wrong there. The vendor bootloader doesn't boot a kernel with a raw NITRD which some of the distributions tend to like that feature enabled and this is a common thing in vendor Uboots is they won't, they'll enable just the features necessary to boot their particular variant of the operating system and then I don't think the Mainline Uboot supports the NAND on board and there's not really, it doesn't actually even have a microSD card or anything so you can boot over USB and then the only file system you can get access to is like a USB stick or something which makes it kind of hard to be a mobile device but so this device, I don't know maybe someday it'll improve. There are a few of them out there but these are kind of some of the stories that they didn't quite work out or they worked out to some degree or eventually they work out. So you get to a situation when you're developing when you're developing software for a particular board or a particular system you kind of get to a point where you have a choice. It's like you can either engage the community but that might involve some sort of conflict, right? Like you're like, well, we need these features enabled, we need that but what people usually do, what projects often do is they just run away and they kind of go do their own thing in a corner somewhere but this has some drawbacks. I mean I'm glad people aren't like getting, you know, getting ready to do fisticuffs over like getting their support for their board, you know, fighting on the mailing list and creating lots of flame moors but running away has its problems too. So I like to propose a different approach and that's to engage, to be physically engaged through like an art like Aikido or typically you're aware of the situation, you're present, you're fully present in the situation but you blend with the forces coming at you, you're involved, you are, you know, they'll grab your wrist and you might do something, you don't actually want to let go and there comes a point where you just got to roll with it, you know, you got to be engaged but you've got to just like move along, you know, work with what's there and be involved. So I would like to propose some kind of middle road approaches that can help prevent a long term problems that we've seen from innumerable boards, I just mentioned three boards but, so the kernel is a really key component. I mean, basically if you can boot a kernel on your board and you have Debian as a user space, you can run just about any kind of board on that as long as it's supported in, as long as it's an RMV7 board or if you use the really old Debian ports, the RMV5 port but a nice middle road is to basically take the Debian stable kernel and then backport patches from mainline to get it supported and there was a talk earlier where they actually talked about how you, that actually optimized and improved their driver code by basically pushing everything mainline and then backporting the features and that's a really excellent way to approach it and then you can kind of have a stable baseline from which to work from or do a custom kernel but continually have a plan for how to keep stuff working in mainline so that you never get stuck on an old version. Similarly, the bootloader, it's the same kind of thing. There are countless forks of U-boot for tons of different boards but in the end that version, every once in a while we'll get somebody, we've been, I think U-boot's been using date-based versioning since 2010, 2009, I'm not sure. A while now and every once in a while we'll get somebody who comes into the IRC channel with like a version 1.1 something, right. Well that's old and then with the date-based version you can see, well, it's exactly that old, you know. So we see even really new boards getting released with like versions of U-boot from 2010. So this stuff isn't sustainable in the long term. Would really like to see people focus on pushing things upstream. I've been involved in some of that and then in the x86 world, a lot of people don't really even think about the firmware. They don't even have a choice really. They've got this BIOS on their machine and maybe the vendor provides updates, maybe they don't but it's a little different in some of these embedded systems where you actually do get the source code to actually replace your boot firmware and we should keep that current so that when unforeseen security issues come up we can actually, that require something at the boot loader or firmware level we can actually address those issues. Another thing, a complaint I've heard about Debian is sometimes the packages are a bit old or a bit out of date or maybe not in Debian but you can follow the stable distribution with selected versions of packages that are relevant to your particular use case or you can include just the selected subset of packages that aren't yet in Debian and that way you can focus on maintaining the actually value added part of the product, whatever it is you're working on and that'll give you a pretty good approach to not over commit yourself. A lot of these projects don't really need to maintain a whole operating system. There are lots of operating systems out there already and I'm talking about Debian mostly because I know Debian but there are other choices and really focus on what is your particular projects specialty so by doing some of these approaches you can really bring that out. I'll talk a bit about the Debian community. There are over a thousand developers in some official capacity depending on what exactly criteria you're looking at and then thousands of additional contributors above and beyond that. For the most part individual packages are maintained by an individual or by a team but it's not an absolute. They're not the only people who can upload that. Any Debian developer can upload any package in the archive technically. There are some social guidelines around not doing that for anything, for any random reason. You kind of need a reason to do that but in general it allows some flexibility between targeted maintainership and the broader realm where if somebody's on vacation we can still get a critical security update out. There are of course online resources available mailing lists which have historically had some reputation for being a massive flame wars but I think in the recent years that I've been involved it's been less so given a few exceptions but IRC is also a great place for more real time communications. They're irc.offc.net or irc.debian.org which is just the C name for irc.offc.net but there are a number of Debian based channels on IRC and then of course a wiki which as many wikis go it's not necessarily well curated. There's a lot of stuff that's quite out of date. Some pages are really well maintained so that's also one way that other people can get involved is if you see stuff that's out of date fix it. That's a huge ethic actually in the Debian community is it's an ethic of duocracy and that's also something that can really if you're an outsider it's not that hard to become an insider and guide Debian in the direction that's useful to whatever your project is. So there are numerous events every year there's a Debian conference and typically hosted in a different city every year. A few years ago we hosted it in Portland this coming year will be in Taiwan and then numerous smaller mini conferences some of the mini conferences like in India are as large as the official conferences but and then we'll have bug-scrushing parties where we might try and get a release out the door or trying to move some new feature over that's been lagging. So I kind of referenced this tracker I just wanted to give you an idea of the kind of information you might be able to see like by going to that page. So this is for example the Linux package and it'll basically show you the versions of the package currently in Debian gives you links to the bugs, build logs numerous other kind of quality assurance checks. So this is a really great URL to know if you're trying to if you are using Debian and you've got a particular piece of software in Debian you can pretty much go tracker.debian.org slash package name and it'll get you to a summary of information about that package. The web interface to the Debian bug tracker is actually read-only, which you know this thing, I'm not sure how old it is exactly but it was kind of around before this whole web 2.0 thing happened and so basically you can browse the bugs and do all that sort of stuff but really when you get down to it it's an email-based system which for my workflow actually works really well I can do things offline, I can queue up mails I can develop bug reports I can download all my latest bugs to annoy me and then read them on a train or a plane or something so you can actually do quite a lot offline and it's a pretty simple format I mean you just email submit at bugs.debian.org make sure there's a package colon and name a package in the body of the message ideally a version and then there are numerous other kind of tags and flags you can set but so for some of you this may seem like an antiquated system but it still has a lot of merits and this is how you get information into the bug tracking system so I've been suggesting that there are ways you can get involved in Debian of course you can report on features or bug reports that's a pretty good entry level that's certainly where I got started some innumerable years ago and it's really not that hard to become a maintainer or developer of a project within Debian if there's a piece of Debian that is useful to whatever other project you're using Debian with typically people are pretty happy to have help and as I said earlier it's a bit of a duocracy so those doing the work define what happens I mean so if you need something done in Debian in many cases you can typically jump in and do it so there are and if you've got a reasonable coding skills background or whatever if you're familiar with revision control systems and at least one programming language you can typically jump in on some project and make a really useful contribution there are also other ways for non-uploading contributors people who help with editing the website or planning conferences there are numerous ways to get involved but given that this is an embedded Linux conference I figured it was maybe targeted a little more towards the technical end and so if you already have a lot of those technical skills the process has been pretty streamlined to help get people engaged so I think this talk was used in one of the slides or one of the this slide was used in one of the talks I mentioned earlier so thank you Wikipedia but this is actually salmon in the Willamette River just like a few blocks from here and the salmon in this region there's a long history thousands of years of salmon swimming upstream they actually contribute oceanic nitrogen deep into the inland country the forests of this region are basically there because partially because the salmon kept swimming upstream and they would they grow up in the tiny little streams and then they swim out to the world and they do their interesting things eating other ocean fish then they swim back upstream die and contribute everything back to the place where they were born I think we should be doing this in technical projects as well you know maybe you need to go off and do something that's not quite appropriate for within Debian but if you bring it back in I think we'll have a whole cycle that really can avoid some of the problems that we've been seeing so I'd really like to thank Linux Foundation for hosting this conference they also sponsored the Core Infrastructure Initiative which funded me for a lot of work on the reproducible builds project which is sort of related to this and Debian of course and there are so many people pushing things upstream and I want to just say thank you that makes as a Debian developer that makes a lot of my work a lot easier so let's see obligatory licensing and questions how does this work I didn't know I was gonna get a microphone this is pretty cool so you talked about some of the boards that were not working out so well do you have any favorite boards that you like running Debian on? I've been really happy with the WAN board series they've just been on the reproducible builds farm they're not the fastest of boards we have but they've been really stable so that's one of my favorites for a somewhat newer board I just recently got a dev board from Thio Brahma Systems it's the Puma RK3399 and I haven't spent a lot of time actually working on it but the board is really nicely labeled with like switches instead of random jumpers or having to short like connectors which I've had to do with some boards just to get them to boot correctly so but probably the one that comes to mind first is the WAN board for something nice and stable it doesn't really require much external firmware or anything which is nice and it works reasonably well out of the box on Debian how many people have worked on a project using Debian? okay I guess the title of the talk might have biased the audience a bit there cool yeah yeah any any war stories from a project you're using Debian on or I see a hand back there hey um so I felt this talk was mostly about evangelism for Debian but I was hoping if you had like some concrete points about best practices for Debian on embedded systems okay like what are some of the takeaway messages real takeaway is get your stuff upstream I mean that's really to a large extent on Debian if you have upstream support in the Linux kernel if you have upstream support in Uboot if you have your boot for boot firmware available upstream Debian basically just works so in that sense yeah I don't have a lot more to say it's kind of stressing that point and there have been a couple other talks talking in a similar vein throughout the conference so yeah okay thanks so if I want to use newer stuff in Debian stable there is something called backports.org can you speak about that a little sure yeah I'm glad you brought that up that's kind of like an in-between place between testing and stable so backports there's backports.debian.org is a site where basically they'll take newer versions from the in-development release of Debian the in-progress release and rebuild it so that it's compatible with Debian stable so I think even I should have mentioned that more explicitly yeah so basically yes I should have included a reference to backports.debian.org on this slide that's a really great service you don't get quite the same security guarantees with backports but in general you get a pretty good process and contributing to backports.debian.org is something you can really a way to really help get all of your stuff integrated if you do need newer versions of software there's also a site called mentors.debian.net or is it .org I'm not sure off the top of my head that allows you to upload packages and then request sponsorship for not just backports but any part of Debian so you don't actually have to become a developer in order to get things into Debian but backports is a great starting place where you can if you need to learn the particulars of Debian packaging that's a nice easy way to sort of contribute yeah thanks that was a great question I should have included it in the first place last call all right what can you say about how to formulate this it can be frustrating to work with a lot of packages because the packaging format as it's always been is tar balls plus patches I know there's been there was a a Git variant of source packages for a time that seems to have stayed experimental or died right I know that a lot of Debian developers use Git repositories and this new GitLab infrastructures coming in place um what's going on in the community these days when it comes to source package formats and is there ever you think we'll ever see a time when the official source package format is a real Git repository like everything else in the world and not something that's you know ultimately gets converted to tar balls plus patches which are right kind of hard to work with yeah there is an interesting project called DeGit I don't know if you've heard of that but that basically allows you to kind of do that kind of a workflow and then the packages that get uploaded as a result of that are are then imported as like I don't know if it's a commit or a tag in your Git branch but it basically is a workflow where you can basically just commit to your Git commit and then you just you use the DeGit tool to upload and then it'll prepare the source package and it'll prepare all those artifacts that you don't kind of want to deal with um so there are people working on other approaches but yeah the DeGit source or the Git source format as far as I'm aware nobody's actively working on it and like I said Debian is very much a bureaucracy and it's also thousands of individual interests all kind of working together but also kind of working independently which is both a strength and a weakness of Debian so I can't really promise you anything any sort of timeline or anything I haven't heard of anything in the works to really change that angle of Debian but have you seen anyone try has it been a discussion that resolved in some not in recent years we have this general philosophy about it and yeah and I haven't seen any recent activity on that um but yeah uh I mean the biggest change right now is we we're basically the only official hosted infrastructure is all going to be Git based we used people can still use whatever revision control they want but the the provided hosting infrastructure is just going to be Git based from now on and it was the vast majority of packages already simply using Git so so sorry to not give you better news but don't think that's the status of it thanks oh uh somebody mentioned Git build package yeah there there are a handful of tools there's Git build package there's Git DPM there's D Git and they all basically serve as a way to generate the tarball plus patches that we then upload to the archive and then the build machines go and rebuild all the packages I've been given this microphone but I think the speaker just went over what I was saying so thank you okay any other questions or comments or hopes for the future all right I have a general question since I'm already holding this which is do you think um especially given the slower release cycle of Debian as compared with many other distributions I haven't gotten the impression that the community as a whole particularly values supporting embedded platforms which is just because the landscape changes a lot faster in the embedded world do you think that understanding is accurate or do you think it's changing or so on well what's really interesting about Debian's workflow is we produce the stable release but the majority of developers are on a rolling release you know so we kind of have a bit of both which actually I think can actually be a positive thing it gives us the time to like really go over and make sure everything's complying with policy you know having those releases I'm very partial to a stable release I kind of have a bit of a system administrator background but I do think Debian actually has a nice blend of the rolling release and stable release kind of format because packages are constantly rolling into testing you know it's it's maybe not technically a rolling release or whatever but it has a lot of the same properties so so yeah that that that's a whole another approach I was definitely recommending targeting stable and then basing your stuff on that just to give you so that the rug doesn't get pulled out from under you in the middle of your you know development process but if you would rather go with newer stuff definitely targeting testing or even unstable in some cases may make a lot of sense just as a response to the earlier question about Git I know of two arguments against one is that the cryptographic provenance checking for current tarball based Debian packages is considerably stronger than that is that's used for Git and second is that when dealing with several layers of derivatives out of Debian it is sometimes easier to rebase by having separate patches in Quilt than doing it in Git thanks for coming everyone and I hope you got something good out of it