 Hello and welcome to this talk about Debian and Yachter project, a tale of two distros, one of which is not a distro. I shall explain. So I'm Chris Simmons and let's get on. So this is the obligatory license screen. I'll leave you to read this at your own leisure. This is a little bit about myself. So I'm a consultant and trainer. I've been doing stuff with Linux as an embedded operating system for over 20 years now. I've written a book on the subject and you can find me speaking at many conferences, including this virtual embedded Linux conference. I'm very happy to be here. You can also follow me on Twitter and you can link to me on LinkedIn. Okay, so that's me. Meanwhile, let's get on with it with a show. So this tool then is about Debian or Yachter project, which one should be you be using for your future embedded systems. And it comes down to discussion about whether we should be using off the peg or bespoke. So off the peg by this I mean using something like a Debian based distro. You can just grab a copy, grab an image, put it onto your board should pretty much work out of the box. And then you can install the packages you need. And you should be up and running in almost no time. Or contrast that with the bespoke approach where we build everything from scratch using something like your project, which means that we can tailor the system for exactly what we want. And we are in total control. So those are the ends of the spectrum. So let's have a look and see which way you might want to go. So Debian, I'm using Debian here as a bit of a placeholder for any other distribution that you may be using. So pretty much everything I'm talking about applies also to say Susie Fedora and your own favorite distro. But Debian is commonly used for embedded systems. And the main advantage of Debian is it's a binary distro. So you don't have to cross compile anything. You just pick up the binaries away you go. Also Debian is familiar to a lot of people in the guise of Ubuntu. Ubuntu is based on upstream Debian. On the other hand, we have YoctoProject. So YoctoProject is a build system that creates packages from source code using a tool called BitBake, which is the job scheduler, and metadata from both the YoctoProject and another project called OpenEmbedded. YoctoProject and OpenEmbedded and BitBake are three of a pair, three that go together. So I sometimes refer to YoctoProject when I should be referring to OpenEmbedded and sometimes I do the opposite. So Yocto and OpenEmbedded have been around for a long time. There are literally millions and millions of devices running software created using these tools. So the thing about Yocto is that you are using it to create your own tailor made distro. You decide exactly the rules for selecting the packages and building those packages that go into your system. And you should end up with the ideal system for your particular hardware. So binary distro easy to use and set up, but you are restricted to do the things that were decided for you by the builders of the Debian distro. Or build your own distro using Yocto, you give you total control, but you're going to have to pay some more time and effort or spend some more time and effort putting this together. So the next topic I want to look at then is the board support for both of these systems. This is kind of crucial. There's no point trying to use either one if it's not going to run on your hardware. So Debian supports a range of CPU architectures. Follow that link there for a complete list. I've selected the ones that I think are most useful for embedded systems. So we're looking at x86, 3264 bit and various versions of ARM, including the 64 bit ARM distro. So selecting the architecture for your board is kind of part of the job, but it's not the end of it because as well as support for a particular architecture, you need support for a particular board. So we refer to this as the board support package or BSP. So just to be clear, a board support package is the means of booting the board, which on ARM would be the bootloader typically based on U-boot. On x86 is the BIOS that comes flashed onto the board when you get the board. You need a kernel, obviously. You need the device drivers for that kernel that are specific to the board. So when I say kernel, that comes with a whole bunch of device drivers already built in, but there may be some additional device drivers you need for particular peripherals. If you're using an ARM based board, you will need a thing called a device tree. So the device tree is a description of the hardware. And that is obviously highly board specific. You will need to get that from the board vendor. And then finally, there may be some libraries specific to various components on the board. And in particular, I'm thinking here about the graphics, particularly ARM based devices. They all have or most of them have a GPU built in. And those GPUs are very specific to the silicon vendor. So Qualcomm use one, Rockchip use another one, NXP, the IMX chips use another one. So you're going to need support for that particular GPU if you're using accelerated graphics. So in practice, what have you got? Well, if you are building a system based on x86, things are relatively straightforward. So these are generally called embedded PC boards. There are many, many, probably hundreds of manufacturers of these boards. And they should pretty much work out of the box because it is kind of a small PC. The BSP itself then is really just the BIOS, which is flashed on the board when you buy it. The rest should be off the shelf upstream from Debian. ARM systems are more highly integrated and things are therefore more complicated. So you will probably need to get the full BSP from the vendor. For BSP, I mean the kernel, the bootloader, the device tree, the drivers, the libraries. And remember when you're doing this, the vendor is maintaining this stuff, not upstream Debian. Things are a little bit easier if you have a 64-bit ARM device. 64-bit ARM, the variation between the kernels is somewhat less. And so you can in most cases use a stock off the shelf Debian kernel. Whereas with 32-bit ARM devices, the kernels tend to be very specific to each device. A quick mention for Raspberry Pi, since it was mentioned in the blurb for this talk. So Raspberry Pi's are very popular, not only for general purpose computing, but also in embedded. And I'll argue whether that's a good thing or not, but it does get used for a lot of embedded devices, particularly the compute modules, which are specifically designed for embedded use. So in this case, the Raspberry Pi organization creates and maintains the BSPs for you. Historically, it has maintained a distribution called Raspbian, which is a port of upstream Debian for the ARM V6 architecture, which is the architecture used on the original Raspberry Pi's and also the current Raspberry Pi zeros. More recently, they have switched the name to Raspberry Pi OS. And there is a 64-bit version of this still in beta testing. So this is for both the Raspberry Pi 3 and Raspberry Pi 4. And it's based, it's a pretty simple port based on the upstream Debian ARM 64 port. So that's all nice. What about the Beagles? So I'm talking about Beagleboard, Beaglebone and such like, well, these are also very popular, especially in embedded space. So the Beagleboard org look after all this stuff and they maintain a bunch of BSPs for their boards. So Raspberry Pi's and Beagles, pretty much no brainer, just off you go. Okay, flipping it round now to Yachto, how does it look here? Well, it also looks pretty good. Yachto has better board support than Debian in this respect because pretty much anybody who makes an embedded board, particularly if it's based on ARM, they will have a board support package for that board. Within Yachto, the board support package is normally provided as what we call a meta layer. And I've got a slide on that a bit later on. But most of these board vendors do indeed have such a meta layer. So Yachto is obviously the winner here. If you have boards which are not supported upstream, you can create a board support package using Yachto project. And in my experience, at least that's much easier to do than it is for Debian. Debian is really great if it happens to support one of the boards that you are choosing. And there are quite a few of those. Okay, the next thing I want to look at is the root file system. So the root file system, this is the main part of your operating system. It's the programs, libraries, scripts, configuration files, all the other bits and pieces that make your system run. And in both cases, both with Yachto and with Debian, the components you put together are put into our packaged as packages. So a package can be just a single program, or it can be a library, or it can be a bunch of configuration scripts, or something larger. So a package could be, for example, GCC, if you want to install a compiler, that's a big, big package which has lots of things in it. Or it could be something like open SSH if you want to install the SSH package, which is a bunch of libraries and the SSH tools. So you need to get, sorry, your embedded system is basically composed of a bunch of these packages. So there are two ways of doing this really. You can either start small and get bigger, start with nothing and then just list one by one the packages that you need. And there are tools that do this for you and work out the dependencies. You don't literally have to sit down with a piece of paper. Or you start with, you start bigger and slim down. So you start with a known working system, typically desktop distro, and then you strip out the stuff that you don't need. So looking at this from the Debian point of view, essentially you have your root file system. You can do things like apt install and install package in this case XYZ that goes and fetches it from the configured Debian repository and installs it into your root file system. Seems easy. And this is what I call the search for the golden master. So this is kind of the start big and slim down approach of things. So you typically begin with your full desktop image. For example, for your Beaglebone, you've installed the Beaglebone image, the Debian image for Beaglebone. And now you want to tailor that to do whatever it is you want it to do. So you take the full image, you start stripping out stuff you don't want. You add in the stuff that you do want, so particular packages for various libraries and tools that you want for your particular project. Make a few tweaks, typically to the boot script so that when it boots up it boots into your application, maybe lock a few things down. And once you're happy with all that, you take a copy of the memory using the DD command or something similar. And you create what I call the golden master, because you're now going to take that image and just replicate it to everything that you ship. Okay. So I mentioned this because I have seen this done so many, many times. And it always goes wrong. Why is that. So the golden master is not a great idea. It is, as I described it, the way you created is kind of interactively. You make a few changes, you try something out, you make a few more changes, try something out again, and you keep on doing this until it works. But almost certainly you didn't write down every single thing that you did. So that makes maintaining the golden master quite tricky. Since you don't know how you created it, you can't create a new one, you can only take what you've got and keep on making more, more changes to it. And only that, it probably leaves some kind of fingerprint behind. So you may have left some accounts or passwords lying around somewhere. Your bash history might be stored in the dot bash history file. There may be some log files that you forgot to clean out. The whole thing is very handcrafted. So second pass, this is the right way of doing it. What we need is robust reproducible builds. And there are some tools that will do this for you. I would highly recommend looking at either LB or Armbian. Go look at those links there. And those both allow you to give essentially a list of Debian packages and some configuration information. And it will generate your root file system reliably and reproducibly from those scripts. Looking at things instead from the Yocto project point of view. This is illustrating the way that we're building stuff using upstream source code. So we have a bunch of metadata, which are called BitBake recipes. We have a list of packages and what we call an image recipe. So that's a kind of meta, meta recipe. And then BitBake will build all of those packages for us for each one. It will download the source code from the upstream location, wherever that may be. It builds it into a local package by default using the RPM format, but you can change that. And then eventually it will put all those packages together for you into the root file system. So this is very, very deterministic. And yeah, it's the way we do things. So one of the great things about Yocto project is this trinity of the distro, the machine and the image. So the image, this is my order, the distro. So the distro is the set of instructions you give to BitBake. These are basic policies to which versions of packages don't want to use, run use the latest or the some more conservative version, which builds, sorry, which in its system don't want to use, which logging system don't want to use, that kind of high level policy stuff. Then I can select the machine, which essentially is selecting the board support package I want to use, another which hardware I'm going to run this on. And then I can select the image, which is a kind of super recipe, which lists the packages I want to build. And the nice thing is you can change these things around independently. So I can keep the distro and the image the same and just change the machine, and then I can build exactly the same distro for different hardware. Or I can change the distro and I can keep the machine and the image the same, and I'll get the same, a similar image, but using a different packaging, a different set of packages. So it gives you almost total flexibility as to what you want to create. So comparison of that stage then. Yocto, since we're building from source means we have very fine control over what goes into the packages. The typical bit bait recipes allow you to select individual or very, very finely grained configuration options. And so you can create packages that are as big or as small as you want them to be. Not only that, when you are compiling the source code, you'll be using the compile flags from the appropriate machine configuration, which should be tuned exactly to the CPU you're using. On the other hand, on the other hand with Debian, you're using binary packages, so they will be compiled using whatever options are given for the upstream Debian port you're using. And these will tend to be more generic. They will work on a range of Intel or ARM processes, but they may not be ideally tuned to any particular one. Also, the way the packages in Debian are put together, they tend to be bigger. And so you will end up using more storage, sometimes quite a lot more storage, maybe three or four times as much. And quite often they will end up selecting options that you don't necessarily want, and quite often this ends up using more RAM as well. So a Debian system is going to be bigger and require more memory. And of course, do not, whatever you do, create a golden master. Okay, the next thing I want to look at is developing code. So by this, I mean, how are we going to compile the code we've got for the specific thing we're using the board for, and how we're going to put it onto that board. So at the top level, it's really a comparison between cross compiling and native compiling. So normally we use cross compiling. So this is the traditional embedded way of doing things. So this is quite nice, it means that you're compiling on usually a fast machine, usually your desktop or maybe a server, which is probably an Intel-based device. And then you are deploying on the target, which most likely is running an ARM SSC of some kind. So you get the best of both worlds, you get a nice fast compile environment, and then you can deploy the final binaries on the target. The other way of doing things is to do native compiles. So you compile on the same machine you're going to run the code, which in this case would be the target. So Yocto is very much based on the concept of cross compiling. And this is facilitated by the way that you can create an SDK, System Developers Kit. So that's an option within Yocto, an open embedded, and that generates an SDK which you can then install onto another machine, and that contains the cross compilers, it contains development packages for the libraries, it contains some debug tools, and some profiling tools, and some other useful stuff. So this is an easy way to deploy and develop code for your target system. WM by contrast is really built around native compilers. So that means that in the simplest case, you would have to compile on the target using the libraries and such like installed on the target. But obviously if you're building small IoT systems, you don't have a very fast CPU, you don't have very much memory, and you probably don't have very much storage. So in practice, unless it's something as simple as a Hello World program, this doesn't work so well. Quite often you'll find people recommend you to use QEMU to get around this. So QEMU is an emulator, it allows you to emulate a whole bunch of target architectures, including the various ARM machines. So you would set up a QEMU environment, and I think Debian supports this pretty much out of the box, for the architecture you're using, say ARM64, and then you would provision that with a lot of memory, and of course it has access to all the storage on your device. And so you can do much bigger builds within QEMU, and then that generates the executables, and then you can put those onto the target board. So this works perfectly well. It's a little bit of a faff, but a few scripts can help you get there. So in my opinion, cross-compiling is where you should be, and Yocto does that because that's essentially the whole way it works. So I am much happier with Yocto for this than using Debian. Next topic I want to talk about. I want to talk a little bit about long-term maintenance for these things. So this is a big issue. The devices you build and deploy today are likely to be in use for two, four, ten, maybe even longer years. And particularly as things become more and more connected, then support and fixing the security issues with those boards becomes important. So both Debian and Yocto have support policies. Neither of them are 100% what you want. But in the case of Debian, so Debian releases are supported for typically five years. And if you go look at that link there to the Debian LTS project, there's information on how they do that. Note that when I talk about Debian support, I'm talking about essentially the root file system and maybe the kernel if you're using an upstream kernel. But the remainder of the board support package comes from the board vendor. And so it is their responsibility to maintain that. And so you probably need to talk to your board vendor to find out what their support policies are. The Yocto project has in the last year introduced a long-term support release. So there's one LTS release per year. And the LTS releases are supported for two years. So that's not quite as long as Debian. And when you get to the end of that two years, you have a few choices you can make. You can either go on and self-support. You can see if you can seek support from a commercial vendor or even from the community. Or you can update to the next release of the Yocto project and rebase everything on a new release of Yocto. The latter one is in many ways the ideal solution. But it's also the most complicated one because it means you've got to do a complete rebuild and maybe even a complete validation exercise, which may be problematic. And I'll emphasize again the LTS Yocto support is only for the core of the Yocto project. It does not in particular contain the meta-layers which are supported by, in this case, the board vendors. So that is a quick run-through of all of the topics I want to cover, looking at various aspects of Debian and Yocto. So can we draw any conclusions from this? So I want to say, first of all, it's not either or. On several occasions in the past, I've tried to draw this as a conflict. Obviously it's not a conflict. Both Debian and Yocto are good choices in various cases. So I really think Debian is the best solution if you're using commodity hardware, which has good upstream kernel support. So I'm talking about x86-based systems and I'm talking about things based on the ARM64 Debian port, which is well supported. Of course, you need to build your root file system using a suitable tool such as Elba or ARM-bian. But if you do those things, yeah, it should work. And it's especially good, of course, if you want to do proof of concepts and prototypes because you can probably get a Debian system up and running more quickly than you can equivalent Yocto. However, Yocto has its own niches. It's great if you have, well, it's essential if you have custom hardware, which has no upstream support from Debian or other distros. And that's probably the majority of cases where it gets used. And also it's really great in that it's optimized. So the memory usage and the storage usage will be less in a Yocto-based system than they will in an equivalent Debian system. So you choose which is the best for you. Okay, so that is everything I have to say right now. So over to you. And if you have any questions, please go ahead and ask them right now. Thank you very much.