 My name is Manuel. I'm from linotronics, and I wanted to talk with you about Debian-based embedded systems and some tooling around that called Elbe Let's see what's the plan for today. So Basically, I want to answer the question. Why should I use Debian? Why should I use Elbe? What other methods are there to create such Debian-based systems or What are other things you could do to build an embedded systems? Then I want to talk about the features Elbe currently provides I will show you an example by the Texas Instruments Beaglebone blackboard to build an SD card image for that board and then we will have a look at customizing this image for a specific need and and We show how to develop and integrate own applications into that image And the last thing will be we will think about what's next what's the direction of the development in Elbe itself Maybe you have also ideas. What should be possible with Elbe? so if you are in the need of having a distribution of having embedded Linux-based system you have a decision to make one is I want to build my own distribution something we heard in the talk before and it's Something you need to do so you have to write receipts and do maintenance work and so on so you you know that and Maybe it's too much for you. So maybe you crash with that because you don't have the development power and so on and So It might be harder than you think at the moment So you can see the Yachter project or something like that. That looks really good maintained Is it in effect also? but you Have a huge knowledge base you need if you want to adopt this system. So By the way, you need to wait until every package is ready You need to compile it you just can't like on a desktop say apkett install this in that library and try something out You always have some effort to make that it is on the system then the binaries that come out of such a build process are maybe in a Binary format, you are the only one that uses that because you have special configure flags and so on and You are so the probably only one who is testing those binaries And this is also the effect you might be then alone with some bucks that are in those binary packages and at least you are responsible for tracking the security of your whole self-built distribution and What we ask ourselves what are is this really needed do you we really need to build our own? Distribution or is there not anybody else who could do this job for us? So we found in the desktop fraction a lot of distributions that are providing ready-to-use binary packages and we ask ourselves could we use this on an embedded board on an embedded system and If so, which one is the best to choose of all those distributions that are around to provide binary packages We resulted in the choice of Debian because this graphic is a bit overloaded I think but basically what I want to tell you you can understand Debian because it's an open community based distribution there's no vendor behind Debian and All the tools you need to rebuild a Debian distribution are available for free they also have a really good documentation and open development process and You have interfaces to retrieve all the build logs of the binary packages and so on so basically You have the same information as you Have if you build the distribution yourself in on your boards. Maybe it's better because it's worth web applications around and user interfaces and so on and The process is really open. You can get involved as deep as you want if you invest some time So basically if you want to use Debian on an embedded system It's not that hard. They already provide a lot of tooling for doing that By the way, Debian is available for a lot of different Architectures so you have of course x86 support but also two different kind of arms MIPS power PC and so on They provide binary packages for all of these architectures So what you basically do is you use some Debian repository available on the internet and then use a tool called the bootstrap it's a tool from Debian you can just install it on your desktop PC and You tell Debian what you want I want an arm-based system with decent that packages inside and what you get back is a Debian based rule files Root file system in some folder on your desktop PC. So just extract it and put there Then you have to do something by manual Typically the bootstrap has a second stage that needs to run on the target You can use emulation for that or you can just copy the stuff over on a SD card and run it on the target and This is the what you typically do and then you do some manual Modification remove documentation man pages install some additional packages and so on and then you get your customized root file system for your product and After that you can use some open source tools to produce a firmware image. That is later Flasher ball on a nan flash is the car or whatever for doing that you have several Possibilities to do that. So you can use trust as I explained in the slide before the bootstrap plus Bunch of hell scripts. This was the way we started in the early days Because shell scripts enable me to reproduce the same thing but as we Started with a second and third bird We learned we need to modify ideas shell scripts for a specific board Then we copied it over and hell began of maintenance So there are different options you have For example, they are the bootstrap whoppers that provide exactly that infrastructure to you They allow customization scripts that can be versioned in git. They allow creating images at the end for Basically, it is we am for virtual machine So they started with virtual machine disk images, but nowadays they have support for a lot more image formats So this is something you can use Then there is Isar that is a project that uses the yok to infrastructure to Just run the bootstrap and other scripts so that it you can use yok to to maintain your custom board And Then there's another interesting thing you might have heard of it's meta yok to it's another yok to based thing and I Try to use Dabian Source code and use the yok to infrastructure to build binary packages out of the Dabian source code Which is a good idea because you have those maintained and really wide use sources with Exactly the same patches Dabian uses and you have a reproducible build But you have the thing you always need to build the stuff by your own The question is why are Why should I use Albin out there are still a lot of possibilities to use and It's Quite simple to answer because every produces brilliant firmware adventures with no box inside Yeah, I think you don't believe me So I'll be stress Python code that uses Dabian infrastructure and tools is the real answer so what we try to do is To have a project description in a single XML file so every project is one XML file We don't want it to be distributed in different layers folders and so on to get together a config in some more or less magical way Then we said we just want to integrate Dabian binary package. We don't care of building our own software For the upstream project so for all the basic tools like bash busy box Firefox or whatever you want to have on your system But it allows you to build own software components And as we say build on software components We may mean modified packages from upstream where you say I need an additional version or something like that Or we mean your own application because normally you don't want to sell embedded board with a plain Dabian and no other application on top Then we want to produce flashable images. So we want that Every produces output that can be used directly to flash it by JTAG or whatever way And we want to be able to regenerate the images Maybe a few years later and regenerate the whole build and where I am meant also So what we thought about is the easiest thing to do is to run much of the Elvis stuff inside a virtual machine where we know what is installed in and That also is created by LB so if you put in LB XML file the virtual machine will be created if it's not available and then the whole image is built inside this virtual machine and Transfer back to your host PC and What goes out is not only the image that you flash on the system It is also included a rebuild CD We are all the debut and binary package are on that are needed to build this in it for M this initial virtual machine and the target image We also provide a source code CD run Where all the debut and source packages are there that is something you might want to give to your customers that you Say this is the source code in a format. You can reproduce the whole system. So because Licenses say you need not to ship only the source code you also need to ship the information how you transfer the source code into the binary format and With a debut and package you have all those information you need Then we create some license files that you see what licenses are inside my embedded system And this is all generated out of the debut and repositories That are out there on the internet By the way, if you don't want to connect to the internet during build you also can Do a replication of a debut and mirror on your own company side? Yeah, it can be installed the virtual machine is just a plain debut and install relation With the LB software components called LB demon or LB build environment installed We package them also as debut and package that it's easy to install them inside the machine But you also can run the LB source code the LB demon and so on on your host PC. It's not a problem. You trust you Lose those repute usability of the images because you don't know which version of G part that is installed on my host PC At the time I generated the image and so on it is generated by LB. It's safe So basically at the moment we have a LB stable version with the number one dot one it was a Version one was released Christmas last year. So I hope getting a version two until Christmas this year the version one supports target images for armlittle-endian arm ha f e x86 32 and 64 bit and power PC Debian supports a few more Architectures, but we just didn't test LB on that. Maybe it works, but no guarantee about that. We don't use it Then we support some Image formats. This is MS-DOS Formatted discs also GPT for market discs then trust the top all you might want to use as NFS root file system Or something like that CPI over in it are these Squash FS and for NAND we have UB and UB FS support included Then we have the availability of fine-tuning targets So you can describe any your XML you want to copy over some files from here to there or move them Or you want to execute a command inside the target root file system Then we have the possibility to extract an archive over the whole target root file system This archive file is tau ball. That's trust embedded base 64 encoded inside the XML file It is not meant to integrate your own application. There are big binary plops It's just meant for having some config files for your Apache web server or something like that We also are able to Generate target related sys routes for cross-compile tool chains. This is important for developing your own Application against the stabian based root file system. So what you can do you can say elbow generate me assist route for this or that target and then you get Tau ball that includes all the libraries that are on the target including the headers for those libraries and the development versions of the Libraries that are there and we just resolve that By using informations from the debian project. So we look What binary pack what development package fits to which binary package and install them additionally into those sys routes And as I told we get CD ROMs with all use debian source Packages and one with all the binary packages and there all the packages from this initial virtual machine are also included on this CD ROM so you can just archive this binary CD ROM and Say elbow in it for M create and give the user image and the whole thing is regenerated from that step Another nice thing is you can check for updates of a generator pack Target image. So what we do is we store inside the XML file during a run Which packages are installed on the target with which version and then we have a command elbow check updates you put in those XML file and it looks on the reference mirrors if there are any updates available for this target and Typically what goes out of some debut strap pack? Process is a debian with all those essential packages. So essential is a tag that a package can have in debian and Essential is everything marked that is needed that apt is running. So the package manager But maybe you don't want to have a package manager on the target and you it has a dependency to pearl and so on and this Is the reason why images get quite big even if you just install those essential packages And so we thought about a mode to generate a shrink target out of this Essential debian system to even get smaller Then we have another version of a bit This is the one we are currently working on it will be the next major release called 2.0 And the development version is called 1.9. I think 70 or something like that at the moment And of course we have some new features in a bit testing That are not in the stable release So we just want to get them stable and if they are stable we make the new stable release That you can use always the stable release without big troubles So what we have done for LB 2.0 is LBP builders Think that you can use for building own debian packages So you have a source code and every produces the debian binary package Then we have support for arm 64 targets We now show the build log because every 1.0 Just shows weight project is busy as long as the whole target is generated and this made Tike half an hour or something like that and just see weight project is busy And you have no idea how far you are in the process and so nowadays we display this build log instead of this busy Log this is something some user requested, but it's definitely useful Then we have some tooling to generate SPDX files out of the debian license informations because the new License information format of debian is human is machine parsable And so we can transfer it to XML and then have some tooling to Transfer those information into SPDX files However, not all packages in debian provide machine parsable License file and the licenses in debian the terms That are used there are different ones than in SPDX so you need to do some mapping and We just developed some tooling there that can do that Then we added extended partition support for MS-DOS hardy So basically these are the things that stabilize now the biggest thing is those LBP build a thing and If you want to use every you can and you have Debian installation on your host PC. It's quite easy You can trust at the Linux repositories and install LB with apt If you don't want to install it into your system It's all so possible to just clone the git repository and use it in tree So the first thing you need to do is to create this initial virtual machine This is done with the command LB init we em create and you can give a directory where this init4m is stored The init4m is as I told you a basic debian installation Chassis at the moment including the LBP package It is used to build those firmware images inside the init4m and as I told you it's Reputed so built one from this bin CD-ROM output The init4m is not started automatically if you if you boot your host PC so you always need to start it explicitly with this command and Now let's have a look at a concrete example So if you want to have a SD card image for a beagle bone plaque board, how does this XML file look like? so what we have is trust Schema description for our XML file. It's called after the development name of LB because as we started in 2007 I think we called it Debian build system for embedded devices Doops fit but our marketing thought it's no good name. I don't understand that But we kept this in the Schema because nobody sees it typically so What we have is a project description here So we have some name of the project a version a human readable description and the build type what is basically the name of the Debian Architecture you want to use its arm hard float in this example Then we have a reference to Debian mirror and we distinguish here between a mirror That describes where you can redrive all those packages that are marked essential So the bootstrap runs from this mirror There's the availability to add some additional Debian repositories here and some thing called URL list There you can add some other repositories, but it's important that the Top one is one where the bootstrap can run on Then we specify which Debian suit you want to use you can also use Chassis on a bit one dot zero even if the chassis Even if the elbow one dot zero in it for M is based on we see so Because you can trust the bootstrap a chassis system on a VC system. This is no problem Then we describe our target Basically, we give them a host name. I don't may name to it This is something every trust feeds into those files that have been expected You can set a root password there and specify a serial console. This one is used on x86 for example That bootlog is displayed there. So we set up the crop config that it is Entered there. It is used that a get he is started on with a Login prompt is started on this serial console and so on and Then you just specify the output here. We define some MS-DOS based partitioning shame And the output file name should be SD card dot IMG with a size of one dot five gigabyte and Then we have one partition with 50 megabyte size that has a label called boot and the bootable tag should be set on this partition and Then we had a second partition that should be used the remaining size of the whole image and called RFS and on the next slide you can see The file system table. So this is the thing where you describe which file system should be on which partition And various should it be mounted? So we just reference these labels here So as you can see on the slide before we had a label called RFS here So this one is mounted on slash formatted with extended to file system And we have another one called boot that is mounted on boot formatted by Faufett because the TI bootstrap code Loads the bootloader from this partition and Then we have some fine-tuning section here. I think I just showed it in the order Schema needs so it's better to have a look at package list first So what we install is a Debian based system that comes out of the bootstrap plus U boot package and Linux image package. So Debian also packaged U boot for ARM and Multi-platform kernel for for ARM including device trees for a lot of common evil boards So you don't need to have built your own kernel your own boot loader Of course you you can do that But there is no need to do that for a common evil board So I want to show you here But what you need to do is for example all the device trees that are packaged This is not the only one that is in this Kernel package in Debian. They have all the device trees for other birds So I just copied this one over to a place the bootloader expects it and The same for the U boot image and for the first stage boot loader The next thing is what I do here is I just say echo U nth command set and boot arcs and so on to boot U nth txt so this is the way I set an Additional configuration to U boot this U nth txt while it's read during boot from U boot so I can Set up something here With just echoing something. This is also possible. You could also put this File into the archive that is bay embedded base 64 encoded inside the XML file Would be just another way to do it But the XML file is then not so human readable then if you do it this way so this is basically everything you need to describe to create a bootable image for a beagle bone plaque and and If you want to get a build by the init VM, you just say LB init volume Submit and I use an extra attribute here called keep files Which means start a project inside the init VM the default is to delete it after successful build Because if you run nightly builds and Jenkins or something like that you run into the situation The LB init volume is just full with a lot of embedded projects. So we use this keep files just for development and normally And I give the XML file as an input we have a lot of examples inside the git repository Or we install it and user share doc LB doc if you used our Debian packages If you inspect the build output You see the spina re cd rom That is needed for reproducing everything than an every part txt Where we just list all files that are on the target system and who produced them So is this was this introduced by fine-tuning was this installed by some Debian package and so on and some more Interesting informations about your target those text files are in the ASCII doc format by the way So it's easily possible to transfer it into a website or a PDF document if you need that Then we have this license txt file and basically the same information as an XML file But the XML file can be used by this spdx tooling to continue Generating such as pdx information Then we have a log file where we just capture all the output the bootstrap and so on generates Actually our image we want to use DD later to dump it on an SD card And we have this star six ML file, which is basically the thing you put into the On the top But it was added with some sections where you can see which packages are installed out I installed by the bootstrap or as a Dependency of one package in your package list or did you specify it explicitly? Including the version of the package so that is the thing LB a check update Users to see if there are updates available for your image Then we have the source CD around that includes a Debbie and repository with all the source packages that were used And at the end we have some validation txt file. This is useful You can use this one as input again And then we verify if all the Debian packages were installed with the same version And with no deference and if we say a difference we log it just in this validation text file Yeah so the question was where there are Debian packages that I can update images inside the field and Where are Debian packages with debug informations and so on that I can debug my thing Answer to the first question is they are all in a Repository that is stored on this spin CD or on so you can extract it from there and you have the possibility We have no brilliant control interface for that at the moment, but technically it's possible to also generate Some archive file with just updated packages if you give this source XML arts input file again But we can discuss this later if you are interested in something like that or on the mailing list The second question about the debugging symbols and so on This is something we generate with a special command called every control Build this route that you get an archive with all debugging informations development files and so on In LB 2 this was the output of LB 1 in every 2 the images are just cheats Compressed because we transfer them from the virtual machine to your host PC and this took quite a long time For big image files and they can you can reduce the size a lot if you just treat them so what we do here is just unzip it and push it on an SD card and Then you can boot your beagle bone plaque Maybe you are happy with big Debian beer in your hand But everything is a lot bit bloated for embedded you you know, so What you want might want to do is shrink the whole Debian image to get just Just to fit into smaller Pieces of flash so some ideas are you can remove man patrons. You can remove unneeded locale or package lists This can be done in this fine-tuning sectioning in the XML file This is so if you remove the package lists, this is something apt-get update retries Unneeded locals and man pages you can reduce the image size by about I think 80 to 100 megabyte Then you can that there no recommend tag inside the XML file so Basically Debian says a binary package can have dependencies that are hard that the the reference packages need to be installed on a system that the One package is working and there are suggestions So they recommend to install an additional package and the feature of the basic package is The features that of the basic package is higher So if you set this no recommend tag we just don't install these recommendations because app does this by default and So if you have a long package list this can also reduce its significant later image size and If this is also not enough and you say maybe I don't need apt on the system or something like that We have even two modes that can be used the diet or the Titan mode to Copy over the output of the bootstrap and additionally installed packages to the target directory so Basically what we call change route in Abyssal go is the output of the bootstrap plus the packages that are installed from the package list inside the XML file and Then in default mode, we just copy over this change route directory in a directory called target and On this target directory we extract the archive and run those fine-tuning rules and the image generation works on this target directory and what the modes are basically doing is just Alter the way this copy is done So the default mode is just a one-to-one copy as I told you the diet mode looks at the package list in XML and Calculates the runtime dependencies of all those packages using apt and Then installs just these packages or copy over just the files from these packages to the target directory so you have basically a runnable packages, but Not everything that is on an essential devian system is installed on your target So what we also do is we run all those post in-store script you might I don't know if you that deep in devian and that made file that they run on The target directory because not all essential packages are there and maybe it's a pearl script and the pearl interpreter is missing And we just ignore those errors. So the result is definitely a target image But you miss some you might miss some Files like for example ETC pass VD or something like that because the generation failed And then you need to use fine-tuning or archive files to add those files you need again, so it's harder to Produce a diet image than a default image, of course Then we have another mode Called Titan. It's basically the same as diet mode But we don't even resolve the runtime dependencies. So just the files referenced on the Package list are other the files from the packages referenced on the package list are copied over to the target system I love this Titan mode for creating a really small systems like just having busybox static inside the package list And then I just get a CPI image with a busybox system inside of course the customization part in the XML file gets bigger as far as you go down in those copy modes and At the end if you use such a copy mode and use fine-tuning and so on you can shrink the Debian based image to a quite small size and The interesting thing is you can repute use this after and after again because it's all described in the XML file and You don't need to have custom scripts and so on for different purposes But what's still missing is you want to integrate your own application into the system I think or add users and so on for adding users We also have fine-tuning commands by the way and groups But adding own software in elbit 2.0 is for example that easy If you want to integrate for example the lib GPIO we wrote some years ago you can trust reference a git repository and Optionally specify a revision tag that is useful if you want to integrate always the same revision of Some software if you want to integrate always the latest version just skip this revision tag and What we then do is we use a People a mechanism to transfer this source code into a binary Debian package some details of that follow soon And what you need to know you need to specify this package in the package list again because out of one Debian source package you can generate and Debian binary packages for example the deep one including the debugging symbols and so on and You don't want to have all of them on your target typically, so you need to re-specify what should be on the target You need to Debian nice your source code if you want to use that so What this means is you need to produce a Debian compatible information how to transfer source code into binary format It's something like bit-baked receive But it's split it out into different files But basically you need to enter the same information you have also in the octo You have those basic classes for auto tools and space projects and so on you have a similar mechanism in Debian called that help us that you can use there and there is a quite cool tool called DH make that just produces a template for your application where you just need to fill in some more informations about your package and Then you can use this ABP builder feature to build the application We don't have invited all this P builder functionality basically it's based on the PDE build project and QA move user to emulate on targets and so on so it's a common way to build packages It is documented in several places and Debbie and Vicky and so on and we just made it to be To reuse this out of and regenerate all of these environments out of the XML file Of course all those build dependencies need to be specified correctly because P builder always works in an in an Debian environment where only the essential packages are installed and all other packages Need to come out of your build dependencies that need to be specified in your Debianization The build environment is created from the same Debian mirrors and so on as your target image so if you use for example that mirror to keep a static version of Debian with no updates and so on in your company You can rely that you always build against the same binaries Then the ones that on your target even if you create the P builder environment later than your target image Of course for application development. This is not feasible because P builder Installs all those build dependent disease then builds your software packages as Debian Package puts it into a Debian repository regenerates the whole image this needs of course Some time and if you just want to develop an application. This is overkill So we have L be built this route for this. This is the feature. I told you where you can integrate Into an cross compiler toolchain all those headers and libraries that are Fit to the ones that on your target So after that, I hope your little Tux is happy and has a good party because it's stable good Debian based system with your own application and so on and We hope to fix all those open issues for LB to till Christmas or something like that this year and Then we think about new ideas for the next version of LB and What we are just thinking about no guarantee that this will be an LB 3 or if you trust Do some other things? Maybe you have ideas One is at the moment We just create those the bootstrap change routes use to aim a user to change into the target fire systems And there are some nice things like Linux containers where you Can start an in it of the whole target system And then you have a the whole target system running inside the innate VM with all those demons and so on And this is something you need if you for example want to install my sql that just insert something into the database As post install step so they want to have to my sql demon running After this install that they can insert default users and so on Another thing we we need to work on is improving the logging and report generation at the moment We are quite happy that we get most informations out of app because they do really crazy file descriptor pipeline stuff and The output and somewhere and now we think we corrupt all that we need and The next thing we want to do is we want to We have some build jobs inside the innate VM so ever you submit something There's a build job generated and so on and this can get an idea and then we can log all together into some database using log for pyre or something like that and then generate a Customized log file out of this database because we we learned that some of our users are Interesting trust in the output of their fine-tuning rules or trust in the output of the debut strap process And so we can filter that better Then we want to improve the session management because at the moment you have one in it VM and you can build one project per user per time So you can say elby con elby in it VM so submit can also have a user and Password argument that you can run it on some central installation and we Have the limitation that one user just can walk at one project at the moment This is some something not really technical argumented But what it was the way it was developed and we need to change this that one user can work on several projects at the same time And then we think about sharing some source code with tools like VM debut strap because it's also right written in Python they also generate disk images They also have the problem to install crap there and so on and they basically I have had a look at the source code these days and Trust the Python classes look nearly the same than ours So we I think we should share code with them and not having the problems and to code repositories and fix them in two ones Yeah, that's basically so my thoughts about what might be in elby 3 So if you want to use ever have a look at our website There is also reference to our devil mailing list. You can also ask user questions there. That's no problem And the source code is available on GitHub, but the recommended way is to just use the Debbie import Packages you just want to try it out. So my conclusion is For all the things you can get help from other projects let other projects do the work Let Debian do the maintenance work. They do a great job Don't try to build everything by your own if you don't really need to do it Then use cross tool chains with a little bit generated this route to do your application development, then you can use LBP builder to integrate it nightly into your image and and Then you can use the LB image build to produce those flashable images and Focus on your application development and not building your own distribution so thank you a lot for your attention and I think if we have one or two questions, we might can answer it now and Our hand I will be out of the room later for questions. Yes What is your use of It's arm little and then it's the name as Debian calls this architecture basically it's for arm v5 and earlier No, not really in current projects, but Debian supports it and as we started with LB it was a common architecture So we've still supported Then We we thought about that but This is a combines with something like co-chairing with VMD bootstrap and so on But basically I was in contact with the embedded Debian mailing list Some people like everywhere other say yeah, there are other tools and scripts that do the same job We we don't need every but I think it's no problem to integrate ever the only problem at the moment is Debootstrapping this initial machine Uses the Debian installer and for that purpose we build a Debian package including the Debian installer and this would be a source code replication of the Debian installer and This while it's a day Debian policy and this is definitely something I want to fix and have some other tool or some other mechanism to recreate the initial virtual machine But this affects also the easter generation and so on and it's a bit tricky But it's something we we have in care, but it's a bit of work to do for that Yes Yes, it's possible. You can trust use app get source patch the upstream package and then use LBP builder to Integrate it in your file system Yeah, it runs automatically if you put a section like this into your XML file where your reference here needs to be the patched source code and Then need to be the step in directory inside and then LBP builder builds it and here you reference it That is also installed on the target. Mm-hmm Yeah, it's quite easy. You get this license dot XML file and then you can just write small snippet of code Where you can say is what packages use this license and get a list of these packages Should be I think four or five lines of Python code No, that is not possible So we can trust have a look is it used inside the image and then we need some manual Modification to remove it from the image. There are tools helping by doing this for example, you can trust trim a live image and Then you can use LB diff or LB package diff to get the difference to your original image And then you get recommendations how to order your XML file to get this result You're welcome Yes No currently not we thought about using up cash or for example inside the init VM But then the init VM image gets rather big. So it's something we recommend to use as you can use app cash As a transparent proxy for example Yes, adding is possible removing is not possible at the moment because removing is quite hard Yes With default mode on our arm system I think it's 300 megabyte or something like that and as I told you you can easily remove those Package lists low-cales and man pages that would use it by about 100 megabyte And if you want to get really down it's possible to get images with five megabyte and trust a busy box inside with those modes So thank you a lot. Oh We use something custom at the moment, but it's something we want to get rid of Yeah, thank you a lot and if you have any questions