 So, my name is Bozhan and in this talk I'm going to present how to build Debian-based embedded systems or products using Debian and BitBake. So, the plan is our motivation where we started, why we developed the whole thing at all, a brief introduction on either how we do this. There are a couple of slides about the mechanisms, I don't think we will have time for that today, but I included this for a reference so that you know what to look in the documentation and a couple of news since our last presentation on embedded Linux conference. Okay, so where we started, we wanted to build a product and we knew that we are going to use Debian as a base system for this project. Actually, this was back in 2004 and I don't remember whether open embedded existed at that time, at least I was not using it then. So, what we needed, we needed to have a Debian system and produce complete bootable file system and firmware images. So, it means including U-boot and kernel and so on. There should be binary Debian packages used and there should be our own application packages that are going to be included on the system. What we also wanted is one command on demand image creation and what we also didn't want is to touch every upstream package and at the end, after this is a product, we want to build an older version of some snapshot afterwards. There are some further requirements on how to handle the whole thing. So, we want to be able to customize upstream packages if you want to. We want to build several products from one repository because they are, for example, similar or share components and we also would like to incorporate upstream code from software or hardware vendors in the repository and we have to maintain this for at least 10 years. If we look today at the existing art, basically we have binary distributions and source distributions. So, Debian is a typical binary distribution where you get pre-compiled binary packages and you can use them right away. Yocto, on the other hand, is a typical source-based distribution where you get the whole sources and you have to build them yourself first in order to use them. So, I'm not going to go through this whole table and also please don't take it very literally, it's really an over-simplification to give you a feeling in which direction every project goes. But to summarize, Debian has many standard tools. It's tested as a whole distribution, which is important for us. It has clear licensing, long-term maintenance or relatively, depending on the definition of long-term, and fits small products or the whole product's lines. Regarding Yocto, in Yocto we also have clear licensing. We have BinBake, that is a single command integration tool, which is very flexible and efficient. And we have collaboration policy in form of layers so that software pieces could be grouped together according to some criteria. Also, last but not least, one difference between them is that Debian focuses on certain pre-selected architectures. For example, if you take ARM, then only two of them are currently available. And with ARM CPUs, you have a wide range of optimizations that could be important for you. So, with Debian, you only have two grades of freedom, whereas with Yocto, after it is source-based, you can fine-tune your build for your required CPU capabilities. So, what is Ether? It started 2004 as a bunch of shell scripts. Of course, with the time, we saw that it's very difficult to maintain it and we switched to BinBake. First off, Ether is not a distribution and is not a distribution builder. So, we just enable to use Debian distribution as a base system for our product. And we have some mechanisms to build our own applications and install it into the firmware image. And we also like basically the structure and workflow of the Yocto project, how they organize their directors and we also use this in Ether. So, this is what we do and what we don't. Basically, we reuse binary packages from Debian because they are tested because they have security updates and because we don't want to rebuild them every time. And we use BinBake, the tool behind Yocto to bring the whole thing together. Like in Debian, we have distributed source repositories and like in Yocto, we have centralized metadata repositories. We want also to change as little as possible and to build as little as possible. This means we try to really reuse Debian packages and only where we have to, then we modify them or build our own applications. So, this is an example how we did this in this particular product. We used Debian packages as is. We modified some small amount. We built with them once and saved for future use. So, we use them also pre-built. We also built our own U-boot once and used it as a binary. And the rest we rebuilt every time because it changes often. This is an error also in this slide. U-boot, we don't rebuild U-boot every time. I took it above. So, kernel drivers, libs and applications, we rebuilt every time. So, how does it work? Do we have a pointer? No, we don't. Okay. So, first, we create a build change route. This is a change route. It's a Debian installation where we build our packages that are needed, like our applications. Then we build, actually, build these packages there, like in this example in HelloGit, in this build change route. So, this build change route in current implementation, it is an ARM change route and we run it with QEMU and build the packages with standard DPKG build package tool set. After that, after all packages are built, we call multi-strap again to create a root file system. Then we install the packages that we built there and create a bootable, complete firmware image, containing also U-boot, kernel and root FS. So, these are simple five steps and they are all incorporated in one command, like we know from Yachter. So, how we build images, we clone our repository, we initialize the build environment and we create an image. So, if you look inside, basically, the whole thing consists of five files, so-called recipes, and these recipes do each every small step, like build change route prepares the change route only and HelloBB is responsible for building the Hello application and user image base and debug are recipes for building the whole images. These recipes consist of tasks and order, build order is defined by task dependencies and performance is achieved for one hand on using binary packages and on the other hand through parallel execution of this. The whole thing is organized in two layers. The layer meta is a core layer and it provides basic functionality like build change route. And meta either is the product template layer so that you can copy this template to your own project and modify files there. So, this is how it is organized. You see here two metas. The meta core is going to be pulled from time to time from our site when it is updated and your meta product you modify it once and maintain then you don't update it. And you see that your application Hello and your images are in your meta product and here you can see for example dependencies between the tasks which define the order how the whole thing will be built. So, how you create your project? You clone either, you push it as your own repo, you define your product images, add packages and add boards. Images are defined in their respective recipes so that you have a copy of your product recipe and you can customize it. For example, if you want to install more packages in your target router first then you say you modify the image pre-installed variable inside it and say okay I want to have apt and I want to have open ssh server and so on. You also might want to provide images with more functionality for example for two different products or for debugging or development tools. In this case you just include the base image recipe in another recipe and add additional packages to image pre-installed like gdb or else of all strays and so on. So all variables in these recipes like bb and also configuration files are added to a single global namespace. Often there is a question why not use a single configuration file? The problem is if we want to build multiple products that share components then if we use single configuration files then we are going to repeat ourselves. This is the problem and that is why this modularity and that is why we have to find a suitable place for that. Now in the long term we want to reduce the number of configuration files and to organize them in some way maybe we need a GUI. We have to get more experience with that. So adding packages basically you should have an own project or application repository and it should be already Debianized. After that you create a three line recipe for that referencing the URL of the repository and source revision and then you say inherit dpkg and this makes the whole thing built. For now we have to list the dependencies of Debian packages here in the recipe again but we are looking for solutions how we can add building DSC files directly from a bit bake backend. Finally you have to add your application package name to image install in your image definition file. Also please note that we don't port all Yocto tasks here we also we only have those that we need and at the time those are fetch and build. For a board definition it's called machine in Yocto consists of settings that are necessary to bootstrap Debian and get a root file system. So what you have is a reference to the machine file. It's QMO arm is really the name of the QMO arm conf referenced below and you have distro Debian jc. It's also a reference to a file that defines the Debian jc distribution and distro arch is arm hf which is the architecture Debian architecture name and in QMO arm you just list the kernel image and internal day image and pre-install packages that are necessary to bootstrap the root file system and here is the example of distro definition files so that you can install the right suite with right components from the right app source and afterwards configure it with a certain configure script if you need to create some device nodes and stuff like that. So for effective collaboration we want to organize codes in different directors because once we put them into these directories it's not finished. In a few years we have to update these files with newer versions and this is where layers are useful. So we define different layers usually by vendor or by company or by division and define split also products in their own layers so that every layer has more or less its defined life cycle and update cycle and so on. There's maybe basically in one central repository or distributed in different git repositories that it doesn't matter and you can use tools like repo for example to clone all repositories. What is important is that these layers have only to be listed in the BB layer conf file and that's it so you have after you create these layers you list them in this in this file and they are then included in the build. So one common use case is overriding an upstream package for example with custom configuration or modifications. So a quick and dirty way for doing this is just to hack this in image recipe. The right way that we follow is to fork the respective package but of course this means that as soon as you have one line difference in init tab you have to fork the whole sys5 in it. So our vision is to have a recipe that can override the default package with some patches and then that it will create the source package and the binary package for that but this is not implemented yet. So this is for the reference. We can return to them if there are questions regarding that. So what's new currently in the development branch? We have added we started with ARM support. Now we added Intel support. We had some issues with Jesse that are now fixed and we also after we have done U-boot and in general boot loader installation manually in the recipes we are looking at image creation with WIC which is Yachtos tool to create images. It has some a number of dependencies and we are looking what would be the right way to incorporate that in ISR and we also have numerous documentation fixes. Okay, so to summarize we tried to combine the best options that we see and Debian provides a stable tested distribution with security updates. Yachto provides BitBake which is very flexible and efficient. We provide the template repository that you can use right off and you can use it as a template and modify for your own product and the whole thing after you get running you want to work it fast and this is last and not least why we developed this system. Here are the references so we are on GitHub. Manuel is in the wiki and for mailing list we decided to abuse Debian embed it for now till we have great much great amount of traffic and we could move anywhere else. Well that's it for today. Any questions? Yeah well actually sorry in Yachto there is a possibility to generate SDK. Do we do this or don't we need this? Well I personally don't need this but what you get is actually in the first step you get a complete build truth that you can use in a virtual machine or you can use it in a changed environment. So we could provide a BitBake shortcut to carve this so that it is like an SDK that I can hand over with one file to a developer and that they can use it. Basically technically the answer is we provide it but not in the form that people are used to in Yachto community. We could think about adding this syntactic similarity so that people find stuff where they expect it to find. The same is also for DevShell so you basically change root into the build truth and you get the development shell. We compile the kernel or something like this. Generally we are using GPS but we have a package it's called QMLux so because we have problems with the performance does it have the same problems? Well actually as we started we used the question was excuse me whether we had problems with QML performance for building kernels and whether we provide a QML axle package to mitigate these problems. The history is we started with cross compilation in a native system so the build truth was not really an arm truth started under QML it was a native Intel change root and it ran natively but you had to cross compile your kernel package so this means basically you don't want to reuse your Debian kernel in your product that is why you can hack a small kernel package for you and cross build it this is not a big deal so this is how it started. Now we had a hard requirement from a customer who didn't want any cross compilation in their system and they insisted that we ship this with an arm build change root that is built under QML and this is where stuff gets slower so to answer your question today we haven't looked at QML acceleration we might look at that however if you need performance then my suggestion would be to go to native compiler to get it cross compiled. Okay so we have two minutes or okay okay so we we don't have time but I'm available here if anyone has questions then please contact