 Hi everybody, thanks for being here. My name is Luca Ceresoli, and as an embedded Linux engineer, I've been working for several years on Buildroot in the past, getting to know it quite well and become a contributor. Last year, I've moved to AIM Sportline, where I'm designing the next generation of dashboards, data loggers, and cameras for motorsports, and in that company, the Octo was being used, so I started to learn the new tool, and it was a big effort because it's so different from Buildroot even when you have to do pretty much the same things. So here are the most interesting experiences I gather from that, in the hope that they'll be useful for those of you who have to switch from one project to the other. Okay, so first of all, this talk is not a tutorial. You're not supposed to learn the tools now. This is not a feature comparison to help you choose the better product for you. If you need one, there are several. There is a talk, a blog entry, some articles to check those out. They are very interesting, and those are very fair comparisons, I think. So the conclusion of all of them is that both tools have advantages and disadvantages, and even more important, each of them is better than the other in several relevant use cases. So this is probably the reason they still coexist, and each of them is actively used, developed, and so on. So just comparing them on the technical side shows one very important similarity. They both maintain a graph of dependencies and use that to build the various steps where each step has instruction to do something on top of the dependencies. That's pretty much all for the similarities, technically speaking, because they are implemented using completely different tools. Buildroot is built using on top of two big pillars, which are the K-config configuration system from the kernel, which you probably already know. This is used to tell Buildroot what we wanted to do, so to configure it, and then it uses the even more well-known make tool to do the actual build. So Yocto is completely different. It uses a single tool called BitBake, which is pretty much used only for that purpose, I think, and it is tailored to that use, and it has its own language. It doesn't have a real configuration engine, so configuration happens through source files, but we'll see more on that later. Another important difference is Buildroot does, it's focused on embedded Linux devices, so most typically they use a root file system and they upgrade the entire root file system for several reasons, and so what it does is build a kernel, a bootloader, a rootFS, as well as a tool chain and so on. So it directly builds the root file system. Yocto instead is meant to build an entire distribution, and so what it does is it builds packages, IEPKG, RPM, DPKG, whatever, and you can use those to put them on a package feed on a server and devices can connect and do things like APT install, to install packages with dependencies, so it's really a distribution, and actually it can also produce the same thing as with root, so kernel, bootloader, tool chain, and most important, the root file system, as a sort of byproduct of packages. So to build a root file system, it just creates an empty directory and installs a list of packages into it. So this is very different. The goal of this talk is to cover the topics that exist in both build systems, so I will not talk about packages, I will talk about the root file system, because it's tuned for embedded devices strictly, and of course I will speak only about the topics I have some experience with in both build systems. Yet, even with those limitations, the list of topics is really huge, it will not fit in 40 minutes, so these are the topics that I covered in my slides. So how you bootstrap a project from zero, naming differences, how the two tools name the same things in different ways, basics on writing recipes, layers, and what's similar in build route, building and commanding a build, understanding what's going on under the hood, customizing the output that is generated, especially the root file system, and finally, and more advanced section on working on recipes. So I won't be able to cover all of them, so if you have any topic that you are more interested in, you can raise your hand and ask now, so I will focus on that mostly. Okay, so I'll switch to my default shadow, which goes mostly on the topics that are at least that were less obvious to me to learn. Okay, first thing to set up a new project from zero, well, it's all covered in the tutorials and manuals, so I will go very, very quick. The main difference is build route, the concept is you have a unique repo with build route, pretty much like the Linux kernel, while Yocto has several layers, so you just clone one repository to start working with build route. With Yocto, there is one repo, which is called Pocky, which collects the basic tools and the most widely used recipes, so you can use it for a reference design, for a reference build for like Beagle-Ballback, but if you're doing a real project, most likely you will need at least the open embedded layer, which has a huge list of recipes for a lot of pieces of software, and probably other layers will be useful, maybe one from your system chip vendor, maybe one for additional software like Qt-5 or many, many others. You should just go on the layer index side, as you are out there, it's very useful, it has a list of many, many layers that you can use with all the information about them. Okay, so after that you have to configure the build system to tell you what to do, and with build route you have K config, so you can ask it to show the available default configurations, which are minimal booting configs for many boards, you just load that configuration and you're ready, or if you want to start from scratch, you just enter menu config, and the basic you have to tell it is the kind of processor, so architecture, and the essence one, and the tool chain you want optionally, and then you have to tell it where to find a working kernel and a working boot loader for your target, which is probably from your system chip vendor or mainline, and then you have to tell it which dev configuration to build and which device to build, and that's it, so here's how you configure a kernel. With the Octo, your first thing to start after cloning is you must execute this script, we need to build the MV, you must source it, so there is a dot, hope you see it, and then you should look in your SOC vendor layer for a configuration for your board or a similar board, and then set that into the machine variable so it knows if you want to build for that machine. Okay, running the build is just a matter of calling make for build root, and it will start all of its process. With bait bake, you have to supply an argument, you can start with core image minimal, which is an image with just the basic to start, so both of them will build a minimal system to boot into busybox. Okay, of course the two system do many of the same things, but they name them differently, so the common thing is they maintain, as I said, a graph of dependencies between packages, but for each package, they also have a set of sub targets, of sub steps to accomplish, like configuring, installing, building, of course. So the naming is, the word package is used in build root where the word recipe is used in the octo, so it's pretty much the same thing. It's a description of how to build a package. In both cases, you can just call either make or bit bake with an argument that is the name of the package and they will build the package along with all of its dependencies. That's pretty much the same. But the packages compiled for the development host natively are calling host name of the package in build root and package-native in the octo, so it's exactly the same thing. The sub steps needed to build each package are called just steps, usually in build root, there is no formal name, while in the octo they are called tasks, and tasks are really first class citizen in bit bake, they are in a different namespace. So for example, you can ask it to just run up to the configure step for some packages with dash c configure, while build root creates a new make target for each of these steps, so you will make name of the package dash configure, name of the package dash build, and so on. But it's the same thing more or less. Okay, here it's a picture of how the default normal steps or tasks are organized in the two-wheel system. They are very similar, except for the naming, so source becomes fetch, and so on. Okay, the build step is kind of ambiguous because in build root it means run the build, the compilation itself, so before, without installing, while in the octo it is in a special task, meaning build everything related to a package, it's what happens when you do bit bake space name of the package. Okay, installing has two different steps in build root to install in staging and to install on the target root file system, while the octo manage this internally, so there is only one install step. And finally, what the octo calls a deploy step is install images in build root, that's to install just images not in the root file system. Okay, this is just a recap of what I said with naming one by one as a reference table, and along with the name of the files where this happened to be implemented in the two systems. Okay, it's impossible to talk about the octo without talking about layers, so just a few notes, when you start a new project, there is a default layers file that is created for you with a simple list of paths to the layers, this is the default, it has three layers, and then you can add more as you want. I suggest you use a relative path, not an absolute path, otherwise it won't be portable with your colleagues, for your colleagues, for example, or on to different directories. Okay, and what happens with layers is a recipe for a package is defined as a .BB file in one of the layers, but then other layers on the higher levels can have a BB append file, so there can be zero or more BB append files, and what it does at the beginning while parsing your code, it will just pretty much concatenate the files, appending one after the other, so the final recipe will be a concatenation of them, and your highest level BB append will come last and can change what has been set in the previous files. Layers are very powerful, but can also create some problems, especially some of them have a very good quality, some have not, and but most important, there can be conflicts, which is that your build will fail. This happened to me, for example, with Gstreamer where a vendor layer wanted to implement their own video codec hardware, and so tweaked on the Gstreamer recipes with the BB append, which is not compatible with what other layers do, and so in those cases, my personal algorithm to fix the problem is, if at all possible, fix the offending code in my BB append, maybe it's just a matter of changing one or two variables. If that's not enough, you can just blacklist one recipe, so it will not be considered by big bake, and then you reimplement what's needed on your own layer, and finally, if the whole layer itself is creating problems, just don't use it and copy over the part that is useful in your layer. So these are possibilities to escape this kind of problems. Okay, and finally, you're not supposed to modify the existing layers, but you should create your own top-level layer where you add all of your customization specific to your product, and maybe fixes to lower-level layers, and everything should go in there. Okay, BuildRoot has mechanisms which doesn't really have a nice name, but BR2External is the Kconfig variable to enable it, and it's also how it's usually called. It is somewhat similar to layer technically, but it's simpler. It's also simpler in its goal. Its goal is not to give something that can do whatever to change every recipe, but just to keep your own local customization in a separate place. So you are developing a product, and you have your own packages, your own board configs and so on. You can keep them in a separate place. If you find bugs or problems or missing features in a BuildRoot recipe, the policy that you suggested to follow is to submit your improvement upstream, and then it will be part of the whole BuildRoot. Okay, this is how you use it. You just have to call make whatever, like make menu config with BR2External and the list of your external trees. That's it. That's saved in your DevConfig, so you don't have to type it again every time. What happens basically is the top level BuildRoot make file will include your own make files in your external trees, and the same happens with Kconfig files. So it's all pretty much a part of a unique system then. So in this sense, it is somewhat similar to layers. Okay, writing recipes, I will mostly skip on the section because it's well covered in tutorials and manuals. I just wanted to highlight how, as a bird's-eye view, it looks very similar to this one. The syntax is different, but with BuildRoot and Yocto, in both cases, you are just telling where to download the code from, how to build, how to install, and that's pretty much the same you do with Yocto, with the difference index. Okay, so actually in BuildRoot, you also have to define a Kconfig file to let your package be visible in menu config, but that's very simple, at least for normal packages. And okay, in Yocto, you have classes, so if you have an auto-tools package, you don't have to re-implement everything in the same way every time. There's a class doing all the basic stuff for you, and it's extensible. Exactly the same happens in BuildRoot. The syntax is cleaner in Yocto. You do inherit auto-tools, which is very intuitive. It is a bit less intuitive in BuildRoot where there is this syntax because that's based on make and it uses make features to implement classes. They are called actually package infrastructures in BuildRoot.h. It is used in the same way to implement packages. Okay, if you use classes, but same applies to BB appends, you can add code to be executed before or after every task. So you can have things like do install append and it will be appended to the existing do install step from your base class or base BB file. In BuildRoot, this is called hooks. So for every step, there is a pre-hook and a post-hook. So you can append code. Again, the syntax is a little bit more verbose, but you can do the same things. It's completely equivalent. Okay, so here, just an overview, showing that both tools have created several variables for you to use in your recipes. Like for example, PV is the package version in Yocto and name of the package, and the score version is the same thing in BuildRoot. There is always the name of the package in front of variables for BuildRoot, and that is because BuildRoot uses make and in make, all variables are in a unique namespace. So they need to have a prefix. With Yocto, these variables are assigned per recipe, so they don't need a prefix. It's a little more concise. There are variables to know the various directory you have to work in, the source and build here, where you should install and install images. This mean a bit different things in the two tools because BuildRoot installs directly in the root file system while Yocto installs in the temporary directory, but that can change in the future, and so you should not rely on that. Just use the variables and don't make assumptions is always the best thing to do. Okay, finally, you will need to add patches to packages that you are not developing directly. So in BuildRoot, it is rather simple. You just have to put a .patch file with an any convention in the package directory, so it's going to look like this. In your package directory, there is the mk file, the kconfig file, and there is the patch here. In Yocto, it goes into a subdirectory, at least the convention, but you also need to set in source URI the name of the patch, otherwise the file is ignored. If the subdirectory does not have a standard name, you also need to list in files extra patch prepend. Okay, otherwise you can have a patch downloaded from the internet. In this case, it is the name of the package and the score patch in BuildRoot, and you just mentioned it in source URI with Yocto. Finally, if you need to patch packages, which is quite likely, that are not in your layer, but are in lower level layers, the way in Yocto is to just put them in your layers and mention them in a bbappend file, so that's pretty much the same mechanism as the first line table. With BuildRoot, there is a configuration variable called global patchdir. So this should point to a directory where you have a tree of patches and it will pick patches from there without having to change them in the BuildRoot tree. And it can be used also, of course, with BR2 external. Okay, so building, and I mean invoking a build is happens in a very similar way although the syntax is different. So what I've already shown some of the common lines is just an overview of other possibilities. Maybe the most complex one cover here is make busybox reconfigure, which is equivalent to bid-bake-upperkc-configure-busybox, which means forced to rerun the reconfiguration and everything else to build that package. So they can do pretty much the same thing. Okay, let's skip in on these. Okay, with BuildRoot, you can do out of three builds if you want, like from the same BuildRoot code to test different configurations, you can just put them in different subdirectories just like you do for the kernel and then you can launch two different consoles and build in parallel to two or three or more to see how a change in your source affects several boards. This is not present, I think, in Yocto, but you achieve the same probably in a better way using other techniques. So you can just bid-bake core image minimal and then bid-bake my image and it will build only the parts of my image that are not already built from the core image minimal. So it will recycle the builds very effectively and also you can change the machine then and build for a different machine and it will also recycle whatever is common. So if the machine is using the same target architecture, it will not reveal most of the code, only the board-dependent code. So you will save both on this kind of time. Okay, dependency tracking is very different. Build route keeps track of completed tasks using a stamp file. The stamp file is an empty file whose date matters and this is not covered in all cases because if you build a package then change the configuration and it doesn't know it has to rebuild it because the stamp file is up to date. So in that case, you need to manually rebuild and to trigger a manual rebuild. Well, the most simple and safe way is just to run make clean. With make clean, it will then restart building everything. Of course, it takes more time. If you know what you're doing, like you know why just change something affecting this specific package, you can just reconfigure or rebuild that package and save a lot of time. So with some knowledge on the packages that you are having on your target device, this gets to be relatively simple in many cases. Okay, Yocto has a much more powerful mechanism. So it keeps a hash of everything that is in the recipe. So if you change one variable, one function in your BB file, then it notices the already built version was built with a different hash and it will rebuild it automatically along with all the venasses. This is very powerful and handy. You don't have to think much about what you're rebuilding. On the other hand, it forces you to rebuild things that at times might be necessary. Okay, if you still want to force a build, you can do it with the dash F flag, which forces rebuilding a step even when it is already built, useful for debugging. Okay, so after finishing your build, you wonder where are your output files. It took me quite some time to understand with Yocto because it has a very deep and huge directory hierarchy. Well, with build route, it's very simple. There's the output directory. It's the default name if you don't change it. It has a build subdirectory and then there is a subdirectory which builds each package. For example, busybox. While Yocto has a much deeper hierarchy, where this is where you're working, the build here, and then the tuple like x86, Linux, blah blah, and then the name of the package, the version, and then there are many directories. There are many because it does many more steps. It does out of three builds when possible. So you have source and build output. And then when you install, it doesn't install in the target file system but in any directory called image. And then with internal tasks, it takes from image and puts in package and then in package splitting the development files and the files for the target documentation and so on. So it has several steps that are useful to inspect where your build started going wrong, for example. Okay, the root file system is generated with build root in the output slash target. So this is pretty much your final root file system, more or less, but it's where you find all of your files basically. In Yocto, there is no concept of the root file system but there are images, you can have multiple. So they are pretty much like the regular recipes, although they don't really build stuff, they just collect stuff. So in the same hierarchy as before, more or less, there is this core image minimal and it has a subdirectory rootFS that contains your root file system which is collected after building your packages. Finally, the images that you really deploy to your devices are in output slash images with build root and in well, pretty much the same hierarchy, but instead of work, it has a deploy here. So here are your images and they are more or less the same result now. So this is where you find them. Well, the Yocto hierarchy is a lot deep so you'll use a lot of your tab key definitely, but after some time, you get used to it. Okay, this is to me a very important chapter because it is where I spent a lot of my time in understanding why I was seeing red text on my screen, the build was failing and it's very important to understand what the tool is doing. You know, these tools are not mind sweeper, they do a complex task, things may fail in many different ways. So you'll definitely spend some time in scratching your head with a question. And so one of these questions is what will the tool build and why does it build so much stuff? I just want busy box, it's building tons of packages. Well, build root has a neat tool to do that. You just type make graph depends and it will produce a PDF with all the dependencies. I don't know if you can see it, but there are two different shades of gray here. The dark gray is target packages, the lightweight host packages that don't go on your device. So it's very easy here to understand why the tool is building, I don't know, host zealib because it's needed by somebody and also useful to fine-tune what goes on your target and to save space. If you only care about some of the information, you can filter them. There is an environment variable which can do many things. Among these exclude the host packages. So I just want to see what goes on my target and then you can, where is it? Ooh, make name of the package graph depends. It will only show the name of that package and its dependencies. So it's nice to see, okay, if I use this tool for implementing this feature, how many dependencies will it have? How much space will it take on my device? I think there is no similar tool in Yocto. Well, there is one. It can generate a similar graph in a dot file but it's not really usable. It takes hours to generate and then the graph is not really readable. I think the closest thing is task explorer which task dependency explorer which is invoked this way, bit bake dash G dash U task expert and then the name of the package or word to see all the packages. And it doesn't show a graph. So it only shows for each item, its dependencies and reverse dependencies. And it's also much more fine grained because it shows tasks not packages. So it shows a huge amount of information. It's not really easy to find out what you're looking for. Okay, then you want to understand what the tool is doing, of course, especially what went wrong when build fails. And so, well, build route by default shows a very verbose output. So I'm starting this step and then the whole step output. And when things fail, typically, the error is in the last few lines on your terminal. If you want to concise output, there's a utility script called brmake which shows only the start of each task and date and save everything into a log file. Yocto has a different output. It shows in a very visual way what is happening in this moment. So if you just turn your head and then look back into your screen, you don't know what happened in the meanwhile. But it's very effective in saying what is happening at the moment. If you want a sort of concise output, very stupid trick I found is just pipe it into cut. So it will show like, I'm starting this task, I'm finishing this task, which is also what happened probably in your CI machines because it doesn't have a terminal. So it's more useful if you want to look after at what happened, okay? And when something goes wrong, you can inspect the build logs. For each step, Yocto saves a separate build log. It's in one of those long directories. So for the package, there is a temp directory with all the logs. It's very useful because you can inspect whatever happened. But if you want to see the error happening live, then you can just rerun it with the dash v flag, which means verbose. And in that case, whatever is logged is also showed on screen. Okay, and understanding what the tool is thinking, it means you wrote a recipe, you set the variable, and you thought it had the fact, but it doesn't. So why? So you can inspect the internal thinking of the tool in build route using the print bars feature. So you just, you can just look at the second example. It's a typical usage, make dash s, you give a pattern and then print bars. It will show all variables starting with busybox. So if you don't find the variable you just set there, maybe you have a typo or anyway, you can see the value of each variable after expansion. While if you want a much more low level tool, you can use directly make. Make has this dash q dash p means it prints the whole make database. It's a huge output where all variables are before expansion. So it's a lot harder to understand, but in some cases it could be useful. And it also showed the rules and the action it would execute without executing. Yachto has a similar possibility with the dash e flag. It's a bit big dash e show only the global environment, the variables affecting all of the build entirely. If you pass a recipe name, it will show the variables and the function for that recipe along with where they have been set. It is very useful especially when a variable is set and modified in several different files. So you can see which files actually cause in trouble to you. So that these are very important tools to understand what the two build systems are doing. Okay, now I'd like to give you some ideas about how you customize your root file system and your build output in general because with a default build you have a default system but you don't sell big old bones with the default busy box image. You sell products so you want to customize stuff. And for this, well, as I said, build root has its configuration system with its configuration file. You use menu config to customize. And well, the command make safe that config will overwrite your previous configuration with the current values. So if it's version, it's very handy. You are ready to commit it. With Yachto, the configuration actually goes into BB file so there is no separation, no strong separation between configuration and build. Anyway, the layout that is probably quite common is to, well, define all of your build option, the tool chain, and especially your target machine in a configuration file in your top layer. Well, you could use the conf slash local conf file in your build directory. It works but it's not meant for that. It's meant for really local stuff. But basically you can choose where you put your file and then you can set that. And the system configuration, so what those options that affect generally the system must go in several places and you have to look for them. And finally, to add packages to your root file system, you use an image recipe, as I mentioned before. Okay, so adding a package in build route is very simple. You just enter menu config, go into the packages submenu, and if it is implemented, you will find it there. It's very easy, you can search. And then if you removed or modified the configuration of some packages, you have to maybe clean and rebuild. If you, or anyway, then you save your configuration and your package will be in your image. With Yocto, it is a bit different. There is no menu, but you can ask BitBake to print the list of all the packages that it knows, so those that are in the layers you have enabled. With BitBake layers, show recipes. If you don't find it there, you can go again on the layer index. It also indexes recipes in layers. And so you can just look for a recipe and see which layer is implemented. So maybe that layer is useful to you. You should pull in that layer, probably. And then to create a root file system, you create an image recipe, like the core image minimal shown above. You can create another one. Basically, an image is a collection of packages. So this is a very simple image recipe. You do image install plus equal a list of packages. Well, not only packages, but also package groups. Well, the package groups is a subgroup of packages, so you can just include a package group for debugging and include that or not. So actually, what BitBake does, it builds packages and then from that, it picks those packages that you listed in the image and installs them in the root file system. Okay, there are many root file system customization that affect the whole system and build root has a really nice menu for that. So you just enter menu config and system configuration and you'll see options to change important things like the system, their management, passwords, root password and many, many more. So actually, what happens is everything is stored in your configuration and most important, it is very easy to find all of those options in a menu. It's very handy. With the update, it's more difficult because for each of the features, you have to find where it exists in which recipe and so my personal algorithm to do that is first I look into the Yocto Reference Manual which is very comprehensive. If I don't find it there, then I grab into the pocket sources because maybe there is a variable which is not documented and it does what I need or otherwise you just search the internet. Then for, I listed here just a very small selection of possible configuration. It would be too huge to leave them all but starting from very basic and mundane things like setting the host name. This is the menu entry in build root on the left so you just go in system configuration and find this and set your host name. In Yocto, you have to set this variable here which come from the base files recipe in Pocky. This happens in similar ways for other settings and more important things are the internet system, the dev management and this really affects the whole system. Okay, this is just a brief list as an example. Okay, then if you need extra root file system or image modifications, then build root has two tools. One is the root file system overlay directory or directories. You give it one or more directories and at the end of root file system creation it will just copy your overlay over the root file system one by one. So it's actually really an overlay of everything. It's typical for configuration files and similar that are global and not specific to a package. If that's not enough you can ask it to run a post build or post image script. It will just be run after each of these two steps. It's called script but it can be any program actually. And then you can do everything on your root file system or your images. The same thing can be done in Yocto with the root FS post process command and image post process command. It's very similar. Okay, I think I'm kind of out of time. So there is probably time only for one question. But if you have more questions I will be available during office hour on Wednesday on, well it's in the built-in sitting. It's just over here on the right. So I'll be available for an hour if you have additional questions. So if there is a question now I think we have time to reply. Yep. Require not to include packages, recipes with specific license. And Yocto provides us this mechanism to exclude recipes with, for example, specify the list of licenses. How can I do the same with you? Okay, the question is in Yocto I can ask it to explicitly exclude recipes with a given license. How can I do it with build-root? Build-root does not do that directly. Actually, one of the thing is build-root typically is used to build a small to medium system where you have quite a good knowledge of what goes in your system. So you just have a look at the other penances. Build-root on the other side can produce a very detailed reporting about legal or related information. You just do make space legal info and it will produce a directory with all the license files, the tar balls and so on. And also a report, a manifest, which is a CSV table with all packages on the target and all host packages and with their license and additional information. So you can look into that to see if you have any package with a forbidden license. It's a very useful tool so you can do that with a little manual intervention. Yes. Maybe just another question before we go. Okay. Do you have a field for vendors supporting the opto or build-root where it's gonna go in the future? Sorry, do we have? Field for which is gonna be more prevalent in the future? More? Which tool is gonna be used more in the future by vendors? Which tool is gonna be used more in the future by vendors? Well, I really don't know. I think the opto is more used by vendors. Probably, yes, I think it's more used by vendors. However, this is related also to the philosophy of the project. So with the opto, it is very typical that each vendor has their own layer. So nobody, they don't have to go through a patch review process. They just put their things online and they are there. This is a pro and con so they can publish freely, but on the other hand, there is no quality assurance and complete testing. With build-root, most typically, to implement a new platform, you don't need much in build-root. In most cases, you just need a dev config, saying pull the kernel from here, pull the bootloader from there. This is the configuration to use and that's it. So except for some more complex architectures, that's all you have to do. So I think some vendors just don't do that because I don't know, it doesn't look so serious stuff like having a layer, but it's actually a matter of having a working kernel and a working bootloader. Well, a little chain, of course. Okay, time is out, so thank you very much for your attention. All right.