 Hello everybody and welcome. So this subject is setting up Yocto builds using official tools. What that means that those things are available directly from Polky repository. But first I'd like to say a couple words about myself. I'm Alexander or Alex. My specialty is distribution engineering and Linux middleware and I try to help companies make custom distributions using the Yocto project and I also try to find time maintaining an open embedded core and keeping it up to date and fixing bugs when I see them. So before we go to the subject I'd like to quickly review what Yocto actually does and then we sort of zoom in into the problem and the tools to solve it. So here is a professional looking chart that explains all of Yocto on a single slide and as you may know Yocto is a set of tools to make custom Linux distributions. That's like a single sentence that you can describe it. So why would you want a custom Linux distribution? And the answer is usually because you do not want a distribution from a vendor. And why you do not want that? Well that we live out but so you decide to make your own distribution and the distribution is made of those things on the right. So images, packages, those are things that go into your product and then there are all those things that are needed to ship a product to the market. So bills of materials, license information, you probably want to know if the product is broken or not. So you need to run tests and have test reports. Maybe you need to know if it's vulnerable to exploits. And all of this is produced by a tool called BitBake and BitBake itself doesn't actually know anything about embedded Linux or building software. It's a task processor. So it builds up a graph of tasks with dependencies between them and then it works its way through the graph until it reaches destination or it finds an error and then it stops. And the tasks are Python or shell scripts that BitBake runs. So what defines the content of the tasks? Those come from things called layers. And today we do not actually talk about what is inside the layers, right? Layers have stuff. Stuff defines tasks and BitBake runs them. And another thing that is an input to BitBake is your local build configuration, those two files, localconf and bilayers.conf, that's where you make choices about how you basically parametrize your build and I'll get into that in a minute. So what is configuration of the layers? How do you set them up? So for each layer, you need to get it from somewhere, from some repository. You need to use a specific revision and you need to place it somewhere on a local disk so that BitBake can find it. And there can be several layers. They all essentially get repositories and each three things need to be decided for all of them. And that defines how your layers are configured. Then you have your build configuration that are those two files that you set up yourself. And later on I will show a video of that. I actually will show the example of what those files contain but that's where you need to make decisions before BitBake can start. The decisions are like what is your distribution because that defines how the layers are built and what target hardware do you actually want? For which target hardware do you actually want to build images and packages? So those two things go into localconf and then there is file bilayers.conf where you say to BitBake that I want this, this and that layer to be included in a build. And then there is a third file that is specific to your build machine like where you run BitBake where you can specify how much the build machine can basically cope with how many tasks can it run in parallel. And also things like download cache and estate cache go there as well. But what they mean, I don't talk about that. Once, okay, here's an even more professional diagram. Once you have set up a build locally and maybe it works perfectly and you're really happy with it or maybe it really doesn't work and it throws you some cryptic error. At some point you need to actually give all those things to other people. Maybe because they can help you with the error you're having. Maybe because they would build on top of your work add more things to your distribution. Maybe you actually don't want to give it to people but to your continuous integration system so that it can run tests on real hardware or the kind of tests that are not easy to run locally. So you need a way to kind of replicate these three things exactly on other people's machines. And by the way, the third thing is the target for BitBake that it needs to reach in order for build to be successful. It's what you say on the command line BitBake and something. So now we kind of get to the problem and the problem is that historically Yocto didn't offer any solution for this. The answer was typically that you need to use some third-party project external to Yocto or maybe you should write a custom script which is even worse because custom scripts are usually never documented and they are usually not understood by anybody in the organization. So please avoid custom scripts not just here but anywhere. So the tools like the pieces needed to solve this problem are appearing in open embedded core and this talk is based on Mikhail Dorr, the latest release that happened in April. So three months ago and what to do if you use something older than that I'll get to that at the end of the talk. So now we split the problem just one more time further into like two sub-problems. First is that you need to have those things layer config and build config that we just covered in a format that can easily be moved around to other computers so put them in some kind of box and you can give it to other people. And the second problem is the reverse. Once you get that box from somewhere you need to turn it into the layer configuration like layers set up, laid out on a local disk correctly and build configuration and then when you run BitBake, BitBake will do exactly the same thing as was done previously on somebody else's computer. And one kind of nice to have feature here is that to set this up you don't need any special tools. Ideally you just need to be able to run scripts and shell scripts and Python scripts and have the git on your computer and then the rest will be just handled automatically. Right, so now we finally get to the tools and here is like condense too long, didn't read, didn't listen version on one slide. But I'm not actually going to talk over it. It's just kind of one slide reference for you to like a cheat sheet that you can refer later on. So I'll cover those things in greater detail on the next couple of slides. So first we, I'd like to talk about how do you save and restore build configurations of local conf and layers conf things. So it's a two-way thing and the way to save them is to use this command bitbake-layer-save-buildconf. And what it does, it turns your build configuration into something called a build configuration template and that is the same two files but stored in a special location in some layer that you need to choose like conf templates and my template. So when you run that command you need to choose two things, the name of the template, like what your build configuration actually does, what it's for, what its purpose and which layer should it go to. So you need to consider that you should have a layer in your build that stores those configurations, typically a product layer like meta product or maybe meta person like meta alex. And the reverse action should be well familiar to you if you used Yocto. It's this dot or yeah, you need build nth, a tool that has been around since forever. And it takes as a parameter, not like there is a slight mistake there, but it does take as a parameter the path to the template in some layer and it takes those two files from that location and writes it into your build configuration. And what happens if you don't save each template you want, it will go to the default location in poke or open embedded core and copy the template from there. And that's how you end up with this familiar to everyone local.conf that has lots of settings and comments, like the starting point for any Yocto build. So this tool, oh yeah, you need build nth. It's not really friendly to users. I don't like using it at all. So we can do better. But because I'm talking only about things that are available in master branch and Michel do right now, that will be covered at the end in the future directions. Okay, so the next process is saving and restoring layer configurations, like layer locations in the internet revisions and where do you place them on disk. And to do that, you need to have active Yocto build, like something where bit bake runs. And then you say bit bake layers, create layers setup. And what happens is that the tool will ask bit bake to tell what are the layers that have been configured in this build. And then it will write information about those layers into the JSON file. And it will add a nice little setup script next to it. And that's all there is to it. Then you can place those two files anywhere you want. You can attach them to email and send it by email. You can place them on HTTP server for download. You can also place them in a layer so that people can clone the layer and run the scripts and all the rest of the layers will be set up. So that makes it like a bootstrap layer. And the reverse action is actually restoring all the layers. And to do that, you simply run that script. And that's all you need to do. It doesn't ask questions. It doesn't require parameters. Well, it has parameters, but you can just run it. And once it completes, you have the layers set up and ready to go. And this is a self-contained script. So it doesn't need anything except Python and Git binaries. All right. So we have gotten to the video. So now I'd like to show you how all of this works. So let me start the video. Okay. And I guess I need to go out of presentation mode. Yeah. How do you actually put till this work? Yes. Excellent. Okay. So let's restart the video. And the video actually shows how to set up a yoke to build from nothing at all because I didn't want to show some existing yoke to build. I wanted to show how do you start from nothing. And then, but let's just start it. So let's make a directory where we do all the things. Then we clone Pocky's Mikldor branch because that's where the tools are available. And that's where BitBake is also located. So you can run it. And it will take a bit of time. And it's not a live demo because it includes cloning things from the network. And I didn't want to tempt the demo gods by doing it live. Things will go wrong, I'm sure. So it's prerecorded. And it's me talking over it. Yeah. Let's make it, I can try to make it full screen better. And not really. Okay. Let's, yes, maybe I will re-record this with a bigger font. Yeah. So and now what is happening is that we need to create the cloned Pocky, but we need to create an additional layer where we can store things for later use. So the build configuration and as a template and the layer configuration. So you do this by BitBake layers, create layers, create layer meta Alex Kanavin, and that puts the layer on disk. And then you issue this add layer command to add a layer to your, to your build. And you can see that in Conf VB layers, it has appeared. So you don't need to edit that file manually. And the second thing is that we add a BSP layer meta Intel so that there are actually useful target machines in our build. And then we check what are available machines. So we pick the one that's most, most useful. And we go to local.conf. And that's where you set, say, I want to build for this hardware target. And the rest is deleted because it's not really needed. And the other thing is you just say, I want to build for this hardware target in a way defined by this distribution. So we pick not Pocky, but alternative Pocky, something else. Just to show how you can make this choice. Usually you have your own custom distro somewhere and we leave the configuration version as well. So now there is a ready build configuration. And the next step is that we save it somewhere in this meta Alex as a configuration template. So say build Conf. And you need to say which layer and what, what is the name of the template. So met Alex and the name of the template would be Alex's Gizmo because I'm working on a Gizmo. So it's saved and it tells you to actually document what the, that build configuration does. You need to explain to other people what is it for. And this is how you edit the Conf-Nodes text in the template directory and like write as much there as you find useful. But for the purposes of the demo, we say this configuration allows building Gizmo, building a Gizmo. And you will see later why this is useful to do. Now let me pause this for a second if I can. So this shows that in the template, the list of layers is not specific to a disk, but it has this magic token or a root that will allow restoring them on different locations on disk. And now that we have a build template, we need to create a layers configuration so that people can restore all the layers exactly as we have them on this, in this setup. And that's another bit bake layers sub command, create layers setup. And they need to say where, where does that layer configuration go? And one choice is that it goes to the same layer meta Alex. So let's see what happens. So it doesn't actually succeed because to save layers configurations, all the layers must be clean. They shouldn't have uncommitted changes because if they have uncommitted changes, that means that people who restore the layer configuration will not get the same content as what's on your disk. So we make meta Alex clean and actually a proper git repository with a commit ID. Just make one, one commit. So it's all registered in git and we try again. Yeah. And now it, it it worked. So it wrote the JSON file and the script that operates on that JSON file. It's a Python script. It's self contained. It's short. You can just run it. And the JSON file is okay here. I pause for a second. It just lists the layers where they come from and which git repositories and which revisions do you need to get. And meta Alex does not actually have a git location, but that's okay because it's like a bootstrap layer, meaning that you get it from somewhere and then you run the script from that layer. By the way, the videos are available on the conference schedule next to the slides. So if you can see this, you can run the video on your local laptop. And there are also like git describe strings that tell you a little bit what is it you're getting. So it's not just some random harsh string, but you get the rough idea what the setup contains. So this is first video that shows how do you create build configuration and the layer configuration. And there is a second video, shorter one, that shows once you have those things, how do you restore a build exactly as it was. So let's start that. And again, restoring a build. So I wouldn't, okay. So we make a Yachto demo copy directory and we copy this meta Alex layer there, but normally you would clone it from somewhere. But to simulate this cloning, we just copy it there. And then we run the setup layers. It's a standard name. If you see it, then you run it. And then it goes over the JSON. It clones everything that is listed in JSON, except the layer that which we already have, meta Alex. And cloning takes a bit of time. So we need to wait until it's finished. Okay. And then we need to set up a build from the Gizmo template. So we check that in meta Alex, there is Alex Gizmo template. And then we give that location to oh, yeah, you need built and that is in pokey. And that will set up the build exactly as it was in the first video. Okay. There is a type of here which will be corrected in a second, like an extra space. Yeah. And okay. So let's see. It prints like the standard Yachto banner. And after that it prints like the documentation for the template. Like this is where you explain to the users, okay, what do you do now? Well, it says allows building a Gizmo, but probably you should also say now run bit bake image, some image or some other bit bake target that is useful. And anything else that the users would need to know. And yeah, the bblayers.conf is set up to work from the copy. And the local.conf is as well set up as it was. And that's really all there is to it. That's all I wanted to show. So let's go back to the slides. There is really just one more slide. And that's the future, what still needs to be done. So one major thing that I think is missing is that setting up a build from templates is not super friendly and we can do better. So I'm proposing a tool called OES setup build that is nicer to users. And it's nicer to users because it can tell you what templates are available. It can tell you help you choose a template. It also picks a build directory for you so you don't need to decide where you want your build directory to be. And an important point, I suppose, is that it doesn't replace or up bit bake. It sets up a build for you and then stays out of the way. So the code for that is written. It's in review and hopefully it lands in in Yocto Master soon. Then the other two things are more hand wavy, meaning that it's just ideas floating around. No code is written. One is config fragments. So you can, like, config templates are static. Like once you write them, they stay as they are. You can't really modify them without modifying the whole template. But fragments can let you tweak your build just that little bit. And if we can have some set of useful fragments, then it should be easier to add features without reading documentation. And the other idea is an even higher level tool called OES setup, where you would hopefully be able to express what do you want out of Yocto on a high level. And then the tool would figure out how to make it happen. And the final point I suppose I want to make is that all of it is available in Mikildor. And if you are using some older Yocto, then you have to use something else, something external to Yocto. And the big three choices there are CAS repo and Git sub modules. And the CAS is actually the subject of the talk happening right after mine. So if you want to know why use CAS, why not use CAS, how to compare this, then you should stay here and listen to that talk as well. And I guess please ask any questions. And thank you. You mentioned the repo tool, and I'm familiar with quite a lot of companies that use this tool for Yocto. Do you think that what this new approach will be like better, or how is it better than, for example, using repo? Well, it's better. First of all, it's better because it comes out of the box. So if you're setting up a new project, you have the tools available right there, and you don't need to seek solutions somewhere else. And on the other hand, if you have existing projects using repo, then by all means continue to do so. If it works for you, I'm not trying to convince you to replace it. This is the primary target for this is newly set up projects. But if you look at repo specifically, there are a number of problems with it. Other than its external project, the specific problem size, it's a Google project. So Google sets its direction. It's very badly documented. And it's using XML, which I personally cannot stand to look at. Related to the last question. So basically, as an alternative to repo, you could use West, which is used by Zephyr. It's not owned by Google. And we're currently using it and you can make plugins to it easily. So basically, what you had there could be made as Python plugins to that. So you could just have your West workflow for you. Yeah, I just want to point out that these existing tools are all battle proven. So yes, people have shipped a lot of products with it and they work. This is not battle proven yet. It's just been released. But it has the benefit of being inside Yocto and being documented in official Yocto documentation. So people who start with Yocto, that hopefully will be the first thing they look at. Quick comment. So basically, adding more tools like OE something is not very helpful, because you need to read all the documentation to know that the tools exist. So you can just get like git help style of. Yeah, that you can do. Last question. How would I integrate this into continuous integration? So you didn't say how I could run a build with a single command. Is that possible? Yes. Well, you need to execute commands in a sequence. So there is no single command that would do all the steps, but each step is a single command. Can I put them in the script or because OE setup environment normally sets up another environment? Well, you put them in a script, yes. But we build things from small pieces that do one thing and it comes together. So trying to make all purpose tool will not work in the Yocto upstream world. Also because everybody has different opinion on what tool they think is best tool for the job. If you ask five people how we should do it, you get seven different opinions, which is happened right here. So we try to solve small problems one by one. Thanks a lot, Alex.