 Bonjour et bienvenue à ce taux d'optimisation et d'octobre qui nous présente les meilleures pratiques. Je suis Alexandre Belloni et je travaille comme l'engineur Linux en Boutlin où nous faisons des développements, des consultations, et nous avons aussi des trainings. Nous sommes complètement concentrés sur le Linux et surtout sur le kernel, mais nous faisons aussi des travailleurs et des systèmes de bouts. Nous sommes souvent utilisés pour intégrer ce que nous développons à l'intérieur de l'Octobre ou de la Boutlin. Je suis aussi le maintaineur de l'Octobre Linux et je suis le co-maintaineur de l'Octobre pour le soutien kernel pour différentes familles de microchips. D'abord, let's talk about distributions. First, I want to define what we are actually talking about. For example, you see there that the Octobre project is not actually some code that you can use. It's actually an entity that is composed from multiple member companies and they are working on different projects or code repositories. So the first, the main one, let's say, is Pocky and Pocky is the distribution and this is actually what you can use to build your root file system or your embedded project. So Pocky is the distribution and it's composed of open embedded core and BitBake and then a few other layers that are Metapocky and Metaioctop BSP. So Pocky is the reference distribution. It is not really tailored to your embedded system. For example, it will always include OpenGL even if your SOC doesn't have a GPU. So not only you will waste time building OpenGL support but also you may generate an image that will not actually work for your platform. So it can definitely generate demo images and it can be a good starting point to know whether you can actually flash a platform or if you want to quickly show something working on your platform. But it's not really meant to be used as is on production systems. Also you have to remember that the included features are not stable and it's not reliable. What I mean is really that the number of features or the kind of features will change and one of the examples I have there is that it switched from XOR to Wayland a while ago and that may be breaking whatever you were doing by using Pocky. And like I was saying, Pocky is just bundling open embedded core, beatback and two very, very small layers. So one of those layers is Metaioctop BSP and it is a BSP layer that is adding support for reference boards from the Yoctoproject members. And also you have Metapocky, that is the actual distro layer that provides for different distributions. So that will be Pocky, PockyTiny, PockyBleeding and PockyLcfg. So I'm not saying that the Yoctoproject is not doing a lot of work because they are actually doing a lot of work but the bulk of the work is not actually in the Metaioctop BSP or Metapocky layers and more likely they will be improving open embedded core and beatback. So for your project, I will say that not using Pocky as some advantages I will even say that it has many advantages. So when reporting bugs, for example, it will most likely be necessary to reproduce with no distro builds, that is without using any distribution and that's a way to ensure that it is not a distribution that is creating the bugs or issues you are seeing. It is also easier to start from a no distro and create a distribution rather than tuning a distribution that is starting from Pocky.conf because as I was saying, Pocky is trying to include a lot of features and enable a lot of features so that the build testing will be able to compile as much as possible and so most of your tuning will be removing features from Pocky.conf And it's way easier to add features than remove features. It is also easier to work with the open embedded core repository when you want to send batch upstreams. The main reason is that then you have the actual commit ashes because the Pocky repository is combining the open embedded core repository with the other layer directories. So the commit ashes will differ. And also, one of the advantages will be confidentiality because Pocky is defining the premier variable and that variable will point to downloads.yachterproject.org meaning that it will leak the name of everything that you are building that is fetched using version control so anything that is fetched over subversion, git, mercurial and well, other version control. The main drawback that you will have there is that you will have too much open embedded core and beatback. So for open embedded core you will have the usual open embedded branches name so that will be, for example, dunfell and for beatback you will have branches with a version number and so you will have to match them. So, what I really recommend there is that you will want to create your own distribution and it's not actually that difficult. You simply have to create one configuration file and that configuration file will be in conf slash distro in your own layer usually depending on the size of the project usually what you will do is simply include the distribution configuration inside your BSP layer but you may also want to create one layer with your distribution separately from your BSP layer. So inside that configuration file you will define whatever policies that are distribution wide so that will be the tool chain selection and that tool chain selection will include the leapsie selection that's also where you will want to select in it so system 5 or system D you will set the distro features variable you may set a few preferred providers for example for the libjpeg and you also want to set package classes there so package classes will say what kind of packages do you want to build for that particular distribution which is also for example one of the difference between pocky and open embedded core because the default for open embedded core and a node distro build will be IPK and the default for pocky is RPM something also that you may want to set in your distribution configuration will be the QHX that you want to do and if you want to switch some warnings to errors or create new warnings or new errors or new QHX that's also where you will do that so something else I want to talk about will be local.conf and I want to say that local.conf is really for your local configuration so that will be for example the CPU number of your current building machine or the disk space or you want to react when the disk space is low and for example if you want to remove the work directory or that kind of things really anything that is really local to your machine and that does not affect the output of the build the main reason is that the distribution of the changes that are made in local.conf is not easy because that file is really a configuration file that should only live on your machine and you should not care about distributing that file or putting it in version control so you don't want to affect the reproducibility of the build with the variables that you are putting in that file or defining that file so there is also a huge drawback is that when you change something in local.conf beat bake will parse all the recipes again which will take like 1, 2, 3 minutes every time which can be kind of annoying after some time I will say that it is fine to carry changes in local.conf for development testing it's not an issue as long as you then move those to the proper location when you are done with your development and testing something that I really want you to do is to avoid the numerous tutorials that are saying that you can do those changes in local.conf so I will have some examples one of the things you can do is that instead of putting variables in local.conf you can have them inside.conf which will be used for site-wide configuration and that's where you will set up the company proxies, mirrors the shared estate cache location that kind of thing so anything that will affect all the build machines inside your infrastructure unfortunately that file kind of suffers from the same local.conf distribution drawback meaning that it's not that easy to provide a site.conf and ensure that it will remain synced with the other sources from your layer so the other location where you will put variables that you are changing in local.conf will be inside an image recipe and it's actually the most abused variable that is put in local.conf and that image install happened and that's definitely what you can see in tutorials from many vendors it's working but it's not really something you want to do it's not even easy for beginners to understand why they have to add that underscore append keyword after image install and that's mainly due to the parse order because you can't use plus equals there because your image recipe is parsed after local.conf well also this solution to that is simply to create your own image recipe as soon as the core image something so core image minimal or core image sato recipes are not enough anymore and it's again it's not that difficult it's mainly inheriting from image base and then setting the image install variable or the image features variable so it's not that difficult to create your own image recipe all the machine related variables have to go in the machine configuration and that's also something you can see some people are putting in local.conf the preferred provider virtual kernel preferred provider virtual bootloader and the preferred version for those so that would be preferred version linux preferred version uboot or those kind of variables something also would be the image fs types so you could argue that image fs types should go inside image recipe but I would say that it kind of depends on your platform because maybe your platform has a specific image type requirement depending on I don't know whether it has emmc or nand you may want to select different image fs types depending on your actual machine and not depending on the image that is generated in a few cases you may also have some image install append in a machine configuration and that is for example to install the kernel in the root file system because that's where in that case the bootloader will look for it so I will do that in again I will do that in the machine configuration as opposed to the image recipe because depending on the machine maybe the bootloader will look for the kernel inside the root file system or maybe it will look for it in a separate row partition or maybe a fat partition again I would prefer to have it in a machine configuration because it's actually depending on the machine then you have the distribution configuration itself so that's where all the other variables should go and that's where the remaining preferred provider and preferred version will definitely go that's also where you will define the package config for packages so package config allows you to define what kind of features you want for a particular package so this is definitely something that should go into your distribution configuration and you will also want to set what kind of license is incompatible with what you want to provide and what kind of license you want to whitelist so if you want to whitelist any commercial license I will do that in the distribution configuration but it's kind of also where I would put anything that you could also have in the site.conf because while the distribution configuration is easy to distribute, it is part of your layer and so for the ease of distribution and version control you can move for example premirons or proxies configuration inside the distribution configuration so release management there are multiple tasks that open embedded or bead bake base project will let you do on your own build reproducibility and the fact is that using bead bake you will be able to generate reproducible images but you have to get those layers first so that's the main thing you have to do first so that's the first step fetch open embedded core, bead bake your layers, set that up create your local.conf, bblayers.conf so that's up to you to do that and also something else is that the release tagging it's up to you to select your own so usually you will not do that manually every time and you will use a separate tool the usual solution will be combo layer that's what pocky is doing combo layer is the tool that is used to generate the pocky repository and again the input open embedded core, bead bake and those two metioctobesp and metapoky layers and so using combo layers they are combining all those repositories or part of those repositories inside a single repository so that's also something you can do you have some documentation on the yoctoproject wiki and you have a link to that there what you will kind of not so often see will be the use of git submodules and if you have a look at the combo layer documentation it will explain why you shouldn't do that but I would say that's still something you can do so usually you don't have just the git submodules you usually also have a setup script there and you have a great example in ui distro so just have a look at what they are doing you will see that it's mostly the setup script and that setup script is setting up the submodules for you so that's a nice way of doing things on my side I am mostly using repo and that's because I was first working on android and so repo is an android tool and so I'm kind of used to it so I'm using repo you have an example in metafree scale so that's how I started and well I'm now used to that so that's what I'm using and I will have an example of repo later you also have CAS it's fairly recent I think from 2017 and CAS is a dedicated tool for open embedded and really it's making your life really easier and I'm not using it that much yet but really that's definitely what I will be using from now on so like I was saying repo is used in android to distribute its source code and the source code of android is split in many git repositories and it's basically the main use case that we have here where we have layers in different git repositories we fetch them and put them at a single location so that they are available to be used so the repo configuration itself is stored in a manifest file and that manifest file is usually in its own git repository that's what free scale is doing that's what android is doing now I've been used to also simply put that in a specific branch of my custom layer and that just allows me to have one less git repository to take care of so it's kind of not obvious at first but well you can have a git repository with 2 different projects your actual layers there and then in a separate branch inside that same repository you will get your manifest file so repo will only handle fetching the code and local.conf and bblayers.conf will have to be handled manually so the manifest example is there so you can see basically you will define some default values in that case the default value for the revision will be donefile so that if you are not defining the revision for a particular project the donefile branch will be fetched and then you define your remotes there we have 3 different remotes one for open embedded, one for free scale one for our own server and finally you define your project so what you want to actually fetch and where you want to put them so first project will be the remotes we use will be the free scale one and we will fetch a project that is named meta free scale and it will be located in sources slash meta free scale then we fetch open embedded core and then we fetch bitbake so for bitbake as you can see it's path is inside the open embedded core path it core will look for bitbake then we have meta open embedded there and finally we use our own private server to download our custom layer in that case it is named meta custom and as you can see for that particular project we have a copy file entry and that entry will say ok I want to take something from SRC and put it in test so in that case it will take buildconf setup environment from inside the meta custom layer and it will put that in setup environment in the root of where you are running repo so that's a way to distribute your setup scripts and that setup script will usually set up the bitbake environment so it will most likely source in it but it will also set up local.conf if they don't exist yet again you have an example with what free scale is doing so I will let you have a look at that to tag release a few steps have to be taken so you will optionally tag your custom layers it is really an optional step you don't have to do that but what you do is for each project entry in your manifest you will want to set the revision parameter to either a tag if it exists or a single commit hash and finally you commit and tag this version of the manifest and so it will ensure that basically all the everything that will be fetched that ensures that from a release point of view your tagged manifest will always download the same sources so you also have CAS CAS is the specific tool developed by Siemens for open embodied you have a link there and CAS will fetch layers and build the image in a single command so that's really a nice it's even nicer than Hippo in that sense because then Hippo is not specific to open embedded but CAS is so you can use a single configuration file that can be either JSON or YAML and usually it will be part of the custom layer so from the beginning it is designed to be distributed with the custom layer something to note too is that it can generate and run inside a docker container which is really convenient because then you don't even have to set up your own docker container it will do that for you which will lower the risks of host contamination of your build it will be able to set up the local.conf nbblayers.conf automatically from what is inside the configuration file which is really important so the CAS configuration looks like that so you have a small header defining the version in case of where you will be updating CAS just to know so then you define your machine that will become the machine variable inside your local.conf you can define your distro that will be the distro variable in local.conf you can define a list of targets that will be the target that will be passed to BitBake so usually it will be a single image target but it can be multiple packages targets or whatever it's not an issue and finally you will define a few repositories so in that case you will use metacustom you don't define the URL because metacustom is the first layer that you will download to actually get that particular configuration and CAS will be smart enough to know that then you want to download BitBake and OpenUmbitCore so you have the definitions there and if you see for OpenUmbitCore you have layers variables that is a list of layers and under that you have Meta and Meta is a single layer of OpenUmbitCore and that will instruct CAS to add the path to that meta-layer to bblayers.conf so the same for Meta FreeScale so you pass the URL to that layer the branch you want to download and then you have Meta OpenUmbitCore and in that case Meta OpenUmbitCore you will get 3 different layers which are Meta OE, Meta Python and Meta networking so it's really nice because then it allows you to well CAS will generate automatically bblayers.conf for you something you can see there is that so you have a ref spec and it's kind of the same as the revision from the repo manifest once you are ready to release you just have to switch that ref spec to a single commit hash or a tag and then tag your own layer and that will tag that particular configuration file and meaning that then you will always download the same sources from that particular configuration file so something else you want to do when doing your final release is that you definitely want to ensure that all the code is already available internally and that will be either on your local build machine or at least on local internal mirrors because if you want to ensure reproducibility you don't want to depend on external servers or sources to be hosted somewhere you don't know so the first step will be to ensure that you don't have any SRC ref equal auto ref in any recipe and then you want to generate the mirror tables for everything that is coming from git repositories or version control and so for that you want to set the bb generate mirror tables variable and that will generate the tables in dldr for you you fetch all the sources so either you beat bake your target completely or you just do beat bake-c fetch all target so that will fetch all the sources that are necessary to build that particular target then you can archive dldr and make those tables available internally so if you only care about the local machine you just have to keep that dldr but if you want to provide the tables as a mirror then you will have to make them available on an HTTP server optionally you can build once with bb nonnetworks set to one that will check for missing target but at that point you shouldn't have any missing target but more importantly it will check for the remaining auto ref that you can have in any recipe finally you want to point beat bake your own internal mirrors and you can do that using premirrors or you can use your own mirror class and set the source mirror URL I will let you have a look at those variables but in the documentation but it's easy to use them and final step will be to build the actual release from scratch premirrors only to ensure that beat bake will not be allowed to go to the external network but only use the premirrors that you have set so that should be your internal mirrors so build optimization you can share this estate cache which is sharing the shared state cache you can share it across multiple build machines and usually the steps to do that will be to first set up continuous integration or nightly builds then you use a dldir from those build machines to populate the premirrors and finally you will share the estate cache so the estate cache is located in the estate dir and you share it over nfs or sttp usually it's fine to use sttp because you definitely want that to be read only and not as un populated by whatever machine that is on your network finally on your local machines you can use the estate mirrors to point to that particular share meaning that your build machine will fetch any state cache or state object artifact from the mirror that should speed up your build quite a lot especially if you have a fast network which is usually the case and this only works well if all the hosts are similar enough and usually meaning that they have the exact same distribution running the exact same distribution and distribution version because your host will have influence over some estate cache signatures or checksums so something that can help there will be the containers and that's why I was saying that using cache is really nice because it will generate those containers for you you can also want to clean that estate cache because it will only grow over time so you may want to clean all data using the estate cache management also one of the topics that is quite important is license compliance and something that is nice will generate license manifest for you and it will list all the software that is present on the target image and that will be in the license directory so usually it will be in tmp deploy license then the image name and then license.manifest so you have an excerpt there of that manifest so it takes back the package name its version from which recipe it is coming from and the license that is used so that will help you in your license compliance efforts for some licenses you have to provide the license text and to do that you have two different ways you can select to copy all the licenses inside your image road file system you can do that with copy leak and copy leak manifest that will copy the license directories and the license manifest inside your road file system but then if you are using package management on the target you will have no way to add new licenses all the other solution will be to use license create package set that to one and that will ask BitBake to also generate the packages that are including only the license and so then you can install the required license packages if you need a license to be installed which is finer grained but a bit more difficult to setup for some licenses you will also have to provide sources and open embedded provides the archiver class to generate tables of the source code that is compiled so you will want to use it with inherit plus equal archiver so you put that in your local.com for example and then you want to setup the archiver mode variable so the default is to provide the patch source which may not be enough and if you want to provide the configured source you can use the archiver mode SRC equal configured and then it will provide a table of the configured sources depending on how your lawyer is interpreting some licenses you may decide that providing the compilation scripts is also necessary and you also have a mode to do that so have a look at the documentation for archiver mode to know how to do that c'est tout je suis heureux d'avoir des suggestions et si vous avez d'autres meilleures pratiques ou si vous vous inquiétez qu'est-ce que vous faites c'est bien je vous demande merci pour l'attention