 So, welcome everyone to my talk about doing fetching, configuring, building your Yocto project with just one command. Well, I'm also cheating a little bit, we will see about it. But first of all, who am I here? So I'm from Siemens, Siemens Technology specifically. We are a smaller group or larger, depending on how you look at it, of Linux experts called our SIF, the Linux Competence Center, or shorter these days, Linux Expert Center. We are now consultants to Siemens units, enablers of many Linux, not all of them by far, but many Linux products. So possibly have ridden one of our products by taking a train. We are gateway to upstream communities, not exclusively by far not as well, but that's also one of our roles. And we are also maintainers of several open source projects on behalf of Siemens and beyond. For myself, I'm doing it for 16 years now at Siemens, trying to navigate the ship these days more than doing it. But still, I'm also a maintainer of too many open source projects, well, you will see. So yeah, that's for the background. So before starting with the details, so first of some quick survey here, who of you has heard about CAS before actually reading my abstract? It's a praising. Who of you has actually used it before? Still quite a lot. Who of using it with a CAS container? Thanks for the back. Anybody actually using it for Debbie and Isabel's? That's not so many. Okay, so a little bit of a summary of what Alex was also talking about. And keep in mind, this is starting six years from now in the back, well, where we started to work on these problems. So how did you set up at that time specifically a Yocto project? Well, first step, fetch some of the repos manually according to readme, not that uncommon, with the help of a custom script, not preferable using the repo tool was mentioned before. There's some other tools as well. Next step, creating the configurations, pblayer.conf, local.conf, well, usual patterns. Using the right template is also one, obviously, which is preferable, but it wasn't done on all cases. Then run the build environment, which part of include the former step if you use the template? Yes. And then choosing the right target to build. So far, so good. You were happy. Well, you forgot two steps. Pray that you didn't miss anything about it to find out that you did, because in some cases actually the host environment makes a difference. And while your local user configuration can also make a difference, despite all the effort that Yocto is taking and back then not so much these days, even more to prevent that this actually makes a difference, it did in the past. So that brings basically what to the question now, what does cast now mean? Is there any native Bavarian in the room? I saw someone, I heard someone. So cast means in Bavarian, cheese, yes. What else? Nonsense. And that was the impression that my colleague had at the sense when we're looking at this, this is nonsense. So out of that impression was part of this was influencing the name of this tool. Think of cheese, but who invented it? The Swiss word again. The first commit actually came from Daniel Wagner at that time at Siemens, working on his first Yocto project and trying to solve this problem in a generic way or more generic way it was done back then. We went pretty early public with that in June 2017 and I remember him discussing this problem with Richard Purdy back then here in Prague at the EOC in 2017 and trying to pitch a little bit that something should be done there and I'm happy to see that something is now done, but anyway the project evolved more or less under the radar of most people I guess. We got actually a first contribution in 2018, in 2019 we thought this is now done, it's 1.0. We also headed over the project management, Daniel went for enterprise, I'm still doing embedded. So I took over that what he was starting and in some point and I think the guy is in the back as well, someone talked about CAS as a nice tool maybe useful and we suddenly got a lot of contributions from people and first one to mention Paul Barker who was actually contributing significant parts from Yocto people on that tool and since then it was really taking off. There are many other contributors I think in the room here. So where we are today we have a tool which is now version 4.0. The versioning has come out locked with the built environment we are using but anyway this is where we came from where we are. Now what is CAS again? So CAS is really about that single command to build up a build to get to the image and that's not only but also targeting Yocto users who are used to this but it's also targeting users who are more interested in the image than in the process. So you have that command you run it and you get your image or even if it's not delight by everyone you take it in the container and you don't even have to worry about all the dependencies you need first of all to run the native command or these days we'll talk about more you run into a menu and you choose basically what kind of combination you want to have from this reference layer maybe. So one command for them all. Okay you have to clone the bootstrap repo first of all to be fair so it's not really one command but that command easily migrates as you can imagine also in a CI environment and that's one of the goals as well of that. So now how do you get to this configuration file that we are referencing here and how does it look like? So it's a journal file that we use we also support JSON if you like to have more brackets or curly braces in your text file but anyway for the readability I took here the Yumble version of it that's so to say a self-contained example to build a curable image from Pokey so you have your header version you have here so to say pulled out out of the local corn section which comes below the distro the machine and the target you want to build with this specific file then comes the most important part all the repositories here both regarding where they are coming from but also what layers they are carrying for your build here so you can configure them both in one place and then comes the section optional in some cases recommended usually in other cases pieces you want to add to your local conf header because we are also generating some bits of the local conf which can be derived from the setups and you could also do the same but this is not needed in most cases to add also pieces to your bb-layer conf this information here is enough to generate the bb-layer conf out of this thing so then you are done and this is actually executable and you get a QEMO image out of this how do you interact with bitpick yes sorry we are wrapping so as I said normally you specify with one file what you want to build and that's done if you have variations of it you can still overwrite it and say I want to build another target so you kind of modify on the fly your configuration file you can also run specific tasks of these targets also possible you can also pass forward commands like give me the environment of the whole thing to bitpick directly or if you really don't like all the wrapping you just say cast shell and you are in that environment where you can natively interact with bitpick or other tools of the yachter ecosystem on that shell level so these are the kind of wrapping you have around that another feature which is I think kind of unique here is that we can already combine certain configuration file include each other that allows to build more complex configuration without repeating yourself obviously so here's the top level configuration file on the left bottom left top and you have some base configuration that you want to reuse possibly across certain files so you state here please include that file that file can be in the same repository as the main file it can also be in other repositories as you were just fetching them it can become quite complicated but in most cases it works what happens then if cast pulls the top level file it kinds of merges these two together so you get a virtual not not directly visible configuration file in the back where you pull in certain say the things which are only specified on one of the files and where you can override other things like in this case the machine configuration you can override what has been decided in the base file by the top level file this kind of merging and on overriding via the include files happens from top to bottom regarding the include ordering in your in your top level files and then depth first regarding who sets the first value and then you go up and up and up who sets the final word so to say on this overrides again you can do obviously here complex things I wouldn't recommend to go become totally complex but depending on your use case that can be quite useful to avoid repeating yourself in this kind of configurations now another scenario that we quickly evolved because the initial idea was okay one file to rule them all then you found out okay maybe another file to rule them in the debug build and then came to another board and it comes another configuration for the number of configuration file group with the base idea of having just one file so the next logical step for us was okay we have some variation here some combinatorie so let's make a configuration fragments so have multiple files the includes to such a thing is contributing that already so let's say you have a main configuration file for your distro and your main repositories and then you have a selection file for your different machines and maybe some configuration fragments or configuration fragment files for features you can roll them out and what you can then do also on demand is combine them on the fly on the command line of cars by just colon appending them onto that so again what happened in the background is that cars is building some kind of include it virtually a rewriting the main configuration file and adding these includes of the fragments that you were appending on the command line and that allows you to to provide to your users here a combinatorie that you don't want to roll out and and write to individual files in your bootstrap layer possibly quite powerful but obviously also with some limitations because the command line doesn't restrict you and what you combine you can combine also impossible combinations maybe one machine is not really supporting a certain feature or is not really prepared to these kind of configuration settings and that led to the question also in the combination the question is also how to the explore what kind of pieces are there you can define conventions where files are located okay there's a folder which is called machine probably there are machine fragments and there's a folder which calls feature or options but in the end your user has to should be given a certain information about what is reasonable combinatorie available with these kind of configuration and that basically led to the idea of having an interactive menu for that so just like you know well build would have something similar the kernel office has something similar where you can just go there and and explore so to say interactively what kind of combinatorie is available but also express that's why we chose here in this case the k-config language what dependencies exist in these things which are kind of choices which are kind of features and and all these stuff and you can even enter here some values if you like to in the manual so again we chose an existing language and this was all pros and cons so the pros definitely is we didn't have to invent an own language which usually goes wrong in the first attempt we could also even leverage in existing this case Python library to implement the language and it actually proved to resolve quite a few problems in our deployment scenario specifically if you have end users on the end other end which you first of all has to explain what all of options exist there nothing is for free it also comes with some cons you will see it in the next slide on the example side that if you actually express that combinatorie in the conflict file and as we didn't change the language the k-config language you have to map it somehow on the cast language the cast configuration that mapping is not really natural it works but it's well bit cumbersome in some cases anyway and also what always comes but this is actually also true for the previous slide don't misuse this you can now build weird kind of combination you can add a lot of numbers and you will maybe end up in something like build route in the sense you can configure everything that's not maintainable obviously this is really thought to be for some high level choices I have ABC boards I have CDE features let's make them combinable maybe I can choose a certain version or like now michael door master or you name it of my this of my build system these kind of variations really what it's about so how does it look like from a technical perspective the conflict k-config file well has also a header section where you define the menu the title of that then you have the main file you want to include then you can build choices here for example to select the machine I've highlighted here some special syntax the cast include conflict snippets those are basically which are magically interpreted by cars as being okay this is a string conflict variable it's referencing a cast configuration file so please make out of this something in the end which can be interpreted as cars as a configuration combination so you basically represent what you have in the cast configuration yaml files you map it in something which is here interactively selectable and also you can express here as you see on the right regarding dependencies that the feature be actually depends on not being used on the qa max 36 machine also as I said you can also specify here variables which takes input values if this is needed matter root password by default measure if this is a good idea but anyway you could do that you can pass in here also a variable numbers so that's what what cast happens now if you if you run that menu command and after asking or answering all the questions maybe someone's step back keep that menu here in mind you can save that or you can save directly build the whole thing so again the user doesn't have to worry about what's the next step if you save the thing what happened here is a small cast file is generated the dot config not h or something like it but dot config yaml file which can actually just be passed to cars for building so it's technically compatible it contains a little bit more than this one and you can also use it just like you're used to it from the kernel possibly to go back and reconfigure your configuration or to store it if you like to but that's what's happened in the background with that so how do you now manage your repositories your workflow regarding updating repositories and that was something well we discussed also for the time we had our own workflows dealing usually with much less layers than you have in a in a larger yachter project but anyway there's one way obviously do this manually yeah go through your cast configuration files edit there the shards and bump things around doable doesn't scale we know we have now a different workflow available as well so you can basically select branches in your repository configuration and say i please check out the latest version of micldor for example make it run and if i'm happy with the configuration i can lock it down so i can generate a log file which then again is interpreted on the next build as being actually the ruling file defining the revision that you are actually checking out or that you're actually sharing with others when you are exporting the whole result so that's one way to maintain basically floating testing environment while then locking it down for production or for handover another thing that's possible from the workflow is it's also quite unique that you can patch your layer and we had a lot of discussion actually i was initially against this feature as well because patching your layer means that you're doing something wrong so either you are not doing bbapend or something like this wrong from the philosophy or you're not doing upstream first also wrong for the philosophy so if you really have to fix something infrastructure however it turned out to be that this is helpful even if you are playing according to the rules it helps to be upstreamed to test to be upstream patches we are doing this heavily so the rule usually is you can patch a layer in this cast configuration file but that should be gone the other week or the other months at least if there are some discussion but it allows you basically to feed in your upstreamed or to be upstreamed patch in your ci in your workflows and already deal with it locally while it's being regulated also upstream so if you want to patch i don't know if you want to patch bit bake or if you really want to patch something in infrastructure classes or stuff like this this is normally not doable with bbapend and co you can actually test it as if it were or upstream already without creating a fork repository which obviously can also do for your for the upstream poke repository and things like this but this is really for my perspective helpful to be proven helpful if you don't abuse it we had one aspect recently and this was almost the point where i got my first cve or where i was about to file my first cve as a project maintainer so there was a problem in cast and this is really cast problem historically we had just one revspec which was able to be used both on specifying a commit id and both specifying a branch a floating thing or attack and that allowed attackers theoretically if you if you if they are in control now of your mirror repository for example to sneak something in in your build flow it turned out after discussion with the reporter that there's more behind it well it boils down in the end right now in our ecosystem that our source code management tools use char one and if this is a coalition as possible at some point you are burned in any cases still we improved on that so yeah the first attempt was or to fix this maybe sustainably by adding proper check summing to the repositories you are checking out so basically like bit bake is also allowing you to say to take a fetch from the bit repository or take a snapshot of a tar ball and then char some the tar ball something like this could be added to cast and i played around with naive approaches with less naive and then with own approaches in the end we discussed it in the community the problem is it's not really it's not really a cast problem it's rather an ecosystem problem of the scm so maybe they find a better and smarter and more sustainable solution than cast and in the end also is it already realistic to have a shark collision on your git or hg so we all support bit curial repository today debatable the current is a statement yeah you have to trust your upstream repositories in this regard make sure that you're not pulling from random repositories anymore but still this is what we've discussed and so the current situation is we still resolve the initial problem of the you and not clearly identified rev spec we have now changed a little bit the syntax that you have a separate commit and the branch key to specify okay this is really just a commit and nothing else not a tag not a branch and nothing else and that is a branch where you want to write explicitly and that allows now to avoid this kind of original problem we had above without solving the issue of the shark one but that's again i don't think really a cast topic in the end so now coming for the containers that's also a feature which we added quite early also in the light of ci environments in the light of that basically everyone our team had a different distro running on their machines how to make the environment the built environment portable and also archivable so casc provides along its releases since the very very early days also corresponding containers nowadays hosted on github one is for the open embedded dependencies resolved build dependencies resolved using a debian input feed the other one is for the debian either built same story and then they have so to say the common dominator of both is a number of packages you need in order to do the connectivity for cloning stuff and doing downloads of artifacts also interacting for for example the aws bitbuckers and stuff like this that you can fetch from there also artifacts so this is pre-installed comes ready to use we internally standardised on that and we also recommend it because it makes the story easier for users to get started without discussing which ubuntu version they have to install first that comes again with a script to replicate so to say casc inside the container to the outer world so it's a wrapper script sorry again which calls into the container and runs casc there with all these build dependencies resolved decoupling you from the host environment without really having a different kind of user interface than you would have with casc on the on the native environment that also allows you to lock down your built environment along with your bootstrap layer so we now have established the pattern that you can just take the script and check it in into your bootstrap layer that script contains the version of the build container that was using so you basically documented this way in form of code what version of the built environment your layers or your layer setup was tested and built against it works with docker obviously for the past but nowadays also nicely with sportman for the oa build it means also unprivileged obviously so it's it's nice and we also have an arm64 container if you happen to have an arm64 host and don't want to cross build so that also brings the point of the casc sandbox concept so we have a stronger decoupling from the host than yorktoe in an open-bedded body fault has at least had so when casc is run it creates a separate home directory throwing everything the way basically what you have in your own configuration and that even without using containers and you can only pass in so to say parameters variables from the environment that are explicitly permitted so allow list based basically that gives a higher level of control or what's actually supposed to come in independent of what is happening with the version that you're using there it can be weakened though specifically for development environment at least our inns at least without the container scenario you can weaken these kind of rules by saying preserve the environment my user environment however if you want to export this to a ci environment you have to think about actually what you're just pulled in implicitly or explicitly from a beginner perspective i often heard what is this thing doing here i'm confused so it's possibly one of the most dislike beginners features but we are kind of stuck to that with good reasons so again we came from a time where this was really very needed and helpful some people told me recently oh yeah there's something not enough of cases where this would be really useful and is really useful again we also want to foster the migration between local and ci environment by giving this kind of yeah guidance around it and last but not least we really try to keep the cars container usage as smooth as possible also for the compared to the native usage and there you need to know what you have to forward from your host into the container it doesn't have magically as it was normally with a native build now i don't want to repeat what alex nicely explained regarding what is the bit bake layer approach by now again we've saw it already there is now the file to or the files being traded in to have the the repository setup repreciated and there's a way now to generate also the the templates which gives you quite a bit of pieces so pros of this approach clearly is this is how yokto is lived and it obviously is also kind of consistent with the yokto community in the sense of how you should use it while while we are not in the part of the yokto community with this thing so we may also get something wrong there's a risk although we have yokto contributors yeah this is definitely the official way no no doubt about it still it has still some limitations some of them might be temporal others might not be desirable to solve from our perspective as we learned there is an idea now not yet totally flashed out regarding how to make this course-grained interactive configuration maybe possible one day maybe we can help to inspire this as well or to rule out how to not do it i don't know the cross layer reuse of the configuration file not subsistence scope already or if it's considered to be useful because it is possible layer patching no no but the official opinion on that is it's coming with cars the older versions of cars and the older versions of yokto and an isa they are not addressed this is basically agnostic to it but this is just a matter of time i would say even for isa it's solvable when we do the next bump of our internal tooling what's not in scope not sure what the official opinion this is the hosty coupling so the container thingy if you like it you may want to look at this if you don't need it and you don't like it well then you are fine with what yokto is doing natively on that so with that let me quickly summarize so cast makes your big bake builds easier portable from our opinion single line bootstrap scenario stronger decoupling from host stable build containers which resolve the dependencies kind of automatically and you have this interactive configuration which is quite useful i guess for also for many of the sdk board support vendors combining certain things and making a broader set of configuration available to their users and add some discussion about this as well as well already so what's next for cars um we had quite a couple of times the discussion could you just add this or that to your containers to make also the test working including a complete curio environment um so currently i'm pushing back on this still because we want to have build containers not test containers but maybe it's time to rethink it and actually fork off or also provide test containers i'm open to the discussion about that um and as now the the native tooling is evolving it's obviously time to observe this and maybe discuss also where we should actually step back as cars and say okay this is no longer our business there's an alternative to this maybe make some bridges or whatever to to make use of both of them while focusing on the steps that the native environment the yokto environment may not be willing to address for the upcoming future and in the end this is something it's your call specifically as there are way more yokto people here in the room than there are debian folks so tell us what you are interested in what you like to see what you do not like to see where you see basically chances of improvements in both directions also regarding in the interoperation just for the reference there's the project well there is the documentation url beyond mating list and there are also some example layers here reference an external one i think ross is somewhere in the room for metaharm there are two layers from the isa world but conceptually as i said this is very similar to what yokto is doing yeah with that i'm ending my talk thank you for your attention i'm happy to take questions so not really a question but just a information so i gave a talk about the crops generator which is a new tool to generate crops containers so those are build containers for your your yokto environment so we will help to solve the host os problem by letting you build a known host environment into a container and that will support repo get some modules costs and the new oe core tooling so it's it's still a little bit young we also chose to use cake and fig for exactly the same reasons so just want to let you all know about that thanks a lot for the overview one question that came to my mind you pointed out at the beginning that cast um yeah hides away the yokto stuff from the users so users that are not interested don't have to deal with it when you're done building the users are there and say where's my image so it's stuck somewhere in the yokto build tree and also sometimes you have some custom post processing you need to do did you ever think of adding some post build steps some post build hooks that you can add as a user thinking about already but it becomes quickly quite complicated so what is the common dominate you can abstract you can provide in form of okay now you have to run the following command to flash your image or i don't know to to take the artifact and put it somewhere where it's useful i'm open for suggestion right now we often still have for that step also read me now you can go to the following folder pick the following artifact and go for your usb stick and get it to the device for example maybe there is some room for improvements it wasn't just the major point point yet even though even the cases where we are facing end users so to say but yeah i'm open for that kind of suggestion if you have a good idea come forward and we could discuss a design about that okay thanks good hi you were really talking about the layer setup but i think you said that is also this is all done by big big layers did i understand that correctly um that was basically the reference to how big big layers working if you choose the other path the non-cast this is the non-cast path that's for reference it's not we didn't integrate big big layers into cars at this stage so generating a cars config like alex is doing with the big big layers so the setup layers config that's currently not what we do with cars could be done everything is done so by the way i didn't mention cars has a plugin concept as well you can write plugins to this i think even local ones if you like to wouldn't recommend it but anyway it's technically possible you can do anything on top of this if it's reasonable is a different question so currently really the layer setup is something which is more or less manually done in most of our cases you don't have to do it all the time because of the tooling to later on maintain them but yeah that's could be a theoretical problem if it's solving enough people's pain it it's it's audible but right now we didn't have anyone who stepped forward and said yeah that would be really make it even more useful so yeah and the layers that are in the cars yaml config file that are just to generate the bb layers exactly exactly so in most cases you can just get it from there and and it's even easier if you have a single layer repo you don't have to specify anything it's assumed to be the only the way it's around if you have a bit big repository you actually have to say this is not a layer by the way excluded doable so this is the default assumption is that this is a layer or is this has it has multiple layers included yeah okay thanks i've seen that cast container can be used for example for gitlo ci directly for the immobile i've seen that it is it is used by bent arm for example but they have a quite complex configuration because they build a lot of targets so i'm asking you if you have a more simpler example or if there's some documentation directly inside cast to use continuous integration with gitlo ci so how to use it in in ci environments yeah so there are there are two patterns actually you can use and i think you find both of them here in the oops in the referenced isa layer sorry um so the the isa ci pico we have a dot gitlab ci yaml file there we reference the container we want to run the scripts in and they then the so to say use cast natively inside the ci script that's the one case um the meta id 2050 our product layer is running on a github a github actions there you get a vm and you can run the cast container script natively um so we have to so to say you you have to cast container command line in your ci script so both is possible it's transparent so to say it heavily depends on rather what your environment is able to handle um also depending on the york too it's easy you have the free choice it's usually unprivileged built with isa we have a more complex things because we are taking debian bootstrapping things you need a certain privilege environment and there you need a runner which is compatible with these environment if you want to use the native approach um but it's doable obviously if you're in a vm then you can do every everything so both is possible and it's really like okay this is the command the user is also running take it put it in the in the ci environment you have the same thing this is where we started actually okay thanks that was another question is it possible to insert credentials for artifactory or api tokens into yeah so obviously you shouldn't you shouldn't check them in um but you the normal pattern is to pass them in the environment or you specify also if you take for example as this age keys you can specify a folder which contains the the employment keys for your built and pass them in so for ci usually these be your preferred approaches to have them environment variables ci environment variables protected ones and those sneak and then into the build and you can do the same on the local environment to have it on par that's normally the pattern we use here and you have to configure them or whitelist the environment as they come off go for the the documentation here about the environment there a couple of them are predefined so the pre-wired listed so to say that's for the common cases for ssh keys for example or ssh demons even and others you would have to pass an explicitly by saying this environment variable I'm taking here and I'm handling in ci as well as locally but quite a few cases are already covered and if you find something which is generic but that isn't covered yet please come forward we can discuss this we just had it with some more details but most case actually are covered by now okay and entering then into the config file or local con would not work it would work but it would create the danger that you check in the credential that's the whole point so if you have it in your cast configuration hard-coded you will get attention to get it from the environment and then make casts inserted into the local con file that's generated technically possible if this is needed depends really on I don't think you need this for for bitpec to run properly and have the fetching working downstream there that you've entered there can you also handle site.conf somehow we can't actually this has been discussed a couple of times so honestly I don't have a good design pattern for that yet again this is a call if you have solid cases for that we can discuss how to model it properly right now it's not generated it is not consumed it is not handled this way we handled the site specific configuration basically environment variables in most deployments so yeah that's the situation thank you that's time thank you Jan thanks a lot thank you