 All right, good morning everyone. Thanks for joining my sushi class. No, today we'll talk about speeding up BitBake. So I'm, hello, yep, I'm Jan Simon Möller. I'm the release manager for Automotive Grid Linux and I'm also representing AGL on the Yocto project board. If you have any follow-up questions, feel free to reach out to me. The slides are uploaded on SCAT. So if you want to see the presentation later on, please go to SCAT and you can grab it from there. So today I want to talk a little bit about the BitBake build process. So essentially that is what AGL is using to produce its images that we show in the booth, that you can download at home, that you can build yourself at home. It's using Open Embedded and the Yocto project and BitBake is the engine that drives that. I'll show some ways to speed it up locally. I'll show ways how to speed up in a network environment so that would be for your work group, for your team, for your company, essentially. And then there are additional ways that you need then in this case, you need a PR surf, you need for additional speed, hash surf is then a solution. That works also for just your computer but it works best if the more people it essentially use it. And in the end we have time for Q and A. So let's first take a look at how the BitBake build process works to set the stage and to see what is going on and where we can speed up things. So that's essentially the input and output from Yocto project from BitBake. Essentially we take everything on the left side and top. We take all the sources, we take all the configuration, the recipes, the BSPs on the left side here. And this is our input, right? And then we crank the wheel, we let the engine do its work and then in the end we have an image or an SDK. So essentially the engine will start with fetching the sources. It will then go on, patch, it will compile, it will package up, run some checks, run some QA package up in the selected format. In our case, let's say RPM. And from there we will create an image. So that's the high level view. How can we speed this up? Essentially, well, we can speed up the downloading. We can speed up at the binary side after compilation. We can speed up by, well, not building if it's not strictly necessary. Open embedded Yocto is a system that is designed to build from source. So that's the feature. But we can try to only build the necessary. The tricky part is here when and what is possible, especially if we are talking about the compiled the binary version of it. It is really easy for some host dependencies to creep into the binary that is produced. Even there have been great efforts and this is an upstream project called reproduciblebuilds.org, right? So there is a project that focuses on this very topic, how to produce binaries that you can reproduce and it's the exact same thing at the binary level. So this is not an AGL, this is not a Yocto problem, this is a general problem, right? Just think of it, if there's a single file that is, for example, generated during the build process and it might have a date inside or a folder name, it's not reproducible, right? Because the next build will have a different date or a different folder. So that's a generic problem. Yocto has great support for reproducible builds and that's what we leverage and that's what helps and enables this. Let's take a look. So we have local caches. One local cache that we have is the download directory. Essentially here, we download the sources that we want to build and this can be cached. So any build that will be done later on can use those artifacts, tar balls and so on. There is one thing we need to do, especially if we use sources from JIT. Those are then taught up and there is a feature that you can enable to produce tar balls. Okay, so simple, local on your own machine or on your build machine, you can set download here to point to always the same folder in your builds. That will pick up the downloaded artifacts and you can speed up and skip the whole fetch phase, essentially. Okay, number one, that affects, as I said, the source fetching. Number two, we look at the binary side. So we have essentially, once we have the software compiled and installed into a staging directory that is one state that we could preserve, we have another state, the analyze, the Q8 version, the split package, the split means here, we could have the binary, we could have the debug information separate. So there are different post-processing steps that happen. And in the end, we could have essentially the generated RPM packages. This is, those are binary artifacts that we can cache. This ends up in a state here and for your local build, you could reuse that directory across different builds. The inputs, so download here and state here. There's a nice way how to handle this. On your local machine, you can define a file site.conf. So I have a site.conf, which is maybe a little different on each machine that I use, but it's for this machine here or for my build machine over there, they all have their own site.conf. If I set up a project to build, I will link that file into the conf folder of that project. So conf site.conf and I just use my template site.conf file. And this stores my desired download here, the desired estate here and so on. Now, the interesting thing about those binary artifacts here is we basically can reference them in the next run for the next project through a hash. So if the hash, basically a hash over the inputs, which is the sources, which is the configuration, if that matches, we can reference the binary artifacts we don't have to build. And that saves time and speeds up the process. So the inputs essentially determine a desired hash. And if that is found in state here, fine, we can short circuit and skip right to the generated RPM and go on. So this is important, this can be reused. Now, this is local on your single machine. Here's an example and I put that, as I said, in a site.conf and throw that always into my project environment. These folders could also be on NFS. This works. The Octa project uses that on their build infrastructure. So that is also possible. Now let's go one step further. If it's not just you, if it's multiple people, a team, right? Then you end up with, well, NFS would be one solution or you end up with networked caches. Okay, again, we have the source side of things. For that, we would use generate mirror tables. So the fetcher would then go ahead and for example, git checkouts would be tarred up for later consumption. And that basically means the host that runs with this produces a download directory that we could throw on a web server and that server, we can put into the variable pre mirrors, pre mirrors. So that's again affecting the fetching. And this is a live example. We use that URL as mirror and that's an easy way. You just specify source mirror URL to the top level folder of your mirror and then there's a class called own mirrors which will do all the work to define the pre mirrors variable. It's essentially a match. This is the first part of the source URL and basically that's what we inject. Note there are actually two mirror variables. One is called pre mirrors, that's what we used. And one is called mirror. So what's the difference? Pre mirrors, we will query in the fetcher first. So in a Yachter recipe we define where the sources come from in a variable source URI. And pre mirrors will be queried first. So if you have a pre mirror, a web server loaded with the content of the download directory in your local network, it's faster than fetching the original, yeah? So that saves you time and well, essentially internet bandwidth as well. So this is then in your local environment. If it's not in the pre mirror, we will fetch the pristine source URI. If for whatever reason it's no longer on the internets, there's the fall back and with mirrors. Okay. The Yachter project for example hosts such a mirror. And if whatever tab all vanishes, we still have to back up here. So even old builds will continue to work. Both have their use. Note the method over essentially sharing the download directory and enabling BB generate mirror tables is one method to produce a mirror folder. If you need to use the archiver anyway to have a nice folder of your sources, you can configure the archiver to actually look like the desired mirror, right? And then you don't have two steps. You just have one step, the archiver that you, well, run anyway and that archive is then a mirror or pre mirror right away. Okay. Whatever of the two methods you use, they produce the same result. It's basically one folder with a lot of tar balls in it. Now, a good way to try if your setup works is to use the flag BBNOR network. BBNOR network will forbid any network access during the build process. Okay. So it would prevent anything from being fetched. But we could allow the pre mirror. So if this works, you know for sure, A, the build works even without internet and the second point is your mirror is complete. You have everything that you need. So this is actually a good test. Let's leave the sources and go over to the binary artifacts again. Similar to the estate directory. We can take that, put it on a web server. Then there's a variable estate mirrors that we can configure. And Yachto, the big bake will then query if the binary artifacts, remember, they are identified by a hash. So we can calculate those hashes without building. And when bitback runs, it will go through the recipes. It will go through all the settings of our config. Then basically calculate the hashes for each package and say, okay, I need hash X essentially, right? Is it on an estate mirror? So that can be queried. If we have that result already, and in brackets, giving you in your connection to that survey is faster than building locally, we save time because we can query the artifact right away. So we can basically short-circuit and go straight to the result. So that's the estate mirror. And here's an example. So this is the setup if you use a server. Note the marker path that's required. And this is the setup if your estate mirror would be on an NFS. Normally NFS would be read wide. And this would make your NFS share read only. So you could have, for example, you could have a builder with the NFS mounted read wide and everyone else uses it as a mirror to not write into that. Also one option. The other setup is then to have a web server and host that. Again, the closer you are to the web server, the better the connection to that web server after the whole process. So how do you fill your web server? Well, either you have a nightly build, you could clone an upstream web repo. Yeah, there are different ways to do that. Mostly most projects fill it with a nightly build or release build in a different folder. Similar to the allow premirror only, there's an estate mirror allow network. So that's the equivalent. So you could build completely with the internet shut off but still access the premirror. Then the estate mirror in the same way allow access to that. So again, the artifacts will be pulled down then from the web server and they will be stored in the defined estate here. But only if the desired hashes are existing on the server. If not, we have no other trends and must basically mark this task at build locally. So we basically will query the server and say, yes, we have it, yes, yes, yes, yes, no mark for local build and proceed. So watch out, the hashes should be free of host dependencies. Yachto tries hard to make that as host independent as possible, but sometimes it's really hard. And if a host dependencies creeps in, you basically end up with a different hash. Your machine is different than my machine and we don't end up with the same hashes for the binaries. So key is here to have similar build environments that helps be it the containerized system, for example, that can help or in your company group, you have Ubuntu set up whatever, so pick your poison. The more close the systems are, the better the result in the end. To, there is also a tool in Yachto which is called the build tools table, which can help with that. So it's not a container, it basically brings in a couple of extra binaries that lets you smooth over that. That's also an option. To help here. A few lessons learned. So what to avoid? So in early releases of AGL, we produced applications and we basically had them always point at the very latest tip of the tree with the autoref here. So we did not lock the apps down. We ran our QA and said, okay, latest, greatest is fine. Autoref means that the build process always has to query and find the latest revisions. So there is no easy way to short circuit. We'll always go and find what's the revision of the day. So autoref has been replaced with static source refs in the recipes now and that improved build speeds because we can more easily parse and reuse the hashes. Yeah, host tools, so host tools is a facility to expose host binaries into the build system. So if you have some custom binary that's needed during the build process, this is problematic. Similar, this goes into the reproducible build allay, if in your software project, you generate files at runtime, inspect them, take a look if there is a date or a folder or created with by, right, created with this file, this generator and oh, there's a full path or oh, there's a date and that can basically hurt you because your binary output might be different every time. Okay, so that basically showed you the ways how to speed up the source side, speed up the binary side, both local and in a networked environment. There's another aspect and that is called PRServe. PRServe is mainly needed if you target package manager. So if you use RPM for updates, that's the main case because any package manager depends on an always increasing version number essentially, that's the key stop. Now if I build on my own, right, my first build would be R zero, that's what this PR defaults to. So if I build on my own on my little machine, it's revision zero. If you build on your machine, it's also revision zero, right? So this is fine if it's only on your machine but once you are in the position that you need package updates that you need basically a binary feed for basically you could also use a binary feed with an imaging tool to produce your images. You could basically short circuit that and say I have a binary feed and an image generator. Then you need those always increasing numbers. For that we have a so-called PRServe which just provides the records so we can do an always increasing revision. So if I do rebuilds, it would just be that simple but if multiple people, parties, well if your release or CI engine is involved, pick your poison, you basically want that each different developer release whatever branch build does produce a consistent and ever increasing number. And for that to keep that consistent that's for what we set up in PRServe. The PRServe, the server side is quite simple. You just download Big Bake and find in one of the subfolders in the bin subfolder there's Big Bake PRServe. You provide the IP address where it's hosted. You provide a port number and start it. AGL sponsored in addition so we can expose the same in a read-only manner to the outside. So essentially that's the use case. You have someone, tier one, tier two that builds against a released platform and he would rebuild one of the existing packages. He would be number X plus one. So that would ensure that. And that could be the read-only port. This would be started in the same folder. Note that, just different port numbers because it uses the same SQLite database. So same server, just different ports and one more flag. On the client side it's easy. You just define IP address port in the variable PRServe host. And Big Bake will then query that machine. There has been a relatively new addition and that's like half a year, nine month sort of. And that's the hash equivalence server. What does it do? So again, we build from source. What happens if a package gets updated, right? Well, we have kind of a tree structure. So let's say this one is updated, right? What happens? We fetch, we build, we get a different set of outputs, essentially, right? So all the subsequent, everything that depends on this here needs to rebuild as well, right? Yeah, everything that, and this basically triggers more and more and more and more. Okay, now the question is what if this change was purely documentation or was just a comment or a typo in a non-code section. Essentially, let's just say we had a fix in documentation and this doesn't end up on the target. What if this binary is the same as the built before? We can meanwhile detect that. Again, by hashing the output, the built output. So they produce hashes on the output. And if we know, if this build here generates the same hash or a known hash, then we can say, wait a minute, we have the same outcome. So we do not need to start the avalanche to rebuild subsequent packages because the binary output is known is the same. And this is what the hash serve records. So essentially with the hash serve, we do rebuild this piece here, this lib y, but we detect the outcome is known. So the hash serve, we just throw hashes at him and it says, whatever, package name hash and it says, oh, wait a minute, I know this hash and this comes out of this chain. So we know this chain and we can then short circuit here and say, okay, we do not need to rebuild this, okay? So this saves a lot because it short cuts the whole downstream builds. Again, this is relatively new. It works best if you use, well, reproducible builds. If there's a date in here, we end up with the rebuilds. So this needs to be paired with the reproducible builds, which are meanwhile default in the Octo project and BitBake. So the latest master, there is no specific reproducible build class that you have to enable it is built in. And that server has a similar setup. We have the hash serve and we have to read only version. Again, similar setup, same folder, your trust start, both up on the same folder, select two different port numbers and then your users can either it's the night rebuilder, it gets the read white port or downstream users get the read only port. That is useful if you share an estate mirror because then your users, here, over here, your users can query this, this and this and this because it has already been built on your estate mirror. There is also the possibility to run this on your local machine and specify an upstream. So you have kind of a local clone of this server which makes the interactions faster because you are then in the local network or on your local machine. So this is meanwhile also if you go to the latest Octo, it will start up and hash it with by default just locally. But as you see, the more hashes we know here and that's a crucial point, the more equivalency we have recorded, the more hits we will get, right? Okay, so the default setup is actually this here. This is enabled by default, we need to enable the hash-equip handling, we need to enable the hash serve. That's the default, if you fire up BitBakeMaster, you will end up with this. You can also tap into an upstream. So this is basically this setup automated in this way in your configuration. Or if you have a local server with fast connection any way you could tap directly into that. So the difference is replace auto with host and port to have a direct connection. Now watch out, if there is no server on your local network and you go out to whatever, the internet, the global Octo instance, the AGL instance, we expose the read-only port as well. The initial clone will take time, right? We basically need to pull down the information, what is equal and that will take time. You see it, that BitWare will basically stick at 40% while it queries the data. You can avoid that if you have a local server with upstream in your network. So that helps. A little, well, room for improvement. So it occurred to me that you can add multiple estate mirrors. So you could have multiple binary artifacts here and there for three, four, five, if that's useful is a different story, but you can have multiple estate mirrors, but only one hashed with server. So if you have kind of multiple binary sources, fine, but you have only one hashed with server. Okay, are there questions? Okay, yeah, so we had, as I mentioned, there was the auto read, so we did fetch and build our apps and the query basically brought the build times up by at least 20 minutes because the query went out and for whatever reason it tickled the build, we needed to install all the requirements and so on and so on. So just changing from auto read to static hash and then we had a lot of changes brought the build times down by at least 20 minutes, if not half an hour. Yeah, so that's why avoid auto read if you can. There are one or two things that you can do on top, but they are dependent on your setup. There are classes that allow you to enable, for example, C-cache, have multiple hosts contribute to the build. There is a class for ice cream, which is a fork of C-cache. So depending on your setup, you could add these in your local networked environment to speed up an individual machine by letting multiple machines contribute to that build. Note, I know that from the ICC class it will change your hashes, yeah? So then it will kind of enforce a rebuild. If you wanna use it, inherit the class but turn usage off. That is one option how to use the class. Just inherit the class by default for everyone and just flip it to off in your configuration and someone can flip it on if he wants to use it. All right, any online questions? Okay, then thank you and enjoy the rest of the conference. You can find us at the booth and we can chit chat some more. Okay, thank you.