 Next talk we have is from Grace about the AOSP build system. What do you Grace? Okay, thank you. So yeah, welcome to this talk. I'm going to try and demystify the AOSP build system. Don't guarantee that I'll succeed, but I'm going to try. Let's skip that. So I'm going to start off with a kind of high level view of the AOSP build system for those people who may not be totally familiar with it. And then I want to break it down into the various components. I want to talk about Soong, Carti, Ninja, and then I'll finish off with Basil. So let's see how this goes. So Android is a large chunk of code. I counted, well, I used a C lock to count the lines of code, 50 million lines of code, mostly C++, big chunk of Java, bit of Kotlin, bit of Rust. And you need a build system to build all this thing. So this is Google, so they wrote their own build system. But the way it works, if you're familiar with, for example, open embedded or build route, you'll see that there are a lot of similarities. A build system is a build system is a build system. They all basically have to do the same thing. This talk is based on Android 13. And so big warning is that, first of all, the build system does tend to evolve release by release. And we know ahead of time that Android 14 is going to have some big changes in it as they begin the transition towards Basil. But we'll come to that later. But the full details of that aren't really apparent to me at the moment. So like I say, you'll have to invite me back next year to talk about the Basil version of this. Notes at the bottom there in small text. So the ASP build system only builds the ASP operating system and the kind of upper levels. It does not build a kernel. It does not build a bootloader or any other bits and pieces. Consequently, when you get ASP from your favorite Silicon vendor, they will have bundled with the ASP build system, their own build system on top of that, which is going to build a kernel and the bootloader and resulting in some truly weird stuff. I wrote stuff. You can make up your own word. So yeah, it's not entirely ideal. However, so first of all, going through just the quick top level thing, just a quick recap to make sure we're all on the same page. So the ASP source code is a large collection of Git repositories, over 1100 now. So you begin the process, you get repo, of course, and then you use repo init to go get the manifest, which is an XML file which lists all the actual Git trees you want to clone. So we do repo init, all that kind of stuff. Optionally, given a tag and in this case, Android 13 release 35. That just downloads the manifest. That's a fairly quick thing to do. And then you do a repo sync and that will then go through, iterates through the manifest and is essentially a Git clone for each one of those 1100 trees. So it's going to take a while and in Android 13, it's about 150 gigabytes that you download. Yeah, notice the contrast here to say build written Yachto. We build on Yachto, basically it downloads on demand. So as you download, as it goes to build a module, it will first of all download the code and then combine it. A repo and ASP don't work like that. You basically go get everything in one go. Even if you don't need those things, you're going to get the entire thing, including BSPs for stuff you may not be going to touch. So that's one reason why it's so big. Then you set up the shell environment. So you source the script build M setup. And this includes a whole bunch of shell functions, one of which is lunch. And so the lunch command allows you to select a build target, basically a BSP. So in my example, I'm actually building a Cuddlefish target for x86 phone. But the procedures are the same, whatever target you choose. These targets, then, they're defined in Android product.nk files. So if you go looking through the device directory of ASP, you'll find there's a whole bunch of Android product.nk files. That's what populates the lunch menu. And so the particular example for the Cuddlefish phone I'm using, that actually comes about because of that file there, device Google, Cuddlefish, Android products to mk. And within there, you'll find common lunch choices equals da-da-da. So that's how it becomes selectable as a product. So you run lunch, select your target, and then you build it. So building is very easy. You type M return. That's the easiest part of the whole process. And then have a coffee and have another coffee and take a vacation because the build is going to take, well, if you have powerful hardware, it may take about an hour. If you have more normal hardware, it's going to take four hours, six hours, whatever. So plenty of time to take up a hobby and learn some crocheting or something. So when it's finished, oh no, sorry, step back a little bit. Yeah. So how did it know what to build? So you selected a target, Cuddlefish in my case, and the packages that get built are all bundled together into this makefile variable called product underscore packages. And if you look through the various components that make up the entire build, you will see things like product packages plus equals and then a list of modules. So Cuddlefish service, Vsock input service, blah, blah, blah. And if you actually, you can actually dump out variables. You can use the tool get build var, for example, get build var product packages. That will then list all the product packages that are in your product. And it's not particularly easy to read because it's basically one big long line because the names are just separated by spaces, not by new lines. But if you do that on a typical product, you're going to find that it consists of somewhere between 900 and 1000 individual packages. So when you type M, it's based, amongst other things, it's iterating through product packages and it's doing a build on each one of those modules. And then it all finishes eventually, and it generates stuff in the out target product, product name directory, which in the case of Cuddlefish, turns out to be Vsock x8664. And if you look in there, amongst other things, you'll find these are the image files. So there's the boot image, there's the vendor image, there's a system image. And then typically you use fastboot or something to flash this onto your target device and then you go and test it. Okay, so in a couple of slides, that's essentially how Android build works. These Android modules then, there are currently defined in two different ways. You can either define them in android.bp files. So BP stands for blueprint. Blueprint is a language specific to ASP as a way of defining a module. This is basically a recipe. So if you want to think back into Yachto terms, this is somewhat similar to a BB file. And in Android 13, we have something in an excess of 8,000 Android BP files. There is also a legacy format, android.mk, which has been deprecated for quite some time, but there are still a lot of them around. I counted them. There's just a little bit less than a thousand Android mk files in ASP at the moment. So you'll find both of these things. And when you do the build, it basically will iterate through all of the BP and mk files and it'll build whatever it is you specified in your product packages. So now I want to dive a little bit and look under the hood or behind the curtain or whatever it is and see what actually is going on when you do this build. So it turns out that there are three components to building ASP. There's Soong, there's Carti, there's Ninja. It's kind of complicated. So I've got a graphical example to kind of help you and kind of put this all together. So first of all, start with Carti. So Carti is a genetically engineered augment and she is a follower of Khan Nyunian Sing. Okay, this is Khan Nyunian Sing, but he has nothing to do with the story. So instead we need to talk about another Nyunian. How many Nyunians can there be? This is Dr. Nyunian Soong. So Dr. Nyunian Soong is a AI genius inventor of data. Okay, this is data. So data is an Android, but not as we know it, or at least we don't know. I mean, does data run Android TM? I'm not quite sure. It's not made clear in the TV series. The other thing is the same actor. I mean, that's cheap. They could at least, they could at least employ a different actor. Anyhow, data is also not part of this story, so we can forget about him. And then the third character is Ninja. So Ninja turns out that well, the Ninja are a cast of Japanese soldiers, most notably known for assassinations and spying and all the other kind of daring do's. So those are three characters. We have Carti, we have Soong, we have Ninja. Okay. And so essentially, Carti and Soong are passing information to Ninja. And then Ninja with some deft sword play is creating for you an Android from maybe a tree stump and quickly carpet. So there you go. I hope that helps. Everything becomes much clearer now, I think. So how do we get here? So if you look back to the first publicly available source code release of Android, which was the Cupcake 1.5 release, 2008, actually maybe in 2009, strictly speaking. So originally it was all done driven by Gunnu Make. So the Android build system was a bunch of Makefile fragments, .mk files. And there's a whole thing about whether it's recursive mate or make or make file fragments, go Google or search using your favorite search engine, recursive make files considered harmful. Anyhow, initially it was just being file based and it was fairly simple and not particularly surprising. But it turns out that make doesn't really scale that way to really large projects. So as Android got bigger, the limitations of Gunnu Make became apparent. So slow to start up. The processing of the internal logic is not brilliant. There's no progressing indication. So you may remember in the early days, you type make and then you had no clue as to how many cups of coffee you had time to go drink. So you had to keep coming back after each cup of coffee. Has it finished yet? Oh, god, no. And then another coffee. So now we do have progress indication, which gives you some ideas to whether it's nearly time to stop drinking coffee. So in Android 7, Nougat, they introduced Kati and Ninja to overcome those particular problems. So essentially with Kati, Kati basically parses the MK files. So Gunnu Make is no longer involved. Kati parses all of the MK files, generates a Ninja manifest, and then Ninja works to the manifest, does all the work, and it gives you a nice progress bar. So that's good. And then for mysterious reasons, in 2017 in the Oreo release, they added Insung. So Asung is its own thing. It has its own format, blueprint, but it still generates Ninja manifests. In fact, it also actually doesn't quite even do that. It actually generates a make file, then Kati processes the make file into Ninja manifest, and then Ninja does the actual work. I think the original intention of Asung was that Asung would take over the entire process. But for various reasons, which I really don't understand, it kind of stalled and hasn't quite achieved that. So here we are in 2023 with the upside down cake release, Android 14 coming along in a couple of months time. And the big idea now is to add in a fourth component called Bazel. So how do you solve a problem with software? You add another layer. So hopefully the Android developers will follow through and actually use Bazel to replace all the other components, which as I understand it is the is the aim. So let's see what happens. Okay, so like I said, I want to go through not in particularly great detail, but I want to give you a little bit of an idea what each one of these components actually does. Starting with Soong. So as I said, Soong reads Android dot BP files. They're written in blueprint, which is a quotes Jason like language. And they are also quite similar to the format of Bazel build files. One of the key things about blueprint and Soong is that the blueprint files do not implement any logic. They are quotes declarative and quotes. So in other words, they're just a bunch of name value assignments. That's that's all you get in the blueprint file. So the logic is actually implemented separately in Soong modules, which are written in the go language. And we'll come to that in a moment or two. So the code for Soong, you find it in the build Soong directory. Don't ASP by the way is just my shortcut for wherever your ASP code based route is. And there's some documentation. This bit is looking at the startup. And actually, the main thing that I was have been worried about for a while is when I type make, how come it runs Soong. So I eventually dug this out. It's a little bit hard to read. We don't need to read all of it. But the main thing is that there's a bit of logic there that says, if you're at the top level, then when you type make, it actually runs M. What actually runs Soong UI and Soong UI actually does the invokes the build process. So if you're at the top level, if you're in dollar ASP, M and make are identical. If you're in any other directory, then make actually does do make, which is confusing. And we also have these variants. So as well as M, we have MM and MMM. So if you type M or make that will build the entire thing. And it builds seeking towards a target called droid. So the default target is called droid. If you just want to build a single module, then you can just change directory into the directory which contains the Android BP or Android MK file. And you can type MM. And that will then run the build system with the target of just that one module. So usually that's much faster than doing a complete build. And then we have the three M's kind of similar, except now you can give one or more directories as parameter. And it will essentially CD into each one of those directories and run MM in each individual directory. So the driver for the whole thing is Soong UI. So when you type MM, MMM or make, you always end up in Soong UI. And that is what actually then runs the various jobs. Soong UI takes a command line parameter. So you can actually run it from the command line if you wish. But it has basically these three modes. It has make mode, which is basically what happens when you type M. It has dump var mode, which is what happens when you type get build var. And it also has dump vars mode, which is actually used internally with to generate the variable list, which CAR T uses later on. Quick complaint to anybody who may be listening from Google, for example, it'd be really great if dump var mode will take a wild card or something. I'd really would like to type dump var mode and give some kind of regular expression. But it doesn't. Okay, help with make or M. You can type M help, and it gives you the list of possible... Well, no, it gives you some hints about possible targets. Common goals are. So you can do M clean, which just deletes the entire directory. We have droid, which is the default target, and some other things which you can read from the slide yourself. When you run M, if you don't give a minus J option, it will choose as many jobs as you have processors plus two. Again, the plus two is a little bit mysterious. But if I run this, say, on an eight core machine, it's actually going to run 10 jobs. So it's equivalent to minus J 10. Here's a very quick look at an Android BP file, just in case you've never seen one before. So this is a CC binary. So CC binary. So this is the type of module we're going to build. So it could be Android app is another one or Java library or whatever. So that is the type of thing we're going to build. In this case, a binary compiled from C and C++ code. Every module has a name in this case log cat. And as somebody was talking about just now, that name has to be unique throughout the global namespace. So you cannot have two modules with the same name. Kind of makes sense unless you name spaces, but that's another topic. List of source files, list of libraries, list of C flags. So that is going to invoke the cross compiler. It's going to build log cat. And it's going to install it by default into the out directory into system bin log cat. So at this level, it seems pretty simple. A quick note about dependencies. So when you write Android BP file, first of all, there are implicit dependencies on any libraries. So since here we are linking with lib base, lib process group. Those are implicit, implicit dependencies. So the system's clever enough to realize that we need to build lib base and the process group before we can build log cat. And then you can add in explicit dependencies using the required keyword. So you get a list of modules. These modules will be built before it builds this module. So I said that the CC binary is a type of module. And here are some other examples, which I'm going to skip over. Oh, yeah. If you want to see the information, the documentation about the various modules you can build, then you can type m soon underscore docs. And that will then compile the documentation. And then you can go read the file. If you're looking at soon docs, soon build HTML, that then lists all the various module types and all the various attributes each module type can have. So these module types, like I say, the logic is written in go. So here's an example. First of all, the source code for soon is in build slash soon. If you have a look through that, you'll find things like this. So have register module type, CC binary, and then binary factory is the subroutine that's actually going to do the processing. There are, by my count, there are over 300 module types in Android 13. And the fact that there's been quite an explosion of different module types over the last several releases. And somebody was discussing just in the presentation before about the possibility of doing certain things with Android BP files. And yeah, if you actually want to do anything other than just the simple stuff, then you would have to add in a plugin and define your own module type. So when you run soon, so you'll see that it's running soon. It actually appears as a line on the one of the first things you type after typing M. Essentially, it is writing this file called build.ninja. And it says analyzing BP files and generating ninja file at blah, blah, blah. So this is a big file. So it's a text file, but it's usually between five and 10 gigabytes. So that's why it takes a little while to complete this stage. And also, it has to regenerate this whenever you change anywhere in the system. If you change a single Android BP file, that is going to force it to regenerate the entire build.ninja file. Now, the interesting thing, then, if you go look into that file, it basically has build rules for every single Android BP file. And that's because at this point, soon doesn't know what the target is. Only Carti knows what the target is that we're trying to build. So soon just blindly goes through every single Android BP file it can find, puts some build rules into this build.ninja file. So even if there are components that you're never going to use, it still has to generate the build rules for them. It also generates a .make file, Android dash, whatever, which uses input to Carti later on. Oops, I meant to delete that slide. Okay, so there are some, let me just skip. Yeah, okay. TBD, describe what you can find here. I didn't do that bit. So there are some log files. I'll leave you to work out what's in them. I mean, the thing is extensively logged, but the contents of the log files is not 100% obvious. And I was trying to find a kind of definitive reference as to what is in the log files, but I couldn't actually find it apart from reading the code. And I ran a design reading the code. Okay, so how are we doing? Okay, I better speed things up a little bit. So that was soon. Let's talk about Carti. So Carti is a new make file clone. It parses these .mk files and generates the Ninja manifests. Here's a typical Android .mk file. This is similar to functionality of the BP file we looked at a couple of slides back. And oh yeah, comment at the bottom, looks a lot like build route. Yeah, this is essentially how build route works. Again, there are dependencies, there are implicit dependencies, libraries and such like, and you can add in your own explicit dependencies in this case with a required local required modules. Oh yeah, we think because of the way it works, you cannot have a dependency from an Android .mk file to an Android .bp file. So you can do it the other way, but mk files can't have dependencies on modules defined in BP files. So when you run, I've confused Ninja and Carti. This should say Carti outputs. So Carti generates a Ninja manifest called something.Ninja, for example, build ASP CF blah blah blah. And this contains the build rules for all of the .mk file stuff. It's still quite a big file. So it's less than a gigabyte, but it's quite a few hundred megabytes. So that then brings me to Ninja. So Ninja is basically stitching all this stuff together. Let's get that. So Ninja is its own build system, although the Ninja rules are not intended to be kind of written by humans. They're mostly generated by tools such as Ninja. But there's a quick idea as to what is in Ninja. So we have some variables such as C flags. We have a rule called CC, which tells you how to compile a component. And then we have a dependency. So build food depends on food at C, and it's going to use the CC rule to build it. That's basically what it is. So Ninja is started from suing UI. And I dug this out because it's kind of interesting to see how this kind of rolls out. So you can see the target there. So the default target is droid. You can see the parameter, the minus j parameter I gave because I wanted to limit it to 16 jobs. And it is going to process the file combined ASP, Cuttlefish, whatever, .Ninja. If you go look into that file, it is this little file that lower down the slide, it is running a sub-Ninja on all the other Ninja files that have already been generated. So the first two were generated by Carti, and then the last one, the build.Ninja, was generated by suing. So in that way, it brings all the Ninja files together. It knows what the target is. It's droid. So it can work out the dependency tree. It can build the whole thing. The interesting little thing that I discovered is that you can actually run Ninja directly. Ninja has some useful tools. So one of the things that I've always wondered is how do you generate, how can you see the dependencies? So you can do this by calling Ninja minus t, minus t query. In this case, I'm doing a query on logcat, and it gives me the dependencies on the logcat module. So we have two lists. We have the input dependencies. So these are things that logcat depends on. And we have the output dependencies. So these are the things that depend on logcat. So that's quite useful. Pushing a little bit further, you can actually get it to draw a graph of this. So minus t graph, generate dot dot file, then you convert the dot dot file to, for example, a PDF. Not as helpful as I had hoped. Maybe I could tweak this a little bit, but yeah, send me patches if you know a better way of doing this. And oh, we're doing quite well, actually. Okay. So that was Ninja. Final thing. I've got a couple of slides on Bazel. Like I say, Bazel is still a little bit upcoming. So we don't know for sure exactly how it's going to be deployed because it's not released yet. However, the eventual goal, as I understand it, is to replace all of those things, CartiSoon and Ninja, with one tool called Bazel, which in theory should be a good thing, as long as they actually do that. And it should speed the build time up a little bit, particularly the start of the build. So all this messing around, generating dot Ninja files, most of which you don't need, and then generating make files and then reprocessing them in Carti, it's a little bit crazy really. So if this works, then this will be a great improvement to the Android build system. So my understanding then is that we will see the first deployment of this in upside down cake. And it's going to be gradually rolled out, I believe, over the next few releases. So maybe in three or four years' time, we'll have a completely Bazel-based build. Right now in Android 13, the only bit of the build process, which is completely done in Bazel, actually is the kernel. So I kind of skipped that a little bit. But if you go down, download the Android common kernel, the ACK, there is a Bazel, set of Bazel files there, and you can build it using Bazel. And there are some references. Bazel in one slide. So this is as much as I managed to discover in the short time on the plane over as I was writing this stuff. So how do you know it's Bazel, first of all? If you look for these two files, if you see a workspace file in uppercase and a build file, you're probably dealing with Bazel. So the workspace file, this defines the workspace, basically the top level directory. And then the build files, these essentially are the rules. And then the, sorry, these are the equivalent of the BP files. And then there's a Bazel logic is put into BCL files. The language is called starlock apparently. So, yeah, looking at the master branch, if you do a build of the master branch, you'll see that Bazel has replaced Ninja, but we're still using Soong and Kati to feed information to Bazel. And I've got a couple of minutes free. Yeah. So there you go. So that is my interpretation of the Android build system. Does anybody have any questions? At the end, you said the kernel will be built by Bazel. Will it replace the scripts for the GKI when we want to build your own kernel? There are lots of scripts based on the GKI architecture with a common kernel and specific kernel. Yeah, basically it will. So in Android 13, the official way of building the kernel is still to use the build.sh script. Yes. But there is also a Bazel build mechanism and you can invoke a Bazel build, Bazel blah, blah, blah, and that works as well. I think in Android 14, the build.sh script will disappear and then Bazel will be the only option. But the GKI will be the same. GKI will be the same. The GKI is the same. I mean, in theory, it produces the same binary. It does the same thing under the hood. Is Soong actually built as part of the build process? So if you add new rules to that, do you have to rebuild a custom version of Soong and invoke that or do you just change it in the source tree and it will work automatically? Yeah, how do you add modules? Okay, I'm going to have to say I've never actually done this. But my understanding is you can just add in the module and it is controlled. I mean, it's essentially Soong being built Soong. So you should be able to modify the, or add in your own .go rules, .go files, add them to the appropriate Android VP file, rebuild everything. It should rebuild Soong with your changes in. Has anybody here actually done that? Can you confirm that I've got this right? Okay. Okay, right. So answer from the audience is, yes, that's basically how it works. Anyone else? Thanks for the talk. Quite interesting. As someone who's also patched some parts of the Soong things for some special tasks we had in another project, you talked about the modules and you quoted plugins for Soong. Do you know if there is something similar in Bazel plant or actually there? Well, you're stretching my knowledge a little bit. But again, my understanding is that the logic that used to be, or is currently implemented in Go, is instead implemented in Starlock in these .bzl files. And it says Starlock is a Python-like language. So it's a scripted language so that there's no compilation part. I guess it's the important bit there. So yeah, I mean, these are a little bit like bb.class files in Yocto. Do the BP and android.mk files live in the various source directories as downloaded as GITS? Yeah, so when you write a module in Android, the BP or mk file is in the same directory as the code you're trying to compile. So does that mean you have to maintain multiple build systems for your component if it's used outside of Android as well? It depends which Android versions you are targeting. But assuming you're targeting anything since Android 8 ish, then an Android BP file should be sufficient. But if I want to build the same code outside of Android? Oh, sorry. Yeah, I'm a miscellaneous question. Sorry, you can't, basically. So the Android build system is hermetic, meaning basically the Android build system only sees the stuff within the Android build top directory. So you can't separately compile a module without building the entire thing. That just doesn't work. What probably what do you mean is if you have an external application, what I did in the past on the external library, you usually have, for example, the classical mk file there. And to build inside the Android, you pull the source code in and add something like an Android mk, an Android dot BP, and then call the mk file itself, something like that. So you have to add an additional build system if you want to have an existing module, if you want to build it inside the Android and outside. Yeah, thank you. Yeah, that helps. So typically that kind of stuff is done in say Android Studio, you build an APK in Android Studio, then you include it into ASP as a binary. That's typically how it works. And you said that you cannot have dependencies between the mk target and BP target. Is this bidirectional or only one to the other? Because there is the droid, the first, the last dependent one is droid, and then you have, should have dependency on both of them, something like that. Yeah, I only know that it doesn't work. So if you have an Android mk file and you want to have a dependency on something that's declared an Android mk file, that definitely doesn't work. I haven't quite got to the exact reason why that is so. Maybe because there are two different tools that calculate the dependencies. I think it's something to do with the way that Ninja, you know that that sub-Ninja slider slow showed. I think it's something to do with the order in which those dot Ninja files are included. But I've not had chance to verify that. Okay, thanks. Thank you, Chris. That's it. Okay. Oh, and don't forget to come along to the bath two o'clock this afternoon.