 So hello, my name is Chris Simmons and I'm here to talk to you about Yocto project and Android How are they the same and how are they different? Here's a license for this stuff and Here is some background information on me But the main thing I want to talk about is Yocto project and Android So previously I have spoken about Yocto project and Debian At embedded Linux conference and various other places as well I've also spoken about Android and whether Android makes a suitable embedded operating system So now I want to combine these two threads together and talk about Yocto project and Android How are they similar? How are they different? So on the compare side Yocto project and AUSP of the Android open source project are in some ways the same They are both open source They are both build systems And they can both build Full operating systems that suit the kind of needs that we have for embedded devices So they are both used by many people All over the world for a whole range of embedded products and yet they are also quite different So Yocto project is a classic community-driven open source project Designed for maximum flexibility and to produce a wide range of operating systems for embedded devices AUSP on the other hand is very much sponsored by Google and they Control the direction that it goes Mostly to target the devices that they care about smartphones So the differences really come down to ownership control of the project and the community around that project And I will look at all that in a little while But first of all, let's have a look at some technical stuff. I'll begin then with an overview of Yocto project So Yocto project is as they like to say on their website It's not an embedded Linux distribution, but it can build one for you So it is based on the bit-baked build tool and open embedded and Yocto project specific metadata and Together these can produce Operating systems of a wide range of different capabilities So let's start off by getting a copy of Yocto project So you can do that by using the git clone command. I give it the top of the slide there and what do you get? well, you'll get roughly 55 megabytes of tools and documentation and metadata and Also quite a lot of git history So it is quite small one of the reasons it's quite small is that You only have the tools and the metadata to build things you don't have the actual source code That is that comes along later So having cloned your copy of pocky You then cd into the pocky directory you source a script and that sets up the build environment for you And it also will create for you if you're doing this the first time a build directory And by default that directory is actually called build then in that directory it will create a configuration sub directory and It will put in there a file called local.conf so here you set a whole bunch of Project specific configuration including the name of the machine you want to build for So for example, I'm building here for a beagle bone So I put machine equals beagle bone and that will pick up the beagle bone board support packages But it could just as well be rise be pie or Generic PC or whatever So the metadata I keep referring to Is a bunch of things, but especially it is recipes so recipes are files that are consumed by the bitbake build tool and A recipe typically will download source code from upstream and Compile it and produce a binary package and then you'll take those binary packages at the end of the process and build for me images For the root file system and other components So just so you have an idea where I'm talking about this is a bitbake recipe so this is a BB file and It's going to build a simple hello world program This particular recipe is not typical in one way in that if you see down here where it says source URI This is referencing a local file Most recipes don't build local files. They will reference here a remote server Using HTTPS or something like that or they can reference a git repository In which case the recipe will download the code from that server or git repository and then build it for you So now I want to tie together the configuration the recipes and this idea of layers So a layer is a collection of recipes and We have two in my example here called meta and meta pocky the layers you're using in your project are Defined in another configuration file, which is in your conf directory. This one is bblayers.conf So if I want to add more layers to my project I just extend bblayers.conf and then I also have the local.conf there Specifying the machine I'm building for for example Beaglebone and the distro Which by default will be called pocky So amongst those recipes will be image recipes So an image recipe is a kind of top-level recipe which lists the packages that you want to build into your target Here are three examples of such image recipes There's one called core image minimal. This will generate the smallest possible Linux embedded Linux system Which will be just enough to boot the system and give you a root shell and not much else But there are many others Including examples on the slide there. And so the process goes something like this I bit bake a recipe typically I bit bake a image recipe Bit bake will work out all the dependencies and Then you will iterate through downloading those dependencies from upstream source So for example, these will be tarred up GZ files. It downloads those Configures and compiles them to produce packages by default. These will be RPM binary packages although there can be other formats and Then as the final stage it will take those RPM files those packages and It will extract them to generate the final root file system So this idea of layers or meta layers everything is meta in your project Actually makes the Octo project very extensible and very powerful thereby So we can Starting from a simple configuration. We can add in the layers for whichever components we might need So there is a semi official list of layers at this link here Let me just click on that for a moment So this is the open embedded layer index and it lists. I haven't counted but quite a few hundred layers Some of these layers are labeled as being machine or BSP layers So these have the metadata required to build for a particular machine If I scroll down a little bit bear with me for a moment Beyond that we have layers that are labeled with type distribution So the distribution is the policy as to which packages you should you should choose When bit bake is building stuff For example, there is a distribution here called automotive grade Linux or meta HL And this is the layer you would use if you're doing stuff with automotive Head units and that kind of thing and then scrolling down a little bit further I'll skip that for a minute There are a bunch of layers called software So these are a variety of things they include things like graphics libraries I See there's one here for Amazon AWS if you want to integrate AWS into your product and that kind of thing So switching back to the sides Yeah, so the layers as you can see there's a lot of layers there So we looked at machine layers meta Raspberry Pi would be an example of that distributions Software layers and then there's another category called miscellaneous Which is basically stuff that doesn't fit into any of the other categories In addition to these layers almost every SOC vendor and Board vendor will have their own layers So you can incorporate and run Yachto project based operating systems on their hardware. So what can you do with the Octo project? and the answer is in Principle anything you want well certainly a very large number of things anyhow, so here are some things that I personally worked on So I worked on a ticketing machine Issuing tickets for trams and trains and that kind of thing I did some work on a pure water purity testing machine It's got to be done a Little smart entry phone device. So that was a fairly simple device with a touchscreen interface Which is a replacement for the classic entry phone thing I've done quite a lot of work over the years on automotive systems based for example on a GL and also on Communications including 4g and 5g base stations. So these are just some random Examples that I've worked on There are many many many other examples that I'm sure you know of Okay, so that's Yachto. How about ASP So here we are talking specifically about the Android open source project This is the open source components of Android so ASP consists of a build system a Base operating system which we call the native layer So the native layer includes an init program the demons that init will launch a command shell all the libraries to support all that stuff and The hardware abstraction layer then above that we have the Android framework and The Android runtime So these are the core parts of the Android operating system For the most part written in Java and then at the top level as part of ASP There are some fairly simple demo apps. So there is a home screen called launcher 3 There's also I think a launcher 2 in there somewhere There is a settings app so you can go and tweak the settings of the operating system and There are some simple apps like desk Clark and calculator and that kind of stuff Just demo is nothing particularly useful Now all Android devices actually are based on this very same code base everything is ASP and Then there are layers on top of that so that the the Android phony or pocket has a lot of Polish applications put on top of this but underneath is all the same stuff In terms of licensing The preferred license from Google is Apache So most of their code is written about using Apache 2.0 license But there are also some components using BSD licenses and maybe some others that I've not actually tracked down yet So ASP is open-source Which is nice. It means we can do whatever we like with it We can take ASP modified tweak it and ship it and This is exactly what people do So Amazon for example use ASP for their fire branded devices including the Kindle fire and such things and We can use it to build embedded operating systems so-called embedded Android Kind of things you would use it for So test and measurement is quite a big thing essentially anything that has a touchscreen is a good candidate for embedded Android So test and measurement so oscilloscopes Frequency analyzers that kind of thing Digital advertising as a big area typically based on the Android TV and set up boxes based on Android TV and One particular case I've noticed is marine navigation So when you have your yacht parked in the harbor Yeah, okay, I don't have a yacht, but I could have you never know Then that yacht would have a navigation system and there's a good chance that navigation system actually will be running Android for its user interface Now in all of these cases you probably wouldn't be aware that Android is being used because these are deeply embedded These are mostly single-use single application Android devices So let's have a look at what a USP is composed of So AOSP is built from a fairly large number of components each one of these components is a git repository stored In the default case at least at Google source comm The list of Projects Let's try that again the list of Git repositories that you're going to use in any particular Implementation are held in this thing of this manifest file So this is an XML file with a particular format and you can see down here where we have Project so each one of these refers to essentially to a git repository so the repository in this case would be platform slash art at Google source comm and When we get this project it will be stored locally in a directory called art So this is just the first few lines of a manifest a typical manifest is eight or nine hundred lines long And then there's a tool called repo that can process these manifests So you use repo in two phases first of all you run repo in it and You give it a pointer to the manifest you want to grab Again, this is the canonical Manifest from Google and I can optionally give a tag to clone a particular Version of this manifest So repo in it is a fairly fast operation That's just going to copy the manifest and the components that support that and Then I run repo sync So repo sync is going to iterate through that manifest file and do a git clone of each one of these several hundred repositories Now here's a difference between AOSP and Yachto project Whereas Yachto will download the code as needed on demand as it builds things With AOSP when we run a repo sync that is going to download absolutely everything all in one go So a repo sync is going to take quite a while and it's going to download in excess of a hundred gigabytes of stuff Here's another difference. I didn't actually mention this in the Yachto Overview but when you run Yachto one of things it will build will be a toolchain for whichever supported platform you have selected for example the Beaglebone AOSP doesn't build the toolchains on the fly instead the toolchains are supplied pre-built as binaries in a directory called pre-built and You will find in their various versions of the Clam compiler and Since Android 11 also a Rust compiler and these toolchains and the things around them will support the AOSP architectures which are ARM and AOSP86 in both 32 and 64-bit variants So the fact that the toolchains are supplied pre-built does have some implications So whilst it reduces time during the build phase It You end up downloading more stuff during the repo sync phase So each toolchain is quite a few megabytes And we end up if you look in the pre-built Clang directory and dig down a little bit You'll see there are eight or ten different versions of clang in there for backwards compatibility. I guess So this inflates the size of the pre-built directory and currently so I'm talking about Android 11 here It's about 42 gigabytes Okay, the build system so the build system for AOSP is called Soong and Soong parses recipes written in a language called blueprint These are both basically unique to AOSP Here's an example of a Blue site of a blueprint recipe So the recipes always are in files called Android dot BP And you can see here that This is what we call a module. So this is the type of module CC indicates It's a C and C++ code. So we use a CC compiler The binary part means this is going to produce a binary executable and Then within the curly brackets, we have the attributes of this module So this is a very simple module it has a name all modules have to have a name Then a list of source files just one in this case a list of libraries and optionally some C flags So when Soong processes this Android BP file, it is going to compile LMKD.C link it with a couple of libraries and It will end up with an executable Which will have the same name as the module will end up with an executable called LMKD And that will be installed in this system slash bin directory In addition to the components written by Google If you go look in a directory called external, which is part of a USP You will see there are Several hundred upstream open source projects and in the listing on the right there I list I just have a random collection of them So these are local copies of upstream source source projects Why do we need to do this? well Basically, it's because the Soong build tool doesn't know anything about open source packaging So it doesn't support order tools or CMake or even make file. It can't pro it can't even process just a regular make file So what has happened here is that the Android developers or somebody has included these upstream projects within the externals directory and Somebody's had to sit down and handcraft an Android BP file for it And that can be simple or it can be difficult depending on the project USP comes with a bunch of board support packages for devices and these are mostly in the directory devices, but they can also be in a directory called vendor So for each device there is a sub directory and within that sub directory there will be a file called Android products mk So this essentially defines the device It adds it can also add the device to the lunch menu, which I mentioned on the next slide and It gives a pointer to the device make file So altogether this is going to define the software configuration of the device There in other words the list of packages and the type of packages used on this device The other important file is called board config dot mk So this as the name suggests is the hardware configuration So in here we specify the CPU architecture and sub architecture. We Give a list of the flash memory partitions and their sizes and similar stuff so you get a default set of board support packages with the AOSP download and Typically when you're working with a particular board from a particular manufacturer They will supply their own device configurations for those boards So Android products to mk like I say that points to what we call the device make file and this contains the software configuration for the for the device One of the things you do here is you have you inherit a base configuration for whatever type of device it is your building So in this example, I am inheriting a USB base telephony dot mk So this would build for me essentially a smartphone As well as smartphones, I can select configurations to build a tablet or a TV or An Android automotive OS system. So as we said the Android products to mk file adds A product to the lunch menu So in order to build something you first of all have to set up the environment So in my example, we source build slash M set up to sh you always have to do this and then you can run lunch and Lunch will give me a list of things configured in my devices directory So say I want to build a for a Dragon board 845 So I select DB 845 C, which I see is number 38 on my menu So I type 38 and hit enter. So now I'm configured to build for that particular device And then I can do a build I type M or make if you like and then I go for a walk Quite a long walk actually because it's going to take a few hours to build AOSP however doesn't do the whole thing for you. It really only produces the root file system In addition to that then you're going to need a kernel and of course a bootloader Well, let's just talk about the kernel for the moment So historically It's been up to the OEM or the SOC vendor to supply a kernel for the hub for whatever hardware you have The Android developers produced a reference kernel called the Android common kernel, which you can download from Google source.com And then the SOC vendors take that and they modify it with usually thousands of in-house patches To make it work on their particular hardware This whole process takes a little while Which is one reason why the Linux kernel versions used on Android devices tends to be quite old two or three or more years behind upstream kernels This bit however is changing Since Android 11 we have a thing called the generic kernel image or GKI So this is quite a big change to the infrastructure for Android The basic idea is that instead of everybody producing their own kernel There will be just a single kernel or at least a single kernel code base Which is produced by the Android developers and then everybody whether your phone is from Samsung or Nokia or someone else entirely they will all be running exactly the same kernel Now, of course, there still needs to be some device specific kernel modules in there So each vendor will provide a set of Linux kernel modules which add in the necessary features for their particular chipsets and I have a feeling that most of these are going to be binary only modules So I think the net result of this is that there's going to be much more binary only binary blobs in your Android device Which brings me on then to the question Can you make a 100% open source Android product and the truthful answer here really is no Why is this well because of these things I've just mentioned in addition to which we have the support for OpenGL for a given graphics GPU chipset These are almost always proprietary. We have the hardware abstraction layer the house These are usually proprietary and binary only Even before GKI we had binary kernel modules typically for Wi-Fi and Bluetooth and the datamodem GKI I think is going to accelerate that so we have many more binary kernel modules and Finally, there are a bunch of binaries that you need to load into various parts of the system as you boot up So these binary blobs are to be honest a real pain They ties into a particular vendor and to a particular version of the software and They also ties into the support time scale of that particular SOC vendor because once they stop supporting a binary component There's nothing we can do about it So binary blobs are a pain ideally they will go away, but they seem to be doing the opposite We seem to be getting more of them Okay, the next few slides so from time to time people observe to me that Android is the most popular Linux distro ever and In a very trivial way that is true since Android is based on a Linux kernel Android is the most popular version of Linux But it is not a distro in any real sense For AS for Android to be a distro. I would need to be able to take A project an open-source project that for example runs on Debian and have it run on Android Now naturally I can't I wouldn't expect to do that as a binary I don't I don't expect binary compatibility necessarily, but I would like source code compatibility But that doesn't exist either So the next couple of slides I highlight four particular areas Which illustrate why Android is not a Linux distro first of all incompatible and incomplete shared libraries So the most obvious thing here is if we look at the C library The C library in Android is called bionic, which is a very stripped-down C library and which is not POSIX compatible. It emits quite a few POSIX functions So if you're compiling your code against Bionic You'll find that a bunch of things don't link and you're going to have to change your code to fix that But not only is it the C library There are a whole load of other system libraries you would expect to find or be able to install Onto the system that is simply missing in Android and as we mentioned already it's difficult to add libraries to to Android because the build system doesn't understand open-source packaging, but it's not only that The library loader LD-android.so Has its own peculiarities In particular, there is a thing called the VNDK, which is the vendor native development kit Introduces part of project treble in Android 8 Essentially, the VNDK is a fairly complex set of rules for linker namespaces and It essentially limits where you can load libraries from So you can't simply plonk a library in The VNDK quite likely will not allow that unless you go and change its rules Which you may also find difficult Another big drag is SE Linux and the security policy in general Now SE Linux is a good thing. Don't get me wrong. I like SE Linux. It makes the system more secure But the snag is that the policy written that you for Android and supplied with AOSP is very much geared towards the use cases that interest Google and You can't defeat it So you can't disable it. It's enforced by the init program. If you disable it, the init program will simply refuse to run Maybe you could write your own init program, but then you're into big changes And also the rules are enforced at build time by soon. There are various Checks in there, which will give you build errors if you try and do things that don't fit the policy so this makes things unnecessarily complicated and Make it difficult to add stuff at the lower level of the operating system and then fourthly I've mentioned this already in fact the build system soon does not understand upstream stuff and so if you want to integrate something you're going to have to sit down and write an Android BP file and Resolve the other things I mentioned on this slide and the previous one So adding stuff to Android is not easy Also, you should realize that AOSP is not a stable platform By which I mean that the internal ABIs and the SE policy and the link and namespaces and the locations of Files and various things change from one Android version to the next So if you have added Extensions to AOSP which depend on any of these things API's or policy or whatever Then they are going to break when you try and upgrade to the next version of Android and There are usually two responses to that the most common response unfortunately is to just not do the upgrade And so you end up repeatedly with embedded Android devices based on ancient versions and vulnerable versions of AOSP But the cost of upgrading to a later supported version is just too high The other response is to buy the bullet and actually go and make the changes. Yeah, but not so many people do that So the only ABIs that are stable in Android are the ABIs Associated with Android applications Obviously The Android developers don't want to break Android applications because they need to keep on working from one version to the next And also the vendor Howe is a protected ABI Because it is important to be able to switch board support packages without having to Upgrade the main operating system or actually more likely the other way around To what great the main Android application The main Android operating system the system image without upgrading the vendor Howe's But beyond those everything all bets are off. So how far can you go with embedded Android? So here I'm kind of rolling back a little bit on some of the things I may have mentioned in the past so essentially The main message is don't mess with the base operating system Any changes you make which depend on these ABIs I've just been talking about will most likely break in the next release Now this isn't to say you can't make any changes to AOSP It's fine to add additional native services or demons The API's for that are fairly robust and stable. There will be some breakage, but probably not that major If you want to go a little bit further it is also possible to add system services and The API's to do that again are reasonably stable or they're not so stable as for the native demons If you're going to do these things however make sure that you are not doing anything too intrusive Otherwise you are building for yourself a support burden and you're going to make it difficult to upgrade Later on so the best thing to do actually is not modify the base operating system But actually keep as much as possible in applications the APK layer is stable and If you want to do lower level stuff you can put stuff into you can use NDK to bundle Libraries and some libraries are supported upstream for example if you want to use G streamer then the G streamer project has support for Android to include these into applications via NDK and Opencl and opencd and all that kind of stuff Okay, so Let's move on then to have a quick look at the development development processes of both of these projects so the development process for your project is Classic open source project so there are mail lists when we can communicate and See the decisions made by the developers The git repositories are open and we can see every git commit and the messages associated with those git commits and Though we can also delve deep more deeply for example There is a weekly weekly project status report which you can look at and receive on the mail list So you can see exactly where the opto is Week by week day by day even hour by hour ASP is the complete opposite Most of the development on the ASP is closed so we can't see what's happening from day to day or even month to month The roadmap for the operating system is not published There are various hints that leak out from time to time, but with no real idea for example today In early September, I have no idea what is going to be in Android 12 even though It's probably going to be released before you see this video If we go to the git repositories at google source comm You will see there is a master branch there or maybe it's called main these days and You may think that is the next release it really is not Let me show you what actually happens So this represents two timelines the top one Represents the timeline for the master branch. So this is one that's publicly available But like I say, this is not where the development actually happens at the beginning of a development cycle The Android developers take a copy So they create a private copy of master and then they do most of their development on this private copy here And they will work on that for eight or ten months or whatever When they're ready usually in September or October They will do a release So essentially they'll make this branch public and then over the next couple of months they will merge then The new release into the master branch. So by the end of this period here The master branch contains all the changes that were in the Android 10 branch and Then once they've done that they repeat the process They grab a copy of the master work on that internally for eight to ten months. That's your next release So The development is the license why that says it's open, but the developed method development methodology is very definitely closed So let's talk a little bit about community community and Yachto project Yachto product is all about the community and the developers on Yachto project are very supportive of the community They are available on mail lists and via chat channels of various sorts You will find them at various conferences and they're always willing to help but nearly always. Let's be fair There's a steering committee, so you know the goals of the project and you know where it's going and How far it's got ASP well, you know what I'm gonna say here ASP is the opposite. There isn't really much community at all Google don't really encourage Community they like to have complete control over the Android operating system They do development in secret as we've just said The Android developers don't make themselves available via mail it mail this so it's difficult to ask questions about How it's meant to work what you meant to do with things and why certain errors are happening So ASP doesn't really have any community. Oh, yes and resourcing Well, you know again chalk and cheese Yachto project and I'm sorry Android. It's not with Android here Well both Android and Yachto are loss making obviously they are open source projects They do not of themselves generate any revenue. That's obvious But in the case of Android The Android operating system produces a very large revenue for Google via various indirect ways advertising search information all that kind of stuff and So it makes sense for Google to sponsor ASP and to maintain a fairly large development team Yachto in the other hand Doesn't have That kind of benefit. So it has some sponsorship from Linux Foundation who sponsor Some developers and there are donations and sponsorship from other organizations, but it's on a much much lower level So the net effect is Android evolves quickly But in the direction that is specified by Google Yachto project evolves rather more slowly But in a way that is compatible with a large a much larger range of people So nearly done. Let me bring you then to a conclusion So my conclusion Essentially is that with a USB with Android what you get is what you get So if that fits what you want to do, that's fine. Go ahead. Everything's everything's good but I Have encountered people who have tried to mold AOSP into places where it's not really intended to go and I've seen people Essentially wasting time trying to modify a OSP when it probably would have been much easier to start from scratch And build the product using Yachto and then my final Point is really a call out For community based projects and Yachto project is one such but there are many many or many many more as well So this is a call out for these these projects and the people who work on them. They all deserve a little love and We're appropriate. I think it would be good for us developers to push the awareness of these projects up the tree within our organization and Maybe Get chance to commit a little bit more upstream maybe Get to the point where we can actually do donations or least though the people we work for can donate some money upstream That would level the field a little bit So thank you very much That is the end of the presentation If you have any questions, please go ahead and ask them on the appropriate channels And I will be there to answer those questions on the day Meanwhile the slides slides are available at that link there So thank you all very much for listening and watching and I hope to see you in person at the next embedded limits conference wherever that may be