 Hi, good morning, Android people. So I just want to remind you that we have a sequence of presentations on AOSP today. So I hope you find them all interesting. So following from Anna, talking about building Android systems that are good and the bad, there is Amit Pundir talking about dev boards and Android. Then at 11 o'clock, there's me talking about the Android build systems. And then crucially, after lunch at 1,400, we have the buff. I hope you all come along to the buff because I want lots of contributions from everybody for that. And then just before the closing game at 1,650 or something, we have Chris Haynes from MemFault talking about debugging and logging Android systems in the field. So a lot of fun stuff. I hope you enjoy it all. Good morning, and welcome back. In the first talk of the day, we have with Anna Lena building embedded systems with AOSP. Over to you, La. So thank you. Nice to be here today. I think the most of you come from the site building embedded systems with classical Linux. Today, I want to take you with me in the embedded Android world. I want to tell you a bit about why you should consider using AOSP for building embedded systems, in some cases, and why not, and share some of my personal best practices and pitfalls when doing so. But first, some words about me. I'm Anna. I'm embedded system dev at Inovax. My most important topics are Linux kernel embedded systems with Yocto and embedded Android. Cool. I have a background in computer science and embedded systems. And because I'm not a fan of the most hardware people, I start studying it myself so I can do even better. And so let's have a look on how we can build systems with AOSP. I've put some pictures of actual products running with AOSP with me, rather they build from people in my company or from myself. The first one is picture of a German e-reader. That's one of us. That's running with Android. The second one is security camera where the whole ecosystem is open for different vendors. And the base platform is built on base of the AOSP. And the third one is a card diagnosis system, which also is built on top of the AOSP. But I don't want to go more deep in there. It's just an example for you on how AOSP is actually used in real life. It's not just something I tell you here. There are actually products out there where the AOSP is used beside phones. But so, yeah, let's start with why you should consider AOSP for embedded systems and a small introduction in the AOSP wordings and some backgrounds. Something to the wording first. I probably will mess around with Android and AOSP in this talk. The actual name is the Android open source project AOSP. And there's a small but important difference between AOSP and Android. Because when talking about Android, it mostly means the certified Android. You have to pass some test reads and certification and so on to call your product Android compatible, even if it's mostly never the last case. And if you want to ship the Google Play Services to make it real, real Android, you have to pay some additional license fees. And the Android system itself is, as you might know, based on the Linux kernel with a completely new engineered user land from Google with a lot of abstraction layers, complete hardware abstraction layer, system services, and mostly the best of it, the app runtime. So yeah, it's a lot of layers. We don't want to target this here. But I want to say some few words about kernel. Android always had an issue with keeping the Linux kernel up to date, shipping the updates to the manufacturers, them shipping them to the actual phones. It was really a mess. It's not really maintainable the way it was. Currently, Android is pushing the generic kernel image approach. So the idea is having one kernel image shared between all phones, all Android devices with pretty well-defined interfaces for manufacturers to put their external kernel modules on. So the generic kernel image can be updated standalone and the dependencies are intended to not to break between kernel updates. But this is just a side note, as the most of you are probably rather interested in the kernel world. So let's go to the things Android is good at and what gives you really a plus when using Android. Let's start with a systems engineer's perspective. When using the AOSP as a base platform for your product, you get a really nice proven, stable, and really solid engineered open source platform, which is based on Linux. You get this really nice, secure runtime for untrusted apps on top. You have a really good graphics, video, and media support. I know the most of you are aware on the situation with media and graphics and all this in Linux. It's always a bit hairy. Android is much easier when using this. And the same for the connectivity stack. You get Wi-Fi, Bluetooth, NFC. All of this is even an issue in the desktop Linux systems. I think Bluetooth is a mess on my laptop for years. Android is, on this point, very well-tested. It runs mostly smoothly out of the box. That's really something nice to have. As said, you have the generic kernel image and lots of well-defined abstraction layers in all places at a kernel between kernel and the system as the hardware abstraction layers, between the system and the actual app side, between the apps. So the Android API, you get a real nice support for different target devices, such as different target device glasses, like phones, tablets, cars, watches, whatever Google is currently pushing. And you get some kind of power F and G and power optimization out of the box too. Yes, AOSP is big. It's not that optimized than you can do with a microcontroller and ultra low power, whatever. But for some things, the optimization is really better than when starting with a bare Linux embedded system. Then let's go further to the security perspective. On a system level, Android is already stuffed with a lot of things that are basically no cost for you when to start. SELinux is already down in place with pretty nice defaults. Yes, it's a mess if you have to work with it and if you want to integrate your own stuff. But that's always with SELinux. You have Asan and a lot of other sanitizers already in there. And enabling them for the build time is basically no cost. I think it's one configuration you have to change to use them. The invariity is already there and mostly enabled per default. Android is enforcing. You have a hardware key store. For example, trust zone or implementations of the arm trust zone in there. And you already get a nice APIs to work with them on top. And over the air update mechanism is already built in the system. No, the server is not. You have to provide your own server, keep your server up to date, maintaining it. And still have to test the updates. But the client side mechanism is already there and really bomb proof. And some rather not that cool parts. We do read only file system. There is a good engineering too. On the app level, you have the really, really nice app concept, which has a secure runtime for untrusted apps in different levels. So there are different levels. If you have a system app, if you have a vendor app, or if you have a third party or even instant apps, that's different glasses with different rides and different levels of isolation and security. And it's pretty well sandboxed. So if you really want the app concept, this is really one of the plus sides. And if you have a look on your app developers and users, they mostly really help you when you choose Android. On the user side, the app concept, your icon concept, is already known worldwide. So people mostly have no issues with actually using your apps on the device. And you have the whole world of Android developers that already know the base ecosystem with the standard APIs. There are lots of well-maintained and really good third party libraries and even the Android provided libraries. Most of them are really good documented in depth. And yes, you have a large ecosystem of libraries, already existing apps. And you have really a lot of Android app developers, much more than you have for QT, for other embedded graphics frameworks. But there are also downsides when using the AOSP. And let's go to the pain points of the AOSP itself. One is the code size. When you do a rare AOSP source code checkout, it's easy, around 200 gigabytes of source code only or more. If you start building it and have built intermediates lying around, one AOSP tree gets around 500 gigabytes really fast. So if you have one or more or three AOSP trees to maintain, you really need lots of SSD space. And you really want to have really potent build hardware. It's not fun to build the AOSP. It takes hours, really hours. And the same on the system requirements on the target side are high too. So if you have pretty limited hardware, the AOSP is may not for you. And when starting with the AOSP development from a system engineer's point of view, it's a mess. It's not really well documented. It's a really steep learning curve to navigate into the system with lots of layers and abstractions within the system. Google is constantly morphing around in the system beside the APIs. Nothing is left untouched mostly between the releases. The build system, it's a point itself. I think Chris can tell you more about later. But Google mostly likes to ship a new build system with new Android release. And this makes version updates rather painful. But I'm sure Chris will tell you more about this. And let's go further to the pains of Google, SOC vendors, and OEMs. One important thing even is the AOSP branded as open source. It's not really open as Linux is. You can download a source. You can work with it. You can build your own products based on it. But it's not really open to contribute. It's not completely impossible. But it's rather hard. And it's not the way you can influence the way AOSP evolves. It's completely controlled by Google. And yeah, if you want to brand your product as Android, you have to do this whole compatibility things and require a certification from Google. And the whole process with partner ecosystem license agreements is not really transparent and easy to go. And in general, SOC vendors, OEMs, and Google are not really interested in helping you if you don't do phones or really support the devices they want to push. None of them is interested in selling rather low or medium quantity products. If you want help for your use case, don't expect anything of them helping you. And that's the last point on this slide. But I think even more on it is the BSPs. If you want to build your own product based on AOSP, you get a BSP from an SOC vendor. And the quality is really varying between the vendors. It's not fun every time. But I have a bit more on this later. Let's have a look at the general downsides first. Google is focusing on certain types of devices. Currently, it's mostly phones, tablets, a bit rear and cars. Everything else, if you have a headless use case, it's not really supported in the Android system. So may you have to find hacks around if you want to build a product with a headless use case. As said, it's intended for large volume products. And especially if you want to build your own ecosystem, then it's nice if not that much. If you want to do OTAs, what's pretty much recommended, you need to run your own server and manage it. And in general, you have a large number of app developers, but really view AOSP system engineers available. And mostly when building a product, you need lots of them. So that's a bit a minus on this. But yeah, when is the AOSP nevertheless a good choice? And why not? I would recommend you using the AOSP if you want to build a whole ecosystem. For example, you want support various devices, maybe from different vendors. If you want to run on different hardware targets, different classes of devices, for example, phones, cars, whatever. If you want to make usage of the really nice app concept and really want to instrument this a lot, then AOSP is really nice. If you make a clear advantage of the UI concept, it's cool. The same for the media framework. If you have a use case that uses extensively media communication and so on, then Android is may worth considering for you. And a bit not that decisive point if you need a GPL-free system layer. But I won't decide for Android just because of the GPL part. And on the other side, don't use the AOSP if your use case has not a clear advantage of using it. So for example, if you don't care about graphics and media at all and the app concept is not needed for you because you have only one app running, then AOSP is pretty much not the right thing for you to use. If you're really in a performance-sensitive context, if you may have to run even real-time tasks, the AOSP is not for you. It's just not manufactured for this. If you have a really small use case, for example, only one or two apps running, the overhead you get with the AOSP is much too much for the things you get on the cost side. And if you don't have the person power and the number of engineers really well-trained with AOSP system engineering, I would not recommend you to start a project with AOSP. Because there's no really support. There are not much forums. There is no general community support there. But if that's not the case for you, you may can consider using the AOSP. And when you want to get started with it, I have some general tips and tricks for you. The first one, when starting, I think it's pretty much the same with Linux. Use the source, read the source. There are things that may help you, for example, code search engines like codesearch.android.com. In the past, there were even more third-party code search engines, but I think they're all gone now. But code search is really nice and fine to work with. Besides, you can generate an Android Studio project out of the AOSP sources. I've listed you how to do it. For me, it always worked pretty much fine. A colleague just helped me. It did not work for him at all. It was a bit strange, but I had no time to investigate in this. And there's some pro points. If you use the AOSP tree in Android Studio, you get the general features of Android Studio on top. Mostly, they're intended for Java debugging and Java development, but the most of it is available for native sources as well. For example, it helps you with type information, find usages, syntax highlighting, and auto-completion. It's really comfortable if you want to execute tests in the Android Studio project. For example, if you have Gradle build files for your tests too, then it's possible to run the tests directly in the IDE. That's really nice and sweet. And if you have the AOSP source imported as a project in the IDE, you are able to attach the debugger directly from Android Studio to the system process, which makes debugging really nice. But there's no general support for building the AOSP from inside the AOSP. But for the rest, it's really helpful. If you want to start, you need to select some points on board to start with. Mostly, the question is between starting with a phone, tablet, a deaf board, or the emulator. Mostly, I would recommend starting with a phone from Google because they are available. They are not really cheap, but well supported. Using the emulator is pretty much cheap, also available. But depending on what you want to do, it does not behave exactly like real hardware. But for most cases, it's fine too. Deaf boards is mostly an issue. Currently, most of them are not available anymore. Sometimes, even the ones Google pushes and supports are out of sync in the tree and not really well maintained anymore. In general, the availability is still an issue. If you want to use a reference board from a third party OEM, you have also available ability issues. And you have to deal with the OEM things too. OEMs tend to make their BSP a bit different. They have hacks in the build system and so on. So it's always a bit of a mess to deal with vendor-branded AOSP BSPs. I have a list on this. If you want to start, I mostly would recommend starting with the vanilla AOSP supported by Google. It's the standard way of working. It works as documented. All versions are, in general, available. But on the downside, it's not really suitable to build products with it because the hardware Google supports in there is in common not available for you as a third-party vendor to build your products with it. And the deaf boards supported are not a good option for production. But it's great to start with. When dealing with vendor-branded BSPs, the BSPs are supported by the vendor. They're brought to you by the vendor. They're often filled with strange build scripts and work completely different than the standard. Sometimes they're really slow. Maybe Chris says a bit on this vendor build things too. I'm not sure. But this is the only option, the only real option if you want to build your own products. Then you have to deal with this side. And in this part, I want to mention this really nice graphic from Karim Jagmur, which makes a bit the point where you placed in all of this mess. So as a third-party people not working at a BSP vendor or at Google, you're completely outside. And you have to deal with all the mess you get. On the tools and platform and workflow side, I really recommend you using Garrett Code Review. Of course, it's possible to build the AOSP with GitLab, GitHub, whatever in background. But as it's a multi-repo project, it really tends to get a mess. Garrett is intended to handle multi-repo projects. It enforces a really clear way of working and forces well-formatted, nice commit messages. It has a proper handling of cross-repo commits. So yes, it's really ugly. It's hard to learn to work with Garrett. But please use it. It makes your life pretty much easier. Then, please use continuous integration container setup. What you use is, in general, not that important than using it. Container are needed because the AOSP tends to behave a bit weird on different platforms, different host systems. And you make it much easier for your developer if you have a container setup ready for them to start. And I think we don't have to talk about why needing a CI anymore. It reduces risks at merging in such a large project and AOSP a lot. But I want to tell you what is, from my point of view, really needed in your CI pipeline. One thing is billing for all the targets you support. The next is check for the code style. If you allow different code styles in such a big project, it's get a mess. And the next point, the most important one, is testing and a lot of tests. If you can do it, have tests on real hardware integrated in your CI pipeline. If not, use virtual or emulated devices. Use the available test suits in Android. Enhance it, do your own ones, and in general, write lots of tests, really lots of tests. And as a last point, Android has a real nice emulator supported to Android virtual devices. It's based on QEMU. And it's really easy to make a emulator configuration that matches your real device as good as possible. It allows you much faster round trip times in development. It's really easy to handle within the CI's. You can switch around with Android API and general Android versions much, much easier than when working with real hardware. And it's portable. It's home office friendly. You don't have to put your five dev boards with you every time. And it's really nice if you don't get enough real test hardware from your clients. So you have to share the hardware between developers. So parts of it can use the emulator. In general, AVDs do not replace the need for testing and developing on real hardware. But they can help you a bit and speed up the development. So thank you. Time for questions. Any questions? Thank you for the talk. Do you have any experience with using Linear.js as a base for getting started with Android development or maybe porting it to supporting new devices? What? I didn't understand the name of it. Linear.js. Lineage. Yeah. No, we currently did not start with Linear.js. But I think as it's based on the ASP2, it's not much different. Yeah, you mentioned Java in your talk about which development tools are available and integration and so on. But I had the, well, I developed a lot for Android in the Android 3, Android 4 days. But these days, I hear that Kotlin is very important. So how well is Kotlin supported in the Android Studio and tooling and so on? Kotlin and Channel is very well supported. But on the system side, Kotlin does not really play any role currently. The system side is mostly Java still yet. If there are any questions? Hi, Anna. So talking about emulators, you were talking about the AVDs and the Goldfish emulator. Do you have an experience with Cuttlefish and CVDs? The person next to you knows about it. So the follow-on from that is that in my experience, the CVDs are better for CI than AVDs. That's what they're designed for. I think I personally used the AVDs more. But we are experiences with Cuttlefish too. Just ask Florian. OK, thank you, Anna, for the talk. And yeah.