 Alright, well thank you guys all for coming out. I hope you're here to hear about some embedded Linux build systems. If you're not, you're in the wrong room, or maybe I am, I'm not sure, but I do appreciate everybody coming out. I know there's a lot of crossover, a lot of interesting talks here, and I do appreciate you guys coming to hear what I may have to say either rightly or wrongly about these things. So as I said, we're here to talk about some embedded Linux build systems and distros and various options for getting software up and running on an embedded Linux device. Things have changed quite a bit in the last few years in this space, especially due to systems like the Raspberry Pi, making it a lot easier to get in and get started for low effort and low dollars, so there's a lot more options out there than there used to be. So I'll start with just a brief overview of what we're going to discuss here. There we go, are we back online out there? Sweet. Alright, so I want to start with just a little bit of the challenges of embedded Linux. I suspect the majority of the folks in this audience are well familiar with them, but I figured it was worth a review of some of the things we face as embedded Linux developers that are different than your traditional desktop or web developer in the issues that they will be facing. Define what a build system is from my perspective and just some criteria for evaluating them for use in your particular system design. Discuss a few popular options. Obviously, I'm sure we all have our own favorites here. In my current role, I'm actually kind of stepping back and talking to more customers and seeing a lot more questions about systems I knew nothing about. So ultimately, this talk was really to give me an opportunity to learn about some of these build systems and I figured I'd share what I learned with as many people as possible. Ultimately the goal is to help new embedded Linux developers get started. I'm not here to teach you guys how to use Yachto or how to use BuildRoot or how to be an expert in any of these systems. Obviously in 45 minute talk, that's quite a tall task, especially if I'm going to be talking about more than one of the systems. My goal really here is to give a beginner's view of some of these systems and like I say, things you might want to consider when you're designing your system and deciding what build system you want to use for any new upcoming designs you may have. So just briefly about me, my name is Drew Mosley. I'm currently working on a project called Mender.io. We do have a booth over in the sponsor area. It's an over-the-air updater for Embedded Linux. It's an all-open source, dual-AB root file system, full-row busness, that kind of thing. I've been in the Embedded Linux and Yachto space for about 10 years and I've been in general embedded for even longer than that. As I said, my current role is a technical solutions architect. So I'm actually out talking to customers, figuring out what they want to do. I have a long history of working on the Yachto project, but in my current role I do get people asking about other systems. So thus the impetus for this talk to kind of figure out some of the new systems that are rather the other systems, not necessarily new systems, but new to me systems that are out there and when they might be useful and how I can make better recommendations to our potential users. So briefly, let's talk about some of the challenges Embedded Linux developers face. Obviously, the variety of hardware that we have to deal with is quite a bit broader than most folks. If you're doing web development software, you're pretty much isolated from the hardware. You're worried about software APIs and that kind of thing. But in the embedded Linux space, the variety of hardware you will deal with on any given project can change quite a bit from previous projects. Storage media tends to be a pretty big issue for most of our users. Some systems have SDMMC, some systems have USB mass storage, some systems have UBI. And each of those has their own challenges and ways of dealing with them. So embedded Linux developers typically have to deal with those kind of things. Additionally, a lot of the drivers and board support packages for the various pieces of hardware we're using in the embedded space may not be in the main line. They may be developed in their own forks, maintained by the semiconductor vendors or various other places like that. So gathering all the right components for any particular platform can be a challenge. And for those coming from the non-embedded world, cross-development kind of takes some getting used to. The idea of developing code on one machine, running it on another, having to have some kind of connection to the machine, that kind of thing. That does present some challenges for folks moving into the space. And as I said, with a lot of the customers we're seeing now, we are getting a lot of new embedded Linux users coming into the space, say from the Node.js communities and that kind of thing, doing IoT devices. So a lot of these things are very relevant to folks that are new to this space. And the initial device provisioning, and what I mean by that, is how do you get that initial image on the device? That can be tricky. Every device is different. Some of them are simple. If you're lucky, it's an SD card. You can take out, plug it into another machine, and write the image to the SD card. If you have a raw flash-based device, there may be some kind of, you know, you've got to get a short two pins together to put it into a certain mode. So those are all going to be very device-specific. And those kind of things do tend to slow down. You're getting started in this space. So just a few facts, embedded Linux systems are very large. In case you were unaware, there are typically, in any of these systems, anywhere from hundreds of thousands of packages that you'll need to download, configure, build, and get up and running on your system. Dependency Hell, unfortunately, really is a thing. I actually Googled it, and it's a very big meme. There's lots of interesting photos. I didn't, I decided not to actually include any of them there, but getting the right combination of packages and versions for any particular configuration can be tricky. I actually, at one point, worked on an embedded Linux project where the entire build system was, I think it was like a 10,000 line make file and a 15,000 line shell script. So those kind of things don't scale very well. So again, we're talking about embedded Linux systems. Working with simple make files just doesn't cut it anymore. There's a number of good options to help you get started without having to deal with a lot of this complexity yourself. In general, these builds take a long time and take a lot of resources. If you've ever tried to build, especially something like QT, cross-build that for an embedded device of some kind, it gets hairy. It takes a lot of disk space, takes a lot of build time, a lot of coffee breaks while you're waiting for things to finish. Well, sure, we can do that too. Sword fighting breaks, coffee breaks, whatever. Overnight breaks, I can't tell you how many times I start the builds and go to bed because it's just not worth waiting for, especially when you're building from scratch in some of these cases. Some other things to consider. In general, when you're talking about embedded applications and use of these systems in the embedded space, they do require significant customization. It's rare that you're going to take a system and use it completely out of the box and have it have everything you need for your particular application. You're going to need the ability to customize these systems. And so any of these build systems that you look at, they're going to have to provide some means for you to customize these systems as you move forward in your design. And similarly, a lot of the systems will provide defaults and some of the systems make it more difficult to change those defaults, which may not be appropriate for your environment. So that's another thing to consider is how well your requirements map into the defaults of whatever build system you have chosen. So what exactly do I mean when I'm talking about a build system? So what a build system is, it's a mechanism to specify and build a full embedded Linux system. So it gives you the ability to define your hardware, whichever hardware platform you're working on, whichever SOC you might have on your board, that kind of thing. Gives you the ability to integrate user space applications, either yours or someone else's. Generally speaking, most of these build systems, they will bring in a lot of user space applications just pulled down directly from their canonical upstream location. And then they will also give you the ability to include your own custom applications in scripting and that kind of thing. A couple of the things that are part of a build system, ideally you've got the ability to reproduce your build. A lot of times you see people, especially in the hobbyist maker space, they'll log into the system remotely, they'll install the packages using apt-get or yum or whatever, and then they'll actually, once they get the system of the way they want it, they'll actually copy the system off as a binary and that's their means of reproducibility. And that's fine for certain environments, but I don't recommend it if you're going to try to go into production and get people to pay you for devices and you're gonna be deploying large numbers of devices. So you need to take a look at your build system and figure out, okay, how am I gonna reproduce this build? How am I gonna make the same build and know exactly what's getting installed on my devices every time? They also need to have some support for multiple developers. Again, if you install on an SD card, log into the system remotely and try to configure all your files in slash Etsy and install all your additional packages, that doesn't scale real well as your development team scales up. So you need some means of doing the development away from the board so that you can have your larger development teams contribute. Ideally, they allow for parallel processing. As I mentioned in the previous slide, these systems are huge, the builds take forever. You've got to be able to do builds in parallel as much as possible. Cross-tool chains generally should be considered part of your build system. If you've ever tried to build a tool chain yourself, you're probably familiar with the number of additional requirements that that brings in, getting all the components matched up. So there are tools that can help with that and I've got a link for one or two of them later in the slides, but ideally the build system you choose will take care of the tool chain development for you. And another big thing that a lot of people miss, especially early on, is license management. If you're just doing a hobbyist project at home, that's usually not a big deal, but if you're working, say, in the automotive space, I know they have some very strict requirements as far as licenses that they allow in their code. So ideally, whatever system you choose to develop your binary image for your target will give you some ability to track the licenses that are involved and be able to exclude and include specific licenses and throw errors if you try to include the wrong thing. So real briefly, a couple things that typically people think about when they're thinking about build system, but from my perspective, I don't consider that these components are part of a build system. It's not an IDE. These build systems that we're discussing today are not really built for iterative development. Most of the development, say, if you're doing kernel porting or whatever, you're not gonna be doing your hardcore kernel development within the Yachto build system or the build root system. The turnaround time is just too long to go from a source code change to binaries that actually run on your target. So you might use one of these build systems to provision the system initially, but then you'll typically go for your actual application development, you'll do some work outside of it in whatever mechanism is appropriate for you, ideally using the tool chains that are built by the distribution and the libraries and that kind of thing. It's also not generally considered a distribution, although the Raspbian view of the world is kind of changing that in some of the hobbyist spaces. A lot of them are using things like Raspbian and Debian directly out of the box, but distribution is generally something that's kind of layered on top of some of the build systems, although more and more of customers and users are looking to use things like standard desktop distributions in their target devices. It's also not typically a deployment and provisioning tool. So the output of these build systems is generally a build image of some kind, but they generally don't include direct mechanisms to deploy to the system. There's just too many variety of ways that you actually have to get the images deployed to your system. As I mentioned earlier, you might, if you're lucky, you've got an SD card that makes it simple. If you're using something with raw flash, there's generally some kind of USB interface where it comes back into your system and shows up as a block device on your development system. So they provide, typically these systems will provide tools to assist in this, but generally speaking, you're gonna have to go back to the manufacturer or the board to figure out exactly how to get the images from your build system into your deployed devices. And generally they're not considered out of the box solutions. It's most of these you're not gonna take, run a build, install it, and have it have everything you want on it. If you do great, you're ahead of the game, but generally speaking, you're gonna want to customize these things quite a bit. You'll add your own custom code, you'll add your own custom drivers, that kind of thing. So let's jump into a couple of the big names in the space. Yachto project, I suspect most of you have at least heard of it. If you haven't, they've got a booth over in the sponsor showcase. I encourage you to take a look. As I said at the beginning, I've spent quite a few years in the Yachto space. So this is the one system that I actually know the best. You can see the quote at the very top. That comes directly from the Yachto project website. It's not an embedded Linux distribution. It creates a custom one for you. So the way Yachto is structured, it consists of recipes, metadata, dependencies and configuration. Recipes are basically just that. They're sets of instructions that tell the Yachto build system how to download, build, compose, and everything else you need to do for all the packages that go into your build system. The primary output of a Yachto build is a package feed. And what that means is a directory somewhere on your disk with all the packages that are built as part of your configuration. Typically that'll be in RPM format or Debian format or IPK format. And then the secondary output is the actual boot images. So with Yachto, you can actually install the package management utilities directly onto the target and still do things like installing packages dynamically at runtime using AppGet or IPK or things like that. It does generally build all the components from source although when you're working with some of the, say the semiconductor manufacturers, sometimes there are binary blobs that need to get downloaded for various components in the GPU and that kind of thing. And its focus really is more mechanism than policy. There are defaults in the Yachto system. For instance, I believe right now the Yachto default is to use sysvnit. I know system D is becoming more widely used. It wouldn't surprise me if their default changes over time. However, the Yachto project is generally pretty good about allowing you to substitute your own policy on top of the bare mechanisms that are provided by the Yachto system. So you got a list of the products there from a standard Yachto build. It's a root file system image of some kind. It'll be an EXT3 or 4 image perhaps. It could be a tar ball if you want to deploy your root file system over NFS. It could be a UBI image. It just all depends on your board configuration and how you want to use the files. The kernel, the boot loader and the toolchain are obviously all included as well. Those are typically stored outside of the root file system image although for some of the deployment models, for instance the kernel and the device tree could potentially actually be inside the root file system image and loaded from the boot loader there. And as I mentioned, the package feed. That's the primary output. And you see the link there. If you haven't gone over to the yachtoproject.org website I encourage you to do so. Go talk to the folks at the booth and there's a lot of people with a lot of knowledge about Yachto. So just going into just enough detail to kind of give you a flavor here. One of the strengths of the Yachto project is its layering mechanism. That allows individuals that are not directly in the Yachto project to produce their own layers with their own custom functionality. So for instance you've got the stock layers from the Yachto project that you can download from their GitHub. And then for most platforms you will actually get an external layer that adds the board support for that particular platform. So for instance if you're looking at the Raspberry Pi platform and you want to run Yachto on that. If you go to yachtoproject.org and download directly from them you won't have the kernel and the board support package as necessary. You actually have to go get the Meta Raspberry Pi layer. Advantages of that are it allows the systems to develop at their own rates. The Yachto project has their own release cadence. The Meta Raspberry Pi layer has its own release cadence and everything can be maintained separately. It's both a blessing and a curse because getting started that first time can be a little tricky if you're not aware of this. And the recipes in general in Yachto they're developed in Python and Bash. So I would imagine most of us are reasonably familiar. There's snippets of code in the recipes when for instance if you want to create a custom compilation recipe it's generally done up in Bash or in Python. So it's fairly easy to expand and store your code in a separate layer. I would also encourage you when you're doing your system design for one of the first steps you need to do if you're using Yachto is to create your own custom layer. That's where your customizations will go. You might use the upstream images from the Yachto project but you'll want to add certain packages and that kind of thing. You're better off adding those in a custom layer than keeping local patches against the upstream Yachto project. It just makes it easier for separation. The Yachto project can move forward and it makes it easier when it's time for you to move to a new version of the project. If your code is completely separated then you can have that code develop at its own pace. The Yachto project also has a very integrated SDK mechanism and the idea with this is that it's easy to separate between your system developers and your application developers. So your system developers are the ones that are running the full BitBake builds, generating the images from scratch and producing the actual target binary images. The application developers may not need that level of detail into the system. They're developing applications in whatever language is appropriate so you actually export an SDK from the system developer and that consists essentially of the library's header files and that kind of thing that the application developers will need to link their applications with to be able to work with a smaller subset of the system. And this is, as I mentioned, is one of the easy mechanisms for allowing multiple developers to contribute. Some optimizations that the Yachto project provides. If you've done a build previously the pre-built binaries will be stored in a mechanism called the shared state cache which allows for reuse of those binaries if the components going in to generate them have not changed in subsequent builds. So it really gives you the ability to reuse these components that haven't changed from build to build. And parallel builds are very well integrated into the Yachto build system. So the Yachto build system will take as many cores and as much memory as you can throw at it and it will parallelize as much as possible. A previous talk I saw from ELC, I think about a year and a half, two years ago, estimated there were about 8,400 software packages available as documented in the Yachto metadata index. That doesn't include layers that are not necessarily indexed by the Yachto project so getting an exact number is hard to pin down. In that 8,400 there's probably quite a bit of overlap with different versions of the compilers and QT and things like that. Just to give you an idea, at one point it was somewhere around 8,000 to 9,000 packages available. So what I've got on the screen here is the minimum set of steps to get up and running with a Yachto project build. What this will do, obviously it'll check out the Rocco version of the current Yachto project. Within that Pocky repository, there's some emulated platforms and I believe a Beaglebone platform and maybe one or two other platforms that have the platform support in that layer. So this set of instructions here will actually build an x86 emulated platform with the minimal image, core image minimal you see there is one of the image recipes provided by the Pocky repository there and it also provides the scripting to actually easily run that emulated platform on your build system. So in this case you see the image here that actually shows the Yachto system booted up to shell prompt. At this point it's obviously emulated you're gonna wanna expand out from there but with this set of four commands you can actually get a Yachto system up and running to start nosing around with it and get a feel for how this system might be useful in your next design. So just in summary about Yachto, it is very widely supported by board and semiconductor vendors. A lot of the manufacturers out there actually maintain their own layers with the board support for their particular platforms. So if you are on a particular platform you definitely wanna check what is provided by your manufacturer. It has an extremely active developer community, the mailing lists get a significant amount of traffic. I know there's an active IRC channel, pretty standard open source community model. Wide, lots of functionality enabled by that layering mechanism. There's a lot of platforms that can be added outside of the main community. There's a lot of functionality, application layer things that can be added outside of the main community. So that allows for a very widely diverse set of functionality. And it's extremely customizable and expandable. And one of the nice things about it is generally speaking, they try to avoid putting too many requirements on your host system. So where possible, the Yachto project will actually build appropriate versions of the native tooling that are required for that build, which allows it to easily be supported on a wider variety of host operating systems. Couple of downsides, it does in general have a steep learning curve. It's easy to get started quickly, but to be functional and understand all the ins and outs, especially when you're adding additional layers, it can take a while to get used to that. It's an unfamiliar environment to not embed the developers. And that's really true of all these systems we're discussing today. And generally speaking, it's pretty resource intensive, especially the first time you try to do a build for a new system. If you're putting a lot of target code on the target, the initial build times can be pretty significant and the amount of disk can be pretty significant as well. Mitigation for that, of course, is that shared state mechanism that I mentioned. And if you have a central build system, of course, you can actually pull from that shared state from your CI system to any of your developer build systems. So there's a lot of mitigations to this, but those are some things to be aware of. So moving on, build root is probably the other, the next biggest one that you'll see around here. I know the fact they got stickers up here, I guess somebody left it here when they were given a talk the other day. You see the quote across the top. Again, that was pulled directly from their website. The primary output of build root is boot images. They don't generally support the RPM style, package management, although things are changing all the time. So it's possible that has changed since the last time I dug deeply into it. I know I've already received at least one email correcting a couple of things I had in here earlier. So it's because it doesn't generally consider packages outside of the boot images that you're building, it's sometimes referred to as a firmware generator just to kind of give you an idea of how it's used as opposed to something like Yachto, which is providing a little bit more functionality at the package level. Again, similar to Yachto, it builds all the components from source and the focus really is on simplicity. I'm new to build root, but I was actually able to get systems up and running pretty simply. So the focus on simplicity really is an advantage in terms of the speed of getting started. The main products, of course, are the root file system image and the kernel bootloader and tool chain. Very similar to Yachto, you're gonna have very, it's trying to do a lot of the same thing. So obviously the products that you're gonna get out of the build system are gonna be very similar. Some of the details, it uses the make files and K config. Obviously that's very widely supported, well-known if you've ever done any work in Busybox or in U-boot or in the kernel. You've seen screenshots like the one on the slide here. By default, the default images are intended to be very small. Where possible, configuration options are disabled in the default configuration. So if you just come in and run one of the standard def configs, most of the configuration options for the various packages that are built as part of the build will have been disabled by default. They do have a BR2 external mechanism and it's similar in functionality to the layering mechanism in Yachto. It's a way for you to store your local additions outside of the main build root source tree. However, unless things have changed, it's really one place to store it. So I can't store a layer and then have someone else store a layer and be able to pull them all in. There's the build root main sources and then there's the BR2 external sources which would be my local configurations and customizations. As I mentioned, in this case, recipes are typically defined in the k-config and make syntax, so it's fairly familiar for I'm sure a lot of us and there's lots of documentation out of it. They did recently add an SDK mechanism similar to what the Yachto project has where the application and system developers are able to work in different environments that are more appropriate to them and a previous ELC talk estimated about 1800 software packages. Again, that number is probably low now but since there are things that are generally not included like the on target tool chains, the total number of packages that are available for build root is gonna be typically lower than you will see in the Yachto project. So similar to the slide I had for Yachto, here's our getting started for build root. It's just a couple of commands. Pulls the latest version, the 2018-02 version directly out of their get tree, runs an emulated def config and launches it. In this case, it actually launches two windows. One is the frame buffer window and one is just the serial console window. But in a similar number of commands, you can get an emulated platform up and running that will let you get started and play a little bit with build root. So summary, and I put the first bullet in both the pros and cons. There seems to be less corporate involvement with build root than with Yachto. That may just be the bias of the users and people that I talk with. But that can either be a pro or a con depending on your point of view. So I've listed it in both places and I'll let you pick which side makes sense for the way you feel about things. It is very quick to get started. That's one of the primary design goals. And it's easy to understand with, it does have an active development community and fairly broad architecture and board support. And I know they're expanding all the time. Configuration changes. If you're trying to switch from one board to another, generally that's gonna require a full rebuild. They don't have quite the number of places for storing configuration information that they do in Yachto. So typically if you have, for instance, you wanna run the same basic image on two different platforms. There's some manual syncing to make sure you get the same def config between those two platforms. So with Yachto, it's separated out a little bit more. You can kind of pull from different areas with build root. It's a little bit more manual if you are building multiple platforms and want the same basic configuration. There doesn't appear to be any reusable shared state by default, but there are some settings in the K config that seem to enable that. I haven't had a chance to play with it. I see some faces around here that probably know a lot more about that than I do and can certainly speak to that if there are questions about it. So moving on, OpenWRT, you see the quote at the top from them. It's a fully writable file system with package management. If you're not familiar with it, it basically started as replacement firmware for the venerable Linksys WRT54G router. That router has been around for, I don't know how many years now and is no longer being maintained, obviously, by the manufacturer. People wanted to continue using it, so they started developing their own custom firmware. It's primary focus of this system is on networking. The number of devices that they have custom firmware for is pretty astounding. I didn't realize there were that many consumer networking devices out there, but there's quite a few if you scroll through their list of devices. Unlike build root and Yachto, OpenWRT seems to be primarily focused on the binary distribution side. They do have a build system, but it's very much, if you go to OpenWRT.org, the focus there is really on downloading the binaries and deploying them to your system. So they provide pre-built binaries for a number of systems, and expandability is provided in OpenWRT through on-device package management, very similar to what you would see in a desktop type distribution. The products of an OpenWRT system are the firmware image, typically in a device-specific format. I have seen some of the makerboards coming out where the default operating system for them is an OpenWRT, so I know they're definitely adding functionality. But also, generally speaking, for a number of the consumer boards that they provide images for, there are package repositories available that you could actually download directly to over the internet. So if you install OpenWRT, for instance, on an old router you happen to have at home, the package repositories are there if you wanna install something like OpenVPN or some of the other packages that are not installed by default. So as I mentioned, there is a build system. Unfortunately, I didn't get a whole lot of time to play with it. This is one of the areas where my content is a bit thin here. So if anybody has any experience with it, I'll be over at my booth all day today and tomorrow I'd love to pick your brain about it. I know it's consistent to make files and patches. It feels very similar to build root in that respect. It does obviously generate a cross-tool chain root file system image, uses K config. There's a link for it there. As I said, its goal is to do very similar work to what both build root and Yachto do. So I would imagine it's going to ultimately look very familiar to anyone that's familiar with any of the other systems. So just a summary on OpenWRT, it is a great choice as a replacement firmware if you happen to have a consumer device at home that may no longer be supported by the manufacturer but you don't wanna necessarily put it in the garbage. Go check out their website. They may have a pre-built binary image that is maintained and has quite a bit newer code than what you have on the device from the manufacturer. It is a good choice for any kind of router networking device. I have had in my role customer facing now, we do have folks ask about OpenWRT typically for router devices and that kind of thing. And if your application does require package-based updates, this is potentially a good choice for you as well. Some potential cons. It seems less flexible for general embedded applications, although I'm willing to be convinced otherwise on that. As I said, this is the one that I'm just starting to dig into. And it seems to have a bit more policy imposed by the OpenWRT design. It doesn't seem that you can easily customize it as much as you can with the other options we've discussed. And if you are using package-based updates with a large device fleet, that can lead to problems down the road just due to version explosion. If you don't know exactly what's on, say, you have a fleet of 10,000 to 15,000 devices, it can be a little bit difficult if they're all updating at the package level on their own schedule. So package-based updates, you might wanna rethink that depending on your use case. But again, that's gonna be very specific to your particular application. So moving on, as I mentioned, the desktop distributions have seen kind of a more use in the desktop space lately. So why can't I use my favorite distro in the embedded space? And the answer really is that you can with some caveats. If you've used a Raspberry Pi, they come with a Debian variant on it by default. The Beaglebone comes with, I believe it's Debian by default. So certainly these packages are out there. The basic workflow is you take the installer from the whatever favorite distro you have and you install it on an SD card and you pop the SD card in the device. And if you need to add additional packages, you use on-target package management to install the packages, that kind of thing. It does impose more policy than some of the build systems we've discussed in potentially significant amounts of policy. Each of the distributions has their own set of defaults that may or may not be applicable for your particular use case. You're dependent on the vendor distro generally for upgrades and the decisions they make as far as what the defaults are gonna be, whether they're using Wayland or Xorg or SystemD or Sysbian it. So that's definitely something to consider if you're considering this approach. Most of these device desktop distros, they're not targeted embedded applications. So just in general, they may not be the best choice. They certainly are easy to get started with but they're not very cross development friendly typically. I've seen a lot of people trying to run the tool chains actually on the devices and while that works, in the short run, I think in the long run it tends to be pretty painful to actually do on-target development for some of these smaller device systems. So just to summarize that, there are a lot of choices to start with. You saw all the images I had on the original slide. There's Debian, Ubuntu, Raspbian, Arch, there's plenty of choices. Your developers are likely already familiar with many of these systems. So that definitely helps you get up started quickly with that. Typically there's a large selection of pre-built packages for these things and it's quick and easy to get started and if you need on-target development that's generally possible with these systems. The decisions by the vendor may affect you negatively if they make a decision that you don't like. Usually they'll give you some means to undo that but it may or may not be very easy to do so. Reproducibility is complicated in this environment. Typically if you're installing packages by hand on the target it's hard to come up with a means to produce that image from scratch. On-target images are gonna be slow and then off-target builds may be difficult or impossible especially in the arm space getting a cross-tool chain that can build for the particular variant you may happen to have can be difficult. So with that I'll move into just a couple bits of other criteria that you might consider when you're deciding on a system for putting your embedded Linux system together. If you have already chosen your hardware and the hardware vendor provides support for one or more of the above systems that may drive your decision for you. Frankly that's I think probably the case for a lot of systems is you just, the vendor prides it and it's the path of least resistance to start with that. Training and documentation depending on how familiar your developers may be, all the systems we discussed today there's lots of documentation, some of them have more or less training available and if you do want to use a commercial vendor to provide your base support that can also help drive your decision. And frankly whatever developer experience or whatever experience your developers may have and preferences they may have for build systems to use. So just real quick I'll mention just a couple of related tools that you may come across in your research if you're new to the space. UCLinux is a port of Linux to systems without a memory management unit, at least that's the way it started. I haven't kept up with it over the years. According to their website they're still back at kernel 2.6 but I haven't dug in it too deeply to know. But they do provide user applications, libraries and tool chains. So if you're just getting started in the space that may be something you come across. And also CrosstoolNG, which is a Kconfig-based configuration utility for actually generating Crosstool chains. So this is actually useful if you need to generate tool chains for bare metal, if you need to generate tool chains that go to various RTOS's in a bit of Linux. So that can be handy as well if you do have say a Debian and you want to get a Crosstool chain built up from source for it. So and I just wanted to mention a couple others that I came across. I've got the links here. I know nothing about them other than what I've seen. I've used Android on a phone obviously. I've not had any experience with it in the embedded space but it's obviously becoming more common there. ELBE and ESR, as far as I know they're both systems around the Debian pre-built packages to allow you to configure a custom system without having to build everything from source. So basically if I'm understanding them correctly, they actually go pull the binary packages from the Debian package repositories and kind of repackage them into the root file system image. So those are some other options. There's lots of things going on in this space. Things are changing all the time. So definitely keep your eye out, things will change. I'd like to eventually add a little bit of more content on some of those other options to this talk and maybe at future conferences I'll be able to give a little bit more details on some of those other systems. So I just want to, we've got just a few minutes left. I just want to kind of, these are some use cases that I put together. These are my thoughts. Your mileage may vary. If you're a beginner, hobbyist maker, the commercial debt and you're using something like a Raspberry Pi that you can get easily, something like a desktop distro might be a good choice. If you look at some of the maker blogs out there, most of them start with Debian and work from there. If you're in commercial use with a single configuration, build route is a good option. If you're in commercial use in multiple configurations, Yachto Project might be a better choice just because of the way things are structured and allow you to easily layer things on top of each other. Again, there's so many decisions that go into anyone, any one application I don't want to presume to tell you exactly which one is right. There's no right answer for everybody. These are just some general ideas that I threw together. And I'll kind of leave this slide up there and kind of open it up for questions. This is, again, my thoughts on the kind of the matrix of things you might want to consider as you move forward with your system design. So with that, I think we've got about 10 minutes for questions, so I will open the floor, yes. Okay, so the question is, a lot of these commercial boards today come with ready-made Yachto support provided by the vendor, typically. And is there any easy way to import that information into build route? Is that basically your question? Not that I'm aware of, but that would certainly be nice. I know the recipes are very different, so there may be some tools that can help with that, but I doubt there's any real explicit plug-and-play mechanism to just take the BSP out of Yachto and drop it into build route. Yes. Yeah, so the question is, how does this plug into DevOps environments and is this appropriate for your build systems and that kind of thing? It depends obviously on your workflow. My current company, we do, our CI system uses Yachto builds because that's how we deploy our system by default. It certainly can plug in easily into things like Jenkins and other systems like that. And I believe there are actually extensions for Jenkins and various other CI systems to make it easy to do things like this. I haven't really spent a whole lot of time on that side, but anything that allows you to run a shell script, you can actually run pretty much any of these build systems. Yes. Have I looked at Gen2? No, I have not. I know on the desktop side, Gen2 is basically a build from scratch desktop operating system, right? Yeah, I have not. Do they have decent support for non-X86 architectures and cross-development and that kind of thing? Okay, great. Well, I'll add it to the list that'll make this talk even longer. And I know the SUSA guys are here and the fact that they supported ARM was news to me. So as I said, this thing is changing on a daily basis. So, yes. Okay, so the question is, how do you get more desktop style user land with something like a Yachto or a Buildroot which is a build from source type kit? Well, they all, you know, all those systems will have some level of user space applications and they can all import other applications. You just have to take the time to write the recipes to do them. I don't know how many people in the embedded space would want a full Ubuntu GNOME interface on their target device. There may be a layer for it out there already because there is a lot of different layers out there that people have used. So it wouldn't surprise me if something like that exists. Yes. Okay, so the question is, are there any web interfaces to the build system? I haven't come across time to send in many years so I can't comment specifically on theirs. I know the Yachto project has a project called Toaster which is basically a web front end. That's the only one I'm familiar with. I launched it once as part of a previous job and that's as far as I've gotten with it but I know it's under active development. So the question is there a preference on CPU architecture for individual build systems? And certainly there is. I think they all have a, you know, RMX86 MIPS are generally supported across the board. I know Buildroot has some other more esoteric architectures that may or may not have Yachto support. I don't know. I tend to focus on the more common ones that are supported by everything. So certainly that's one of the first criteria you'll need to look at. If you have something that's not a commonly used platform you'll definitely want to investigate which systems may support your chipset and your boards. Yes, in the back. Sure, so the point was I don't have a line in here for maintainability. And the idea being if you're using something that has an upstream community like Ubuntu or Debian you get a lot of maintainability for free. Whether that's a good thing or a bad thing depends on your point of view. You know, if the software changes out from on you that may not be a good thing. But certainly if you're pulling from a prebuilt binary system like that then that does save certainly some effort and build time on the part of your developers. All right, anybody else? All right, very good. Thank you so much. Thank you.