 Hi, my name is Bruce Ashfield, and I am talking today about binary artifacts and the ease of use on ramp for the Octo project. I have done this present parts of this presentation in the past with Mark Hatley. So I want to give credit to the inputs that he's given to that we just weren't quite organized enough to do to get both of us presenting this time around. A little bit about myself, I am one of the maintainers of the Octo project through various components. I take care of the reference kernel, as well as the meta virtualization, which is containerization and traditional virtualization technologies. I've been around for about 10 years. And recently I've been doing more work around the binary outputs of the Octo project and trying to make things a little bit more reusable and work on that on ramp that is mentioned in my title. So what I want to cover today is, you know, I'll do in a level set. I'll talk about the use cases that we are trying to solve. I will go over some of the technology that underpins the ability to do binary outputs and binary artifacts in the Octo project. I won't go into a lot of detail on each one of those because there's other talks that have been done in the past that cover them in much better detail by people that probably know them better. And then I'll do some examples, a quick demo and future work. So what I'd like everybody to get out of this presentation is to understand and get an introduction to the various binary artifacts produced by the Octo project and open embedded. I want to just do that exploration so you'll know what to look for in the plumbing of the binary outputs of the project. And I want to show how ease of use and open embedded in the Octo project are not mutually exclusive. So, first of all, I like to start with a little level set on the definitions because everybody has something different in mind when they hear binary artifacts and ease of use. So for our purposes in this presentation, when I say binary artifacts, I'm talking about the outputs of a defined build that can be used or installed on a running target or to construct a target image. The architecture, the optimization and different settings like that are defined by the build parameters and can impact the level of reusability. In particular, we work with that Xilinx, a lot of ARM platforms, so they have some different challenges in that, you know, the instruction and optimization techniques can vary between platforms, and that a level of optimization does conflict if you have a high priority goal of running common or generic binaries. So that's binary artifacts. Ease of use in the sense it should be obvious and clear how to complete at least the initial steps towards a goal. Meaning you want to build an image you want to boot an image you want to update a package it should be pretty obvious how to do that. And if it isn't well then you've you've got a barrier to entry and that's what we're looking to address. Details of you know how to do those things can vary by the use case. Also for ease of use, we've thrown in that transitioning between use cases is supported and documented and you'll see in some of my slides later why that's an important addition to ease of use and the scenario. So I always do a little talk about, you know, some common questions and comments that we get when talking about open embedded and the Octo project things like, you know, our binary packages supported. Do you always have to start building from source, you know, are these binary outputs that you finally produced are they compatible with third party packages. Whether it be from another distribution or some other build or some other process, you know, are they fully optimized for platform X or software stack why, you know, and then after you get into it it's a little bit of you want to pull the curtain to find out you know what is behind the binary artifacts, you know, and what does that. You know, we'll point out here that it's you know open embedded core in the ecosystem metadata, the recipes in the configuration that is behind those binary outputs. And it's not the sources of other distributions or base binary package feeds from other distributions so in this sense you know what I'm talking about our binary artifacts that are built from open embedded and its ecosystem not grafting in from other sources. Another question that we get or you know can I apt DNF update my target. Can things like Docker build work against these outputs, you know, and of course a big one is why would I use the Octo project party artifacts versus distro X. And there's many more I don't plan on answering all of these questions. During this presentation, but I wanted to throw these are some of the concepts and the ideas that are where I'm looking to explain an answer. In particular that last one it really is you know why would you use the Octo project binary artifacts versus distro X that really does come down to a lot of different reasons and there's been other presentations on just this topic at past embedded Linux conferences and various other forums. I tried to throw together a little bit of a sort of chart representation of these sliding scales of difficulty. It's not completely perfect as I like it but what this is trying to show you is, you know whether you are building from source on the the bottom. Or binaries on the top and then this left and right axis it's sort of easier and harder so you know it depending on again it matters you know what's your point of view are you a developer. Are you a user. Are you trying to do production and commercially supported products. So what your use cases are vary by your point of view. And then they will have a sliding scale of whether maybe binaries are easier place to start sources easier place to start and then how hard or easy they are. And of course there's. These are not precise there's all kinds of different shades of of gray that aren't shown on this slide so for example you know, if you want to do extended or SDK or produce SDKs. Maybe that's easier if you start from source than if you're going from binaries license compliance again might be easier if you start from source and build from sources rather than starting from binaries but you'll see later on in my presentation we're doing work around binaries and license compliance that are built from the active project to hopefully, you know make that a little bit easier anyway so I won't go through all these but you know optimization level container builds reuse it all. Depending on what you are where what what your point of view is and what you're trying to produce. The difficulty will slide around on the scale between binary and source based builds. Another question or represent attempt to represent the same sort of concepts or decision points is you know this slide about is a binary distribution appropriate can you reuse parts of other builds directly without needing to start your own builds. So, you know, in this case, these are the different points of view right you're an it or an application developer in the green box on the lower left, and you're doing a customized optimized device specific distribution in the top right. And in the middle, you're obviously you're somewhere in the middle between the two. So, in the green you know if you're doing an application developer or sort of a traditional enterprise deployment you know really you can use a binary distribution because you are not modifying the base parts of the operating system and the distribution at the level. If you do a partial customization, you might be able to use a binary distribution or you might be able to use parts of a binary distribution, as long as the tooling has a way for you to craft your own customizations into that binary distribution starting point. And of course if you are doing a customized optimized device specifics, then really you can't particularly use somebody else's binary distribution you could certainly create your own and the overlap in the square shows that of course there are areas of gray or decision points where yeah you might be able to use parts, and you could slide between the different use cases. And at that point at this point you know what we've established is there's, you know, various use cases and points of view for binary artifacts, you might, and you might not be able to reuse somebody else's binary build. And the question is okay so what would we consider our binary artifacts out of the Octo project so open embedded and the Octo project has a long history with my artifacts, maybe a lot of people don't know or maybe a lot do. But the point is that they have been part of the project for a significant amount of time and some of the example binary artifacts would be either the build appliance images, or some of the containers that are able to do builds, and that are output from the Octo project, the build tools, which is an SDK to augment older hosts, we have tool chains, and of course what everybody normally thinks about when they talk about body artifacts which is the BSB and machine artifacts whether it be device tree blobs, bootloader kernel images, and these different things for testing these are things that are produced that are available today from various Octo project servers and components. And so what the binary artifacts in the Octo project are designed to you they're designed to support either complete reuse, a bit of customization or total customization, and you should be able to slide between these different scenarios. One thing that I'll point out is and I'll talk about a little bit more before I close the presentation is that, you know, a work in progress that we're looking at right now is sort of a reference binary feed for those that do not need to customize the base or the standard packages, or I guess as well for anybody who wants to embrace and extend some of the base packages so a sort of tested reference set of feeds that we know will work, and that they can be reused by those that want to get a quicker start. So, that's great with the, you know, those are the components that are sort of output as binary artifacts from the Octo project. So the inputs, so how do you, you know, that can make them different or make them vary that you have to pay attention to of course is the build configuration. So that would mean some of the local settings, the layers that you use which makes different sets of packages and different packages and configurations available, as well as your site and local configuration so that would mean your distribution of your local.conf where you can change the attributes of the build which can produce different types of or modifications to the output packages so in particular in site and local configuration if you are looking to generate a reusable binary feed of some sort you need to minimize the types of configurations you do there to make them more broadly usable and hence more appropriate for sharing. So what you get for outputs, you know, some of them some of the binary artifacts, you might not even be able to see their internal to the bill they're internal to the process they're internal to advance configurations and others they are user visible in this list. One thing that we would that I would consider a minor artifact would be the shared state, which you can think of as a build cash. You can use hash equivalency, which works with shared state in order to promote cash reuse by calculating hashes on the inputs to know if an existing estate object can be used to accelerate a build. There is the PR service which is the package revision service, and that is available it works in coordination with all the rest of these components to manage your package upgrades by producing an autoincremented package, and hence you'll have a linear lead increasing package number and your package manager can see a newly modified built updated package as a new upgrade and pull it in that way. We have, and those are all largely behind the scenes now we get into some of the more visible ones which would be, you know, the package feed, which would be could be the package manager of your choice to whatever be the RPM IPK, and of course non o f components boot loaders DTB firmware can all be built from the same bill the doctor project. We have, you know, you might have pre build images which would be starting points for somebody which would be downloaded run, and something that we work on in meta virtualization would be OCI images which are open container initiative images. And run and deploy those types of those outputs in artifacts as well. An example of what you know the different users of open embedded and the doctor project in the ecosystem, where I work at Xilinx, you know what we provide would be some of these binary distribution artifacts of part of our build and release process which would be, you know, we release the system configuration which is the build scripts the layers and components like that that are used for setup. We also have intermediate artifacts which would be the shared state and the various other components to allow build on top of what we've done. We release DTB we have WIC images, and the scripts that create them. And of course we also produce SD card images, ESTK and SDK so we provide SD card images so you can get started and boot quickly, and we provide SDK so you can either build applications or you can customize those base images. And none of these are perfect, but it shows that you know there's various levels of leveraging and reuse of the binary distribution capabilities of the doctor project, depending on what part of the ecosystem, and what vendor you're working with. One thing that you know since I've been contributing and working on the doctor project and others have seen as well, is that the use cases of building embedded quote unquote embedded Linux have changed over time. And one of the things is we are running into more generic images, more application developers, more software stacks that are deeper and don't need that visibility into the lowest levels of the operating system at the beginning. So the big thing is in that case with involving use cases like that is we want to hide the learning and complexity curve until it's needed. You know the there's things behind the scenes that are solving problems you might not know you have yet when you're deploying a node JS application, or you're doing a go build for example there's dependencies there's reproducible builds there's licensing, but you don't really need to see that immediately, just to get started. We're also running into heterogeneous systems so we are running into more systems that have a mix of Linux. Our tosses they have firmware they have MC use and to have we need to have the ability to build for those types of systems. Also, we're there's more and more demand for more than just images that you can flash onto an SD card and boot. So we have things like containers in binary deltas which I'm implying there are be the various different types of over the year updates and management of a target. So we have deeper software stacks, we have multiple flavors of Kubernetes, you know, full Kubernetes K3S available that you can deploy software that way. And of course, building really small microservices with the octa project so you can deploy them that way. We're also running into that merging and blending of embedded Linux, increasing in complexity and enterprise Linux distributions coming down to maybe smaller slash, you know, edge system so you end up with blended features from the two. So you want more than a package installer. You want accelerated containers that can make use of hardware assist to do AI image processing, different things like that we want potentially safe and secure containers through technologies like run x leveraging and a componentized system. We want low footprint run times on both as an example so we can bring up the density of the services on a smaller device. And, you know, we want more maintenance and in service upgrades versus again, back to more than just a flashed image. So the big thing is between these different use cases is the mobility between them is important. You know, is there a defined and structured way to change your use case so the way that these are enabled needs to be the same for all the different use cases that are not yet is available only when you need it you shouldn't have to dive into multi config to do your heterogeneous system until you know you need it, for example. Another important thing and this goes to the hidden capabilities that are that you shouldn't have to worry about until you need them so you know beyond packages things to consider. So you need to do reproducibility reproducible builds. This is a core yachto project capability, and you can check out the details at reproducible builds.org. And it turns out that a lot of the work that has been done for reproducibility of course also leads into more efficient binary artifact building with hash estate reuse and different components like that licensing S bomb. Also, are you know that you may need to do to consider more than just producing by binary packages so that is also a core capability. And there's multiple ways if you've built from open embedded core the yachto project that you can consume those outputs and accompany it in a company your binary deliveries customization. You need to how would you if you need to modify package some way is there a streamlined way besides pulling the source on to your target configuring and building how do you customize the target what level of customization. So will you get support from the ecosystem. If you've modified extended or use them directly you know what is, what is the way that you can get help whether it be in the community or commercially if you modified or extended packages. How can you extend a platform is it possible, or do you have to start from scratch. And also be considering both application and system developers don't if you cater to uniquely one or the other you'll either be too simple or too complex and having a system that can support moving between the two is absolutely important. And then of course the typical, what are you going to do for support maintenance and updates once you've done your first binary build or reuse the binary package feed from somebody else. Over the years one of the things that we've noticed that when you you when you add some of these capabilities or these concerns to your traditional distros or enterprise distros you know it can be a little bit ad hoc, or maybe you don't follow the guides directly, and it can result in something that's potentially less structured than what you get out of the yachto project. So at that point. If you look at the learning and complexity curves, they're roughly similar so the, the, the comment that the yachto project in open embedded is hard or is complex. It tends to blur so you need to view the complexity through the things that you need to consider after you've built the packages. So what I wanted to do for about the next five, five or 10 minutes would be to go over and do a little bit of a demo image with a package feed. There's been other demonstrations of this in the past, but I wanted to throw it in here to show that the stuff that I'm talking about is is available, and it does works. I wanted to do most of it fully live because the build server that have available can take two to three minutes per some of the operations so it takes a little bit too long to fit into our a lot of time. But what I would like to cover first is that you know what I've done. The package build is is is cloned. The octave project or Pocky reference distribution. I'm using the minimal the minimal image that is part of a week or flash, you know Pocky and I've done I've configured that image for package management system D, and I'm using it for the base, the base image. I wanted to show package management DNF in this case because we're building the default which is RPMs commands on target. I will show how you know we can add packages to the feed after you've done your initial build, how we can install and upgrade those same packages on the target. This is something new that's maybe not quite as commonly demonstrated. I'll show a little bit about our capabilities with meta virtualization and those types of binary outputs and show how I can on that same running image I can add Docker. You can use pod man or some of the other available runtimes that we have. We can add those dependencies onto the target. I will show how we can create a simple base container image. We can add that image to a registry and then pull and run that same image onto the target. So I've captured in the slides, an abbreviated set of details so they could sort of be used as a reference, because it's hard to show it all in the demo. So, you know, here's a little bit how what I cloned and a little bit of the configuration that I mentioned in my summary I've enabled system D. We have an SSH so I can get into the target. I've also added a little bit of extra space because Docker and some of the other builds will be doing actually need a fair bit of storage. The only other things of interest in this configuration you can see where I have set the package feed URI to which is the IP address of my build host in this case. You would set that to whatever your build or you could actually set it on the target after you've booted. Also, there's other various configurations where you can use a remote or a shared package feed coupled with shared estate PR server and hash equivalency. There's lots of complex and different types of configurations you can do. I'm using basically the simplest. So in order to, you know, to get this started, I bit bake core image minimal on my builder it takes a little while so we're not going to do that. I already have it this base image with what is in core image minimal built, and then you bit bake package image. Sorry package index, and that goes through and it creates what is needed by the whatever your packaging system is in order to create the binary feed to be served to the system. And then you need to have some sort of HTTP front end to serve those packages so on in my example I'm just using the built in Python web server and I've gone to temp deploy and started that server. And then booting the target. Once the target is up and running. You can see that it's a it's a recent build from last week of five dot 13 dot 12, and that we initialize DNF by running DNF make cash. And that goes through it will contact the package server it will create the you know the default database, and then we can search to see what is available on the target. And in my example I've gone through I've searched for busy box, you can see that it is available there's different versions it's installed and all of the different utilities. And I searched for Docker in this example and it is not currently even available in the package feed so that's something that I'm in the demonstration that we can go back to our build host added to the package feed and it will appear. Here's an example as well, something even simpler than adding Docker because it builds quite a bit more dependencies is that say we wanted to add them to core image minimal because it's not available or they it's got the built in busy box. Vi and you want something a little bit more elaborate so we can go back to the build host we could bit big them, it will build them and all these fantasies and then re bit big the package index. On the target, you would refresh your caches, you can search for them, see that it's available. dnf install them and as you'd expect, we get about 23 megabytes of downloaded packages, and them is now available in your package feed and it is installed on your target. The transition to make that permanently part of any new images you would go through and do the normal define a distro and an image and you could make it always built and installed in the future if that's that's what you want to do but I my demos are all about adding packages to a running image without needing to reflash and restart the entire image. This this part of the demo is showing how we can use the PR service to trigger a package upgrade on the target so in this case I'm trying to run expand which is not available in this busy box based image. So we get command not found. So if we go to the build host, and we add config expand to the busy box configuration, we would do a bit bake on busy box, and the, as mentioned the estate, the hash equivalency and the PR server would figure out the features, see that this is a new configuration the package and it would produce an updated revision to the package, so that it would be available as an upgrade so once a bit big busy box and redo the package index, you can see that I now have a busy box and dot dash are zero dot three, I had a couple of iterations, the dot X is what is increasing by the PR service so we now have a zero dot three RPM available. If we redo the make cash to check with that package feed I can do a DNF upgrade on busy box, it sees that zero dot three is available. I install it and of course now I have expand available. So that was all managed through all the plumbing that you don't need to know about it immediately but you know that if you do customize the base image, you can have it flow into the package feed. So that same theory holds if you're using a base package feed from somewhere else but you have access to a shared PR server estate and various of the internal artifacts, you could rebuild busy box and have your own extended binary show up in the package feed as a newer version and supersede what you're initially handed in that base image or package feed. So this demo is again a little bit more complex, and I didn't build it because there is a little bit more to the rebuild in this case I've added meta virtualization, meta security and meta open embedded to my build. I've added the virtualization and set comp feature to the distro and that triggers a bit more of a rebuild and then we have time to sit through. And I've, and once all that is configured and then I've baked the Moby version of Docker that is available. And a note that I'll put here is this actually rebuilds the kernel because of the virtualization features. It adds some of the different configurations that you would need for proper Docker Docker operation on the target. So you're either going to need to do a DNF kernel update, which is one of the things we're looking at it still may or may not have issues or you may you will need to rebuild the whole image and start over. So you bit big Docker Moby, the kernel, all this dependencies get built. And again, as I've been doing all the way through this, we bit bake package image. And then you have the two options. Are you going to rebake core image minimal or option B that I did is I did a DNF upgrade on the kernel, it pulled in the new kernel and the dependencies for headers and various things. And then I shut down QEMU to make sure that we reboot with the with the new kernel. So in this case, I think I might actually attempt to show some of this live. If it fails, we will go back to the reference slides. So in this case, I have the build that I've been doing that I've been using to do to capture the upper for these sites. So if I do the run QEMU that you see on the captured versions were booting that same image. So this is, you know, one that was built on Monday, August 23rd. And we don't have, we don't have Docker available. So using the package feed and the various components that I just built, we can do a DNF install Docker Moby. Actually, before I do that one thing that is interested to see is that so we can show the difference if we do a DNF. If we do a DNF list installed kernel star. And we see how many there are for kernel packages on the target. You'll see that we have four. And so after we do a DNF install Docker Moby. We're going to download 236 package 51 megabytes and add to so 200 and some megabytes to my image so we get all of the IP tables dependency modules we get a bunch of kernel modules. And in this case we are going to get some util Linux individual packages to supersede the busy box built in components. So you'd expect it's contacted the the package server. It's pulled in those possibilities. It's running their install scripts on the target and so in this case we're taking care of everything from it's installing the package. It's running update alternatives to make sure these are pointed to instead of the busy box alternatives. And you know generally making the packages available. So it's you know we're, again, I'm throttled by the speed of my build server. Versus how long this would take this would vary based on your and of course as these kernel modules installed we're also getting debt mod and other installation post installation. So we're going to run to make sure it's available. And one of the things that we're looking at, and I'll talk about briefly in a minute is we're looking at making sure that those post install scripts and installation steps are tested and and working consistently such that you can count on your package feeds being able to do things like this so there we're finally now to the point we are where we are installing Docker Moby. And we should be done very soon. All right we are complete. So now if I rerun DNF list installed kernel there's now 33 kernel package that available so you could see it wasn't just Docker, we got a lot of the Docker components installed. So a system control stop Docker, and we will then start the Docker demon, and due to the networking configuration of my build server this can actually take up the two minutes to get sort of things started so I will switch back to the slides while while Docker is busy starting up. So I capture these commands we don't need to go over them we're starting and so in this example you could see, you know, once Docker is up and running on the target, we can see that it's 20.10.8 it was built recently, and that we can do a Docker poll on busy box which is the standard reference container from Docker hub, and that I can run that busy box target. Sorry, container on our target without needing to do any modifications. Now that's great that we can run a reference images from from Docker hub, but you might be more interested in being able to run your own container so in this in this example it's again a very simple example. There's something called container base and meta virtualization that's we're working on making more useful. If I bit big container base that is the minimal sort of configure container configuration it skips a kernel and some dependencies and puts a shell into the entry point. And it uses the image OCI back in a meta virtualization, and it creates an OCI container in temp deploy. So you know you bit the container base it builds a container builds the components or reuses them in this case. And then from our build host we can use gopio to copy it to in this example, my Docker hub. And we can, it's something called container base, and then back on the target I can do a Docker poll container base, and it brings in the artifacts that I just built, and I can run it just fine I can drop into that container, and it works just the same. You know, as the busy box reference and it was completely built from open embedded the octa project and of course you now have the associated licensing and source information behind that container if that's we're using because you can now use that as a base container for for other Docker builds and other reuse scenarios. Quickly check to see if Docker wanted to start. And if not, we won't okay so it is up and running so we can do a Docker version. And we can do that Docker poll on on. We can do a Docker poll on Iran you can see using tag latest. Pulls the layer, and so I can do Docker run. And we drop into the container, and it's just exactly what you would expect. And then of course you at that point you're free to do whatever you want you can keep and do you can retag push and use a sort of a typical workflow to pull and update the container in closing. To the presentation I want to introduce this binary artifacts working group that I'm helping to drive as part of sort of the medium to long term planning in the the octa project right now through the summer it's been on about a monthly sync meeting. It will get hopefully a little bit more. So that's what if needed, going through the fall, and that is a focus on code and tangible outputs it's not just planning. The goal is to actually to produce reusable parts of the ecosystem. So, in particular, we're looking at making sure the components that have always existed, making sure that there's good documentation around them that the guides are easy to find they're easy to follow, and that we're looking at it consistently to put some QA and some structure around that binary infrastructure. So we can test improve and know that it's working on sort of a nightly base basis. We want to create a place for collaboration and technology sharing. It's all about the Xprate embrace and extend we are not trying to implement anything that we don't have to whether it be configuration and set up whether it be testing whether it be the base configuration with the package feeds is not about creating everything from scratch. It is the overarching goal is to address that ease of use and on ramp questions how do you get started more quickly. And also to unlock, you know, the full potential. And also to produce sort of this tested reference binary feed. As I mentioned when I was going through my demo with attention to some of the upgrade paths and package management issues with the sharing of a state PR server to making sure that they both scale and work consistency, and to produce some eventually some maybe some reference containers and different types of binary artifacts that so people can test and use them quickly on different types of systems, whether they have or haven't been built from open embedded and I will point out that the non goal of the working group is to create a commercial and supported binary distribution, or to replace existing open embedded based binary distributions. It's all about creating that reference and making sure that the underpinnings that are available to both doctor project members and open embedded community members are working consistently and working, you know, as advertised. If you're interested in binary artifacts, or helping out in this effort, you know, you can join the join open embedded, or the doctor project and help us unlock the potential of binary artifacts and these types of scenarios. And my final thing is I will close with sort of this future work slide. It's not it's sort of what you would expect is to get up and running the reference distributions there may or may not be that first package feed available by the time. This presentation is viewed as part of embedded Linux, but it won't be too far after. We would like to produce some maybe some pre built images for, or at least advertise the existing pre built images out of the the auto builder, or different infrastructures that are available. And I said auto builder but this is not tied into your project infrastructure it should be working on GitLab and various other CI pipelines, we could potentially have some sort of installer. Go back to my we're not looking to reinvent. Another thing would be to extend those reference feeds outside of our core because a lot of the interesting packages and things that people would like to do are part of the Yachto project and open embedded ecosystem whether it be open open and meta open embedded or meta virtualization security, some of the different layers. And of course, as I mentioned something that I'm working on in particular is reference based containers, both the pre built variants, and the recipes and base definitions to produce those containers so somebody can do it on their own. And that is it for my presentation. I'm now available for questions. If we have anything that we'd like to adjust. Thank you very much for your time. I hope everybody got something out of the quickly presented and broad ranging material that I had.