 So I'm Scott Murray and this presentation is building container images with open embedded in the Octo project It's a little bit about myself. I've been using Linux since 1996 The first few years I did some hobbyists stuff for my own amusement and then in 2000 I started actually working on Linux for a living doing embedded and non-embedded desktop software stuff The last few years I've been working for a console group Currently a principal software engineer for them It's all good groups a services company We specialize in embedded Linux and opensware software and kind of run the gamut of hardware software design build development and We do training and we're a Linux foundation training partner If you have any interest in that you know come up to one of us and ask about it So it's a San Jose company, but we're quite distributed I live in Toronto, Canada, and we have people in Europe and all over the US So quick agenda of what I'm going to try and cover So a really hopefully quick overview of open embedded and the Octo project I'm hoping that everyone has got a reasonable familiarity with those a little you know sort of spiel about containers Probably also something that most people here have some knowledge of already and then I'm going to talk about what open embedded can bring to the container sort of situation and then I'm going to walk through some example build configurations that I've been tinkering with to try and build both full distributions and application sort of micro service type containers and Then a little bit about nesting images to try and build a Like factory install sort of pre-installing application sandboxes in an image during the build process Which is surprisingly not not an easy thing necessarily A few kit caveats I am not a container expert by any stretch of the imagination when I submitted this talk I thought it would be good. I'd you know good opportunity to learn some so I have been doing a lot of a lot of googling and learning things But this presentation I do not really go into the details of how you would take the images that are being generated and Actually using them in with talker or run C or any of the other tools As well container technology every month or some significant change I might have missed something you know in the way of tooling that If you know of something quite useful either come up to me afterwards or during that I'm gonna you know take questions at the end. Hopefully, you know throw something up And I am assuming a bit of open embedded and you know yachto project and you know maybe a little bit of you know Docker knowledge Because I'm not gonna go deep dive in some things So our hopefully quick summary of open embedded in the octa project Open embedded is a build system and the metadata to build embedded Linux distributions It is not itself a Linux distribution The octa project is a collaboration project. It's been running for eight years To aid in the creation of Linux products based on you know open embedded as the technology behind it And yachto project provides a reference distribution called pocky Which is built with open embedded and provided by the octa project and then so that's the goal is to have that pocky acts as a starting point I refer to Pocky and some of the configurations, but I'm trying to do things in my example Configurations that would also work if you just use plain open embedded yourself and don't use the octa project pocky So quick feature list here So one of the big strong points of open embedded is has very broad cpu architecture support Basically because vendor support it with BSP layers that they actually maintain themselves and contribute so the layers are basically a Way of customizing your configuration So it's basically has stacking metadata and that's one of the key strengths of open embedded and yachto project Because it's an embedded, you know Linux distribution sort of tool That's sort of the focus on with it. I have worked on projects where we've built server side stuff with it But the goal is usually in an embedded system to build small images So that is kind of one of the focuses of open embedded It's notable for a regular release schedule basically every six months. There's a release pretty much One of its strong points is integrated license and source publishing compliance tools Which make it quite easy to Publish your license, you know information for an embedded system and actually, you know archive your source to provide to somebody who asks for it One of the sort of current projects it's ongoing for the last couple releases is full binding reproducibility which Debian has also been working on which is a you know Boost your confidence level in your images that you can reproduce the contents and it's also this some security benefits there as well So containers No surprise hopefully to anyone are basically operating system level virtualization as Opposed to virtual machines that basically virtualize a whole machine Linux implementations, you know have been around for a while. I mean basically this is like, you know going from shrewd to something more substantial Nowadays it's namespaces and C groups the Linux control groups feature So I mean the sort of the old school tool these days is Linux containers LXC But then Docker and run C and even system D has system DN spawn Which allows you to actually even with the stock Linux system with system be installed run a you know system of containers if you put your mind to it To muddy things a bit Nowadays there's actually is work on it what that you know they refer to as lightweight VM technology So clear and cat a container is actually do Virtualization at the machine level, but it's aims to be so lightweight that it's doesn't have the performance impact and startup that Typically VMs do and that's because that's one of the key strengths that people have for containers is quick startup Then ease of of starting and lightweight resource usage. So these are new technologies to try and get the benefits of VMs in the same space Fortunately, that's all I'm gonna talk about in that for about Cat a containers there isn't really a integration In upstream open embedded there are some layers floating around but what I was looking at that looking at it there a bit stale So there's probably some work to be done there Container images are you know run the gamut you can do a full Linux distribution install in there or Nowadays it's more typical to have small images with potentially a single application and just its dependencies That's pretty common now in this micro services model and you know having kubernetes orchestrate a big system So continue that vein the common use cases are Sort of the more old-school thing is is trying to run an application that has you know gone stale It doesn't actually you know on a new new system doesn't have libraries that will work for that application You can sort of run it inside a sandbox Sort of the container and actually run it with its own set of libraries and actually get that it will run on a new system one of the uses for that is builds you want to keep your Potentially builds running on newer machines, you know when things shift over time in the tool chain or in libc You can get stuck with you know your build doesn't work anymore on a new system. That's one of the use cases Sandboxing an application to isolate it So this is more of the security sort of idea where you basically can't see the whole file system or all the other processes There's you know debates on how good you know container security is versus virtualization It's approving all the time, but I mean nowadays or I mean it's a heavily researched area You know the black cats are always trying to fork out how to get out of a container environment break out of a namespace But it has come a long ways in the last few years And so the other use case I sort of already mentioned is microservices with this like starting on demand for orchestration and you know Potentially migrating services from one machine to another So typical container construction These days is you start with a minimal system a Debbie and or you boon to or very commonly now alpine Linux Which is quite small and then you add your required packages that you need to run your software or your particular application And then you might actually compile some things Actually install the tool chain build a particular piece of software Using its build process and then you actually prune down you might remove actually the tool chain that was in there and Everything other you know the source for that application You might have compiled to try and make a very small image and that's desirable because it reduces the attack surface Less software is better in the container Makes it easier to maintain again from having less software in migration time and systems where you're actually shifting containers from host to host and like a you know cloud scenario Smaller is you know easier to transfer less time so What are the potential drawbacks of you know some of these types of things? With these sort of common, you know scenarios of how people build containers Reproducibility is not necessarily great In Docker you sort of rely on People keeping the labels sort of sane and not overwriting an image and keeping the same label And being good about actually versioning things That might not be obvious if you're actually running a Docker file It might pull down something different than the last time you ran it and unless you're careful to pay attention That would be a surprise and that comes into like the base Debbie in an alpine images might upgrade You need to keep track of what's going on And then you see things in Docker files, you know I'm not a Docker expert But looking around people do things where they do an automated app get update and upgrade because they want the latest bits Which is good. I mean security wise. That's that's smart But unless you're actually careful to track what's happening when every time you run the Docker file to make a container It might be quite different each time, right? I mean You have to go look at the package, but you know a manager and see what's in there And as well there's things where you see people talking about what you can pin a package version in you know an app tour APK but I have seen reports of things where people go to reproduce a container and Alpine basically don't archive necessarily all the old versions of packages And so you go to reproduce and it's gone and you can no longer build That without doing a very intense web search to find an old version of the the package And that's not ideal really I mean if you if you want to actually be able to tell people this we know what bits are in these images and we can Reproduce it. That's not good As well, I mean so that as I've sort of been reiterating You actually have to put some effort in in this scheme with like, you know using doc files or other mechanisms to compose images And So some of that is a trust issue that the upstream is not going to change on you As this comes into this transparency and sort of a security aspect of it You do have to trust the builders of the base container images And so if you're you know paranoid or your lawyers are paranoid You want to really be sure that you know the origin of the bits that are in the image And so I mean distributions. I mean you this you can build up a you know confidence level You know trust Debian that the developers actually aren't going to sneak something in But then there's the you have to be sort of sure that none of the Debian package maintainers are going to sneak something in and get it by people so it's you're kind of at the whim of maybe distribution update policies on the security front and Having you know if you're pulling a large amount of software in a container for something large like node Or you know Python sort of software framework It's it's definitely a challenge potentially and of course is also anyone who's done much container work It's very common refrain of well containers if you you know Open SSL has an issue. You have to update every container image to pick up the you know CVE fix So I mean that's something that I'll address about I Mean you can just do your app get updated rerun your docker files or whatever and build new images, but It's not necessarily ideal The other thing is licensed compliance Potentially we're not talking about a lot of software. I mean Alpine is is quite minimal in a base image It's only several packages But I mean if you do have a legal team who wants to be able to produce a list of the software the Licenses be able to produce a source archive of everything that's in an image That's not straightforward necessarily Without being quite familiar with the package manager and being able to script up pulling it out of you know apt or Fetching all the source packages and you know scripting up a mechanism to build that So that's not ideal Customization is where things really get more sort of wonky for if you're you know It's one thing to have a docker file. It does you know W get such-and-such start tar dot gz configure, you know make blah blah blah But if there's a package that's in the distribution and you for some reason you want to change its configuration and need to rebuild it Ideally you would do it with the something that's compatible with the package manager And that kind of gets into the state where you might actually need to know more about building packages with the distributions build process You know using its package build things like Open build server ease that a bit, but then it gets into things where if you were trying to do Containers with a distribution for some reason on an architecture that the distribution didn't support Bootstrapping something like Debian or Alpine is a pretty big job if you've not done it before And so that's not something the average person is going to do And so an embedded system say you wanted to do sandboxing and you want to use a very small Distribution and you want to use Alpine and you're on a MIPS system You're going to be in our you can't do that. Basically. You're kind of stuck Unless you really like build systems So He's open embedded in the octa project a solution to these types of problems. I'll sort of run through the same list Reproduce the wise we have a good story Basically, you can rerun the build using the same set of metadata and Should get the same results that's improving all the time as I sort of alluded to at the beginning Ideally within and a couple releases. It should be the case that with the right configuration You should get the same binaries every time Several distributions are working on that, but you will actually be able to pretty much do that yourself Transparency wise because everything's bootstraps and scratch you have all the source in hand if you want to audit things you have it It's pretty straightforward The security front there's 18 months of support per open embedded release or octa project depending on This is a fine line of who's actually doing the QA there, but It's Sort of been discussed about long-term support releases for for open embedded, but it hasn't really taken off yet So on the other side we have debbing is pretty much five years For one I can tell the Alpine is a two-year promise with some potential exceptions For certain packages. They might actually do updates for So that's not actually open embedded isn't look too bad in that sort of like Um license compliance, that's pretty much all built in Basic a bit default to get you know license manifests You can do license text archiving even inside the image if you want to have it contained with the container Source I carving is a pretty straightforward thing to do So wait, that's probably one of the stronger arguments for using open embedded for containers The Customization front because it's a you know pretty much the purpose of open embedded is to be very flexible The layered metadata in the build process you can almost customize anything, but it comes into your skill level I mean simple config option tweaks are pretty straightforward But if you get into wanting to change tool chain security flags and stuff that becomes more involved You'll have to be somewhat expert, you know in tool chain stuff to do that As well pretty much any architecture that that a BSP layer exists can actually be targeted That's useful for embedded if we wanted to sandboxing with container tools We can do it on platforms that don't actually have mainstream Linux distributions that are geared towards containers We can build our own So then we get into some more general issues package availability of you know, what's what's the story there So Debbie you boot to it's really hard to come up with a good sort of solid number But that's tens of thousands of packages even for arm. It's these days It's tens of thousands of packages Alpine from what I can tell they pretty much build 5,000 packages With their system they have a custom System so that but it is in the same ballpark, but not as dramatic as Debian Open embedded at the moment is about 2,300 packages by going by recipe count Some of those are going to be tool chain ones So it's not necessarily a really solid number there, but that's in the open I'm at a core and met an open embedded layers, which are pretty much heavily the heavily Qaid layers so those recipes are likely to be usable right off But there are many more recipes if you actually start searching around but then your confidence level of Reproducibility and how well the light of the recipe for that software is going to work will be not as good one thing to note is Open embedded node and Python module support is is this quite a few like of the commonly used packages that there are recipes for But given the you know the sort of rapid development and that's sort of wide array of new modules Popping up that people want to use in their stacks That's probably a bit of a problem. You might actually have to write recipes for some of those yourself in order to use them Which would be good if you could then upstream them to the project and the cycle would sort of continue but Then on the easy use fronts One of the things that you know you get with Debian or Ubuntu or Alpine is the like app get or APK install Sort of experience where you're pulling from a package feed You can do that with open embedded It's a bit more work to configure than probably the average person is really going to be comfortable with necessarily right off It's not the most well-documented part of the project in my opinion But it is doable There are enough wrinkles that it's probably not something everybody would do And so that kind of means that it's maybe more the case that small relatively Fixed content images are going to be the thing that will be more straightforward to handle with open embedded Resources it's a new thing to learn if you haven't used it before and you want to jump in and start building container images You're gonna be a you know, there's a little learning curve It's a build process so there's potentially quite significant horrible resources if you're building, you know much software And a lot of term maintenance might involve actually dedicating someone to keep track of the project and you know sort of focusing on it So that said so there are pluses and minuses I'm not saying this is gonna change your life And this is the way that everybody should build containers. That's not a slam dunk So on the To get to the sort of mechanics of of what we would do with open embedded So back in the pyro release last year, I believe we're called correctly or late 2016 There was a container image type added and so on the up here In your configuration you can just set the image file system type to container And it basically produces a tar ball with there's no kernel It actually disables post install scripts because you're not gonna have a normal init process There won't be run and then you have to set the kernel to be the dummy kernel Which basically won't build anything and that's it you actually will get a tar ball with just what's configured in the image So that's how we would build a container If we actually want the tools to do something with that container like on target basically There's a meta virtualization layer It provides the sort of gamut of you know Linux containers run C Docker Which is not quite the latest version, but it's in the current master branch is close As well as got the open container image tools And this configuration frame fragments for the Linux Octo kernel that when you pull in the meta virtualization layer We'll add the required kernel configuration to actually run Container tools One thing to note is there's currently no support for Building you're like an OCI or you know a Docker image during the build Which would be mean like running a Docker file or doing Docker compose type of stuff because Docker basically has a demon that it has to have running in order to do most of these types of operations and That's kind of a difficult thing to achieve with the build process I'm poking around there There are so many web pages where people talk about alternate ways to actually use different tools to try and actually build a container without using Docker itself if you actually want a Docker compatible container because it has its own format I haven't actually figured it out yet If anyone has any ideas maybe come talk to me or throw it up at during the question period As well Togan labs they have a booth down in the vendor showcase They have a oryx linux Distribution that they support and it's basically based on open embedded and they offer a commercial support package for it And they actually have container support as part of their system in oryx and they use run run see from the open container folks as the this sort of container Running system on the target and there's the URL there. I haven't played with it yet might poke around at some point just to Kind of see how they do some things, but it looks pretty much like they use meta virtualization and grab the tools But they go talk to them at their booth if you want some details, I think All right, so I sort of rushed through the Frozen cons why we might want to use open embedded and you know, Yachto project for building container images I'm gonna try and get through my examples before I drastically run at a time So I have a few things in mind when I sort of submitted the talk one is It'd be nice to have a container image that I could actually run an open embedded build in It's I mean, I you know, most people carry around like a Debian image or something to kind of give them maximum compatibility for doing builds But it's been a thing for a while that open embedded has I think called the build appliance virtual machine image Which is inside the open embedded distribution you can build it so he can self host its own build And I was like well I'd like a container to do that and not have to worry about building the VM image and stuff So that's one of the things I've tried to accomplish Another one was to see well, how close can I get to something that's like alpine? Linux Size wise, you know hard how hard is it to sort of get sort of similar functionality and So look a little can some configuration options to do that And then there's the building a single application container. So the typical microservice type of thing and then my sort of ideas around Building an image that's pre-loaded with application containers and this would be potentially useful for factory image builds if you're doing sandboxing of applications on your target, all right, so this is my Example for a build container And so for most of these I walk through how you would just sort of ad hoc do it by putting things in local.com Which is the main configuration file for open embedded If you've used it at all, you've probably tweaked things and open in local.com So here I'm actually just setting the machine to the x8664 cumue compatible machine I'm setting the FS type to container which as mentioned gives us a sort of bare bones image I'm actually setting the kernel to dummy because that's required by The container image type and so one of the things is I actually have to set the locale to be included so I'm setting the language here because Nowadays open embedded the bit bake tool use is python 3 and it needs a locale in the image to actually work So I have to actually tell it I want one Because I'm going to build a core image minimal image, which normally wouldn't have any locales in it And then this is the magic these two package groups are defined in oe core in the build appliance Stuff there's actually a package group self-hosted Recipe that these are two package groups out of four or five But these two are enough to actually build an image on top of core image minimal These two package groups give you all the tools to actually do a bootstrap build and so When you build this it's about 150 megabytes for cumue x8664 The build process is interesting because the the self-hosted package group that the build appliance Got a graphical desktop an extremely minimal one But you when you do a bit bake that package group gets built all of it And so you're it's interesting because web engine or I think it's built Which takes a lot longer than you'd expect building something like this should take but then it doesn't actually end up in the image So that means if you were trying to build a minimal image or you wanted to reproduce this quickly You might tweak some things to try and speed that up And maybe prune out a few things So because the container type Image type turns off post install scripts Things like the volatile directories functionality that's on by default and open embedded Break is maybe too strong a word. It doesn't do necessarily what you expect because Without those init scripts running. There's a few directories missing and some software works perfectly fine But the build does not So to work around that we need to sort of populate a few things and it's a var volatile temp is one of the sort of big ones So we could actually script up a little bit where we take this container image that we get and actually run the Nitscript that ends up in it to create those handful of directories We could tweak it as well in our like make an image recipe and and basically Create them by hand as well. That's there's a sort of couple of different options there The base files packages what controls the creation of these volatile directories We could BP append that and actually tweak the recipe to change the behavior As well your user for doing your builds needs to be created the build appliance VM image actually adds a special It has a recipe that adds a builder user We kind of want to work around that necessarily you might actually want to have your own local user so that gets into more like Docker configuration or System D and spawn tweaking with N spawn config and as well your build tree to actually get at it You need to do Docker volumes or do some bind mounts But it works perfectly fine. I was doing test builds works fine And this is something I actually probably tinker with some more because I would like to actually have support for this an upstream open embedded It actually seems quite useful So the follow-up this is actually an image definition of Sort of some of these ideas without actually having to do it in local.com. You could make a layer Put this image definition in it and it sits on top of Open like this OE core pretty much and if you bit bake build container You'll actually get an image that actually pretty much works and you then you can actually run bit bake inside that container One of the things to note. I have thrown in adding the x4 type That's sort of a convenience thing for system D and spawn You can just stick an x4 image into the barrel of machines directory and just ends the run system D Container and spawn to have started so and this is my sort of hack for fixing the temp directories Using the root of s post-prost command hook. It's not pretty But it does work So one of the things is You start poking around is using ecumu x86 64 It would be nice if I didn't actually have to set preferred provider virtual kernel to use the dummy kernel and There's a one extra little package that ecumu x86 64 pulls in that is not useful in Container so this is like a machine definition you could throw somewhere as you know like you know make a small little Bsp to use for containers That would basically turn off. I think it's v86 D is the package and force the use of when it's dummy without having to add it to let local.conf all the time Okay, so that's Pretty straightforward that didn't seem that bad actually it was took a little bit of you know wrapping my head around what the Self-hosting package groups were going to be needed and things like that and some tweaking to but it's not that bad It doesn't take much configuration do that So let's try and raise the bar. Let's try do like alpine ish type of image So one thing about alpine is it uses the muscle C library instead of glib C That's probably the biggest size reduction that that they get is doing that I've you know the previous talk, but probably I assume that was discussed I've spoken back in the spring also what size reduction muscle is the biggest bang for your buck and reducing sizes of open embedded images So this pretty much gets you not that far off actually you can bit bake core image minimal with this Container sort of configuration and just switching to muscle produces something that is not that big This is the image manifest And so there's a few odds and ends in here that you wouldn't think you'd need This doesn't use my machine definition. So it has v86d in it, but It has a few other odds and ends that actually has assist five in it hooks It has the update alternatives for the old package package manager I'll talk a little bit about you can prune these things out actually Because you're not gonna actually have init scripts run normally in one of these containers You might because by default alpine supports that you could do that You could actually run in it inside this container and actually start up multiple things But most people don't actually do that. You can still go ahead and prune those out So as I mentioned on this slide, so this actually is about 4.8 megabytes But there's no package management and so alpine uses its own at package manager APK And so if I actually configure with o package as the package manager in the image We do end up at eight and a half megs, which is in the ballpark of double the size of an alpine image, unfortunately One of the other features of alpine is it signs all its packages That's configured, but we can actually do that with o package and open embedded We can also do it with rpm But when you turn on rpm as the package manager you get an image that's about a hundred megabytes So that's pretty much a non-starter. I assume for anyone who wants to do microservices That's quite a bit larger than you could do with pretty much anything else and like I said further pruning is possible one ways with a custom district configuration and As well all the like update alternatives and things like that. There is an option That's not really obvious called force our own remove which is force read only remove Because when you build a read-only image in open embedded I mean, it's pretty obvious You're not going to actually update the alternatives links when you install packages because you're not going to install packages And so when you build an image that's read only all this stuff gets pulled out But a couple releases ago a knob was added to force that behavior So if you set force our own remove a few other things get pulled out of the image Which in our use cases if we're not going to use package management. That's what we want So this is a example of a sort of custom district configuration I Called it schooner. I can explain that to somebody if they ask me later But it's this very minimal just for like sort of ad hoc thing the big takeaway here is I am basing off a Pocky, but I you know and change the name of stuff, but I am setting muscle as the default C library and This is the sort of big win here is I'm actually setting a very minimal distribution features And so this cuts out quite a bit of stuff If you look at distribution features with the big money C on this regular Pocky It's like three or four lines of stuff a Bunch of is is coming from this variable But just all the other variables like blue Z and pulse audio and things like that Which we're not going to want any of that in our container image and these variables are used by the core image sort of functionality the default image types in Open embedded and so if you set these a few other things get pulled out So the init scripts and the unit manager and things like that So if you're going to make your own image type that doesn't base on the core image You don't need to worry about these, but if you want to make it this is the fault district configuration It's nice to actually have these defined to do what we want All right So remember a little bit of that. I'm actually going to the some of these subsequent examples I actually use schooner.conf when I'm building them So we've built our sort of alpine alike thing to see what we can do Let's try and build a minimal app container just a single application as dependencies So this is my base for that where I attempt to build as little as possible and so Basically, I do you know sort of set the container type I Attacking on x4 because I want to use this in the end spot at one point But here I'm actually turning off all the image features by default no language support and I'm disabling recommendations, which is a mechanism where you might not have a hard dependency on something But you can recommend that it be installed for utility You can turn that off and then it makes if you need something you have to explicitly say you want it and Then I actually define I would do want Things like the default directory structure. So like Etsy and dev and things like that So they they're available for things to be mounted on them by the container tools That's base files base past WD gives us that sort of initial sort of blank You know or whatever default past WD for things to actually be able to look up UIDs and stuff and then net based son I think taxing Etsy hosts sort of default values and then I also sort of hook up the my sort of Fill fill in for now my scheme for actually making sure that the volatile directory stuff is handled so If we say that's our template And then I want to build something with that So my first example here of doing that is I actually build a minimal image with light HDPD and so I pull in that previous image definition and add to it and So as the comment says Light HDPD actually requires BNSH when you start poking around and you're trying to run it It actually execs some things and needs a shell So that previous image you will build you won't actually get a shell in it. So We have to add it to get it and Because we disabled recommendations We have to explicitly add a couple things that were sort of recommended interestingly when you build an image Without those two things and go to run light HDPD it fails because the default configuration refers to them so That's maybe an interesting thing I need to poke at somebody because that sort of goes beyond recommendations if it doesn't actually start without them I'd almost say that's more of an R depends as opposed to our recommendations, but what does this produce a very small image? It only has these packages in it Three of them are the ones we specified in our image template and then basically we get just the dependencies muscle for the C Library and then like libcrypto and lib adder basically and I believe a PCRE one I think is also gets pulled in by muscle and It's quite small. It's just the application So let's try something a little more substantial another web server engine on X It basically is very similar, I mean that's the goal of having that templated app a container image BB file So we can just basically image install plus equals So That gets us an image with just that I think I'm getting close to running at a time here, but so a couple of notes there bash might get pulled in There's a mechanism that looks for scripts and might pull it in so let's be aware of that If something is actually going to exact things you need to have busy box And because post install scripts get disabled you may have to do some tweaking So I'll try to quickly run through The sandboxing example because this is some of the more interesting bits that I haven't seen too many people talk about So we've been building container images with those previous examples and you know So you can docker import them you could run docker compose or some of the other OCI image tools And then you actually on your target that you've built docker into the image you could fetch them or other tools run C But what if one actually do a factory image that already has some of those like light HTTP or similar type of app container images? How can we do that? So right now it's a bit constrained System dn spawn it seems doable from you know experiments the other things it's not really obvious how to do except maybe doing an Image for the target device that used a post install script to do an import of the image that we could sort of orchestrate that So there's two approaches the simple ish one Jeremy Rosen talked about at embedded recipes last month. He gave a somewhat similar talk and so it basically is sort of straightforward, but has some restrictions on machine distro and libc configuration Between the container and the the target image As well, I sort of did some other sort of tinkering this multi-config feature and newer versions of open embedded Gives us some more flexibility where we can actually mix and match between the container build and the host build Multi-fig if you want to look that up there's this URL here to the documentation and Just note that multi-fig dependencies are a somewhat recent addition and the newest version of open embedded And so I'll mention later this some things happen All right, so I'm down to a few minutes So here's my simple example so we sort of follow on I make a recipe to package up the Image that we previously built So this is the app container image light hdb Httpd I'm using it as a source file and I mean basically Including it into the target root file system And then I have an R depends on systemd container because that's the tool that that package includes the tools We're going to use for systemd n-spot So this is an interesting thing where That's Pretty much all that's required if you bit bake this it'll build the image Then run this recipe to basically stick the image that was built into a package And then that package will go into the root file system and To use it it's as easy as this you make a host image definition and just image install You just add that recipe to it and when you bit bake this you end up with a root file system That has the container already in it Just pretty straightforward. I mean it's not it's interesting. It's a I won't say hack it is interesting I Was interesting to see it and when I looked at Jeremy slides to see if I had any point in giving this talk after his but I Did find that the multi config Stuff he hadn't really talked about because it's quite new So if you look at Jeremy slides from embedded recipes He sort of walked through that example. I use the container class in mind But multi config is a system where you can actually configure for different Basically, it's almost like you have different local comms. It's essentially So in your local comm for you to set BB multi config to list the different config types And then you make a multi config directory and put a separate comp file for each of them so in this one It's a lead to hopefully quickly run through this. I am basically Defining the host has system D. That's what these options do and the container. I'm using my schooner distribution class or distro definition and my machine type and I have I'm actually sending it so that when the container stuff builds it goes into a different temp directory and So here's our multi config version and this is the magic here So this dependency Actually makes it that our do install in this recipe doesn't run until the other configurations build of the container image has completed and So if you look at the documentation, this is the pretty much almost the example that's in the The documentation of one image. We're relying on another image and this also works. It does give sort of the same result The build process looks a little different than the output from bit bake and this is We actually accomplish things a little differently in our image the This is the MC depend multi fig multi config depends And so this is the dependency that we're actually defining and this is an image class We're not actually using a recipe to grab the image. This is and actually we're doing it in the image definition and this is the example that's in the documentation for open embedded right now and and This works the other approach when I actually try and make a recipe to pull the other image with the multi-config dependency it fails so That's something I've started poking. I talked to Richard about yesterday the open embedded maintainer I have to generate some log files and we have to sit down and work at what's going on And I mentioned I had a set tempter That is probably there's some meaning compatibility between muscle in g. Let's see building the two images at the same time Ideally, it shouldn't have to be required that you set tempter and then one of the drawbacks is that I Have to sort of dig down from the top level to get at the image It'd be nice if I could just say tempter and deploy dirt and avoid having to specify all this so sort of a work in progress and And That's my talk. I sort of rushed through there trying to fit it all in I don't think I actually have much in the way of time for questions It is the coffee break. That's a good point So if anyone has a question feel free unless you're going to go for coffee the terminology, right? So yay The Linux dummy are you using the upstream Linux dummy from OECore? I am. Yes. Okay. I just paste its link to a slightly longer version that installs the headers So external modules don't fail, which is useful for Well non container. I don't care. No, but a package like Lighty can recommend an IPv6 module which will trigger Kind of things so I needed it for my containers. Okay. All right. That's good to know. Are you gonna upstream that? maybe And Another question you derived schooner from Pokey and overwrote basically everything pokey does so well, there's a bunch of other stuff in pocky I didn't feel like trying to fit on a slide if I was doing it for real if you were doing for real You probably actually do a full file. That's what he's getting at. You wouldn't actually require pocky.com You just actually fill in there's like another 20 lines of stuff I think you can just delete that and then to depend on what OECore sets as defaults because the faults work nowadays Yes, that's This is a linux foundation talk so So your project is the approved project Well not proofs to stronger word, but it's the It's the sort of linux foundation back project. Thank you, but Any other questions If you can come up to the mic or hopefully So one of the things you said running as a challenge of containers was the library updates can affect multiple containers So how does this address that? That's sort of the same problem. Although in this case my sort of expectation is you'd be building all this yourself, so It's very easy to script builds with with you know bit bake and open embedded is that my imagination would be you'd pull the update And you'd run your build process and you'd recreate all your container images in one go It wouldn't be stuck going out and tweaking Docker files potentially and doing you know compose or you know Whatever you would do to actually regenerate all your different containers This might get it all in one shot potentially if you've are you know Put the care in to set up your build process, right? Because the tools this is all command line tools and it's all pretty much Bit bake this image and so it's pretty scalable to do builds other the Didn't really mention it, but the state cache for Open embedded if you're making these container images And you're only making like a change of adding a package to an app container The builds are going to be actually quite quick after you get the initial tool chain all that stuff builds So if you're just tweaking one library and then just do bit bake of like these a bunch of different images It'll probably be very quick So there might be a benefit Yes, sir. Yeah, one last thing I wanted to just your your open question about if you wait about two to three weeks meta virtualization will have a bb class to build the Oh, yeah, oci containers. So what's your how did you cook up a scheme? It's using and it's using the tools that are in there already. Oh, yeah The image tools and un-mocky and all the things that are there already so That's good. Good. And if you jump on the meta virtualization mailing list, you can find her So you do you end up with a docker native in no oci oci compliance So you can run them with run see your docker. Okay. All right. Cool. That's good here Thank you. Thank you bruce anybody else Well, thank you very much