 Good morning everybody. Thank you for being here in such large numbers even though John is right now giving the current report Which I would have liked to go to myself Today I'm going to talk about a Some things that I've experienced and that I think Are an opportunity to improve how we build embedded systems Let me first say a bit about myself. My name is Agnaut van de Kapella. I'm a embedded software architect whatever that means with focus on integration of Linux based systems so embedded systems I've been a consultant at mind since 2008 and so there my work is doing exactly that I've worked for plenty of different companies in different projects in many different domains and I'm Maintainer of one of the four maintainers of build routes I have to say though that what I'm discussing now is also a surprise for my built-root colleagues So it's not the opinion of build route. It's my own opinion. I Have checked it with my colleagues though at mind. So it is more or less minds opinion so what I want to discuss is some things that are missing when we Use the existing build systems to build an embedded system When I see it my customers How people build in a method system? There are a number of issues that? Always pop up that always have to be done and for which tools exist for which Solutions exist, but there's still some reinvention that needs to be done And there are four things that I want to highlight here There are probably more things as well that are missing but these are the four that I could think of First is field updates So updating the software after deployment Second is persistence and related to that this factory set The third one is how to do manufacturing and Related to that also provisioning So how to get the software on the device Before it goes into the field and the final one is verified boots Now all these topics I'm I have some familiarity with it But I'm not an expert in as I sit there is still some work to be done And so I absolutely welcome feedback during the presentation or afterwards So please interrupt me if you have any remarks or questions or whatever and I'll try to Get this in a more or less organized way to the end Let's start however with discussing a bit about build systems, so I Expected most of you are familiar with some kind of build system to to produce an embedded system there, too, which are very well known and Almost everybody uses them. That's open embedded or yokto I'm going to use yokto in the remainder of this talk and not try to do this distinction between open embedded and poking yokto In built routes there in fact very similar. I use both of them And So what what what is good about them is that are very configurable that? They have Everything that you need to build your embedded system, so they provide all the the tools all the packages that you're going to need But there is still In my opinion some redundant work that you have to do to get to an actual product Then I'm going to do this to introduce two others which go further And that is open-debatee and foundries. I owe there are many others But these are just some some Highlights to this thing. I mean to explain why There these problems are solved but not in built route in open embedded So in open-debatee What they do is basically make some Make a lot of explicit choices about the kind of system You build so there is less freedom for You as a developer to make these choices the choices already been made for you And open-debatee has kind of niche choices. It uses you was it uses progd so it's pretty specific it's also pretty focused on the Routers as the embedded system it can theoretically be used in other contexts as well, but you know, it's like fitting a Was it a round hole in a square peg or something? And there's also some things missing like rust There's no rust support. I think there's also no mess on support or well, there's some student things missing there So it's pretty niche And then there's foundries I owe Well foundries I was bigger bigger than the build system, but it has the Linux micro platform build system Which is basically a derivative of Yachto also making some specific choices like using OS 3 for the update system Using a rightful root file system and combat later to what that implies And so what I'm basically trying to claim in this presentation is that We should make these kind of choices or at least offer these options In an integrated way in other build systems as well Now the thing with foundries I owe and others like that so I think Balena is similar It's very it's tight pretty tightly to their foundries I owe cloud based system and so also it's Pretty much focused on using Docker for application development. So it's so again fairly niche. Well, it's it's applicable very generally But it has very specific choices with which are not appropriate for all systems So the traditional distros are another thing sometimes used in embedded systems, but not often So with traditional distros, I mean Fedora, Debian, Ubuntu and the like They're obviously more focused on the desktop in the server use cases and not on embedded use cases in the way They are typically Configured is to boot an installer and then updates the package manager and Everything is writable. None of these are really a I mean useful in embedded systems Or we actually want to actively avoid these things in embedded systems And then it turns out that they're Not even ideal for the desktop use case so Leonard's puttering made this blog post a few months ago about Basically the same things that I'm discussing now It's a kind of coincidence that he came up with this idea at the same time as me but I've used a lot of his His thoughts to make this presentation So absolutely Advice to to check out this post. It's not directly applicable to the system systems, but the ideas are there so now going to the the missing things in embedded build systems The first one is field updates when you deploy a system every system needs to have a way of doing field updates There is simply no way that you can deploy something and then hope that it's going to work like that forever At least for Linux based systems And differently if it's network connected Usually this is called over the air updates, but yeah, sometimes there are different mechanisms like with USB media that you plug in or something So that's why they use field updates here and almost always the Approach that is chosen is using an AB update. So having Two copies of the root fastest in the kernel device three and other supporting materials And swap between them and then if you you boot the new system if that doesn't work you fall back to the old one And tools exist for this so software update route mender and there are more So there is you could say, okay, this is a soft problem Because the the build systems integrate all these tools so all these three they exist in in both open embedded in in built routes So you can ask, okay, what what is it? Why why is this still a problem? Well in terms of integrating this there are still some some gaps the first one is that we Almost always when you build a system you have to distinguish between the The image that is used for manufacturing which is the complete image which has the the boat or which has a space for both partitions which has a bootloader that is going to To be able to to swap between them so is able to choose better to boot a or B And then you have the update image which contains just the which doesn't contain a bootloader We just contains one of the two systems Which possibly Well, which contains the the only device three kernel in root file system and which puts this in some kind of image format which corresponds to the to the update system that you use for instance for SW update it's a CPIO archive So the Basically what you need to do in your build system is to create both these images Of course, this is this is not rocket science. It's easy to do that. It's just something you Still have to do when you use the build system If it was just this I wouldn't make a presentation about it, but there's more There's also some things missing outside of the build systems For example in you boots You would does not actually directly support this use case basically what's what all these Update systems do is they they provide you would scripts to do the swapping or to choose the the A or the B Which basically means you have to either patch the default environment in your you would configuration or You have to have a custom environment And they also use the ubut environment to choose between a and B that part is fine now the problem with that is to having the script is that it's So much fragile and then when we come to verified boot for instance It means you're it's hard to protect your ubut environment for a verified boot Because the script is in there. So if you if you do the swapping between the two It's it's not signable And but the script which does the boot is in there. So yeah, you have a kind of difficult situation there Plus the ubut environment can be corrupted in that case you fall back to the default environment. So basically you have to make sure that this A be logic is in the default environment So again, there are solutions Without patching you would but it would be easier if this logic was directly in you boots and then there's no risk you Which I Which is semi solved also, but yeah, it needs thought by the developer again. That is that you wouldn't be of a itself Often also need to be updatable So in many cases I've seen that after a few years there is some use case for actually updating you would or updating TFA in the fields which is Annoying if you didn't foresee that when building the initial system On you we avoid the situation is better because there we can just Skip the bootloader entirely and boot kernel directly using the evice stop integration of the kernel and rock for instance, I haven't checked with the update, but rock and directly sits the evi boot order And boot next to basically implement the AB. And so you have nothing there that that interferes And there's also system deboot, which I'm not sure if it's suitable for embedded systems, but it's an option That can also be a choose it only works on you evi systems that can also choose between A and B based on a version number. So you just the file name Determines which one is going to be booted So now back to the Build systems. So yeah for software updates, okay, there is some Some friction there, but it's not too bad But it becomes worse if we take in persistence so persistence because we do this AB updates We can't simply have a writable root file system and put our Persistent data in there because then when you swap to the B partition, yeah, you lose everything which was in the old truth of this Which means there is a you need you need a separate volume for the persistent data a separate partition So that must be that must be created that must be mounted somewhere and it also must be populated so the Usually there are some initial directories that you need on there sometimes some initial files So yeah, there there is stuff to be done there again, it's not rocket science. There are a bunch of solutions for this I have three here and you can probably invent more So one solution is putting an overlay of this on on route itself Which is really easy because you can just act as if the thing is writable, but there are some dangers with that so it makes Roots writable, which means that if an attacker can somehow write to do something Then it can actually modify your root file system can modify any file It's a bit safer to have a limited scope of where modifications can be done Or it doesn't necessarily be have to be an attacker. It can also be accidental, of course In practice, you usually need an in-ear time of this to make this work. I mean you can Mount it directly from the kernel command line, but it's a bit involved in For instance with system D it's not possible because if you have any system D Confideration file in slide GTC It's not going to be read by system D If it's passed on Whatever so in practice it's often twice in interviews. I'm actually not 100% sure if it really always is the case You can also directly mount for instance Slash far and then make sure that anything that needs to be written is in slash far Now often you need some files in slide GTC to be writable For instance a machine ID if you use system D If you use network manager the network manager settings are inside GTC So there are a bunch of things in slide GTC that you probably need also to be writable So then you need bind mounts for those or you need to have sim links Yeah, so there's again stuff to be done there It's not like out of the box And then the the solution that's proposed by by Leonard buttering is to have a writable slash writable roots, but if user read only because if you have unified user concepts then Everything which is executable is under user. You're never going to execute anything anywhere else So the security thing from earlier is then solved by that way by having a read only user And you can write ETC and you can write Var and anything else if you need things like for something for instance to put Like docker images or whatever you can do that But that one definitely requires an interest to to make that work And then as I mentioned earlier, it has to be populated except for the overlay because in the overlay case you can Populate it as part of your the root file system image So you need some something that populates its first boot or when producing the image System D temp files can do this but there are some Limitations for instance, for example, it can't remove files Sometimes when you do an update there are some configuration files that are or or writable files that are no longer needed It's not possible to remove those With system D temp file, so it's it's always a bit tricky So what can Build systems do here it actually goes a bit beyond the build system But I think the build the build systems can be a driver of this It's basically standardizing how persistence is done. So we have all these options Maybe you should just like choose an option and always do that Similar to like we've kind of converge on doing a B updates We can do the same for persistence saying, okay, this is the approach that we're going to take And and really supports that approach And a build system can help a lot there because the out of the box configuration could just support that could just say, okay You can assume that your for instance your roots is with only and var is writable Still leave some things to be done for the developer like if var is writable Any sim links or bind mounts that have to be created in ETC But you know, it can it can help or it you can Okay, it takes it away a bit of freedom from developers But it makes things easier for developers as well because they know what to do Plus there are a lot of integrations considerations Regarding which parts are writable and which parts are Read only regarding this the bind mounting so you need an in-script or an System the units to do those bind mounts And and yeah, the build system is in a very good place It's like a distro basically it's in a very good place to to take into account those considerations So if you install this particular package the the logic to To apply the persistence that it needs Can go along with that because you know if you have a database for instance, yeah Persistence is going probably going to be needed for that database. Otherwise, there's usually not much point So the default installation of the database can make sure that it has a writable location in the persistent partition And then of course it ties in with the update system Not so much with the update system itself, but rather with this a be logic and the fact that I mean the partitioning The having an a a b in a persistent partition available so basically what I'm advocating here is that a build system instead of Generating an image which is the minimal image to be bootable on certain targets is to immediately generate an image which has an a and a b partition and which has a separate persistent partition. That's basically my my Minimum expectation And then related to persistence is factory resets So if factory is it isn't effectively simple. It's just wiping or reformatting the persistent partition But then we basically need to be able to repopulate it and That ties in with Populating that was to be done at first time So it basically suggests that we should do the population on first boot Not in in installer or something like that Because then we can reuse it for factory resets Another thing to take into account is recovery from a corrupted Persistent partition because this is writable. Even if you use something which is supposed to be power fail safe Things tend to go wrong at some points So you need to have a recovery mechanism for that That is typically something that if an individual developer has to take care of it They're not going to get around to do that. So that's I think we are having this as part of the build system can help a lot because then the build system can just make sure that so for instance a file system check is done and That if if the file system check fails and it's not recoverable, it's reformatted and okay You lose your data, but at least the system still works and What I especially want here is to be able to improve so that over time we see From use that's okay There are some use cases that are not covered and then we have a central place with it where it can be improved if everybody Develops their own solution. There is no learning there and you know Somebody falls a problem and nobody else can make use of that. Yeah, I I'm not sufficiently familiar with doctor to be sure but possibly But I don't think that it's also going to reformat in case the fastest and check is not recoverable but yeah, I mean it's it's definitely I mean There I mean we are not in We're already somewhere we don't have to start from zero and I think in these things in general Yachto goes further than built roots and of course I'm more familiar with built roots. So I'm Possibly sometimes a bit pessimistic about what I already exists. Okay Then we come to manufacturing I see I'm taking too much time as usual So I'll go a bit quicker So manufacturing is basically when you Have your board and you need to install an image on it There are again many ways that this can be done, but it always has to be done And so developers always reinvent how to do the manufacturing And so here again same message. I think that there is Opportunity there to have something common Which is probably should be something that is I mean it's not that tightly related to the build system But the nice thing about the build system is that it's a place where people already Converge where people already Collaborates and so it's a good place to to have the collaboration place already Again, Yachto is a bit has something there already the week format which The covers one use case where you have where you flash it to an EMMC or SD cards and it expands to take the whole The entire space Provisioning that is something where I have not seen any solution that Discommon at this point. So the idea is that when you manufacture a device an embedded device You typically need to put some information on there Some persistent info information, which is device specific So it can't be part of root file system because it's different for every device some of these are Well, you can just generate offer first boot like an SSH key That's easy And doesn't actually need to be part of provisioning However, there's some things which need to somehow come from a manufacturing beta database or needs to have a relationship with the outside world A serial number for instance a MAC address that needs to come from your set of allocated MAC addresses Password so that you have a unique password in every on every device, but you still know what the password is So, yeah, there are a bunch of things that either have to come from a manufacturing database or have to be stored in the manufacturing database So this is something that people have to reinvent all the time Both things actually one is the the fact that you need to store this somewhere. So you need to separate partition for it That you need to somehow link to the rest of the system And It cannot simply be the persistent partition because this has to persist even over factory reset You don't want to do a factory is it and then lose your serial number or lose your password or lose your MAC address so Yeah, it's there's They really need to needs to go so often to this Ideally also there would be some common way to communicate with the manufacturing database But I don't think we're there yet because yeah this manufacturing database is There's nothing at all that exists to To cover that so yeah, what can we do? Make sure that there is some common tooling and integrates in a in the build systems Yeah, so here it could be outside, but Yeah, the build systems are a nice place to discuss it And then the final point is verified boots in my slide code screwed up for some reason So basically this is something I don't really have experienced with myself So I've used devices using verified boot, but I never implemented it myself. So here my knowledge is just hearsay But the important thing is that it's complicated There is a bit of Standardization I mean there are two standards approaches for doing verified boots so just Verified boots is basically that's when the system is booted. There is some hardware that checks that the first stage bootloader is What you expect and then that bootloader is going to stay check the second stage bootloader And that one is going to check the kernel and then the kernel verifies the Signatures on modules that it loads and there is a chain That needs to be maintained So the first steps are a bit standardized. So the you you advise secure boot chain It's supported in some bootloaders. It's supported in the kernel Then arm trusted firmware on our trust platform Although there there are still some sock specific thing that has to be done on the very first stage So as soon as you go into arm trusted firmware is the same approach everywhere, but still there is no Standard I mean this this standardization is not really implemented in in build systems And Then there is still also the problem of the root if is itself this I'm going to skip completely Let's just say for the root if is there are plenty of different Well, not plenty of but there are a few different options and there's no one solution that is used by everybody and The options are also not ideal. For instance, it's hard to get working on UBI so hard to get working on non-flash So what what needs to be done there? It's the same thing again Some common tooling is needed and Although this is kind of outside of the scope of build systems its Build systems are a good place to Discuss it the common tooling here is Not so much in doing the bootchain itself because that is in in the bootloader in the kernel in etc It's rather in generating the images for it. For instance for the invariity you need to have an The verity partition I mean the the file system itself in a separate Checksome partition that has to be generated and there's that part that needs to be integrated in a build system and also linking them together. So in your in scripts making sure that the The verity is taken into account And why is commonality important here? It's to have some standard change that we know are trustable So that I mean it's like all security stuff. It's better if this is not in the open in there where the tools are Open and only the keys are secret And so although the implementations are in separate tools the integration It's can be part of the build system. So that's why I think it should be discussed there Plus of course, there's also an impact on partitioning if you use the invariity for instance. All right, that's Concludes what I wanted to cover in this talk So the the main takeaway is that Currently developers developers still have to reinvent the wheel They it's not really reinventing the wheel, but they have to put the pieces together themselves in and there's some not really gaps there, but there's some choices that they have to make which has bigger impact than that you would think at first sight and Build systems can help making the right choices there by maybe offering a few Alternatives but having some flows which are the supported flows Ideally I prefer to have the freedom to Do it differently But make it easy to do it the standard way And so I think that this stuff should be part if in the Yachter case should be part of open a bit at core and not in some external layer Because the the core is where you know, there is most exchange and most Commonality can be exploited But the build systems are not alone. We also need some support from upstream tools like bootloader for updates To to make it all work together. Thank you. Are there any remarks or further input or questions? I'm going to repeat for the recording And I'm not going to remember everything so I'm going to repeat now So the remark is that he was in a similar. He also had a product where you needed manufacturing SD cards that would install stuff But it's an SD card. There's annoying because it gets lost and it's can Break when you push it in And it can actually break the board if you put it in a bit too strongly It's actually mentioned in the slide so what he suggests is that an alternative approach instead of using an SD card and an installer approach is to Use a USB connection. Almost all Socks are able to boot over USB where you have some tool on the host On the PC that loads an image over USB in memory that image is an typically a bootloader But can actually be anything The the image is booted and then in that image you can do anything And for instance from you boot if you use you boot as an image and a specialized you boot You can have all the scripting in there to download another image either over USB or over Ethernet and Flash it to the emmc or not Thank you, were there any further questions. I'm not sure how we are for time I would not like to go So I would love to see Building blocks that I can easily enable One example So Lucas remark is that What he wants is to have the freedom to still Design the system that he likes and so what he wants is that the build system provides the tools the building blocks and even connects them together to some extent, but that there is still That the details in the little Decisions are still taken by the developer My own take on that is that I think that the developer should be free to do that However, I think the build system should have to the and the default of the build system should be something that just works out of the box And doesn't need further configuration But while still allowing further configuration, it's similar to how you have for instance a kernel configuration All the the build systems they allowed you to use the entry dev config But then extending the dev config with your own fragments so I think we should have something similar for All these things that there is something that works out of the box But with the possibility either to just override it and not use the out-of-the-box system Or to add something to add some plugins or hooks or whatever to slightly modify to your needs You were first I Can I can I repeat it already So this is a very good remark actually so I gave basically four points where some Stuff that developers need to do all the time There's actually a fifth point and that is debugging all developers need to do debugging now for debugging You need debugging tools on your target And the typically you don't want to ship a product that includes those Debugging tools so you need a way to have those tools in development and not in final product and solution that they chose was to To install it in an overlay But make sure that it's it needs to be an overlay because it really needs to be in the same place that it's expected like phalgrinds Expects it's libraries to be in a very specific location. Please continue Yeah, yeah, so that's a good remark again so by enabling a Debug tool in in in your configuration You actually have an effect which goes further down the chain. So there are dependencies that it pulls in for instance apparently Valkyrie Changes something in a bootloader as well So you basically you don't I mean if you have a configuration for your production and a configuration for debug The it's not that the debug shows some additional files So it's not so easy to make an overlay. So this is an assault problem basically. Yeah, okay I'm afraid we're out of time. So I can't take any more questions, but I will stay either here or in the neighborhood to discuss things further Thank you very much for your attention and I hope that I gave a similar talk ten years ago about software updates and Voila a few years later. There were a few tools for that. So I hope that in the same way this problem is going to be solved as well Thank you