 Welcome again, folks, for Speaker Houdre. Okay. Welcome again. My name is Antonio. In this next talk, I'm going to talk about device provisioning at the Edge, specifically what the team did over the past two to three years to make it possible to provision a device for the Edge. I'm an engineer at Radat working for eight years. Right now, I'm working with a team of eight engineers on Ralph for Edge. This is the slide, like somebody who's already seen it. I've already seen it. We build Ralph for Edge. We take care of Fedora IoT too, along with the upstream community led by Peter, and we develop and maintain all the technologies that make Ralph for Edge, Ralph for Edge, and Fedora IoT, of course. For instance, the FDO, secure device onboarding, Greenwood that I've shown earlier, and the simplified provisioner, which is the topic of this talk. Not in picture, unfortunately. The agenda for this talk, we're going to talk about how do we provision a device at the Edge, what the requirements are for this process, or at least what we've been gathering in the field, and based on experience by talking to many hardware partner and vendors too, and which tools have we developed in order to meet these needs, and what we've learned so far, and what's next to it. At the beginning, I really struggled with all of this. Peter Robinson has been preaching this division in the flow. At the left side, you have the build phase. The build phase is when you actually build Ralph for Edge or Fedora IoT. That part for us has been done through ImageBuilder. You basically specify the package set, some configuration option, maybe some files to inject, maybe some kernel arguments to tweak, and then what you end up having is an artifact. This artifact can be a raw image, compressed raw image, can be an installer like the simplified provisioner, or it can be on a condit itself, can be an MEI, anything that boots can be made at this phase. In this phase, you're not playing with the hardware, with the device, you just build the artifact that you want to put on the device, or that you want to install with. Then this topic is about the middle layer, the middle balloon there, and it's the provisioning part. The provisioning part is the part where you have 10,000 devices, and you want to effectively and efficiently flash an OS onto the device and make sure that it's ready to boot. Ready to boot here means so many things. The provisioner or whatever provision the system can just flash the image onto the disk, or it can run and integrate with things like secure device onboarding, can talk to the hardware for security sensitive, things like talking to the TPM, storing keys, all of the things that has to be done at provisioning. The analogy here is, for me at least, the way I understood it is the Microsoft part of the world. I know this is a Linux conference, but pertinent, but still. Microsoft can ship pre-built laptops that already have windows, so they provision them very early, and then they ship that to you, you power them on, and they onboard. Usually you have a Windows key or something connects back to the central Microsoft data center and says, okay, this is actually a system that has been registered with us and it works. So the provisioning part is usually done at the manufacturer, where you have, again, thousands of devices. You want to install the very same image of the operating system onto them and just simply ship them or sell them. And then there is the onboarding part. Still a reference to the next talk in this room, because it's about the onboarding part for us that is done securely by two mechanism, mainly. We use FDO on ignition in the Rene and Sarita. We're gonna talk about that later too. For this talk, we're focusing on provisioning. So at the beginning, we tried to gather some of the requirements of what this process should have. And the very first one was it has to be unattended. Nobody has to interactively type on a keyboard or watching a screen, like a zero touch, yes. So we don't want that. We basically want somebody, for instance, to grab a USB key or boot the system, the provisioner remotely and just install on a fleet of devices. And the number here can go pretty high. Like, again, we're talking about thousands of devices that you want to flash in the very same way with the very same base operating system. And so the first requirement for this provisioning part was unattended, hopefully fast, which is somehow different from normal installation and installers where you're actually there, clicking the mouse, selecting the disk, maybe doing something else. You want the provisioner to go as fast as it can and to be able to scale to thousands of devices and just works, just install the system, ready to power on. And that's why the second requirement here is should support remote installs. At the beginning, we were talking about Pixi and other remote install mechanism, Pixi is insecure. So what we've settled up was just use UFI ATTP boot. So that has been a requirement from the beginning for us. You basically have a web server that serves the NITRD, the kernel, a couple of kernel command lines to drive the actual installation and then install it, or you provision it. And yeah, the other requirements are similar to this. We want to be able to tweak something into the installation or the resulting system. An example is the console kernel argument, maybe a thousand of devices where the kernel, yeah, the console is on TTI, yes, something, or TTI something in general, and you want that to be the same over the same hardware. So the provisioners should support something like this. Partitioning is also one of the requirements that we got, partitioning and injecting files and directory. We're not doing that right now at the provisioning time. We're doing this mainly at build time, but we're gonna find a solution maybe later in the future to also do this at provisioning, because we've been asked, maybe, maybe, maybe. Yeah, that's why I'm just mentioning. Especially on the edge. Exactly, I mean, I'm listing this there because those have been discussions and points that we've taken from various parties, customers, partners, users, and for the ROT community, maybe. So we have these requirements and as I said, these two specifically are done at build time. Interact with the hardware like TPM, that's something that of course, if you have an installer booting, that's doable, so we've been able to do this just fine. And it has to be secure. Many definition of what this means for us, even just interacting with the ROT TPM was enough. And yeah, so the idea that we came up with was having a, Peter called this a minimal, tiny INITRD installer at the time. And what that meant was just create an INITRD, pack something in there, like a kernel INITRD, some tools to actually write the image onto disk, and any other tool that we need for this. And then make it also bootable over HTTP boot. And of course, flashing an image onto this, that's the very first requirement. You have an image, you run the provisioner, you wanna provision the device with that image. And the other one, as I said, that we come up with was the ability to inject kernel arguments, interact with on-boarding. As I will show, we made the simplified provisioner, the provisioner that we use as a DRUCT modules, and that means that it's pluggable. We have a couple of other DRUCT modules, like the FDO-1, which is part of the on-boarding, that we can quote-unquote easily plug-in. So the whole system is made pluggable by using Eclipse DRUCT for us. Just to support encryption, right now there's also, again, the part where we already built the image encrypted, but in the future that may change, like we can do library encryption at provisioning time, maybe that's, there will be a thing. Right now we're doing that at on-boarding, though. And we're building the image already encrypted. And lastly, this is one of the things that we get tasked every time. These taller should extend the raw image to the full disk capacity. So we have a raw image, maybe it's compressed, uncompressed, it's 14 gigs, something like that. We DD the image, we basically flush the image onto a disk, which is 100 gig, for instance. And at that point, what we've been asked is, okay, but I need to grow that root file system to take on the full disk space on my device. Because of course you wanna leverage the whole hardware that you have. And so this provisioner has to support something like this. So we initially created the simplified installer, but that was causing a lot of confusion. Peter is already looking at me like you shouldn't say that word. And that's when we called it the simplified provisioner, just to distinguish between the phases. So this is the simplified provisioner that we came up with. We're still calling it simplified installer somewhere in the code, especially in OS build, but we are not struggling, but we're working towards removing that, you know, warding, basically. And it's, yeah. And as I said, it's a tiny ISO, like we've been able to pack everything we need into an ISO that contains just the kernel, the nitramefs, and the raw image. So there is no real system running there. We boot the nitramefs, the kernel, we have the image. We're gonna work with that. All of the simplified provisioner, as of today, is driven by Dracop modules and system services into the nitrd. Basically, this allowed us to, again, plug as many pieces that we want to control the flow of the provisioner itself. For instance, with build, we flash the image and then maybe we wanna run FDO to do other stuff. Maybe somebody wants to mount the root file system and inject some certificates, things like that. By using Dracop modules for us, this was the way to make this possible. And the reason for having an ISO, like a normal ISO 92 something, is because all of these components, like you can actually unpack the ISO and then you can have this component served over HTTP with HTTP boot and then this stuff can just boot and install the system and run the nitramefs and the kernel. So what we need to do, of course, is before all of this, once we built, well, at the point we built the simplified provisioner, we of course need a raw image to work with because that's gonna be the golden image that we're gonna provision over 10,000 devices, for instance. So we need to build this raw image. We're using image builder again. And what we're gonna do, as I've shown in the other talk, is first build an OS3 commit that contains the base system. With that, we're gonna just create a raw image and this is when partitioning and encryption are taken into account. So at this point, image builder is gonna create the raw image out of an OS3 commit for us. What we do with this image, we embed that into the ISO so that the simplified provisioner code can use it, uncompress it if needed and flash onto the devices. Okay, so the way we produce the installers of the provisioner, as the slide is wrong, is through image builder again. We have created, over time, a couple of stages. That's how image builder works by having stages that do something onto a root of us. For the simplified provisioner, we created, the team created a bunch of stages that we put together in order to basically have the resulting ISO that you can see in the image. What we do is, so image builder is basically creating the raw image. It's compressed, so it takes up less space and then it makes it bootable via, you know, ISO Linux, FEI and everything else and then embed also the unit LamaFS and the kernel. So all of this for us is done with an ND blueprint and command from image builder. So now onto the requirement on how do you write the image to the disk? Like you have this nice ISO, there is a raw image and now what we need to do is actually writing that on the disk when we run the simplified installer. So for this reason, we have chosen to use Coros installer. For those not familiar with it, it's a project from Coros, the Coros team that is widely used in OpenShift and it's basically the way to flash an image onto a disk. You know, in its simplest form, that's what it does. But it can also do something else. It supports embedding ignition. As I said, it supports compressed raw images so you can embed a raw image that is compressed and then when it writes it down to the disk, it gets uncompressed on the fly. That's a huge win for us, of course. We can keep the ISO itself more concise, basically. It has encryption support. Coros installer can be run as a Dracut module itself. We've created, the team created the Coros installer Dracut project and that's a way to run Coros installer in a Dracut module. And being Dracut and being just command lines, tools, we were able to integrate all of this using SystemD, basically. That's the framework that we're using. It allows us to order units and execution the way we want, so we're using SystemD there too. So, now Coros installer is really neat and the way we made it work, it's even so, I think, because the whole execution is driven through kernel arguments. And now, this is, you know, this is, again, neat because if you wanna boot this provision remotely, then you wanna be able to drive the full provisioning through something like kernel arguments, because, you know, every device is gonna come up with this kernel command line that's gonna execute this. And so, we were able, by using a SystemD generator to leverage the SystemD generator in just parse the kernel command line and then say, okay, we want to install the image onto that VDA, this is a virtual machine, maybe your hardware has dev SDA or something. So, if you have a thousand devices, all these you want to install on dev SDA, you can configure this via kernel arguments and then boot this via HTTP boot so that every device is provisioned in the very same way. There are other things that can be tweaked, like where the image file is, can be remote. Actually, for HTTP boot, you want these to be, you want the image file kernel argument to be image URL, so you can optionally, CoreSysTaller can optionally fetch the image from a remote location every time. They may be slower, but it works. It can be made to work. And so, this is how we achieve conditional execution in an actual. So, yeah, this is basically what I said. Apart from parsing the command line, just to derive the execution and configuring, you know, the installation device we can also tweak how we behave post install. You know, somebody can install the, well, provision the machine and then maybe you want to reboot. Maybe you don't want to reboot. Maybe after provisioning, you want just to get the hardware, pack it, box it somewhere and then ship it to the address. So, you know, anything can be, well, this flow can be tweaked by post install kernel arguments that we provide. Another thing that CoreSysTaller does, our drag module is mounting the ISO so that if the image is local to the ISO, we're gonna, you're gonna, we're gonna have the image available into the, in E3D so that we can flash it onto the device. And then at the very end, this was the system degenerator for CoreSysTaller. We're just gonna run the CoreSysTaller service, which is just a command line saying, CoreSysTaller dash dash install. And then the target disk, post install behavior, where the image is, all of that. So we've been able to achieve this with a system degenerator basically. And as I said, you know, the CoreSysTaller service, this is just a snippet. But at the end, it's gonna just echo a command and then it's gonna get executed by system D itself and it's gonna run. Like, I hope this time the demo works because it's really tiny. So you can see the actual command line, CoreSysTaller install and everything else. Yeah, so all of this kernel command line options can be configured by an image builder. So I'll show a couple of blueprint that, you know, that we use daily to build the image so you can configure the install device, you can configure FDO and ignition, there are various options for FDO, you can configure certificates, you can configure the manufacturing URL and other things that we've been explained later at the FDO talk and ignition talk. For ignition, you can attach and a kernel argument to say, this is where you're gonna fetch the actual ignition configuration. Again, you can configure the target disk by a blueprint, all of this together by an image builder. As I was saying, this also integrates very well with FDO, which is secure device onboarding, and I'm repeating myself, but FDO is also a drag-out module so we can order it after we write the image to the disk, we can add FDO run and do whatever it needs to do with an installed system, like registering keys to the TPM and other things that, you know, at provisioning we do with FDO. So for this demo, like I'm using a normal virtual machine, what I've done is I've created a simplified installer based on Redline 2 in this case, but all of this is coming to Fedora really soon because we got the latest bits of everything we need, so expect it in the next month or so. All of this can be tested, it's pretty exciting, as you can see Peter laughing, it's my introduction. So all of this can be tested with also Fedora. And so I've created a simplified installer using this blueprint. And as I was explaining earlier, like you can see I can customize the installation device, but this is a simplified provisioner meant to install on the dev VDA disk, that's usually, you know, Kimu and like. And then what I wanted to do also was add a first boot kernel argument for ignition so that the first boot of the system also runs ignition. So this is really easy to digest and grasp, it just follows the image builder blueprint options. Another thing that I can show is also the FDO blueprint. Well, the FDO snippets in the blueprint too, you can see we can configure a bunch of options for FDO itself, in this very simple case we configure the manufacturing server URL, maybe in a factory you have 10,000 machines, one FDO server, you create a simplified installer for that server so you can have the very same image that installs on 10,000 devices all at once by just building this image. And then there are a bunch of security sensitive options too. This example is running insecure for various reasons, but we're not building this, this is just to show that there are aspects and pieces of the blueprint that are made just for the simplified provisioner. And this comes from the integration we had in the past with the image builder folks, really? All right, seems like 10 minutes left. So what I'm gonna do here is run a simplified installer provisioner, sorry Peter, to install on a virtual machine. You can see somewhere that, no, it's red dot now, actually it's red dot three. So you can see here this image, this image unpacked, you have already seen it just the FEI directory images with Kernel and NetRamaFast, the ISO Linux, standard folder, and then the raw image. So if we go back here, this is the simplified provisioner booting. You can see the very same similar configuration, kernel options that I've shown in the slide. So we're gonna install to the FBDA and the image file is run media ISO, that is something that CoreS installer drags itself, it's gonna mount for us and then CoreS installer can install it. So if we boot this, at some point it's gonna, yeah, in a moment it's gonna show, I know this is tiny, hopefully everybody can see that, but you can see right in the first line 64 that there is just a normal command line. It's CoreS installer installed where the FBDA as we've configured and the file is there. So the image file, compressed image file is there. Then there is insecure tool, I've explained like just for demo purposes. You can see all of this went unattended. I just edited the kernel command line just to show you, but otherwise it's like 60 second delay and then this thing is gonna boot on itself, it's gonna run unattended. At this point it's still writing the image to the disk and you can see at least, this is a virtual machine, so it's pretty fast, but we've been testing this on FitLets and other devices like I think Intel Nux 2, ARM devices too. So it can be slow depending on the actual hardware that you have, but in general it's a fast process. And you, yeah. So yeah, so now it's gonna reboot into the actual Relforage system. This is not really important, but I wanted to show is that the whole process can run unattended, it's pretty fast too, which was one of the requirements that we got at the beginning and it's really tiny, just an ISO, you can put it on a USB stick to install manually, you can boot from the network in order to streamline this provision. So yeah, what we learned so far and bear in mind we're still learning, requirements are coming probably daily, depending on who we talk to, what their actual needs are. So what we learned so far is that many of our users slash customers don't wanna use raw images and this is legit to some extent, although raw images buy us something like security, like IMA and things like that. So we're trying also to be opinionated into this whole process. We're saying, this is probably the best way, it's a bit arrogant, but trust us. Like depending on the various talks that we had, exactly, that's changing that process. Exactly. That's an education process that we need to do. Exactly. And so they just wanna use, they're used to have an installer like Kanakonda. They go there, they script that with kickstarts. If you're familiar with that, Chad is definitely familiar. So you script that, you want just an installer, you use the installer on a fleet of devices or that can be also be heterogeneous. So like even different too, and they want to have the logic to derive which system it is on the kickstart itself. That is cool, that works, but again, having something like the single raw image that can be FS verified and all the security mechanism that can be done, what we're saying is better for security reason too. But as Peter mentioned, it's an education process that we need to do with the customers and with users and the community to help them understand this is definitely a better way. I think because you basically have two single devices to deploy to guarantee that a dead device across tentability and change, so that it's... It's an education process that, this is what we're fostering basically right now. This is probably a good way to provision your edge devices. As I said, if you don't have a raw image that is already packed as a whole, things like IMA and other security sensitive things and processes do not really work. It's very difficult to make IMA work with Anaconda. There have been talking, but we're not there yet and probably never will be for Anaconda. So this make it easier or at least this is what we've been told. And then the other things that we've learned is that many people and users and customers want to have partitioning at provision, at the provisioning time. Again, they're used to have something like Anaconda where you can actually script the kickstart and say, I want var here, I want attc here, whatever. You can use our imagination there. And with the simplified installer, what we do is basically pre partition the raw image. And so we're gonna tell you like, this is the layout that mostly works. If you wanna tweak it, use the image builder blueprint and then you can have var on a separate partition and again, use our imagination there. You can do whatever you want. For partitioning specifically, there are a couple of areas that we thought about that we can do this. One of it is ignition. Although ignition does partitioning live on first boot, but it's often pretty, pretty much expensive. Meaning right now, OpenShift uses this, but of course, OpenShift doesn't have any resource limit. They can have like 32 gigs of RAM. They can have tons of CPU power. And this doesn't work, of course, for the edge. Maybe for some hardware works, but you know, if we- Also, OpenShift, in the data center, you bring all sorts of other things like that on the edge. Like if you lose an app and you prove repetitioning, that device is close to the first step in the process again. And so it's, there's a lot of factors on the edge. You know, sharing the resources, various other bits and pieces don't matter. Doing things on the edge without a human to interact with, so, bit to zero touch is very complicated. And the more that you can take out of that and do it in the image build time, so that you're provisioning the device, you've got minimum vital decisions that need to be made, which means minimum amount of things that can go wrong. The more likely you're going to get a successful sharing that you can then sort of customize more so with something more reliable and accessible. Yeah. And yeah, so at the end of the day, leveraging Ignition in OpenShift is okay, as Peter explained, for us at the edge is probably a no-go to begin with, Ignition can do root reprovisioning, so it can completely change the layout of the device system on the disk. On OpenShift, that's fine. At the edge, not likely. So yeah, so what we came up with was maybe we need to support partitioning for the raw image at build time. So that makes sense. That's something that's on the roadmap already. Maybe you can have a blueprint option just for the simplified installer to, well, not for the raw image to partition it the way you want. Again, maybe you want bar on a different partition. Again, as I said, we can look into on-the-fly reprovisioning of the file system, although that's tricky, or we can do first boot reprovisioning. Maybe we can optimize Ignition itself to avoid, I don't know, reprovisioning a 100 gig root file system because that would be impossible to fit into memory too. So those are just scenarios that we're gonna run through and think about. The last one is something that we've been talking in the past. We didn't get any traction. We're at the conference. If you like the idea, I want to contribute. That makes sense. Right now, the biggest thing that we've been told is how do I test it? Relfar Edge and Fedora IoT easily. I don't wanna build the image. I want to have something that boots and I can just navigate into it and understand if it fits my needs. So right now, we don't provide a live image, but if there is any interest, you know, fraction from the community itself, we can also explore that. And that is meant like the slash F-Cos, well, start F-Cos systems do provide a live image so that you can actually jump on the system before installing it, maybe create a template of the installation that you wanna produce with chorus installer and then just boot that. So that is something that's not officially on the roadmap, but that I think we would be keen to have as an input from the community and maybe some out of time. A couple of links here, just, you know, from chorus installer to some reddit documentation on how we do simplified provisioning with FDO and ignition. That's it.