 Okay, so thank you everybody for attending. I'm very happy to see you all here. My name is Ana Guerrero Lopez. I have been contributing to FISOWA for like 18 years now, mostly to the Debian Project. Probably you have seen my name before, it has been something to do with the Debian. I have been working at Colabora since earlier this year. And today I would like to present something cool that we have done at Colabora that is called DevOps. So let's start with the presentation. What is DevOps? So DevOps is a tool written in code that is used to create Debian images and you also can use it for Debian derivatives. So when I say images here, it's raw media images, docker containers, hard disk images, developments, it's a root. Everything you can imagine. So DevOps creates an image following a series of steps that you are providing in a recipe file. Currently we have 12 actions that you can use for creating the image. And one of the most interesting things is you don't need to be root for creating the, for using DevOps and creating the images. We will see later why. So from the very beginning when we designed DevOps, it's because we needed something that we could plug easily in our continuous integration system. So from the very beginning, the focus of this tool has been easier to work in this kind of workflow. It's also modular, so if somebody wants to implement a new action, it shouldn't be very difficult to do without interfering with other actions. The recipes are written in YAML, but it's not fully YAML. We also will see later with an example why. And of course you can run DevOps in any Linux system, but if you are not using Debian, probably the best bet is using a container. So which architectures are supported by DevOps? A priori, you can use it in any architecture that is supported by Kimu, and it's also available in Debian. However, we only have tested it in a handful of them, mostly ARM, MIPS, X86. It also can work potentially with Rift 5, 64, that is the Debian port of Rift 5. I say potentially because the Rift 5, 64 is still not fully functional in Debian, the pieces are coming together. It will be in the future, but not yet. But the day is there, you will be able to create a root affairs with DevOps. So this is maybe the most complicated slide for the presentation. DevOps is using something called fake matching under the hood for allowing you to use DevOps without being a root, and also for set up in the Kimu so you are able to create images in architecture that are not the one in your host matching. So fake matching is going to create and spawn virtual machines. Setting up Kimu system using the slash USR from your system. So once you have that, DevOps is going to use Kimu user emulation for being able to run binaries of foreign architecture. What I mean with foreign architecture is probably your host matching is in the 64. Well, I am talking in Debian port. Maybe you want to create an image for ARM. So that's what foreign architecture means on this context. And then DevOps is going to have a root privilege in this matching that you have set up. Something also important is for fake matching to work, your user needs to have permission to a slash dev slash QVM. This is actually one of the things that is doing the trick that you don't need root privileges. And also if you want to have good performance, you really need that. So also it's important to explain what is not the boss. So it's not a real system. You are using Debian behind or your derivative distribution. If you want to put something that is not available in Debian, you will have to do your own package, you will have it in your own repository and you will install it later in the image. Also it's important to say that it's not the official way of installing Debian. That's the Debian installer. Maybe the result is the same, but when reporting books or a problem that you phone is something important to specify. So probably if you already tried to build a Debian image in the past, you have realized that there are plenty of tools that you can use. There are so many, I cannot really make a comparison of them. But if you are curious, we have a page on the Debian wiki where you can check all of them and find if there is something better for what you need or not. So who is using the box? So there is the Apertis project that is an infrastructure tailored for the automotive needs and other things. It's an infrastructure, Apertis, and one of the things they are providing are repositories of packages because it's a mix between a wound to a Debian derivative. And they are using the box for creating the images they are providing. So if you are curious to see the images, they are in that URL. They are also publishing the device recipe they are using. Other project is Kernelci.org that is a service that is distributed, is testing Ustream Linux kernel. Basically what Kernelci does is build many trees of the Debian kernel, makes boot tests, also have some test plan for testing some subsystem that is also running tests. And the box is being used for creating these images that are used with the test for testing the kernel. If you are interested in the Kernelci project, I am giving a presentation on when is it about it. When I was preparing the presentation, I decided to check if somebody else was using the box out there. And actually I found very nice examples. The two first examples there with the Debian ECME system are two embedded boards. And they have a good example of what you can do with the box. The third one is from Plasma Mobile. Plasma Mobile is a project that you use in Debian. Actually they are showcasing it in the QD box if you are interested. They are using the box for building the images for testing their systems. So they are doing continuous development. And the box was for them the perfect tool for creating the images and testing that the laptop using Plasma Mobile was working fine. So I think after the presentation, the best we can do is doing a small example of what you can do with the box. So this will be like a minimum recipe that you can create. So here we are targeting ARM half-load that is like the Debian port for 32 bits. Half-load. Is there was strapping there are two steps. The first one is there was strapping a Debian such a route. We will see later what there was strapping is exactly. And the second one is just making a target with the such a route. So this is like a minimum you can do. Of course, most of the time you want to do way more than that. Here you have a small example of how the application run. You can see that every step there was strapped and packed is marked here when this is starting. So it's easy to see which step is being run every time. So thing also interesting is you can see that here the image is taking something like five minutes to be more or less. And then there is a powering off of the virtual machine and suddenly there is two hour more. It's because the images are created in a virtual machine with UTC time. So I am two hours ahead of that. So our first example was like hard coding. I wanted to do an image for ARM in Debian in stretch. But what happened if you want to do the same for several Debian releases or even you want to one? And you also want to change the architecture. So you can use parameters. This is actually one of the cool things we can do because we are using the Go template package. Text slash template. So we might want to run for example Devos and say sweet stretch architecture and 64. Or we might want to create a image for Buster that is the next Debian release, the next stable for ARM 64. So we have an example with parameters. So here we will be using the architecture parameter and in the top we are putting the result. So in case we don't provide a parameter in the command line, it will take the default one. The same for the suite, the Debian release. And then we have a few parameters that is the image name, the torbol, what's going here. That we can create from the other two. So we will have every torbol with a different name depending on what we are building. Or the nice thing of the templating system of Go is we also can use conditionals. So we can have some actions that are implementing depending on the parameters we have provided. So if we have a parameter called type equals to tools, this step will be RAM. But if not, Devos will skip this step. You can do this at level of action but you also can use as a small part of it in this example. So I say in the beginning that there are 12 actions of Devos. All of them, all of these here. I am not going to explain all of them, especially the all three ones kind of require a new presentation. Introducing what is all three and so on. In our former example, we use Devostrap and pack. So Devostrap is something that is very known for everybody who is using Debian to try to develop a bit on Debian but probably is not known to everybody. So I wanted to explain it quickly. So basically what Devostrap does is allowing you the installation of a Debian based system from scratch. So it's going to load all the dev from a Debian mirror and it's going to create a Seache root within. We are using DPQG or APT. And this Seache root is going to be like a Debian based system. I mean that you cannot boot, you cannot do anything but it's a Debian Seache root. If you want to know more about this in the Debian Wiki, there is a Deostrap page that is explaining everything quite well. And I think we can go to explain the most relevant actions. The first one is APT. I think this one is, if you have used Debian, it's kind of obvious. After you have a Deostrap assistant, the first thing you might want to do is installing packages. So with APT, you put action APT and you list all the packages you want to install. Here is the example from the beginning where I have replaced the second step for APT. So there is actually an option that you can use that it recommends. My default in Debian, you are installing a package with all its dependencies and all is the recommends line. You can use the recommends parameter to avoid installing the recommends, especially if you want to create a small system. If you are a user, you probably want to install. If you are using Debian in a desktop machine, you probably want to install the recommends. So here in the example, I have done something different to the sample from the beginning. In components, I put main, country, and on free. So I put on free because I want to install free where Linux. That is not available in the main repository of Debian because what is Debian actually is Debian main. Everything else, country, and on free are like two extra components that are there for people's convenience. But they are not part of what is Debian proper. So often we want to install free where, so we need to have them free for having it available to APT. So don't load. Obviously when we are building images, often we want to don't load files maybe from the internet, from other place. There is a command for doing that and also integrating them in the system. For example here, I am downloading a target of firmware and I am telling the box that it will be available with the name of firmware. Also, there is two options that is unpacked because especially we want to download a few files that will be compressed. The type of compression we are using. It's important to say that this action is not going to place the files inside the system we are creating. We are just going to refer it and if we want to copy them in the system, I will show you later how to do that with the Overlight action. So we want to install the file we have downloaded. Maybe we want to copy files that we have in our system into the new image. So we do that with the Overlight action. So Origin, we use this when we want to create something that we have downloaded before and we have referred with a name. If not, we don't use Origin and we only need to use source for pointing out what we want to copy. And then the destination directory in case we don't want to copy that in the root directory of the new image we are creating. So this is a bit complicated to see only with the action, so the best here is an example. So we have here three actions. The first one is the download. And then we have two actions that are overlay. So the first action is exactly what we saw before, is the loading firmware from somewhere. I'm naming it firmware. The second action is overlay where we are saying, oh, we want to copy things from our firmware that we have downloaded before. And then we say, okay, so inside firmware, there is a directory called firmware, firmware version. We are using the parameters boot. And we want to copy that in our image in boot firmware. So for example, we have downloaded, I don't know, firmware for the Raspberry repeat. We use copy exactly this, the boot directory from the tarmac into boot firmware. The second overlay action is actually copying what is inside an overlay slash auto login directory that we have in our directory next to the device recipe. In this case, we are not specifying any destination. So whatever is inside the overlays auto login is going to be copied directly into the root image. So for example, with auto login, we are copying a system file. I have to put a slash, USR, leave, blah, blah, blah, system D until the file inside auto login. We have to create the full tree. So action run. Actually, this is probably the most useful action because everything that you cannot do with other actions, you are going to run. So you cannot run a common OS script. Inside the target file system that you are creating or in the fake matching host. If you are running something in the target file system that you are creating, you will be root. So you actually can do everything you want. If you are running it in the fake matching host that you are using with fake root, you cannot do everything because it's like you are a user. And you will have a label, a few variables. So you will have the variable root deal that points to the root of the target file system that you are using. You will have artifact deal that is a directory with all the file that you are using for creating your image. Image that will point to the image if you have already created something. And recipient actually for most of the time is the artifact deal. The directory that you are using from the directory you are launching develops actually. So, again, this is more easy to see with an example. So, yeah, again, where you are creating a file system, sorry, as such a root. The first action is run inside the image that you are creating. So that's what I say is such a root, true, because it's been running in such a root. And with this command, we are creating a list of the packages that are installed. Given that we want to keep that file after we have finished running DevOps, then we need to copy it outside. That's what the second action does. So, in this case, we say such a root false. And we say, I want to copy the list file that is installed in the root of the image to my artifact deal, because the artifact deal is what I am going to keep after the DevOps execution. A second example, you also, of course, can use variables in the commands. So, in this case, I am just using a command. I wanted to do a script. I will only have to pass the variables, well, the parameters to the script. And it will work fine. So, what if you want to create images partition with DevOps? That's what we do with the last two actions that are image partition and system deploy. So, with the first one, we actually create the structure we want to have. And the second one, most of the time, is use action and file system deploy. And we'll copy the image we have created in the partitions. So, sample to finish. So, here, actually, I am only creating one partition, but it's enough to get the idea. So, I am naming it operatives, because it's an example taken from operatives. So, this is the name. This is the size. This is the kind of partition I want to do. And, well, this is easier to understand if we check first partition and then mount points. I am defining a root partition using EXT4. It starts at 0% of the 4G. Ends at the 100%. And then, here, I am saying, okay, I want to mount this in slash root. This is the root partition, and, of course, I want this to boot. And the second action is pretty much saying, okay, copy the image we have created before into image partition. So, I have a longest sample here. I would like to show mostly to see a big example. So, here, I have a directory with all the overlays I am going to use. So, see, I have an overlay for installing the firewall config. Firewall, no. It's the firmware, sorry. Another one for the networking, and another one for your boot for the options. And then, I have a few scripts I am going to run. I have all of them here on the scripts. So, let's see the full recipe quickly. Because if you follow the presentation and look at all the steps, it's all the steps we have seen before, but now making a big recipe. So, here, I am saying I want to download the dispersion of the firmware from GitHub. And I want my final image to be called like this. So, here, I am downloading everything from GitHub, from the Raspberry P repository. It's the first thing I do. Then, I will strap a Debian Buster, install all these packages, especially U-boot and the kernel. When I was using this example, I found a problem. That is the U-boot version that was in Debian Buster. It didn't boot with the kernel 4.18. So, I have to add a script, downloading the U-boot version from Debian Experimental and running that. So, I am not going to open the script here, but if you are curious, this is in GitHub. You can check it out later. I have a script for set-uping of the user. So, when I boot my image, I will have a user-user. Setting up a hostname called Raspberry RPA3. Copying the overlay for having network. So, basically, it's copying a system unit. This is doing whatever that needs to be done after for having a work. Here, this is the same example than before. I am copying the firmware. I am removing some files from the final image that I don't need. So, I just remove them from there. Here, I am copying the U-boot files for the Raspberry P3 into bootfingware. I am adding the configuration for U-boot. And here, the configuration for U-boot menu. And also, I want to access to my final image by using the serial cable to USB. So, I am adding this to the kernel command line. Here, I am running U-boot, finally, because I have all the configuration I need to install the package. I am running the U-boot update, so my image will boot later with U-boot. And here, I am creating two partitions. So, you will see the first one is for the firmware. And I only want it to be 64 mega. That will be called firmware. And then, I want a second partition. That will be the root of my system. That will start at 64 mega and will go to the end of the image. And, of course, I want this to boot because it's where the kernel will be. In this case, it's going to be a partition type of Microsoft DOS. With this, we tell DevOps to copy in the image partition that we have created or imagined. And then, basically, the last two steps are for creating the blockmap file with a VMAP tool and for compressing the final image if you want to. So, if you get DevOps from the end right now and run this, you should have a good table image for the Raspberry P3, 3 plus. It's the only device I have test. I have another sample here also for the Raspberry P3 for the firmware model. If you are curious, this one is installing a stretch, but you will see it's pretty much doing almost the same. So, going back to the presentation. So, the future plans. So, we don't have any great plans for DevOps in the future. We don't have a roadmap or plans of things we are going to do. We are basically developing it when we find something that we need or when somebody actually contributes something. There is a file called to-do file, but actually that name is wrong. It's not things that we want to do. It's more like things we would like to do, like ideas more than to do. And, of course, we are continuously improving the documentation, but there is always something to improve here and there. So, if you want to know more, you have all the relevant links here in this page. Follow there. Eventually, here in Go DevOps Recipes, we'll go the example I showed before. If not, in the meantime, if you want to check, it's GitHub, Anna, DevOps Recipes. And that's all. Thank you very much for your attention. If you have any questions, please. I've got one. So, that looks great. As you know, I've taken interest in this for a while. The thing I've always found missing and the reason we got multi-strap was the ability to use two repositories to make your image. It wasn't entirely clear whether after you've made your image we can then use apps on another repository or, as well, multiple set repositories to produce all the pieces. You want multi-strap, I guess, yeah. Exactly. And Debootstrap doesn't do it, which is really annoying. So, that's one of the things in the to-do file. Right, okay. Actually, if I have time tomorrow, it's one of the things I will implement because it will be super useful. Right now, I am afraid that we don't have that. Okay. Do you know, Olam? A reply into work is you can run the bootstrap and then you get the minimal Debian system and then you can add the repositories you want and then you can, through a script, and then you can install whatever external packages you want to install there. Yeah, he wants multi-strap, that's possible. Yeah, but... You have a minimal thing and then you start growing from there incrementally, adding all the crap you want to add. Do you mind calling me? Yeah, thank you. What do you suggest for building your own application? I mean, I have my custom application to install on the root of the system. I have to, now a bit on my own and to produce my package and install it later. So you want to know how to make a package of your application so that later... Yeah, if I can put it all together. Basically, you need to learn how to make a Debian package using the helper, so following the tutorial and the package. If it's only for your own internal consummation and if it's not for Debian, actually it's very easy to do because when you are sending a package for Debian, you need to check plenty of things. Then you can set up a repository using Reprepo. Once you have that, you only need to add that repository. You can do an action copying the URL of your repository and installing it from there. If not, if you check my example, you can do as I have done with variable. You use W get your Debian package and you install it. If it's only one package, you don't need to set up a repository. You just copy the package in the image and then do dpkg slash e and install it. If you have more, you want a repository. I'm also thinking about having a continuous security system for my application to build the package. Maybe my application use also some other Debian packages that depends on... You will have to install the dependencies and the dependencies before and install your package. Maybe even if... Let's say if I want to have the continuous security system for my application to produce the Debian, maybe I can build the image with your tool and having at the end, before completing the image, downloading the sources, building the package with a run action and copying then the dev file somewhere and deploy it somewhere on their procedure or something else. That's what Apertis is doing, in fact. They are using Debian as a base. They are adding some packages from Ubuntu. They are adding other packages. They are creating images. Okay. One more. Do you know the Debian tools that I'm using like yours? Use KVM or QM to emulate ARM mainly, I think. Do you know if there's some plan or something like that that I can use to do the BIOS on a VM or something on the cloud or something like that? I'm using Jenkins. Okay. You use Jenkins, but I want you to have the build machine because I bought a physical Debian machine because now I have to have a physical Debian machine because I need QM and so on. If you are running devos from a Docker container, you can run it everywhere. Okay. But I cannot run it on a cloud VM or something like that, I think. Yeah. If it's installed with Debian, you can run it directly. If not, for example, you have a cloud instance where you just pull the Docker container and run it. Okay. Thank you. Hi. Two questions. The first one, are there some benefits of using DevOps over Meek Image Creator? One, so I can't read it. Okay. And the second question is if you could compare this with... There is a tool from OpenStackGuys. This image builder. Yeah. It does, I think, almost the same apart that they are more focused on the cloud than embedded machines and couldn't we achieve pretty the same just using that one? There is a wiki page I put on my presentation that I don't know. There are, I don't know, dozens of different tools. Some of them are still maintained, some of them are still not maintained and there are some of them specified for the cloud. Some of them who are the devian installer modified, they are also like helper. I mean, yeah. I know some of them, but it's not the tool that you mentioned, sorry. Yeah. Okay. I think that this image builder is really similar. They are also using YAML. They also have this layered idea, so I think it might be worth checking it. I'm happy with the most, but yeah. Maybe we can install some ideas. Okay. Any other question? Thank you very much.