 One, two, three checking. Can you guys hear me in the back? Cool. So thanks for coming. My name is Lukas. I work on the Formman open source project and also satellite 6 product. And today I'll be showing quite a little hack I've built yesterday and last week, which is really about I have this idea of deploying bare mental hosts using images. So if you have any comments or requests, send me it over to the LZAP at redhead.com. So today I'll do a short motivation part. Why do I care? I'll show some tools. I've built my patch. It's actually a patch that's not yet a pending review, but will soon do. And I'll show you discovery and format. And then I have a demo for you when I'll be showing you not dozen, but actually one host provisioning. I had ideas that were simplified a bit. So if you look around in today's data center, you'll find a lot of virtualization cloud containers, maybe. But it's really the virtualization technology which changed the way we provision systems. It's because it allows something that is different. We don't usually do with bare mental hosts. And that is cloning, image-based provisioning, cloning, templates, quality you want. And virtualization still does support booting via OS installer like Anaconda or Pixi booting. But it allows this nice feature. And it would be nice to have this feature on bare metal. So it's really the cloning that is interesting, that really impresses me, like being able to deploy a lot of hosts using an image. So when you're deploying virtualization on cloud, you have this little problem, chicken egg problem. You really need to start, although the system itself, the engine usually supports this kind of provisioning. You need to somehow bootstrap the system. You need to install the virtualization itself or a cloud. And there are a couple of approaches, like if you install OpenStack, there is a OOO000, OpenStack that installs small. OpenStack that installs a big OpenStack. In Overt or Rev, you have one kind of deployment, which is actually, I think, it's called self-contained deployment, where the engine that the manager really is a VM inside, so it manages itself. So it's interesting, but it always starts with a bare metal host. And what's also relevant, I think, is we do see increase of ARM servers. And these are small, small servers that are not as powerful as Intel's, well, not yet. But while the ARM Server 64 platform does support virtualization, I think it's reasonable to squeeze the power of the system and try to use them as a bare metal host. So it's very relevant, I think. So the question, the big question I made for this talk is, can we just deploy an image on a bare metal host, a Linux image, and the most importantly, will it work? So obviously, we can try. I've tried. We can try it works, deployment of an image. And the answer for the second question is, yes, it works. The wonderful thing about Linux is its batteries include it. You have kernel, and you have all the modules and firmware, usually. And several megabytes, right? But the great thing about it, and what differs this from other systems like Windows, is you don't need to usually search for CD-ROMs and install a third-party of firmware and drivers. It's all there. So basically, some of us have been there when somebody asked, hey, we have this REL 304. It's disconnected from the internet, but it's a production system. Can you please migrate to the new bare metal host? And what do you do? You basically transfer an image. And with some amount of luck, you have the REL 4, REL 3 that can't be really upgraded for some reason on a new bare metal system. So the idea is simple. So what do we need? So first of all, we need an image. So obvious thing to do is to create an image. So you install a system. So you run through the Anaconda in case of federal REL, install it on bare metal or virtualization. And then you create a snapshot, right? Using any kind of tool like in virtualized environments is easier. So I would suggest to do this in a virtualized environment. Two things I want to say is, mind the size. It's easier to create a smallish, smaller image. Typical OS installation consists of several gigabytes. So if you create six 10 gigs image or I'm in volume, it's fine. You can always expand that. And it's also the transfer. I'll be showing it's a little bit lengthy operation. It's doing it over network. So mind the size. And also, important step is, un-configure. Because when you install a system, it will deploy things like SSH keys, things like random seed, seed file, or subscription manager information. You want to un-configure the image. So there's a nice tool called VIRT SysPrep. It's in, I think the package name is LipguestFSTools. And there are a lot of tools starting with VIRT-. It's a wonderful package. I highly recommend. And it will do, you provide an image, provide option, and it will mount the image and do the work for you. So it's great. Option number one, OK. Option number two, there are a lot of images floating around the internet. These are cloud images. So Fedora have one, or REL have few, CentOS have few, and et cetera. And I had this idea to leverage that. So it has been tested. Why not? The only drawback is these are locked down. For security reasons, of course, we don't want to have a lot of un-secure images running on AWS. So you need to provide it what we call a seed configuration. They usually use a thing called CloudInit, which is a small Python script that runs during the boot at several stages and can do things like set root password, deploy SSH key, or what's typical is expand volume and expand file systems. Great. So yeah, you can use that, of course, for bare-battle provisioning, image provisioning. But you need to somehow set your infrastructure for CloudInit. So I think the way it works is it downloads that from HTTP server on your network. So you need to set up this kind of server. You can also work around this by inserting a floppy. If your server has floppy or USB or something like that, it can take the configuration file from there. Also an option. My third recommendation would be use WordBuilder, which is distributed in the very same package I said, libgustfs. It's a wonderful tool that RichW Jones, I think, maintains a repository of pre-paked images that are compressed by XZ. And they are each about two megabytes in size. And you will find CentOS, Fedora, Ubuntu, Debian on all of these. We also have an internal repository within Red Hat that provides Rails. And with a single command, WordBuilder CentOS size 6 gigs, it will download the image if you don't have it and mounts it somehow. It works like a magic. And it will expand it for you. And you can use it. It's a base installation of CentOS with what you can do is you should set a root password, of course. It's locked by default. And so first approach could be you can create an image that will be exactly the size of the bare metal hose you want to deploy on. Second option could be, and that's much better, that that is actually what I'll be showing in a minute, is this command. I've created an image of 6 gigabytes that will install this command. WordBuilder has a lot of options. It can edit files. You can put files into the image. You can do a lot of stuff. One of these is install in a package. So this is Cloud in RPM with dependencies. And you need to configure the Cloud in it because by default it requires some configuration. So using this trick I've learned yesterday, you can say, just do the defaults. And the default is expand disk. And that's it. And we can set the root password here using WordBuilder. So this is a nice technique. So we have image, okay? What to do next? We have some tools to transfer the image onto a bare metal server. So every year we have a right-hand summit. And we are building a classroom of satellite workstations. And every year, I haven't been there, but every year we struggle with this a little bit, walking around with the USB sticks. And if you want to provision 30 servers, 30 workstations won't work. I mean, unless you want to spend three nights doing that. So we've been using, I think, BitTorrent or stuff like that. And I was searching the internet trying to find a tool which we could use. And I found this wonderful tool, UDPcast. It's a 20 kilobytes package. It has been added to the EPOL 7 and 6 last winter. So it's in RHEL. It's great. Of course, it's in Fedora and other distributions. And it works great. You just give it a file or a block device, and it will transfer it to the receiver. Now, the great thing about this is this is a UDP multicast. The protocol has been built from the day one to transfer of a huge amount of data. So it has some kind of error corrections or stuff like that. And if you're doing actually one or is multiple receivers one center, it will send the image just once. And it will take you just one particular amount of time. It has a lot of options, but the menu page is quite short. And it just works. I love it. So we have UDPcast. We have an image. So what to do next? So there's a project I'm working on. It's called Formen. And I'll just give it a minute to introduce you to Formen. Formen is a lifecycle tool management lifecycle systems management tool. It helps you with managing servers and even workstations. We do support. You can install it on Fedora, RHEL, or Ubuntu, or Debian. And with it, you can manage almost any kind of Linux, any kind of, well, Windows. We have also some Windows support. I haven't tested myself. We also do BSD switches, like C-score. I'm not sure which switches do we all support. It can help you with provisioning. It can help you with running configuration management. Integrates with puppets, with Ansible, with Chef, with all of the major configuration management. It does some reports and stuff like that. So it's a pretty wonderful tool. I really like it. Now, one of the features of Formen is Formen Discovery. So of course, you can install systems with Formen. You can pick-see boot systems via network. You can generate boot disks that will allow you to boot systems on DHCP less networks, which is interesting for customers like banks and agencies. And Discovery is one of the features that we provide. This is a screenshot of a very old version. I love it because of the logo. And it also basically describes what it does here. Discovery is a small-life CD we've built. It's based on CentOS, Rael, or Fedora. So the great thing about this, we don't strip unnecessary things. So it contains all the fiber drivers. So if you have a certified hardware for Rael, it will, you know, Discovery will just work just fine. And what it does is puts all unknown hosts on the network, puts into the Discovery mode, and gathers some facts about the system, and send them over network to Formen. And in the Formen, you can decide clicking on Provision button and rebooting them into Anaconda. You can also Kexec into Anaconda if you don't want to reboot them. If you are limited, if you don't have a DHCP server, you can also automate this via Discovery rules and stuff like that. So it's a very nifty tool. And yeah, so it's pretty obvious. I maintain, I co-maintain the Formen Discovery image. This is the image that you basically put on your Pixie server or on your USB stick, stick it into a server or boot from network, and use that. We also do support what's called Pixie less mode when you don't want to use or can't use Pixie or DHCP. And that provides you some nice text user interface when you can configure network, specify some options, and then Discovery host and it will Kexec into Anaconda. So what I prepared for this talk is a patch in Formen Discovery image that will present you a new screen where you can start UDP cast and deploy an image onto a Discovery host. So I'll start with that. The demo is, by the way, prerecorded. Sorry about that. Okay, so what we have here is one VM, it's a LibreVM, that is, it has attached the Formen Discovery image development built I've been working on recently. And I'll just boot it so it will boot up and show you what we call text user interface for Pixie less. It's booting in the Pixie less mode, which is meant to be attended when you will see a menu that needs some input from the user. By default, usually, if you're not putting the Discovery image, it will do automatic discovery and it awaits orders. But this is what we call welcome menu for the attended mode. So from here, you can either enter network credentials and do the traditional discovery and then K-exec that into Anaconda if you want. But I've added this new button here, Image Transfer, which will allow us to basically, all I've done is I've added the UDP cast, so the image is slightly bigger by several dependencies. I've also added a few other tools like SSM, K-part D, so if you want to change the volumes on your server. And thanks, but this is really it. So I'll go ahead and click the Image Transfer button in a second. And what we'll see is an extra screen where Discovery image will ask for interface I want interface to use for UDP, UDP Cast. You may notice that the UDP Cast, I was not on the slide, but UDP Cast requires interface to use if you have multiple interfaces. Unfortunately, I only have one interface over here, network interface, and that's it. So I can decide to manually enter network credentials or I can just go ahead with DHCP. That's what I will do because on my network, testing network, I do have a DHCP server. And this is my work. I just wanted to show you and start some discussion about it is this is a new screen that allows you some kind of ad hoc UDP Cast Image Transfer. Okay, so first we need to start the mode. Obviously we want to spawn either UDP Receiver or UDP Sender process. The Sender is would be useful if you wanted to migrate to migrate existing server onto another server. Volume is obvious. We're in the Libvert environments. We'll use a VIRT IO device VDA in this case. Depending on the image, you want to partition your disk or not, if you have an image that has master boot or grab on it, so you just simply can drop it on a physical device, and that's the case. Now UDP Cast requires a port, which is by default 9,000. It's actually port-based. It uses 9,000 and now 9,001. So it's the communication protocol uses at two ports. It's a UDP, don't forget. On the image, on the developing image, I've enabled through up to 9,100, so you can use any kind of port. If you need more, you can reconfigure that. So good idea and good technique would be to have a UDP Sender on your network, run through Chrome every 10 minutes, let's say. So on port 90, you could be sending every 10 minutes, send OS standard installation you want to use. On port 9002, you will be sending, I don't know, real sticks if you're using that every 10 minutes. And both receivers have some kind of batch mode when it waits five minutes and until no receiver shows up, it will exit. So you could do something like this, okay? And then extra options, you can search in the manual pages. It has a couple of options you can provide. What I would like to provide is two more fields. I'll talk about this at the end. Now, as you can see, this is very simple, you're using interface, I didn't want to do, this is a redhead new T library, you may know it. It doesn't provide a lot of options, so I decided to make a short message. And now I'm doing a UDP send dash dash file and my image on my second VM on that network. And if you want to see more, you need to use a second console when we run by default a journal D. It's a journal D following the system log. And from third console, you can, there's a getty, there's a possibility to log in. As you can see, UDP receiver started and it's receiving, we are doing great job of the image is six gigs, so we have a halfway through, it's doing some very quick provisioning at rate of 1,030 megabytes, oh, dropped back to six megabytes per second. And this is it, really. In a moment, we'll see this screen to fall back to the main menu. I would like to add a checkbox, like reboot after transfer is done, and another checkbox or radio box compressor to use, because the raw image usually contains a lot of zeros and we don't want to spend a lot of time transferring zeros. I'm not sure that Ethernet or TCP UDP stack would handle that. So I'll just reboot manually, there's a reboot option. And we'll see discovery booting into REL 7.3 that was actually deployed using image. So it's image onto bare metal, okay? Last point I want to mention before we start Q&A is that compression is a very good idea to do. So you can of course pipe it through GZ or there's a nice compressor LZOP that is almost as good in the ratio but it's four times faster than that GZIP. So it's a good idea to pipe it through a compressor and so I'll add some support for that. And also make sure that if you're creating an image, you are creating a raw image. You're not creating a VDMK or VMware image. You're not creating a COW2, which is a copy and write format that is only used in LibreD. You need really a raw image that will be just one-to-one mapping to the physical device. So this is it, this is it. We'll see it in a minute. It's rebooting into REL 7.3. I would love to go a little bit further with, Forman has a nice plug-in called remote execution, which is essentially SSH in a loop but with a nice interface, with a nice features, with the support of templating. We have a nice templating system in Forman. I would like to add a template, an integration with a discovered host. So basically what I need to do is add a SSH key to every single discovered host and basically allow remote execution to reach over to the discovered host. And we could do the same using a simple shortcut, right? We could partition disk and start UDP Cast on the correct port. You would receive, if we're running it from Chrome on manually, you would receive the image and that's it. So, yeah, questions. Do you like it? Do you, okay, go ahead. So the point was if we can use the imaging for deploying some kind of a DNS server or management server, yeah, that's, so if it's possible to use this technique to deploy anything in the center, of course, yeah. If you bring in your images, you can just use UDP Cast and start sending them over the network. Sure, it's interesting idea. Please. Yeah, yeah, good question. So what happens if you don't start all the receivers and it will start, okay. So there's a handshake protocol and UDP Cast is a wonderful piece of software. There's a handshake protocol. You can customize a lot of that. So wait five minutes or wait until I have at least five receivers or something like that. So we can configure a lot of stuff. Basically, what will happen is if the transfer is already in progress, the receiver will wait. And according to its configuration, it will wait forever but you can also configure like wait 10 minutes and then exit. So if you'll be configuring your infrastructure to send every 10 minutes to send the image, it will receive the image in the next batch, basically. So that would happen. So this was the last question. Thank you for your time. Please get back to me if you have any comments and hope you'll love this feature. Thank you. Because this is actually what I have done in my data center. We had bunch of computer, they were actually PC for testing purposes. And I have managed to have a setup where I, after installing some software on my laptop, I would go down there. I was actually provisioning those bare metal PCs using a preset configuration and you go to network images. But that setup worked just fine. It was perfect. So I would simply go down with my laptop. I had a mask installed. I had a list of MAC addresses that I am to provision. So it wasn't discovered. I knew it wasn't done before. I went down. Yeah. Yeah, and after I started rebooting, those would simply pull the configuration from my laptop and install the software I wanted to install from them. And then I used actually Ansible to further provision them because I already need the IPs, new Macs, et cetera. Those were installed with my preset configuration. So I think that would be a great idea to include your project, the kind of setup. Well, we have a...