 Welcome all to the talk about discuss booting of Raspberry Pi's and more in generic the Raspberry Pi 400. You should see them here. Who's of you, for example, or has Raspberry Pi style at home and works with Raspberry Pi's? A lot of good. I like them. A good company. By the way, I have two of them here. I got two more there and a little bit of luck. We're also going to use them all in the presentation. As you know, these are actually very rare and the big guy actually at the outside of the room that's actually protecting me in my Raspberry Pi. So, might you get any of these? Okay, good. Let's get started a little bit. I will tell you also a little bit about myself. What I normally, actually this ramp on course is actually a little bit normally out of my comfort zone. I'm not really doing a little bit of all the things and that kind of stuff. Actually, this was a really nice project that I picked up actually with my sons a time ago. Okay, let's see. First, the introduction. Yes. What we're going to do is I'm going to tell you where I'm. Then I'm going to tell you a little bit about the Raspberry Pi 400. I do see that a lot of you people actually already know it, so we'll keep that short. Now, actually, okay, what were actually the reasons why we started this project? Because actually the idea was quite nice. These Raspberry Pi's stay are really powerful. These things are actually eight gigabytes for CPUs, ARM based, that kind of stuff. All the world is not Intel only nowadays. Luckily, have other processes like these arms, like the wrist files and that kind of stuff. One of the nice things actually that got introduced in 2020 in November were actually these Raspberry Pi's that were encased actually enclosed in a keyboard. And they're actually really nice actually to use as a, you know, computer for classrooms where you actually don't need to have the big laptops or if you don't want to have desktops, etc. So I'm going to tell you a little about it. Then I'm actually going to show you and tell you actually how we actually did solve the problem, that kind of stuff, the multiple ways actually to do that. There's still some quacks and some open issues. I'm going to try to demo it. The nice part is that you see I did bring a lot of stuff actually here. A little bit of a mini data center, a switch, an odroid, all things ARM by the way. So let's see if we can work again. Like with many demos, this is my only option. I do not have a backup plan, so we're going to make it as good as it gets. Good. And then actually for any questions that are left. Good. Let's first answer the existential question, who am I? Well, I'm Pascal Van Dam. I've been a trainer also for Linux Foundation there since a long time. Started being a trainer actually in 1999 with Uli Peckert. Started actually delivering courses for HPOX. A little bit of the old iron with the property unixes and that kind of stuff. I'm also the architect in this case for ATOS. I'm doing actually in the Netherlands their Linux and their Kubernetes work and that kind of stuff. Next to actually being in Linux for a long, long time. Actually being there since 1991. So even when we had these floppy disk, for example, I was already onboarding it. I did some kernel hacking and that kind of stuff, created a process resource manager and encrypted file system, that kind of stuff. Nowadays, I'm also doing trainings on Kubernetes and also creating, for example, the operators. So the deep technical stuff, that's reason why now this is actually a little bit of a challenge for me, but I really like it. Kernel Internals, Node.js and lately actually developed two courses for Go and Rust. And I'm proud father of four sons and the two elders actually now have joined me in the company. They're actually now at the university. And one of the reasons actually that we started this course is because my elders actually decided during a sabbatical to join me in the company and we had, okay, let's do some really nice and we'd like to do something where we have some common technologies like iSCSI, like NFS, like TF2P boot. And how can we actually integrate that? And for example, in a project to boot Raspberry Pi remotely. Okay, let's see. So Raspberry Pi four and four hundred. And they're actually the four was introduced in 2019, a successor of the successful range of the Raspberry Pi, the two to three and that kind of stuff. What's really nice actually was that the Raspberry Pi four is a lot powerful. Actually Raspberry Pi as we have it now is actually powerful, for example, than laptops that were actually introduced four years ago. The arm a based nice is actually in 2020, we even got a gigabyte model. And that really makes it nice because the next project actually had okay, I really love these Kubernetes actually these Raspberry Pi's in a Kubernetes environment. What would be greater just to have a Kubernetes that I can just switch on, add new notes actually as Kubernetes of as Raspberry Pi modules, etc. So that's also what we did. Now I've got one gigabyte ethernet port in SD card slot, two times USB three and two times USB three. We'll see a little bit later on is that that SD card is actually a little bit of the Achilles heel of these Raspberry Pi's. Now what we have there is actually we need to put in SD card. And even for example, if you're running Kubernetes on this kind of the condition of writing, for example, on the LCD database will completely vary on this card. So that's not really something that's really nice. So we need to need something else to see a lot of people actually nodding. So you know that one. The other one came a little later in November 2020. And that was actually in a Raspberry Pi four, but then enclosed in a keyboard was actually based on an upgrade of the C. In this case, the C zero stepping and allowed also for higher frequency. So instead of the base frequency, we could actually step it over 200. Even if you were lucky, even more, the Torback was a little bit, they only had four gigabytes of RAM. Actually, if you open up such a Raspberry Pi 400, for example, what you see is a major heatsink with a computer on it. And one USB port less, because the keyboard has actually also needs to be routed. And also this one had an SD card slot and actually ideal for running Linux desktop. The only thing what you actually do is you plug in a monitor, you make sure that you put your SD card in it, a mouse, and actually you're completely in business. Okay. Oh, well, also, if you have any questions, let's keep this as interactive as possible. Just raise your hand and I will give you and you can actually ask me and I will try to answer. Good. Okay. So the shortcomings. The first one I'm always running to is these things do not have a real time plug actually built in. And that means that when we switch them off, they actually have a way to store the time in the file system. But for example, if the shutdown doesn't work, go very well. Actually, it means that the RTC, for example, the time in your Raspberry Pi will be somewhere in the past and that kind of stuff. Would be actually really nice if we would have a battery backup RTC there because that would make life a lot easier. For example, with running these things as Kubernetes nodes, for example, that is really mandatory. In the other way, we already told that the SD cards, they actually proved to be very fragile. And the problem is that the SD cards are normally being used for things like cameras and that kind of stuff. So actually, sequentially, right, there's not much reading on that one. If we use them actually in computers, and we have a file system, like XFS or xd4, for example, we have, for example, a database actually hammock on down, you see a lot of wear and tear and actually, yeah, the cheap ones, they actually die within a few weeks. And the longer ones, they die when you do not want it. Good. So the Raspberry Pi foundation on itself also, of course, I found out that the SD cards were not ideal, actually. So they also actually improve something. The nice thing about the Pi 4 is that it has its own eProm, its own boot eProm, actually, that it's actually bootloading that one, they can actually upgrade that one fairly easily. And gradually, the last firmware version that we have now is one actually from the second of September 2022. But gradually, in the past, booting from USB storage was added, actually, two forms of USB had a mass storage and the normal one, booting from network, using TF2P was added, that's actually what we're going to focus on in this talk. Lately, booting from network using a TF2P was added. And the nice thing about this form actually is that you can now, for example, in print as a scribe store your SD card with the OS image, without actually having a, yeah, have to put a for example, in the second computer and write there your SD card. That's actually nice. We're also now working actually on that to see if we can actually boot it directly so that we don't need to have it written first to SD card. We're actually using it, but we're still in in the investigating that one. And the other one, but that's only actually for the CM4. And the CM4 is actually the third version that we have of the Raspberry Pi. They actually allow us to use NVMe devices and to boot from that on the Pi 4 and the Pi 400. Unfortunately, that's not possible. Good. Challenges. So actually, why do we do this? So now, first things that we are delivering courses and doing to COVID, pre just COVID, I would actually allow to travel a lot and go in a lot of time in the island into India, South Africa and that kind of stuff. So actually there, you work with people and also there, the systems actually that we use for classes are mostly on the cloud and that kind of stuff. But classes like the kernel internals, that's nice to do that actually on a public cloud. But yeah, the machines get rebooted a lot. We need to do box stuff. A lot of the clouds actually have their own kernels optimized and that kind of stuff. So actually, this works much better with physical machines. So actually having a Raspberry Pi 4 or phone that actually really nice works with that. We have another class, for example, kernel management, exactly the same that works the best if we have a small machine that we actually can do in the window, actually would stand not any ballooning in the public cloud, etc. And this was a nice one. Yeah, next to being in the IT engineer, I'm also doing stuff actually with children's coaching for children. And I'm also, for example, teaching computer science or ancient Greek for classes of highly gifted children. And actually, to have them work, for example, with logo, actually, these Raspberry Pi 400 are really actually nice to set up. But actually, we'd like to make it very simple. So that the teachers also can do that. So I would like to have a fast setup. Actually, plug it in. Or once it actually plug it in or plugs in, it should actually know, okay, I'm, for example, this Raspberry Pi 400. And we're going to, for example, boot it up with a Ubuntu distribution. Another one, for example, we're going to plug it in. And as well, actually, start up with a Pi OS distribution, for example, and then load it, for example, with a kernel rust implementation, etc. Central management, and important, not relying on the SD cards, or USB storage. USB storage works better. But still, the little things actually also classroom, for example, they get actually, people lose them and that kind of stuff. So we really would like to do it without any storage. So the second project was actually nice. And actually, I offered this also to the keep calm and that kind of stuff. What really would be nice, and actually also to teach this, for example, in Kubernetes classes, you know, in the public cloud, we can actually do something called horizontal note scaling. That means, for example, if I would like to scale up my cluster, add all the extra worker notes, I can actually do that. And what would now be nice, if I could do this actually with Raspberry Pi 4, the nice thing about these guys, actually, not this one, but this actually recipe by inside, we can actually put a PoE plus hat on that. So it actually means they get fat by the network. Not only that, that actually means that we can start them up, power them up over the network, we can power them down. That means actually that I could create, for example, a controller inside of Kubernetes, just a process, which I'm going to say, I would like now to scale up my cluster and it automatically actually starts up, opens up one of these PowerEE ports, for example, this one powers it up, waits actually till the Raspberry Pi automatic consensus boot request and boots with the image actually to directly being able to connect and add another note to it. So actually to deploy Raspberry Pi Kubernetes cluster on demand, scale up and scale down scale up by just not doing more than actually enabling the power on a PoE plus port and scale down actually by disabling the PoE. So of course, first, we need to do that in a nice way. So we're going to shut down the note and actually scale it off and et cetera, and then actually power them off the port. Good. So how does a Raspberry Pi actually boot? Good. One powered on. What actually happens there with the Raspberry Pi is that it starts up. You see these red flesh light flashing. If it flashes too much, then you know that something's wrong with your SD card or with your boot process. The boot EEPROM, sorry, I forgot an E here handles the boot order that actually since 2020, we are actually able to configure that. You can use in this case the so-called RPI EEPROM config. I will show you that a little bit later. And then you can actually change the boot normally. It's actually default on starting from the SD card. And what you actually can do is put a number there. For example, this is F241. And that sounds a little strange, but actually we're going to read it from the sketch right to left from my side. If I'm in the front of your view. One means boot from the SD card. Four means boot from USB card. And one means of the two means boot from the TFTP from the network. Why do I do that? Well, quite simple. This will always help me, for example, to intervene in the boot. For example, I need to flash the ROM outside of the network boot and that kind of stuff. Or I need to do some other stuff. If I actually have first let them connect to the network, as soon as I actually find my TFTP server, it will directly boot from that one. And the other actually means try it an infinitive time. So quite a little try this complete boot order. It will do one, four, two, and again, one, two, four, etc. Always. So that way we can make it a little bit field proof. Good. When the network boot is actually the one that's been selected, then a TFTP request will be done. And actually that's really nice. I will show you that in a few moments. Let's see. I don't have one. I have one half. I'm going to show you that because I need to change my screen. The TFTP request will be done. And I will actually look for a directory that's actually named after CPU serial number. You can find that one. I'll show you when I actually have my laptop ready. When you do, for example, a CodProc CPU info, you will see there the serial number. And that serial number, that's actually where the TFTP will do, actually, and check where it needs to file. So with normal PCs, for example, they advertise, for example, their MAC address. And it actually looked there for the boot files. In this case, it will actually look for the serial. You can change that if you would like to, but I really love this one because this is actually means that I can recognize each and every individual pie for that one. So it downloads the config.txt that actually consists of some of the device tree declarations, for example. It sets, for example, what's the speed of the CPU is. It sets, for example, the which kind of HDMI mode it needs to start up, etc. etc. Then it also downloads command library text that will actually tell which kernel needs to be booted. That will actually also tell, for example, what command line options are needed. These are very important because the whole goal is literally that we are going to try to boot the Raspberry Pi completely over the network without any local storage. So things that we could use, for example, full NFS. The problem only with NFS is that it doesn't scale to is a bad NFS is not something, for example, that would use for my etc database of Kubernetes to run on. So actually we chose in this case for ice because he discs and also going to show you that. So actually in the command line context, you will find actually the command line for the kernel to boot up with the target ID, the target initiator from the ice cozy and actually where to find in this case the ice cozy provider, for example, plus a lot of other stuff, either to have a quiet boot or to have a more noisy boot and that kind of stuff. But also, for example, a panic equals 15. They'll actually tell, for example, if something goes wrong and within 15 seconds, we can actually not get a boot root volume for example, then I'll actually restart and we'll try again. Okay, after that, the kernel will be loaded, the initial RAM disk will be loaded and then actually the kernel will be started, will boot from the initial RAM disk and then actually later on this will actually pick up the ice cozy disk and boot from that further on in the second stage boot. Okay, so the kernel is downloaded in it are these downloads, the kernel is started. What are we needing for the solution? Okay, I need to boot because the whole boot principle of the Raspberry Pi is actually to boot off the USB. So what I need is a TFTP server. Of course, I can use one from Peter Andrew, for example. Another one that's really nice actually to use is the one that's already available in the NASMASC. The good part is actually that will give me one ADHD server. So I get my IP addresses. It will give me TFTP and it will actually give me a caching DNS, my DNS server. So also all the pies, for example, that get booted automatically get a name in my DNS. So that's actually one solution, which is really small and actually fits them all. So that's the reason why we chose actually for DNS mask. And the nice thing is actually this one. And again, this is not a Raspberry Pi. The reason why I chose this one is actually this in Odroid. So definitely still are. This one has the ability to have an NVMe disk actually inside. So he has a two terabyte NVMe disk that will actually have all the logicalized QC disks for the Raspberry Pi on it. If that would be possible Raspberry Pi, for example, I could also use a Raspberry Pi for that. Or for example, I could use Raspberry Pi and add, for example, a USB stick or for example, a SSD disk to it. Good. Now we need to use a nice QC server. That's also in this one. And I'm going to demo that a little bit. I'm actually going to show you actually how this works in a few moments. The QC server actually serves the disks out, the root disks of all the Raspberry Pies that are attached to that one. The nice thing actually we're making use in this case of logical volume manager two. And of course there are more ways because important of course is that we try to create a kind of a template for our Raspberry Pies to provision. For desktops, what we actually pick up is a Ubuntu image for example. We need to clean it a little bit. We need to actually add some extra drivers to that one and then actually we need to copy everything over to the logical disk to the iSCSI one. We created actually a ansible playbook for you for that. So I'm actually going to show you that also. And another nice thing of actually using LVM2 is that I have the choice for the iSCSI actually to have my devices, my logical devices being false. Or for example having them as logical volumes. Now we chose actually to use logical volumes in this case. And one, quite simple, it means that I do not have the falseness interface that I have less over at. And the second part is really nice. With LVM2 for example I can actually create snapshots of existing logical volumes. And that means for example I can have a template and I can actually snapshot this template and actually have another Raspberry Pi for example work from that one. And that will cost me one less time to copy it. And the second one actually that will give me a faster startup and actually less space of users of storage space. Good. Then there's another one that's actually nice because what we have now is actually a system, a Raspberry Pi that can boot from TF2P. It will get this kernel because actually that will be an a TF2P actually file system. I'm going to show you a little bit of a drawing and a little bit. And then actually we do have the iSCUSY device. That will be given. That's great. But there's one problem because these Raspberry Pies they have this and depending on what kind of distribution you're actually running. If you have an example of Ubuntu, then your firmware files, your kernel and all your device refiles, they actually boot firmware. That means that boot firmware is actually copied to your TF2P server. That's nice. But if I'm going to actually, OK, if I'm actually going to upgrade, for example, the kernel, it means that these files, the RAM disk, the kernel and maybe even the device refiles, for example, they get updated. But TF2P is a one-way road in this case. So I need to make a way actually to copy back to. So actually what we do, this boot NFS of this boot TF2P, boot firmware, we actually also mount that as NFS. And that means, for example, if I'm going to upgrade the kernel, it will automatically put these things on the NFS file, shared file system, and that's actually also directly available on the TF2P. Then another one, and that's less needed for these ones, for example. But we use that a lot for the Pyforce, for example, when we actually deploy them in a Kubernetes environment. I am making use of CloudInnit, which is actually a tool that has also been used by the big public clouds and that kind of stuff. And with that, we actually can post config or Linux distribution. Now what we can change here, for example, is your hostname because I'm coming from a template and that means actually that from the start, actually, all my Raspberry Pi's will have the same name. That doesn't work very well. So what I'm going to do is actually for every Raspberry Pi that I'm going to deploy, there will be a special directory with the two famous CloudInnit files, user data, metadata, metadata containing the hostname, for example, user data, containing all the items that actually configure the system, for example, changes in the users that need to be made, adding users, adding keys that can stop. Eventually, for example, adding logos, banners, issues, maybe some software, all that kind of stuff is actually possible. And also here, this CloudInnit, what we actually do is in the command line, we actually refer to CloudInnit, in this case, to get it from this web server. Okay. The next thing that we need to do, this is nice. Now, I'm working with a template, but how do I get this template? So actually, what we do is actually installing a Raspberry Pi on the normal way. For example, with an SD card, we need to do it once, or for example, with a USB stick, we boot it up, we install the software that we need. So for example, if I have all my classes in Go, a Rust, for example, I need stuff with a VI, for example, completely configured for Rust. I need to have the latest Rust compiler, for example, and all that stuff we actually put in. If I need a special desktop, for example, for Kits, there's also what I'm going to put in onto it. That will be my base. But we also would like actually to have a distribution that is stable. So I would not like to have, for example, a single file system. I would like to have everything we need and clean in Logical Volumes. So actually what we do then is we have an Ansible Playbook and that will actually run on a remote system. And we'll go to the Raspberry Pi and then actually mount the iSCSI device and then actually rSync almost everything over with some changes. And also for example, they're making sure that the Logical Volume Manager is being used there. So we have far on a different file system. There's a lot of different file system, file log, file log security, all that kind of stuff. Okay. We chose two distributions to actually work the more. Also with, for example, the Ubuntu, the previous ones, like 21.04 and 22.10. This one is actually now the most stable one that we have. Pi OS 64 is also very nice. The nice thing with Pi OS 4, and unfortunately I cannot demo it because the HDMI cable doesn't like the output of my other Raspberry Pi 400, but that one, for example, is running KD right nicely. And also with some of the acceleration actually that the GPU inside of the API 400, for example, does. The reason, so for choosing for Pi OS 64, for example, is using the acceleration, for example, of the graphics what you have, for example, in Mozilla and Chrome, sorry, not in Mozilla, what you have, for example, in Chrome and that kind of stuff. For Kubernetes, we use the 22.04 most. Okay. So, let me see. I need to add a little bit. My laptop refuses to mirror the screen. It could be that my laptop is a 4K and this screen is actually a 2K. Let's see if I put it to the screen. Then I have to make it a little bit smaller. Yeah, I'll pull it just for this. Make it here a little smaller and need to do it a little bit with some apologies. Okay, what we actually have here, this is, for example, in this case, the Odroid. That's actually doing the function of, yeah, excuse me, the server, the CloudInnet server and the TFTP boot server and of course also DNS mask with DNS and that kind of stuff. With our lab, we actually have this on a, running on a NAS, for example, we're using a QNAP for that one, but you can actually use anything that I actually did is port all the functionality in such a small system. Okay, first thing that we actually have is the TFTP boot and that will actually, in this case, provide the kernel and all the boot files and also all the device tree files, for example, to the Python or the Py4 over the network. The other way around, we also have one line because quite simply, when we upgrade, actually, the kernel, all the upgraded images, the upgraded init ID actually needs to be written back in the same directory here in the TFTP boot. Then another stream, actually, that we have is the CloudInnet. So once the kernel actually has been booted, in one of the first stages, actually, what it will do is it will execute CloudInnet. That will actually look for the user data file and then actually apply all the customizations, actually, to the image. And of course, the ISCSI, which actually is providing the real root disk, actually, for the system. Okay. Okay, Quicks. Yes, we still have them. What happens, and we'll see that actually, when I demo it, we'll give two demos, one actually for the Py400, to see how it actually boots into Ubuntu and that kind of stuff. We'll check then also on the Odroid, actually, how that works. And the second one will be, we'll try to actually boot up two of these guys. They actually will completely set themselves up, so they're completely empty. The Py400 actually here has already been set up, so we don't see too much of them, actually, that it just works. But Quicks, that we have, for example, is in the CloudInnet. Sometimes we run into a race condition. That means that most of the times, actually, and we might see that here with the Pyforce, for example, we see that it does have actually access to the Nginx set, to the web serve, actually, for the CloudInnet files, but still isn't able to do that and actually will fall back then to the fallback resource and actually means that we will still be stuck with the old hostname that's actually on the template. The way to fix that manually is actually just logging on to the system and then, for example, the way CloudInnet is cleaned, for example, and then that will actually fix that one also. And another one that we have, for example, is, for example, Ubuntu is there notorious for. It has the option, actually, to do all the updates, actually, of security features. And sometimes that doesn't go really well, but actually leaves out certain drivers and what, of course, is disastrous for or set up is if the LVM2 driver is gone or the exclusive driver is gone, which actually means that the system cannot boot again. We can easily fix this also by just copying, for example, the files of the previous backup files on the nasty files, in this case, on the boot supplier. Good. Let's see if we can get it demoed. Okay, I need to do this a little bit, switching one and forward and back. Again, apologies for that, but let's do that. This is my Raspberry Pi. I'm going to actually now unplug it from the power. So here it is. The second thing that I'm going to do now is I'm going to plug this one off. Now we are, I plug this one in. Then the Raspberry Pi will get power. I believe we'll get an image in a few moments. Yes, here it is. I'm going to, this is the new boot screen action. This is a fairly new, actually, firmware. This one has already the HTTP boot in it. We're not going to use that. What I'm going to do is I'm going to use the escape key here to, and again, it's not looking really very readable. But what it does here, you see here the boot order. In this case, the USB, then you see here twice again for the USB, says that it twice the SD card. And now it actually goes into the boot process of the network. You saw the TF2P boot. You saw that it actually got its own IP address. It did find the IP address of the TF2P server. And this is, of course, the Chroma screen that we have. And it's actually that now it's going to boot the kernel. This shouldn't take too long, but every time that you look at it, it's taking its time. Okay, this means that the kernel has been booted. And in a few seconds, we should actually see that the video adapter is actually getting screen output again, okay, there it goes. Again, here you see it booting and goes a little bit fast, but you have to believe me. This is actually the cloud image. You can see that here with the pluses and that kind of stuff. On the top of it, you could have seen, if you read very carefully, the attaching of the ice because he devices, actually the volume groups are actually being activated. That's actually now coming here. Now it's going to start up the desktop environment. Now you see the cursor of the desktop environment. And here we have a nice 22.4 and again, you can check it. There's no USB cards. There's no SD card in it. This is just booting actually just from this cable, from this switch and from that apparatus. Okay, what I'm going to do now is I'm going to switch back again to my laptop and I'm going to try to get on board of the Odroid. So actually you can see the locks of the DNS mask and I'll show you that. And we can see, for example, what happens on site of the Raspberry Pi. Okay, let's do that. And again, it's a little bit unfortunate that the screens don't work. Here we have it. I'm typing a little bit in the blind. So forgive me my typing mistakes. So I have to look up here sometime. Now I'm actually on the Odroid. Let me see if I can make this a little bit more readable. Okay, we have to do it this way. Again, here, this is just the normal login screen what I'm going to do now. I'm going to, for example, yeah, it works better. This is readable. This is my DNS mask print code. Now we'll see some special stuff actually in this one. This is just the address I'm listing on. This is actually the internet device that's actually here. Here, I have a private range in this case from two to 100. This is will be given for 24 hours. This will be the router, which will be that thing itself and putting up the NTP server. The good part is this one normally does do have a battery backed up RTC clock. Unfortunately, it did not survive the travel with the plane, so I need to check in that one. The DNS server, the net mask, et cetera. Then what I do here, I actually put in the different, so here, we have, for example, some of the DSP hosts actually going to fix, for example, the MAC address to a name. I'm not going to use that here. This one actually does, but that doesn't work for me. Here, I'm going to log all the communication actually the DNS mask and, for example, for the TFTP protocol, that's very handy. I actually see what's happening to see if actually the Raspberry Pi picks it up to see which files, for example, getting sent and that kind of stuff. For example, if I didn't prepare my TFTP boot, for example, for the serial number of the Raspberry Pi, I could see that that it couldn't find a file, for example. Here with, I'm actually enabling TFTP and also setting the directory. We're going to look into that a little bit further. And actually what I'm doing here is this needs to be specifically there for the Raspberry Pi. Normally, we do not do things, for example, for x86 boots or UFI boots, for example. But by using this, PXE serves as zero Raspberry Pi boot. We can actually recognize or have TFTP recognize Raspberry Pi is actually to do a TFTP request. So here, this is the TFTP boot directory. Unfortunately, this is a horrible color actually to read. What you can see here, these are actually the serial numbers of the CPU purchase. So you see here, a few of them, you see sometimes a little bit of a different name. There's actually where I copy the snapshot, for example. Where can you get these? Well, quite simple. Let's see, for example, first, what the IP address was that we booted from. So, it's of which Pi actually booted from us. Let's see that. You see here, the TFTP boot process with all the options that have been sent. What I'm looking for is actually the IP address. Let me see that. We'll leave this one came in at 712. If nothing is actually down the other line. That's my start. It's a request for a 7.3. So most probably we have 7.3. Okay, now logged on to my Raspberry Pi in the DTCN05. And what we can see here, for example, if I pick up the FSTOP, you can see here nicely, see here, these are actually the file systems that have been created here important. You see here the NFS boot of the distribute with exactly this number. And that means actually that the serial number of this Raspberry Pi, you can see that here is indeed this number. You have to strip off, by the way, the first numbers. So here, 669, 2D, E2F, that's actually my serial number. And so every Raspberry Pi, when it boots off DSTOP, it will look for files in a directory that's been called after this serial number. And if we go, for example, in this case, this is a Ubuntu one. So you can see here, all the files that are in the DSTOP, well, that doesn't look like it. You see here, all the files that are there in the DSTOP boot directory and by investment synchronized, actually, with the boot server. And you see here, for example, for an update for a new eProm. And this is also possible. We could, for example, put a new Pi eProm, for example, a package here. And when that's actually been picked up by the boot, it will first upgrade to eProm and then actually do normal boot and that kind of stuff. You'll find the run disk, all the overlays, a little bit under that, you'll find again here, we have the command line context. That's also nice, actually, to look into that. This looks impressive. We can see here, in this case, a ZSTOP enabled, that's actually for the newer Ubuntu's, they have that one. Here, it's getting actually interesting. The console is TTY1, root of S type is XFS, we did change that. This is actually telling me that my root disk will be on a VG root, LV root. This is my initiator and this is my target. So actually, there I can get my ice cruising devices from. In this case, learn one, for example. I could also have multiple ones like this, for example, for the Kubernetes one, we actually add an order disk for the file containers of our Docker, for example. And here we have Dan, the root bytes, the elevator deadline, the root weights, which is actually for root to be active. Fix the RTC, this is the fix actually to get the latest time actually on the file system when the system actually stops. Panic is five, if we didn't face in seconds, we do not have a mountable root disk. Now we actually go to reboot. And this one is the cloud init one. So actually the provider is no cloud net. And then actually from this one, which is actually the IP address of the engineering server there, we're actually going to, in this directory, get the user data and the metadata files. Okay. So that's how a 400 actually boots and how we did do this. Okay. Let's go a little bit back, see where we are now. I'm on the other road again. That's good. Okay. So the other stuff on the other road, we can find in VARWW. Let's see. Yes. That would be a nice one. Yes. Now I don't have it installed. Now in the bad of actually, the bad luck is actually, I did have a second monitor actually also to show me that, but I didn't bring that with me. So in this case, we see here the different systems. Let's pick up the five. Here we have the user data. There's a lot of stuff actually here. What we actually have here is so the preserve hostname, that means you're going to reset the hostname. And normally we would have to reset the hostname actually here from the template. Here I'm actually going to define all the users that need to be created. And at the end, we're actually doing one more trick. Let me see. This is all the users. And then we have here run command. And that actually downloads a script and actually does the last configuration that what we actually could not express ourselves in cloud in it. So in this case, this does the following. I'm going to do this one, then I'm going to cloud in it. I have here, yep, let's go in that stage. For example, what this one does is actually putting a banner and then also make that actually readable when you're actually using a console. For example, the other thing that it does, it makes sure that the ice-coosy, because that's also coming from the template. The ice-coosy initiator is actually rewritten that kind of stuff. We're going to delete, in this case, the old registry actually from the ice-coosy devices. I'm doing some check here, the CPU info, because we also have the old FSTAP entry actually from the template, the Raspberry Pi actually that was used to template it. So we need to change the serial number. Put that in SFTAP. And then actually the last thing that in this case user data does is reboot it again. It will then reboot with the new house name, it will reboot them with the correct ice-coosy initiator and that kind of stuff. And then actually then we're in business. Good, one more thing for this one. This is actually a dance for Playbook graph, which my older son actually has created when I asked him this is all actually a pick up a Raspberry Pi where we actually have installed a template Ubuntu or a template PiOS for example. And it will then actually transfer that to a ice-coosy device that we can actually use as a template. So I'm just going to do it on a high level. What we do here is we're going to check it. Then we're going to do the disk layout. So I'm actually going to describe everything. Let me show you how we do that. Everything with rules. Those of you who did attend the previous talk here in this room, the guy was already talking a lot about Ansible and that kind of stuff and a lot of what he talked about. You can also see here. So we go to disk layout as I roll. Then we go to tasks. A few things that we do here. First of course is making sure that we get the ice-coosy number one, then actually creating the volume groups. And once we done that, creating the file systems, we have a special dictionary. Actually it creates all the files and we can actually customize that kind of stuff. We are going to write everything in a temporary FSTAP file. Then we actually going to write it back in the write format. And then of course we're going to pick up the serial number and also put that in this case in the FSTAP. So then it's actually quite simple. What we're actually doing in this case is the R-sync to the new, the R-sync is what we do here. We actually write it to the ice-coosy device that we have. And then it's quite simple. Then actually we can copy that ice-coosy device. You can just do that by DD. You can also make a snapshot from of it and then actually have that one attached to your Pi. Okay, let me show one more thing. This was the Pi 400. I'm going to hurt it a little bit. So I'm going to say, I don't like this. Okay. All we're going to do now is to see if it works. I'm going to provision one of the Pi 400s. Lucky, change this. The problem is most probably that the screen is not so readable. Let's see that. The good part is this one has a power button. Yes, now it's starting. There it goes. This one has a little bit of a older boot loader, as you can see. No, actually not. It's the same as new one, but this one actually directly starts. You can see here that it tries to boot from the USB. That one won't work. So in a few moments, it will actually go over to TF2P again. Okay, link is ready, it says. You see, not a time. You see that it actually tries to find the boot images by actually looking for the serial number. This is again the Chrome screen. Obviously, after that, we'll actually boot up the Ice-Cuzzie again. And what it does here, in contrast to the Raspberry Pi phone, which was already initialized actually, when we're a little bit lucky, this one will actually initialize itself using CloudInit, so it will actually boot twice. And unfortunately, the screen is not readable. And I cannot change that, unfortunately. As long as the screen is calling, everything is okay. And the good part is that this is actually how we then also add, for example, Raspberry Pi's as nodes to our cluster. It goes really fast. Here you can see, with the pluses, this is actually the CloudInit that is running. You see multiple sections actually. So this actually means this is the initial one. Let's see how far it comes. If it reboots, then actually it's okay. And it does do its job. If it doesn't reboot, it means we have this race condition. Okay, there it goes. So this is CloudInit. Having the final reboot, what it actually did now is, it changed the Ice-Cuzzie initiator back to what's actually used for this system, named after those names. It did change the everstop. And so that actually the boot firmware, for example, is now connected to its own NFS directly on the TF2 Reboot. And then it actually reboots again, and I have to wait. But it in, yeah, let's say 80 seconds or so, we can have a new node actually there. And actually using an answerable playbook, what we do is we actually pick up the new node. We actually go back to the master nodes, request there the join key, and then actually directly join this one. And that means that Kubernetes is on itself, for example, and it's two minutes actually to pull in all the images. And then actually everything can be just useful production. Here it goes again, the chroma key. Are there any questions already? Please. You did put a lot of work in all of this. Are you publishing? Yes, yes, definitely. I will make sure that I already was busy with it last night. I did miss the Guinness. A lot of you all did it also. But yes, I'm going to come to put it on GitHub so that you can all get this kind of stuff. Yeah, you're welcome. Now a lot of people actually work with this kind of stuff, but not many people actually have done the eyes because you won. And actually the biggest problem that most people face is actually when a kernel gets upgraded, how I'm going to actually fix that. And actually we did it with NFS for example. It's not the most elegant way, but actually it works like a charm. The other way for example, yeah, what I really would like to do is put some more effort in it. For example, I have this one function as a NTP server. So all the Raspberry Pi's actually directly when they come up, they do have to use it here. Using other technology, actually making the best of these Raspberry Pi's because they're really great work and great toys actually to work with. More questions. Good, now you can see it's online head. Now it's actually has its own logo and that kind of stuff. If you read very carefully, you can actually see it's host name. Good, oh that's a very good question. If I, within SD card for example, depending on one from Ali Baba or one from the AM, how do you call it? The Samsung ones for example, the premium ones, you get 80 to 90 megabytes per second. On this one with ice-cozy for example, and again, this is just a one gigabyte, so it is not really super, so the processes in these are actually a little bit more powerful than in the Raspberry Pi, but I get some 320 megabytes per second read, 320 seconds per second write. So for Kubernetes, that works. More questions. Yeah, you could do something like it because the question was not, is it possible to have a single disk actually? And then you can do two things, a single template or really a single disk? Yeah, what you could do in this case, we chose for LVM2. So yeah, you can actually create one disk and actually give the older ones actually a snapshot of that disk. So they still have their, because they need to have their own storage, of course. These are not containers, but they need to have their own storage. And the other way that you could do, for example, is overlay it with a temp file system, for example, a ROM disk or something, that could open. Well, my choice would be actually do something indeed, have one real disk, have the other ones actually snapshots from that one. And give them some extra space, for example, 10 gigabytes, for example, so that they can make the changes. And later on, for example, for some of them, you can even merge them back if you'd like to. Yes. Yeah, what you have with it is really nice actually to see. And you see it more, for example, when you have, and we can also do these things actually with highly available Kubernetes, for example. HCD is a key value store. And that means that it actually makes really fast updates. And actually the whole idea is that HCD doesn't go and search, for example, what was the next thing that I need to write. It actually tries to, each time, write in a different place and that kind of, things is actually horrible for an SD card. And in this case, for ice-cusy, yeah, we don't see the different. For NFS. Yes, for NFS, yeah, sorry. Yeah. Yeah. Because you have all these ops, the mutiny of the file system, that kind of stuff. If you have a really big NFS, you should get it to work. But actually, yeah, okay, let's do it as it's actually supposed to do. Normally, you don't run actually SD database on NFS also. And that's why we actually chose for, yeah, a normal logical file. More questions. A very good question, by the way. Okay, yes. Yeah. Good. More questions? Okay, then let's round it up. Many thanks for your attention. Really love it. It's late on the day and that kind of stuff and yeah. Thank you.