 So, hello and welcome to the third day of GPN. I guess I can still tell good morning to everyone because we are at a hacker conference, so I'm not up that early. We are here to listen to some, yeah, new things regarding GO that you can do on a Raspberry Pi for whatever you want to do on a Raspberry Pi. We had a quick, yeah, questionnaire here. A lot of people have a Raspberry Pi in the audience and a lot of people do home automation and a little backup on there. And I'm hoping that there is a new way of doing that. Maybe some of you will then go there. But without further ado, I'm here to introduce Michael. He will, yeah, tell you a little bit more about it. And a warm applause for him. Yeah, thank you very much for the introduction and thanks everyone for coming. So the title of this talk is Build Go Appliances for the Raspberry Pi using Go Crazy. So first I'm gonna give you an overview of what I'm gonna talk about. First of all, I'm gonna introduce what is Go Crazy and I'm gonna do that by showing you a couple of different aspects of the system, like how does it work, but also notably how does it look and feel like when you use it. I'm gonna show a quick demonstration of how you would just install it and get started. I'm gonna talk about which hardware you can use it with and how we implemented fully automated Linux kernel updates. Then we're gonna switch gears a little bit and talk about a couple of notable Go Crazy appliances. So these would be things that other people have built that you might find interesting and wanna run. Then we're gonna talk about notable Go software that you can run. So if you wanna build your own appliance but you don't wanna like write your own programs, you could decide to just use existing Go programs and run them in an appliance setting. And then we're gonna talk about building your own stuff with Go Crazy because the conference here is called Gulasch Programmier Nacht, so we better do some programming. And then a very quick outlook to future developments and we're gonna have some time for questions and answers at the very end. All right, so let's jump right in. What is Go Crazy? Well, I told you already it's for appliances. You can use it to deploy your Go programs as appliances to a Raspberry Pi or a PC. What do I mean when I say an appliance? An appliance is sort of the opposite of a general purpose computer. So with the general purpose computer, you just get like, let's say maybe an Ubuntu or a Debian installation or what have you and you can add anything and everything you want, right? An appliance sort of looks at it from a different way. It's not like you want a broad base computing platform and then you can do anything and everything. It's like you have the one program that you really wanna run and you don't really care about all of the rest that is typically part of let's say an Ubuntu system. So specifically Go Crazy is 100% written in Go, right? And what do I mean by that? The only non-Go parts of the entire system are the Linux kernel itself, which you might know is written in C and it would be futile to try to reimplement it, right? We really wanna use what the Linux kernel already offers. And then there is the Raspberry Pi bootloader, which if you're into the Raspberry Pi, you might know, you need these binary files in order to start up the system. But aside from these two components, there is no C user land on the device. This means there's no G-Lip C, there's no open SSL with security vulnerabilities. There are no package managers. There's nothing like that, right? There's only the Go programs that you decide should be on the device. So the philosophy is sort of a build minimally from the ground up and add only what you need. And that way also keep the complexity and the attack surface very low and self constrained, right? Notably also, like why is it written in Go or what does that give you? Well, you can enjoy all of the strength of Go uniformly for the entire system. This means that all of the components are managed as Go modules. So if you think, oh, there's like a little bug or a little feature missing in the DHCP client, off Go crazy, you can just say, okay, I'm just gonna take that Go module, change a couple of lines of source code and then deploy it very quickly, right? So in order to do that, you would use the same replace directive that you might already know from working with Go. Also, you'll have all of the other strength and to pick just one that I personally like very much, that would be the very quick compilation times. So working in a system that is written in Go is great for interactive development. It's very quick to make progress and the same is true for Go crazy and I'm gonna show you in a little bit. But first, I wanna show you this demonstration video to give you an idea about the look and feel. Welcome to this quick demonstration of how to first install Go crazy on your Raspberry Pi. I'm gonna assume that you have already done the preparation work of installing Go and installing the Go crazy CLI. I have prepared an entirely empty SD card and we're gonna jump straight to step two of creating a new Go crazy instance. With the Gock new command, we can see that we now have a new instance created and we can use the Gock override command to write it to the empty SD card. Now I'm going to insert the SD card into my Raspberry Pi. Now that the Raspberry Pi has started up, we can copy and paste the link of the Go crazy CLI gave us to open up the web interface of Go crazy. Let's add another program to this Go crazy instance. We're gonna use the web stat command and this time instead of using Gock override, we're just gonna say Gock update. Now that the device is back, we can reload the web interface and indeed we see the web stat command has been added and it should now be available on port 6618 and indeed we now see the same output in our web browser as on the connected monitor. So I think this should give you a good overview of what it feels like to use it. Now let's explain in a little bit of detail what happened. What did we just see? So the stuff that we're writing onto the SD card on disk, what's happening here? Well, you enter an empty SD card or you're just gonna overwrite the existing contents of it and then Go crazy is gonna add four different partitions. The first partition is called the boot partition. On this one, you will have the bootloader files I already mentioned for the Raspberry Pi you need these binary blobs to start it up. If you're installing Go crazy on a PC, you don't need those and then the boot partition is your EFI ESP, right? Go crazy also supports the older MBR based boot protocol instead of only UFI. Then after the boot partition, you have not just one root partition but you have two root partitions. These are called root A and root B and they're using an AB update scheme so that typically the first time you write it, so in the demo that we just did, we have written boot and root A and then with the GOG update command that we used, we actually wrote the new system to root B so that if anything goes wrong in the update, we would still have a bootable system and then in the reboot it switches over to start from root B and then the next update goes to root A and vice versa, right? So it's a classical AB update scheme and the actual contents of the root file system are read-only compressed SquashFS images. So the idea here is that most of your appliance typically is stateless but obviously you can only get so far with a stateless system and then typically you will want to store some configs, some data, some what have you. For that, you will have the fourth partition for your own usage. This is called a permanent partition and you can have an X4 file system there or really any file system that the Linux kernel supports out of the box. So you could also use ButterFS if you wanted to but you cannot for example use CFS because it's not part of the Linux kernel. It's a separate module that we don't support yet. So the fourth partition you can do with it whatever you want, you can also say, oh, I don't want a file system on there at all. I just want to directly write to the block device if you have strange custom use cases. That's entirely up to you. Go crazy, only manages boot and root. All right, so what happens with that SD card once you plug it in? What happens at runtime? Well, first of all, obviously the Raspberry Pi bootloader starts up and then the Linux kernel starts up and then once control is handed over to us from the Linux kernel, the go crazy init system starts. So the init system is the core component of the system that brings up all of the other processes. So this means it supervises the installed programs. So for example, the WebStat program that we added in the demo, it will be supervised by the go crazy init system, which means that whenever it exits because it crashed or something, it will just be restarted, right? It's a very simple supervision mechanism. The init system is very integrated in go crazy, meaning it's also the component that displays the web interface that we just saw. So when you click on stop or restart in the web interface, you're directly telling the init system what to do. You can also configure it to send the logs of all of the programs centrally to a syslog server. Now, obviously this depends heavily on your setup, whether that makes sense. If you only have one Raspberry Pi, don't bother with remote syslogging. But as soon as you have like five or 10 or the numbers start to rise, it makes sense to like centrally aggregate and filter. The init system is also what provides the network update functionality. So the gock update command, it will send over the new stuff to that system, which will then write it and reboot into it. Aside from the init system, the only components that we really, really need are a DHCP server and an NTP client. So the DHCP server is for IP address configuration so that the device can actually participate in your home network or in the internet. And the NTP client is required because the Raspberry Pi does not actually have a real-time clock that would survive across the power loss. So whenever you start up a Raspberry Pi, you will need to set its clock from scratch, which is why those are like the three components that you really need in the user land on the Raspberry Pi. Then there's one other component which also gets added by default, which is gonna be very handy if you develop anything on Go Crazy and that's a component that we call break glass. It's used for interactive debugging and it is an SSH server. It's Go Crazy's SSH server. It's called break glass because you sort of break the glass of the model. The model is you have a stateless appliance, you have only Go code, there's no C code running on there. With break glass, you're starting up an SSH server that's still written in Go. But then it starts up a busy box, statically compiled environment, which is an embedded Linux tool set for those who haven't used it, which is written in C. And the whole idea of break glass is that you bring your own software. So this is like, I'm gonna say remote code execution as a service, but for a good purpose, right? So you could, for example, if you wanna debug, I don't know, what your CIS calls your program is doing, you could use S-Trace. If you wanna debug what's happening on the network, you would copy over TCP dump. And that way, even though the system is very small and self-contained by default, you can still add all of these useful tools that you might need during development. All right, so now that we established what happens on disk and when you started up, where can you actually plug in that disk, right? Which hardware is supported? The first device that was supported by Go Crazy is the Raspberry Pi 3. You might wonder why exactly version 3? Why didn't we start sooner or later? And the Pi 3 was the first ARM64 Pi that was supported by the upstream Linux kernel. So in case you're not familiar, upstream Linux is developed for all of the devices, whereas the Raspberry Pi Foundation maintains their own fork of the Linux kernel that is specifically targeting only the Raspberry Pi. For the longest time it was necessary that if you wanted to use the Pi, you would need to boot the kernel provided by the Raspberry Pi Foundation. You couldn't just use any arbitrary Linux kernel. The downside of that is that whenever there's security issues in the Linux kernel, you need to wait until the fixes have propagated to the Raspberry Pi kernel foundations, the Raspberry Pi Foundation's kernel fork. And that was a delay that we didn't want. So we thought it would be nicer if all of the components could be updated very quickly, including the Linux kernel. So we were so happy to see that with the Pi 3, we first had a CPU that was supported by upstream Linux. Then later on came the Pi 4 and also the Pi 0 2W, and all of these are supported and recommended for usage with Go Crazy. You can also install the system on standard PCs. And as long as you have an AMD64 CPU, this can be anything, right? It can be a very small embedded PC, like the PC Engines APU was sort of my go-to example that I would recommend. Unfortunately, PC Engines is no longer extending the series of APU processors, so they're like selling all the stock they have and then it's game over. So if you want the PC Engines APU, now's your chance to buy one. But it could also be something more abstract, like a virtual machine. So you can totally run it in like your Proxmox cluster that you have on your Home Lab or something like that. And as I mentioned already, Go Crazy supports both the more modern UEFI-based boot but also the older MBR boot. So whatever embedded PC that you have, as long as it's enough PC-like, you will be able to use Go Crazy. Now, this is the officially supported stuff. And what that means is that whenever there's a new Linux kernel version, it will be automatically built, imported, and then tested on a range of continuous integration devices that I have at my home. However, there are also other alternatives, community-supported alternatives, where the community has stepped up to provide the Raspberry Pi Foundation's Linux kernel fork without as much testing. But if you, for example, have an older Pi, like the original one or a Pi 2 or a Pi 0W, which are not supported by the official upstream Linux kernel, you could still use Go Crazy on these as long as you're okay with using the community-supported kernel. And then lastly, there's also alternatives to the Raspberry Pi, such as the Odroid family of devices, which are also supported, also community-supported. So I can only do so much, but other people are interested in running the system on the other boards, so they're providing the support for that. All right, so typically, you don't only wanna boot an operating system, you actually wanna use some hardware with it. The most common request for hardware that we got was certainly using Wi-Fi. And on the Pi, the situation is that Wi-Fi is the same chip as the Bluetooth chip. It's like an integrated thing. So as soon as we add a Wi-Fi support, we can also use Bluetooth. Now, let me qualify this a little bit. We've have had Wi-Fi support since the very beginning, just not encrypted Wi-Fi, and people really wanna use encrypted Wi-Fi typically. And that was pretty hard because we thought you would need to use a WPA supplicant or something to do the Wi-Fi cryptography, and turns out there's actually a cool feature, which is called Handshake Offloading, where the Linux kernel tells the firmware, in this case, like the Cypress or Broadcom firmware of the Raspberry Pi, hey, why don't you do the crypto? And then you have a very nice API with which you can just connect to a Wi-Fi network. So since March 2022, you can use encrypted Wi-Fi on all of the supported Go Crazy Raspberry Pi models. The configuration is very easy. You just give it like this JSON file for a configuration. And then I mentioned you can also use Bluetooth, but I do wanna qualify that because what you can use is the Bluetooth Low Energy part. So this is called BLE, and devices that will be popular with it are any sort of IoT sensors, right? Like a temperature sensor, humidity sensor, that sort of stuff. They use BLE, but what you cannot use is like a full Bluetooth stack. So don't expect to like connect your wireless keyboard or your headphones or stuff like that, because for those you need like a lot more software components which are just not available in Go because nobody has bothered to write a Bluetooth stack from scratch in Go. So those are sort of the limits, but encrypted Wi-Fi works, and that's very exciting. So then I mentioned we have automated Linux kernel updates, and I wanna talk a little about that. The goal here is to make new Linux releases available as soon as possible. So whenever there's a new Linux minor or major release, usually they will also address some things security critical, not always, but usually. So we wanna pull in these fixes ASAP, right? Because don't run unpatch stuff in your home network. And the idea here was to make it really as easy as possible. So the way we designed the automation is to actually mirror what a human would do if the human were to import the new version manually. So we have a daily cron job that is running on GitHub Actions. It looks for whether there's new releases on kernel.org. And if so, it submits a GitHub pull request which just changes the URL from which the kernel is supposed to be built. Then that pull request triggers a GitHub action, CI run. The automation builds the new Linux kernel, which by the way is built reproducibly, so it really doesn't matter if I'm building it on GitHub CI or I'm building it on my local machine or any of you is building it on their laptop, it will be bit for bit identical. Then the newly built kernel will be added to the existing PR, which will then be force pushed. That triggers another run of the whole GitHub Actions CI automation, which then goes on and deploys the newly built kernel on these sacrificial Raspberry Pis that are running in my living room. So for those, they're just there to support the CI infrastructure. It doesn't really matter if I break them. So it's better that I break them where it doesn't matter rather than you break them where you're gonna be sad afterwards, right? So if those devices boot and pass the test suite successfully and they're all hooked up to like a serial port on the coordinator, then the PR automatically gets merged. And at that point, you as a user and pull in the new kernel versions, right? And the beautiful thing about this is that if any of these steps fails, it's very obvious where it failed and how you continue, right? Because the automation really does exactly the same steps that I would be doing if I were to manually update the kernel. Now you might be wondering how well this works and the track record for this automation is that we have most Linux kernel versions available in less than 24 hours. And mostly this is just constrained on the GitHub Actions cron job running just once per day. If I were to subscribe to kernel.org with like a pops up feed or if I were to pull more aggressively, I bet we could get it down faster. But at some point, you also need to stop. If there are issues in this automation, it typically means that there was a crazy specific kernel patch that fails to apply to a new version. And in that case, you would need to change the patch, rebase the patch or drop the patch or whatever is the appropriate action. Over the last couple of years, we've tried to minimize the patches that we need and it's only gonna say less than five or so at this point. So, you know, this typically works really well. All right, so now that you have a rough overview of, you know, how it works, how it feels where to run it and what hardware is supported, let's talk about a couple of appliances. Like what do people actually do with this system? One appliance I wanna highlight first is called Conserve. Many of you might have like a home lab at home or are responsible for a small rack of computers at work. So, you might know that the serial console is very handy as soon as you have an issue with your server where after you rebooted it, it doesn't come back online on the internet and you can no longer log in using SSH. So, that's why many people still have a serial console as sort of an out of band access mechanism. Now, either you can buy these very expensive serial console appliances or you just build something from scratch. If we're only talking about one, two or a small handful of servers, right? And this is what Conserve is. It's an appliance that uses a Raspberry Pi. You plug in via USB, a USB to serial adapter. You connect the serial end to your server and then you can actually SSH into Conserve and directly get onto the serial port, right? So, it's not like you would need to log in and then start mini-com, figure out which device, et cetera, right? No, it's much more convenient and integrated and configurable. What's notable about this particular appliance is that Matt Leher, who wrote it and runs it for his personal home lab, started the project on stream together with me. So, if you really wanna look at, you're starting from zero, how do you build such an appliance? You can just watch the recording of that stream and you'll be walked through it. All right, so not every one of you might have a serial port on a server or something, but I bet almost every one of you will get physical paper mail in your inbox, usually more than once a week. So, maybe you might wanna do something about it in terms of organizing it. Like, I bet not every one of you is diligently scanning and organizing every mail they get. Maybe you don't need all the mail, but typically you would want to subset. So, what I built at some point is an appliance called Scan to Drive, where the idea is that whenever I get a letter in the mail, I just put it into the scanner that you can see on the picture here and then I push the button and it automatically will be scanned and uploaded into Google Drive, where the cool thing about that is that for the PDFs that you upload into Google Drive, you have full text search automatically using OCR that you don't have to implement yourself. So, from my perspective, I'm just putting it into my scanner. The Scan to Drive appliance converts the full colored scan into a black and white PDF, compresses it so that it fits into the size limit for OCR on Google Drive, where you can't have more than a couple of megabytes on your PDF, I think, if I remember correctly. And then you can use full text search over whatever many years you have stored in there. So, if I needed to pull up an urgent, I don't know, insurance invoice from three years ago while I'm here for the conference, I could totally do it, right? So, this appliance supports the Fujitsu Scan Snap, which is the very model that you see on the picture. I actually implemented a driver for it in Go for this particular model, but it's a couple of years old at this point, so many of you will not have this exact model, I think you can't even buy it anymore. The good news is that any AirScan compatible device is also supported. And what does that mean? Well, AirScan is the standard that Apple introduced and uses for its scanner slash printer support on the iPhone. So, if you go into, I don't know, media market and you look at which boxes say compatible with the iPhone, you just buy any of those and it will work with Scan2Drive, which is very cool. If you don't want to use Scan2Drive, you can also use just the regular AirScan library or tool, both of which are on my GitHub. Under the covers is really just HTTP and JPEGs, very nice small little protocol. So, if you have ignored printing and scanning for many years because you just couldn't, then have a look at it again, it's much better now. You don't need all of the crazy software. All right, this is Scan2Drive. Let's ramp it up one level in complexity. This appliance here is called a router seven. You can see the original version on the top right. This is a PC Engines APU that I mentioned before already. Like many of you, I was using OpenWRT for my home router for a number of years. I was using it on a Turris Omnia, which is a nice open hardware router from the Czech internet registry. This was nice because it's also an auto updating device, but at some point, the auto update feature really annoyed me because in May of 2018, there was an update in the DHCP v6 client which broke my IPv6 connectivity. Now you might be wondering, well, okay, why didn't you just fix the bug? And I tried, I reported it to the author and the author said, well, this is sort of a weird edge case. The standard doesn't really cover it. No, I will not fix it. Talk to your ISP. So I talked to my ISP and the ISP was, well, yes. I mean, the DHCP platform is old, we're migrating to a newer platform. We cannot help you find weird bugs in the current platform. Just wait until the new platform is rolled out. So I can totally understand both the author and the ISP saying, no, we can't fix it. Talk to the other one, but it didn't leave me in like a good position, right? So I figured, well, I could either turn off the auto updates which defeats the purpose for which I bought the device, or I could pin the DHCP v6 client indefinitely which defeats the purpose of having auto updates and security updates and stuff. So I decided it would be fun at this point to just build my own router. So I did. So router seven is entirely written in go. It contains a DHCP client server, DNS forwarder, all of the different components that you need in order to run your small home network. And because I implemented the entire stack myself, that also means that I could focus on a couple of goals that I really wanted to try. So for example, router seven is built to maximize connectivity. What does that mean? Typically, if you have a DHCP client and it cannot reach its server anymore, the standard says the DHCP client should expire the lease, not use it anymore, give it back, right? But if you're talking about DHCP between your router and your ISPs access platform, these are like in VLANs anyway, there's no other clients to take care of. So it's actually better to ignore the standard and say, well, I can't really reach the server, but my lease is gonna be fine. It'll still keep working. And that actually helps me get over brief connectivity issues like that. So the DHCP servers of my ISP are much better now, luckily, but whenever they had small blips and issues, I would just stay online with this approach. The router seven is also very reliable as its goal and also very debuggable. Meaning that for example, if you notice that there's an issue in your local network, so for example, when my girlfriend got a new Mac, it couldn't really connect. And it turns out I had a little logic bug in the DHCP stuff. But in order to debug this, it was very nice because I could just say, okay, I noticed now that your computer can't come online. I can just connect using Wireshark to a ring buffer that router seven maintains, which will give me the last couple of hours of network configuration traffic retroactively, right? So it's kind of like time travel debugging a little bit where you don't need to know that you're gonna debug something in two hours. You can at the point when you encountered a problem, just jump back in time and get all of the stuff. And this is very simple to do whenever you're controlling the entire platform because on the PC Engines APU, I have a couple of gigabytes of memory. So I don't really care to set aside a 50 megabyte ring buffer to just proactively capture all the network traffic. However, for OpenWRT for example, they run on much, much smaller routers where you don't have that RAM so you can't easily do that, right? But I found it very instructive and enlightening to see that approach work as very nice for development. So you might be wondering why the two pictures? Well, the picture at the top is the PC Engines APU, that's sort of the small version with which I started out. As I mentioned, it's a very nice device. You can see it has like the serial console at the top. It has a couple of ethernet ports. You can even connect like a little teensy microcontroller to its power and reset pins and that way like really fully automate the device. But at a later point, I needed to upgrade because my ISP started offering 25 gigabit connections so I built my own PC and that's also the reason why Go Crazy now supports both MBR for the PC Engines and UEFI for the regular PC. But both are well-tested and both work. All right, so you don't need to go as far. You don't need to like build your own router from scratch. Nobody's asking that. In fact, you can go the other way. You can say, oh, I don't really have a use case to build my own appliance but there are a couple of cool Go tools that I would like to run and maybe that's also something for which Go Crazy is interesting to you. So I'm gonna list a couple of examples and the one that I wanna start with is called Prometheus. Prometheus is a monitoring system and time series database. For those of you who haven't seen it, I would encourage you to check it out. It's pretty cool. It comes with a couple of different components. For example, the Prometheus node exporter is the software that you would run on every one of your nodes in your network. So this allows to export metrics which in turn allows to monitor the Raspberry Pi's CPU usage, RAM usage, whether your disk runs full in a couple of days, that sort of stuff, right? You can also go one step further and you can run what's called the Prometheus black box exporter which you can somewhat compare to like traditional monitoring like Nagios or other stuff where you probe targets to ensure they're still good. For example, you could probe your company's website to ensure the SSL certificate is still valid for a couple of days and you would get an alert if that's not the case anymore. You can run that on Go Crazy as well and you can even go one step further and you can run the entirety of Prometheus yourself on the Raspberry Pi. So including the time series database storage and all of that stuff, but I would encourage you to use good durable storage, not like a cheap SD card that you found in your drawer, right? Because if you write that many time series database updates, you will want to have something that survives for more than a couple of months. So I don't know, like a USB SSD or something like that on your Pi. All right, let's switch to the next example. This would be Tailscale, which markets itself as a secure network that just works. To put it into more technical terms, Tailscale is a zero-config VPN which uses WireGuard in a mesh network configuration. So that means if I'm here at the conference and I have my laptop and I connect it into Tailscale, I can reach the computers in my home network without having to deal with port-for-wordings, natural-versal and all that sort of stuff. And the traffic doesn't go through a central server, it goes directly between my machines. So that's very appealing and you can also just install Tailscale on Go Crazy. And then as soon as you have installed it, that means the services that you're running on your Raspberry Pi, be it your backup console, your media center, what have you, will be available over Tailscale. You can go one step further and you can run it in what's called a subnet router configuration where Tailscale will not only make all of the services on your local Pi available, but all of the services in your entire local area network. So this is very appealing because it means that, for example, for my parent's place, I could just prepare a Raspberry Pi, install Go Crazy, install Tailscale as a subnet router, place it at my parent's place when I'm next visiting and then just leave it there, update it remotely and have access to it and all of the other devices, PCs, laptops, printers, tablets, what have you via the Tailscale subnet router. So very appealing use case, I think. Okay, and then most notably, we can also run Docker containers on Go Crazy and this really is a great feature that we added in April, 2022. The reason behind it is that Potman is written in Go, well Docker mostly is as well, but Potman has the additional advantage that's available as a standalone static build which lends itself well to be packaged for Go Crazy where you don't have most of the traditional components of a Linux system. So we did that in April, 2022 and this means that we now have an escape hatch for running non-Go programs on Go Crazy. You might sometimes need or want to do that. One of the examples I've put here on the slide is the Ubiquiti Unify Controller which you need to configure the Ubiquiti Wi-Fi access points, that's like a pretty complicated thing that uses, I believe, Java, Redis and a couple of other components. So you can just run that as a Docker container and then you don't need to install it on any of your systems, you can just run it, right? Similarly, maybe it's not so much about third-party software that just isn't written in Go but maybe it's about older setups, right? And one example I have here is that for chatting on IRC I still use IRC which is very old software at this point but there are a couple of Perl scripts for it that are not in the mood to rewrite in anything else. So I've just put that into a Docker container and I'm running that myself, right? So this is very interesting because it means that you can really start using Go crazy with nothing else but Docker containers and then sort of move all your stuff in there and then gradually replacing Go what you want to replace in Go, if at all, right? All right, so now you have an impression of what appliances can you build? What software could you just install to build an appliance without really building much? But let's talk about what if you wanted to build something from scratch, right? How does that work? Well, the only thing that you need to supply is a runnable Go program, right? The command line flags with which it will be started you can customize entirely, you can customize the environment variables. These would be typically the stuff that you would expect to be customizable in traditionally a system disservice unit but in the case of Go crazy just in your Go crazy instance configuration. There's a couple of more details in terms of what environment variables does Go crazy additionally provide to tell the program where it's running, what's going on? Like is this the first attempt at starting it or are we talking about a subsequent attempt after the supervision figured out, hey, you crashed? Or are we talking about a program which shouldn't be supervised at all? All of these sort of nuances you can look at at our documentation where the process interface between Go crazy and you as the software author is clearly defined. Then aside from the runnable Go program that you provide, the stuff that works out of the box is essentially network because we ship the DHCP clients, the NTP client. So from your perspective, the program that you write you can totally use the network. You can assume the network is there and working. So one example here that I found very instructive is that at some point I needed an MQTT server for my home automation and I didn't have one. So I looked at which MQTT server is written in Go. I found this one here at github.com slash FHMQ and I just installed it on Go crazy and I didn't have to change anything. I didn't have to set a flag, I didn't have to set environment variables, nothing. I could just run it and would just work by default because the network is there, you connect your devices to the MQTT server and everything just works. So if you're mostly interested in home automation or in network services and that sort of stuff that's very convenient in Go crazy. Then in terms of hardware support, what hardware can you actually use? Well, whatever is included in the upstream Linux kernel. Sometimes there might be a little surprising differences between the Raspberry Pi foundations Linux kernel and what is upstream, right? But mostly most of the features that you would expect to work also do work. Just last week for example, I merged a pull request from someone who enabled USB webcam support for Go crazy. So that just works. And another thing that you might wanna do is connect GPIO pins on your Raspberry Pi to I don't know, LEDs, speakers, all that sort of stuff. You could use Perif IO for that, which is like a little Go library for GPIO access. And there's a documentation page about how you would do it, how you would verify that it works using a multimeter, that sort of stuff. All right, let's do a little live demo about how to add a new Go package from scratch, right? So let's see, wanna have a terminal here. Okay, so here I'm in my Go crazy directory. What I wanna do is I wanna create a new instance called gpn. So I use goc-igpn new. You've already seen this in the demo, right? So now we can go into the gpn directory. We see there is a config.json, which we have just now created. And let me switch back to the slides. So this is the first command, create the instance. The second command is, okay, now we need the actual program. So I'm gonna call it hey for hello world. I'm gonna say, go modern it, hey, as you would do typically in Go development. And then we need sort of a minimal program that just does nothing. So this is what it would look like package main, func main, it's empty. Let's say, hey.go. And now I say, okay, use the goc CLI, goc-igpn and add the current directory to it, right? So now I can use goc override dash dash root temp root dot squash fs and just write it out, right? So this is what it would look like. And within a couple of seconds, you have the SD card image prepared. We can see in the resulting file system, there's a lot of system stuff, but then there's also the program that we just added, which it would be empty. So it doesn't really print anything, it doesn't really do anything, but this is what you need to start with an empty directory and you can just write your go code in there and then it runs on the device, right? All right, so once you have that, the interactive development, you could use goc update for it, but if you run it in real time, the demo was sort of a little bit accelerated, but if you run it in real time, it takes about 45 seconds on the pi four. This, as you just saw, like the go crazy part of it is very quick. It generates the entire root file system within a couple of seconds, or even less that if you're only changing a little bit of it, but then you need to also transfer that over the network, write it to an SD card, have the pi reboot, and that all takes a little bit of time. So what I prefer to do instead of using goc update, which updates the entire thing, is I just use goc run, which is a command which just cross-compiles what I'm working on, uploads the resulting program into the RAM of your running go crazy instance and then just restarts that service. So just like a very quick edit run loop where if I interactively develop for the system, that's how I do it. Okay, so now I've already told you that there's like no C on go crazy, right? But that's not actually strictly fully 100% true. There's a couple of escape hatches that you can do if you really want to. If you want to stay go only, that's fine. You can do it, that's the default, but sometimes you might need like the one C library or the one routine that you have. And one example is for the scan to drive appliance that I showed you, at some point I noticed that the image slash JPEG package that comes with go is a very nice and readable JPEG implementation. So I really recommend you read the source if you're curious about how JPEG works, but it's not as optimized as the other implementations that we have for JPEG. So the standard implementation is lip JPEG turbo at this point, and it contains an optimized version that uses the ARM neon instruction set that really accelerates stuff on the pie. And this really makes a difference. So this changes the scan time of a single page from, I don't know, 30 seconds to 10 seconds or something on that order of magnitude. So it's really worth it to use the optimized version of this library. But then I told you, okay, go crazy, it's 100%. Go, no C user, no G-Lip C, so how does it work? How can we still use that library? And the answer is that some of the C libraries can be used if they're statically linked, if they have no dependencies on G-Lip C or anything else, right? And lip JPEG is a perfect example here because really it's like you give it bytes, it does an encoding, it gives you back the bytes, right? It's what we call a compute kernel in the sense that it just does compute, it doesn't have any dependencies on the outside world. So these sorts of libraries you can totally use, you would just build them with like a typical cross compiler GCC in your environment and then you specify a link mode external and you give the external linker the static flag and you see it's a bit involved but you can do it if you want to, it's supposed to be the takeaway of this slide. There's one more escape hatch that I wanna show you which is that if you wanna run C software maybe temporarily on the go crazy device, you can just put it into a Docker container but sometimes you don't want that. Sometimes you don't want it containerized in the way from the host. Sometimes it really needs to run on your host. What can you do in those situations? The one example that I have here is TC, the traffic shaping and control program on Linux where if you just run TC, it doesn't support anything. It really needs to dynamically load the traffic shaping plugins at runtime before you can even configure what sort of bandwidth limit you want on your Linux machine. So when I used router seven to run an event network much like this conference network here, I wanted to cap the traffic limits to not overwhelm our uplink and I used TC for this, right? So how did I get TC to run when it obviously needs to see user land to the extent that you can dynamically load code at runtime? Well, there is a little helper tool which we call freeze, which looks at the binary that you wanna freeze and it looks at all of these shared library dependencies and it freezes those as well. And then it generates like a little tar ball that contains your program and all of its library dependencies and you can just copy that over to your go crazy device and run it there. And obviously that's not great because the software is frozen. So that means it doesn't get automated updates, right? It's like entirely outside of the system that go crazy manages. But this is very helpful for stuff that isn't security critical and doesn't really change. So one example is that we use this approach in order to ship the MKFS program so that we can create the permanent data partition using the X4 file system. We didn't implement like, creating an X4 file system from scratch in go. We just use MKFS and it's easy enough to just freeze it because it doesn't really change. All right, so with that all out of the way, let's talk a little bit about future developments and then I wanna open it up for questions and answers. So one project that one of our contributors and I am working on is called GUS, the go crazy update service. And the update flow that you have already seen is sort of the default update flow, the synchronous update flow where you run GOK update and it does the update right then and there. But sometimes that might not fit the model of devices that you have deployed. So if you have a number of devices that are only sometimes online, you don't necessarily wanna coordinate between where you create the update and run it and where you wanna deploy the update, right? So what GUS allows you to do is one to many updates. So you build the image once, you upload it to GUS and then all of the different Raspberry Pi's that are running the same image can come to GUS at their convenience and pull down the update, right? So it just uncouples these two steps. All right, so with that, I'm at the end of my slides. There's more information at gocrazy.org. I've also laid a couple of go crazy stickers on the table here up front. So if after the presentation you think that's cool, you wanna grab a sticker, please feel free. Also, if you wanna talk about any aspect of go crazy, feel free to talk to me while I'm still here at the conference. I love talking about the stuff and I will also give you a sticker as a gift if you want. Please also scan the QR code and give me feedback on this presentation either now or later. And with that, I'm gonna open it up for Q&A and say thank you for your attention. Thank you very much for the great presentation. Yeah, the question in the front row, please. Thank you. Right, I'm gonna repeat the question for the recording and stream. The question was, do I develop this on the side or is this like a commercially available thing? This is entirely a hobby of mine, which I'm using for all of my Raspberry Pi's and more and more computers at this point, but there's no business behind it. In fact, we've gotten the question at some point where somebody was like, well, I wanna build like my IoT system on this, on go crazy, like are you okay with it? And I'm like, look, it has crazy in the name. Do you really wanna base anything on this? But if it works for you, be my guest, right? Just no support other than what I can provide best effort. Thank you very much. Any other questions? Question in the front row. Right, yeah, I'm gonna repeat for the recording. When did I start with Go and what was my motivation for it? I actually looked at Go when it was first announced in 2009 and then I tried a couple of toy demos and I was like, oh yeah, this looks cool. And then I put it back until Go 1.0 was released in 2012 where they introduced what's called the stability guarantee. So all of the stuff that is in Go 1.0, it will not be broken until you know, go too hypothetically. But that was really like the status, like the signal for me to really use it, right? In earnest. So I started building services such as Debian Code Search, which I'm still running to this day. So if you're interested in searching through open source software using regular expression matching, you can go to codesearch.debian.net and grab over all of what's in Debian. That is a Go system that I built as my bachelor thesis back then. So I was like, oh yeah, this looks like a natural fit for network services and for that sort of stuff. And I'm enjoying it to this day. And I'm doing more and more in Go as you can see. Question in the second row. Oh, there's Mike, can you use the mic? Yeah. What is your experience regarding the performance of Go because you said you'll vote some drivers and so on, is it compatible with C or needs it some fixes or tweaks or something like that? Right, in general, my experience is that the performance of Go is always sufficient and in the very few rare cases where it isn't, such as in the hot loop that's encoding the JPEGs, right? That's the spots that you find using profiling. The profiling story is very good in Go, so it's very easy to figure out where are your bottlenecks in terms of CPU usage or memory allocation or that sort of stuff. And then you implement that more efficiently, right? Most often, I try to just optimize the Go code itself, but sometimes if that's really not sufficient, like for example, if you wanna use the ARM neon instruction set for which the Go compiler doesn't have any intrinsic support, you will need to use the escape hatch and say, okay, now I'm gonna use C or I don't know, maybe you could even use Rust or something else and just pull that into your Go program and keep the rest in Go. Yeah. There's a question at the very back, right? The question was, is there any separation between the processes and the rest of the system, like Seccom or Landlog or that sort of stuff? Yes, but you're responsible for it on your own. So for example, the NTP client that I mentioned and I believe also the DHCP client to a lesser extent, they use privilege separation, so they sandbox themselves essentially. So when you're running on Go crazy, you get full access like sort of as root, right? To like anything you wanna do with the Linux kernel. But if you think, oh, for this particular use case, Seccom would be great or Landlog would be great, then you can totally do that, but it's something that you need to do yourself. Yeah, there's currently no facilities to make that easier, but I'm also not opposed to adding something at some point if it turns out, oh, most people actually need to implement this and that, it would be nice to have a common helper for it. And another story where it's pretty similar is with the Docker containers that I mentioned, which can also sometimes function as a sort of boundary. For those, you can currently run Docker containers, but there's no sort of container manager or something, which you could configure to just tell it, look, just run this and that. So there's no Docker compose essentially, right? So for that, I'm also into space where it's like, for now, just do it yourself. So because you know what you need, you know it best, right? And if it turns out that there's a common need that we should address with like a central component or a repository or a library or something like that, I'm open to add that in the future or to merge that if someone wants to step up and contribute it. Another question in the very front, right? I was wondering since it was also attached to Google, did you ever meet up on the topic of how it involved this with Google nowadays? Because I saw that there's very little information in general from any of these points on the internet. Right, right. I'm going to try and summarize that for the recording. The question was like, you know, for tail scale, there's head scale. So for other parts of the Go ecosystem, are there other replacements like that? For example, the Go module proxy, which is currently centrally run by Google. Fun fact, for this presentation, I actually did use a module proxy that was running on my laptop locally. So that is totally supported. I don't think for the context of Go Crazy that there's really any dependencies like that. You can sort of mix and match everything and anything. So if, for example, you say, well, you know, I don't trust the Google engineers. I want to use a different compiler. You can use GCC Go, right? Or you could use Go LLVM or whatever other implementation you have. The Go Crazy system itself is not really tied too much. It does use the Go tool though in order to build software, right? So, you know, just getting all of the modules, working with the modules, all of that, we don't implement that ourselves. We reuse what the Go tool already provides. So I think that would probably be the one dependency that you really have and that you can't get out of. Right, that's written in Go and you could provide a replacement, you're right. Yes. Yeah, I think, you know, for, I think to summarize it and feel free to talk to me later outside, but to summarize it, I think for any level of paranoia and as you can get comfortable with the system as long as you're willing to exchange a couple of parts here and there. Yeah. Further questions? Still have a couple of minutes, so feel free. Okay, no one wants to ask, I have a question. Please, I'm interested in your build pipeline. You mentioned, yeah, Raspberry Pi's at home. Like, how do you ensure that if they break or like if a separate job comes along, how do you do the duplication of that? Right, so there's actually locking in the coordinator. So the CI software, like the CI pipeline that's running on GitHub Actions, it talks to my coordinator at home and it tells it, look, I want to deploy this new image. Can you do it? And then either it immediately gets the lock and it has access to all of the devices or it doesn't get it and then it just waits. And the reason why this needs the locking is that not only are these devices coordinated in terms of which software is running on them, but in fact, they're also turned off to save power. So the coordinator actually talks to a smart plug, which then turns on all of the devices, lets it boot, et cetera. And then the CI pipeline only uses the energy that it really has to use, right? But yeah, the jobs can't stomp over each other because there's locking involved here. Yep, another question in the front. Okay, the question was, which was the trickiest part I had to build and which one is the proudest part? I think for the proudest, I'm just gonna give you the origin story of how we built it, which was we were sitting around at a Chaos Communication Congress at some point and we were like, isn't it so super involved to have all of these distribution installations that you need to manage? So one example we had at the point was like, back in 2013 when I did my last big house move, I set up an installation on a Raspberry Pi and that just left it running for years and years without really updating it because it was so involved to update it and I didn't wanna risk anything. So that was sort of like, we were like, isn't there a better way? And then a friend of mine was like, well, what do you really need? And then I figured out, well, if we have the Linux kernel, could we just give it in it equals and then a go program that we statically build and cross compile? And to statically build it, you just need cgo enabled equals zero and to cross compile, you just need the go arch environment variable. So I just entered a simple command to my computer and out came a binary and we just booted that up on a Pi and it worked. And we were like, wow, that was way easier than we thought, right? So could we just build the rest of the system that we need? And then it sort of spiraled from there, right? And then the trickiest part that we had to build was probably when go did the transition from using go path to using go modules, because that really changed a couple of different things in particular, because we're sort of a downstream user that's built around the go tool, right? So you really need to be in tune with all of the nuances there. And that also changed a little bit how everything is structured on disk. So that was sort of a bit of a disruptive change from our perspective, but I feel like at this point we have mostly gotten everything, right? But we can talk about the details if anyone wants to. I think we might have time for one more question or so. So don't be shy. Yeah, yeah, about five minutes left. So there's definitely room for more questions. Okay. Okay, but if you can't think of anything, as I said, feel free to approach me outside any time, talk to me about stuff, or just send me an email, open a GitHub issue. Contributions are very welcome. So I'm gonna say thank you again and see you all soon. Thank you very much.