 Welcome to the second day of GPN 2022. And yeah, hi. This is a talk about Ersing, which I personally use every day to transfer files around. And Michael Stabelberg prepared a talk on his re-implementation of this in Goal. Probably some of you know him already because he's the creator or behind the I3 time manager in Linux. And I personally am very curious to learn something about why you did this and what you discovered doing it. Thank you so much for being here and enjoy the talk. Thanks very much. So, you know, as you see, the talk is titled Why I Wrote My Own Ersing. In order to explain this to you, I'll need to first explain a little bit what Ersing is about. Then we're gonna talk about Fiber 7 and about my projects, Router 7 and Go Crazy. And hopefully, by the end, it will all make a little more sense to you. So, can I get a quick show of hands? Who here has used Ersing before? That is most hands. Who has never used Ersing in their life? One per two, oh, okay, cool. Okay, so let me show you a quick demonstration, also maybe as a refresher for all the rest of you. What I'm doing here in this video is I'm copying the slash ETC directory of a Debian server with SCP on the right-hand side and with Ersing on the left-hand side. And you can see that Ersing just completed almost instantaneously. It's much, much, much faster than using SCP, right? You've probably made this experience yourself, which might be one of the reasons why you're using Ersing. When you look at the documentation, the way that Ersing describes itself is as a fast, versatile, remote file copying tool. What does that mean in practice? It means that whenever you have two file sets that you wanna synchronize, you can use Ersing. There are many use cases for which this is a useful tool. One of the original usages actually was to synchronize mail directories. So if hosting your own email server is your kink, then you can use that to read emails on your laptop and not on your server. Then, obviously, many of you might be doing backups using Ersing. You can use Ersing to do partial backups, to do full backups, to do incremental backups using hard links. Really, when they say it's versatile, they mean it. There's an option for almost any behavior in Ersing. One of the things that I like to use it for is to mirror source trees. So typically, before I travel to a conference like this, I wanna just grab my work from my PC at home so that I can continue it here. One option to do this is to commit everything nicely, create branches, et cetera, but I'm way too lazy for that. So what I do is I just Ersing my dirty git directory over, and then I have all of the state, right? I don't need to clean it up beforehand. It's very handy. Then, if you've ever used a Linux distribution and installed a Linux package, that package will have touched at least one, most probably multiple Ersing copies in the process of getting to you because the whole Linux distribution network is all built using Ersing. And then lastly, if you just wanna upload your website to your server, to your web poster, or if you wanna go all the way and build your own CDN across multiple continents, Ersing is a great tool to just distribute all of your content like that. So a little bit of history. Some of you might be surprised to know for how long Ersing has been around. The guy on the right here is Andrew Tritchell, and he is famous for starting the Samba project in 1991, 1992. A couple years later in 1996, he first published a post about Ersing in the CompOS Linux Announce group. Then a couple of years later, he published a PhD thesis that outlines how the Ersing algorithm works. So if you really wanna know about it in detail, you can check out the PhD thesis. But there's also a presentation by Andrew at the Ottawa Linux Symposium just a year later where he develops Ersing from first principles, essentially, in the presentation. And I thought it's a very nice presentation. So if you wanna get to know a little bit of the reasoning behind it and how you would come up with a design like this, then I would encourage you to look at that presentation. All right, so how does Ersing work? I have a sketch here of protocol version 27. Why is it protocol version 27? Because in later protocol versions, the protocol got a lot more complex. In fact, most of the alternative Ersing implementations and there's multiple of them. For example, there's open Ersing from the BSD world. They implement protocol version 27 because it gets you most of the way and then afterwards you have sort of diminishing returns. So in protocol version 27, you can see that on the top side, we have the Ersing sender and on the bottom half of the diagram, we have the Ersing receiver. The protocol works in one direction, but the roles of a sender and a receiver are not tied to the client's server roles. Again, Ersing is very versatile and flexible, so both the server or the client can be the sender or the receiver, it really doesn't matter. These are distinct from each other. The Ersing sender starts the protocol. The sender walks the file system, so that is the file tree that you wanna synchronize, and it sends the file list over the network to the receiver. The three arrows that I have here on the diagram indicate that all of this is pipelined. So there's no synchronization here or anything. Like the sender just goes, it walks the file system, it sends as fast as it can. It doesn't need to wait for the receiver to acknowledge the transfer. Then afterwards, or actually let me dive into the file list. So I'm saying send file list, receive file list, this is what the file list looks like on the wire. You can see here the smallest file list entry is about 18 bytes. This is pretty compact for a file list, and Ersing uses a couple of tricks to make this file list very, very small. For example, device node data and simulink data is only sent when it's actually non-zero. Or maybe even more complex, the file lengths are sent as an int32 by default, and only if they need an int64 to be represented, then it switches to an int64. So this makes like the encoding and decoding of the file is more complicated, but also a lot more space efficient. All right, so back to how does it work. After sending and receiving the file list, there is one step where both of the sites need to synchronize with each other. And that's because they both need to sort the file list so that they have it in the same order. And then afterwards, the protocol only uses handles into the sorted file list. So it's kind of like an index into that array. In later protocol versions, this step has also been eliminated. And our sync became even more streaming and incremental. But in this older version, that's the only stage where you have like one synchronization. Then afterwards, it's no longer the sender who is active, but now it's the receiver who starts doing something. And the receiver goes through the file list that it has received. And for each file, it either creates the file locally and sends checksums if the file already existed, or if it didn't exist, then it just tells the remote site, hey, I just created this in you and I don't have any contents yet. Now, based on the checksums, the sender side does what's called the hash search. And I'm gonna cover that in more detail in a second. This is sort of the main thing that most people think of when they think about our sync and the algorithm, right? Because here is where a rolling checksum is used to then figure out which parts of the file are already on the other side. And the sender can then either send a token or actual partial contents if the contents are not on the other side yet. And the receiver will use either the tokens or the contents to reconstruct the file from scratch. And again, here I have these three pipeline arrows. So all of the directions are heavily pipelined and are only constrained by the bottlenecks that you have on your transfer. So that might be your CPU or your disk or your network, any of these, but our sync in general has no problem maxing out any of these dimensions. All right, so I mentioned the hash search is like the key of the synchronization algorithm, right? So how does it work? Well, our sync first determines a block length for the algorithm to work. The block length is chosen depending on the file size. It's actually the rounded square root of it. And then the receiver sends all of the checksums of the existing file. We talked about this just a slide ago. And the sender goes through each byte of the file and matches existing content. So this sounds like it could be super slow, right? It needs to go through each file. So how can it be efficient? Well, our sync uses a multi-tiered checksum architecture. It has two different algorithms. It has the so-called sum two that's the more expensive algorithm that's a 16 byte MD4 checksum. And why is it MD4? Because end-retritual happened to have an MD4 implementation for Samba. So you just reused it. And then you have the sum one and the sum one is much less expensive. It's essentially two U in 16s. And then I have here this sort of multi-tiered structure where you first have a tag table. The tag is one half of the sum one. You first calculate the sum one. You look into the tag table and if that gives you a target's index, you check if the target's tag two also matches. So now you've verified the sum one. This is very, very cheap. You don't need to do any MD4 calculation up until this point. And you can use this in a rolling checksum way. Then with that index, you again do a lookup into the sums. And in here, these are the transfer checksums. You have the full metadata about the block, right? You have the sum one. You have offset length and index, but you also have the sum two and that's the MD4 checksum. So R-Sync can be efficient because it tries to do as little work as possible, right? All right, and then as I mentioned, the sender now sends either a token, which is a reference to existing file content or the actual file data and then the receiver reconstructs the file. You can actually visualize this quite nicely to verify your understanding of the algorithm. You can just look at the R-Sync receiver and sender and look at the system stats while you start an R-Sync process, right? So you can see at roughly at this point here, I started the R-Sync process and I started it in order to transfer a full Debian server with like 21 gigabytes in 800,000 files. And what you can see here, the rectangle marks the file list sending phase, right? You can see, okay, the server now reads from disk with approximately 40 to 50 megabytes per second and it sends out to the network but with only one megabyte per second. So here you can see sort of the compression rate of the R-Sync file list encoding, right? It needs to read 50 but then only sends a megabyte to the other side. This takes approximately 17 seconds and then you can see that the transfer characteristics change, right? On the other side, previously you had like these reads from the network and if you watch very closely, you can see that it's reading from the network and using more RAM because it needs to buffer all of it and then sort it, right? But then afterwards, once you're at the end of the file list building, you can see that now it actually reads file data from the network and writes it out to disk and on the other side, you can see, okay, it reads the file data from disk and writes it out to the network and these go as fast as possible, right? So this slowly ramps up to a full gigabit which is the connection I have to this particular server. Okay, so now you know roughly how R-Sync works under the hood, right? The question remains, why did I write my own version of it? So previously I was running the server in my kitchen as one does, right? And it was hosting R-Sync D. The server was connected to the internet with a project of mine called Router 7. Router 7, you can see it at the bottom right of the slide. It was running on a PC Engine's APU. It's quite a nice little embedded device. It has a one gigahertz CPU for gigs of RAM but most importantly it had three different gigabit ethernet ports. So it's like an ideal platform for doing a firewall, a router or any other sort of network appliance. So then at some point I had to upgrade the hardware for this router because it was becoming a bottleneck in my network. And I decided, well, if I need to upgrade from like a small little embedded device to a fully blown PC or similar, I don't want to run like a server and a router and both of them consume power, I want to merge the server and the router machines. So why did I need to upgrade? So I'm with this ISP called Fiber 7 and they did a hardware upgrade recently. They upgraded their core network from 10 gigabit connections to 100 gigabit connections. But more importantly, they also upgrade their access network, which is the network for the customers. And that network used to be limited to one gigabits and now they added 10 gigabits and also 25 gigabits. So obviously it was clear what I had to do, I immediately ordered the 25 gigabits. I have a blog post about this whole upgrade if you want to learn more about that. It's linked here, the slides will be online afterwards. What you can see here on the right is a picture of the actual hardware that INIT7 runs in their access network. So these are like the new Cisco switches. Each of these lines here is a fiber to a residential customer and they can run with either a one gigabit SFP module, a 10 gig or 25 gig. And then the pink wires here, they go up from the access switches to the aggregation switches. And then from those, you have another fiber that would then be like the previously 10, now 100 gigabits between the pops. All right, so how do you do 25 gigabits at home? Well, you can no longer use an APU embedded device. They're just not fast enough. Instead, what I ended up doing is I evaluated commercial off the shelf hardware, but the problem was that these are typically made to like plug them into your rack. They're like loud, they have fan control that spins up every minute or so, it's very annoying. It's not living room compatible. So instead, I decided to do my own build. I did a custom PC build for 25 gigabits and the way you do that is you just build a regular PC and you plug in fast network cards. On the right hand side, you can see the left card of the two. That's an Intel network card that can do two times 25 gigabits. So that's where the uplink goes into. And then the right network card here is four times 10 gigabits. That's essentially distributing it across my home. So if I have multiple machines that need to plug into fast internet, they just go into the 10 gig ports here. So that way I don't need like an expensive switch that has a lot of switching capacity. I can just do all of it from this one custom PC build. And if you're curious about more details on that, again, there's a blog post I've linked it here. Feel free to browse that later. Okay, okay, so now this is why I had to upgrade. Now I was talking about router seven, right? What is router seven? Why did I build it? Well, previously I was using the tourist omnia. And I suspect many of you have heard of this router. It's a great project. It's open source software. It's open hardware. It was even sold on the official in its seven store and they subtitled it. You can see the ultimate nerd router, right? So I was super excited to get it. It worked well. The reason why I originally got it is that it does auto updates, right? So it automatically updates to newer versions of the software, meaning I no longer need to care and shepherd it across the updates. However, in May 2018, it did an automatic update and it broke my IPv6 connectivity. And that's just not great. And the reason why it broke is that the fiber seven DHCP servers, they always sent an option with it where essentially the option says, here's your DHCP lease. And for any further traffic, please contact the DHCP server directly at the following address. Now the open WRT DHCP client used to ignore this option. It was just not implemented. It started to honor this option at some point and it turns out that the address which fiber seven gave out was actually not reachable. So obviously it wouldn't work, right? And I contacted both of the parties in order to get a fix for this. I contacted fiber seven and they told me, well, it looks like a bug, but we don't have the manpower to investigate it right now because we're in the middle of a full DHCP platform revamp. Like any debugging work we spent now just delays the actual project. So it's not worth looking into it too much. Also it didn't actually affect many customers. And the author of the open WRT DHCP six client was actually against my suggestion of making the program more robust. I suggested to them that, if you can't reach the DHCP server at this address, instead of timing out and expiring the lease, why don't you start over? And they wouldn't have it, I don't know why. So unfortunately I was stuck between the rock and the hard place, right? Nowhere to go, like the provider can't fix it, the client can't, what do I do, right? So obviously I can just pin that release and stop the auto update, but that was like not the point of the device, right? I really wanted the auto updates. That's why I got the router in the first place. So at that point I decided, well, you know, it would be fun and more reliable to build my own stuff. And when I say build my own, I don't mean just installing standards like network software on Debian or open the BSD or something like that. I mean, I'm gonna rewrite the whole thing and go, right? So that's what I did. So router seven is a home router that is built on Go Crazy. Okay, now what is Go Crazy? It turns your Go programs into appliances. And what does it mean? It means that no longer do you need to like install Raspbian and update it and maintain it. No, instead you only have four moving parts. You have the Linux kernel, you kind of need a kernel. You have the Raspberry Pi firmware, you need that to boot the Raspberry Pi. You have the Go compiler and standard library in order to build your programs and then you have your actual programs, right? Those four things are the only things that are running on Go Crazy. And notably there's like no C environment, there's no dynamic linker even, no shared libraries, nothing like that. There's no memory unsaved programming languages, no security holes like that. There's only the security holes that I program myself, right? So that's comforting. Then Go Crazy, it originally was created for the Raspberry Pi three. It also runs on the Pi four. We recently added support for the Pi zero two W. And for router seven, I added support for running it on PCs. So with Go Crazy, the way you would use it in your day to day is you just have this one Go Crazy packer command that you use to override an SD card. And the packer command creates a reproducible read-only root file system, which afterwards you can then update over the network. And I have a demonstration here. It's embedded as a video, so that I don't have to have like the hardware here, which is inconvenient to show. So let's sit back and enjoy the demo. Quick demonstration of Go Crazy. In this demonstration, I assume that you have already done the preparation work. So I'll skip straight to step one. In step one, we're gonna run this watch command while I insert an SD card into my computer. You can see that the new device has appeared. And now we're ready to move to step two, where we create a new Go Crazy instance in a new directory. Now, we're gonna use the packer command and tell it which SD card to override. I'll remove the SD card from my computer and plug it into the Raspberry Pi. All right, let me switch over to the Raspberry Pi's monitor output. We can see that Go Crazy started up. The system is now obtaining its IP addresses, time, and showing the utilization on the connected monitor. Switching back to the computer, we can now use the URL that it printed to access to Go Crazy Web Interface. Here, we see all of the running programs and now we're gonna add one more program. To do that, we're gonna modify the packer command for I, specify the new program, and also instead of overriding an SD card, we're just gonna tell it to update the existing Go Crazy instance over the network. All right, now that the device is back and ready to use again, we can see that the new program we installed is listening on port 6618. And indeed, we see the system utilization in our web browser now and can disconnect the monitor. So this is like what the day-to-day feels like for Go Crazy, right? Now, you might be wondering, well, what did I write this platform for? And one of my favorite use cases that I have is what's called scan to drive. So whenever I get a new paper mail in my mailbox, right, I just unpack it and immediately throw it away if it's not important. But if it is important, I put it into the scanner, I push the scanner button, and then a couple of seconds later, the connected Raspberry Pi will have scanned the whole pages, arranged them in the proper order, converted them into a black and white PDF, which is then uploaded to Google Drive where I can use full text search on all of the documents that I have there automatically, right? So this is very convenient. There's also an option to not store it on the cloud if you prefer to do your stuff locally. Scan to Drive is compatible with any scanner that can use AirScan. So maybe it actually works already with the device that you have at your home. But going back to router seven, right? What does a router seven, a router appliance look like on Go Crazy? Well, you don't have to have that many pieces in order to implement a router. On the left hand side, you can see the internet, right? And then in order to obtain access to the internet, what I need to do is I need to at least get a DHCP v4 lease, right? So I talked to my ISP with my DHCP version four client. I also do the same for DHCP version six. And then both of these clients create a lease file, which is then picked up by what's called the net config D, which is a demon that continuously runs and configures network interfaces. Okay, so at this point, the router itself actually has internet connectivity, right? It has obtained an address and it can reach targets on the internet. In order to pass along that internet connectivity to your local network, you need to have a few more components. You need to have a DHCP version four demon so that, you know, your computer on the LAN can also obtain an IP address. And for IPv6, you need to have a router advertisement demon so that your client can use Zerocon to obtain an IPv6 address. Strictly speaking, this would be enough, but it turns out there's one more component here which is useful to have and that's the DNS forwarder. It's not necessary to have it, right? You can just configure a public DNS server on all of your clients that works fine, but it's good to have it for this particular arrow here where the DHCP four demon feeds the host names into the DNS server so that you can just, you know, do a ping laptop and it will just automatically work, right? So this is like a simplified architecture diagram of how router seven works. There are a couple of other features that I decided to add into router seven because, you know, while you're at it, why not experiment with a couple of nice features and see where it goes. So one of the features that I specifically wanted is I wanted to keep DHCP addresses in order to maximize internet connectivity. And the reason behind this is that the DHCP specification actually says that when your DHCP lease expires, you have to un-configure the IP address. Now, that's typically good advice, but in the specific case of the connection between my router and my ISP, I know that the ISP has a feature enabled which is called port security, which means that if I have the wrong IP address, I will not be able to send out any packets into the network anyway. So the only couple of times where I've seen issues with a five or seven connection is when they had trouble with their DHCP servers. So sometimes what would happen is the DHCP server disappears, and then a regular router would go offline after a couple of minutes because the lease expired and it un-configures the address. But with my router, it will actually just leave the address there until it gets a new one. And chances are that this is good enough to like paper over any connectivity issues that you have because by the time the DHCP server comes back, like you have not even noticed that there was an issue. So this is like a very small tweak that has a big effect, right? There are a couple of other measures that I did to ensure that there would be no more breakages of my internet connectivity, such as the one that I had experienced with the Taurus Omnia before. For example, for all of the unit and integration tests that I wrote for router seven, the test data is actually packaged capture files from five or seven. So really like the specific bytes that I'm sending and that I'm receiving from my ISP, that's the test data, right? So it's very unlikely that I'll introduce a bug and not notice it at unit testing time. Another super cool feature is that router seven is maintaining a ring buffer for any sort of network related configuration packets. So any sort of DHCP traffic, neighbor discovery traffic, that sort of thing is automatically captured into a ring buffer and you can then attach wire shark via SSH to view the contents of the ring buffer and whatever else comes into the ring buffer anew. And essentially what that means is that whenever there is a bug, like one time I had a bug where DHCP addresses in my local network would be reused too quickly, like with the modern Macbooks, Apple does like super quick configuration stuff and apparently there was a bug in there. What I did was when I noticed that the Mac was complaining about a duplicate address in the network, I attached wire shark and got like the last couple of hours of traffic and just scrolled up a little bit and saw, oh yeah. This is like the host that previously had the lease, right? It's super convenient. It's like traveling back in time. So especially if you have the RAM, just capture this and then expose it for wire shark. It's very cool. And then lastly, I wanted to have safe updates and especially also automated rollback so that I can very quickly recover any issues that the router has. In fact, I've decided to set it up by using network boot. So the original installation is via network boot and then all subsequent updates, they either go over the network or they fall back to doing rollback with network boot. And in fact, the safe updater, it actually looks at whether the machine is currently connected to the internet. So I have another demon on router seven which is called the diagnostics demon and that demon essentially does all of the troubleshooting steps that you and I would do if we noticed that our internet connectivity might have some trouble. So for example, it looks at is there an IP address configured on my ethernet interface? Can I ping the default gateway? Can I resolve host names? Can I then establish a TCP connection to a host name, right? Like all of these sort of troubleshooting graph steps, it does them automatically and in parallel. So if you ever think, oh, like, I don't know, my Netflix is hanging, is that a problem with my connection? You just open up the diagnostics demon and it will immediately tell you, no, all check marks, everything's fine or it will tell you, look, your IPv6 connectivity has a problem. So we can not only use this in order to save ourselves some troubleshooting time, we can also use this programmatically. So the save updater will actually query the diagnostics demon and say, well, if you're not online, I'm gonna abort the update to not make things worse. But if you are online, I'm gonna do an update, check if you come back online and if so, everything's fine and if not, I'm gonna do an automated recovery. So this is sort of the gold standard for updates, I think. And it's very good that it can work like this. And I've been automatically updating my router every day for the last couple of years without any trouble. Okay, okay, so I mentioned router seven, it needed to be updated, right? It needed to have better hardware and it needed to be merged with my other kitchen server. So what I had to do with was I had to move all of the server software from the Debian installation on my kitchen server to go crazy. And that means because there's no C code on go crazy, no user land, I need to use Go programs, right? Luckily for many of the services, this is not a problem. For example, I see a couple of torrent images of my own Linux distribution, I'm gonna tell you that in a second. That program was already written in Go, so just install it on go crazy, configure it, it runs perfect. I was serving files with Nginx, but instead I could use Caddy, which is a good web server that's written in Go, just configure it on go crazy, it runs perfect. But here comes the problem. For R-Sync, there's an R-Sync server, the original one, but there was none in Go at the time, right? So obviously the right way to fix this is we're gonna first do some prototyping, run the original R-Sync and then re-implement our own, right? So for prototyping, this is a relatively laborious procedure, but just to illustrate how it could work, right? So on a Linux computer, there is a tool that's called the Freeze program. And what the Freeze program does is you can point it at R-Sync, for example, and it will copy the R-Sync binary and all of the libraries that it depends on. So this includes the linking loader, this includes, I don't know, lib SSL, lib Z, whatever else that R-Sync links into. And then it creates a tarball of that and you can actually unpack this on your Go crazy installation and just run it there. But then obviously you don't have the benefit of having a unified installation where everything is in Go. You don't have the insight into it and most importantly you need to manually maintain this, right? Like if you do a snapshot of your software, you are on the hook for updating it and the whole point of Go crazy is to have auto-updating devices. So this is not a great permanent solution. So the better solution is to build an R-Sync on Go crazy and this Go implementation will then automatically be updated and deployed using just a Go get command and using the Go crazy packer that you've seen before in the demonstration. All of this can be automated with a simple cron job that you just run every day. And while we're at it in implementing a new R-Sync implementation, we can achieve a little bit stricter security by default for the particular use case of serving data to the world. So in the Go crazy R-Sync implementation, all of the configured modules are mounted read only into a Linux mount namespace and then obviously as most servers do, the server will drop its privileges and change route into that mount namespace. So even if there is like an issue, if there is a bug in my implementation, all an attacker can do is look at the files that I told them to be world readable anyway, right? So the attack surface is massively reduced by that. And then there are a couple of other nice properties that we have here. I mentioned the privilege dropping. It will actually verify that the privilege dropping is configured correctly by trying to write a new file into what you told it is a read only module. And if that succeeds at just the boards, right? There's no chance that you can accidentally misconfigure it. You need to be very explicit in what you configure. Then obviously Go is a memory safe programming language as opposed to C. So just by virtue of doing like a clean re-implementation with better starting properties, we end up better off. I've also decided to build in an SSH listener, which is very convenient in Go. There's great ecosystem of SSH related tool. The SSH listener can be anonymous for if you just want to have a secure transfer but allow anybody to do that transfer, or it can be authorized. And it's entirely separate from your system SSH setup. So you don't need to create a user account where you could have misconfiguration or data leaks or anything like that. It's entirely separate. In fact, Go Crazy by default doesn't even have an SSH demon running. You can start one, but it doesn't have one by default. And yet, our sync on Go Crazy will work because it brings its own SSH listener. And then you have everything that Go brings you. You can do a simple yet performant implementation and you can leverage the full Go tool chain and ecosystem. So you can use Pprof profiling to figure out where the bottlenecks are. You can add Prometheus metrics to monitor your stuff. The list goes on and on. So what did I actually implement in this project? When I say an our sync implementation, that sounds like a lot. So the first program that I implemented was called Go Crazy our sync D for demon. And this would be the role of the sender as we saw in the architecture before and the endpoint of a server. So colloquially, the use case would be to serve files to anyone who wants them. Then I later added the Go Crazy our sync tool which is a receiver because somebody was like, well, now you have the sender but the receiver would also be cool. I have a good use case for it. So I decided, okay, let's look into it. It wasn't too much work. So we have the receiver role as the endpoint of a client. So this would mean it downloads files for you. To be done are the other remaining role and endpoint combinations. So we don't yet have a sender as a client which would be the use case of uploading files. And we don't have a receiver as a server for receiving files. But given that the functionality of the others is already there, this should be, I'm gonna say an elaborate refactoring exercise but nothing groundbreaking. All right, I mentioned that Go allows for simple yet performant implementations and I was actually surprised when I benchmarked this myself because it turns out that at protocol version 27, the original Tridge our sync achieves about four to five gigabits and Go Crazy our sync without me even trying too hard was already faster at five to seven gigabits. So how can that be? Well, the bottleneck is the MD4 checks and calculation and the our sync protocol is sequential as you've seen on the slides before. But what you can do very easily in Go and what is much harder to do in C correctly is you can move the hashing, the MD4 hash calculation into a separate Go routine and that way use two cores instead of just one core and that makes the whole thing much faster. So you get two extra gigabits out of the same protocol. Now, Tridge our sync actually didn't stop at protocol 27 and one of the cool features of the newer protocol versions is that they can negotiate which hash algorithm they want to use. And in particular, they default to the XX hash algorithm which is much more modern and much faster. And then you can actually reach 10.4 gigabits with Tridge our sync. I actually tested it on a 25 gigabit line. So this is really, it's not constrained by running on a 10 gig line. It's really just achieving 10 gigs and no more. I haven't been able to test how good Go Crazy our sync would be in that situation because we don't implement the newer protocol versions yet. So that is a future optimization that we could do as well. Cool, okay. So I've talked about the kitchen server but what does it actually serve? So let me introduce you to Distree Linux. Distree Linux is a Linux distribution to research fast package management. What do I mean by that? If you measure how fast common Linux distributions are when you install your packages, like when you upgrade your system to many new packages or when you install new software on your existing system, it turns out that Fedora, Debian, and NixOS all achieve about six megabytes per second. Arch Linux comes in slightly faster at 12 megabytes per second. They have switched to using the Z-Standard decompression algorithm, for example, that gives them a good boost. And Alpine Linux is the fastest of the bunch that I've tested, which comes in at 18 megabytes per second. And Alpine is sort of like this very small stripped down thing that's primarily targeting containers. So Distree Linux tries to approach this issue of how can we install packages faster or how can we install them as fast as possible by using a lazily mounted read-only SquashFS image for all of the packages, including system packages. So you might be hearing, oh, SquashFS images, that's kind of like, I don't know, App Image or Flatpak or one of these other new application container formats or distribution formats. And those, yeah, they do use images, but what I'm showing here with this Linux distribution is you can use the same approach for any package, even system packages, not just your application. You can use it for everything on your machine. So with this architecture, package installation becomes just inserting a new file into an append-only package store. And this you can do very quickly because you don't need to synchronize your rights to disk. It's not a big deal if the file is like half added to the package store because it's not effective, right? Like any sort of temporary file, you can just clean up afterwards and retry. And with this architecture, you can saturate even 10 gigabit network links. So you can install packages at 920 megabytes per second. And this is not just like a synthetic HTTP download. This is really like, you type distree install, your packages, and at the end, it clocks in at 920 megabytes per second, including writing to disk, right? So this is like much, much, much faster than even the fastest of Linux distributions. And it just shows what huge potential we have in order to make our stuff faster, right? There's no good reason that it has to be this slow. Many more details about Distree Linux you can find in my talk here at the GPN conference from 2019. Recording and slides are linked at my website. All right, so achieving these 10 plus gigabit installation speeds requires that obviously people who wanna try it out have a fast internet connection and that they have servers that are closely located to where they are. So I need efficient distribution of all of the Distree files across continents. And as I mentioned before, Linux mirror distribution is one of the prime use cases of R-Sync. So with a good R-Sync demon, we can sort of build out our own mirror structure. I also wanna do more work on figuring out how to set up this infrastructure. So to that end, I've recently started implementing like a little 25 gig test lab setup at home to really like max out where we can go. Now, this was the original motivation, right? To have like a good auto updating, secure safe R-Sync server for Distree. But then I realized, wait, now that I have an R-Sync implementation, we might as well use it for other things. For example, circling back to the scan to drive appliance with which I scan my paper mail into the cloud, I can now actually use the R-Sync demon as a backup server for it, right? And the way it works is the packer command that we talked about before. I just add the R-Sync demon so that it becomes included in the image and installed on my Go Crazy instance. Then I configure the flags for it. This might look a little laborious, but it's not too much. The two flags we need are just the Go Crazy config file path and the fact that it should start as a demon. And then in the Go Crazy config file for R-Sync D, we tell it to not actually namespace. So the file permissions are actually honored. This is not well readable data. This is not Distree. This is personal private data that should not be well readable. So I wanna actually authorize everything, right? I have the authorized SSH listener configured and I configured just one module that is just the scans module. So all of the scans that I do, they go onto Google Drive, but also onto the SD card. And then I use R-Sync from a PC to do a backup of all of that. So on the PC side, it works as you would expect. You just configure the SSH ports and the hostname and a passwordless identity file so that you can then actually use a cron job to automate that without having to enter your password. And then afterwards, you can run R-Sync via SSH as usual, right? Just like any other R-Sync demon, you can just use this command and it syncs over your backups onto your computer. So at this point, you might be thinking, okay, that sounds cool, but how does one approach such open-ended projects, right? Like if I wanted to reimplement something, where do I even start? A couple of tips that I can give you is to start in the right mindset. You need to identify really the smallest possible milestones and then focus on achieving just that. And then afterwards, extended step by step, right? It really helps to clearly document the current status and any limitations that you have, not just for yourself, but so that you can also then publish it as a community project. And then you rely on community contributions. So when I started the project, as I mentioned, I just implemented the serving side. I published it, somebody asked me for the receiving side. Somebody else actually thought, hey, it would be cool to use this as a library and not just as a program. So they contributed a couple of pull requests to turn it into a library. And then there's a couple of other people who are interested in even like paying me to implement features into this, right? So it can go quite quickly, quite far. And you never know what people might be interested in before you actually publish it and see what the response is. Then in the process of doing the building, you need to build tools first, right? You need to focus on visualization and custom tooling in order to make your job as easy as possible. At least for me, I know that I need everything visualized. And a quick step for this is in R-Sync, there's already debug output. It's not graphical visualization, but it does visualize what R-Sync does under the hood. And maybe you can just add a couple of more print messages, recompile R-Sync, it's quick to recompile it, and then go from there, right? That could be like a first step. Maybe you also want to do like a diagram, right? Or some other sort of graphical visualization. Then in the test suite, compatibility tests really help. So in order to be sure that my R-Sync server actually serves stuff that other people's clients can use, I did extensive compatibility testing against the Tritch R-Sync implementation, that's like the original that you all know and use, but also against Open R-Sync from the OpenBSD people. And if there's other R-Sync implementations, it might make sense to add them to the test suite as well, just to make sure that they're all compatible. All right, so that's all the slides I have. I have put a couple of links up here to the Go Crazy application platform, sorry, to router seven, to the Go Crazy R-Sync implementation, if you want to try it out, just one go install command, you have it on your computer. I have actually another repository I haven't talked about yet here, which is R-Sync Prom. If you wanna have like Prometheus monitoring and alerting for your R-Sync transfers, like if you have automated jobs that you do for backups, for synchronization, then you can have an alert when it no longer works. And you can also track how big are my file sets, how long does it take to transfer them, et cetera, and build like a nice dashboard. And then lastly, a link to Distry Linux. So if you're curious about that, feel free to try it out. You can just download it on to like run it in a VM or download it onto a USB stick, put your computer from it and try a couple of cool things that it allows you to do. Lastly, if you have any feedback for me for the presentation, you can use the QR code here or later on just click on the slide to fill in the feedback form. So with that, we're gonna open it up for Q&A and I'm gonna say thanks for your attention. Thank you so much for this thorough explanation. If you have any questions from the audience, raise your hand, I will come to you with the microphone so everyone can hear you. First question coming up. Pass the mic, please. Yeah, thank you for rewriting this in Go. I'm asking about synergies because I'm also in a place where we just rewrote a ton of things in Go, which is the Eurot project. I guess you know it. So what we essentially have is a rewrite of everything you find in Busybox and also a bunch of other commands and I will definitely pull this in and try it out. And my question in turn would be because you're using I think the C Busybox, right? Right, yeah. Have you considered using Eurot or some other Go clone of that? Yeah, we could totally do that. So the only case where we're using Busybox just to be clear is whenever you use the break glass program. And the reason why it's called break glass is because it departs from the Go crazy model where the primary idea is everything on your device should be in Go, right? But sometimes it's just so useful to use like a Busybox shell and interactively poke at I don't know the sys file system or proc or whatever else you wanna do, right? And for that we have the break glass program where essentially it breaks the class in that it opens up an SSH listener, you SSH in and you bring your own user land. So what I did is I just built a Busybox statically myself copy that over as a tar ball and execute that. But you're free to like build Eurot yourself copy that over and use that. And if you think that works well, if you try it out we could even like replace the standard Busybox or have like a prepackaged Eurot that people could just pull in very easily by just adding the package name to their Go crazy packer command line. I think that'd be cool. It's certainly in the spirit of the project to use Eurot, so I'd be happy to make that work. Yeah, awesome, thank you. And regarding break glass, it sounds a bit like the CPU command that we also brought. Yeah, it's a bit similar, but it doesn't go quite as far. Yeah, the CPU commands definitely very cool. So if people are unfamiliar, they should check it out. Yeah, okay, cool, thank you. Further questions from the audience? No more hands. Okay, well if any question comes to your mind please feel free to find me at the conference. I'm always happy to chat about this project or any of my other projects. You know, just look for the Go crazy t-shirt. I should be easy to find. All right, thanks, and enjoy the conference.