 Welcome to the Home Lab Show, episode 64. We're gonna talk about tail scale and head scale and hopefully make heads or tails of it. And any other pun I can insert. Well, you know what? Any episode number that's a multiple of eight has to be a good one, right? Yes, well, I actually say that as well. So we are excited to talk about this subject and before we get deep into the show here, let's thank a sponsor who's very relevant this time and it's Linode. And why is Linode relevant? Well, when we get to the part about head scale, one of the important things is going to be that you have to have a public IP to run the head scale server on. If you don't have a public IP, it's not gonna work very well for you if you plan to use this in a way that you share out the network with everyone. So Linode's been a sponsor as show since the beginning. They've been a great sponsor. They have just a really solid tool set that you can use for building a lot of these projects, building it through automation, building it through their market or just spinning up raw Linux servers, which is probably what you'd wanna do to set up a head scale server. It doesn't take much. Even one of their really basic plans will handle it because the traffic does not pass through as we'll be discussing in this video. It also is where you'll be setting up DERP servers. So yes, we will talk about what a DERP server is as well because it's part of in there. So run your DERP servers on Linode. And we have an offer code, the homelab show to get you started with them. So thank you Linode for sponsoring and yeah, it's a great place to run a DERP server. I like that they took the time to come up with DERP server when they were designing some of the tail scale stuff. So. It's funny I've been using that word a lot myself because for people that don't know when you use, when you're editing video, if you use what we use, you know, DaVinci Resolve, like the file extension is dot DRP. So I'll say to myself, I'm going to back up the DERP file for the video that I'm editing because it's DRP. What else would you call it? Yeah. Sometimes. Yep. And there's this actually DERP as in designated encrypted relay for packets. That's better actually. I don't know what the file extension, it's probably DaVinci Resolve something anyway. Yeah. But it doesn't DERP. We're going to go DERP. Well, I'm going to. All right. So what is tail scale and what problem does it solve? Now, before we get to the head scale part, I put that in here because we're going to talk about first tail scale and everything that it can do because that's a lot of complexity to kind of navigate and I'll have, well, I shouldn't say we'll have, I already have links that are in the show notes so you can dive deep into this because they've done a great job of not just documenting things, documenting how stun works, how natural reversal works, all the different trickery you can do for natural reversal. It's a good lesson in network engineering. The product uses an entire bag of network engineering tricks that are all documented and things you can, well, do with the, that's within the limits of the RFC along with more fun that they're having with it. So they've done some really clever things to make all of this magic work. So that's why I think it's a really interesting open source project. It's a really interesting network engineering project. And by the way, it solves a problem that we've discussed many times on here. And that specific problem is how do you handle if you're a firewall? And recently PF Sense added, and I did a video on this, the integration for Tailscale. How do you handle when your firewall does not have a public IP, but you would like to VPN back to it? And Tailscale is just a dead, simple, easy solution to do this even if you have several devices or several firewalls and you like to mesh them all together into a big network. I think this was hands down the easiest site-to-site VPN I ever set up with PF Sense. Like it takes you minutes, literally just minutes to get it all configured. There's no, it just, we'll just say it makes it simple. That's why I figured talking about Tailscale is just a lot of fun. And I already see comments in the UDP hole punch for the win. Yes, that's essentially how it works. And we did a full episode on that, didn't we? I thought we did. We talked about overlay networks in general. I wanted to narrow this one down to very specifically Tailscale. Now the other overlay networks out there are good, Zero Tier and Nebula. Those are pretty cool. Nebula is a different approach compared to the other ones. I kind of single them out because Nebula is only self-hosted. There is no central Nebula server, but Nebula also doesn't have as many things in this bag of tricks because it has a different use case. It's kind of a DevOps tool. It's actually used by Slack and it's developed by one of my friends. He's CEO of the company and we chat from time to time about it. That's actually one of the reasons in the video. I thought this was really cool because it's the open-source infrastructure that manages the back end of Slack. It's how they manage, and many other companies too. Every server they get deployed is part of a Nebula node and it's like a overlay network. So you have a locked down management control plane across all of your public IP space. So they actually don't traverse any commands over standard like even SSH or VPN tunnel. They actually have access to everything over there. Now you can do all this with Tailscale as well. The only reason Slack and why Nebula was developed is because they were aware of other protocols out there, but they also said we are Slack, we host our infrastructure, we don't do third parties. And at the time, I don't think HeadScale was an option. HeadScale came later to the game part we'll talk about which is allowing you to self-host it because even though Tailscale is open-source, the control mechanism in the site is a product. It is a service and they have a free tier so you can absolutely sign up for it. So this is all, there's a lot you can dive into and I think I forgot what episode number it is on overlay networks, but yeah, we did cover some of that. Mm-hmm. Well, that's great. So we can dive into a technology even deeper which is always a great thing to do. So I'm very interested because this might actually be something that I'll implement myself after we're done recording this. Yeah, the cool thing is with Tailscale is they've done such a nice job with documentation and putting things together. It is supported on a lot of devices. So you can load this up, not just on PF Sense, but Synology has support for it. So people who want access or would like to build a network of Synologies where you could say, I need all these Synologies and one of the use cases that we have like business use cases is companies that want to set up a bunch of Synology surveillance stations and they go, but I don't really want to or may not have access to opening up a public cloud for these. This is where Tailscale can assign an address to each one of your Synologies and let's say a client has 20 locations, those 20 locations can be tied to Tailscale and then their phone can be tied to Tailscale and they're easily choosing between all 20 locations to view cameras. There's some slick things you can really do with it like that and it's open source, you can run it on Linux, Mac, Windows and BSD. So you got lots of devices you can load it on to get access to all these different networks. And of course it also supports routing and exit noting which the routing protocols means it can see adjacent devices. And this is what it's implemented with in the PF Sense system where you actually turn it on as a route and you say, I want to show or share with the Tailscale the other Tailscale nodes that talk to it but then also see everything in your routing path. So everything on a particular subnet that you want can also be shared with it. That way you're only loading Tailscale on one device but then if you have a bunch of devices maybe you don't have access to share Tailscale on, it can do that. So it's got a lot of really cool features on there like that but I want to start with who is Tailscale and what they're doing because it's an interesting company because they've got a pretty big team. There's a lot of people there that have worked on some very large projects previously. And I think this is really cool because I like what they're saying here and I'm going to read a little bit of their company page. We're building the new internet, small trusted human scale networks. We're returning to the original vision of the internet we want to help everyone create their own secure networks built around their social connections. Whether you want to connect with your co-workers to share a prototype, the company database for security access information or family members to share files and photos, Tailscale makes networking safe and understandable. And I think Jay said it best when we were first talking about overlay networks you're like, this is how I thought VPNs always should have worked. I think that's how you said it. Well, it wasn't necessarily VPNs. It was how before I knew how networking worked, networking in general, when I was first coming into this before I even learned my first thing about networking I'm like, there's probably like some kind of application on both ends that are kind of like creating a connection and that's how it works, right? But no. I mean, at the protocol level it gets much deeper obviously but I really thought that that was the way it worked. And then I remember when I first started an IT how much money the company was spending that I worked for just having a wide area network through AT&T was ginormous the cost in the early 2000s. And it's like nowadays what I really enjoy seeing is when companies like disrupt everything like you get this thing that's super expensive to implement to purchase. And they're like, yeah, let's just like do it not that way but we'll just give it away for free because there's a way to do it. And then you can, depending on the company pay for more features or whatever but still it's not like a couple of grand a month per connection. I mean, it's just like so within reach more than it's ever been. And I think that things like this making everything easier is exactly what the industry needs. Yeah, absolutely. And it's part of their ethos because they're, you know, this is right from their page we are working to remove the overhead and complexity from the long tail of software and operational problems that people face every day by making connectivity easier and more secure. We empower small teams to build systems at scale without scaling overhead. And I think this is just really cool because even though they're a business and a product they're really dedicated to open source and this was something I found and I didn't really go deep into vetting it but I've seen a good discussion on Reddit on this when I was doing my research on headscale they even take the time to contribute to the headscale tool which technically subverts their business in some way because, you know, you're still tail scales open source but you know, they want you to buy they have a free version or a free sign up to use their control plane but obviously they have a business model and they have a fee they charge for it that pays for the development of it but they still take the time to one contribute actively to the go community that they do a lot of they keep the client free and open source so anyone can look at it which of course they even don't have hard-coded within the product where the control plane server is so they built it allowing people to kind of mess with it and then take it a step further someone comes up with the headscale project they're among the people contributing code back to it going, oh yeah, yeah, that's cool well, cool, you're building this thing let me help and I think that's awesome they even like their DERP server is open source and they didn't have to open source it if you looked at it from a business model because that's part of the coordination server they open source that too and this is what makes tail scale to be a really interesting product because you can do a lot of this so they have a clear commitment open source which aligns with me and I've always said this I'm not someone who just likes open source because it's free and neither is Shay we like it when there's a business model behind it because that means it's sustainable it gets security audit, it gets updates we look for it as a solution because we think the code should be open but I do like when you combine that with yeah, some type of business model around it especially when it's a fair, reasonable business model for how they want to do things it just makes a better ecosystem I think for me that kind of thing could be a tiebreaker whether or not I want to go with the service is how they treat the community so if I'm on the fence about two equal technologies that just could be the thing that makes me actually consider using them and I love hearing that instead of like sending a cease and desist to a project they're like, yeah, we'll just help you out yeah, they send that memo to Nintendo because they really need to hear that yes, exactly it's one of those things like, how do you deal with someone maybe copying some of your product or just making something available and I see, you know, HeadScale is gonna be probably a lot towards your homelab audience that's why it's gonna be in this as part of the topic or why I brought it up in this particular discussion but it's just kind of cool that they're not trying to stop them from doing it and matter of fact, they're helping them so I think that's just really cool it like that's everything that is, I think that's the way to be nowadays I feel like in the future in order to compete you're gonna have to be okay with not only okay with but you have to be a part of the community otherwise at some point in the future I feel like it's just gonna be hard for companies to get traction because at some point this is gonna be way more the norm than it is now right now it's like they're doing what? That's so cool and progressive that's how it starts, right? But then as it gets going and more companies realize that they can do this and it's a good thing to do then everything's better for it so yeah, I think at this point like we haven't even gotten to the discussion topic yet and I'm already sold on it Yeah, so this is interesting and kind of related so people have asked me and because I've talked about numerous times these different overlay networks and there's a few commercial ones out there that are completely topped about a proprietary in the IT space and I'm not even gonna mention them because they're not worth mentioning and one of the problems I had especially when one of the executives I was in a private forum so this can't be publicly seen or I'd link to it because I wish it could be I called them out because a bunch of people were promoting them and I said, what protocol are you using? Best in class, that's what they reply with I'm like, I'm a network security person I would like to know in more depth I understand it from a marketing standpoint just tell me you're using best in class, cool and I asked a more in depth question and they said, military grade security is how they replied and this person allegedly was a technical representative of the place and I said, and this was turned into a discussion within this forum I said, no, that's not an acceptable answer and then he said, can I DM you? I'm like, no I said, you should be able to post publicly what protocols you're using what your transport layer is how you're auditing I said, I see all these people promoting this and one person admitted, oh yeah, by the way they give me a free copy that's why I promoted it because I'm on their advisory council and I'm like, you guys should all disclose this because you're recommending a product and no one's asked the question to this and why am I the first? And yeah I think you handled it way better than me because that would have been more of an aid hole that would have been like, wow, the BIC protocol I've never heard of the BIC protocol could you please enlighten me more about this BIC protocol that sounds really cool. Yeah, exactly. So you, this is the advantage that Tails God had in the market before people want to surrender their control because this essentially is attaching a extra network adapter directly on your machine your loading software to get this working you're adding an extra network interface on your machine why should probably know the code that goes in there and there's a lot of trust in there so by making all of that code and the DERP server and the transport layer which is just WireGuard completely open source, awesome it is clearly transparently done from a trust level and that is part of the beauty of how this works is instantly I like the software it's been vetted, it's been, it's being used by a lot of large companies obviously so it's hard because anything vetted obviously someone could say what if someone snuck something in, et cetera, et cetera well, Tails has got a lot of eyes on this they're in control of the project they're reputable, therefore not likely someone will slip something in there this is not seeing supply change tax are impossible but a well documented well audited process will give you a lot of that confidence in there and that's what Tails scale is done by putting it in there and I also like that they didn't try to roll their own crypto because I have a feeling that other company was not trying to tell me that they rolled their own crypto because I would have called them out for that Tails scale said, nope, nope, we use WireGuard that is a well vetted, well audited system we're using WireGuard for transport, end of story and it makes it a real important aspect of it that they didn't try to reinvent security because if anyone listens to security now and Steve Gibson is totally right on this particular topic of quit trying to roll your own security unless that is what you do for a living Yeah, he said that many times in that podcast and we refer to that often because we love that podcast, it's great security now by GRC slash Steve Gibson but yeah, he says that a lot because it's true I mean, why reinvent the wheel? That's why we have shared libraries and things because it's a thing already and then they're talking about supply chain attacks I mean, that could happen to any open source project if you look at the I think it was the University of Minnesota debacle Yeah, that was another example of that it could happen to the best of us but the fact that it's open source is always going to be an advantage over not being open source regardless of what could happen Right, and that's why it's so important anytime Steve's mentioned that it's almost always he just re mentions it and then he leads into the story of how a company got completely taken apart because they rolled their own security and someone found a major flaw in it so it's Some of those stories they come up with are just hilarious actually but anyway Endless because there's always someone doing some silliness there Now, let's start with tail scale itself How do you sign up for it? I bring this up because a few people ask me like, well, tail scale do they have two FAs at a normal VPN? They're just a control plane handler that authorizes the nodes to communicate No data goes through in any unencrypted form tail scale and when they sign up they have a WALTH2, OIDC, SAML which means you can sign up with your Google account, your Microsoft account your GitHub account I did that in the demo They don't actually have username and password management They don't want it and I don't blame them So it's always the control plane itself is third-party provider and then you start authorizing the nodes Now, your normal networks work in a way of your hub spoke You know, everyone VPNs into work Everyone VPNs to the VPN concentrator the firewall or whatever the device may be on the node but obviously this is hub spoke and has some limitations to it Tail scale answers the question of what if we did it like a mesh? So if you have a mesh of and as I said, the control plane is WireGuard and the only thing you have to do for any WireGuard node it just needs the public keys of the other nodes to communicate pretty simple and WireGuard's lightweight it's low noise it uses UDP So you're able to start talking to your friend So if me and Jay we want to access a common resource but sometimes me and Jay would just like to send a file to each other we can do so it allows interconnectivity between all of the different nodes and just by handling the keys Now, it's only the public key the private key never leaves tail scale hence the reason being open sourced and wanting to make sure that your machine as it generates a private key it doesn't leave Well, you can look through the code and see it doesn't need to that's not how WireGuard works you only have to share the public key out that's why WireGuard is such a nice simple way to do this Now creating this mesh network sounds fun but let's do some math Tunnel sounds simple but a 10 node network would require 10 times 9 or 90 WireGuard tunnel endpoint configurations to manage that's just for 10 people or 10 devices that you added to the network So every node you need to know owns its key plus 9 more each node would then have to be updated every time you rotate a user in or out that's actually what's being done they're just bumping public keys and rotating them in or out as you add things in or out of the network this is really simple when you see them you can see all the ones within that network you can see who's on there but if you want to remove one of the devices from the network it just updates going we've removed this device please remove that devices public key from every single node so that is as simple as it works your private key stays private this is where everyone thinks well if I'm using tail scale doesn't that violate the privacy now the only security thing that you have by using tail scale is nodes can be added and approved by tail scale but it is up to you to lock down your nodes to say these are the nodes I added these 10 nodes I'm only going to accept commands and this is done through ACL rules from these 10 nodes end of story that's where you stop if someone were to get into your tail scale account someone were to maliciously take it over in some way they can add nodes but because you didn't allow those nodes well they can't talk to you now of note the default behavior is all nodes are allowed to talk to each other so it does require some rule creation they made it simple they went on will allow nodes to talk to each other and the control plane just says all right everyone can talk to each other unless you create rules not to so it's kind of up to you if you want to lock it down a little more a lot of people probably don't do that and for the home web use maybe you know you just like let's leave all the ports open but it's still up to you to create even on the individual devices because it's an extra network adapter your IP rules still apply you can still say SSH but only allow from this IP address and matter of fact probably a good idea to do that after you set it up and the reason why is because let's say I know my computer's a node and my laptop's a node and that's where I do my administrative work from so all the devices I add to it I say you can't accept SSH except for from these two devices and that way if anyone ever to get in my tail scale network or for some reason take over some other node they're like hey I managed to take over a node or I took over a device and it happens to have this tail scale access on it then they want to pivot somewhere else well you're still following the standard security rules or principles of least privilege and blocking that down so I don't think it's as much of a worry as people do using tail scale using their hosted service which right now you get 20 devices for free that's probably enough for a lot of lab people to get set up their pricing is only a few dollars if you want to use it using their control plane is as big a security risk as the comments immediately on YouTube the discussion on right about this it's everyone's like I don't trust it because their control plane they could add a node to my network and where they also make the assumption that the data traverses their nodes so I'm like nope, nope, nope none of that happens at all It's hard some time to I mean privacy is very important but at the same time it's like is the person worrying for a legitimate reason or are they just worrying because they worry it's really hard to tell the difference and it's not that I'm trying to invalidate anyone here I'm just saying it's just sometimes hard because especially for us who are doing videos it's like we need to know if there's a legitimate concern here but we also have to figure out which concerns are actually legitimate sometimes it can be hard to get through but more to your point I vet it as best I can I think that's the best you can do and then you make the decision for yourself Yeah and that's a really it's just understanding it and this is why looking through and having open source code lets you validate all this and this is extremely well documented on your site I highly recommend taking a technical look on your site because if you're not understanding how Sun and Nat protocols work they have such a good write up and it's in the link in the show notes that will help you dive into just how Nat traversal works and all the tricks they've used for it so that documentation level helps Now, this is a walk through the steps real quick each node generates a random public private key pair for itself and associates a public key with its identity pretty simple this is all done on the node but the private key as I said never leaves the node the node contacts the coordination server that you set by default is going to be tailscaled public server but headscale allows you to set your own we'll get to that later context coordination server leaves it and leaves its public key and a note about where the node can currently be found and the domain it's in really simple it says, hey, here I am this is the IP address I have and it figures out the public IP address because the coordination server sees the public IP but it's also looking at how it got there so if you're behind CG NAT there's no port opening at this point needed for any of this it's saying, hey, this is where I'm at and it traversed through maybe several networks before hitting a public IP before hitting the coordination server the node then downloads because it reached out to the coordination server and node then downloads a list of public keys and addresses that it has access to this is that namespace you set up going this is Tom's namespace these are Tom's devices and I've authorized all these to be on the list and then they gather up all the public keys that were handed out by all the other nodes that contact the coordination server the node configures its wire guard instance with the appropriate set of public keys pretty simple to get that working and essentially it's almost like a zero trust at that point because you're only trusting things that were put on the coordination server for public keys now this is where the really advanced stuff gets in this is where it gets kind of fun Talescale uses several very advanced techniques based on the internet stun and ice standards to make these connections work even though you wouldn't think it should be possible this avoids a need for a firewall configuration on any public facing open ports thus greatly reduces the potential for human error for UDP the rule is very simple the firewall allows inbound UDP packet if previously saw a matching outbound packet for example if your laptop sees a UDP packet leaving a laptop from 2.2.2 1, 2, 3, 4, 2, 7, 7, 7, colon 5, 6, 7, 8 it'll make a note that incoming packets from 7, 7, 7, 5, 6, 7, 8, 2 2, 2, 2, 1, 2, 3, 4 are also fine the trusted side of the world clearly intend to communicate with the 7, 7, 7 so we should let it talk back in probability theory this is what's kind of fun of how they did this because UDP and I want people to stop and think about this just for a moment put your network engineering hat on UDP unlike TCP it can be spoofed this is often how some of these UDP reflection attacks happen because we can spoof where it's coming from and we can lie and we can make changes because well I would tell you UDP joke but you might not get it it's because it's not verifying it now that playing on that is a lot of fun because it may have a UDP port open when it talked to a coordination server but the coordination server is essentially going to talk to another server and trying to get it to land on that UDP port going all right you came from 111 and you came from 222 network and you're both talking to me at 333 but now I'm going to lie and say you should redirect your packets over to the other side and by doing that you're kind of it's almost like just bending it so you're spoofing it but it requires figuring out those ports and this is where the birthday paradox comes in and it's a clever way because we know there's only in 65,000 some odd ports but you don't have time to scan them all and you don't want to be scanning everything so then you say all right we can eliminate the first thousand of them because usually things aren't going to be on the low ports when it comes to firewall but they do basically a probability with the birthday paradox and in probability theory the birthday paradox as for the probability that any set of randomly chosen people at least two will share the same birthday the birthday paradox is that the counterintuitivity you counterintuitively think about this the probability of shared birthdays exceeds 50% in a group of only 23 people so they're actually able to just by scaling down and guessing at ports able to guess these ports really fast they've got this really well documented of exactly how they solve the problem they even have a little free Python script to do the calculation that they threw in there if you want to know how fast you can figure out what ports these things are using now once you've done this there's a couple of things that are really interesting about the way they can traverse NAT they're figuring out the UDB ports, awesome and that sounds easy enough because you're usually one hop in but what if and this is where things get a little bit clever and I want people to think about this from an interesting perspective of how this works is if you have me and Jay let's say we were using the same ISP that ISP is CG NAT it and if you're not familiar with carrier NAT it's like having a second layer so the carrier has a public IP address but within that network the ones they would hand out to me and Jay's firewalls are CG NAT or private IP based so they're not publicly accessible I can't open ports but if me and Jay happen to be using the same carrier and tail scale has some external server outside the carrier it says both you guys come from the same IP address and we can see that it's CG NATed because the hops in between it can actually form a connection between the two private IP addresses provided they are able to talk to each other within the same network I actually demoed this when I was going over some of my overlay network and tail scale demo if you have two different subnets even though they don't have direct pass to each other they can route through PF sense without going to an external network and by doing that you're actually not going and exiting out to the public internet it actually kind of created a connection between two different subnets even though the rules may say there is no connections now you're kind of thinking isn't this some type of security violation isn't this breaking firewall rules slightly the rules have to exist and it depends on the firewall but let's say I have a subnet that can contact subnet A can contact subnet B but subnet B cannot talk to subnet A because I've got rules because maybe it's my IOT network but if you reach out from your network and you don't have something implicitly to block it subnet A can reach out to subnet like to get to my camera and because I requested it it's going to be able to send back data it's utilizing that principle to essentially get the packets to talk to each other it's a really cool firewall trickly so it's just really slick how they're able to make all that work now there are certain times and I like the way they actually worded this some especially cruel networks block UDP entirely so this is where and this is a rare exception but there are weird times when you run into firewalls to do this they go you know what we're going to do we're going to eliminate UDP on this network that's a terrible idea one the QIC protocol has become very popular it's basically 443 it's a way a lot of websites are traversing things faster because UDP doesn't have the extra overhead that TCP does it's fewer packets so you ruin people's web experience by doing things like blocking UDP but let's say you're on a network that is super locked down with UDP being blocked this is where it switches over to that DERP protocol we mentioned earlier which is DERP designated encrypted relay for packets once again it's still using WireGuard as a transport layer but this is where your DERP servers come in and Tailscale has a lot of them HeadScale has one as well and it's basically going okay we're going TCP over 443 because well if you're on the internet that's probably not blocked and we're going to just traverse that so you're encapsulating it now that is where you're going to see a significant slowdown by doing it but it's really cool that that is like the absolute fallback so they've covered pretty much all the bases of all the different ways you can get Tailscale connected without opening ports that's the part that really impressed me and it's all well documented step by step in a massive NAT right up the half Wow that's actually amazing like I'm literally installing it and learning it at the same time just seeing I mean because it's going to take me a while to dive in but yeah so far this sounds like an amazing technology Yeah it's just kind of it's a lot to wrap your head around to think about all the different things that they can do like that to be able to figure all the ports align them and make it all work and on that worst case scenario someone has decided to do this it goes well I'm going to go to the DERP server and we're going to wrap it in TCP because it's the only way this firewall will let this stuff out of here is 443 TCP that's what it defaults to and they have a breakdown of that now when you're setting up and we'll get just a head scale if you don't you don't necessarily have to set up the DERP server it's disabled by default when you self host it's enabled by default when tail scale uses it because there's some extra steps you need to do to get the DERP server running but for the most part I've not needed it because I haven't been behind any networks that go as far as implicit port blocking I don't see them as much anymore I see them occasionally I mean there's high security environments and what some companies may do and I had a couple of people that kind of overthink this but they go shouldn't firewalls block all outgoing traffic and I'm like not really because you're going to get a headache because when you do that you'll find so many services that break and this is a common question just in general firewall setup shouldn't it be block all outgoing unless I implicitly allow it I'm like good luck buddy you've now created a job for yourself and because you'll realize there are of your ways the first time you run an apt update command yep you'll run in so many problems by doing that doesn't mean someone hasn't done it doesn't mean someone's not going to go I want to implicitly only allow connections to leave this network that I have designated that's fine but you just got to remember if you want things to work you'll start finding it just breaks everything so it's why it's not as commonly done it was actually more popular years ago because we're so few things on the internet that need to get online people just need to get to websites so you would only allow that so you know bring us all the way to 2022 where the average general home user has a dozen devices that all reaching out in beaconing you're like oh yeah I can't just block outgoing because I'll spend a lot of time creating firewall rules to it so usually allow outbound so generally you're not needing it but it's cool that they have it it is a fallback now that's obviously where things get high latency and slowed down because once you take a fast UDP wire guide protocol and we're going to wrap a UDP and a TCP and relay it and bounce it off of an external server to get back to where it needs to be yeah now you've got your latency is going to go up it's not the most ideal situation but good that you can still connect so I can still SSH into things I can still get some data across so all that's still functioning and working so I'm glad it has that fallback on there all right now what about hosting it yourself now a couple things about tail scale it's all done and so is the head scale tool it's all written in go and for the most part I probably should have mentioned this earlier but this still applies to both head scale and tail scale it's really only made to support TCP and UDP connections and they have a reassembler essentially this is in their documentation as well for doing ICMP packets so you can ping, you can use TCP and UDP what it's not designed for and this is important it's not designed to route every other protocol that you may want to use you're like well what about MDNS and using my Chromecast over it I'm like you need to use certain streams for this so this is not exactly ideal for that right there so that's yeah you're also going to lose some speed because it's all written with the wire guard go implementation that makes it portable and works with everything and I know there is a kernel implementation wire guard for both BSD and for Linux but the way the encapsulation works is for that so all this is written in go so there are certain overheads where go just isn't going to be performant because the context switching because go operates in user space with this so those are all notes on there now let's get over to hosting it yourself and this is kind of clever because they have head scale tail scale is a modern VPN is how it starts out and it works for like an overlay network as we said everything in tail scale is open source except the GUI, clients for proprietary OS as Windows Mac and iOS and the control server and that's what head scale is is replacing that control server control server works in exchange point for wire guard public keys as we said and head scale aims to implement a self hosted open source alternative to the tail scale control server head scale has a narrow scope and instance of head scale implements a single tail net which typically what a single organization or home personal stuff will use so they're only building a single tail scale network which is fine you can add separate name spaces within there so there's a lot of support for doing you can still connect multiple servers most time people you spin up a head scale server so you can do a single thing so here's the full features it has full base support for tail scale features configurable DNS including split DNS node registration and single sign-on very open ID connect pre-authorization keys now the pre-authorization key is important because that's how some devices want to register there's different ways you register tail scale devices you're going to run the tail scale up command and say I just want to register this device and it creates a link you click the link and you're signed into your tail scale account and away you go by creating the pre-authorization key which is another one you're basically saying tail scale use this key and then you pointed at the server that generated the key by default if you put no server it goes to the tail scale servers but you can specify wherever your head scale is now head scale has to be run somewhere hey and preferably like on our sponsor Linode you have to run it on a public IP address you could do testing internally absolutely but obviously that tail scaling around your own network you could already talk to everything on your own network so it doesn't seem like the most ideal situation you want it on a public IP that way if me and Jay wanted to talk we're both stuck behind CGNAT we can't open up firewall ports we can't figure out how to communicate but we like to share something we like to share a server we can set up a tail scale node we'd set up a head scale node I should say and Linode we need the most minimum because the traffic's not really traversing there unless you get all the way into the DERP server not likely it's just coordinating the key coordination it just sends kilobytes it needs like nothing you can do the lowest level what's the lowest tier that Linode has is it like $5 a month right now? Yeah, it's the shared CPU 1GIG one CPU I believe I'll verify Yeah, but it's like the most basic so it's not intensive even if you have a large scale network it's just sharing some if you look where you see how small the keys are like that's it that's all it does I'm like yeah it just coordinates the keys no nothing complicated there now there is and I have not used this this is a feature that they added since I did my original video on tail scale there's like tail drop file sharing it actually has that in there so they've got some tools for just sharing files with tail scale as a protocol so it just goes through a tail scale network so that's kind of cool it also supports multiple IP ranges in a tail net supports IPv6 which will make somebody really happy I've only set up an IB before it does support route advertising ephemeral nodes and of course a DERP server now the route advertising as I mentioned is a feature that's really cool that in tail scale for setting it up on PF sense or really if you wanted to build and handwrite your own Linux router could do that too and then you say this is a route and I'm advertising these routes to all of my machines on this particular tail scale namespace and then you can say all right I would need my phone to get to this router as my PF sense is the example I use from there and now I can get to all the devices on there so this is supported not just in tail scale but in head scale as well and it also supports that ephemeral node thing I think the ephemeral node is kind of a clever functioning at it if I only temporarily needed one of you to access something I could create an implicit rule and say here's your ephemeral key and ephemeral node go ahead and attach one I can delete it or set an expiration date on it but two the moment you disconnect you can't reconnect again so if I had a temporary server I needed to transfer files to or temporary thing I needed to share with people I could lock it down with some ACL rules I know when you're assigning these what IP address you're gonna get I can say here's your node connect get your thing done and this is good for maybe even a temporary contractor who only needs to be in for a certain amount of time you can allow them to your tail scale network to help administer something so I can say hey Jay I need help with setting up this wiki Jay can get an ephemeral key he logs in he does this thing hey Tom is it working? Yes great as soon as he just connects that way if I forget the ephemeral node means it goes away he can't reconnect he can type it again he doesn't do anything I have to generate a new one so this is kind of nice when you're using these pre-auth keys now the one thing about tail scale tail head scale head scale does not have a UI at the moment and this is you know recorded in July of 2022 so this is all command line managed now it's all simple end curses you know type the command you spin up the server they don't even have an installer for it essentially you just download the binary they have an instruction how to do their config file it's all pretty basic I haven't done a video tutorial on it yet but I probably will to get people's steps on there it will work with just an IP address but I'd probably recommend having a DNS entry for it that way if you ever move the server or the database around which it's really simple it just sets up a SQL like database but it actually can use if you did this at a large scale install you can use other databases I believe it's under a roadmap I think something I seen one of the write-ups for someone said they were using Postgres with it but even with a SQL like database you're not really tracking much you're just tracking who's got what public key and how many nodes you have on there and I imagine it's quite scalable even with just a SQL like database you just control even the command line though for setting these up viewing them but for the most part once you've added the nodes unless you're adding and removing nodes a lot you're really not doing much with it it just sits there idols and telling the servers that beacon out to say here I am and if their IP address changes like phones do where you're moving out the laptop just coordinates making sure that the connectivity is facilitated like hey, Tom's laptop is on this IP address now we traveled, now it's on this IP address so it's always just creating that route back for you so it's not much on there and of course if anyone were to compromise back to what we said if someone compromised tail scale someone compromised the server you set up because whoops you forgot left SSH open to the world and you didn't have a strong password you didn't have key authentication what could they do? Well, turn it off that would be annoying second they could add more nodes but you would also notice the nodes being added and back to my original hey practice principles of least security you know what IP addresses are being assigned they can't assign another IP address that's already assigned on there they can't re-register these nodes to impersonate you easily there because you'd have a conflict so yeah, as long as you're practicing all the principles if someone compromised it the worst case scenario is that they'd be able to add more nodes and hopefully you've protected against that because well, you've locked it down to all the nodes you wanna have pretty simple well, simple in theory complicated if you actually take the time to read all of it on there so well anytime you have network things and you know being developed I mean the amount of work the people spent making this happen is immense so yeah, there's definitely a lot of layers yeah someone said could it be run on a DDNS address yeah, you could run it wherever you want the you know it just needs a common place if you're using DNS which I think it'd be preferred rather than IP wherever you run the headscale server as long as it has a common place because you've specified I think mine is headscaldemo.launchsystems.com I think that's what the I'll actually be revealing that so people can poke at it so I'll leave the server up and running so people can do the thing because it's all just demo stuff I'm setting up for the video but I leave it at that now if I ever change IP addresses I'll just update my DNS entry I have for it so yeah, it shouldn't matter because all the nodes I'm connecting they're all looking for that same DNS entry this allows me to shuffle it around so that's why I recommend DNS so I don't see any problem with using DDNS someone says will it work with CGNAT like Starlink absolutely it'll work with CGNAT and of course Starlink is becoming more popular and yeah, that solves that problem too for Starlink cool yeah, it's a lot like I said, I'll probably work on the tutorials the write-ups are solid I misread a few things definitely use their config file and read their config file properly and do the things they say in your config file because I thought I was aggravated with it when I was talking to Jay I think on Monday I expressed some aggravation for lack of documentation turns out, I was RTFM-ing wrong so I overlooked something and once I stared at it and realized my mistake yeah, there was a problem now I haven't tested this yet and this is the part that I need to do for the tutorial I wanna make sure I get the DERP server set up without the DERP server it's really easy with the DERP server I need to make sure I'm understanding how they're using Let's Encrypt because they have all the ACME stuff built in for setting up Let's Encrypt because DERP servers do seem to need Let's Encrypt to work because they're working over 443 of the cert in the case of a DERP server you have to be using DNS to make that work so that makes sense yeah you need to have that so it can do the challenge response and get a certificate installed and I see a few people mentioned and this is the part of the documentation at least for me it wasn't 100% what it looks like for the DERP server they set up a reverse proxy with Nginx and it's not necessary to do but it's something you can do so I'm not sure exactly why I think it's because Nginx works better but I can't tell if that's part of someone else's old right up in Reddit or if that's something you need to do now because it appears to be built into the head scale itself but for like most part not everyone's gonna need the DERP server I got it running without it even with my restrictive firewall rules and things like that playing around with it I never had a problem getting all the devices connected or PF sense yeah I'm running through the install instructions right now they're super easy like I'm just copying, pasting commands and doing exactly what they say to do and it's, I mean things are happening the way they will so it's great when you read them and then you read them properly and then they work you know it's like a crazy now yeah a little bit overboard doing it right the first time, come on who are we I know why spend 10 minutes reading documentation when you can spend two hours mashing the keyboard and carrying and you know all that other stuff doing, why don't it work why doesn't it serve me so well you're the DERP this time yeah yeah I did see and I don't have an answer for this is head scale good enough for production I haven't run head scale over time long enough to say that I would trust it for production the project's been around for a little while but I don't know I just don't know the answer does it seem very actively developed? Yes does it when did it start? I don't know this is not an old project this project's not been around for a long time but then again, neither is tail scale so I don't know where the exact start date was for this project but they do seem to be doing a lot of releases which seem to be mostly feature enhancements so that's at least a good thing let me see what was the first release how long ago was this project been around I guess that's probably a fair question 2021 so yeah I whether or not you want to now the project's based on this and as I said version one started in June of 2021 so it's not a particularly old or mature project so I don't know but if you're a good network engineer I don't see why not spinning it up and trying to see what happens on there worst case is you switch back to actually using tail scale for the control plane so you actually have a fallback plan if it doesn't work well, there you go yeah so hopefully that makes sense I don't know a project has been around for a year that's open source that has no business model behind it that's just a bunch of code contributors I don't know that I'd run it in production is probably the where I land on it so if one of my clients said hey Tom, should we run this in production I'd say use tail scale they're solid they've been around for a few years they're a big company with a good staff they care about privacy, security and everything else so I would absolutely say use that product I trust it it is trusted by many large companies for home lab users worst case is you lose some connectivity that you somehow had connectivity to establish a connectivity so you have a fallback plan and you're just inconvenienced if it goes down yeah, but not no, I agree I think for production use I mean, enterprise stuff you want it to be around for a while and prove itself for a bit longer than it has so far yeah, I mean, they have a lot of people contributing to it but that doesn't necessarily mean and they have a sponsor button and I may sponsor it so it comes down to currently, if those people that are contributing get bored or get bored is probably not the right term this happens a lot in open source projects where they start out really well we get excited about it but if they have no way to do it or it's a hobby project for someone because they have a day job their day job may become more demanding or in some case they may have children or something that become more demanding and they go, I don't got time for this project and then the project just stops and then that can be a big problem and that's how I feel about this it's in its early beginnings unless the person commits full time to it I haven't really dug into the person it seems like a side project for them but hey, nonetheless, it's open source it can be forked and someone else can pick it up and run with it so that part's a little fuzzy I would say I'm certifying it for homelabies here you go that's fair enough it's a great learning opportunity especially just understanding network engineering better you will come out of this understanding things better more than anything else you can play around with firewalls more you can start anytime you want to understand security or how things traverse the firewall having a task, having a goal of I want to get this set up I want to spin up a server I want to make this happen I want to set up an exit node to tunnel my traffic through where you have to do a few extra steps those are all things that will help get your network engineering so if you decide to just do this as a hobby or do this professionally you have a better understanding of how things actually work by going through those steps take notes along the way of that you did because even though I didn't read the documentation properly I did make notes that's how I knew how I screwed up because we're Tom knew where you went wrong good boy yeah but that's what we have for Tailscale thank you all for joining us that was a lot of fun so I learned a lot myself this is going to be I'm just going to keep plugging away because I've been listening and actually like many of our audience members just following along and having fun so these instructions are pretty easy and straightforward so if you're interested I think it's probably a fun project yeah this is where you know not everyone realizes and where Jay is really good at automation I have a little bit more experience in some of the networking stuff that's why me and him will share so much things is he helps me out with a lot of Linux automation stuff for commands and then you know sometimes thinking about how some of the networking stuff works there can be some of those details it's it's a lot there's a lot to know that's why we bounce ideas off each other that's why Jay was a little bit more quiet on this one this was more of a consuming of that knowledge I think we pretty much take turns being the quiet one because it could be something like I know very well or something that you know very well I feel like though as long as we've known each other it's always like we kind of compliment each other's I don't want to say weaknesses because they're not it's just like your focus in my focus are completely different but when you put them together they equal like one main thing so it's like content just flows pretty easily at that point so yeah that's it someone asked in this question TCP UDP ICMP that's where the protocol level stop in here and now we have BIC oh yeah BIC protocol I think I'm going to call call it that whenever I deal with marketing and they sound like they or they're trying to sound like they know technology I'm just going to from now on just say oh that's the BIC protocol got it and that'll just be our inside term before it's BS yep but we put in a BS protocol that's too obvious when you're talking to people so yes absolutely alright well thanks everyone for joining us links are already in the description of this video and of course if you're listening to the podcast you're going to be in the show notes there but it's basically all the stuff it's all the stuff that is on the tail scale site you can check it out I also left the link to headscale easy enough to find so plenty of reading to do get started hopefully all of you have fun with this project alright thanks no thank you