 Tom here from Orange Systems and we're going to talk about tail scale. Now I just did a video on tail scale versus zero tier. I'll leave that link down below. I'm also aware of the other project that's referred to as head scale that somehow ties into the tail scale clients. I will leave a link to that GitHub project for those of you interested in it, but it's out of scope of this video, but I am aware that it exists and I will leave links to it for those of you that would like to play with it. It is currently not on my roadmap and also because I have not deployed commercially and outside of just my lab and testing any type of tail scale deployments. My reason for doing this video is well, I compared it to zero tier thought I'd do a dedicated video just to bring up exactly what tail scale is for an explainer. And just because a lot of people seem to like the project and I mentioned it to me, I know a lot of IT people said they like it. So I've heard good things about it is basically where we're at. And of course I did my testing. I found it relatively easy to set up. But I do want to make sure it's clear. Tail scale has nothing to do with this video. It's totally sponsored by me. The opinions are my own. I've not reached out or talked to anybody at tail scale. I did tag them in Twitter when I posted the video the other day and they retweeted it. So there's that. That's my complete affiliation with them just so we're clear up front of this video. Now we are going to talk about a few of the details, a few of the service offerings and of course security concerns I have with this and really any other product and I brought that up in the previous video, but I will be repeating that a little bit in this video just to make sure we're clear on where all that stands before we dive into those details. First, if you'd like to learn more about me and my company head over to LawrenceSystems.com. If you'd like to hires for a project such as network consulting, there's a hires button right at the top. If you want to support this channel in other ways, there's affiliate links down below to get you deals and discounts on products and services we talk about on this channel. Now the first question I want to answer is how tail scale works. Kind of a general overview, but don't worry if you want to dive deep into everything, it is a very well documented system here. So tail scale solves the problem of connectivity with WireGuard, but it doesn't work like a normal VPN system. Now I have videos about WireGuard and how to configure it manually and for those of you that have not watched that, it's a little bit more in depth and it's not something that maybe the average dabbling user would say is easy. Now that's a very subjective whether or not something is easy to set up, but WireGuard is just a protocol and not necessarily a tool to manage users and devices easily. It does allow for easy management with other tools and that's what tail scale is a tool that sits on top of WireGuard to solve connectivity issues and they have demos of what a traditional VPN gateway might look like where you have clients on the outside, some servers on the inside and a subnet, everything traverses into this traditional gateway. The way it works and we'll jump right over here to so-to-speak like the node here, the WireGuard protocol that they use allows it to add an extra network interface on each node that you add. Tail scale does all the orchestration to allow all these nodes to figure out where they are and we'll cover that when we get to the demos. And each of these nodes can talk to another node. Essentially it's like having a statically assigned address that no matter where these nodes wander around, whether it's double-naded, triple-naded, whether they move between networks, it is always able to find them at that same node address that tail scale assigns the devices. This is an important aspect to how this works and it will automatically as the picture depicts here, just connect to the node and figure out the best path traversal to get there. Now this does not require any firewall rules, this does not require any special settings, tail scale takes care of that for you. How does it do such magic? Actually this is a really solid right up there here. This is just great reading if you want to understand NAT traversal and how that works. They dive deep into NAT traversal as a topic, including how their servers work, and especially some really interesting when they're talking about picking at, hard NAT, what they mean by each of those. Everything you ever wanted to know about NAT, they've done a good job of explaining and explaining how they absolutely figure out how it's going to work. So one of the things they have in here as well for people asking like CG NAT, here's your double NAT situation, but one of the clever ones that they can do, and this is when you're doing KeriGrade NAT, is create a loop between two separate home routers within the same KeriGrade NAT that is able to get through a series of this NAT traversal trickery that they use, and it's well documented. It almost looks like magic, but you can read through it and document how it works, but they're able to, without leaving the carrier to the greater internet, get these devices talking to each other. My demo I set up is going to work a little bit like that, but in the case that there was no connectivity between there, it does offer relaying and all this is done completely behind the scenes and facilitated by the tail scale system, and we'll do a demo on how fast that works. Of course, the next question that people want to know is pricing, and I do have that pulled up here. This pricing is as of right now, August of 2021, and things can change because that's life and prices change, but this is what they do have. And if you are wanting to get started, it's one user, 20 devices, one subnet, secure peer to peer connections, SSO MFA, sharing magic DNS and more. So pretty good feature set for hobbyists or people want to get started with it. Easy enough to do of note supported SSO identity providers. I like that they put this on here. And this is an important thing to think about. They do not have like your own identity management that they're doing inside of here. They rely on you signing up with a third party to do this. So they do support that, but no, however the tail scale never handles authentication itself. You can send enable these like MFA features with your identity provider. This kind of gets them away from dealing directly with identity management. And they do say right here, we do not sign up with email addresses by design, tail scale is not an identity provider. There are no tail scale passwords account recovery, etc. I actually kind of like that they've done this. It puts them like away from it. So they are supporting all these different SSO providers and not dealing with usernames of passwords themselves. I have no problem with that. But it's something to note in case you're wondering how you sign up. You'll either need it. If you're signing up for the free tier, a Google account will work, a Microsoft 365 account will work, any of those identity management ones, or you can also use a GitHub account to sign up. Next question. What does tail scale work on? Well, tell scale has clients for Windows, Mac OS, iOS and Android. So they've covered the major, most common things that you'll run this on. But they have Linux support, which is really enough to say, all right, and when you go to the instructions, this gives you quite a wide variety of devices, because they don't just support one version of Linux, they support everything from the Amazon, Arch, Cent, Debian, Fedora, Susie, Oracle, Red Hat, Raspberry Pi's. So yes, they do have if you want to include Raspberry Pi's in this, because that's running on an ARM processor, they do have the one specifically for that. So it's not just Linux, but also Raspberry Pi's specific version of Linux or specific compilation there. And they do have the 32 bit and 64 bit variants. And once you've downloaded the client, whether it's for Linux or Windows, it generates a little URL when you follow the instructions, after you say tail scale up in Linux or load the Windows client really straightforward, and then it joins you to the dashboard here. Now the dashboard is pretty straightforward and simple. Here's all the statically assigned IP addresses that tailscale assigned to each of these nodes. And this is pretty automatic. The all or external is an interesting feature of tail scale. The external ones are when you share a device somewhere else, and you can click on this. So I have my Debian lab one lab to my Windows 10 lab, I've been testing this across a couple different things here. And if I just want to say, all right, we're going to say edit machine name, review route settings, because you can build routes between there, but share the machine would allow you to share it with someone else, as in someone else with a tail scale account. And you want to have your machine also available to them in that account. I think this is a neat feature. If you have a friend using tail scale, if your couple home users want to be able to share one device in another network, but not give them access to everything, you can share and have access to it. The services list is also interesting because what this allows you to do is see the services that tail scale has on the nodes running as in what it sees running. So I have IPERF that was running on this one. I'm doing speed tests. We'll do that in the lab here. SSH is running. So it sees these IP addresses it can scan for services and say what is available on there. And you can then click like copy SSH command kind of neat. Now, what about access control? Yes, they do have ways you can create rules between the devices. So this is goes a little out of scope for diving into exactly how to create those rules. I don't use tail scale commercially. So I've never really played with it, but they once again, plenty of documentation on exactly how to create everything from ACLs to breaking down all the little traversal rules that you want of exactly how you want the nodes to be able to communicate with each other. So they've completely covered all of that in here. They also have some DNS things that you can do said once again gets a lot of scope. It's called their magic DNS to you register unique domain names, or you can also push your own domain name servers in there, which may be something that you want to do when you're dealing with a decentralized network essentially like this creates, you can get really creative with all this is kind of my point on here. All right, now just by default, all the nodes can talk to each other. So we left everything at default for this lab, we're going to sit up here. And now let's talk about the lab. Now, here's a little diagram of how the systems in our lab will be set up and how we're going to do the testing. Here is one cloud server that's actually sitting in digital ocean connected to the internet, of course, then here's our firewall at the office. Here's a lab firewall we have at the office. So we have this firewall, this firewall, this firewall. So we've double-naded this and put it behind two firewalls. And this is on a separate network. So this is on the 192 1683 network. This is on the 172 1669 network. And matter of fact, let's put this little dot, dot in here, this can communicate with this. So I'm able to ping from DB and lab one over to Debbie and lab two, but the reverse isn't sure. That's why I put the little arrow on here to represent that it's kind of a one-way street. I did this on purpose because this is something clever that tail scale can do because each of these devices, that's what these red dots represent, are beaconing out to tail scale. Tail scale then uses, see the natural reversal article that I will be linking to that I mentioned earlier, and it will use all the natural reversal tricks it can to figure out the best way for all these different nodes to communicate to each other. And it does this very fast and dynamically. So we're going to actually break the connectivity and show how it can recover very quickly and choose a different route. And when no route is available, which is including things we're going to break. So we're going to make sure that these two devices cannot communicate with each other. It will then result and fall back to relaying. So that's one more trick it has up its sleeve. If for some reason, in this case, the reason being I create firewall rules that blocked these devices from acknowledging each other in any way, then tail scale will relay it off of one of their external relay servers. And that'll be part of the demo. And that's what this right here right now exists a rule that does allow these to talk to each other. And then we'll actually put a block rule in that says no, they can't talk to each other. But please note, these are the different node IP addresses. So the tail scale IP will be the same 100, 102, 68, 99. And then each of these has a local IP address screenshot if you want. But trust me, I'm doing all of these to show back and forth how it pings. Now, when we go back over here to the demo, I've split the screens up using Tmux here. So we can show here's the devian lab cloud. Here's a devian lab to the devian lab to is the local one in my building here. And then to see the connections on any device. And this does work in Windows as well. It's tail scale status. Windows has a little UI that shows it in the bottom corner. Same concept when you're doing this, but tail scale status, it talks about the connection. Now, we're going to watch the connection literally watch dash and one, which says update every second, the tail sales, tail scale status. So right now, we're going to ping devian lab to local address, make sure we can ping it. That's this computer up here. And we can ping the local IP address proving that we have connectivity. Now, the opposite is not true. This is not able to ping back to this particular computer's address. And we can do IPA. And it's 40.39 completely different subnet. And if we try to ping it, it's going to fail. And if we try to ping 3.217 that firewall that is behind it's also not allowed, we've isolated this on a network, which means it's essentially that, like we said, one way communication. So now we're going to go ahead and ping clear, ping 192, whoops, actually we'll do it, devian. And we'll say lab to tail scale IP address, which is that 101 19 2115. And you can immediately see it created an active direct connection from this computer here through that firewall, and is now talking to it over the IP address. And we can see the data flowing back and forth. As a matter of fact, we can actually run IPerf on here. So we'll set this up as an IPerf server s. So it's listening, and we'll do IPerf three dash C for client and we'll say WN lab to tail scale. And we're able to get pretty reasonable speeds on here just under 300 megabits here. So not bad. What if we went local and I'll bring this up because if we contacted local IP address, what is the potential that you could get across this network? Well, these, because of the devices are routing through and the layers are going through, it's only at one gig right now connectivity between them. So yes, they're able to talk at one gig with the overhead of tail scale on this local network, they're able to talk at at the speed of these machines and how it's configured pretty reasonable speed right now and just under 300. So not bad in terms of speed. And you're not always going to run into that where they're on the same network, usually more likely dealing out devices that are in the cloud. And it's not going to be absolutely the fastest transfers, but there are some limitations based on CPU usage and things like that, that you'll run into. But that's these machines here, I'm not going to get too deep into it because testing every scenario and every machine for their speed and what wire can do on each machine is going to vary just an FYI, not a speed test video, just connectivity. Now, this active connection that we have here, we're going to break. Now, by the way, even though we have this active connection, if we wanted to say, let's talk to this lab one, we can't get to it via its local IP, but just so you know, while that connection is established, I am able to whoops, got to ping right ping. It has no problem connecting to it, even though, as I pointed out, it can't connect to the firewall or anything else. But back over to the point here, we're going to show how it breaks the connection and how it's going to move into relay mode. So actually turn IPERF three back on. So we were getting, you know, about 300 here. And that's for server. So let's go back over to here. And this is the rule that I have inside of PF Sense, rule creative, but not enabled. So it can enable that rule. Now, of note, depending on the firewall you're using, having the rule doesn't necessarily mean it will automatically stop communicating. Matter of fact, if I'm willing to bet that it's still working an active direct connection. And just for those of you that aren't familiar with the way firewalls work, you have states that are created, and there's those states that we have. And we want to make sure we kill those states off. Make sure to just kill them like this. There we go. Filter. All right, no more states. Just want to filter them, find them, destroy all states that were between there. And now there is no more way for it to communicate. We have force blocked it. And that's what expires those states out was me forcing it. They'll die over time. But sometimes when you create a rule, unless you have your firewall configured to really kill those states, they will stay up a little bit longer. All right, the rules are reloaded. The connection is set to relay. So now if we do that same speed test, we're going out to their relay, and it says NYC also means New York City. So it's relaying out to coming back in, which gives us a much lower connection speed. It's probably hitting probably, like I said, just under 20 here. But I also have restrictions on my network. So my lab stuff can't take up too much bandwidth. To that matter of fact, if we go to the cloud here, and we do another IPERF test IPERF three client and it will be WN cloud tail scale. It's able to go out hitting very similar speeds. But let me show you what happens when we hit the public IP of it. It's a little bit faster. There's those drops again back down slower. There's speed restrictions on here when it leaves my network out to the main internet that, you know, like I said, it's a bandwidth restriction we have internally that we're doing. So it's actually able to go faster, but the restrictions will keep it from going faster here. So mileage may vary, tooth factors are of course, how fast is your ISPs internet, how fast the internet on the other end and how fast is the machine to be able to handle all the wire guard packets. But back to these connections here. So we have another active direct connection here. This one's in relay. This is the part that I found really interesting and kind of have to split the screen to do this. We're going to go ahead and get this rule ready to be applied. We're not applying it just yet. Let's go ahead and kick off a speed test while we're doing it. So I want to do it under load so to speak. So IPERF three C W lab two tail scale. So we're doing the speed test here. Apply changes. And it's taken a while in the background PF sense, reloading the rules. And instantly it says, Nope, we're able to go direct again. It's kind of neat how fast tail scale was able to switch that I was playing with a few different scenarios on here. I just wanted to cover this one for the video here. But as fast as you set these rules, it is able to reestablish those connections. So I was overall impressed with that particular feature being able to go through change rules, kill it, change it actively. And I didn't have to do any service stops, reload or wait a few minutes, literally seconds later, it was able to figure out the best connection and redo that connection back to direct from relay. So overall, I thought that was pretty impressive with there and pinging other devices was not a problem either. So if we go and we'll go ahead and ping this one. And once again, instantly establishes, this is yet on another subnet again that I have the windows on, it's on the 10 dot 13 37 network and no problems it is able to establish that by the way, the 10 13 37 network has no access back either at all. So it's able to traverse this and you can do the same if I wanted to initiate from the wind 10 lab network, because tail scales figuring out the best way to traverse all of these all the time. So my overall on any of the testing I've done on my lab, pretty impressed, it seems to work quite well. Now that's all fine that I tested in the lab. And as I said at the beginning of the video, I have not tested this commercially. And my thoughts are though, after testing it in the lab and breaking things and of course, putting a few out in the cloud and going back and forth and playing with it just for the last few days for the first demo that I did we can zero tier and of course the demos I'm doing right now. I think it's a pretty solid product in terms of functionality and how it works. Now let's get to the security topic, because I don't want to leave you with that being unanswered. And my thoughts are on security of using wire guard is a great idea, where guard is a great well vetted VPN protocol. They don't use the keys inside of tail scale. And what I mean by that is tail scale does not have access to the device node keys. So that means they don't have visibility into the traffic that's going there. They know that there is traffic between node A and node B or any combination of them, but they cannot see within the traffic. So that's great for security, where there is potentially your threat surface with these type of services and specifically tail scale, but this is more than just tail scale. I mentioned this one at the other video with zero tier is if someone were to take over that control plane someone else gets your single sign on and gets into your tail scale, people could add other nodes, if those nodes are malicious, then that could be a problem. Not to mention, if any of your nodes become compromised, that's a threat surface, you always have to think about that if you have deployed this as a solution, because while it's easier than setting up a VPN, when you have a lot of people doing the work from home, and you want them all to have connectivity, if one of those nodes goes bad, and you don't have ACLs that say no on talking to each other, then that could be an attack node. That still doesn't really change from other VPN solutions from that aspect, but the thought of someone getting control of the control plane and adding potentially bad nodes in there, that's something that has to be considered. Now, one thing to say about that IP authentication or just knowing the IP address and being within the network, that's just a layer that you're protecting against. And obviously, if they get inside, one layer is peeled away, but it should not be the only layer of security you have. And I say it like that, because let's say you have tail scale tied to some type of line of business application, or whatever it is, the thing you want to have access to your server of sorts, that runs whatever things you want. Generally speaking, you're not going to accept any connection, you're going to accept connections that have been authenticated with username, password, or whatever challenge response set up you have an addition to. So it's just something to think about it does overall make a pretty secure setup, because you don't have to open up any firewall ports or do that configuration. But there is of course that risk of if someone gets a hold of the control plane itself itself of adding those bad notes, but you should be mitigating that just in general, having username password is the same thing with VPNs, where VPNs keep people from being able to see some of those privatized applications. But once you're in with the VPN, you generally still have to authenticate again. Now this gets more complicated if you have a lot of Windows nodes, because of the recent 2021 discoveries in Windows that allow people to kind of easily, as of right now in August of 2021, escalate things, especially things like print nightmare, and there's a lot more that can go wrong than we realized previously, or maybe some of us that worked in security kind of knew there was something hanging out there with it. So there is a lot of different things and considerations on there, but that just goes out of scope of this video and topic. It's not really a tail scale problem. It's a hey, how do you deal with someone getting on the network and being able to get to your domain server? And do you have all the different services disabled that are potentially problematic? That's topic for a different discussion, just the thought about security in general. Overall, despite not using this commercially, all my testing with it has gone well. I think it's a pretty neat product. And hey, you can't beat the price of free to check it out and test it out for yourself and see if it's a solution that works for you. And as I said, these are my own thoughts have no affiliation directly to tail scale. There's no offers, there's no affiliate link, there's no nothing to sign up. If you're interested, click their single sign up systems and external identity and management system and sign up for an account and delete it if you don't like it. That's about it. All right, thanks. And to have a more in depth discussion about this can be reached over in the forums or say hi on Twitter. All right, thanks. All right, Tom here wanting to add one more little piece of information to so we're clear on this. The lab systems are running on the same server. And this server is a Intel Xeon E5 2670 at 2.6 gigahertz. Each of these do have 16 cores assigned to them. And I wanted to do that because of the speed test thing and show you what it looks like with the cores. So first we're going to do a local I have them on the same network right now not through any firewall so essentially direct connecting and not using tail scale, they're able to achieve about 15 gigs a second between these two servers. But when we go back to tail scale, and even though they're local and on the same thing, and these can talk at that rate, and we're only getting here that same number right around 300. And this is what the processor usage looks like when you're doing that. I just wanted to add that little bit. I didn't try every possible scenario, but I figured 16 cores with this Xeon processor is quite a bit of horsepower to add to it. And I tried bumping the cores up. It was the same at four cores. It's the same at 16 cores that transfer rates aren't really any different here. And just so we're full disclosure on how it's done, tail scale and status. There's the connection active direct connected by 14. And we'll do this right here IPA. And you'll see that this one is assigned that one seven to 14. So just to be clear on the testing side of it, 16 cores each assigned to this and still no faster speed than that. Despite the ability of these two on this, because currently for this they're on the same subnet able to talk to each other at the 15 gigs, it seems to be just stop set about 300 megs a second versus the 15 gigabits it actually is capable of this wanted to add that in there for those asking. But yeah, these are the hardware spectra here. It's the Intel Xeon E5 2670. And hopefully that helps if you're curious about some of the specs and the lab equipment we're using. All right, thanks. And thank you for making it to the end of this video. If you enjoyed this content, please give it a thumbs up. If you'd like to see more content from this channel, hit the subscribe button and the bell icon. To hire a sure project, head over to laurancesystems.com and click on the hires button right at the top. To help this channel out in other ways, there's a join button here for YouTube and a Patreon page where your support is greatly appreciated. For deals, discounts and offers, check out our affiliate links and the descriptions of all of our videos, including a link to our shirt store where we have a wide variety of shirts and new designs come out well randomly. So check back frequently. And finally our forums, forums.laurancesystems.com is where you can have a more in-depth discussion about this video and other tech topics covered on this channel. Thank you again and we look forward to hearing from you. In the meantime, check out some of our other videos.