 Tom here from Lawrence Systems, and we're gonna dive into zero-tier versus tailscale and versus is not exactly a Head-to-head competition where we're gonna decide a winner each of these tools solve a problem in a similar way But have a transport layer that makes them different and some nuances the way they function that of course make it a Different product all together for which one we're talking about now the reason I chose these two products And I'll also give an honorable mention to the Nebula tool, which I have reviewed before and I have reviewed zero-tier And I have several videos I'll be linking to below with zero-tier But then tailscale comes up a lot because it is commercially supported just like zero-tier is both of these being popular Commercial and Cryptographically sound as we know of today They've gone through levels of code review with the underlying technologies that they use are good commercial solutions that are Trustworthy now that is not to say there's not other solutions out there It is not to call them untrustworthy But one of the challenges vetting any of these particular products of what or not they've gone through some type of review process Or have commercial support before you decide to deploy them Other reasons I'm covering these two people ask about them There's real obvious answer But the other side of it is they do have free tiers and public-facing pricing So it's really easy to demo these different products without me engaging with either one of the companies which I have not I have used the product. I've used tailscale on a limited basis just for the demos of this video I've used zero-tier both as a solution that we've deployed for clients and recommended for clients And of course I've been messing with it for oh, I don't know maybe over a year since I did the first video on it It's actually been a popular solution But I wanted to cover these and cover some of the nuances cover some of the differences and talk about SD WAN products in general And I once again will leave a link to the video I did on nebula because the one thing we can get out of the way up front is there's not an easy way to Host zero tier as a completely self-consane self-hosted system and the same goes through tailscale If you are looking for something like that, but that comes at the expense of not having an easy web interface I will recommend nebula if you're looking for not those two but one other solution But this video that's as much as we're going to cover on nebula I have a completely separate video on that and it to me is more of a dev ops tool because well the challenge in Learning curve of nebula is a little bit harder But it is being very actively developed and to give you a little bit of background if you haven't watched that video yet They came out of the tool that slack uses to literally manage your servers globally So it's also a well vetted system. All right now back on to the topic of those two Those are what we're going to be covering today But first if you like to learn more about me and my company head over to Lawrence systems comm like to hire short project There's a hires button right at the top if you want to hire for network services That's where you would do that by the way if you want to support this channel in other ways There's affiliate links down below to get your deals and discounts on products and services We talk about on this channel now. Let's dive into the topic. All right, let's start with a few bits of a rata to get out of the way and The first thing I want to talk about is the what abouts I know everyone has their favorite service or different tools that they've used and I will actually leave you that a link to this reddit post Which mentions I don't know exactly how many but quite a few different tools In here and discussions about their merits There's a few different github projects and links in here and actually this starts as a benchmark comparison of SSH 0 tier And tail scale so I'm kind of expanding a little bit on this and it's funny because it wasn't until after I started working That I even found this particular post But unless it's really some interesting things that were pointed out in here and of course lots of different tools available for it So that is another further discussion goes out of scope of this particular video And this is also the defined networking and the nebula I'll leave links to all these things down below and of course the video I did on nebula So if you're interested in diving into any of those absolutely great But let's go back to getting right on topic with how tail scale works This is the first thing you're gonna notice is that both tail scale and Zero tier have solid documentation This is a great thing when I can go to a company find out pricing and Documentation of how a company works or how their product works. I should say because that matters I want to understand the protocols so we can understand the security of how they're done now the data plane The data plane is wire guard for tail scale and for zero tier. They're using their own protocols, but Rolling your own crypto and to your own protocols can be dicey But they have gone through a code review and the code review has came out good This is a security assessment from March 23rd 2020 and they didn't actually roll their own crypto It's just the way they implemented the crypto was with their own zero-tier protocol. It's really clever It's open source. It's well documented and it has been vetted So both of these companies because wire guard has been vetted and so has the zero-tier in the way they implemented the protocols That comes back to the trustworthiness and this matters a lot Obviously if you want to have a secure network and those claims should hold up not just by the marketing team But by actual security and pentesting companies that go through it. That's just an important thing to look at now This is over at the tail scale documentation where you talk about the traditional VPN gateway where we have a Subnet over here with some servers on it And maybe some office and you have your clients externally and they would log in through the firewall VPN device Or maybe a dedicated VPN device to reach those devices that are on the other side And then you handle all of your authentication and brokering right here now the way it works with both tail scale and Zero tier the problem they're solving is The ability to have the clients be anywhere but have an extra network adapter Essentially that has them always connected as if they're on that subnet without having to go through and authenticate each time through a VPN kind of an always on type connection and Whether that connection is internal whether that laptop for example is external to the company or internal to the company It has access those resources also. Where are those resources? Let's say you have something hosted in a cloud or even adding another location You start tying each of these devices at each location With a series of access rules and you say all right These are the groupings of the systems that can access them and it doesn't matter where they are you add them to your tail scale or zero to a network and they're always are accessible at that address This is will be demoed a little bit more detailed here when we break it down But this is what they're essentially solving so the IP address of some type of service That you're running is always the same no matter even if the public IP changes whether you move that server to a new location Whether the endpoint does the endpoints that you have attached to the network also always maintain the same IP address this way the Networking is private so to speak and the global network the WAN can just kind of change dynamically They can move behind a nap behind two nats behind three nats later And it will figure out how to broker the connections So when you're connecting to the resources, they are always connected IP address Of course, this is also going to be substituted with DNS names and everything else But we're stick with the fundamentals and I get to out of scope on here Now lots of documentation as I said and how the protocols work and everything else and the data plane specifically is wire guard Now couldn't you just run wire guards yourself? I'll bring that up and one of the reasons why tail scale exists I have a detailed write-up on getting started with wire guard and a walk-through video to go with it And it's not that simple as you may have looked here. It's a lot of steps getting wire guard set up It is a great tool. It is a great fast VPN But to build your own wire guard server or go a step further have to manage a series of clients There's a challenge to it The goal of tail scale is to solve that problem of managing wire guard because wire guard itself is more of a protocol But not a complete solution when you look at a VPN I say it like that because you need user authentication and management and basically a way to control all of that That's not natively built into wire guard wire guard focused on the implementation of the protocol and that's where tail scale comes in and sits on top of it and The zero tier is solving that problem back in the same way. As I said, they're both just a Creating a series of nodes that all have an IP address on there And the way you do it over here in zero tier is the zero to a protocol Sets up a zero tier network adapter on each device and it uses the zero tier protocol to have all those devices in there And talking to each other now a couple differences where zero tier is a little different in tail scale With the way zero tiers configuration works You can set up your own private root servers And there are methodologies by which you can host it goes out of scope of this video And then they don't make it so to speak one click easy You don't just spin up your own but they do have your own routes and they call them moons So there are other ways you can do this to keep things private But generally speaking the overall and the way you're going to configure these is to buy their service and Go and have them do the hosting of the control plane of all of this Now that does not give them access to your data Neither zero tier nor Tail scale have the ability to decrypt the data that is going between the nodes They can see that there is traffic between the nodes But they don't have the ability to unravel that traffic and break the encryption That is used in there because the encryption used is solid and it's vetted as we stated earlier But that does not mean there is no threat surface at all The threat surface is a little bit different The systems that you connect together should not just assume IP authentication They should still have whatever username and password for whatever applications you've tied together For example, if you have a node that is a zero tier or tail scale server And you have some business application on there that requires a username and password to authenticate great Then the nodes that you've authorized on that network would be able to get to that interface in order to Try to do those authentications What if though someone took over tail scale or zero tier and someone gains access to their control plane? Well, this once I said before will not allow them to actually Decrypt the traffic but it would allow someone to add more nodes And this is where that secondary layer of security that hopefully you have configured Where there's suddenly another node on your network that was authorized by someone else Not you who may have control over the control plane They would join and add another node authenticate And now that node if it was an attacking node can start attacking whatever devices are within there That's why there is still a threat surface But it is not from a protocol leak as much as it's from a System where someone gets into one of those control planes also of no I want to make sure this is clear for the rest of the video There are no special firewall rules needed at all that also helps reduce threat surface To only being the control plane because as we do this demo I won't be changing any special firewall rules for any of these as we move the different nodes between different networks for the demo There's no firewall rules set up. There's no firewall change. We actually have a lockdown firewall that does not have upnp turned on So it'll just be relying on nat traversal Although these services can and look to things like upnp to try and gain better levels of access In this particular demo, no firewall rules are needed This is actually a really nice thing about both of these services is not having to open any ports Which does overall help the security and limits the control plane being the big threat surface of these products All right. Now, let's talk about the lab and how we have it set up We have three devices that we've attached here We have got this devian lab cloud which has a public IP address right here Which don't worry. It'll be destroyed by the time this video is uploaded. We have devian lab one devian lab two And then we have my firewall. These are located at my office here in my lab These both are on the same subnet with each other. So they're able to talk directly But they're also running zero tier and tail scale simultaneously and zero tier and tail scale as I said work similarly where they add an extra IP address that's essentially assigned to these devices in a static form So we always know that this device has this IP address no matter where it wanders around on the network So it's local IP or even the public IP can change and what firewalls behind can change And that's what these little dotted lines that are moving a representative of is those change notifications So the way the clients work both for zero tier and tail scale is they reach out Through the firewall cross the internet and talk to the zero tier and tail scale servers to say this is where I'm at How do I connect to all my other nodes? Now by doing this, they are going to broker the connections an example of tail scale Wire guard will allow for one side of the connection to be behind a nat So it understands that and this being on a public IP means tail scale can say This node here can go through the firewall through the internet and connect over here So it'll establish those connections and take care of those details In the case of both of these being behind nat it's going to use some different protocols and same with zero tier There's going to use different methodologies to get these devices talking to each other such as udv hole punching All right, those technical details by the way are very well laid out in both the documentation for each of these respective products let's Go over here now and here is just tmux splitting the screen. So we're logged into the devian cloud loud, which uh, By the way, I misspelled devian. So yes, you can point that out in the comments Uh, and devian lab 2 devian lab 2 being the one that's local. This is the one that well is in the cloud That's why we named it that way. Let's go ahead and throw some iperf on there And iperf is for speed testing. So we'll go ahead and do that iperf 3 s But we'll start with a ping test because you can't go wrong with pinging things or let's Go, uh, ping devian cloud and we'll say public ip And that's the public ip as listed out in there and we're seeing we're pinging about the mid 33s or so So reasonable ping times for being going across the internet here. What happens if we ping it on zero tier? About the same first couple pings are same over here in our 40s and came down 42 so it's not adding too much overhead plus It's traversing the internet. So you're going to get some variation in ping times What about tail scale same thing so the Added layers of running this encryption and running the protocols that they use does not add a dramatic amount of latency and overhead Based on this right here to going across the internet. So Not a big deal there. Now. Let's start the iperf test. So The client is going to be devian cloud lab and we'll say public first. Let's get a baseline here Raw connection across the internet. What kind of speeds are we going to see? The limitations and restrictions I have on a network means we're going to see about yeah 20 meg connection here. So we've seen about 20 megabits a second there What about if we went through tail scale? What do we see? About the same now just gets capable of Faster speeds in this but like I said, that's why I did the public ip first to show you what it was getting Which is not the full speed. It's just the restrictions on there That we have on this network and we do not have a full gigabit connection to the pipeline to the internet here Sorry for those of you that hoped we did But we'll test local as well and we'll go ahead and test with zero tier and once again Out the same with zero tier. So no problem. Now, obviously there are some scalability problems I don't know for certain but maybe wire guard might be a little bit faster depending on how they implemented it This is where I don't know for sure. But let's talk about the local test. So let's ping locally now And so you've seen that's where all these connections are going on this side of the pane now if we ping locally This will be the lab two and we'll ping the local ip address, which these are both in a virtual server Really the same stack the same hypervisor. So they're going to ping really fast It's not even really leaving outside of the virtualized network adapters, which also means if we did an iperf test to devian lab to The local ip address The connection speed that these can connect at is about 15 gigs a second right now For the hardware that we have this running on So that's more of a limitation of where the hypervisor and what it can connect at But either way we're getting 15 gigs a second on this particular machine. So I would say that's pretty good Then we're going to go and try it through tail scale. Now, this is where tail scale not a hundred percent I'm assuming is using some type of udp hole punching be able to do this and It's able to get a pretty reasonable speed here of You know 281 not bad now I know that is pretty fast But that is not as fast as we have a connection here overall what we can do But I believe that connection isn't necessarily looping out and we'll talk about that a second when we do a little bit deeper in the Demo, but this is what's interesting about the way zero tier does that If we do this with zero tier We're able to get about 400 to 500 So the first couple is a little slow But then it jumps up quite a bit and when you're comparing this to the 281 it would appear that zero tier can Navigate things a little bit faster But I also want to break this down one more level by adding more complexity to this particular setup So right now these are on the same network devian lab one and devian lab two So when we do these local connections, as a matter of fact, let's run that same connection again Once zero tier establishes the Essentially udp hole punch that is allowing this it is able to go faster even at first So now that we ran this again, we're at 482 versus 400 That's because these first couple were a little slow before figured out how to talk to them locally To my knowledge, I don't know and I didn't see anything in a tail scale config that would allow this That would allow them to talk any faster It seems to be pretty consistent at this speed even though they are both local But now we're going to put the devian lab one behind one more firewall And the goal of doing this is the cgnat question that people have asked or when you're double nat it So carrier grade nat is when your isp has a nat for you and then you have another nat that Protects your network or what your network is behind. So what about those scenarios and how do these things handle it? That's the next test we're going to do right and here's how we change the complexity up a little bit We took the devian lab one. We put it behind this firewall. So double nat it essentially This firewall has a wan ip address of 192 1683 dot 217 behind it is this devian lab one So it now has a local ip of 192 168 40 dot 139 now to Just to keep everything as complicated as possible here. We have this going to here going to here going to here so these 3.217 and 172 1669 dot 50 are also on separated sub debts So we've created some complex routing where someone's on a different segment of the network where the rules state that this May not even talk to this and this can talk to this can talk to this but can talk to this But then also has to peeking out to zero tier and all and tail scale So now that we've got those complicated rules in place How well does that work and that's where the next test comes in? So once again, this is on different subnet than this. So we're not going to get those same speeds but go ahead and Go here and we're going to ssh root at 192 1683 dot 217 And I brought that up because that's the firewall and we have a NAT rule that allows me to pass through this firewall to here My computer is actually on This network. That's why I can't just get to this directly. All right now that we understand that And let's go ahead and ping the devian lab to zero tier interface See able to talk to it And of course if we ping it locally We still can because the rules say That the three dot network can talk to the 172 but not vice versa, but we're able to get to this And we see, you know A little bit different when they're not on the same subnet even though they're still in the same machines Passing through a firewall, but then we can do the iper test lab two and uh zero tier address And we were getting that 500 megs And this is where it's kind of impressive Even though we've looped through there and done this kind of confusing network setup We're still able to establish a pretty fast 400 meg rate between these and I It's just kind of the magic of how they make all this work. It's not magic I shouldn't say that it is well documented, but it is Really good network engineering to allow these devices to be able to talk to each other despite being on separate networks Now of course, we got to do the same test with tail scale now And tail scales about the same whether they were on the same subnet or now double nated and on different subnets Within the same network in my building here And they're still able to this and probably you're wondering well, can it still talk to the other device over here? Yeah, it's no problem. Even though this is double-natted it can still Go ahead and just ping the devian cloud lab zero tier And we're able to ping that device so it took care of double natting And what about if we were to ping devian lab cloud tail scale Same thing no problems can ping so getting the same one So even though it's double-natted both tail scale and zero tier are able to traverse these so overall I would say It handles these complex things like double nat or wherever these devices are and especially for people Who may be interested in dealing with double nat or cgnat situations both quite well Now while I do really like the way that zero tier when things are on the same subnet was able to utilize essentially like udb Hole punching and hairpinning in order to get this working in a way where you can get really fast transfers I don't know that that makes zero tier some type of clear winner It is faster in that particular context But overall as you've seen and the more likely use case for these devices being deployed and these services being deployed Is going to be where you have a series of clients that are scattered around in different locations Maybe work from home users and a series of servers Maybe some in different cloud server stacks or some that are on prem And you want to have a single common easy to manage ip plane where you access all of these Both tail scale and zero tier solve those problems Now I won't talk about pricing to try to make any determination of which one you should choose because pricing kind of dates the video They both have public facing prices on their website to tell you how much it costs to use And of course, they also have contact us if you are a Very very large scale user and want some type of custom implement Application that you want to use this for so there is a basic tier pricing fillers You want to use a free tier for anyone at home just wants to sign up And do the same things I did this demo cost me nothing This demo was not endorsed or even aware I did tweet it so I can't say they weren't aware that I'm testing their product They were aware but had no influence neither company did on the video or any feedback or commentary that fed into this I've done several videos as I stated on zero tier. It's the first one. I'm done on tail scale I may do a dedicated video to tail scale if you leave some comments below and think that's necessary But overall I will say both services are really easy to use really easy to deploy And pretty simple and both are good. It just comes down to what you would like to use So either one works So this is kind of my answer video to people who have asked me a lot about these two different products for a comparison And of course that third option I mentioned at the beginning nebula I think nebula is still a good one nebula, of course is not on the easy side and for those you just go You know what? I'm just want to run raw wire guard. Hey, I'll leave a link to that down below All right. Thank you guys for watching this and uh, see you in the forums And thank you for making it to the end of this video if you enjoyed this content Please give it a thumbs up if you like to see more content from this channel Hit the subscribe button and the bell icon to hire a sure project head over to laurance systems dot com And click on the highest button right at the top to help this channel out in other ways There's a join button here for youtube and a patreon page where your support is greatly appreciated For deals discounts and offers check out our affiliate links in the descriptions of all of our videos Including a link to our shirt store where we have a wide variety of shirts and new designs come out Well, randomly so check back frequently And finally our forums forums dot laurance systems dot com is where you can have a more in-depth discussion about this video And other tech topics covered on this channel Thank you again, and we look forward to hearing from you in the meantime check out some of our other videos