 Tom here from Lawrence Systems and we're gonna talk about Nebula, the open source global overlay network from Slack. And before you're probably wondering, well, do we need another network VPN style solution? Well, maybe. And I bring this up because I started digging into this. I can't remember if it was a comment from a YouTube video or a forum post. But I think more than one person had mentioned, hey, Tom, can you look at Nebula? And that was a little while ago. And I say, hey, it looks pretty cool. Well, then I didn't realize just how cool it was. And I like right here on the Slack's announcement page. And yes, Slack is essentially where this was developed, tested and vetted. But then it's all released as open sources, 100% self hosted type project that you download the code and run yourself, which is an important factor in this. So what is the easiest way to securely connect tens of thousands of computers hosted at multiple cloud surf riders and he doesn't locations around the globe. And they had to invent this tool to solve this problem. And you're probably thinking, well, doesn't VPN solve this? And well, not exactly. And I really encourage you to watch the video over here on Define Networking. They have embedded which Define Networking is the new company that kind of came out of this. And my affiliation is always like to get this clear upfront is I found this project interesting. It's open source. I had a great conversation with the CEO, Ryan Heber. I actually went back and forth on the fundamentals and where the project's going, which was a great conversation. And you know, I am as enthusiastic as he is about networks. And we were having this conversation even on a Sunday afternoon on Twitter. Anyways, not to get off topic. They have plenty of deep dives where they've talked about some of the impetus for inventing it. I wanted to cover it here on my channel because I did some testing with it and find it really to be a fascinating solution, especially because security in this is October 2019. He talks about the essentially zero trust where Slack did not want some third parties to be part of the control plane for the security of their network. And that seems pretty like a good idea in October 2019. And with the events that took place in December of talking about supply line attack and things like that that occurred in December of 2020, it seems like a wonderful idea to completely self host a secure networking solution that doesn't have or a reliance, I should say, on any type of third party, which is what makes this a really interesting tool. And this is why I'm bringing it up. Also, I want to bring a few things up about some problems with it. And that's this right here. I just want to get this out of the way before people take the time to watch this video and go, Hey, it doesn't work with my NAT setup. Yes, if you have two devices behind NATs, they are going to cause problems because at this time in January 2020, but it is on a roadmap for a future release to have this solved, they have a lot of NAT problems. This was really designed for a lot of public cloud infrastructure that they wanted to have private access to with groups and security. So it works wonderful. And that's the solution that we're going to show in action where you can have a node behind a NAT talking to a public cloud server. But if both nodes are behind a NAT, each nebula node behind separate NAT, then you may run into a lot of problems. And that's what this long discussion breaks down. But once they support relaying, and this is, you know, very similar to zero tier, which I've covered on this channel before, which does support relaying, that's how zero tier solves this problem. If two firewalls have trouble talking to each other through UDP hole punching, it can go, Okay, I'll just relay the traffic. Now, in zero tier, if you had watched that video, I'll leave a link to it. They refer to them as your moons and planets to talk about how it works. Nebula uses a slightly different nomenclature, but very similar functionality of, they call them lighthouses. Lighthouses are the directors of traffic, if you will, to understand where all the nodes were and basically create an index of where all the connections are coming from. And we'll break down how to set that up shortly here. Before we dive into these details, let's first, if you'd like to learn more about me and my company, head over to laurancesystems.com. If you'd like to hire a short project, there's a hires button right at the top. If you'd like to help keep this channel sponsor-free and thank you to everyone who already has, there is a join button here for YouTube and a Patreon page. Your support is greatly appreciated. If you're looking for deals or discounts on products and services we offer on this channel, check out the affiliate links down below. They're in the description of all of our videos, including a link to our shirt store. We have a wide variety of shirts that we sell and new designs come out. Well, randomly, so check back frequently. And finally, our forums, forums.laurancesystems.com, is where you can have a more in-depth discussion about this video and other tech topics you've seen on this channel. Now, back to our content. Now, as I said, it's all open source right here on GitHub. You can download the code, compile it yourself. They got a pretty straightforward how to set that up, or you can check out the release pages for the downloads. And they've got quite a few different downloads for a lot of different variety of things. Of note, besides having, you know, ARM and MIPS, which means you can get it working on things like Raspberry Pis and lots of other devices, they do have the Nebula Windows. But the Nebula Windows does have a reliance on the TUN TAP driver from OpenVPN. So you do have to load that in addition to this to get it working on Windows. These are all things that are in the future will be solved. But this is also a very heavy DevOps tool, more so than a end user tool, or something that's easy to set up. It's not going to be the solution you deploy to solve a bunch of end user problems for connectivity, not really what this target is. This is more targeted as building cloud connectivity that was also cloud agnostic. So if you want to move servers around, you assign these Nebula adapter IP addresses to them. And no matter where the servers are, the development tools can just always point towards those IP addresses for connectivity and the lighthouse will redirect and fix things kind of as they go wrong. As you go, okay, I moved all these servers over here. Now I got to go redeploy these IP addresses. Actually, you don't. You can just change public cloud IP addresses all you want. The Nebula IP address is the same for connectivity on the back end. So it's really kind of a clever solution for that. You can also build in groups and a lot of other features that some of this will go out of spec for this. But like I said, watch the video over and find networking where they dive deeper into it. I just want to get the news out there, so to speak, and other people interested in this particular project. And by the way, they do need a lot of help with documentation. So if you want to help an open source project, reach out to them over at Define Networking, they are hiring, and I have no financial vested interest in this at all. It's just one of those, hey, it's a cool project and I want to help them out. Back over here to how to get started with it, though. What is Nebula? We covered that getting started quickly. Downloading up into binaries, start building the certifications or certs, not certifications. The Nebula system works on your CA that you build in the beginning of it. This is actually a really clever thing because when you build everything off of CA, we first create our signing authority, which we'll call, they called my organization, I just called it Nebula demo, we'll cover that in a second. And you create these two files named key CA.key and CA to occur in the current directory. The CA.key is the most sensitive file you create because it's used to sign certificates for individual nodes. So essentially you establish where you're going to sign all these nodes. You can sign them privately on your computer and keep this locked down and then distribute all the certificates through probably some level of automation, however you want to build that out, as you see fit. By the way, they do have a quick start vagrant right here file. So you can use some of the automation. It's a framework. It's not complete, but it kind of gets you the idea of how to get it set up if you want to build this with automation. They did all this with scale and automation by using this type of file to do all the creation and distribution of keys, et cetera, customize it. I like that they give you kind of an idea and a template to get things going, including how to install it as a service, which is going to go out of scope of this. But I want to bring it up that if you're going to install this as a service, they absolutely have this you know, file and everything here to do it, but it's not going to do it natively just by grabbing the binary. You can just run the binary for the command line. But yeah, those things go out of scope of this, but I do have installs of service and a couple of the servers. They give you an example configuration, which will walk through how the lighthouse is configured and how the nodes are configured. They also show you how to sign each individual node. Now the default signing for both the original CAU create and each individual node is one year, but that can be customized through the command line to say I want it to go longer. Now the advantage of doing it this way is when you define things like your groups and that is for security reasons. So when we get to the firewall rules, it's rather clever because you can say, all right, I assign this group called servers or this group called users or well, let's say we had a group called developers and we want to give all the developers access to these particular ports on all these hosts. Well, you just define that group and then you throw the group in there and firewall rules and you define the ports in that rule and now away it goes. The other advantage of doing it when you do certificate based here is that one year. Now, like I said, you can extend it if you wanted to, but this is something that happens in the corporate world all the time, especially when you build things at scale, you sometimes may forget about something. Now, if the key expires on something, someone's going to make some noise and complain about it, but this is a way to kind of auto prune by letting them expire and go, well, they're not accessing anything anymore and let's practice proper principles of least privilege and let those nodes expire. If no one complains, fine, they're no longer part of the nebula. I kind of like that from a security standpoint of maybe even creating expirations that last even faster. Now, this is not standard public domain type signing for those wondering. This is all your own signing server. It's rather simple to manage because you can just reissue certificates each time for the nodes and renew them on a signing server and through whatever automation DevOps tools you're using, push those out to keep everything up to date. And like I said, anything that goes into expiration is just going to fall off. All right, now let's actually look for how it works and show you some of this up. Now, complete disclosure here, I'm going to expose public IPs on both my digital ocean lighthouses and any keys you see will can be, well, they're going to be destroyed before this is over. And yes, this is my home computer that I'm going to SSH into that will expose my public IP, which I'm fine with because it does dynamically change. This is what it is right now. I'll also log in through my IPs here at my office, but that doesn't really matter because I'm not going to be showing a public IP for that. But just to make it easier for this demonstration, yes, I will. I just want to address that because it comes up in the comments constantly. All right, here's that first late house. Now, the first thing about lighthouses is how they work. The lighthouse is either part of the network because you want it to be one of the servers. So you can have one of your DevOps servers be a lighthouse and also something you want to access, or the lighthouse can be just a node that directs traffic in terms of where things are located, but doesn't actually pass traffic. So this computer is going to ask the lighthouse, where's this? So let's we want to get to the server over here. And so it asked that question. That question is then answered by then creating a connection between these servers. So the traffic never routes through the lighthouse. It's not the way to sign. Now, when a future releases, they, when they add relaying into this, they'll be designated relays, and probably the lighthouse could be a relay where when we have that double NAT problem, because this NAT and this NAT means these two can't directly talk to each other easily because the lighthouse, when it when asked, where's the Tom desktop computer? It answers, but because it's behind NAT, there's that connectivity problem that I alluded to in the very beginning. But without the traffic passing through the lighthouse, it's really just a lighthouse, a beacon, if you will, for where all of the things are. So let's go ahead and look at the lighthouse over here. And we're in the ocean lighthouse. Let's look at the config file. So there's the CA dot cert, the keys, the part you keep private, but each server, each nebula node, host needs a CA dot cert file, the original search. So they understand to talk to that key. Here's the ocean lighthouse dot CRT and ocean lighthouse dot key that we created. Then comes to the static host map, those static host map 10, 001, and then the public IP address. You list each of your lighthouses here, first you list their internal nebula IP, and actually just really quick, stupidly simple little script I did that creates the CA name, nebula demo, it creates the sign name, ocean lighthouse, then a home. So this is 10, 001, 10, 002, my desktop is going to be 10, 009, and London is 10, 0044. Each one of these is the signing name for it. And these are the IP addresses. I didn't bother with the groups. Didn't really need them for this particular demo. Then it drops these, grabs these files, the dot key, dot CRT and CA dot CRT and drops them all in each of those locations. It's installed as a service on the ocean lighthouse. And it's installed as a service on the London server we'll log into. It is not installed as a service on my nebula home or my local computer here. So we're just going to run it from the command line on those. But that's where we got that IP address you define out your network. Now I did a slash 24, obviously you can do even larger scale if you want to. Then we go to the lighthouse config and lighthouse equals true. That means this one's a lighthouse hosts. Now in all the other hosts, you'll list out the lighthouse IP addresses in the lighthouse. It doesn't need this in here. It can be empty. Now the default port in the example config file they give you is 4242. I left it. It doesn't really matter. They have a couple other fine tuning options that you can read through here. This is a little bit interesting because you can adjust fine tuning such as the MTU. Watch that video that he did because he talks about the different challenges of defining different MTUs and different cloud services. And sometimes while you want to increase or decrease them for efficiency and fine tuning, but they give you all those options. There is some built in logging in here goes out of scope, but it's a feature it has. And now we come down to the firewall rules. Now outbound and inbound, everything's defined on each nebula node and the host can be any or as it shows down here that's commented out the groups. So you can define a port of 443 protocol TCP group laptop or home. I don't have any groups to find, but that's how you'd set that. And then here we can set the ports. Now the nebula, because it's not actually traversing any traffic, it's not making a list on 4242 public IP side, but internally I would at least recommend allowing any, any for ICMP. I've actually got it wide open here so anything can do it because I'll change it and show you how the demo works. But the same thing with the outbound traffic. This is a good way to narrow everything down. Now anything that's going to be talking on the nebula nodes is going to be able to talk on these nodes because you authenticated them and put them on the nebula nodes, which means you can watch the firewall logs very, very closely because you're not going to have a lot of noise in them. If there's something getting a disallow, it's trying to get to somewhere that you blocked. You have either a nefarious actor on your network or you have a misconfigured service reaching out to something it wasn't supposed to, either way that makes firewall logs very, very actionable. This is that security that I mentioned at the beginning where they wanted to be able to monitor things and not just from standing up a bunch of VPNs. This is a nice, quiet way to do it. Everything on the nebula node has been signed with the certificate and everything should be there. And if something happens, you kind of get to note right away because it's trying to reach out to something it shouldn't reach out to. Anyways, we'll leave these at any of you to get them started. Now we'll go ahead and close this. Now the service is already running. So if we go and we can see it's active running, we can restart the service if we needed to, but let's actually go right here and show right here is my security union where we're going to show the public IP address and show the connections. And I want to show that this computer, this is my home computer, has no active connections to this London host and no active connections to the lighthouse host. So let's go ahead and change that. We'll actually watch this one here and we're going to do a nebula, show the config. I'll just show real quick here. Here's the ca cert and config.yaml. Well, I guess we've probably looked at the config file real quick. So let's go ahead and look at it. There's the ca home ltsca.cert, the home key home.crt, the static host map, which matches the same as the lighthouse. This is where things start changing. Lighthouse false because well, this isn't a lighthouse. It just needs to talk to them. And this is the lighthouse that'll talk to and for redundancy, you can have more lighthouses. Of course, we just have one, which is enough for this demo. This is where the hosts are listed. The hosts you listen here are the lighthouse nodes. And this was empty on the lighthouse one. Other than that, all the config is the same. Actually, one thing I probably didn't change here is it is on default port 4242. And on other ones, you can define it to go on 4242 or set it to zero, which means pick a random port each time it starts. It doesn't really matter because it's reaching out to the lighthouse. But we can delete that, insert zero as an option. Then we're going to come down here to the firewall rules. So we have this all set to wide open any any, we'll change them in the future here. But allow any between the hosts and allow outbound traffic any so we can have anything incoming that's on the nebula nodes that are authenticated anything outbound. And now we're just going to go ahead and create that adapter. So let's go here, write that and I have config currently no nebula adapter attached to this device. So we go here and sudo actually will make it right here. So you can see what happens in the background. So when we kick this off, my password in, we've now established a connection port 4242 because that's what we told it to talk to a nebula. Here's the random outbound ports of 617 and of 54454, whatever it's talking to a nebula node. So let's go ahead and bring up another connection. I'll split the screen numbers using Tmux because people will ask. So let's first ping the lighthouse node. By the way, if you ping 10.1 it adds the zeros in between automatically, but I'm talking to the lighthouse node. So there's some traffic. We can see the traffic going up as I talked to that node. What about the London node? That one has actually a IP address of 10.0.44. So let's go ahead and ping 10.44. And it reached out, asked the lighthouse, where is the London node? And it responded very quickly and is now pinging away on that particular node. So if we go over here, we don't see anything other than a connection that's going out from the public IP address and heading out over to the London node. Now, all that it did was talk to does not relay through the lighthouse, that's the house where it is, and then the connections are established. So pretty straightforward for being able to do that. And we have, and let's go over here to the London node. And let's look at the config on that. So if we, vim slash etsy, nebula, config, and we allow protocol ICMP, any, any. And Tom desktop though can do any, any inbound from nebula. I do recommend you at least have this protocol, any ICMP, any, and the reason for that is at least be able to ping. So you, before you're wondering why a service may not be working or you can't get to a port on it, always try ping. So we can allow ICMP, but nothing else. And what that means is if we are on here and we SSH to root at dead air won't let me talk to it. So let's go ahead and adjust that. So we want instead, I think this one's going to be called home. So if we change it from Tom desktop to home, and let's say a port specifically 22, and it's a TCP. So now we've created this rule that says we will allow home to talk TCP only on port 22 to this particular server. So go ahead and write that or restart the nebula service permission and I bubble key, but it works now. I don't have a key for it. So it says, well, you can talk to it, but you don't have the right key for it. But now it does respond in a civil course will respond to ping requests. And by the way, that's one thing about this is how fast does nebula restart. So we can really show and let's actually do it this way. So in the bottom window here, I'm on the nebula home on the top, I'm SSH into that London nebula. So let's sort of ping over here. So you can see him at the same time. Whoops. 10,00044 restart the nebula service. Yes, the connections are rebuilt so fast, we didn't even I don't think we lost a packet on there. 7% packet loss. So there was a there was a packet dropped inside of there nebulas protocols do establish very, very fast. So restarting service or even when a service moves, it'll update the lighthouses of its new location. And it can reestablish the connections very, very fast. This is definitely an advantage it has and should be able to look over here. We'll see both the new connections were established and the old ones because it's got to expire out those old ones, but they still exist. And I think we can probably demonstrate this again. Restart the service. And you keep seeing the stale connections because they're still existed. They're just getting dropped. So each time I do this, but it only drops just for a moment. So things that are happening in the background and connectivity on there if you have to modify a service. Now there's another way to do this that he examines in a video where you don't have to restart the service. It can reload. I believe it's the firewall rules. Some of the rules can be reloaded so to speak on the fly and change without actually shutting down the service without being disruptive. I don't remember how he did that. This is something you need is more documentation because there's some other methods you can do, but either way you can see how fast it can reestablish a connection on there. Now let's go ahead and add one more to the mix here. So we'll exit this and go over to Tom's computer. Now let's look at the node here I have and we have a config YAML file for me. And we have the Tom desktop. So here's the cert key, et cetera, et cetera. Once again, I'm not a lighthouse. I'm defining the one IP address. I defined the one host. I'm not worried about inbound traffic to me or maybe I should be and probably have these set to none so I don't allow any inbound traffic. I'm allowed to inbound here, outbound, et cetera. But I'm not too worried about any of that right now. This is my host defined and go ahead and close that. So let's go ahead and kick it off. All right, it's connected. It reached right out to the lighthouse and let's look at the adapters on my computer real quick. We now have the nebula one adapter assigned with this IP address. I don't think I showed that on the other computer, but you get the idea. It instantly assigned it. And if we destroy this, for example, I'm just going to close the session. There's no nebula adapter on here. So it destroyed it right away. This is one of the reasons you have to run a sudo to add that extra adapter. So away we go. We've done it and there's the nebula adapter 10 009. So let's go ahead and ping. I can talk to the lighthouse. I can talk to London, but 44 not four. So I'm talking to that one. And we'll SSH to London fail. So let's go over here to our London server. And we'll add port 22 TCP Tom desktop. So we want to allow me to SSH into this nebula. Now this is where that groups would have come in handy, because then I would have just built an SSH group and you just say group instead, like it does down here where we'd say, all right, add these groups to there. So it's not our option, but you know, simple enough for this demo, do that, restart the nebula service. I do have the keys. So I'm able to log right in. And here's my session. Now let's talk about another situation. So we have SSH here and I'm logged in and, you know, able to have a connection here. What if we restart the nebula service? Did it break my connection? No, it broke it for just a moment. And you can see it reestablished it right here because it said, Hey, that stopped. And now as a matter of fact, we can do it from here system. Whoops, restart nebula service, instantly reestablished a connection to the point where I was able to restart it and it didn't break even my SSH connection. So I'm still logged in because I actually use the SSH session to do that. So pretty straightforward as far as how you rebuild these tunnels and restart them and adjust firewall rules. And like I said, this is something you build as part of the automation. But this is nebula in a nutshell of how it works. And it's a project I think is really cool. I'm excited about its future development. It was just why I wanted to show some of the demonstration here and get more people interested in playing with it. And I know a lot of people who build out some solutions. I've actually talked to a couple of my DevOps friends who found this kind of interesting and said, that is kind of interesting. We do handle the security on the back end of all the servers, especially because well, one of them has a ton of bare metal servers running in a lot of data centers. And they're like, Hmm, this could be an interesting way to handle control planes across a wide variety from I was even thinking of ways to integrate it with things like XCPNG for when people have some public hosting on there. And they want like an easy back end way to have one web panel that automatically can talk to everything else. And this might be an interesting way to do that. But there's a lack of documentation. This is one of the reasons I did the video was to raise some awareness of it, find some other people who want to contribute to an open source project and reach out to define networking. I said, leave links to all the videos dive deep into the videos that Ryan did. They're great. They're definitely give you a lot of insight into the thought process of building it's kind of a long video, but they also talk about all the challenges they face and trying to do this as a VPN. I think it's a solid, secure solution. It's completely self hosted. So you know, you're building your own trust, it's open source. So we can look at all the code and modify it. And there's plenty of active parties participation in their GitHub of people discussing bugs and different scenarios and different use cases. So if you're interested in me project, I'll leave all the links so you can join in. All right. Thanks. And thank you for making it to the end of the video. If you like this video, please give it a thumbs up. If you'd like to see more content from the channel, hit the subscribe button and hit the bell icon. If you like YouTube to notify you when new videos come out. If you'd like to hire us, head over to LawrenceSystems.com, fill out our contact page and let us know what we can help you with and what projects you'd like us to work together on. If you want to carry on the discussion, head over to forums.LauranceSystems.com, where we can carry on the discussion about this video, other videos, or other tech topics in general. Even suggestions for new videos, they're accepted right there on our forums, which are free. Also, if you'd like to help the channel in other ways, head over to our affiliate page. We have a lot of great tech offers for you. And once again, thanks for watching and see you next time.