 Hello. My name is Marsha Michel. I want to talk today about some work we've done with some of my colleague, Jesse Hamilton and Annie Weng, where we were looking at a solution for hosts that were located at different locations to be able to talk to one another, but with some constraints on the network. The reason we did it like that is that we were looking for a solution where we needed direct connection between hosts that were at different locations on firewall, not always connected to the same network stack. So we have hosting the Cloud, hosting the data center, some hosts we're working from home, and we even had some edge device that needed to be able to talk to one another. The problem is obviously if you're behind a firewall, you have to be able to talk on the same layer. Usually you go in through a VPN, and then you're connecting to the host. Well, unfortunately here, everybody is behind different layers, so that's not as functional. Overnight networks are the way to go because they would create a network layer on top of the current network that exists and make everybody accessible on IPs that are predetermined. There are multiple solutions out there for overlay networks. We've looked at a few. We've looked at particular as Slack, Sneabula at internet and at tail scale. They are all mature enough to be usable. Unfortunately, firewalls do their job very well, in the sense that if you're trying to connect to a host that is behind a firewall, very often you are going to get blocked even if you're the one reaching out to anything that doesn't have a central relay. So that's where actually, tail scale was the most functional of the lot because it had technology that made it possible for it to relay information through a middle host, right? It also had a different technology to go through firewalls in order for packets to be able to go out. If you're interested, I highly recommend you look at the different block posts that are out there. I didn't put any here, but there are different block posts that talk in particular about how tail scale does connectivity and there are initiatives for the other network to work out those solutions. Unfortunately, in the end, for the program we were trying to work on, tail scale adds some components that were closed source and that made it a little more difficult for us to get approval to use them. So we had to look at another solution. That other solution turns out to be open VPN. The classic open VPN, we use the specific implementation of open VPN which is available publicly on the Docker Hub and on GitHub and we change a tiny bit how open VPN is configured so that we are able to use the client to client communication method. Now, client to client does one thing in particular, it makes it possible for hosts that connect to a VPN to see one another and to talk to one another. The classic VPN configuration is I'm going for the VPN and I can talk to the hosts behind the VPN but I cannot talk to the hosts that are connected on the same VPN as I. Well, when you do client to client that makes it possible for you to actually do that extra layer. So what you see here is basically steps needed to set up the open VPN, you need a volume for Docker. Once you have a volume, you generate a default configuration. Here it's going to be called open VPN cloud. It's going to be over UDP. We are generating a default configuration file and once we have the different configuration file we are modifying inside the open VPN.com there's two lines that you see topology subnet and client to clients. Then that implementation of open VPN makes it possible for you to do authentication using CA so it's basically the classic OVPN exchange. So you set up a CA passphrase, certificate authorities of course. You start the open VPN server and then you're able to go to the step where you're able to create static IPs for each host. And that's actually very important for some of the projects that we wanted this technology for. We needed to be able to access hosts and not have, we didn't have a discovery service, right? So we needed to figure out, we needed for every host to always have the same IP. So we did that by configuring it so that a given host would always get the same IP. So here is an example of how to get your CA file as the first two line. You do the same thing on all the hosts you want to connect. Here we have hosts named server one, server two, ohm one and edge one, user one, sorry, and edge one. And they're gonna be on 10, 20, 30, and 40, right? 192.168.255.10.20.30.40. Then what you do is you basically connect using an open VPN free client. I think it's, you can't see, but there we go. Yeah, there was the footnote that was cut off. You need open VPN free in order to get to work. So what I'm gonna do is I'm gonna quickly share something else. I'm gonna quickly share, I have a Linux VM that I set up just to show you how that works. So I'm gonna share a window with my Linux VM, there we go. Okay, so I'm in the Linux VM. I am going to try to ping one of the hosts. As you can see, it's not answering as we're expecting. There is no way for it to reach the private IP. So I'm gonna start my, I generated earlier today an open VPN file for myself. And I'm gonna do a one-time connection and I'm gonna do a one-time connection. And I'm gonna do a one-time connection and I'm connected. So now if I ping, I can actually ping the host. That host is a GPU box at the data center and it's running and it's connected to the server. So now what I can do is I installed on it a Jupyter, CUDA, DNN, TensorFlow open-cv container. So I'm gonna SSH into it, there we go. And I'm gonna run my container. I'm gonna grab the token. I'm gonna go back here. That was because I was disconnected. I'm gonna reload, give it the token. And I have a freshly running Jupyter notebook with a Python kernel and if I run bash, I can access the commands and show I have four GPUs available to me in that Jupyter notebook, which is running in the data center, remotely accessible and through the firewall that we have at the data center. Now, of course, you need to be able to SSH into the host for you to be able to do that. And you can see, I guess I have dot two. My configuration is dot two. I'm gonna share back my slides. Sorry, here with me, which is that one. For each client, just make sure that you have the right open VPL remote setup. That's kind of important. If, for example, the server is itself, you don't wanna expose the default port for open VPN, right? You need to modify the remote on the OVPN file. But beyond that, what we've just achieved is basically this. We've made it possible for all the hosts that are at different locations. So we have tested it with OpenStack instances we have tested with OpenStack instances behind the VPN. We've tested it with Roost as a data center. We've tested it on AWS. We've tested it on Edge device. And I'm working from home right now and I'm basically just access the data center without being connected to the official data center VPN. If I can talk to every one of those hosts currently. And with that, I wanna thank you for your time and I'm gonna welcome questions. And the information in how to reproduce that will be made available also we expect in the blog post in the near future.