 My name is Spencer Grumb. I work for IBM. This is a talk called Secure Peer Networking with Tink. The slides are on my GitHub, and they'll be part of the scale slide spammage that will come out, I'm sure. Can the people who are on the left side of the room do me a favor and just kind of shift to the right? Because when someone opens that door, they're just going to try to sit down right there. Just makes it easier for everybody. So I work at IBM. I work primarily on OpenStack. I also work with the public community. We renamed GitHub Organization to Vox Pupuli. And that's where a lot of my open source contributions come from. Specifically with OpenStack, I work on the infrastructure project, which means we run the CI system for all of the OpenStack. So sometimes 20,000 jobs a day goes through our CI system, as well as code review and stuff. I am from Portland. So when I travel, I like to show people what Portland looks like, Portland, Oregon. I know a lot of people think rain and other problems, but we're actually a very beautiful city with a pretty epic mountain in the background and more green trees than Californians can even conceive of. This is a talk primarily on Tink. It says that consoles in it in the slides, which is true. Like consoles in it. But if you are here primarily for console enlightenment, this probably will not bring you to glory, unfortunately. So the two pieces of software we're talking about here are Tink and console. And these two pieces of software kind of follow an inverse proportion between usefulness and website awesomeness. So if you look at the console website, there are CSS3 transitions. There are JavaScript starbursts flying around. And it's actually kind of hard to use. If you look at the R-Sync web page, it's got H1s and H2s and divs. And R-Sync works great. Similarly, the Tink website is made probably in 2004. It's got a PayPal donate page, and that's the only image. It's that kind of low key. The logo itself is also there, but I'm not even really sure what the copyright status that image is. I'm not going to comment on that. So this talk is mostly about Tink. It's not that much about console. So Tink ultimately is a virtual private network system. This is the obligatory noisy picture of a VPN. We have some classics like the firewall. Yeah, we have the wall that is on fire. We have the mainframe servers, the strange blue, I guess, film reel highway thing or something. But Tink isn't really like your open VPN. It isn't like your Ipsac VPN. It isn't a point to point kind of thing. Where traditional VPN, you have a fleet of laptops or something that all connect to one or maybe two in a high availability pair of VPN servers and then get access to a network behind that. Tink is a mesh. It's a distributed mesh with routing. So a node will come up and connect to one or any number of nodes and then have access to the entire network. And we'll look at some kind of graphs of what that looks like in the past. But in our setup, the setup we have, every node talks to every other node mostly. So some fast facts about Tink, just some quick information. It's from the era of this cell phone. They started working on it in 1998 in C. It's still going strong. It's got two developers, who've written most of the code with a lot of driveby patches from various humans. I am not one of those developers. I'm just an enthusiast. So it's fun to draft some, you know, the whole talk really is about this thing we dusted off. I've been living in Debian repos for a long time and started getting some use out of it. And I figured I'd talk to scale about it. It behaves just like a eunuch's demon. You don't need a kernel module or anything. You need the Tuntap driver. But you don't need anything special beyond that. Their development is done kind of weird. So it's kind of the classic old school pre-GitHub development workflow that's been modified by GitHub. So if anybody works with Apache, you can still mail a patch to most Apache projects and they'll apply it. But you can also open a pull request and they'll merge it. And there isn't really good standardization around that yet. But there is a dev mailing list. That's very good. There's a free note channel that's really good. And Gus, who's the guy who writes most of the software, is very responsive on both forums. We looked at using Tink internally when I was at a last job. And what we found is that for bulk file transfer we lost about 15% of performance when we're moving our stuff over Tink. So that's there. It's also worth noting that the SSL has not been independently verified by other people. So I wouldn't bet the farm on it without spending a fair amount of time looking at it. But it's old, right? It's old. It's got port 655 and Etsy services. So that's pretty cool. All right, who remembers this, huh? The network is the computer. And I'm not even really sure what this is. It's like a mouse pad or something with a handle. And this guy's really excited about his business flying into the computer age. But this is kind of an idea. So when I was in college, we had an old network that had been written in this era and never changed. And it was a very special network and not really modern. But it had some really awesome things about it, right? Like all the machines, all the printers, everything had a public V4 address and everything. You know, there wasn't really a firewall. Everything was just kind of connected. And this was kind of weird, but it also meant that the distinction between a server and a client was a little more muddled, a little more vague. And so you could just write a little daemon and start running it on some Linux lab machine and other people could start using it. And the next thing you know, it's like a service. People depend on it. Any node could hit any other node. It's fundamentally flat. There wasn't a lot of firewall rules. There wasn't a lot of NAT rules that prevented these kinds of connections. Now we've really gotten away from that. Like everything now is HTBS. It's the only port. We don't even need the others. And that's kind of lame, because if you want to share files or have a conversation with someone, you've got to do it through some abstraction layer built on top of HTTP, which is really annoying. So if you want to run your own services, there's something like own cloud. But if you don't want to do that and you just want like a file transfer protocol, it's basically out the window. You have things like Dropbox and Google Drive, but then you're giving all your data to them. And it gets kind of not the greatest thing. So my friends and I were like, what can we do? We found this tink thing. We started dusting it off and we said, we're going to build ourselves a mesh network and see where it goes. And so what that meant is that most of us, five or six of us, installed the tink software and started the Daemon and configured it so that we would start connecting to each other, which is probably different than 99% of tink deployments, because 99% of tink deployments is somebody who wants to build a VPN for their servers that they control or their network that they control or their laptops that they control. So it's what administrative domain. So we have a different problem because we have six or seven administrative domains and people, this is all free time hobby project type stuff. So sometimes it's off and nobody cares so we'll deal with it later, right? So, yeah, there's a blurry line between the workstation and the server. We run it on laptops. We run it on home routers. We run it on rack space gear that's just hanging out in the rack in a couple of random places. So let's look at what the network looks like. This is a picture, an actual valid picture of tink. So tink is kind of a cool name and it will dump the network state as a dot file every minute to a special place in var, which is kind of cool. And then you render it with idea or something like this. And so we can see, I've anonymized all the names to be just numbers, but we can talk about the graphs still. So the easiest thing to identify is that some of the nodes are on the bus and some of the nodes are not on the bus. They're just not connected. And so what that means is that those nodes which were tink daemons were connected and my tink daemon has not forgotten about them. They're just not connected right now. And so if they come back, we know who they are. They're not gonna come back with a unique name or a unique number. You also see that there are maybe two classes of nodes inside the graph, right? There's nodes like one and 10 that only have one connection in. And then there's other nodes that have multiple connections. And so the way tink works is that you only need to connect to another node that is part of the mesh and then you have full access to everything else. If you have multiple connections, tink will automatically kind of pipe it, sends packets back and forth, it tests them to you and it texts transfer times so that it can find the most optimal path to send your network traffic down on its way. And so it's self-recovering. So if we lost six, or if we lost two, we could five, nine, six instead of five, two, six or whatever. And it solves. So it routes around failures, it continuously prompts for efficient reps and every node essentially is a daemon and that daemon is responsible for two things. It's responsible for a subnet and an IP address. And really the IP address is kind of optional. So in a traditional like OpenVPN, if you connect up to the OpenVPN, you get issued like a 10.something IP address and that's your IP address that is your source IP address as you hit every server inside the network. So with Tink when you connect up, like the node number one is assigned a subnet and as an operator you choose the size of that subnet. So what we did with my friends is we took a slash 16 out of 10 space. So 10.0.0.0 slash eight and we sliced 10.11.0.0 slash 16. So that's, I don't know. It's a lot of addresses. I think it's like 16,000 addresses or something like that. And we decided that the entire Tink network, which is flat, would be represented in that slash 16. And then each node we assigned a slash 24 to. Then we found out that some humans, so that was when we were assigning a human, a node and a subnet. And then we found out some humans run more than one computer. And so each human had a slash 24 and they started slicing their slash 24s up and creating nodes inside that slash 24. So each human gets a slash 24, each node gets something. Usually a slash 24 unless it's like a child node. So these, this node one and node 10 are, I honestly don't remember anymore, but these nodes are probably laptops that are connected up to a superior node and they just have like a slash 31, like one address. People with me kind of? Okay. So making diagrams with open source tools is hard. I did my best. This was a lot of work. I'm sorry. It's not better or labeled or even circles, but you know, where are the open office devs? Take my money. Okay. So it has a concept of connect to. This is literally a word in its config file. So when you start a Tink daemon in Unix, it reads its config file and it looks at the list of things to connect to and it sends sort of the Tink version of SIN. Hey, I would like to connect to you and they negotiate and if it's authorized they make a connection. So a node like this bottom node is kind of the classic laptop node. It just connects into the thing that it knows to connect to. Nodes like this one are pretty well connected with both things connecting to them and I guess you'd call that like a reflexive connection or a recursive, not recursive, but you know, it's a bidirectional connection. And again, this is looking at sort of the connection path. Once the connection is made and established, you have a graph that looks like this where traffic flows backwards and forwards across everything. So this, basically the difference between what makes up like a node like this one and a node like this one is that the node at the top probably has a well-known public IP address which allows something like a laptop or something that doesn't have a well-known public IP address to connect to it. And so there's a couple different classes of nodes like I discussed. If you have a piece of gear in a data center somewhere which some of us do just for reasons because you know, nerds, that's a great node for Tink, right? But if you have a laptop that's running around and conferencing and all this stuff, like it can have any IP address of the world so it needs to make an outgoing connection. Then there are things like in the middle where someone has like, I think around here the ISP is Cox, is that true? The ISP is Comcast. What's the LA ISP? Charter, Joy. Okay, so somebody has Charter and they like, you know, you have Linksys and you port forward off to some well-known port and so you use like DNS, DIN DNS or something like that. So that controls and so for things with well-known IP addresses, talking to well-known IP addresses, it's usually symmetrical. There are also places where people join the network at different times just because it's kind of a friend social group, right? It's not actually administratively controlled by someone with like Puppet or something who's doing it like right. So probably what happened is that this person was one of the earlier nodes and so his or her key is published widely and when this person joined the network, this person never bothered to update configs to connect to this person basically is what's happening. So how is Tink authenticated and secured? So the security layer is kind of beyond me but the way it's authenticated is that every Tink daemon that starts up generates an RSA key, a public and a private key and then you put obviously keep the public key or the private key on the Tink daemon node itself and then push the public key out to everybody else and there's a directory called hosts in the config file which we'll show later and that hosts just kind of has the IP address of the thing, the subnet it signs for and it's key and once you're connected, you're connected. So early on in the process when it was kind of figuring out who our friend group was and who we were gonna let do this and stuff, we got to the idea like I wanna let Alice connect to me but if Alice adds Bob, I don't wanna answer for Bob's traffic. I want some way to shut that down and so we looked into it a little bit harder and so at an IP layer, if you send a packet from one IP address on the Tink subnet to another IP address on the Tink subnet, you get no tracer out there. It just goes in and it comes right back out with zero hops. At the Tink layer, there's really no tooling to say I'm not gonna allow from this network, I'm not gonna allow from this network. There's no concept of like a time to live across it. And then somebody pointed out like pretty smartly that if we already allow traffic back and forth with Alice, like Alice can just nat for any traffic that she wants and so there's nothing I can do to prevent Bob's traffic from getting to me at that layer anyways. So we decided we'd calm down and stop bike shedding it and just trust everybody. It's interesting, this isn't an overlay network like Tor or something where it's public and anybody can join. Like you have to, someone would have to say, Spencer, I would like to join your network and then give me a key and I'd give them a key and then we could start communicating once we configured our daemons and restarted them. So when you get to daemon configuration and control, the Tink daemon is pretty cool. It's like, I don't know, I guess the word is old school, it's kind of before my time, but you control it with signals. So in this example, what we're doing is we're sending the user two signal to the Tink daemon and then we're just looking immediately at syslog and so when Tink gets that signal, it dumps connection state information as text into syslog. And so you can get some kind of useful stuff like the IP address of the connection. So you can see this is a bi-directional connection and you can see there's weight. This is the example network I set up so that I could show you things so that I didn't want to violate all my friend's privacy. If there were more connections, you'd see that they would all have different weights and those weights are used into determining which way we send packets out. And again, Tink is constantly pulling those and making sure they're the same. You also have a subnet list. So you can see that Spencer has a SAS 25, you can see that Becaro has a SAS 24 and this gives you an idea of which hosts are available and which subnets. This is great for kind of just as an operator being able to say I need the status command over here then look, it has a couple other things. So it has user one and user two both dump informational information and int the sig int which is what you get when you control C does not actually kill it. It increases logging verbosity to five which is kind of cool. So if things are getting weird and you want to see what's going on you hit it with an int signal, you go read syslog for a while. Once you've solved the problem, you hit it with five again it goes back to normal logging which is great. It's actually really, really nice. It's better than something like Apache where you have to go through a whole reload restart just to pick up more debugging information. However, if you're running Tink like the first time you're running Tink and you've got it in the foreground and you just control C and you would like it to stop and it gives you more things that's not the best experience. It has two more control signals. It has sig alarm and sig hub. So sig hub does kind of what you would expect it rereads the config file. So if you're bringing a new node in hub will have your machine reread keys and things and so a new node that's connecting to you will be allowed in right away. Alarm is like that but better so it'll reload all configurations and try to make an outgoing connection to any host it doesn't have an established connection to. So you can add more connect to's sig alarm and it'll connect over to them. I decided that this getting status it was cute but not DevOps-y. So, you know, I wrote a utility in go that has an HTTP endpoint. So you would curl at this 9,000 tink stat you just run dot such tink stat, right? And then when you curl at it it would do the humping and it were not the humping but it would send the user to grab the log file and then parse the log file and then dump it out in JSON. And then from this you have a programming interface that you can start to make really intelligent things. Really all I wanted was status in my task bar but you know, why not, go big or go home, right? So tink right now is at 1.0 a 1.1 releases on the horizon and that has a couple of things it has a tink info so you can just type tink info IP address tink info subnet tink info something and it will dump relative information for that and of course as we develop we can figure out what we actually wanted to type after tink info and make sure that that's supported too. Also if anybody's used HA proxy tink took a page out of their book and there will be a little control socket that you can kind of write, you can netcat things too and get things back and that'll be a much better interface than what I've got here. Okay, how are people doing? Not totally lost, awesome. So then we got to this phase the Ian Malcolm scientists. Yeah, okay so you have your friends and you turn on all these services there's a few things that once you have an overlay network with a flat IP space there's a few things you can just immediately start doing like you can just run Apache you can use UPNP streaming just really easily if anybody wants to VLC stream a movie they can, right? It's oh are you over there I'll just stream it to you or something like that which is almost more fun than useful. Also for anybody who still plays StarCraft like me you can play StarCraft over this which is pretty sweet but all right so if you remember we had Googles of information we knew that Spencer's stuff is somewhere in this slash 25 so one of those 128 addresses is probably doing something and one of these 256 addresses is probably doing something for Becaro's but discovering that is super hard so DNS, I need to know DNS I want to go to files.spencer.com.com or whatever you know and so we were like DNS, okay so we look at what other people are doing with Tink and so we did the first obvious thing which is that we started a host file we started copying the host file from person to person and modifying the host file, you know and that actually worked really well for a little while for those of you who were around at the early days of the internet I'm sure it worked really well for like a couple of years because while it's just being added to that kind of eventually converges on maximum file size, right? But then once people start repurposing servers and IP addresses and turning services off it becomes a nightmare that's way out of date and inconsistent. So okay, well there was this thing they did after host files, they set up DNS server so we set up a DNS server, somebody set up a bind and a couple of people set their resolve.conf to look at it and that was kind of that because when one person runs a bind on their server that doesn't help me at all, I can't push data into it you know, I don't know how to do his own update and so that the server sat there for a while and it got no utilization we were right back where we were using host files so the bind server didn't work and so kind of we stepped back a little bit and we said all right, well we have multiple administrative domains but we want like shared mutable state and it needs to, more than that it's like obviously we could just write to like a MySQL server or even like a DNS zone file or something but that server is gonna turn off randomly because this is all just weird hacker gear and Raspberry Pis and stuff so we wanted shared mutable highly available state and we were already DevOps so there's something that has shared mutable highly available kind of, you know we could do hip stuff, we could do the hipster things and we could set up etcd. How many people know what etcd is? That's like a third of you. Okay, so etcd is software from Corio, I mean CoroS it is an implementation of the RAF consensus protocol and is largely a copy of like ZooKeeper or Chubby if you worked at Google and the way it was described to me the reason they needed etcd is because when you start Docker containers they don't have a writable root or writable etcd so if you can't write the etcd to file system you can write it to the daemon so it's the etcd daemon. I don't know if that's apocryphal or not but that's a good way to think about it if you're not super familiar with it. This class of software along with Chubby and ZooKeeper and stuff is sometimes referred to as a distributed lock manager so like on the local system when we're in Unix we write a PID file and that's kind of our lock like I'm running this, I'm already running something and that way your daemon can check the PID file before trying to reserve the port and getting kicked out. Distributed lock manager brings that to clusters so you can have multiple machines and they say hey is somebody already running Nagios and they're like yep somebody's already running Nagios and they're like cool I don't need to run Nagios which is good because I don't want to run Nagios. And so when you look right at it it's like an HTTP interface to this weird go daemon. It supports like one, three or five or nine etcd nodes and it replicates data all between them and has a strong, the RAF consensus algorithm gives you a strong guarantee that if you write something if the write is accepted then that write is written and committed to and if you read you also have strong consensus and the cool thing is that it supports the idea that you can have multiple masters and one of the masters can just disappear for a while and they'll just keep on chugging, they'll just keep on chugging and if and when it comes back it'll get new data and so we kind of turned etcd which is again run usually by one administrative domain we decided that we'll use this as the shared way of kind of sharing a data set. When you look at it it ultimately looks like a hierarchical key value store so there's an LS command and you can LS the directory and like get keys out of the directory. Mostly I put strings in if you want to know more about etcd like you can put numbers, you can put IP addresses, you can put like counting locks like semaphores. And so we do the thing that kind of makes sense we start writing like hosts we make a host directory and in hosts we write names and in those names we put IP addresses and it's like it's not DNS but it's a shared way to write that kind of information now that's not just like an etherpad. So we decided to do something kind of stupid. This picture by the way, I just love this picture there's so much going on in this picture so like what are they doing up here? Like this isn't like boxes. This is like some kind of a tank with a compressor is this a compressed like gas? Because if it falls it's gonna burst and like take out a wall. Okay, well what's he doing? Is he holding the forklift? Is that his job? It's still, I got this it's not gonna fall over. Is this guy supervising? He's not holding any controls he's just making the system more unstable. And if you follow all their eyesight right it's all directed at the interesting part but what's interesting is this tire because if you look at this tire and you see the grounds probably like here you know and you look at this tire and you're like, oh this thing is in the process of tipping. Anyways, I would love to see the follow up for this. I don't know what happened. I don't know where it happened. I just I want to know. So we decided to do something stupid. So how many people know about the file in Etsy called nswitch.coff? All right, this is when I'm teaching people UNIX I'm gonna like yeah let's talk about UNIX it's the nswitch the name service switcher is kind of the center of the computer, right? It's this idea that lets us take a number that is the user ID and associate that with a user name. And so when we type LS we can see that the user name is crumb or something like that instead of just 111711. And it works for a couple there's a couple main name service groups. There's like users obviously so the associated between numbers and user names. The association between groups and user names there's the whole get ent suite, right? Where the home name, your office phone, your fax number, all that kind of good stuff. And there's also the original name service, right? Which was the domain name service. So in your name service switcher there's a line that says hosts colon. We all know what comes after hosts colon. It's windbind and then oh no wait. So it's host colon and then the word files and then space and then the word DNS. And that basically means first look at Etsy hosts and then look in the domain name system which probably means resolve.conf and then things pop a line. Well, it turns out that you can put things in between hosts and DNS. You can just write name service switcher plugins. And so one of our buddies, I don't know how much whiskey he had or whatever but he busted out C and wrote libnssetcd which basically just shelled out to curl and hit scd to return the correct name out of this key and shove it into the host name service. Which was a thing that happened and we sort of installed it and it sort of just worked. It just worked really well. And then one of our friends, so this is, yeah. So we set this up, somebody packed it for Arch of course because that's what you do. And then our other friend who worked at a company that will not be named wanted it to be less gross. So right as it was, I didn't even know you could shell out and see evidently you can. So there's this lib curl that's in C that you could use instead of shelling out to curl. So by making the software like 200 lines longer, we started using an actual library so we got way better HTTP handling. And this person felt that this contribution was inappropriate to apply based on where he worked and he didn't feel like he'd get approval. So he committed it as Elvis Presley at example.com and sent it up, okay, all right. And so our buddy who's an Arch maintainer decides to apply this patch to the Arch package. And so they type git patch, you know, apply or whatever. And you know what git 2.0 does is git 2.0 fails to apply the patch because example.com is not allowed by some RFC. So that was fun. The story isn't going anywhere. So DNS is solved. All right, cool. We have DNS. We have a weird pipeline that allows you to write keys into this stare data set. We have a thing that allows your computer to pick them up on the other side. So if you know a name, you can go get it and you can look at each other's cat gifts. Solid. So we find this thing which is way more hip than even at CD, which is console by Hashicorp. And it's really good software sort of. And it's just similar. It's got a lot of the same thing. It's still a RAF consensus protocol. It's still a Godamen that you run three or four different places. It still has that kind of one, three, five, nine, maybe seven, I don't know. And it has this idea of gossiping all the information and being able to refuse a right that it doesn't like. So it's very much the same what we're doing in at CD. However, this thing was optimized for the idea of service discovery in large clusters. And so there's a simpler way to get information out of it, which is DNS. This thing has a built-in DNS server. So you can dig at an instance of this with a name and you will return an IP address. It will actually return you a round Robin IP address of all the things it thinks provide that service. And it actually, to make it more complicated, they added kind of a Nagios check type system, kind of like HAProxy. So what it can do is you can register four things that are supposedly responding for a service. And then it will, in a tight loop, pull them and make sure they're all there. And then if any of them actually go away and start failing the Nagios checks, it'll pull them out of that round Robin. And so the answer you get when you make a DNS request will change. So this is cool. It speaks DNS on like not 453. It's like 8,500 or something like that. So you end up having to set up a local resolved DNS mask inside your system that you can use and it's kind of complicated. But we tried it out for a little while and it turned out it worked. However, there was an issue with it, which is that it doesn't have discoverability of discoverables. So with that CD, there was a simple file system hierarchy. Like it was an abstraction layer, but it was still there. You could type ls dash dash recursive and you would see all responses that are possibly there. And our data set was small enough that that was useful. With console, you had to a priori know what you were going to ask for. And then it was very good at responding to it on a protocol that made more sense. But you didn't know what to look for and that posed a problem. It also posed the problem that part of the point is to get away from HDP, right? So that means that the information that we want to convey is not just really a name, it's a tuple, it's a name in a port, maybe even the name of the service you should be speaking on that port if Etsy services are still working. So we did console for a little while and we ended up ripping it out. And I think the Etsy implementation was actually better, which is surprising because it was pretty bad. Okay. So I want to show you some demo stuff. Okay, so yeah. It's going to work. So if we just look at ifconfig example net, we can see what ifconfig looks like. This is the TumTap device that Tink creates when it's done. This hardware adder looks totally legit. It's an unspecified linking encapsulation because it's just software, right? You just write a packet on this interface, the software picks it up and writes it out somewhere else. So if we want to ping, we get these pings back from another node that is on the network, which is on its own really exciting because that's a conference Wi-Fi for you. And if we do an MTR to that same IP address or then same name, we can see that it's just picking it up and dropping it off. It doesn't matter how many hosts the way it is, from a network perspective, it's directly there. It's also worth noting that the way we've, Tink can be configured in a layer two or a layer three configuration and we've configured it in a layer three configuration, which means that there isn't really an ARP to spoof. There's, it's not like you can trust IP addresses yet, but you can't attack it through ARP spoofing. So with that set up, what we can look at is like protocols. The key advantage here is we have a flat network that's secure enough that we can use all these protocols that have been in the dustbin for like 20 years, which is kind of sweet. So like, what I like is, does anybody know what that command does? That's NFS, the network file system, like Google Drive, you can't touch this. So I don't know if people remember their university experience, but there was this awesome thing called the auto-mounter. And it lets you do stuff like this, where you just CD into a directory that does not exist. Wait a second, it auto-NFS mounts, and my prompt shows you that we're on an NFS directory from example telescope. And now we have this, we can see how big is this directory? Oh, we have three terabytes, awesome. That's the best. We can run that command, which is a full screen video played over the conference Wi-Fi over NFS, over the public internet. And the M-player flags are like, I need a cache, please. That's what that's all about. So there's this other thing you can do though. All right, so the computer has the ability to expend storage from a remote source. Like we said, this is a computer with more storage. So you can do stuff that like, you could make a user, like an NFS user, that HomeDir is mounted over NFS. So the HomeDir of this user is remote Home NFS user, and this directory is here. So I haven't like tried this yet, but you could have a laptop, like this thing apparently has a 3.6 terabyte HomeDir because the Home Directory is mounted over NFS. There's no local state for this user on this machine. It's all network traffic. And of course, Tink is set up as a user levels, as a system level process. So the user is not required to type a password or anything, it's all RSA keys. You could actually extend this kind of pattern to make sort of thin client laptops that just connect back to some mother base. You can bring out protocols that you haven't been able to do in a while, like, I don't know, finger is a command for those of you who aren't around when this was a thing. Not that I was, just that I got excited about it. Finger, in the past, we all had logins on the one big Linux machine. And the way you would like, instead of doing a stand-up, you would write a file called dot plan in your Home Directory. And if somebody, if your manager wanted to know what you were working on, they would type finger NFS user or like it could kind of tell them how many mails you hadn't read and stuff. That's pretty cool. I also want to show you kind of what the etcdctl ends up looking like. So it's just an LS. There's a lot of configuration in the background that you would discover if you tried to follow me in. But it's not, it's not, it's all setting of environment variables and URLs and stuff. It's just giving it a little, a tiny amount of bootstrapping information. Like, etcdctl ultimately needs to know where to go get etcd. But that's one piece, one string that you can memorize or socialize or something. And so you can LS in hosts. Oh, thank you. And so there's the two hosts that actually exist on this example network and some fake hosts that I created for fun. There's no tab completion, which is kind of annoying. And you can get an IP address out that way. You can set an IP address really easily too. You can set any key really. Of course, that's just writing a string into a key. There's no validation on like, you know, I mean, I can do stuff that doesn't make sense, like, I don't know if everybody can see that. But, and you get a little response. And the etcdctl is the easiest way to talk to etcd for sure, but it's actually just a curl. So if you thought about, if you spend a little bit of work on it, you could implement it in Bash with curl or at this point, etcd is popular enough software that there's libraries for Python and Ruby and brain fuck and everything. There's literally no requirement there. Cool. Any questions on the demo? Yeah. So there was a question of, does Tink remote manage the files? Oh, etcdctl, etcdctl, is kind of how they call that sometimes, etcdctl. What is, so your question is what? Oh, the question is, I hadn't seen etcdctl before. Have you used etcd without this? Oh, okay, yeah. So, etcd is just running over here. That's, that's etcd. It's just running in the foreground. And then etcdctl is just a command that basically just makes a couple curl requests. If you look at the help, it's pretty straightforward. It's just kind of, lsget set rm makedir. I really like the file system because as a Unix file system person, that's the kind of stuff I like. It's not perfect. If you're actually used to the shell, some of these will feel a little clinky, but that's better than just mystery database that I can't scan. I'll also show you some of the config files. So, I'll just go ahead and revoke all the keys after we're done with this so I can show them to you. Safe to say that there's a key in RSA, private key. There's an RSA public key which looks like that. I don't think there's a lot of surprise there. In the hosts, there's tink.conf and that says essentially, this is some information. So, we're gonna use IPv4. What my name is, so on that graph where there were numbers, the name would show up there. We're gonna connect to Becaro. We're gonna connect to Spencer Telescope. And the graph dump file is that thing I showed you that will dump the dot file so you can kind of visualize your graph. It's interesting that in tink.conf is where you specify who to connect to but you do not specify with this file who can connect to you. You specify that in a completely different way because that makes sense. So, in your host directory you have hosts files and so if you can't host Spencer Telescope that lets Spencer Telescope connect to me. That's the key, that's the subnet, the port and the address and this does two things. One, it provides information that I would use to connect to Spencer Telescope. The key allows me to authenticate them when I connect in and because this file is here, they're allowed to connect to me. Which is, to me, I would rather have a config line in tink.conf that says these are the things that are allowed to connect to me because this is kind of weird action at a distance. Another thing that's kind of confusing is the tinkup. So, tinkup is just kind of an entry point script if that makes any sense. So, we'll run tinkup after configuring the network and that's where you say stuff, that's where the actual IP address that your node should attain is set and you set the net mass to 255, 255, 255, 255.0.0. Note that .0 right there because again we're taking an entire slash 16 and that's the whole network and there's no routing really. Okay, any questions on any of that? Yeah, can you integrate that? The question is, can you integrate that into network, etsy network interfaces? And the answer is you don't have to because in etsy tink, there's a nets.boot and this file contains the names of all the networks that will start on boot and tinkup is always fired right after tink comes up. So, one of the things that kind of makes sense if you want to provide services onto tink, you can put a hub of your services in tink.up because it's just a shell script, it's actually not an if config line. So, that's what kind of makes sense, is like, hub it so that if it's listening on all global resource interfaces, it grabs the new interface in the front. The question is if A connects to B and B connects to C, can they talk? So, this is probably where we'll do that. So, if one connects to nine, nine connects to six, can six and one talk, the answer is 100% yes. All nodes can talk to all of the nodes seamlessly. Yeah. Okay, so what was pointed out that tink has really good IPv6 support, both inside and outside and that the transport layer that the kind of overlay network is connected via can be done over TCP or UDP. Yeah, is that a, it's perfect, okay. Yes, although in this environment, for all we know, it could be going five, two, six. I guess one, I would have to go through nine, but yeah. And I don't actually have a good sense of like what commands I would use to like actually interrogate that. Yeah, yes. So the question is does IPv6 work on the underlay network and the overlay network? And I think the answer is yes. Yeah. Over there. So this, it was kind of a statement, it was like you can run tink in ethernet mode and then overlay whatever layer four technology, layer three technology you want and use Quago or something like that for routing. Yeah. That wasn't a question? Okay. Yes? Okay, so the question was it has no routing, so how does it decide what path to take? And that was kind of one of those things that I could have chosen better words. It definitely has a concept of which subnets are where and a sense of routing. Maybe this layer, right? Because what the layer we're looking at right now is tink daemons talking to other tink daemons. However, the packets, the IP packets that you're writing and shipping out over these things as a user, you have no routing information. There's no routers that will respond to ICMP, nothing will decrement a TTL. That's what I meant when I said there's no routing. Yeah. So the question is if we're going from nine to two, is it gonna go nine, six to or nine, five to? And the answer is the daemon itself makes a decision and it's constantly probing to figure out which path is the most optimal. And I have no intelligence into the smartness or naivete of those algorithms. I'm in the back, sorry, you've been up for a while. So the question is, are there controls to set up routing, to set priority around it, to set cost on routing? I think there's a command to set a cost on a path, I think, but I'm not sure, I've not done that. Again, this is like hobbyist stuff for me, so I'm not deep into it. Maybe one of the other experts would like to raise their hand and answer that one. Is there a question there in the green? I think, so the question was, can you modify weights? And I think, oh, so yeah, I would refer you to the documentation on that one. The question is, can you set the weights? And I believe you can put cost on a link, but I don't know. This is getting to the advanced stages that I don't have knowledge of in the black. Yeah, so the suggestion was that if you wanna run it in a layer two mode, then you can BGP your heart's content of setting costs and timeouts and things. And gray. So the question is, if you wanted to publish your RSA public key, could you shove that in XD? Well, XD would obviously accept the right. However, the XD cluster is, we've configured it, which again is kind of a derpy configuration. Depends on being, you have to be on the network in order to read it. So, yes and no. There's an advantage in the sense that you only need one connection to the network, to be on the network at all, but you're stronger with more connections. So if you get one RSA, one nodes RSA key from a friend, and then you join the network, and then you could scan the XD to get the rest of those configurations, that's probably a good configuration. Yeah, so yeah. So the question is, can you do something kind of like a split tunnel, or a full tunnel with Tink, where your primary connectivity to the internet is going over Tink? And the answer is I don't know. Several of us, because we're a bunch of friends, which is almost like the word enemy, right? So there's been a fair amount of like, oh, I see you have route, more than one of us runs it on like the home router that's running Linux, and they're like, oh, I see you have IPv4 forwarding set. So they try to get packets going to the Tink network, go to sites for me and get Comcast mad at me. Nobody has of yet, in my configuration, been successful at doing that, even with someone's help. So we've not actually gotten it working. Is that because we're not as good at networking as we think we are? Probably. In the back. That's amazing. So there was a statement that there was, there's a giant free software basement, OpenStack Tink thing, which is probably my jam. Okay, so I wasn't actually done with the talk. Appreciate all the questions though. Okay, so there's some neat tricks you could do with this. So laptops, perhaps the most important thing is it's a flattening of the network, right? So laptops and things that typically are second-class citizens, especially when that's involved, all have known IP addresses, which means that at any time, as long as I'm on the Tink network, my buddy can ping my laptop. Now it's an exercise of the user to figure out something useful to do with that, but it's very useful. It's also useful in corporate networks, because I work for huge companies, right? IBM is a huge company. And it's very common for them to have kind of antiquated networking stacks or networking security models where they'll have completely isolated networks, completely isolated networks. And then your business unit is given three machines in this one and three machines in that one and four machines over here. And you would like to run some services. So by using Tink and an overlay network, you can make them look like they're all on the same network. One of the best things to do with this that somebody did is they made a backup device. So they got a Linux machine, they put like four or five hard drives in it, just a case, and they shipped it and they put Tink on it and they shipped it to their grandparents. And they had said, Grandpa, just plug it in. Grandpa plugs it into the router and then it DHCPs from Grandpa's router, connects up to the Tink well-known IP addresses and now it has a known IP address and can just start being disks that Grandpa won't lose. And now it's just a compute node or a storage node. And you don't have to mess with SSH-R. You don't have to say, Grandpa, what's my IP address.com? You don't have to say, I need you to press these buttons and links us to do the port forwarding. It just works. So brood war over Tink is of course one of the most important things. It's interesting because the, and I'm not 100% confident in how this works, but because your SSH connections that go over Tink aren't really super IP packets anymore. When you close your laptop and you just open it, sometimes they live. They live for like days sometimes. So whatever process is involved in actually issuing the TCP resets that causes a broken pipe doesn't happen. One of our nodes is Trans-Pacific. It's in Japan. So we had a lot of fun setting EtsyD, like a don't time out for a long time flags. I showed you NFS, I showed you a little bit of the NFS HomeDir for the laptop user. Crazy people have run X11 with Tink where you let your machine be a free target and then your buddy who's also on Tink, XIs this is over to your machine. Just for fun. So what's next for me? So now that we have kind of a service index of all these HTTP resets and NFS resources, the files people are providing to the network, I wanna get Reddit daemon that scans those and then puts them in something like a Zapien index so we can search for which files are available on the network. Tink 1.1 is coming out any day now. Gus is working. I know he takes donations. Sounds like there's a lot of buzz in the room about excitement for Tink. So 1.1 is something you can do. You can try the beta, you can donate to the project. Development, I think I'm gonna try, I'm not really a C programmer, but I am a CI guy. So I'm gonna probably see if I can get some kind of a CI pipeline, like a basic like does it compile, kind of pre-flight check, at least on GitHub pull requests. So yeah, conclusions. You can build an overlay network. It's kind of one of those things that it's a part, it's not a solution. So you have to figure out how to interpret this technology to your problem space. Console and NCD are pretty robust. They're a little obtuse. Starcraft is an excellent game. And this is, I think this is gonna be a pattern for my work in the future, is to take something like old, reliable, like Tink, dust it off and combine it with something like new and hip, like at CD, and see what kind of emergent behaviors can come out of that, because certainly we don't have to reinvent every technology. Thank you. Are there any more questions? I know we already kind of did Q and A. Yes, interesting. So yeah, we haven't really done scale checking. At my scale, I don't have that problem. Obviously, RAM is a consideration with any of those open network connections at some point, but probably well before that, one of the problems with routing, right? Like if you think about high-end gear and how it does routing, it keeps the entire routing table for the entire internet in memory. So well before you ran into other problems, you might run into problems where the amount of memory you're consuming, just keeping the state of the graph in memory is hard. Thanks everybody.