 So welcome to the scale We are in here in the container track. So what I'd like you guys to know the scale is a community event as A community event we are dependent upon our great sponsors to help make the rent affordable for you guys And so this room is actually sponsored by Q connect Q connect is a Company that specializes in networking and building relationships Within Southern, California to connect top talent to our clients in the technology industry They actually have a flyer up on the job board that I put up earlier So if you're getting of your guys are looking for jobs or wanting to post jobs, please check out the job board I've got a little bitly link there. So you can add stuff to a spreadsheet So what I like to do now is like to introduce the next speaker and we have I'm sorry Brandon Phillips. There we are CTO of core OS, which is pretty awesome. Thank you Can people can people? Oh Nice or Mike How's everyone doing good? How's the first day of the conference been I'm feeling pumped Yes, I tried to get in here last night But there's some plane problems I guess throughout the country. So there's a little troublesome so what what I'm going to be covering today is a Database that we built called at CD We're going to be talking about what at CD is why it's important and then we'll go through some applications Actually, how at CD ends up getting used in a variety of different systems so the By intro I'm Brandon Phillips on the CTO and co-founder of core OS We're a systems and infrastructure product company Founded in San Francisco about three years ago. Almost three years ago exactly We've built a number of different things which I'll go through and I've been a systems engineer for a long time worked at SUSE Linux and Rackspace and then most recently here at core OS so As a quick introduction since we're going to be talking about at CD and it is really kind of a quarter soon of core OS I'd like to just give a little intro on what that is If you're not familiar core OS got its namesake because we built a Linux distribution called core OS Linux And so as a Linux distribution it runs in a different bunch of different places It runs on this mysterious platform called hardware also not just all these cloud platforms And then core OS is also a collection of open source projects So the some of the open source projects that we're well known for include core OS Linux But also at CD flannel rocket decks and number are the things that you can find on github.com slash core OS We have a large community of contributors, so We've rolled over over a thousand contributors Over the counter so far of contributors the core OS projects We have a lot of people who are repeat contributors that love for any of you to be repeat contributors to our projects And we really do strive to make these inclusive Communities around these projects We're also a company with aspirations to Help run the world servers and to do that in a secure fashion And so we have a number of commercial products one of which is called tectonic Which includes the full core OS Linux stack from core OS Linux through container run times and Container build systems and distribution systems up to a clustering system based on kubernetes If you're interested in commercial support around any of this stuff. It's at tectonic.com We also have a thing for hosting container images and building them as a software as a service called quay or Key if you know how to pronounce words at key.io So Why we built core OS As a systems engineer my co-founder Alex polvi and I he'd spent a lot of time in infrastructure on the cloud and I'd spent a lot of time building enterprise Linux distributions at SUSE And we really felt like with all the times that we'd cut ourselves quite significantly on package management configuration management deploy systems hand-rolled batch scripts That there is probably a lot of better ways of running infrastructure and there might be some opportunities for building open source projects and products around it and so We were really inspired by the way Google runs their infrastructure And there's a really good paper that describes this called the data center as a computer But really the whole concept behind this data center as a computer that Google espouses as the right way to run infrastructure is that it's focused around you as a software engineer Particularly somebody who's trying to take some code and get it running on some servers behind a load balancer So you package up your code into a binary called the container image and that container image is nothing really fancy It's a tar ball that includes everything required for your Application to run you package this up you name it. Hopefully you sign it That's a little detail that we're still working out on a lot of these things, but you sign it And then as an operations engineer, they have a nice boundary So you have this self-contained object and then as an operations engineer you worry about a bunch of servers This data center as a computer isn't one gigantic classic mainframe system It's made up of a bunch of little cells and each of those cells is an individual server So you take those servers You're told something like I need three copies of the application running Some piece of software lands it on the boxes. It doesn't really matter how that happens or what's underneath It's just that that asset is now running as a process behind some sort of load balancer on those machines and so How we actually accomplish that whole story is we build a lot of technology a lot of hard technology to get right to and so It all kind of looks like this at a high-level diagram is that you have a system whereby you have Etsy D which is this database that we're going to be talking about You have some sort of control cluster some sort of machines that people Like engineers and operations people talk to you to say I want X copies of my application running in the cluster I mean you have a bunch of More or less dumb Servers inside the data center that are just sitting there polling for work from the control cluster saying what should I be doing? What configuration should I be in etc etc? So this is kind of the architecture that we're going for it's It's kind of it's essentially a very sophisticated well-designed botnet So that's that's what we're going for And kind of the special property of these systems you'll notice that there's Like lots of the yellow boxes. I didn't really visualize it well for Etsy D But the whole point of these systems is that if any of these boxes go away any of the individual boxes go away The entire system keeps going just like when an individual cell on our body dies We kind of keep going we really don't notice and that's you want to build these systems because You want to build these systems in this way because we're trying to achieve operations sort of nirvana Where if I go into your data center and one of your machine dies You're not getting page if I add more machines to the data center. We increase capacity I think this is what a lot of us strive for if we operate servers But how many people have achieved this on their own projects? Who would be super pumped if I went and pulled plugs from servers in your data center? No, okay. Hey So there's a few sophisticated orgs that do that, but for the vast majority of us, that's not true So it all begins with the operating system these individual dumb machines And those operating systems get through some configuration get joined into the cluster and Then there's cluster operations where we care about what are all the machines doing and really this is a big distributed Configuration problem so we want to make sure that we distribute the configuration of the cluster across a lot of hosts Because if one of the hosts holding on if there's only one host holding on the configuration and that host dies And we have a lot of servers they're out of work So we want to make sure that configuration of the cluster is distributed But we also want to make sure that there's a single point where everyone's coordinating their work. Otherwise, it gets really really difficult It gets really really difficult for human beings to reason about it If there's a thousand and one different places you need to update in order to deploy your app So we want replicated and distributed in that manner, but we want kind of centralized control for the human being So I'm going to be going through a number of demos We're going to get into more Practical stuff now that the introductory stuff that away. I'm going to be going through a number of demos It's on github.com slash phillips slash hacks all these slides will be available After the talk I'll tweet and then they'll be on the scale talk. I uploaded the slides for them, too But that's where this stuff will be if you want to try to reproduce it at home on your laptop So at CD At CD, it's a little bit of a complicated name to pronounce for a lot of people it's It's unclear what this combination of consonants might end up being but the The name came from our idea that we wanted something like slash Etsy but distributed over a lot of posts Now I know this can be very confusing because it's at CD is not a file system. That's actually a key value store So it's probably not the best analogy and it's a difficult pronounced name And we're engineers and naming things as hard. So at CD is what it is At CD has a number of Important properties one of which is that it's open-source software We've been working on it along with the community for about three years It's fault tolerant meaning that it is designed to handle situations where inside the at CD cluster a single machine dies The rest of the cluster continues It's durable meaning that if the entire data center loses power and then gets its power back all of the information remains in at CD and Essentially meaning that we sync every time that we get a right into at CD to make sure that it actually hits the SSD or the platter It's watchable meaning that clients have the ability to see all the changes that are happening to the key space and trigger events on on that which we'll see why that's useful in a couple of different situations and then Importantly, it's exposed via HTTP. So it's readily accessible to things like curl or You know any any language that talk HTTP, which is pretty much anything that we care about All right And oh and most importantly and very importantly actually is it's runtime reconfigurable So there have been systems like at CD built in the past notably zookeeper, which is a very popular consistent key value store but those systems particularly at a time when we're building at CD weren't really designed for Cloud or places where you're reconfiguring your servers where servers are coming on and going away on a regular basis And so while the at CD cluster is operating you can add and remove machines from at CD The API is pretty simple It is you get to get keys you put to put keys and you delete to delete keys So no real surprises there. So let's dive into the basics of at CD and And what it's what it's all about. So a typical at CD cluster is generally three to five members at CD is based on a consensus protocol essentially Think of it as a way of just replicating a ever-growing log across a lot a bunch of machines and ensuring that that that nobody has a different copy of that log at any time so raft essentially is a sophisticated algorithm for doing that And inside of raft there's a strong leader and so at CD has a strong leader and a typical cluster looks like this three to five machines and with everyone talking to the strong leader and Then everyone else who's not the leader being a follower Now the basics of at CD's API Let's go ahead and get to a demo here and everyone Can everyone read my screen all right from the back So what I'm going to do here first is I have a checked out copy of the at CD source code at CD is written in pure go So if you're not familiar with go you can essentially just do this And a copy of at CD will be put into your go path What I'm going to be doing is we have inside of the at CD source code is we have a proc file If you're not familiar with a proc file It's sort of like a service file and system D or something just describes some stuff that we want to run under supervision And in this proc file we set up a three machine at CD cluster and so I'll launch this proc file by typing go reamon start and We'll have different colored log lines for each of the machines inside this cluster But I have a three-member cluster now running on my laptop for the purposes of this demonstration Okay, now that we have at CD setup at CD has a command line tool called at CD CTL That allows us to interact with the API And we'll go ahead and put on that We're gonna want output out of JSON and you can do things like set food a bar and We'll get back up bunch of information about what happened So it's just a basic key value store and it's also a Key value sort of where generally use some sort of hierarchy so you can set slash food slash baz the bar And it has the concept of directories. Although that's changing in future versions Okay Any questions on the at CD? API yeah in the back Yeah, so the question was How is how is service discovery working inside of at CD and how does it actually help do service discovery? Is that the? Yeah, so Yeah, so at CD doesn't by itself do service discovery, but a number of very well a Number of very interesting and good service discovery systems are built on at CD the reason that they build on top of at CD is because Service discovery is kind of the backbone of these security systems and you need these systems to be replicated and fault-pollering So what I'll be demoing throughout the talk will be a DNS server pretty pretty central Service discovery mechanism a configuration service and a load balancer that are all backed by at CD Yep So it's essentially a key value source so you can get and set keys And you can also watch for changes. So if I do Set food a bar and another terminal here. I'll watch on food So when I set this this other Terminal Essentially immediately returns because it's doing a long polling HTTP requests and get far out And so a lot of mechanisms that I'll be demoing use this to update clients in real time of changes happening inside the cluster All right at CD fault tolerance so The the reason at CD exists and we're not using something like Redis or my sequel is because of its fault tolerance, so At CD remains available on side of a single machine failure at CD remains available Which means it's read and write available you read and write from it in the face of two machine failure In the face of three out of five machines at CD can't make forward progress because if we allowed for forward progress You might have half your cluster making one decision and getting one copy of the log and after cluster making the other so It's tolerant of failure up to a point I mean you think of it as essentially requiring democracy a 51% vote a simple majority in order to move forward and then it's Tolerant of leader failures. So again, we lose one machine. It's fine if we lose the leader It's fine within three to five RTT, which essentially is on the order of a second or two at CD will do a leader election figure this out and Restore the cluster into an available state and continue making forward progress Without any intervention of human beings. A lot of these systems would allow you to have Followers without human intervention going away. So I post-gres and these sorts of things But at CD handles this without human intervention Which is really important because we want to build these systems Particularly as they grow in a way that we're not waking up people in the middle of the night to handle situations that computers can really handle by themselves Yes Great question. So the question was how do we know that? The new leader has the most up-to-date version of all the data inside the cluster This is a guarantee made by the consensus protocol that we use called raft Essentially if we implemented rafts Correctly, which we have very high confidence of because we spend a lot of time testing it because a year or so to design the tests Not necessarily write the code we The person while the election happens everyone kind of talks about I Would like you to elect me and here's the most recent version of the log that I have And you can't vote for somebody who has less of a log than you have as long as everyone's voting and everyone has a Correct version of what's going on. So we we continue to have this state of everyone's using the algorithm correctly Then everyone will vote correctly and everyone won't vote for somebody that's less their log and The guarantees essentially will continue through the the life of the of the cluster. Yeah Yeah, so the question Yeah Yeah, the question was whether a system like this is useful across wands. It is Useful across wands. You just have to tell at CD essentially what your expected latency is between machines So if you're running across wands and the latency is 300 milliseconds between machines as long as you can figure at CD It can operate correctly in that environment Google has a similar system called chubby. I've never worked at Google. But my understanding is that they have a Tuned version of chubby that's globally available. It's called global chubby very creative name and Global chubby is essentially tuned to be aware that the latency is such and doesn't cause leader elections unless we go over that latency. Yes Yep. Yeah, so the question was what issues would you expect if the clock skewed on at CD? so This is a this is a great question When I say timestamp, I actually mean raft log index And this is a linearized ever-increasing unsigned integer 64 bit integer Where essentially every transaction increments this number and there's actually two numbers There's the index and then there's also the term which is used for leader election and so These numbers essentially are free from any concern of time Now at CD does do expiration of keys which is used for leader elections If you have a clock skew and expiration of keys This can be bad if a leader election happens and there's the leaders the old new leader have different understanding of clock But generally those Are done on the orders of tens of seconds or seconds And so if you have clock skew on the order of tens of seconds, you already have problems probably in other parts of your stack we have found that people actually do have clocks skew of on the order of ten seconds or hundreds of seconds and expected distributed system to work correctly In this way, I would say that you need to kind of check Some assumptions in your system because probably a lot of other things are going wrong But at CD v3 will handle this situation also And at CD v3 we essentially automatically refresh all leases That we're bound to expire after a leader election and this solves the number of issues Particularly around clock skew. We also warn you quite heavily because we do exchange time stamps internally And we warn you like there's some serious clock skew in the logs If you're paying attention, but I mean if you're clock skew, you're probably already not paying attention. So Yeah Yeah, so the question was how are the notes communicating with each other Essentially, it's a protobufse over HTTP And internal internal communication Yeah, the question was if it's private better to have a private network between the nodes. Yes Like having a lower latency and better guarantees is always better for these systems. It's not it's not 100% necessary But it also just depends on how much how much load you're expecting but put on it Essentially, you need to have some reasonable expectation of latency because in the system We we do leader election decisions and whether a leader is alive or not based on latencies and latency timeouts And so if you have a system where I'm going to expect like a 500 millisecond pause on packets then CD is going to be a little grumpy and do unneeded leader election Yes, yeah, so the question was whether is any concept of authentication? We have a very simple authentication via TLS client certificates We do have an authentication API also where you can lock down parts of the key space using Using like identities so you can have roles and users in those roles and those roles can Have read modify sort of rights Yeah, it needs to be like at CD needs to be told through its API what the rules are for its key space So you'd have to mirror it from if you have an external source of truth You have to talk to the API and say these are the rules for the They can't like directly do LDAP or something Yeah, sorry all right cool So the other bit is at CDs durability. So there's three basic things Around at CDs durability one is that we have a wall file essentially a ever-growing log file This is the thing that raft is essentially Replicating for us. We have snapshots where we take the wall file and we truncate it because we can't have an ever-growing piece of data on disk and in memory that would not be great for a variety of reasons and then just talk about how you back up at CD so When we launched this cluster, we got three at CD directories each of the three machines that are running or Three at CD members running on my laptop are called in for one two and three. So we'll go into infra twos at CD directory and There's two things called wall and snap The wall file the numbers are essentially The numbers of these wall and snap files are related to the Index that I talked about earlier so that ever-growing unsigned UN 64 and these things are pretty easy to back up So just copy it anywhere s3 whatever if they if you get an impartial copy We have CRC check thumbs to the entire thing. So we'll either tell you that it's messed up And if you mess up the end of the file To some limit up here what it is blast. I think it's the last entry We'll just truncate it. So we'll say the CRC's messed up for the last entry. It must have been incomplete right And we'll truncate that entry. There's a CRC mismatch in the middle of the file Of course, we'll air out and tell you we're not going to proceed because some things terribly miss So, yeah, any dumb or sinker s3 copy can back up at CD successfully All right Now at CD bootstrap distributed systems are notoriously difficult to get started There's very simple way of getting started where I showed you just doing this proc file starting But if you're actually running on servers, it can be a little hard We we run this service Called discovery at CD IO, which is a way of helping you bootstrap. Essentially you hit this service We give you a URL that you can use you distribute this URL told the members of the cluster that you want to start say if you're using Pixie then you put this in the cloud config on your pixie server or if you're using a Cloud formation you put this in the cloud in it for your cloud formation on AWS Or if you're using vagrant you put this in the user data, etc. So You distribute this little token everything uploads when it first starts at CD uploads to this token It's IP address and then they all just wait until the size of the cluster that you're trying to bootstrap is reached And then the cluster comes up. So you don't have to know all the IP addresses Ahead of time of your clusters, which is actually super annoying in a variety of cloud and physical gear environments where you have like DHCP or VPS or VPC assigning random IPs Okay, so the process essentially looks like this one machine comes up a couple more come up They're just sitting there not doing anything useful waiting for the full cluster to come up all five machines come up They do the initial leader election From the configuration they get it out of discovery and then they continue and discoveries never used again It's only used for this bootstrapping Okay This whole process can be done totally manually manually to so if you already know the IP addresses and stuff you just Give at CD on the command line or through environment variables the list of all the IP addresses of every other machine in the cluster They similarly wait and then do the initial leader election when everyone comes up Any questions around bootstrapping? Yes No, so at CD will go up to five and then start and then nothing more after that After that you need to use library configuration which conveniently is exactly the next topic So at CD like I talked about has this concept of live Reconfiguration, so there's an at CD command line sub-command called city CTL member list and as you can imagine there's a list add remove and update so you can add and remove members from the cluster live and It is pretty straightforward. So let's say that I have a five-member cluster The one on the far right there is flaky It's network cards going out or something's going wrong with it So I can use that to be CTL member remove remove that machine bring up a new machine with that to be CTL member add and then The cluster never at any point was in an unhealthy state It was just as if this this one was rebooting for an update or something All right, so The the main point of this talk is to talk about why why city is important besides just being a key value store and That's through the applications built on at CD So the original reason that we built at CD was for a use case that we called locksmith And this is Inside of Coros Linux, we have this automatic updating system, which it is very similar to like our Android phones and our iOS phones where Us as Coros Linux on a variety of channels push updates and the Coros Linux machines update To the latest version of Coros This is a very timely week to be giving this talk because we had a terrible slew of security vulnerabilities We have an SSH vulnerability a Linux kernel zero-day vulnerability and then most recently in TPD Vulnerability all of which we patched within about a day and we pushed automatic updates and machines running Coros Linux with automatic updates Simply downloaded Cryptographically verified applied and then rebooted onto the new version of Coros and we have this atomic updating system in Coros where We atomically move from version a it's version B of Coros Linux And we can roll back at the boot loader level. So even if we screw up a kernel update Hopefully we're able to roll back from that now If you're a little troublesome if all of your Coros Linux machines decided, you know what? I need a reboot right now You do not want that to happen. You're running thousands of machines We designed this sophisticated system called at CD so that you know if any individual machine goes down You don't get paged But then if we're completely out of control let everyone in the cluster reboot that would be bad. So we built Locksmith on top of at CD and what locksmith does is it holds on to a semaphore essentially a counter with atomic properties and If a server needs a reboot it goes off and ask locksmith, which is backed by a CD Hey, I need a reboot it decrements the semaphore inside of at CD at CD allows for these distributed atomic operations If if it gets that semaphore it reboots and then when it returns back from its reboot Hopefully successfully it releases the lock if unsuccessful then it holds on to that semaphore reboots start to stall and hopefully a system administrator comes through and and Goes and investigates why that one machine never released its lock So it's a way of automating these automatic rollouts and it's the first application that actually use at CD And we still use this today and a lot of people rely on it. Yeah Yeah No, so if nobody was in the middle of a reboot So every time a machine comes back up if it's running the locksmith agent It will check to see if it itself told the cluster that it would hold on to a lock And so as long as it comes up successfully and we measure success as the ability to talk to a CoroS Update instance and so as long as it's able to talk to the place that it's supposed to get its updates from again Successfully then it unlocks that lock. So if the full data center goes down, essentially everyone will immediately check for Whether it was holding a lock when it lost power and unlock itself All right So that's a pretty simple app Pretty robust. We haven't done much work on that but it continues to be used in production and stuff in a lot of places Yeah, we haven't really done any any of that sort of fencing stuff Yeah So the next thing is a sky DNS. So this is an application that's That essentially exposes a DNS server through at CD. So why don't we go through this? So I Have that that repository that I talked about This is the read me file of that repository. And so what we'll do here is we'll launch sky DNS We'll tell it that the entity server that it's getting its information from is on one two seven zero zero one And we'll launch it. So I already have that running in this window We'll launch it again. So that's a sky DNS being launched All right, and then what we're going to do here is we're going to do a variety of Different entries in the sky DNS and show you just roughly how sky DNS works. So this first thing that we're doing here a little hard to read but What we're doing here is we're setting a key in sky DNS sky DNS has this idea of a hierarchical DNS space With wild cards. So we're going to be telling sky DNS that in production in our East data center, we have a mail server called mail-east and The whole thing is going to be rooted at example.com. So we'll Put this entry in there and then use dig which is the standard DNS tool to get the server could back out All right, so that was me just putting that CDC tail that command and then digging and we see that Sky DNS immediately responds. Yeah, here's the server record for mail-east. All right so the next bit is that we're going to add a DNS entry for a mail server in our Say west data center and very similarly query query the DNS server and get out the mail server for that location and then the interesting thing that I really think is needed about sky DNS is it allows for wild card queries so for those who can't read it in the back We're able to ask well, what are all the mail servers that are in production no matter what data center We're in and for those of you you can't see that We get back the two entries for each of the data centers that we put in So that's sky DNS and it's used in a variety of things including kubernetes Which we'll talk about later, but it's a nice standalone service discovery mechanism and it can do all sorts of different DNS records serve records are just particularly nice because they can be used for load balancers like an engine X and etc All right, so the next one that we will talk about is Vulkan Actually, I'll talk about comf D because that's the next one in the read me. So comf D is a way of using templates and using SCD to hold on to configuration information and then rewriting templates on the fly so you can use this to Rewrite an engine X config or database config, etc. So what we have here is we have We have comf D running so comf D command line tool you can specify the back end is at CD and that Where the templates will come from we have this comf D directory Which I'll show you in a second and then it'll be watching for changes so it'll be sitting there on this machine and watching for changes to the relevant keys and Rewriting the configuration based on those changes. So inside the comf D directory. We have this toml file and it says that The source is going to be my config.com template and that we're going to write out to slash temp my config.com Comf and that the two at CD keys that we need to fill out in this template are my app database URL and my app database user And if we look at the template It's essentially going to write out an I and I sort of file That that reads in those two key values Okay, so we'll go ahead and launch comf D and if we Just launch it. We should see that Our config is written out immediately and then if we set Let's see Here we go, so if we set the value of my app database user from something different from admin 5 to admin 2000 what should happen is that the config gets rewritten to admin 2000 So it's just a simple way of distributing configuration changes across the cluster All right, and then Vulkan D is a really interesting project. It's a load balancer that's driven by an API and it stores all of its configuration data and inside of it CD, so it's it's written in go it can do essentially reverse proxying with a bunch of different algorithms for health management and and Middlewares for authentication and logging and rate limiting of IP addresses and that sort of thing is fairly fairly sophisticated API Focus on on reverse proxying for API servers. You can do routing at l7 layer and that sort of stuff the architecture of Vulkan is Looks like this. It's a little blurry, but You have a Vulkan agent or the Vulkan server at the top Vulkan is coordinating with that CD to get all of the configuration data And then it talks to a number of backends and those backends can be registered to various routes on on the on the instance So the idea would be that you have, you know 20 Vulkan instances that are behind your ELBs or your front-end internet load balancers and this They're doing all of the rate limiting for incoming traffic based on identities of the headers, etc All right, so let's go ahead and demo Vulkan So Vulkan similar to SkyDNS and Comfy You give it where it the location of where it CD is and then some information on You know logging and debugging etc. And so what we're going to be doing here is I'm running See So I'm running a couple of different services the first thing I'm going to be running is a Is a Python Simple HTTP server if you're not familiar with this Very simple way of just exposing the current working directory as an HTTP server And it's going to expose it on port 5000 And so we'll go ahead and do that and then there's this command line load balancing or What I'm looking for tool for Benchmarking HTTP servers called boom. Has anyone used boom before? Okay, I love it only because like first. I love the tool. It works really well. It's written in go It's really easy to get onto your host, but also the Progress bar just says boom That's the best part And so if I guess we've missed it what happened here Where'd it go? Oh Man, where'd my boom go? Okay, let's see. I'm so fast Well, it's more interesting when you when you actually run it on something on the internet Just trust me. It does boom install it And so Well, we'll notice when we do run boom is that we're getting latency it does a histogram And we're getting latency that it's pretty high. So it's like a tenth of a second latency And so what we're going to do instead is we're going to add in go server and goes a lot faster than Python at serving HTTP No offense to Python, but Architecturally go will win every time on this. So this is server.go file Implements the exact same logic as the Python simple HTTP server It serves the current working directory as a thing. So they're doing equivalent equivalent amounts of work So we'll run boom again, and we'll see that we essentially moved most of the work up to a much less latency And what's kind of fun is that we can see here that there's two back ends and we'll notice one is green and one is red Red means that one of them is is being very slow to respond to requests Anyone want to guess which of the back ends is being really slow to respond to requests? It's it's the Python one It's just getting demolished by the go one So so what Vulkan's trying to do is load balance equally across the the two servers Backends it detects that one's acting much slower and shoves a majority of the traffic over to the other one And so that that's kind of Vulkan. It's very difficult to read it 1024 resolution it looks much prettier on larger displays But that's Vulkan and that's used in a variety of spots in production at mail gun most notably But I know of a lot of other people were using it pretty successfully All right any questions on on those use cases So what I'm going to do here in the last bit who's who's heard of Kubernetes Yes, perfect perfect who who who who's used a scheduler-based system or understand scheduler-based systems? Okay, much less confidence Okay, so one of the the other major use cases besides use Service discovery, which is what Confdy sky DNS and Vulkan are kind of about that we built Etsy D4 We also built Etsy D to be really really good at Holding on to information for schedulers and so Kubernetes Backs its scheduling system with By Etsy D so all the scheduler information is stored in Etsy D And that's how you're able to have multiple schedulers running under Kubernetes And you have multiple API servers running under Kubernetes. So What scheduling really is about is about getting work to servers We've all actually written schedulers even if we may not understand them We've all done this This is us being a scheduler a human being scheduler So we compile our application we s copy it over to the server and then we tell that server run the application One thing here that I always like is who's used system D run before? Okay, who's who's who's Ryan an application in production and screen? Yeah Yeah Every time so don't do that System D run gives you the convenience of running in screen with with a way fewer dangers of running in screen And it actually if you're on a system D system, which congrats almost all of you are You you get a system D service file automatically created for you those That service will actually dump into the central journal of the system So you'll be able to go back and actually read those logs even if they scrolled out of a screen buffer And they'll act and behave like a normal system daemon. So use system D run. Don't use screen again I'll find you So this this is you being a scheduler And then a lot of us have written more sophisticated scheduler the four I in machines that I know about scheduler Fabric is a really good one for doing this and so write a scheduler that essentially loops over them one by one and puts runs the application runs in the deploy Or we might get really fancy might get a little sophisticated You'll spin up some machines on AWS So you'll buy some special hardware with like fast SSDs or some GPUs or something and you'll tag those machines special Give them special host names and you'll kind of do some resource aware scheduling where you know where the resources your app needs or somewhere and then You know if we get really fancy Somebody might get get smart and they say you know what let's let's deploy the app to the machine That has the lowest load average right now And so you you write your four I in hosts that I know about script You find that host one has a low so that low average and then you s copy in and deploy it there So we've all written schedulers and what What we can do in a more sophisticated manner and what things like kubernetes are about what Google's infrastructure is about How Twitter runs their infrastructure lots of organizations run the infrastructure is instead of having you thinking about S copying things and writing a fabric script in a for loop You talk to a scheduler API the scheduler Does its scheduling routine and it decides what machines your work actually lands on And it all works on this very basic premise It's a it's a while loop that just constantly is doing this thing where it looks at the current desired state The state that the human being wishes the system would be in so whether that state is I have zero copies of the app running I want five copies or have five copies only want one copy or have five copies running version one And I want five copies running version two And then it looks at the current state of the system Which is what all these agents running on the hosts are doing they're reporting their current state and it says I'm gonna schedule. I'm gonna take that to your list I'm gonna diff those two things take that to your list run it through my schedule routine, and that's it It's a very straightforward thing That's why we're seeing kind of a lot of different interesting scheduling systems being built these days I won't downplay that there's no sophisticated hard computer science in here because Entire PhD theses have been based on how do you efficiently do this? But this is essentially what what the idea of this is and so what? How it's idea is involved in this is that it's the thing holding on to the desired state It's the thing that's replicating what the human being wants across a bunch of machines And that's the thing that when a scheduler comes up. It starts to look at all right so the other thing that SED is used for inside of Kubernetes is it's used as as a backing store for DNS and load balancers and a service discovery mechanism inside of Kubernetes called Labels and so Kubernetes kind of pulls together this idea of Of service discovery, which we've kind of talked about all these disparate tools doing and this idea of scheduling into kind of a cohesive system All right, so we'll go ahead and do a quick demo of Kubernetes So Kubernetes is Is is backed by a CD if you log into any? Kubernetes cluster, there'll be an at CD cluster behind it and the API servers are essentially front-ending That's a deep so what I'm going to do here is launch the guestbook application For those of you who haven't kind of taken a deep look at Kubernetes The service discovery mechanism is very straightforwardly called services And so we have a three tiered app where we have a web app We have a load balancer and then we have a database underneath it and the service essentially says take all the things that Are labeled guestbook and I want you to put them behind a load balancer on port 3000 and it'll act as the TCP load balancer taking kind of the behavior of that Vulcan load balancer that we talked about and the thing that the Scheduler is concerned about is called a controller and the controller is the thing that says I have this particular container image and I want to have this many copies of it running somewhere in the cluster and so similarly there's going to be these sort of manifests for the database and for the front-end and so what we'll do is just first delete the copy of this application that I had running in my cluster and then and Then relaunch it and while that is going what I'm going to also do is do coop CTL I just read this man. I don't think it's watch One second while I remember the verb that is used It anyone remembers that's watch. What is it Vince Vince get events? Yes There we go. Thank you so much Too many verbs everyone chooses different verbs so This will be a log essentially a live log that's backed by at CD events of everything that's happening inside of the cluster so So what I'll do is all make sure I'm in the right degree So what I'll do is I'll go through all of these manifests that describe this application I'll go through each of them and kind of create them and at the same time We'll see everything that's happening in the cluster essentially the cluster is getting the container Launching the containers ensuring that they're up and running on the machines scheduling the containers to machines, etc And so at the end of this what we should get is we should get a set of running applications and the set of running applications Will hopefully be behind a load balancer on my laptop So we go here Woohoo And so we have a running application That's a web app with a database pack in if you want to try it out This is the first time I've tried to run this on a public Interface this is my laptop being load balanced behind this service called in grok, which is pretty cool It's like a reverse load balancer so you connect out to it and then it routes things. Yeah Good guy audience member. Thank you So yeah This is actually us abusing the conference Wi-Fi to a miraculous degree So thanks to the scale team for running such awesome Wi-Fi Yeah, so this is you're hitting my laptop and then while people are hitting this web application I can scale there the The application up and down So what we should see is that now instead of having three copies of the app running? I'll have more of them. Hopefully soon Yeah, so we have more copies coming online because I'm getting such high traffic from the audience here I mean they really scale it up but Sadly, this is all running on a single virtual machine on my laptop, so no amount of scaling will help. Okay, so That's kind of an overview of Kubernetes. I'll just talk really briefly about the Kubernetes label system and then we'll leave a couple of minutes for questions and answers You can take the rest of my stickers and we'll call it a day. So Kubernetes it goes beyond DNS DNS can be kind of constricting in a variety of ways. We talked about SkyDNS Having this idea of a hierarchy, but sometimes a lot of the things that we want to do goes beyond a hierarchy so we have our application and This is a very classic setup. We have environment equals dev and environment equals tests environment equals prod But they're all the same application And we may want to have disjoint sets of these things behind different load balancers It's like a really practical example is like we have test.example.com That is just our testing environment, but then we may have beta.example.com that hits both our Test environment and our production environment. So we see what will happen when we do that rolling upgrade tomorrow And then we have our production services and so you imagine having to In a variety of different scenarios wanting to make various queries or maybe you want to have geo load balancing of beta.example.com so you select This particular region and then from both production and and beta So you can kind of mix and match it's more like it making a database query It's how the Kubernetes service discovery system works and then it nicely flattens it all down to DNS for compatibility with things like essentially every service in the world. So our guestbook example actually Our guestbook example here Thanks everybody Our guestbook example here is Man, I don't know. I feel like I'm gonna get kicked off stage of Jim that comes in so So I Forget the so things like Redis and our guestbook application are actually querying a DNS entry that is Cluster local So, yeah, there's a there's a cluster local DNS server, which I'm querying here So like that is used internally to resolve these sorts of things so that Applications that aren't Kubernetes aware it can still use services discovery mechanisms. They understand like load balancers and DNS All right If you want to play with this stuff, we have a really easy Guide for running this under Kubernetes on your laptop or running this under vagrant on your laptop Coral s Kubernetes. There's docs If you don't get it running under five minutes, something's gone wrong Please talk to me or talk to somebody in pound Coral s on free node You've already done that. Thanks everyone for the really helpful guestbook entries I'll tell my parents about all the nice things you wrote Yeah Just want to emphasize all this Kubernetes stuff even though it's about huge distributed systems that scales down to single systems Just fine it's really about You know when you move into productions with this stuff, you want to have the right Mentality you want to have the right infrastructure and even if you're running on one machine if you get it one running on one machine It's designed to scale out to dozens hundreds of thousands We are hiring Particularly in San Francisco in New York and most recently in Berlin If you're interested in this stuff, we're looking for software engineers and infrastructure people people who like marketing and writing docs ux design etc and Last of all, thank you so much for your attention. It's been a pleasure speaking for you. Thanks