 All right, thanks. And I really should say it's Kubernetes. It's not Google Kubernetes at this point. No, that's not a correct answer. I want to make it clear that we're an open source project. We have actually contributed at this point the project to a foundation under the auspices of the Linux foundation. So it's really out there for everyone. There's a hosted Google version called Google Container Engine that we like to think is a great option for a lot of people out there, but it's really an independent project. So I am Brendan. I've been an engineer at Google for, it'll be eight years at the end of June. So it's been a long time. I started in search infrastructure and spent a long time building the back ends for search. So if you've got something out of Google that's under about 12 hours old, it came out of software that I wrote. And then I transitioned to the Google Cloud Platform. And what I really got excited about there was bringing a lot of the tools and techniques and approaches that we use to make search so reliable that for many people, Google is a test of whether the internet is working to everyone. And so I'm going to try to talk about how our philosophy about that and how I think that systems like Kubernetes change operations. All right. So this is my motivation slide. I've used this a lot. So if you've been to another talk I've given, I apologize. You've seen this slide before. But I find that it's really emblematic in my respects about why the job of operations can be really hard. Because almost no one pushes bad software, right? I don't think anybody knowingly goes, you know what, this is just not going to work. I'm going to push that out anyway. So what happens is actually that it's the coupling that's the problem. There's unintended consequences of whatever it is that you thought you did. I mean, sometimes you just make a flat out mistake and it's a typo, but hopefully you have automation that catches that kind of stuff. More often it's some sort of like, oh, we didn't realize that making that operation twice as slow is going to crash this other system over there. And so this is the point. Most outages are self-inflicted, but most outages are also accidental. And if you think about why this is, it gets back to that tractor trailer, right? It's this coupled system where the thing that is providing the force is on this hinge point to the thing that's trying to move, which is the trailer. And it's kind of unintuitive, right? I actually just spent the weekend trying to help someone, not all of the weekend, that would be really extreme, but trying to help someone back down a boat ramp. And we actually ended up stopping and just hand-trailering the trailer, disconnecting the trailer and hand-trailering the trailer down, right? It's hard. This coupling makes things really, really hard and there's unintended consequences about what happens. And so I think if there's one theme that I like to put out there, and this will sort of hit on in a few places elsewhere about how physical isolation, this morning, physical isolation of teams can actually build barriers and stuff like that, I wanna talk about how we can build systems so that we can decouple our operations into a bunch of one-dimensional problems instead of these coupled two-dimensional problems. And as a result, make our lives easier, right? Job is just too hard. All right, so we're gonna try and break that coupling. And I think that this is the breakdown that we have internally at Google. And I think this is the breakdown that I think that we're trying to put forth to the world in general. And I wanna say each of these layers is essentially decoupled from one another, right? We have hardware SRE who are basically responsible for the physical machines, imaging machines, things like that. They're the people who are in the data center. If you're using an IAS provider like Google Compute Engine or any of the other or some sort of virtualization provider, you may have actually already opted out of this, right? Your hardware SRE already may be someone who is totally decoupled from you because you're using an infrastructure as a service company. And then the next level above that is Kernel or OS SRE. And I think that one of the things that containers have really, well, one of the things that containers did for Google a long time ago and are doing for people now is enabling us to break the dependency between the operating system and the application, right? I can now suddenly say, I don't need the OS people to install Debian or Ubuntu or Fedora or whatever the OS is that I need for this particular binary package. Suddenly, there can be one operating system across and one Kernel across the entire fleet. Thousands and thousands and thousands of machines can have one operating system managed by one team, right? That's a huge decoupling and a huge improvement in process for a lot of people. The next level up is cluster SRE. And this is basically the people who are responsible for running a system like Kubernetes or Borg or Mesos or whatever it happens to be, right? They are the people who sort of provide this abstract API that then applications live on top of. And then of course on top, there's application SRE. Sorry, I should say SRE. Is that a familiar term for everyone at this point? Yeah, okay, sorry. That's kind of a Google, I guess it's a Google-ism. So it sounds for site reliability engineer. It's like DevOps before DevOps, I guess. I don't know. There's a book now in O'Reilly though called the SRE book from Google, so you can go and read about some of the ideas there. And so application SRE, there are people who are actually, I think this is actually the first place in some places where DevOps actually occurs. I mean, honestly, I don't necessarily even know what the word means, but I think that what it means is that the people doing operations have deep knowledge of the applications that they're operating or that the developers themselves actually are doing operations. And I think that that's what this decoupling allows you to do, right, because if you can focus on an application and you don't even think about kernels and you don't even think about hardware and you don't even think about really the cluster API, suddenly you have time and cycles to really focus on the application that you're running and the application that you're responsible for. And it turns out actually, so that's what we say here, like I think this is DevOps. I should remember to move my slides forward. And it turns out actually cluster SRE is DevOps too because the cluster is an application, it's just this special application that everybody else in your infrastructure uses. But it's just an application at some level at the end of the day. All right, and so yes, I really did talk through my slides before I showed them. And yeah, all right, I'm just gonna skip ahead to catch myself back up. All right, so what I think this is the thing that cluster managers like Kubernetes enable you to do is that basically everything below that cluster layer, it just vanishes, right? Just like with IAS, you got to make hardware disappear. With containers and a container cluster API where developers are saying things like, I wanna run this image with two cores and 10 gigs of RAM and the cluster itself is responsible for figuring out where to place it, on what machine, keeping it alive and all this kinds of thing. They can forget about operating system. They can forget about kernel. They can forget about everything below the application that they're managing, right? And that's really the power of this higher level API, this higher level abstraction that is container oriented instead of being machine oriented. And really what this allows you to do is that your machine fleet is homogenous because your developers no longer care about machines. And it becomes actually this just giant C of compute because you have a scheduler that is responsible for figuring out where to run things. And so I no longer need specific machines. I no longer need to tell my application to run on specific machines. I just express the general requirements of my application and I trust that this API, this container oriented API is going to be able to schedule it out. So this is the stuff below this line. It's the not my monkey, not my circus line, right? So this is like application owners don't actually have to care about what's going on with the operating system. They expect that you'll make the operating, the people who are operating who are responsible for the operating system, they expect that they'll do a good job and they'll do the right thing and it'll work. But they have no other expectations, right? They have no other dependencies. This is the decoupling that we're talking about. And it turns out that it works both ways. And so it turns out that for the people who are running the operating system, the line protects them from the application, right? I don't have to care what you're running on my system. I provide everything up to this line and it's your responsibility to provide a working application on top of that, right? So again, we're decoupling these systems so that we can move independently so that we don't break each other, right? So that people who roll out a new kernel don't have to worry that they're going to be breaking some or more often rolling out a new operating system with new LibCs and new Python maybe that those people don't have to worry that they're gonna break some application because that application is no longer dependent on the things that those people are rolling out. This decoupling is really important for reliability. All right, so applications are gonna use resources, not machines, right? Every application just says what it needs. We scale in terms of resources. We don't scale in terms of machines. We just ask for more. Also important is the fact that this is an API, right? And so suddenly you can introspect via an API, you can automate via an API, things that are applications, right? If you're going in and you are running, heaven forbid, you're running manual commands via SSH, but even if you're running scripts inside that machine, even if you're running some of these automation services, it's very, very difficult to get an application-oriented view from automation, right? It's very difficult to write an audit log that can go across and figure out what versions of what software are running where. And that's because the APIs just don't exist, right? The APIs, even if you're in the cloud, if you're not in the cloud, then there aren't even APIs for your machines. If you're in the cloud, there's APIs for the machine, but not for the application. But with containers and with container management APIs like Kubernetes, there is actually an API for the application, and I can go and walk up to the API and I can say how much memory is that my SQL instance using? How much CPU is my PHP script running, right? That's hugely powerful in terms of the kinds of systems that you can build on top, right? Suddenly, I can build a system that goes across my entire infrastructure, and for every team in every group, aggregates the amount of CPU that their particular applications are using. And by the way, I can difference it against the amount of CPU that they're requesting, and I can go up to teams and say like, you know what? You asked for three times as much CPU as you ever use, even at peak, that's probably not okay, right? This is a huge step above what normally you can achieve. Everything effectively that you're doing becomes application oriented. I said this to someone yesterday, but one of the really important transitions that I think that containers and container management APIs are doing and are going to have, one of the reasons why I think they effectively are taking over the world is that the primary key for every single thing in your data center, which used to be the machine, is now the application. Logs are keyed in terms of application. Monitoring information is keyed in application. Your API is in terms of your application, right? That's what you were thinking about anyway. You were thinking about your application, and so why should you have to think about what machine, start with a machine and go find logs on that machine? Why should you have to think about health in terms of machine health? It's just not something that is relevant to your life rolling out software, which is in the end what the job is about. All right, and so this is the real transition that I think is going on right now out there in the world, and the real reason why people are getting excited around container oriented infrastructure is that the primary key of the data center is just shifting, and it's better for everyone. Because if I'm owning a machine, I can focus on machines and not even think about applications. There I want the primary key to be the machine because that's the thing that I manage. If I'm managing the machine hardware or the operating system, I want it to be. Then I decouple myself from those details when I start rolling out applications and for everything else, above that line of decoupling, the primary key is the application, which is the thing that you cared about anyway. And trust me, you don't want me rolling out kernels, right? And I don't particularly want kernel people touching my application. All right, so your data center is now application oriented. And now, hopefully that sounds a little bit cool, but it's a little bit vague. I sort of skimmed along the top and told you how the world was awesome, and I didn't really talk about why this was actually something that could make your life easier. And so I wanted to start with some of those sort of practical, actual things that these application oriented cluster APIs do for you. As an operator of an application. And the very first thing that they do is they keep your application alive, right? So we all, if we've been out there and we've done reliable software for a while, we know that there is, you need something to watch the system that you're running. You need something that software crashes, right? Software crashes all the time. There's lots of daemons out there that will restart your software after it crashes. But they are not really very well integrated. You tend to have to make decisions. Am I gonna do system D? Am I gonna use supervisor D? Am I gonna, like, what is the way that I'm gonna do this? And they tend not to have APIs, right? So I can't walk up to something and say like, give me the place where there's 20 restarts. Give me the application that has had 20 restarts in the last hour, right? Or in another way, is there a difference between how many tasks are restarting on this machine versus tasks that are restarting on this other machine? Maybe that machine is sick because everything that's on that machine is restarting constantly. And so what Kubernetes and what other systems do for you, other systems like this do for you is they allow you to put into the API application-oriented health checks, right? So I can actually, when I deploy something into this cluster, when I deploy a container into this cluster, I can say, you know what? I want you to hit this URL. Just go and hit this URL. If it returns a 200, then the application's healthy and it's good. If it returns anything else, restart it. Right? And so instead of me worrying about installing and figuring out how to configure system D, instead of me worrying about what particular supervisor and how I'm gonna get the information back, just the act of deploying into this cluster manager provides me with this attribute. And I think that's one of the themes you'll see. Not only does it decouple systems from one another, but it actually has this sort of generic set of primitives that you get for free just by deploying into this API. Just like with infrastructure, you expect that if your machine fails, you know, sorry, infrastructure as a service, cloud infrastructure as a service, if that machine fails, your machine should come back someplace else, right? It's taken over health-checking your machine. Now by moving up to this application-oriented layer, the API is taking care of health-checking and maintaining your application. So the other thing that it allows you to do is it's actually health-checking your machine too. Sorry, that's not actually coming through very well. There's a really light gray box around each of those. So this is three machines in a Kubernetes cluster, right? I have my yellow and green jobs running on one machine and two red jobs running on these other two machines. All right, so bad things happen in data centers or infrastructure as a service and machines can fail, all right? So what happens normally when machines fail? Well, if you're lucky, you page, right? If you're lucky, you are checking how many instances of your web server you have, you see that a machine's failed, you page and somebody comes back. But if you've been decoupled in this way, if you've stopped caring about machines and all you care about is applications and you have a piece of infrastructure that's responsible for scheduling your application, what you will find is that the infrastructure itself actually notices that that machine has failed and it actually moves your jobs away from that machine. Because you're decoupled from the machine, you don't actually care. You just said, I want three CPUs. There's three CPUs available on that machine and it gets scheduled there, right? So you don't get woken up just because some machine failed. And it turns out, kernel or hardware or any kind of machine-oriented upgrade, that's just a special case of a machine failure, right? From the perspective of your application and you can do things like, if you want to be a little bit more proactive, you can drain a machine out. You can sort of simulate a failure and drain the machine out slowly instead of effectively just like slicing everything off. But it doesn't really matter from the perspective of the application, right? From the perspective of the application, whether you restarted that machine because you wanted to upgrade the kernel or whether that machine simply disappeared because the hard drive died, the system reacts in the same way and your application continues to run. And so what this means is that if I'm a kernel operator or I'm an operating system operator, I don't have to talk to any of my application owners before I run an operating system upgrade. I just run it. I just go through one machine at a time and I have an SLA, right? Like any other good decoupling layer, like any other good API. I have an SLA and I say things like, when I run a kernel upgrade, I'm not gonna restart a machine more than once every 20 minutes, something like that, right? I have a policy around how I do these machine upgrades, but I don't talk to anybody. I don't tell anybody like, hey, we're gonna come through and upgrade the kernels. I just do it, right? And what this means is my fleet is almost entirely homogenous, which is a huge improvement in terms of maintainability and also in terms of, frankly, team size, right? And specialization, right? I can have a team of eight that is specialized on one particular operating system that doesn't even know anything about any other operating system that runs fleet of machines that is hundreds of thousands of machines, right? That's a huge win. All right. So there's something else that this does if you're thinking about machines and the thing else that it does is that if you've ever planned machine resources or any resource really, network, machines, CPU, whatever it happens to be, you know this problem, which is you go to the teams that are responsible and using your hardware and you say to them, so what are you gonna do? What's gonna happen in the next six months? What's gonna happen in the next year? And they say, team one says, we're gonna double our users. It's gonna be awesome, right? And team two says, we're gonna double our users. And team three says, totally gonna double our users. And you know for a fact that there are probably someone in there either lying or mistaken or something bad is gonna happen and it's not gonna happen that every single team is gonna double their use. But if you're in a world where applications are tied to machines, where applications aren't decoupled from machines, you really have no choice but to say, great, we're gonna 3x our machines, right? Because we have no ability, like team two, they want Fedora and team one, well, they want Ubuntu and team three, maybe they're on Windows. I don't know. Hopefully not. But we have no choice, right? We have no choice but to do this because we just don't know, right? But if we decouple and we abstract and we say like, well, the things that these teams ask for, they don't actually ask for machines. They ask for a container with a bunch of resources. We can actually look back historically and aggregate that information and say, you know what? Historically, every single team says they're gonna double their users. And historically, only one team out of three ever actually doubles their users. And so we're actually only gonna increase our machines by a third. And we're not gonna care whether it's team one that doubled or team two that doubled or team three that doubled because in the end, all they cared about was CPU and so we have the CPU necessary for whoever happens or maybe nobody doubles. Maybe everybody goes up by 10%. It just doesn't matter, right? Because I've decoupled my application from the machines, my planning process, I get to aggregate all your usage together. And I get to buy machines without having to worry in an image machines. And I can just basically look at my usage day to day, week to week, and say like, oh, you know what? I'm at 80% of my machine fleet capacity. I should probably add some more machines. And I don't have to say, well, are they gonna be Fedora machines or Ubuntu machines? It's like, no, they're just machines because everybody just consumes machines, right? So that ability to aggregate usage across all of your teams also is really, really important. All right. So what about application ops? I've talked a lot about hardware and I've talked a lot about sort of the system itself and how it makes managing machines easier. I wanted to talk a little bit and give some demonstrations of how this actually makes it easier for people who are running applications as well. And so that's where I'm gonna jump over for the moment. I'm gonna cross my fingers. I have some backup slides that go through in slideware terms, but hopefully the live demo will work. It is Friday the 13th, someone pointed out to me, which I hadn't, is that really true? Yeah, it hadn't occurred to me, so, you know. So I have right here, hopefully everybody can at least somewhat see it. I have a little Kubernetes cluster here that's running on a stack of Raspberry Pis. It was kind of fun to build. There's a couple of blog posts out there. A bunch of people have done this actually. It's kind of become the hello world of distributed computing. So if you're interested, you should try it out. And so hopefully we will be able to see a demo with that. I guess I'll hold it here with one screen as I move things across. This whole splits. I'm also very used to having video mirroring when I do my demo. I get to type. Wow, this is gonna be really super exciting. I get to type while looking at the screen. All right, so I'm gonna move this. Actually, maybe I'll just turn on this. Yeah, I blame, I really, I'm blaming people who are not me. Um. Okay, so what we have here is a visualization of our cluster. What you can see here is I have the master with Kubernetes. I'm gonna hit reload affairs. I have the master, which is the node says Kubernetes, and then I have node one, node two, node three. I have this thing called a service, which is basically a load balancer. I have this replication controller, which is an object that's responsible for making sure that there are two of any one thing. And then I have two instances of my HTTP application that are running there. So that's all happening right there. And so now, I'll give you the same view. I can say cube, control, get nodes. Is that visible? Can people sort of see that, is there any to zoom in? If I can do that actually with this SSH client. Oh, that's probably true. I'm hitting control shift plus, but there we go. I'm too used to the OS X terminal where it's control shift plus, not control plus. All right, so there's the same picture. I've got my nodes there already. And I can actually say get pods. And there's my two services running. Now, the first thing I'm gonna do is I'm actually just gonna delete one of those. Actually, I'm gonna say dash O wide. And you can actually see, okay, one's running on node three and one's running on node two. So I'm gonna go ahead and kill one of those. So I can say cube, control, pods. Can you briefly explain what a node is? Sorry, sure, yes, yes, sorry. A node is a machine. A pod is an application. A pod is actually one or more containers running, but you can just think of it as an app, as a particular instance of an application. Yes, I apologize for that. So, yeah, a node is one of the, one of these boards is a node. Okay, so I'm gonna kill one of these things. If I could actually. Anyone who learned how to touch type, that's not me. H, D, T, U, W, W, T, three. Okay, so let's go back over to the visualization. All right, there we go. All right, so what you'll see here is we have a pod that is actually, the one that is in gray is terminating because I've actually asked the system to kill it. It's not entirely like a failure because a failure, it would just go down. Since I actually asked the system to delete it, there's actually a graceful termination period. So it actually waits 30 seconds. It gives you a signal that you're about to die. And then it waits 30 seconds by default, configurable, to allow you to drain network connections. So the load balancer drains out the network connections. Your application runs for another 30 seconds to allow you to finish whatever it is you're doing. In the meantime, it's in the midst of starting up something. That's the red one there. And eventually, we'll get back to a situation where we have two things running. And so if I go back over to, you can see that I have two instances running. Again, the same HTTP that was running on node two is there. The one that was running on node three is now actually running on node one. So again, I'm decoupled from application from machine and I just don't care. And when I restarted this thing, it just happened to land on node one. Now, that's a little bit interesting, but we are going to do something a little bit, which is we're actually gonna simulate a network failure. We're not gonna simulate it. We're gonna have a real network failure. And so at this point, all of the nodes are health checking with one another, health checking with the master to make sure that they're actually in existence. And so whether it's because the machine failed or there was just a network partition, for whatever reason, the system has noticed that the node... And you can see there that node two has now disappeared. So it now knows that node two is sick. And what's gonna actually happen is not only does it know that node two is sick, it's actually gonna note that that pod that was on node two, well, it's probably not working anymore either. And so eventually it's gonna be just like I deleted that machine. You'll see now, sorry. You'll see now that there's a gray one there. That's the pod that's been set to terminating. And it's actually created in another instance of the pod. So if we go back over to the terminal, there is that the thing that was on node two, well, because the machine failed, it's in the middle of terminating. It's not actually terminated because it actually can't get the signal, right? The master has actually said, hey, you're supposed to die, but because there's a node part, there's a network partition, it actually has no idea. But it will once it comes back online. And then it's moved and created a new instance. In this case, it's actually running on the master. And that's another interesting point, which is that from the perspective of these applications, the thing that is the master that is in charge of the cluster is actually just another source of compute resources. It's not something special. It's just the place that happens to be where the components that make up the API for the cluster are running. And so your jobs, as long as there's enough CPU, your jobs can schedule there as well. So we'll connect the network back in and we'll go back over to our visualization over here. And we're back and we have two things running, right? So I think this really shows that by decoupling the application from the machine, we're enabling the system to have enough flexibility that it really can react very reactively and responsively to a large number of different kinds of failures. I have one minute 17 left. I can end now or I can go through one more example, I don't know, do we wanna run over? All right, maybe we'll go back and run over. All right, so I'm gonna come back over to you. I don't know, is it better to do things like close the presentation and have the demo work or is it better to? Just wanna go through one quick, this is the review in case you missed the beginning. So then, so briefly, there's this concept in Kubernetes called labels. And basically a label allows you to mark different things that are running in your system, front end in production, beaver ends in testing, allows you to query things and say, I want everything that is production. This turns out to be a really powerful feature for operations, and the reason it's a powerful feature for operations is that I can actually do things like quarantine a server. So if I've said I wanna have three things running, I say, the way that I say I want three things running is I say I want three of front end comma production. Now let's suppose in the middle of the night I get paged because some server is deadlocked and it can't be restarted, right? What can I do in this situation? Well, normally what I would do in this situation is I would, if the system itself hasn't already restarted the machine or the task, I'm gonna go in and I'm gonna actually kill that thing myself because we need to get back to our production service, right? We need three of these things. But in Kubernetes what we can do instead is we can actually remove this label. So we can actually say, actually, we're gonna pull off that production label. This has the net effect of moving the container out of that replicated set of things. A new one will be brought back in its place, but what's important is that the old one is still out there in the cluster running. It's just not serving production traffic. And so this means that when I come in the next day or when a developer comes in the next day, they have a quarantine task that they can go in and debug that's live, that's no longer debugging from logs. It's debugging from a live server that they can poke at and hit with requests and everything else like this. It's a way, way more useful thing to have. And yet I'm still able to restore my system to full capacity without having to, without having to terminate applications, right? And so again, I think this is the labels and I don't have time to really go in and do justice to it. But again, what we've done here is we've decoupled the notion of the load balancer from specific machines, from specific applications even. We've tied the load balancer to just this abstract notion of having a set of labels. And by doing that, we enable behaviors that you otherwise couldn't do that enable your operations to become easier. So I will be around later for questions and everything else like that. Please don't hesitate to come up and thank you very much for listening.