 It's going to be a happy day. Since I'm doing my whole, I'm using ASCII art for the whole presentation. Sadly, I'm not doing any ASCII art. I should have done it next time. So I'll let the room fill a little bit more. How many of you were here for the last meeting? Very few. OK, good. Just fine. Google was just presenting hybrid control planes. And in some ways, that's the peanut butter to this chocolate. But we can get into that in Q&A if we have time. And I refuse to do kube control. I'm going to be a kube cutler until I die. Sorry, Grant. So I apologize for that in advance. For those of you who have switched religions today. And we have a whole bunch to do. How many people have used kube ADM, kube admin? OK, good. I'm going to try and whip through things. And I need to know how much explanation I need to do as we go. And I will start. Hello, my name is Rob Hirschfeld. I am CEO and co-founder of a company called Rack N. I've been involved in the Kubernetes community since the very, very early days, the Oskan announcement. And today we're going to talk about some really exciting new stuff. Rack N focuses on operations, specifically on premises, physical, metal operations. That's our area. We do a lot of other things too. But that's really our focus area. Today we want to talk about something that we think is really exciting, which is immutable bootstrapping. And then we're going to talk about some other places where operations and Kubernetes meet in interesting ways, which is noted mission and dynamic kubelet configuration. If you haven't heard of those things, we'll get there. If you're interested in community side, I'm involved in cluster ops sig. So we like to try and round up operators. Please get involved in that. And then a lot of what I'm going to show you, we're going to start with something called digital rebar, which is an open source project that Rack N maintains for doing physical provisioning. So the bare metal underlay side of this whole equation. And before I get into the immutable boot, it's worth doing a hat tip to kubespray. We've done a lot of work building kubespray deployments since version 1.2 when it was kubespray the first time, and then cargo, and then back to kubespray. And it's a really solid Ansible playbook. It has great features like HA and plugins and things like that. So we're not trying to endorse one installer over another. We actually think if you like Ansible, kubespray is a really good job. But this presentation has a lot of these pregnant butts in it. We think we can do better than what kubespray does in certain circumstances, especially for turning over nodes and reprovisioning immutably. Because we think that we can bring up Kubernetes without central orchestration, so without an external system telling all the nodes what to do without having to build the inventory in advance. We want the nodes just to come up, boot, and be discovered. That would be really nice. Without having SSH involved, so if you're doing immutable infrastructure, I'm going to pause and not assume the room knows what I mean by immutable infrastructure. Actually, I'll define it, I think, in the next slide. Immutable infrastructure. And then we just think it should be much faster. Kubespray is really cool, but it takes 20 minutes or so to run. We'd like to get that down in the sub-5 minutes range. Fast enough for me to demo in a presentation window. So what is immutable infrastructure? How many people know what immutable infrastructure? That term means something to them. How many people think they agree with my definition of immutability? That's about, I know you don't. That's a lie. So immutability has different meanings for different people. To me, the fundamental architectural piece of it is this create, destroy, repeat pattern. So if you're doing cloud deployments, you create a cloud and image. Some people would say AMI, but you create a golden image for a system. You deploy that image, usually bake in a whole bunch of great stuff into that so that by the time the image comes up, you don't have to do much configuration or very limited configuration. And then when you're done, you destroy that VM and you create a new VM with the new, latest, greatest stuff in it. So we're switching from a configuration pattern, which is really what Ansible drives, to an immutable pattern where the image is pre-created with everything in it. You do much less. And then you never patch or fix. You always destroy and replace. We work on doing that on physical infrastructure. So we're literally turning over nodes very quickly. I'll show you two ways that we do that. And the reason we want to do this pattern is it's one, two. The three is faster. But it's a very repeatable and predictable pattern. So that means I can say, this is the thing I want to deploy, and you deploy it. Configuration management tools are really good, but they can be finicky. Especially over time, configuration drift happens and it's very hard to maintain nodes. And that makes everything simpler. And so in operations, if we can increase simplicity and speed, that's a big win. And that's why we think immutability is a very big deal. There's a side effect in here that we think is really important, which is people know how to do it in the cloud. So if your automation works in the cloud and we can bring it back on premises, then that's a big win. Because then you've reduced your tool chains and you've reduced your test differences and you've increased the fidelity within your operational environments. So here's what we want to show. This is the zero configuration. What we want to be able to do is do a three-step cluster initialization. This is normal Kube Admin. If you haven't done it, we just don't want to have to touch anything to make this happen. So we want to initialize the master. We need to retrieve that token from the initialization. And then we need to join nodes with that token. So this is normal. If you've done it, you log into one node, you do things, you log into the other nodes with information. We want to eliminate that step because that still requires some coordination and synchronization manually. If we get rid of that, then clusters are going to come up automatically. If you were in the last demo, if somebody is hosting your control plane already, then you can do basically the skip the initialization step and use a remote control plane. But first, you can't just go in and Kube Admin out of nothing. You have to have some preparation done. So you have to have an operating system installed. You have to attach the disks to the system, so you have some persistent storage. Potentially, we could do everything in memory, but it makes sense to have some persistence for the Docker volumes to go somewhere instead of RAM. And then you have to have Docker installed. All of these, these are your prerequisites, pretty simple, but they need to get done. And then to make the actual bootstrap and go, you need to have some control mechanism. So it's Ansible, or Cloud in it, or Terraform, or what we've been able to do is eliminate those pieces and just use the boot provision infrastructure we build with Digital Rebar. And it looks like this. So we have four nodes in the system. We have to prep them. So we have to install an operating system and install Docker, pretty straightforward. And then we need to let the Admin node, the master node, come up. So it has to be a Kubernetes and create a cluster token. So that token then is going to be used to join the other system. So once that token's available, all the other systems are going to come in and join it. And then at that point, we can just apply rinse, repeat. We can grow the cluster. We could drain and remove nodes from the cluster. Things become pretty straightforward at that point. So this is the basic bootstrapping process. What we want to be able to do, what we want to be able to do is synchronize that process. So we want the nodes to come up just out of as a group. The first one up should be able to grab being master, create the token, and then let the other nodes go. And that's what we'll show you in the demo. And that'll be able to bootstrap the cluster completely by itself. But before I demo, I do want to give a PSA. This is not a new installer. We believe very strongly that we don't want distro wars. We don't want installer wars in the Kubernetes community. So this is using Kube Admin. Using Kube Admin primitives. And it's using Kube Admin commands. There's no magic custom proprietary widget in this stuff. It's just using the tools the way they are. Just like when we use Kube Spray, we just use Kube Spray. And that's our goal, is to be able to use the community tools the way the community tools are designed to be used. All right, so the demo. All right, so this is a Kube Admin rebar, which is digital rebar, and immutable bootstrapping for the demo. I'll show you what these things look like. This is the overview system. So this is a digital rebar console. It's where we're going to start. And in this system, I have put eight nodes because we're going to do two different deployments that are completely independent. And my job here is not to show you digital rebar if you want to do that. Go to the community site, and you can duplicate all these steps there. What I want to do is focus on the actual bring up. And so for that, what we've done, can people see this in the back? That's why I ask. There we go. What I've done here is we've created two clusters in advance for the demo. One is a RAM cluster, and one is a Kubernetes install cluster. And so in this cluster, it basically just tells you that it is a Kubernetes cluster, or CRIB. My naming. Not everybody in my team loves that, but it works for me. And I've created another cluster install that will actually do it. One of them is going to use our in-memory fast booting infrastructure. The other one is actually going to do a full regular CentOS install in this deployment. And then what we have to do is we have to assign these profiles. So in this case, my first four nodes I've already put into this RAM-based cluster deployment. So they're already booted. They're waiting. They're available for me. And what I need to do is go ahead and just tell it to start the install process. Oh, wait. One thing to show you first. And then somebody can start a clock on the demo. In this deployment, we have a workflow. So we've already discovered and put them in the wait state. We call that image sledgehammer. And then what we're going to do after this is we're going to mount the disks. Step one, we're going to install Docker. Step two, and we're going to do the CRIB install. Step three. And while that works, I'll show you a little bit about the actual scripts that are going on behind the scenes. And then we're done. We're just going to wait. And then I'll run Kubernetes for you. So to launch that script, I just have to start the workflow to the Mount Local Disks phase, which I can do for those four nodes pretty easily, four nodes. And I just need to go to Mount Local Disks. So this is the beginning of that workflow. So I've applied that workflow. You can see the stages are mounted here. Going to get some live events back as the system goes through that process. And that'll let us see what's going on with these nodes. So the first thing that's going to happen for these nodes is that they're going to start going through the process of building the actual infrastructure. I'm looking at. They're going to go through the actual infrastructure phase and start running the install process for the disks. And then we should be able to start moving things all the way through. So while we wait for that to kick off, what I'm going to do is I'm going to take the other four nodes that I have going. And I'm going to add them into the other cluster that I have ready to go. And for them, what we want to do is I need to put them in this Kates install cluster. So this is my second cluster. You can see they're in that one. And then I need to go ahead and set them back at the beginning. Sorry, I want to set them to install Centos. And then I want to let them reboot. And so you'll actually see I can expand this a little bit. We're getting the updates in the machines are going through their cycle. And it looks like, did I hit the wrong ones going through the wrong ways? OK. My CTS sitting in my front row, giving me glares or hints as I need it. And so basically what's going to happen in this case is the system's going to go through and build the workflows and identify the cluster with what's going on. So if I check in on, and what we're waiting to happen is in this profile that we've built, we're going to come back and as part of these, oh, I'll take you through what the scripts look like. That'll help explain how we're using Kube Admin for this. What will happen is that as the cluster comes up, the first node that's elected is going to identify itself as the master and will inject I am the master into this profile, and that'll become a shared resource for all the other clusters. So that's how we coordinate this operation of a single node, all the nodes boot simultaneously. One becomes self-elected as the cluster. We don't pre-define that. It's basically a race, one node wins. And then once that node wins, the Kube Admin initialization step will pull back the required token, and it's going to then pull back join information for all the other nodes to participate in. So let me show you what that looks like. So inside of this stage, the crib install stage here, there's a single task in it. And this is something relatively new for us. A lot of what we typically do is we would start to decompose this single task into multiple tasks, and then those tasks become alerts and information that get passed into the system. So you could actually track the install progress of an individual node and then notify other systems of what was going on. So in this case, what we have is a template. And the template is just the bash that you would use to install this node in the infrastructure. So it's going to go in, and then if I dig through in here, I have to install kubelet. I have to come in and actually run somewhere down in all this code is the actual kubadmin init phase. From there, we're going to grab the results of kubadmin init. All the other workers are going to back off, and then they're going to keep going through the infrastructure from that side. So let me see where we stand with the nodes. And so at this point, I have this node in the Docker install stage. And so literally, this node is now installing Docker. And so I can actually see it from here. All of these nodes are in the Docker install step. One of these four I still don't know which yet. When it advances to the next phase in this workflow, we'll become the master node. And it just takes a little bit of time to install Docker. One of the things that we're doing in the background here is this demo. Obviously, I don't have a stack of servers here doing the demo. We're using something called packet.net. If you haven't tried them, we actually have a coupon code, rackn, capital letters 100. You can get, I think, a $30 credit on packet to try out these same types of things. They're a bare metal hosting company. So if you're practicing, you don't have a lab. Or even if you do have a lab, if you need spare metal servers, we're big fans of what packet's been doing. And we have a lot of integrations with their workflow. So in this case, we're just waiting for the Docker install to finish. And then we'll get Kubernetes install happening right after that in just a minute. And then part of what we're able to do in the background is actually see the job progress. So this is our Docker install actually running, and then giving me results of the Docker bring up process. So task Docker finished. And then we should be able to start the Kubernetes. Let me see. Yeah, so now we're actually doing the Kubernetes install process in this case. And then one of them should win the leader election. Let's see where we go. So in this case, node 56 won the leader election. And so that node is now doing the work of doing the Kube admin init. So if you've installed Kube admin init, that's the first step that you're going to do. And then the information that gets initialized in that cluster will show up in this profile dynamically when we get a little bit further in the stage. So I'm right on the edge. I'm tempted to jump back to the slides, but we're right on the edge of doing it. So what I'm going to do is I'm going to flip back to my slides for a minute. And then I'll show you, that way we'll preserve a little bit of time, and I'll be able to show you the cluster actually being built. So in this case, I just applied a profile to nodes. They're all coming up in a Kubernetes cluster. I'm actually building two Kubernetes cluster, one of them in RAM, basically in a RAM boot system. And then the other one is going through a full CentOS install, and then it'll lay down Kube admin using exactly the same scripts. But there's a little bit more to do with this. Adding nodes requires a token. With Kube admin, we could create a long-lived token, but that's not as cool as it would be to actually inject the token and bring it back. Our KubeLit still requires configuration. So if we want to make things sort of different or interesting, or you have site variation, we're going to end up having to inject configuration into the KubeLit. And then it would be really nice to be able to orchestrate this upgrade and run it using some cluster API. So there's plenty more bootstrapping things to do beyond where this goes. The fact that I can bootstrap without having any external things about the system is really handy. So I can just bring up an admin cluster and let it go. So let me see where we are with the Kubernetes install at this point. So in this case, what you'll see is, and this has been really good because I've been able to show you sort of the intermediate steps, we started with just the cluster, and now we actually have the join commands here. Those join commands are then being passed in automatically to the rest of the nodes in the cluster. And then what we also have here is we actually have the full admin conf file that we need to run the cluster. And this is where the demo gets really fun to me. So what we've been able to do, because I now have the admin conf file, I'm just scrolling up instead of typing. There it is. What I'm able to do is using the rebar CLI, I can pull that file out of the configuration values. So in this case, I'm specifying the cluster that I want to talk to. So this is the Kubernetes cluster that's running in RAM. And then I'm just pulling out the admin conf file. So at this point, that same parameter that I showed you is now in my local system, which means I can start running. This is on my local system. The cluster is running in a data center in New York. I can actually go in and use that Kubernetes control admin file to get the nodes that I just deployed. So in the course of this little demo, I've actually managed to bring up a full Kubernetes cluster. I'm actually bringing up two of them in parallel. And then what gets really fun from there. So Kubernetes is up. What we actually want to be able to do, and even more importantly, I didn't have to figure out what my credentials were. I didn't have to figure out my login. I actually was able to create a unique credential set within this infrastructure, boot up the nodes, and then without knowing anything about that cluster, log into it and start getting access to it from my local desktop. But I want to do a little bit better, and KubeCuttle lets me do that. So with these credentials, I can actually set up the proxy. And so when I set up the proxy, now I have access through that proxy into that cluster that I'm using with these credentials. And so if I come back, this is letting me into my Kube admin dashboard. So five minutes, KubeCuttle, no typing, no logging into the machines, no SSH required. I have a cluster that I can log into and start using. It's a pretty big deal. I think it's a big deal. I'm excited. All right, so that makes sense. That's the bootstrapping demo. If you're trying to deploy this type of thing on site, or if you're trying to extend on-premises infrastructure from what Google was showing you with a hybrid cloud, you could eject the parameters, boot the nodes, and it would be done. Even faster, you wouldn't have to do the initialization phase. All right, so back to slides. So there's two other things that we said we were going to talk about that we've been hearing a lot of interesting things for, can we make this faster? Can we make this more secure? One of them is noted mission. Noted mission is a pattern. And in the presentation, if you're in schedule, you can get a link back to these docs and the URLs. Ideally, what we would get with noted mission is that when we booted nodes, we wouldn't need a token. The node would boot. We'd say, hey, I like this node, let it in, and then configure it. That's what we want to be able to do. So we don't have to drive any configuration infrastructure. Should make auto-scaling easier. So noted mission, node comes up. Something says, yes, I have a picture I'll show you. And that should make it faster. We should be able to use Kubernetes to do everything. Sort of a Kubernetes conquers the world paradigm. And it would look like this, using this admission service. So a node would come in. Kublet comes in. It tries to create itself, tries to create the node. That create node action, using the admission protocols, says, I can't create the node until I check in with external service, whatever, webhook. That webhook says, yes, I trust what you're creating a node that you're allowed to do. You could do this with any object in Kubernetes. It validates that the node is a valid node, and says, yes. Kubernetes says, what a relief. I trust this node. It sends back the, it creates the object, and now Kublet's in the system. That's object admission, node admission protocol. And this is a little bit scary. So this is the pattern that somebody described to me, and they started to get very excited about. HSM means a hardware signing module. So you could cryptographically sign the information that you put in that node, and you could start the node with a token from a trusted third party, sign that node by the hardware signing module on that node. So it's unique to that node. And then the encrypted token is what gets passed up. This is tricky to follow. It gets the encrypted token, not the clear token, gets passed up to the API server. The API server forwards it as part of the node admission request. The external party goes back to the PKI to decrypt that token using the public key, and that that system says, wow, this matches the key that I gave you. Nobody else knew about it except me, and everybody's happy in this process. And it sounds really cool. The challenge is that in bringing up the node this way, you already have to inject information that is cryptographically relevant or unique per node. The level of extra effort to do this really doesn't guarantee you that the node is any more secure. So while it's a really interesting pattern, we at Rackin think that this really doesn't provide any additional security from a node authorization, node guarantee space. So we think it's interesting, but you've already got to be pretty sure you're bringing up a node and validating what the node is by putting a secure token on it, an admission token. So marginal benefits for a lot of complexity. The other pattern that looks like it should be super interesting for zero touch configuration is to use Kubelet dynamic configuration. So you want to be able to say, hey, Kubelet, here's a new node. I trust it. We already went through this process. And I don't know anything about it. You tell me everything I know to do that configuration. And that seems really cool, because Kubernetes already has to interact with the node. Why not let it configure the node, too? And there's a line of thinking along these lines, which I call the bootstrap fallacy, which is, if I have a tool, I should have the tool, bootstrap the tool. We've made this mistake before. It's sort of tricky. I'm hinting at where I'm going with this. But you'll see, it should look like this. We centrally configure the system. We install Kubelet. We allow Kubelet to register, and Kubernetes configures the Kubelet. But it looks like this in practice. And I hope we're going to continue to see investment. This is still an early feature, so it might get more like slide one, but right now it's like slide two. You install the Kubelet. You configure the Kubelet. It registers to the API server. You inject your configuration into the API server. And then you reconfigure the Kubelet to use dynamic configuration, so it gets configuration from the server. So you literally have to configure, configure, reconfigure, configure, which isn't really that productive considering you still have to configure. This is the bootstrapping problem. So if I'm injecting configuration into the system, I might as well just inject all the configuration into the system and do it once. And so from our perspective, since we already have to inject configuration and we already have an external system that's managing that configuration, or at least aware of that configuration, we're not really saving anything by taking 50% or 75% of a configuration template and putting it somewhere else. Matter of fact, we might actually be making things harder, and we start having multiple sources of truth and things like that. So it's interesting, but we really don't see that we've accomplished as much because we haven't eliminated the configuration stages for the node. So it's something to keep on your radar from that perspective. But overall, we're really making great progress. The fact that we're able to use kube-admin to bootstrap a cluster this easily, to be able to just let this sort of three-step process automatically configure a cluster is really big deal usability infrastructure. And it's the right direction. So I have to improve, but this is a real step forward in how we're going to build and manage clusters. So thank you for your time and attention. I appreciate that. I have time for questions. The profiles are available. So everything I'm showing you is available. The crib stuff literally, we finished it with lots of time to spare, meaning Monday instead of Wednesday. But so it is available in the registry. So in the digital rebar UX pieces, I will show you this, and I tried to not talk about rebar too much, basically there's a content package for crib, and the content package is available in the catalog for digital rebar. So I don't know which repo we ended up putting it in. But you can repeat this whole infrastructure. We're doing a longer demo of this with a longer discussion on Thursday as a webinar, so you're happy to participate. But this is the type of thing we really do want to see this extended, decomposed. There's a lot of places to go with this. It's just content from our perspective. Another question. Great question. So the question is, what was the state of the sledgehammer machines before? They were pixie booted into an in-memory image, which is based on centos. So we have an optimized in-memory boot environment. And so yeah, so it's literally the default process, the default workflows here, workflow. So in this workflow, discover is a pixie boot, and then it literally just pixie. It adds SSH keys and inventories, and then it waits. So that's the minimal state. It's an optimized boot process for that perspective. So very easy to replicate. Two questions. Go on the right first. So the question is, what do we think of Linux Kit? We did a demo with Linux Kit a while ago. It was pretty nice. We were able to inject some images and makes Linux Kit go. And it's still flooding around in the community, I think, somewhere. We're just not seeing a lot of uptake on it. And so in the community, it's one of those things we'd be happy to see more people trying to use it. Same as true with CoreOS, what we're doing is generic. So you could easily be running CoreOS through some of these processes. Although CoreOS is a little bit different. Some of what we do relies on our post-provisioning agent, which is dissolvable. So it doesn't stay in the system. But our post-provisioning agent will pull down work out of the system. So it's a question of how you inject that post-provisioning step into those immutable images. That's the balance. So Ignition is actually built into the components. I'm looking at Greg's. So we're actually adding Ignition into the Sledgehammer post-config. So you could actually run Ignition scripts in this in-memory boot state and then drive directly to the disk. We're really excited about all those things. We think it's the right pattern. And you had a question. So the question is, can you accelerate this by baking more things into the image? Yes, hell yes. What we did, and this was a very deliberate decision, Sledgehammer, which is our sort of quick start boot discovery thing, is really optimized. So we didn't want to add this into Sledgehammer. But you could create, and this was a subject of some debate before I was just told no. That's the way things work. But we almost made a case Sledgehammer or a Kledgehammer or whatever, Google Hammer or something that would have had these things in it. But the reality is it's so fast to pull these images, it's really not that much overhead. For Docker and Cube, but if you had a whole bunch of other pieces, then I would totally do that. Makes sense. More questions, just about out of time. We're happy to take more questions up here. Rackin does have a booth down on the floor by where the swag stand is. So please come by, talk to the engineers who build this stuff, not the guy who has to do the demos. So we're all here. And they're easily identifiable by their choice of clothing today, kilts. They're teams in kilts today. Excellent. Thank you, everybody. Appreciate the time.