 Give it a minute. Continue your conversation. People just coming in, there are plenty of spaces up here in the front. Okay, welcome to the first session of the day on the container track. Elle came to give this talk all the way from San Antonio, Texas, and it's her first time in Los Angeles, so give her a hand. Howdy, y'all. All right, so thank you so much for joining me today in my talk, containers, what you need to know so you know what you need to know. I know that that's a bit of a verbose title when it comes to a conference talk, and I really think that it deserves a little bit of an explanation. This title came from when, a few months ago, it's probably been almost a year now, I decided I was going to learn containers. I was really excited and this is the new hotness, and I got all my courage up and I walked up to the Kubernetes team and I said, hey, guys, I want to learn containers. How do I get started? They were kind enough. They said, all right, Elle, what questions do you have? I just kind of stood there staring at them because I didn't know enough to know what I needed to know. So they gave me the advice that I'm sure a lot of you in this room have given before. They said, Elle, why don't you go read the friendly manual and then come back to us. So a little bit deflated. I went over to my computer and I pulled up the Kubernetes docs and I sat there really wishing I'd taken high school Greek and I had no idea what these docs were saying. It's been a long journey and I'd like to tell you guys that it was an easy one. Don't worry. Containers are so easy to learn. You'll fall right into it. But dear God, would I be lying to you guys? It's a hard road and it's a hard lesson to go and do by yourself. So I thought, why don't I take all the things that I've learned along the way, all the things that my mentors have taught me and give it to you guys so that you guys can have an easier journey into learning containers. If you came here today hoping for that zero to hero talk, you're going to walk out of here a container expert. Just go ahead and go. You're in the wrong room. But if you came here because you are genuinely interested in containers and you're just going like, how do I get started? What's that first step? And once I have that first step, where do I go next? This is the perfect talk for you. I'm not here to teach you everything you need to know about containers. I'm here to give you the roadmap so that you can go and develop your own journey. With that, I guess I should go ahead and introduce myself. My name is Elmar Kaz, and I'm a community architect for Linux Academy, one of the world's biggest e-learning platforms, as well as being a part of the Jupiter Broadcasting Team. This slide is really where I'm supposed to stand up here, you know, pull my shoulders back, look you in the eye and say, this is why I'm important. These are my certifications, the great things that I have done. That's just not me. Somewhere along the line, my mentors told me, El, don't ever talk about yourself. Let your work talk for you. So when people ask me what I do for a living, I say, I'm a professional noob. I love learning new things. I love breaking them and trying to glue them back together and making that, you know, two o'clock in the morning tweet to a friend going like, so the colonel says it's not there anymore, my bad. Yeah, it's happened. But that's why I love Linux, right? Because at some point you just re-kick and you move on with your life. And that's really what I've learned that containers give you the ability to do. You just delete it and you move on. I spend this time telling you this because it's actually important. It's not just talking about myself. If you're going to learn containers, you're going to break things. And they're going to break all kinds of bad. And you're going to feel that it's your fault. You know what? It's not. Containers is a continuously developing project and actually a series of projects. So when you hit a point when you break something, you don't know what to do, reach out to me. More than likely, I've broken it. But one of the cool things about being a community architect is I build communities. Someone that I know is going to know what you did and be able to give you that next step. So whether that's two weeks from now, just reach out. You know what? I'm not going to tell you to go read that friendly manual. So the first resource I want to give you, though, is my website, LOPunk.com. I am not a web designer. Don't judge me based on it. It's basically a place for me to be able to share resources with you. You're going to find the slide deck there. You're going to find access to my GitHub, blog articles. One of the cool parts of my job is I get to give away free training. So when I do container-based training, it'll be on there as well. And you'll be able to email me and everything. These slides are not stagnant. I'm going to give them to scale. They're going to post them. But you know what? When the technology changes or I come back with a better demo, I'll update the slides and you'll be able to have access to it there. So just go ahead and write that down. And I keep saying it seriously, guys. Don't be afraid to reach out to me if you hit a stopping point. All right. So we've killed four minutes. Hopefully this is the last individual coming in because we're going to go ahead and get started. Today's agenda we're going to go over, what is a container? And I know that that seems like the most basic concept to be discussing. But I'm continuously amazed when I talk to people working with Kubernetes and OpenShift and all these container orchestration platforms. And I start talking about the actual container in the application that's being split up into processes. And they have no clue what I'm talking about. So I think it's a very important step. Then we're going to talk about what's the container made up of. We'll go old school and talk about tributed environments. We'll talk about namespaces and C groups. These are critical concepts to understand if you're going to be developing for containers to be able to understand if something goes wrong, your troubleshooting process. Then we'll talk about why containers. When you have access to AWS and Azure and Google Cloud and there's just so many, I think Oracle Cloud now, why are you bothering with containers? I think that's a really important thing to talk about because containers aren't the answer for everything. We're talking about using the best tool for what your actual needs are. Though I say that there are questions at the end, I'll be checking in with you throughout this talk. It's more important for me for you to walk out with an understanding than for me to get through the talk. Like, we can discuss all of this later. They'll be available to you. So let's get started. What's a container? When I first started learning containers, I did what I'm hoping most of you did, and I'm not alone. I went to Google and I typed in what's a container. And I read blog article and blog post and docker docs, and there's something I kept seeing over and over again. This guy. And the story is that we're supposed to think about Linux containers like these shipping containers, these big metal shipping containers that are put on boats, and they go from one side of the ocean to the other, carrying precious cargo. And the more that I learned, the more I thought, man, what a stupid way to represent containers. Like, people build houses out of these things after they're retired from shipping. So the first thing I want you to do is every time that you see this image, replace it with this. And no, this isn't your mama's Tupperware because your mama's Tupperware is sacred. You lose that thing, you are dead. No, this is that cheap little stuff that you go to the grocery store and your lunch meat is in, and eventually you end up finishing your lunch meat, you throw your lunch in it, you put your lunch in your truck, you go to work, trunk, I'm sorry, you're not all from Texas, and you get there and you forget that your lunch is there, you go to lunch with a friend. A week later you pop open your trunk and there's your lunch from a week ago. And you're really, really thankful for that isolated environment because it kept everything else from getting screwed up. That's how I want you to think about containers. Now I know that there's a few of you who are like, all right, oh, cute story, but I'm here for the technical, I need to know what a container really is. So let's turn to what Docker says. Docker tells us that a container is a standard unit of software that packages up all of the code and all of the dependencies, allowing an application to run quickly and reliably from one computing environment to another. Some of you, especially some of my friends who came to support me here that's a good definition, that's really good. A few of you who are new are just like me going, high school Greek, I get it now. But that's why we're here today. So let's take a step back. When I run a computer, I'm pretty much just running processes. If you're really new, a process is a, the word process is a fancy way of saying I told the computer to do something, now I wanted to go do it. It's a task that the computer is performing. Now we're going to speak in kind of, we're not speaking in absolutes, there's always flaws to analogies, but let's say that one of these processes, process three, is a small Python script that I've been working to develop that someone has given me a review, changed the code a bit and gave it back to me. I want to run this, but I'm not 100% sure on what it's going to do when it starts changing system environments. Like it's going to change my global system environment, it could impact other applications that I'm running. But I still need to run the script. Some of you would be like, oh, that's easy, I'll go spin up a virtual machine. I'm not going to spin up a virtual machine that's going to be. Other people are like, oh, you know what, you just upload it through, you know, like SAP, or you do a bash script and you push it up to a cloud server, it's really easy, you can automate it. Really, you're going to go spend resources every time you want to test this application out. No, what you need is you need a sandbox. Now, those of you who've been around for a while know that we used to really use virtual environments, and you can create different security sandboxes, and really all that it is, it's a way to run an executable file and to restrict that software into a different operating system or to a different subset of the operating system, so it doesn't impact your entire, I said impact, can you tell me how I come from security background? Your entire operating system. The way that this started was actually old school, 1979, so container technology is older than I am, and it started with the charoot command. Now, charoot pretty much does what its name implies. It changes root, charoot, well, kind of. It actually changes the apparent root directory. So I'm going to tell you a little story, because that's how I learn, and the names have not been changed for the story, but I promise that nobody is the villain that I'm going to portray them to be. So, when I started at Linux Academy, I started with my mentor, Amy. We were working on learning containers based on OpenStack technology. My background is in OpenStack. And when I was hired, my manager, Terry, said, you know what, L, I really need you to work on containers, like OpenStack is your past, containers are what you're new to act. But it's really hard to give up what you love. So let's say, let's just pretend, this is just an example, that Amy and I had a server, and we spent a lot of time there teaching me how to develop code, how to work with containers. And Terry one day said, you know, L, we hired you to work on containers. I'm going to need access to that server so I can see what it is that you guys are doing. I could, in theory, go in and go into my servers. You see I have my root directory, and I've got your binaries you expect, I could create a charooted environment. That's your charooted directory. It doesn't have to be named charoot. Within it, I go in and I create new home directories. I create a bin, I create Etsy, I could even put Amy's files down there. You see them at the bottom, so I have new home files. And I give Terry access to the box. He can go in and he can see everything that we're working on. And those home directories would be full of interesting information about OpenShift and cota containers, containers on OpenStack. And my real work could be above. Now, every story sometimes needs an arch-villain. So let's say that Terry's like, no, you don't need to work on OpenStack, and he goes in and he deletes everything. All right, I pretend to be upset, but you know what? All my research is safe from above. So this is really when we begin to see the concept of real isolation. Now, I know that that explanation just kind of explains the what, but not the how. So I never said that I'm good at GUIs, so let me see if I can get my demo going here. But it'll give you an explanation to, where's my mouse? So this actually occurs. Now, hopefully it'll let me pause through it and I'll read through it in case you can't see some of it. So the first thing that I'm going to be doing is I go into my computer rabbit hole, because we go on adventures, and the home directory which I'm creating is Wonderland, as expected, right? I'm going to create two directories. One is going to be for bin, my binaries, and one for lib64. This is going to be because when I want to execute commands inside of home Wonderland, I won't have access to the actual system files. I need to bring them in. I am intruded to that environment. I copy over the files from binls to home Wonderland. If you don't know what to copy over, the which command is going to be your friend. I will show you, though, that there is a little bit of variance and a little bit. So I do an LD binls, and what that enables me to do is actually see what libraries I'm going to need in order to make those binaries work. If I'm kind of talking over your head because you're new to Linux, don't worry. I have a homework assignment that I'm going to assign to you that is going to help you be able to fully understand this. So when you go in, and I will show you, I copied binls over into home Wonderland. However, when I did a whichls, you see that it's actually pulling from user binls. This is something that generally isn't the biggest issue, is whether it's using your user binaries or your system binaries. But occasionally, there is some difference in what libraries it's using. So that LDD command is going to be your best friend because it's going to tell you exactly what to pull. You'll notice I was using Ubuntu for this example. At least I think I was young for sure I was. And so I only use bin and lib64. If you're using a CentOS system, you'll actually need to put in binlib64 and lib, but you'll be able to see that all through when it tells you your exact files here. So the next thing I do is I do an LDD for user binls just to show you guys that it is the exact same libraries. Now you're going to go in and individually pull each and every single library that you're going to need. I won't tell you the first time I did this. I made people sit and watch me create a Trudit Apache environment. It took 25 minutes, and I could see people's eyes just twitching every single time I made a typo. So we decided to go with an easier example here. Now I will remind you that these are all privileged commands, so you're going to need that sudo command. So you've seen me pull in libraries. I can create a Trudit environment now, right? Not exactly. That's because when you try to charoute home Wonderland or whatever it is, you can't. You don't have access to a shell. You will literally only have access to the things that you've pulled. So you have to go in and you have to give it access to binbash or another shell if you're using. Now I know a few of you are kind of like, oh my god, this is taking forever. Imagine being a system administrator before containers and having to create these charuted environments. When people are talking about when you build an application, you only pull in the things that you need. This is what they're referring to. You make your script in what people have access to as small as possible. All right, so we've got everything copied over now and I'm able to create my Trudit environment. Oh wait, one last thing I was going to do, I forgot. I need something to be able to show what we're doing. So I create a file called cookie.text and in this example, what else can that cookie be but other than eat me? So we go ahead and we create our Trudit environment and we remember that it's a privileged command. I leave these typos in so you guys can see that they're not perfect. All of us forget what we're doing halfway through. We can do an LS, then we make some typos, we do who am I, I don't have access to it. That's a command that's accessible in every Linux environment. So I try to cat the cookie, that's it. This is a perfect example, well it's perfect because I did it right, but it's a great example of what we mean when we talk about isolated environments and the mindset that you need to have when we talk about building your scripts and building your environments having the least amount of information available to it. So you only have access to what you need. All right, so if you were thinking, okay, but I can't what you're saying Al but I don't really fully understand like you're somebody that really loves reading white papers and man pages because you help develop that image. I want to encourage you to go and read my favorite white paper. And yes, I'm that kind of nerd that I have a favorite white paper. This was written by Bill Cheswick, it's linked to onelopunk.com so you don't have to go look for it. It's a weird endeared and studied. This is one of my favorite people in the world, I follow him religiously and he was one of the first security analyst for Bell Labs. And at the time, he was having to deal with a cracker. It's a term that's not used very frequently but it's basically a hacker with negative intent. And this hacker was just coming in and he was just owning Bell's systems. He was just tearing them up and Cheswick wanted to understand like what did this guy want? He wanted to be able to study him. He wanted to understand the concepts behind Charoot and he created what became the BSD gel command or there was other people that helped develop it I'm not going to leave it on him but he essentially created one of the first punny pots that we've ever used. So this is a great example of how container technology slowly became developed. This was in the 1990s, long before Docker really became its thing, long before Kubernetes even existed. So once you understand that, here's your first homework. I want you to explain to me or to an 8-year-old what the Linux file system hierarchy really is and how it works. If you're old-score, that's easy. I want you to go find your nephew, your niece, your kids, explain it to them and explain it in such a way that they can turn around and explain to you what you said. If you can do that, I promise you containers become easy because that's really what you're dealing with. The next thing is that you go and I want you to be able to have the same explanation to the process tree. But really be able to explain it to someone who's new. Once you can do that, go create the Truded Apache environment. Seriously, I'll even let you go Google it and pull up a doc that lets you follow command-by-command, but understand what you're actually doing, why you're bringing in the files that you're bringing, why you're bringing in the commands that you're actually bringing in. That's really the first step to getting started with containers. Once you understand that, then we can start discussing the isolation a little bit further. And we talk about namespaces and C groups. Now, I wish I had all the time in the world, but really I could spend an hour teaching you about namespaces and C groups. So I'm just going to give you a little bit of blurb on it, and once you understand it, I'll give you additional resources to be able to delve into further. So we have our username space. Our username space is actually kind of a new concept to be brought into Docker containers. So if you go out and you start looking for blog posts and articles, you're going to find a lot that were probably written with probably a year ago to six months ago and think, well, username space isn't used. Like I said, container technology is continuously developing, so now it's actually used. So make sure that you're looking at the dates that these blog articles were posted in. Anything older than six months may no longer be relevant. That's how quickly things are changing. But your username space basically allows you to have users that appear in your containers that are your root user, that are pseudo privileged users that are nobody users on your system. You have your IPC namespace. IPC is really what controls your message queues in system V. The network namespace, pretty self-explanatory, right? It controls your network. Mount namespace is what allows you to have volumes that are only visible to that container and not to other containers. You've got your process ID namespace, which controls ability for you to have privileged processes. So process one actually would be process 5,000 within your system. Your user namespace, which control, I already did that one, sorry, I'm going out of order here. So you have your PID namespace, your network, UTS which is your Unix time-sharing namespace. This is the one that I think people just kind of brush off, but it's really important. It allows you to have your own dedicated host name within the container. So you can actually save. This is my app one container. This means that when you're scripting things out, you're not having to go through and change your script every single time you change your container to reflect that UUID. I know I went through that really quick and it was probably like, I have no idea what you said L. So go to my website. I have a talk called containers. No, not your mom's Tupperware. That is the 100-level version of this talk. And it really does spend time going and educating you through every one of those as well as going through C groups. The big thing that I tell people to remember is namespaces reflect what you actually have access to on the box. So what you can actually see. Whereas C groups reflect what you have access to actually do on the box. That's really the split between them. This is the one and only time that I will tell people to go read the man pages. I don't know who wrote them, but wow, did they spend time on them. They seriously go down and break into what system calls are occurring on the kernel level to explain how these namespaces work. It's a beautiful work of art in my opinion. All right, so in order to kind of give you a little bit of a visual of it, and before I give you your next homework assignment, we're going to do a little bit of a demo on the network namespace. So let's see this. Okay, so I go in and I'm showing you what box I'm using. So we're in a boon-to-base system. I show you the IP table rules for the box. Simple enough, right? I got nothing here. So the next thing that I'm going to do is I'm going to create a new network namespace. So IP net and s add db1. I'm using a database as an example. I go in, I can list out my databases. Now I'm going to show you the contents of var run net and s network namespace. And you'll see that there is a file handler that is created. This is actually where the work that you're doing is going to live. I know these are really quick. I promise you they're available on my site so you can go watch, but we have a lot of information to go through. So I go in and in my network net and s, I exec execute inside of db1 bin bash. Hey, I've got a bash shell. I'm the root user in here, even though I wasn't before. ID table s shows that nothing configured here. So let me go ahead and add a rule. I'm adding accepting traffic for 3306, which is my sql or any db, really. Here you go. I can see it. There it is, right? I'm accepting traffic for my sql. I exit the container. I do my ip table stash l. And my host sys, well, pseudo, remember I'm no longer that root user that I was in the container. And I can see that that rule hasn't affected my sqlable system environment. Why does this matter? We actually use this in open stack a lot because it allows us to have one server that has created all of our instances and we can give access to instant one to the db that's on instant two so that they can actually be able to communicate with one another without having to give access to the outside world a lot. It really does benefit you to understand how this works because it can really help you implement security policies for your application, hands-on learning. Then I want you to create two networks. At this point, you're laying down the networking that you would be using within a container. And once you're able to do that, you'll kind of understand the next topic. And the next topic is why containers. At this point, if you've done all the homework that I've assigned to you, you have now spent the time and you've created your own charooted Apache environment. You've spent the time and you've written your own demo for a process ID and you've set up networking. You've spent the time and you're thinking why in the bad place would I ever do this on myself and you'll realize that this is what system admins were having to do every time they went to work. So why containers? Oops, that slide did not change over there. Sorry, guys. Like I said, I never said I was good at the GUI. All right, let's see if that works. Okay. So why containers? Because they're easier. So we've gone through that whole process and I'm going to click. Come on, we can do it. No. All right. We're going to just do it small. Okay. So let's say I want to use LXC to create everything that I just told you guys about. I do an LXC list. I have no containers. You can tell I'm not cheating. LXC launch and image. I'm going to use CentOS. I'm going to name it MySent. Now, I did not have this. I did not speed it up because I wanted you all to see it. I haven't pulled this image before. It's reaching out to the web and pulling down the image from the LXC repositories. And so here we go. All right. As simple as that, my container is up and running. That took, what, maybe 15 seconds? It seems long when you're watching it. But look, the network's already done. I don't have to sit there and mess with the networking in it. If I do an LXC launch image CentOS just to show you, okay, it's done. Everything I went and assigned for you guys to do is done with you for a matter of, I don't know, five seconds. That one was up and running. The networking wasn't up when I ran the command, but that's okay. The next thing I'm going to do that you cannot see because of how small it is, but I do clear the screen in a little bit. That didn't work. If I do an LXC exec, MySent, which is the name of the container, tell it I do it. And if I, there you go, I can do a cat etsy red hat release so you can see, even though I was running Ubuntu earlier, I am now inside of a CentOS environment. And this is when I realize that it's an old CentOS image that we're going to go old school here. Cool. I have installed HD2PD and I don't have to create a charuto environment. My isolation is already happening because I'm having this happen inside of a container. What more than likely will take you a little bit longer? So I'm using docs that are already written out to show you how to create charuted environments and namespaces can be done in 30 seconds if I'm slow and my connection takes a while. That's why containers became such a big deal. Is system admins, we were tired of doing all the work by hand. All right. So let's go into, there you go. What is it like, why containers? What's in a name? So when you hear the term Linux containers, now a lot of people are really meaning Docker. But I think that there is a real use case for LXC and understanding what LXC is. So LXC stands for the L and X are Linux, the C is for container, not a lot of mystery there. And it's really kind of a supercharged charute. It allows you to isolate applications or more importantly, entire operating system distros within one system. With a lot of the work that I've been doing with the it's okay to be new concept, we tell people how do you get started in Linux. And one of the things people say, go spin up, just re-kick your computer. Go spin up a virtual machine. A lot of times when people say they want to get started in Linux, they're talking about wanting to get started server side, wanting to understand commands. Go spin up a container, it's 15 seconds of time. And if you're doing random images down, there are always use cases. People can do some interesting things. But if you're doing basic security concepts, you're not really going to do anything to your system. So containers, especially LXD, allow you to go in and try Alpine Linux, and do it in the way that it's safe for learning. So one of things is the concept of between having an application container. This is a distinction that isn't always made for people. What I want you to understand is a container like LXD is going to enable you to run that isolated environment, but you're going to have everything you need to have Debian up and running. It's not the same as that charooted environment where I'm only bringing in specific things. That's the biggest difference between LXD and Docker is really how much do you need. If I'm just starting out developing an application and I don't need exactly that, I'm going to be writing it in Python. But what libraries are I'm actually going to be calling. LXD containers are great because it's all going to be available. Once I've got it written, once I know everything that I need, then I can move on to Docker or Kubernetes or wherever you're going to go, not limiting it to those containers. But the concepts all work together. So people say, why do I need to bother with LXD? I've got Docker. One of the things I want to tell you about LXD is that even Docker on their website says we were never written to. We were passed using LXD for their background, but the relationship is very friendly and they understand that there's used cases for both of them. So when it comes to having both LXD and Docker, what Docker gives you is the ability to have portable deployments across machines. LXD is very much rooted with what you have on your system. So if you write your application and you give it over to a friend and he runs it on his environment, if the resources aren't the same, if they aren't able to call upon everything that you had on your box, there's no guarantee that application is going to work. Docker takes that away and says, if you have Docker installed machine A and you have it installed machine B, it's going to work. And if you pay for their support, they're going to be there to make sure that it works. I'm surprised for everything, right? So I'm going to talk about how you can use LXD as an operating system containers and application-based containers. Developing both. People are always like, really, what should I use? When you're first starting out, I tell people to use LXD because you aren't going to know what you need. As you become more comfortable with the environment, expand your knowledge and move on to Docker. As part of being app centric, they also offer the ability to have automatic access to your application. You can create the process of your containers coming up and running. There's no logging in to LXD and installing Apache. All of that can be scripted out inside of your Docker files. And like I said, I'll show you that in just a bit. It allows you to have versioning. Not a lot of developers are working on applications completely by themselves. A lot of us are working in teams. And a lot of times, if something happens, if something happens to you, you would have in GitHub within your Docker files, which can be critical if you're trouble shooting something that's really affecting your uptime. It also enables you to be able to have component reuse. And we'll really explore that when we look at our Docker files. But it means being able to make very small images. So let's say that I have version one of my site. And I make a change that just pulls in maybe a different module for images. I promise there are visuals to this later, so if it's kind of going over your head, you'll be able to see a visual of it. But it's allowing my system and my containers to be able to share file system space so that I'm getting the most out of my resources. That goes with the ability to share. Let's say I write a really cool app and I want it reviewed and I wanted to be able to just give it to the world or I want to be able to give feedback. I just push it up to the Docker hub and say, hey, can you go ahead and upload your file here? I wanted students to come in and skimmy the most fun way to be able to say hello world using a Docker container. And it was really cool because I didn't have to say, okay, then go to this Google form and upload your file here and I'll download it to my system and pray that it runs. No, I just did a Docker pool, the name of their file, and watched it go by. I really wish I would have put some of those in my slides because they came up with some really cool stuff. It's because it's easy. So I do Docker container run dash D is going to enable it to run detached. I'm going to name it Apache because I like Apache. I used to get this out of it and it's what I know. I'm going to say I'm going to pull down that Apache image latest. It wasn't on my system so it knows, hey, she wants this image, I'm going to pull it for her. Then we go down, I do a Docker inspect, which is like a command line based browser. There you go, Apache, up and running and isolated. Everything that would have taken you that long to get to in a matter of, I mean, what was that? Oh, look right here. It's 38 seconds from the time I started talking to the time that I had an isolated web server up and running. So let's talk about Docker because a lot of black magic just happened, I know. So when I did that Docker run and the container image wasn't found, it was the same as running that Docker pull command. And what you see happening here are a bunch of image layers being pulled down. So we're going to go in and we're going to talk specifically about Docker images, but when you write your Docker file, any time you use a run command, a copy command or an add command, it's going to create a different layer. That's what you see here with the 1.7.7, 6.5. These are different layers. When you do that, you're going to change into layer 3. Layer 1 and 2 will stay the same and only layer 3, 4, and 5 will change. That's when we talk about being able to have shared resources across containers. If I have one container running off of this version of it and another container running off a version where I made the third change, they're still sharing the bottom two file layers. Let's see that in action, though. So if I do a Docker history, it's a Docker history for HDPD. I'm not going to expect you to be able to read this. I'll show you the Docker file a little bit. When I started out with containers, I was like, file system layers. If I make a change, it's going to make a change in the size of the image. That seems so logical to me. So I went in and I started looking and this one has 55.3 megabytes and it added a file. I can tell you just from history because of how much I use this, this is just pulling in the OS layer based on a .3 and I'm like, okay, so I executed and I ran app get update. That's going to make some changes. That makes sense to me. The next thing, okay, 43.1, I go in, I can see that I have some added data and I can see 133, okay, so I copied over file. This isn't making sense. There were five files when I pulled it down. There are only four things that made a change. The change is right here. It doesn't matter if it's big enough to make an actual impact on the size of your image. This was a tiny change. I made a change so it created a new file system layer. This is really important when you're looking to make the smallest images possible. If you write a docker file that's just add this, make this directory, change this and I'm pulling 57 layers to get your image, that's not going to save time. I probably could have gone and written it by the amount of time that I get your file into my system. So something to keep an eye out. So if you look at the docker hub, this is what it actually looks like. I know some people prefer to look at it through docker history. I'm a visual person so I actually prefer to look at it here. And that's exactly what we were saying. So file system one, I'm sorry, layer one, comes in and it's pulling Debbie in stretch. That's what Apache is based off of for a docker container. Then we go in for that run make dir-p, hey, that's similar to Cherook, and it says what's going on. If you'll notice, there are other like this big section here that didn't cause layer changes. That's because we're just changing environmental variables. We didn't run one of those three commands that we talked about. As part of your homework, go in and look at some of these docker files and see if you can see why file system layers are being made. The last one is a copy command here. So that was your fifth layer. Interestingly though, once we look at our docker history, you see this last number it's the very last layer that's given to us. It's going to be the same as your image ID. This has finished building. It now lives on your system. So that is the image ID that you'll be using to be able to interact with your docker file. I know it says missing here. All it means is that basically this wasn't done on your system. This was all done on somebody else's. It was pulled from the docker hub. It's missing because you didn't do it somebody else did. That's something that just nobody ever explained to me. So it was important. I promised I would leave at least 10 minutes for questions. Let me show you really what docker does for us. So I do my docker list and I see that I have my container. So let me run a new one. Docker container run Apache 2. It takes me longer to run this command than anything else. We're up and running again. It's as simple as that. It was 2, 3 seconds and we have an Apache container isolated up and running. So that's what I want you to take away from is the fact that the reason that we move to containers is that it saves time and allows these things to be automated. So with that what questions can I answer? So if you've got a question, raise your hand and I'll bring you in the mic. Am I that good? Like no questions? All right. There you go. And if you are somebody that doesn't want to ask questions like in person, I'm definitely available outside. How does this relate to virtual machines and that connection does it create a virtual machine? So when you're running Docker or LXE on Linux, it does not because it's able to communicate with the Linux kernel. All Linux distributions share that Linux kernel so it allows for the communication. If you're running Docker on Windows like Windows 10 not Windows Server, a small virtual machine is actually created that runs Linux that enables you to be able to make those system calls because at this time, the Windows kernel cannot communicate with the Linux kernel. Does that address your question? Yeah. I have an extension of that same question pretty much. So if the underlying Linux machine is running on one kernel and you want to spin up an LXE running on a different kernel, would that now be able to work or would it just fail or would it create the actual virtual machine or how would that work? I totally put you on the spot. Do you know because I've never actually messed with it. I phone a friend right here. You only get one kernel for all of your containers that are on a shared host and that's part of the kind of blessing and the curse is usually you don't need another scheduler, another kernel, another scheduler per process. Containers give you a lot of the isolation of a virtual machine without introducing a new kernel, a new scheduler and all the extra overhead that goes along with it. So the process is boot faster and share the low level system internals that are capable of being reused. This is my aha moment. This is the moment where in all of a sense, so the secret sauce, the real magic behind all of this is the shared kernel. So it's like how does containers work? Why is this able to work? Why don't I need to have a virtualization here, shared kernel? That's the one that's like the magic little secret sauce behind everything. This is where you give talks right now I know to add that in here. And the way that we achieve that is by on your host machine, you need a very, very new Linux kernel and the rule with the Linux kernel has been that you try to always have the APIs be backwards compatible. And since that has always been true, if you have a very new kernel on the host, you can emulate the kernel APIs of basically any older Linux distro. And so you can have a system appear to be Ubuntu, appear to be Gen2, because all the kernel APIs are all identical. The only thing that really matters is the CH root that you mount underneath. Right. That's it. Thank you. So to piggyback off of that, so if you're trying to, essentially I guess, run a container as a virtual machine, wouldn't that be LexD at that point? There are a lot of comparisons and a lot of people who say that LexD really is kind of like a virtual machine. The argument to me is no, because you're not emulating hardware. That is a big difference between a virtual machine and the container. I'm emulating system parameters and I'm emulating OSs. With virtual machines, I'm emulating and limited to that hardware. So maybe you have some guidance as to documents about design and conceptualizing your containerized workloads. And specifically also, I'm really interested in about persistence. I mean, obviously your container goes away, everything goes away. You kind of go back to the whatever you install, the binaries, but you also maybe have workloads that may want to go away. So I'm kind of curious as to how you do that. I'm sorry, that mic was not really, can you rephrase? So it's kind of two questions rolled into one. One is what guidance do you have for designing container-based apps in general? And more specific version of that question is how do you persist data in a containerized environment? Okay, so I'm going to answer them backwards. Persisting data is really going to take a lot of time. I tend to specialize in Docker. So you're going to be using Docker volumes. And you can use them as mount points or you can use them as part of the image. That gets a little complicated more, your 102 level. If you go to loconk.com and you click on that Docker quick start one, I think it's still free. If it's not, get back to me and I'll make sure that you can watch the video. But it really goes into actually creating mount points and pulling information in and how when you destroy that container, that mount point, you put it in a different container or put it in your script. Going back to actual documentation. Sorry Ryan, I'm going to throw you in here. But go and play with the playground for OpenShift. Because it's where I learned how breaking apart applications actually works. Ryan did a great job and his whole team did a great job of actually writing applications that allow you to be able to make configuration changes and see it happening in your browser as you're doing it. My favorite thing about this is that you're doing a run called, was it like US maps or parks. And you're going in and you're making changes to your config, you're making changes to your application and you're seeing it happen real time. So that's definitely something that would go for. Play with Docker is a great resource and obviously I have to give us a plug. If you want to, you know, visual based learning and getting a chance to do it by yourself, I'd really encourage you to join Linux Academy and our free training. I'm not trying to sell you the branding and possibly that of others. I've heard snaps thrown around kind of similar to Docker. Specific instances would be like Next Cloud. I've spun up Next Cloud in the Docker container. I've done it as a snap. Would Snap just be a packaging utility or is it somehow used separately like an image would be? Or is Pofi when I need him? He's not in my talk. This is so wrong. Alright. So he gets, he can introduce you to someone who can really delve you into Snap. But as far as I understand it, you're just looking at a packaging. You know, it really isn't a containerized. You can use it in your containers. I quite often do. Ryan, share with the rest of the class. Alright. So basically if you have questions on that, get with us. I know Wes and Chris right here know Pofi really well and he's the one that does the Snap magic so he can get you. Alright, we good? Alright. Last one. Last one. I will answer questions afterwards. I'm not trying to tell, but I want to be respectful of the next speaker. So, for the layers, what dictates a layer is when you make a commit, is it anytime you run a command, is it outside of your control? The OS layer, or the image layers? The image layers are run, add, and execute, I believe. Are the three that will build your image layer anytime they use this? Basically anytime that you're making an actual change, not an environmental change, but an actual change to your application or your system. Does that make sense? Alright. Okay, thanks. We're going to take a lunch break and then the container track is going to resume in this room with a panel session of Kubernetes users. Thank you guys. Thank you, Ryan. Thanks, Al. That was interesting. Okay. Check. Check. Check. Check. Check. Check. Testing. We'll get started in about two minutes. Did any of you guys actually see that cannon thing last night? Had never I left the cannon thing? I don't know. Okay. It's 1.30. I guess we'll get started. Welcome to the resumption for the afternoon on the container track third day. This is a session where we've got a number of Kubernetes users who are going to speak as to the experience they've had. Our panelists, I'll let them introduce themselves. We'll start with you, Ilan. Hi. Hi. I'm Ilan. I'm one of the conference organizers here at Scale, but when I'm not doing Scale, I work at Datadog where we're big users of Kubernetes. I run our product and community teams. Hi. I'm Guinevere. I'm a software engineer at GitHub. I'm on the infrastructure team, and we do a lot of things with Kubernetes. Also, I see that we're here in the corner sitting over here, so if any of you are more comfortable with actually seeing our faces, it's totally cool to get to the top of the room. Go ahead and do that. We're not going to be weirded out. Yeah. Hello. My name is Michael Goodness. I am systems architect and tech lead on the Kubernetes cloud native team at Ticketmaster. And we are also doing things with Kubernetes, which probably explains why I'm here. Yeah. So all of you are obviously using Kubernetes. You wouldn't have gotten the invite if not, but let's get into, I want to give you a little bit more detail on how to describe the details. First of all, what form are you using Kubernetes? It can range from you taking the pure open source, building it yourself to using some form of distribution or a Kubernetes as a service hosted in a public cloud. Second, when did you start? How far along the path percentage wise you think you are? Are you like 50% into Kubernetes on the way to 100 or is it 90% and then finally, please describe your workloads. Were these new apps that you lifted and shifted? Things, Greenfield things that you specifically started new for Kubernetes or some mix of the two? And where are you running Kubernetes? Is it on prem in a public cloud or some mixture of the above? Why don't you start it, Lon? No worries. So we use the open source upstream Kubernetes ourselves. We haven't used any of the cloud providers managed service offerings at Datadog around Kubernetes specifically. The challenge we had is when we started, we were starting to use Kubernetes at Datadog was that most those cloud service offerings were fairly, fairly young. So like EKS was in a private beta that we had heard about and had access to, but I think you could run maybe 50 nodes or 100 nodes in there or something like that. So we had not even a percentage of the size of the clusters that we're talking about. So we had to roll our own. Rob, one of my colleagues, Rob Ball, gave a talk in this room about 24 hours ago about a lot of this journey in the container day. And what drove us to do this is actually was moving existing apps. We had a business requirement to spin up a new point of presence in Europe, which we had never, we always had just a variety of requirements where things could not leave the European region. So we needed to spin up something in Europe. And then for business reasons, we were also asked from the business to do this on a different cloud provider. So it's like, let's change all of the things, but keep the application code the same. And so we had to port all that over and yeah. Are you saying you're on a mixture of public cloud providers? Correct. Not running your own on-premise point. And then I think your other question. None of this is, well, there's always some subset of our stuff that is new in Greenfield and being developed from scratch, but the reality is that there's, you know, the company's been around for eight, nine years now. There's eight, nine years of code that was not going to go away and get rewritten, so it was lifted and shifted into Kubernetes, which we can talk a bit more about, but was quite a bit of effort. I think the one thing that's different about our workloads than most people I talked to is that we are significantly connected to the cloud. We have a lot of folks at conferences like this, and they have a lot of stateless workloads, right, where it's just, it's their API servers, it's their websites, it's things that maybe that users interact with but don't store data. We're a time series as a service and an event storage and log storage as a service. So then are you running the state full on Kubernetes? Because some people are considered that to be leading edge. So Kubernetes is there. And then finally, the last thing I wanted you to cover was what percentage of the progress are you towards Kubernetes? Are you halfway there or are you thinking there and in full production use about the same percentage you'll ever be? So it's always increasing. We have regions that are 100% Kubernetes, and we have regions that are, like our original data center region in the U.S. fully intended to be there. It's just sort of a convinced engineers to, next time they do the deploy, do it over here rather than over here. So it's a lot of education, a lot of evangelism within the company, but it's coming very soon. Okay, thanks. Let's pass the baton. Are we taking questions? We'll get into that. We'll have some questions. Yeah, so first I want to start with I a bit of a disclaimer because I started working at GitHub very recently, but I am on one of the teams that runs our Kubernetes infrastructure and one of the really awesome things about GitHub is that we changed to Kubernetes sometime in 2016 and took about a year to gradually migrate to Kubernetes. Maybe some of you know this, but we actually started with github.com the website that many of you use day in, day out and moved all of that to Kubernetes as the very first transition to Kubernetes. One of the main reasons for this is the super supportive and vibrant community that is the Kubernetes open source community. Obviously, being GitHub we support open source. That's sort of our deal. And so in a way we like to keep friendly relations and remain each other's poster children. We at GitHub like to point at Kubernetes as one of the biggest, if not the biggest open source community that uses GitHub and we in the Kubernetes community like to collaborate with GitHub and have a really close working relationship to make sure that we align features work and that sort of thing. So that's just kind of a really cool collaboration right there. We run Kubernetes on bare metal on our own servers, which is super awesome as was already mentioned. We do not run any kind of stateful on Kubernetes. We have a data center team that does a lot of really cool work and we currently don't have any plans of migrating any of that to Kubernetes. As far as this percentage of adoption we have a lot of internal apps that basically almost every new application inside of GitHub is going to start running on Kubernetes. And a lot of our legacy apps are migrating and as far as I would say we have pretty much reached the level of Kubernetes adoption that we want and it's pretty pervasive throughout the company. On my team specifically we are building deployment tooling that makes deploying your app onto Kubernetes, onto our infrastructure as easy as typing in a chat ops command. Okay, thank you. Michael if you want to go next. So we've been running Kubernetes in production for about a year and a half now and while I personally and I think a good chunk of my team would be comfortable running kind of pure unadulterated upstream Kubernetes. We like having we like having somebody behind us. So we have been using for as long as we've been using Kubernetes we've been using CoreOS Tectonic which of course is now has been or is in the process of migrating to OpenShift. Our adoption hasn't been quite as full as I think my team certainly would like. I think we're at maybe a quarter of all of our applications have moved to the cluster. We are the default landing zone at Ticketmaster. We have reached a level of maturity and stability where that's being our platform is being promoted as go there unless you have some extraordinary extraordinary reason not to. Finally it was where is it running? Public Cloud? Sure, sure. When we started we had just made a grand announcement that we were going all in on the cloud and AWS in particular shortly thereafter we got our first quarter bill. Yeah. It was kind of everybody back into the data center. So we've been running hybrid cloud approach for about a year now. The data center clusters are right now bare metal however we are in the process of spinning up new clusters on cloud stack to at least gain the advantages of a virtualized infrastructure. Okay, thanks Michael. Maybe we'll bring it back to Elan but pretend you could go back in time and tell your newbie self one nugget of advice. If you had a do over in your path to adopting Kubernetes what would you tell yourself? Probably better education of end users of Kubernetes within the organization. There's a lot of we ended up finding a lot of teams that were looking for answers on stack overflow or placed on the internet where they thought that's clearly the right answer and it turns out that it might have been the right answer for that person or maybe it's the wrong answer for everybody depends on what they found but it probably wasn't the right answer for how we were doing things at Datadog. So basically doing more of that education more of that training and pairing with teams as they were moving. We just had a very aggressive timeline so it was like Kubernetes there go and while we partnered with the teams we should have done more and so the teams going back and doing a lot of that. I think a lot of the types of things that Gunnevere mentioned earlier around building more into our deployment process into our release process to make it a more standard path to get there rather than like here's Kubernetes APIs enjoy. That tooling and that education would have helped us move a lot more quickly. Thank you. Same question so we're going to do it differently knowing what you know now. Yeah, so I would say the main thing that I would want to do over on is the contributor integration into the upstream community. I think at GitHub we've developed a lot of really valuable tools around our existing infrastructure and we could have definitely done a much better job earlier and sharing them with the wider community at large. I have sort of made it my personal mission to rectify this which is why I kind of bring this up here. We have a couple of really nice lightweight tools that can make the transition from your existing infrastructure to Kubernetes a little bit easier perhaps depending on your use case as always but that would basically be my main regret if I could go back and do that over. Okay, thank you. And Michael, same thing, what's your do over? Sure, so it kept going in Austin in 2017 I gave a talk titled One Chart to Rule Them All the idea behind it being that Ticketmaster had created one helm chart if you're familiar with Kubernetes you would have at least heard of helm. One helm chart that would be provide a common deployment pattern for every team that was deploying to Kubernetes if I had things to do over again I wouldn't submit that talk because what we found in practice is that at least the approach that we took to creating that kind of one size fits all chart in particular just it doesn't work. What ends up happening is teams end up having different requirements to be able to auto scale someone want to be able to you know this or that exploit different parts of the Kubernetes API and by kind of locking them into that helm chart we only have two ways forward for them they either walk away from the helm chart or they add to it and stick in a whole bunch of extra templating and ends up being you demand in helm it's just no not necessarily I think right I think what we would have embraced from the outset is the idea of forking and extending. Next question back to you Elon Kubernetes is of course a traditional open source project that's governed by a steering committee but let's pretend for a moment that instead of the steering committee you're the benevolent dictator and you can put the one feature you want into the next release of Kubernetes what would that one feature on your wish list or extension be? Yeah So I'll answer this more as a person that builds products for Kubernetes unless it's the user of Kubernetes because I think that's where I'm I would there's always a very long sort of roadmap and vision instag instrumentation for metrics and telemetry within Kubernetes will go and if I had my wish we would just sort of like fast forward a year to get to the end of that so that I can then build something on that API it just right now things change constantly and so when people come to me with like Kubernetes one three which yes people do still run I assure you I have to look at one thing and then when they're with one four it's another thing and like every revision there is something different it just makes it hard for me as a person building products for that to deal with I asked my engineering team what feature they would like on Kubernetes and they said they sort of laughed and said could you actually talk to them about removing features there's too much in there so that was actually something that came up at the contributor summit and you're fair to answer this that quit adding new features and work on stability for a few cycles and that would be a fair answer but let's see what others are you kind of took a little bit of the words out of my mouth I also don't really I can't really think of a new feature that I am desperate for what I would really like is a little bit more support around cron jobs and jobs because I think half of the questions that I answer as a first responder for our Kubernetes infrastructures like hi I'm trying to delete a namespace but it won't delete why not and it's usually because there's some hanging jobs and then you have to go in and manually delete the jobs but if then there's a cron job running then you have to get rid of the cron job and so I think at least half of our headaches come around jobs and cron jobs and I would just kind of like a little bit more support around that okay thanks so my feature request would be around the metrics which is a relatively new API to begin with so it's not really an incredible ask I don't think specifically I'd be looking to decentralize it but custom metrics autoscaling is a thing in Kubernetes and more recent versions of Kubernetes however there has to be a place for Kubernetes to pull those custom metrics from right now the metrics API is centralized you have one data source for those metrics at Ticketmaster we run one instance of Prometheus per namespace or per application per product team and they all have instrumented their applications their own way so if they want to scale based on request per second and they've used a slightly different naming a slightly different name for that metric then team B they can't there's no easy way for us to kind of pull in all of those metrics into a central Prometheus and then use that for scaling that would be my kind of yes my selfish request okay thank you next question and we'll pass it back to Elan I kind of know your answer so I'm going to give you a warning here there's a huge number of supporting tools and projects listed in this CNCF landscape it's getting to the point where they're giving them away in the CNCF booth but they need to give away a magnifying glass with it to look at all of these and I know your favorite tool is going to be Datadog so for you only what's your second favorite tool so yes obviously Datadog but I think it's hard to pick one we integrate with we use many of those tools we also have to support almost all of them as our customers may not pick our favorite tools I realize this is a non-answer but honestly I'm sorry I just realized that you have partnerships probably with almost every one on there if you were a king maker you could potentially have to deal with that afterwards you can give a category or whatever or just a recommendation of where people ought to look in this chart for things that will help them out I'll say things like we've internally made the shift from Docker to container D we found that as we scaled up that was a very important ability wise and so I recommend that as folks are looking at what to power their cluster they talk to some other folks that have been running at interesting scales and see what makes sense for them similarly I think on the networking side like we found the lift C&I plug-in is quite awesome for AWS okay thank you yeah so this is a little bit of a shameless plug I have noticed that when so because of the quick release fairly quick release cycle for Kubernetes versions I think a lot of the trend currently is to instead of upgrading clusters in place to decommission older versions and bringing up new clusters and in order to do so what we do internally at GitHub we have multiple clusters and the newest clusters always the latest version of Kubernetes and then the first one we decommission eventually and bring up a new one with a new version and in order to do so and this was also probably part of how we moved over to Kubernetes in the first place we needed to do something to export cluster wide services to an external key value store that then our load balancer could access so we actually have we just recently open sourced one of those tools and it will help you bring your own load balancer to a multicluster Kubernetes infrastructure I think it's really helpful because it avoids having to use ingress for cluster access and management so okay thank you I would have to say our next favorite CNCF project would be Prometheus we're using Prometheus pretty extensively for monitoring so Prometheus yeah that's the short answer I would throw Fluent D in there as well Fluent D is great for log aggregation and shipping super flexible and then we're starting to devil with container D which I also I would agree is kind of feels like the next logical step after Docker since container D is a foundational component of the Docker daemon so yeah okay thank you what are the best sources you found for training newcomers in your organization and then after people are done pass through the newcomer stage what's your advice for getting successful outcomes with Kubernetes so we we sort of took two paths one is because we have so many customers that run Kubernetes we called all of them up and said hey what are you doing what scale are you doing it at let's chat and that was both great from a product perspective I got to know what they needed us to build but from an engineering perspective we got to exchange a lot of notes so I know you're looking for public resources on the internet but honestly a lot of our education came from just talking to peers and industry at conferences like this or one on one meetings and then we built a lot of internal educational tooling around all of that which opens sort of like open the specific educational documents we built inside but instead what we've been encouraging our team to do is share all of that knowledge at community events so Rob spoke here we had Laurent speaking at KubeCon our goal is to make sure that anything that any sort of the lessons hard lessons that we learn we're passing on to others in the community similar to what one of you was saying and so hoping to continue to do that but I don't know that there was great and I know not everybody here is in your position of being able to talk to partners the Kubernetes community in my experience is pretty open to ask your questions in Slack or Zoom meetings yeah so this is this is a little bit of a hard question for me to answer because I started Kubernetes as an intern straight out of web development bootcamp so that was interesting I think that I might be the main resource internally for people who are looking into getting good at Kubernetes because I mean she has run a contributor some of the events so you're free to plug those yeah so at the the past couple of coupons I have helped run the Kubernetes contributor workshop which is an introduction to the workflow in the Kubernetes community and where to get started where to find your first issues how we work how the project is structured how to meet people how to become unblocked how to overcome that initial fear of becoming an open source contributor where everyone can look at your code but also I maintain a list of beginner friendly Kubernetes resources that I send out to various people on a regular basis the other thing that it feels weird to recommend to people but you don't have to take the exam the recommended preparation for the Kubernetes certified Kubernetes administrator exam are actually really good preparatory tools on how to use Kubernetes as a developer and as an operator so the materials that are provided are quite extensive and good you don't have to take the exam though we can talk about that another time and I believe your events were recorded and posted on YouTube correct and the materials are on GitHub so I think gosh it's again a little bit spread out but if you look for the KubeCon Kubernetes contributor workshop on YouTube you'll probably find a series of instructional videos that will walk you through the workshop step by step and the slides live in the contributor playground repository on Kubernetes SIGS and I think there's also a link from the Kubernetes community let's see what else is there so I think the certification classes as well as they have both free and paid training classes associated with those if you go to the CNCF booth that's in the Expo Hall they have a sheet of paper that has those links on it so I'd encourage you to go there if you want to follow up on what was suggesting sure and if you can always follow up with me talk to me, find me I try to wear the SOTY at all times my name's on it then you won't even have to remember my name or peer at my badge or something yeah this yeah this is a tough one I found myself pointing more and more people more and more often to the official DAX actually has specific sections of a given documentation page the DAX open source projects don't always have the best reputation for documentation but Kubernetes DAX have gotten much much better certainly over the last three years that I've been involved with the project to the point that I do actually point some users to those DAX when they have questions I don't just send a link or try not to just send a link that's not a great way to bring somebody up to speed but at least get them started there and then maybe expound a little bit on that I think we've also found that it helps to have a set of what I call Kubernetes whispers in the organization who are available who know the API who know the specifics of our implementation of Kubernetes, how we run it how we offer it as a platform to be able to do some of that translation for people until they you know have kind of gotten up to speed and then of course trying to create tooling and processes and pipelines that make it easier for folks to get up to speed right away Okay at this point this is the end of the prepared questions that the audience members have some this is going to be difficult since we have one hand he'll die okay so let's start with you what's your question okay so I'll repeat the question observe that these people have been successful with Kubernetes the question is now that you've moved was the fact that you went to Kubernetes worth it short answer yes absolutely especially given in GitHub's position of where we are and how we interact with our contributor communities even just from like a like a visual standpoint it's just really beneficial thing for us to even just have that technology readily available running on our platform and then using our platform and then using Kubernetes to run our platform the other thing is that our uptime has significantly improved even just in the last few months it's just gotten much much much more reliable in fact if anyone remembers our last big outage in October of 2018 that wasn't that was not a Kubernetes based outage at all which actually runs into your second question about storage oh did we skip that never mind let's do that later yeah I'd say it was it was worth it for us there's one is you know just those business requirements that we had to meet in terms of being in different regions of different cloud providers I don't know that we could have done it in a timelines we had without Kubernetes it would have been a lot of manual and painful efforts but more importantly we have Kubernetes is one of the fastest growing technologies we see our customers adopting and so as an engineering organization and a product organization that wants to sympathize with our customers and understand their challenges it's really hard to reason about things like Kubernetes and containers if you're still running everything in VMs yourselves that doesn't mean that every time one of our customers wants to adopt the technology and for us to monitor it that I have to go to our engineering team and say you must adopt I don't know Oracle as a database because we have customers that use Oracle as a database you know there's but things where there's like that big of a shift in the way you think about it and the type of data that you collect and the frequency data that you collect and it just it's made our product so much better and it's made our relationship with our customers so much better to be able to say we run in the same way that you do here's how we do it here's the challenges that we've had let us help you and in the same way they've been able to help us so like we've just gotten better as a company as we do provide it to our customers because of it so I regretfully have to say that it would depend on who in our organization you ask that question of and how up how high up they are I think most of our developers would say that it was definitely worth the pain of kind of migrating to containers and learning the Kubernetes API and processes they get things like fast deployments and really high scalability but it hasn't been without as I said it hasn't been without its pains and and I think it also depends on what really I think it depends on what you expect out of it if you were embracing it because it was the next thing then you may be disappointed but if you've done the research and I think those of us who understand what Kubernetes was meant to do it wasn't simply intended to cut costs you know then it's yeah I think you're more you're going to be in a better place to be able to be successful with it thank you and I think his second part of the question was what are you doing for storage how are you accommodating stateful so we do not have a lot of stateful sets deployed in Kubernetes those that we do are EBS in AWS and we have yet to pick a persistent volume solution for the on-prem data centers we're kind of leaving that to the infrastructure team but our requirement is that whatever solution they choose has to have a storage driver available for Kubernetes with regard to that as of this point we do not have any plans to move any of our database stuff over to Kubernetes I don't know if that's in storage for the future no pun intended actually but yeah currently that's not a goal for us at all it's a mix of things a lot of the data stores that we run are actually not off the shelf data stores that you pick up off the internet but rather things we've written ourselves and so they don't always lend themselves to the off the shelf storage technologies that are out there but we do use a you know in AWS we use EBS where it makes sense in some places we use our own tools that replicate the data where it needs to be replicated it's unfortunately not a particularly useful answer for the rest of you all because you don't nobody else runs the data.datastores so okay thank you somebody here had a question okay he's asking what are the prerequisites that people should have to get into Kubernetes you mean like personal knowledge of people or hardware resources okay recommendations for a 101 class on what you have to know before you're going to be prepared to move on to Kubernetes I honestly recommend starting on kubernetes.io the docs website there is there are 101 tutorials they'll walk you through a super basic like this is what it means to run a Kubernetes cluster you don't even have to install anything because it's set up with Kata Kota it's really really nice and very instructive okay we'll go here next and I see somebody in the back your question okay so your question anticipates that you're going to run a big Kubernetes cluster and you're asking how you might take advantage of namespaces for oh okay so his question I believe is would you recommend having a Kubernetes cluster per tenant or per user group or going with namespaces it kind of depends where you want your pain points to be right at github we have multiple clusters that sort of act as one and the same infrastructure to our users our internal users but we do namespace everything but we don't differentiate between where you run stuff not really we're actually starting to group clusters less by use case and more by the properties they have as we're moving out into a multi-region um I it really depends it might be a little bit easier to not mess with namespaces and rbacks as much if you have a per cluster but then you have to manage multiple clusters so there's that I think we'll just answer these one person because I saw a lot of hands that were running short of time and maybe your second part if I I would actually like this one because it's something we're in the process of doing I have been, since I've been using Kubernetes I have been a big proponent of fewer bigger clusters multi-tenancy and then controlling that tendency through network policies and rback however again you know three years in you get some experience it's inevitable and we have some workloads that we are thinking deserve their own clusters their own sets of clusters we have embraced kind of the in place upgrade model updating the control plane in place and when you have big workloads and that's in terms of scale and in terms of resource requirements that causes quite a bit of churn that in place upgrade process creates quite a bit of node churn and when you have these big big sets of big pods it slows things down considerably and actually I think it makes the whole upgrade process a little bit a little bit sketchier than you might like but really it's very situational yeah there is definitely not one answer okay if there's time we'll get to your second question because I see multiple hands would somebody here had a question go ahead and then I'll repeat it for the mic yes so the request is that those resources you referenced can we publish the list so that he can save the search effort yeah we can get that out I'm not sure that we have a vector to post this associated with the session at scale but if we can add them to your slide deck then I'm more than happy to post them on the scale website so we'll get them and I'll I'll send out the slide deck via some means I'll put it out on twitter at least and my handles can't be wrong any other questions okay the question is have you standardized on any tools for updating your clusters kubadm or whatever you might be using for upgrade so tectonic and open shift are built around a technology called bootcube it's self-hosted kubernetes so that's what we've been relying on so far I have worked with kubadm and for building clusters that's really come a long way it's a lot easier than here again than it was a few years ago but day to day we rely on the we rely on the vendor anybody else want to describe what you do to upgrade I think somebody said that you just stand up new ones and port the things over don't we move all of our traffic over to a new cluster and then kill the old one that is currently our workflow we do patch upgrades in place actually so the downside of that moveover do you find I think you need a little extra resource to be able to host that during the time period where you've got old and new living at the same time since we have our own data center we already have the machines it's a little it wouldn't save any AWS bills to not use them yes you always have to have a little bit of extra so that you have room to move sometimes when we move over a larger cluster we have to do so gradually and only use some of the machines and decommission the old ones re what's that called like re-provision thank you I was missing that word re-provision those for the new Kubernetes version and then add them to the new cluster and then slowly move traffic so yes sometimes it's a multi-step process but we have a lot of work dedicated into making this process as easy as possible for the cluster maintainers it's as easy as a chat-ups command to provision a new node at this point for my team thank you and Elan do you want to describe what you guys are doing for upgrades still evolving I'm gonna say no yeah I mean I think we're sort of between sort of which of the two paths to take given some of the stateful decisions we've made I kind of want to ask a question though of the folks here what are you all doing around things like service meshes and inter-application communications and where is that going for you so far I have played the typical ops card of just saying no I'm actually not that proud of it are you saying no because you don't think it's ready or because you don't think it's useful or some third yes I think you know service mesh has tons of potential tons of potential I just I'm not sure if it's there yet especially when you and you know in full disclosure you know I've made these kind of decisions based on you know third party articles and conversations they just you know right now they seem to add quite a bit of operational overhead and we're just not in a place to take that on and you know the benefit to our workloads we're not running a lot of microservices a lot of our applications are you know monolithic and so the benefits of a service mesh I haven't become clear to me and when I ask our dev teams they haven't really become clear to them so that yeah that's why I don't really feel bad about the out of and no your choice you can make him answer his own question about service mesh or throw him another one really quick we actually have a dedicated team that is currently in the exploratory stage of how to do services internally they're called service communications they are also hiring so if service meshes are your thing come talk to me because there's some opportunity there and it actually looks really really exciting so I think that was a question in the back there couple we're no no I'm not we've been exploring Istio pretty pretty aggressively there's been some challenges around performance especially you know on the control plane side and not on the you know not on the the envoy piece is pretty solid so I was just curious what other folks are doing it's a thing we have to do we just haven't figured out how to do it at the scale that we need it yet so okay somebody way back here had a question I'll listen and I'll repeat it okay so the request is that he's attempting to integrate in Prometheus with alert rules and having some difficulties doing that and wants to know if anyone's got advice on how you take care of that I think so we were big users of the Prometheus operator which is you know Coros originated project the operator introduces or somewhat recently introduced an object called the Prometheus rules resource definition we're not using that very extensively however it will work very well for us once you know we've kind of made that migration I'm not sure though if you know I'm not sure that that would work very well for your specific use case as soon as you start throwing red jacks and different environments into the mix that you know it gives me some pause there is no the cube Prometheus sub-project from Coros so if you go into the Prometheus operator Git repo there's a sub-project called cube Prometheus that is using Jsonnet to do a lot of templating and in fact that's how that project is generating their Prometheus rules it's all kind of templated out using the Jsonnet DSL and then you run it through the Jsonnet without a Prometheus rules object that might work but you know we run our clusters we try to keep our clusters as similar to each other as possible so we don't have a lot of variations so I can't really speak to that specific kind of case so I think we might have time for one last question here and I'm listening go ahead okay so he's asking what does the developer workflow look like in your company do developers have direct access to deploy onto a Kubernetes cluster themselves or does it go through some kind of preliminary layer to accomplish that there's there are some that are interacting with the API directly it's not necessarily a desirable approach a lot of our deployments we're just we're using spinnaker and helm at the moment to do that and so that's the encouraged path my company builds a tool called moda which is basically an overlay configuration into our Kubernetes infrastructure so with our tooling you can create a new GitHub repository that already comes preconfigured with a deployment environment and has both the Kubernetes configs and the moda configs and the moda configs has a very very short stanza that is easy to duplicate and edit and bend to your particular use case and then the actual deployment happens via chat ops in Slack let's see yeah so this this kind of gives everyone kind of the paved road of this is how we want you to deploy apps and then moda gives you all sorts of information of how your app is performing it sends links out to our data dog and and you can you can look at your containers both via chat ops and via this the moda website interface most people do not have direct production access but the chat ops spot does we do allow direct access to the clusters the end user access however we strongly encourage deployment via CICD pipelines and this is actually something that we're constantly re-evaluating that decision because we keep getting called into incidents where if they had just made their change through a CD pipeline they wouldn't have gotten themselves into that trouble keep couple edit is pretty cool but not in prod okay thanks I'll give the speakers thanks for coming this was a great session if you want to leave any parting remarks or throw a question out last minute go for it thanks for coming I hope it's been helpful if you're interested in continuing conversation we had taken master pretty open about the way we're trying to be open about the way we run our Kubernetes clusters so get in touch with me Twitter handles everything else I'm always happy to chat any questions regarding Kubernetes how to contribute send them my way I'm Gwen Codes on Twitter I follow my Twitter account pretty closely especially around conferences thanks for coming I'll throw that out there as well where you are hiring a data dog for all things but also people with Kubernetes related stuff so yeah I would check out things I'll put out there maybe a little bit of a shameless plug but we've written a couple Kubernetes related guides that I think if you're looking to know how to monitor Kubernetes whether you do a data dog or not there are some great guides up on our blog that you can share please check out Rob Ball's talk it's already up on YouTube he goes into it in way more depth than I did on this panel and I think it's worth checking out if you didn't see yesterday thanks to the audience for coming the container crack is going to resume in a little while but we might have some of three o'clock containerized desktops thank you thanks everybody one, two okay I'll give the last few time to find a seed and we'll get started here so if you came for the containerized desktops session on the container track you're in the right room Matt came to us from Boston Massachusetts and is working on the containerized desktop at Red Hat and Josh if you've been here for other sessions this day you've probably seen him before he's an organizer of the container track event works for Red Hat and is a long time scale attendee and I like using containerized desktops so that's why I'm co-presenting here so welcome everybody so containerized desktops how many of you know who this woman is anybody? yeah I got a few in here Jesse Frizel one of my technology heroes so Dr. Conant 2015 she showed all of us how she had containerized her whole desktop how she'd put everything into containers and that sort of thing she demonstrated using Spotify in a container she demonstrated using the Chrome browser and routing all of its traffic through another container running tour so she could do anonymous browsing and that sort of thing put up a bunch of stuff on github with all of these .files etc and a lot of us felt that this was a pretty awesome concept it involved an awful lot of hackery and so a lot of people who work for different companies that do container stuff and some people in their own went away from that wanting to actually implement something similar to that only make it a lot more of something that anybody could install and use and not just Jesse and so there were some dockerish things for the U-Know Rancher OS they've got a thing in the pavilion for a while they had a desktop edition I don't know if they still do the us folks at Red Hat created this thing in the Fedora project called Atomic Workstation that was also this containerized desktop concept there was a core OS also somebody spun up an unofficial core OS desktop edition all of these sort of containerized desktop concepts all around having a desktop that was immutable running your applications in containers and installing and removing them sort of in container form I used a couple of these myself core OS desktop and also the Atomic Workstation a whole bunch alright I hope you can all hear me and this mic works seems to be okay alright so yes Atomic Workstation is one of the many experiments or attempts that people have done to do containerized desktops over the years about a year from today like last February at DevCon from Peno a few people who were interested in Atomic Workstation got together and we decided that we want to take this and push it the last mile to make it actually consumable for everyday Linux desktop users so we set out to create a project around it we formed a special interest group and over the past year we've I guess established a project and we set out and found a new name that's Silverblue that's because of us really actually and we got a new logo that you can see there over on the left to get us off the ground and I wrote a bunch of blog posts over the past year about Atomic Workstation and now Silverblue is really interested and I guess there was a little bit of concern on the Fedora side that we were ramping this up as a new thing outside of Fedora and maybe in competition to Fedora but that's why I like to stress that this is actually something that we do inside the Fedora project it's part of Fedora just another variant of the Fedora Workstation and it's actually built from Fedora content like what we use to to create Silverblue is just RPMs that you could also use in a traditional Fedora Workstation that's an image based OS and I'd like to maybe explain a little bit what we mean by that Silverblue before at Atomic Workstation is using a tool called RPMOstree to create an image and operating system files to an image taking a step back if you're running a traditional Linux distribution on your laptop if you use a package manager to update your system what's happening is that the package manager is downloading a thousand packages each contains a few files and then the package manager unwraps each package and puts the files where they belong and slash user so at the end of the day when that is all done and nothing went wrong hopefully you'll end up with a file system image on your system slash user that was created on your system and it's very unique for you what's actually in there depends on the history of your past updates and what exact version of each package every user at the end of the day has its own slash user and a traditional scenario which makes it sometimes hard to understand if a bug that a user sees is actually something that just comes to be because of the history of what's on his system or something that can be reproduced elsewhere so it makes QA a lot harder and makes things less reproducible for that reason RPMOstree takes the approach that yes we want to still use those packages they are very useful, they have been built by trusted packages over the years and that's good content but we move the composition of the image to the server side and create a single OS image with a single version number that's against the important bit here I'll check some and then every silver blue machine will update and they all have really bitwise identical content because they download the same image and in particular we can also do QA and QA on this one image so that things are in good quality so that's how the OS itself works but obviously nobody is installing a laptop just because the OS is beautifully done or updates reliably you actually want to use your system to run applications on it so we need to have an answer for applications because in silver blue we treat the OS images immutable so you cannot just take a say Firefox RPM and install it at runtime and start using the application that's not how it works so we need to have an answer for applications and that's flatback which is just another container tool I would say it's taking all the usual container technologies you know from Docker like namespaces, cgroups, sitcom, bind mounts and applies them to make a system that works well for desktop applications and basically one of the problems that we ran into with the early containerized desktops is desktop applications actually need access to a bunch of things that are not services that are available over at TCPIP port or file handles exactly the biggest one of which being debuts for all of the OS and desktop events and that sort of thing and so flatbacks are containers implemented a different way by the GNOME project in order to have access to the things that GNOME applications need to have access to and have everything else be names based off right so important aspect of being in a container is that you're independent of the OS so flatbacks when you install them and run them they are really not using much of your OS image they have their own slash user tree that matches what the application expects and can be different from what you have on your OS and each application can have a different version even so there's naturally some redundancy built into that you have to just download if in doubt you have to download more file system trees to give each application what it needs but it pays off in the sense that you can actually use the same flatback that I'm running on my raw system and can run it just the same on say Retail Enterprise Linux version that might be much older and might have very different content in its OS image but the flatback does not really get affected by that because it brings itself another aspect of container applications is that we want to have them sent box so they are isolated from your home directory and they cannot necessarily snoop around and steal your secrets and your vacation photos right and where do you get those we have an answer for that flathub which is simply an app store for flatbacks this is how it looks if you go there in your web browser and it's fairly simple but it recently reached a milestone of 500 applications so we are getting there in terms of having some critical mass please it's also nice because it's made for a better experience of when you have to install proprietary applications on your Linux laptop I would rather use open source applications but there are some things, particularly video conferencing these days where I have to install proprietary applications actually I started using this laptop primarily as my video conferencing laptop because on my native fedora laptop the other native fedora laptop zoom video conferencing platform we use a lot for kubernetes not keeping up with the various fedora additions and so they were actually putting me in a position where I couldn't upgrade fedora because I was waiting for the new version of zoom put zoom in a flatback that's not a problem anymore they have one flatback on here it runs on whatever addition of fedora that I needed on right so we have a good answer for desktop applications or at least a somewhat working answer and make some people happy like Josh but the fedora workstation is kind of targeting developers as the primary audience and of course the developers want to use Spotify too but what you really need as a developer is tools you spend most of your day typically in the terminal and you want to run compilers and debuggers and all those things and that's the workstation so flatback is not necessarily the best answer for those kinds of use cases so we have a different answer for that which is called Toolbox which is at the end of the day it's just a shell script that gives you easy access to pet containers that are ready made for developer use cases it's essentially a quick way to get a shell prompt with DNF so basically give you access to working the way that you're used to in the traditional fedora system and underneath it uses podman and scopio and other modern container technologies to do this thing and it's actually something that we kind of inherited from CoreOS the CoreOS project before it was acquired already had something called Toolbox which was basically the same idea it's a shell script that gives you a very quick pet container so yeah oops what happened there nope so so like I said I'm actually really more of a user a silver blue than I am a developer other than just reporting a lot of bugs and so the question is like why why bother, why not just use my traditional fedora laptop for everything there's a few reasons let me go over some of this one is the application isolation the second one is some security and privacy improvements over your standard stuff the concept of fearless updates which is backed by rollbacks and then the other thing is that I am a Kubernetes developer I work on container technology all the time and we really aim silver blue at the new cloud developer so in terms of isolation let me give you actually an example for something that worked out really badly on my traditional fedora laptop anybody here at python dev yes you remember in pythons or requests is like one of the most popular non-core libraries in python and for requests 211 they had a security fix that broke backwards compatibility and if you're like me and you have a lot of python based tools installed on your desktop when you went ahead and updated one of those that required the updated secure requests it broke all the ones that hadn't updated yet at least I went through that and instead of having instead of having all of your sort of tool change shared the flat packs actually bundle a lot of their dependencies into the flat pack itself right which means a bunch of redundancy in terms of storage but it means that also as a software developer I don't have to worry about this one environment that I'm installing here clobbering this other environment that I'm installing there for apps that I use rather than apps that I'm developing right apps that I'm developing run them in a virtual and run them in a container that I don't run into but for apps that I use on the desktop having them in flat packs gives me that same isolation that I get in containers for the apps that I'm developing the other thing here is that for for the flat packs we've taken sort of a key from the sort of iOS world and the phone apps world we really ought to actually have some controls and user discretion in terms of what applications require right because right now the way that it is is if you want to install an application on your laptop and Linux you sudo and you install it as an RPM and that installation process has full root access and it will do a bunch of stuff and you don't know what any of that stuff is and so that means that all of your privacy and security has to resolve around trusting the provider you know so I mean and that's fine when you're getting that RPM from the Fedora project you're getting it from CentOS you're getting it from Suzy or whatever but sometimes you actually need to install packages that aren't coming from as trusted of a source because you need new and cutting edge software and so one of the things that is built into the flat pack installer is actually giving you information about what specific resources the new application is going to have access to and which files on your system it's going to touch so you can actually take a look and you can say hey I thought this was a code scanning tool why does it need access to my audio driver and maybe actually before you move on actually if you look at this in detail not sure if you can really read this all the way in the back but it also is a nice example for showing that that's always difficult to explain to end users these kind of permissions because if you look at the list here there's stuff in there like IPC and network maybe most people will have it and have a notion of what network access means but if anything is there or SSH it's hard to explain and we're still trying to work out the best way of doing that this is the first attempt here's the graphical version of the same thing that has a little bit more clear in terms of the way the permissions again you see a lot of this in your phones these days and it feels like it's time for desktops to actually catch up to this in terms of limiting the permissions and because the flat pack is an isolated environment it means that they really can't access anything that's not on this list which is not something that you can do in a native environment that is you could have those kinds of alerts but then you couldn't actually enforce them right so as I mentioned already that one of the primary audiences for the federal workstation is developers so silver blue fits right in and it provides access to the expected set of tools here like I guess Josh working on containers knows these better than me but like docker, podman, builder all these are available out of the box on silver blue and I mentioned the toolbox already and I think I'll try to show some live demos of the toolbox later on and further down the road we look at doing a little further improving things and maybe adding actual support to the terminal application for easily giving to visualize which container you're currently in like if you're running a toolbox you'll see that later we try to like give you some hint that you are in a different environment but we could make that maybe better by actually making it part of the terminal application which is something that we'll get to eventually so one of the things and so right now it's we mainly focus on a developer because honestly the way that this started was that a lot of us who were in fedora were using the atomic workstation for our own development and so we wanted it to continue in a better you know in a new improved form so it's really aimed at developers right now but we're also thinking in the longer term you know actually and by longer term I mean like next year of the admin use case so one of the other things is that actually this is a new this replaces my previous think pad I got it replaced two months ago for corporate lease right and so I had to reinstall one of the things that I immediately noticed and remembered that I like so much about in image OS is how much faster it is to install like easily you know 10 to 20% of the time that you're used to for installing a brand new traditional Linux on a desktop or server and that's because we're not installing a whole bunch of individual RPMs we are basically just writing bits from the installation media to the disk which is a lot faster for that matter the update process is also a lot faster for the same reason because we are just applying a binary diff to the disk so it is really like an order of magnitude faster updates are also faster because we can actually download them in the background so what happens if you run APM OS 3 update on a server blue system is that APM OS 3 starts like downloading the new version of the image somewhere in the staging area in the background and you can keep using your web browser or whatever applications you have running because downloading the update does not actually touch your running OS image at all and when the download is complete you can just reboot into the new image so essentially the only downtime you have is the time that it takes to reboot the system the other thing is that if you're an administrator and you've got six dozen people so how many people here would identify mostly software developers how many people would identify as administrators what is like one of the number one things that you would like to stop your administrators you'd like to stop your users from doing installing random stuff on their desktops and see the nice thing about in image OS is most of those executable directories are read only to the user so they don't actually have the opportunity to install stuff via an unapproved mechanism and if you limit their access to downloading things like you can lock down other stuff to make it read only so that they really cannot install stuff without your approval not just via security tools and demons running on the system but actually as part of the OS the other thing is one of the tools that's in development is to have right now there's a bunch of hackery tools to do this but working on making an actual knowledge tool for you to develop your customized image right because if you're installing fedora, sentos, rel something for your users you don't want the general upstream version with just it's base level applications you're going to have a bunch of applications that you want in your company that need to be there so the idea is that you pull down the base image from the upstream add your applications as what's known as a binary layer and then redistribute that binary layer out to your users who just see it as an OS image to install so that's I mean you can do it now if you're willing to get into the depths of it in the future we will have a tool that does this so I guess we're switching gears a little bit again and this will be the fun part where we try to do some live demos of course something will go wrong and you'll have to be here with me do we want to see if we can download and upgrade in the background we can try doing that so right now this is running fedora silver blue version 29 the stable version let me actually just start with that up here most is the command line I mentioned that up here most is the tool that we're using but there's actually a command line utility by that name and it gives you some information about the system it shows you things that probably don't make much sense for most of you but it tells you where the image was put from which exact commit it is that we have here and if there was an update we could install it I'm not sure that we can actually we'll try that we'd have to rebase we can't do that can you do the command of the top of your head? I can I'll look it up while you're working on the other things I can do the command I just need to look up the source we'll get back to this at the end of this we're going to rebase this in raw height right so actually I wanted to really start with something else here so this is a silver blue system and this is a shell a host shell I would say on the whole system so I could run dnf install something because I wanted to show you maybe a gdk demo application and oops there's no dnf on this system so this is pretty shocking if you're a fidora user what am I going to do? my hands are tied behind my back we'll see so the next thing I guess to see how the system actually looks like would be to run some other commands and see what we find out at least the basic command line tools are there that's a relief and I was just looking here for how slash user is mounted and if you look over on the right you can see it's mounted read only so there's no way to make changes to the system at runtime that's why we call it immutable right but all is not lost if I run the next command here I get something back so that at least tells me that there's still in this immutable slash user file system image there's still a traditional rpm database that gives me information and it shows that the content is still composed from rpm so that's why the rpm database is there and gives us information that can be useful at times right but what do we do now? we don't have dmf so we cannot install anything here but I did mention the toolbox command earlier so I'll introduce that now there's something called toolbox and if you just say toolbox enter what happens here is that I already created a toolbox container before this talk so I don't have to spend 15 minutes watching the download prompt here with you and the toolbox enter command just basically runs this this container and inside the container is a traditional fedora system more or less and just to point that out here before I move on we try to give some little hint that you're in a different context now by adding a little unicode blue diamond character there at the very left we don't have a logo for the toolbox yet but maybe this blue diamond can become the logo it's just a little hint to tell you in which context you are maybe eventually we'll find a way to integrate that more into the surrounding terminal application and maybe give you a hint in the header bar about what context you are in but for now we just have this little blue diamond there and I'm now in the toolbox and I can try to run that command again that I was going to show you earlier okay so that's unfortunate but again so there's a DNF in this toolbox it's really more like a traditional system and I have my familiar tools that my fingers want to type so I'm feeling a lot more safe already and this will take a little bit to install and did you find that thing that we need yeah we can maybe alright so now comes the totally unprepared part of the demo this will go wrong so part of the idea behind how we treat the OS images is based on the semantics of get so what we have for each image is that we have an original source so do we need to change anything else I don't think so I think I actually just need to change the version yep yep okay hold on so Josh was trying to do yes on the fly reconfiguring the remote for the Fedora workstation image it's basically working around the unsolved problem in the repository setup where we have a different GPT key for each release and there's no existing mechanism to conveniently upgrade them without you manually entering the right URL here so that's why this is a little cumbersome yeah now we are importing the new GPT key here into the OS tree configuration and I'll be amazed that this works okay okay and I think that should whoops whoa what happened there hold on yep I think it's work now so I showed you earlier the uphemost tree upgrade command which is just basically to install the latest updates in an existing release stream and the rebase command that Josh is using here is for switching from one release stream to the next basically what we are trying to do here is take this Fedora 29 silver blue and turn it into a Fedora 30 by rebasing we'll see if this works Fedora dash workstation before that I mean to actually add mirrors here and this is why we love live demos yep it's always fun let me actually try the other URL so what I'm doing here is I'm switching it to a different server switching it to a different server within the Fedora infrastructure I'll give it one more try and then we'll switch back to working demos wait we'll probably switch it back to the same server the previous server what do you think looks like that isn't the same you don't mind alright so we'll try that again in the end switching back to my previous demo where I was installing an application a package in the toolbox you can see the install now completed and I installed this because I wanted to actually run a binary that's in the package and yeah it works so what this shows is that the toolbox is also properly set up to run graphical applications so like the display connection and everything gets set up so that desktop apps can easily run inside a toolbox and display on your host system which is otherwise something that Flatpak would do for us and another part of a demo that I'm skipping here for now is would be to try the same thing with the manual created container like basically I could use Builder to create a container for a Fedora base image and install the GTK 3D build package again and then we'll try to run GTK 3D demo it would not find the display server because Builder doesn't know how to forward the display connection and other things that desktop applications need so I'm going to skip that here because it takes a while to install but this shows that the toolbox is actually very useful even though it's mainly focused for developer use cases where you want to have access to GCC or GDB or whatever toolchain you need but it also works for desktop applications so we switch back to the day for demos then before once that's good Do you have more to demo? No I think we're skipping that I think we're skipping that I talked about this so we skip this as well and I showed this Builder what? Let's go back one I was going to just briefly in passing mention that the toolbox that I showcased here and Flatpak is not the only way to get development done on an immutable system there's other things like if you want to use IDEs as well one IDE that I mentioned here is GNOME Builder which is an IDE that's very focused on writing desktop applications and it's also very well equipped to support Flatpak for development and Richard Hargert who may be here somewhere I don't see him right now is getting a talk about GNOME Builder and the latest developments there tomorrow which you might want to look at it's a fun IDE but you can also install command line tools and build things in container wise ways right so switching gears again I wanted to finish by telling a little story that shows that while immutable OS's are really cool that we want them because they are reliable and they give us fearless updates at the end of the day it's still just software and you can still manage to get into trouble if you're not careful and you don't know how the system works and I wanted to tell a quick story of what happened to me with Silverloo on my day-to-day laptop last February when we decided to start pushing this as a project and so I could write all those blog posts based on my own experiences and one of these experiences is that I one morning was supposed to get on a video call and I still had an hour of time so I figured well with Silverloo and APMOS3 updates are fearless so I can just quickly update my system what could go wrong and now I'm using Rohite which is like the current development stream like the latest bits in Fedora and stuff does break every now and then like it has been like that forever and we're trying to make it better but occasionally there are broken composes so it doesn't boot and in fact that's what happened to me I updated my system that morning and when I rebooted to get into the new OS image I didn't get to my desktop and instead was greeted with this error message that talked about some GBM renderer problem and yeah that was not great because I needed the web browser to get on to my video call 45 minutes from then and I could have just done the right thing and said oh well I just rolled back to the previous image I could have rebooted and it would have been in the grab menu as one other item to choose and everything would have been great but I figured oh well I'm a developer I want to help people make things better so I asked somebody from the graphics team what was up with this and they asked me to provide a little utility called EGL Info and now EGL Info is not in the silver blue image as I found out when I tried to run it so I said oh well RPMOS 3 actually I did not mention that earlier but it actually lets you install packages on top of the OS image it has something called package layering so I figured oh I can just use this package layering utility and quickly install this EGL Info utility so I can get this output for the graphics team and then I'll be in my control in no time so I ran RPMOS to install EGL Utils this command here which actually worked and I got this thing layered on top and then I rebooted because to get into a new OS image with silver blue requires you to reboot into the image and well then I got to the grab prompt again and was suddenly faced with the choice between this broken image that I wanted to debug and the broken image plus EGL Utils installing EGL Utils created a new image version and RPMOS 3 was at that time at least configured to always keep two image versions around so I can always roll back to the previous version under the assumption that the previous version is a well working version but now I was already booted into the broken image when I created the new one by installing EGL Utils and so now I had two definitely broken images to choose from which I guess goes to show that I had not really fully internalized what was going on under the covers and I managed to get myself into trouble here and I had only 30 minutes until my video call so what to do I googled around a little bit and another thing that we say about RPMOS 3 apart from that it provides here is upgrades updates is that it's like git for operating system so I figured well if it's like git there has to be some history and there has to be order commits and what not that there's a command called OSTree log which I could run give it the right ref here for my system and it would tell me which commits are actually available and unfortunately it told me that there's only a single commit around only the one that I had my two broken images were based on because file system trees are big and OSTree decides to not keep too much of it around but default it only keeps the ones that it needs so I guess one or two and I googled some more and I found that there is another command that is very similar to git which is called OSTree pull which actually lets you manually download more commits of an existing ref so I ran that with the option dash dash equals five dash dash depth equals five and made OSTree pull the five most recent commits for my image and then I could manually run OSTree admin deploy to switch back to an older commit and I could reboot again and my system was working and I got onto my video call just in time so that was a I guess near death experience with an image based OS just a quick story I think we are almost at the end here there's a lot of future plans that won't take questions while I actually try to get it rebased or higher again alright so yes future plans many of these things have been mentioned before so federal authority when we set out last spring we started pushing Silverblue as a project we made a little roadmap and we said well Fedora 28 that was coming out that spring we'll have a preview release Fedora 29 is when we have the first release and then Fedora 30 it should be ready for prime time so all the major issues we identified at the time should be solved Fedora 30 is coming out in May and I guess these are all the major issues we identified back then we need to have easy support for installing the NVIDIA NVIDIA driver which is a little difficult for image based systems like OS 3 because I mean slash users we don't need to you cannot easily have a different color or things like that so we needed to have some extra support for this thankfully that has recently landed in Fedora so for Fedora 30 it should be very easy to get the NVIDIA driver working and we also have we also have pretty much complete support for Flatpak and RPMOS 3 in the graphical application that we used for installing and updating applications we saw one screen told of that earlier where we showed how flatpak permissions are actually reflected in the UI now and similarly we also have support for RPMOS 3 in there now so it should be a pretty seamless experience from that perspective in Fedora 30 and we'll have the first release two box that I demoed a little bit that will be included in Fedora 30 and it'll be useful and usable on a traditional Fedora system just as much as on Silverblue so I would encourage everybody to try it out just to get familiar with container based development workflows I guess it's a useful tool to play with and yes we have some Flatpaks available out of the box in the Fedora release we did show Flathub we talked about Flathub here earlier one aspect of Flathub is that we cannot easily enable it by default in Fedora because it's a mix of things that are free software not free software that's something that is always a little controversial in Fedora to enable by default so as an alternative we're actually building we're taking a bunch of the RPMs that we have for desktop applications and we are converting them into Flatpaks that we can make available inside Fedora out of the box we can actually use an OCI registry to serve the content it's something that we did not mention before but Flatpak can actually not only use content in OCI form but it can also pull applications out of an OCI registry yeah those were the plans for Fedora 30 these are the more long term plans for things that might happen after this spring for example integration for the toolbox and one thing we also want to do is provide some ready made toolboxes that are solving interesting problems we are working together with students at Boston University and work as interns at Red Hat and one thing they're probably going to do is work on GPU compute stuff so they need things like tensorflow so one idea was that we want to provide a toolbox that is ready made and set up with all the necessary tools to do that kind of development but we want to make more container-wise make more IDEs work well in this container-wise environment, I mentioned Cologne Builder earlier which is already the first container native ID I guess like it already works very well for Flatpaks and also Christian is working on making it work very well with other kinds of containers like potman containers but there are other IDEs out there that are very popular obviously so we should make sure that those all work well in this environment and then there's a whole category of applications that are kind of difficult to stuff in a container because they either because they run privileges or they set up their own sandboxes like Chrome has its own sandboxing built in and then it kind of interferes with the sandboxing that Flatpaks does so we need to find solutions to make those go operate nicely and VirtualBox is another example of an application that has proposed challenges on kernel module and wants to insert that into the kernel which is obviously not very compatible with being on privileged and sandbox so those are problems that we have to solve for the download okay so there's some links but why don't we go ahead and take some questions while we make another stab at re-basing this to Rahide so if you want to hear this alright I'll start with you will it be possible to provision Flatpaks out of a service catalog in a particularly open shift but other Kubernetes based environment I don't know exactly what a service catalog is to be honest but what I mentioned earlier is that we do have support for putting Flatpaks in a registry like a Docker registry essentially and ship them in that form and I'm not sure what a service catalog is like a registry something similar to that yeah I assume that should be possible it's not something that we have solved yet but I'll look into that so you mentioned customized images do the OS tree composes for those happen on Fedora infrastructure or is it like the layering that RBM OS tree does or how do the customized images work I think Josh talked a bit about that currently you can build your own images using the same tooling like APM OS tree the command line tool can also do composes and can build images if you feed it the right config files it's a little obscure and not very polished as Josh mentioned so you have to dive deep and figure out how to do it but then you can run that on your own system and create your own OS tree repository that contains the image that you built then you have to worry about where the updates come from that is very true you probably have to build updated images and also upload them into your own registry just like we do for the Fedora infrastructure any other questions how's it going up there are you making progress there's one more could you tell us a little bit about how the containers work with the local video presentation is it through you think about things like blue jeans and like sharing the screen and things like that so yeah so there are different ways to do it many test applications still use X as their display connections so what you need to do is make the X socket available inside the container and we do that if the application asks for that so the way Flatback works is it has sandboxing built in and by default you get a sandbox that's kind of tight and does not leak stuff into it but then there's a way to poke holes into it like the application ships with the manifest that says I need access to the X server or I need access to the network and one way is to like have an X connection and pass the X server socket into the sandbox or you use Wayland and then we'll pass the Wayland socket into the sandbox no not really it's really just the same way as like if you run a unsetbox desktop application it would also talk to the display server over the same socket it's just that if you're inside the container normally the socket would not be available to you unless we explicitly pass it in and that's part of the stuff that Flatback does to make applications work in containers if you run like a regular docker container that would not happen for you and you would get a couldn't find display connection error message or something from the application and the intro slide that Josh had with Jesse Frazier I mean she has manually like made the right things happen right by manually poking a hole in her docker container and passing the X server display connection in there you can of course do that but there's a lot of manual work to do that for everything that you use it's not just the display connection it's also like the debus and it's the key ring and it's like a bunch of other things that you expect to work and Flatback kinda is specialized for making all these things work alright any other questions? any progress Josh? well people start leaving I forgot that we don't have a lot of silver blue swag but I managed to bring one stack of silver blue stickers if anybody wants one I have the stack up here just come and get one it's like as a little to turn it off alright if you all come back in two hours we can always roll back alright thank you all very much sorry we moved this part of we implementing the faster downloads we moved around where everything is and that was like two weeks ago so stickers oh good oh hey you are bigger stickers I don't have one for display yeah so by the way when I was talking about faster download and install that 3,000 number right there that is a package count to give you an idea of how much faster this is than updating RPM packages in the normal realm in my current yeah installing a mainline then installing after layer with what it does build the binary so part of the idea is to actually the binary is the layer so it goes my directory so user and min and and that's the one that we're going to have to break something away and change it's still going to be a big deal yeah but some of it's like well and that's actually one of the reasons libraries are libraries are a layer and the idea of it is a lot better and part of this is all stuff that I know we're a player against that yeah with the capability in the well yeah there we go once we decided to go with discourse yeah if you just log into discourse get an account and then just enable everything as you know that's the reason why we use discourse we wanted something that was a little bit more phone friendly and discourse actually had to be okay wait wait we're rebooting into raw hide wait for another 30 seconds here oh yeah well until it reboots I can't figure out where I went and looked at the savings down registration desk check the next door at least it's on the premises okay and there we go yay there we go 31 my wife raw hide reporting is 31 indeed how well does this work with regard to the hour hand testing testing we'll be back in the afternoon going out at like 7 yeah welcome everybody to our next talk if any of you here were we're here for the panel after lunch there were a bunch of questions about storage and so we flew in probably the singular expert on storage in kubernetes solidly who's the are you the chair of six storage right your chair of six storage co-creator of the container storage interface which he is going to tell you all about so take it away right thank you very much Josh so like Josh said my name is Saad I am co-author of CSI and co-lead of SIG storage for kubernetes and I work at Google at the moment I want to get to know you guys first a little bit what is your familiarity with kubernetes how many of you know what kubernetes is how many of you consider yourself app developers to deploy applications that you deploy on kubernetes cluster developers who manage kubernetes clusters okay and storage vendors who are interested in connecting to a kubernetes cluster okay it's a little bit of a mix of everything um so today what I want to talk about is the container storage interface the reason that you would want to pay attention to this talk is if you fall into one of those three buckets it's interesting to figure out how you actually get storage made available to your workloads if you are a cluster administrator it's interesting to figure out how do I actually connect with the storage that I have available within my cluster and if you are a storage vendor of course you're probably wondering how do I jump on this container bandwagon and make sure my solution is available in kubernetes so first I'm going to give a little bit of a background then we're going to jump into an introduction of what CSI is and then we'll talk about how CSI works with kubernetes and finally I'll give a status update and please save your questions till the end alright so first off a little bit about the background of what the problem is that CSI solves so let's start with kubernetes kubernetes has a very powerful volume plug-in interface this is an interface that enables block and file storage to be made available on to kubernetes workloads to pods on demand it allows volumes to be provisioned on demand this is something kind of revolutionary that kubernetes started where instead of having to pre-provision a bunch of storage ahead of time whenever a workload actually requests or needs storage kubernetes automatically goes out and creates that storage for you and then makes it available to that workload wherever that workload ends up on the cluster and if that workload gets killed on one node move to a different node automatically moves it over to that other node pretty powerful stuff but it has limitations one of the biggest limitations of the kubernetes volume subsystem was the fact that these volume plug-ins the extensions to talk out the different types of storage systems were all built in to the core of kubernetes they were actually compiled with kubernetes and shipped with the core kubernetes binaries so what's the problem with that there's a bunch of problems it was very problematic for us as kubernetes developers for three big reasons one was security we had a bunch of third-party code we still have actually at the moment a bunch of third-party code that's running as part of core kubernetes binaries so these volume plugin authors, vendors would write code that would be committed to core kubernetes and run as part of core kubernetes and that could you know it's a security issue it could potentially result in stability issues if there's a bug in one of these volume plugins since it's part of the same process as for example kubernetes it could crash a core kubernetes component and quality was our third concern which is kubernetes as a project does not have the facilities to actually test a lot of these volume plugins so for example fiber channel requires very specific hardware kubernetes has an intent test framework but we don't have the hardware or the facilities to be able to test something like fiber channel so a lot of these volume plugins that we were shipping that are part of the built-in core kubernetes are not very well tested we kind of depend on users to tell us when something is broken and have people who are familiar with those plugins come in and fix it so it was problematic for us as kubernetes developers but it was also painful for storage vendors storage vendors hated to have to align their releases with the kubernetes release kubernetes is a beast for those of you who've contributed to kubernetes you know how challenging it can be to figure out what the release cycle is going to be, when the code freezes are going to be what all the tests are that you have to run there are a lot of hurdles and challenges to get involved with that and if your product is completely unrelated to kubernetes it doesn't really make sense for you to have to commit code to kubernetes to make your product work with it and then of course some storage vendors also did not want to open source their code and by being part of kubernetes they kind of had to so for those reasons we were looking for a better solution an extension mechanism for storage for kubernetes when we started exploring this was around maybe early 2016 what we realized was that we weren't alone facing this problem at the time kubernetes was not the de facto container orchestration system there was a lot of other competitors in this space and when we talked to them a lot of them were facing the same challenge we talked to docker, mazo's, cloud foundry all of them had the same problems that they wanted to be able to connect to storage but didn't really have a good way to do so and storage vendors a lot of them were sitting on the sideline trying to figure out what was going on they didn't want to be involved but not wanting to have to build a volume plugin for say docker and then have to go build another one for kubernetes and another one for mazo's it was unsustainable and they kind of just wanted one direction to go in and so that's where CSI was born out of it was a collaboration between different cluster orchestrators saying hey let's just come up with a single standard to allow us to be able to use users so again the goal basically develop a standard for exposing arbitrary block and file storage systems to containerized workloads and container orchestration systems I say block and file very intentionally the reason is storage can mean a lot of things for those of you who are storage experts it is a huge umbrella that covers a lot of things but the scope of what we care about is block and file storage the reason for that is both block and file have standardized data paths in linux that makes it easy for us to develop control path standardized apis that allow us to make storage available to any workload but not have to worry about injecting any code inside the workload to actually do the reading and writing so what this excludes is for example object stories things like writing to a amazon s3 bucket right that is a proprietary data path that is not a standardized gcs google cloud storage has one implementation amazon s3 has another implementation there's a bunch of different implementations there hasn't really been a standard that's baked into the operating system if you want to use object storage you need to have an SDK that you bake into your application in order to use a very specific type of object storage whereas with block and file it doesn't matter what type of underlying storage you use that your application uses or standardized in operating system so that enables us to basically develop a layer of portability that abstracts you away from storage but that's not to say that we want to minimize or make storage the lowest common denominator quite the opposite we want to be able to expose the full power of the underlying storage system but with a standard way to be able to consume it and I'll talk a little bit about that in a moment so what is CSI CSI is just a specification it is literally 45 pages of proto and comments depending on how large your font size is CSI facilitates programmatic interaction between a container orchestration system and a storage system so the key here is programmatic interaction CSI is not designed for you as a user to issue these APIs directly yourself instead you talk to a container orchestration system be it a processor or something else and that system on behalf of you needs a standard language to talk out to storage and that language is CSI CSI is still evolving we recently yesterday added support for resizing volumes and by yesterday we had a launch of CSI 1.1 yesterday which included this feature CSI is of course an open source project and I'll give you some information at the end of the talk about how you can get involved in helping develop these APIs or the implementation of these APIs so what are the primary use cases that we're aiming to solve with the container storage interface they are number one the ability to create and delete a volume, a piece of storage I want a block or file volume that I can use with my workload being able to make that intention known to a storage system and have the storage system return some handle for that piece of storage that I want this volume to be attached to this given node kubernetes and mesos and all these container orchestration systems manage a large number of machines virtual machines, physical machines and we want some way to be able to tell the storage system hey I have a workload that has been scheduled to node B and this volume A I want you to make it available on this node B so make some way to be able to say attach or detach a volume next step is mount or unmount a volume on a node for a workload so this is like the last mile once a disk is attached to a given node how do you actually take that disk and expose it all the way into the container the workload that's going to be consuming it so this mount or unmount call would be a way to tell the storage system hey on this node please make this disk available at this particular path and then we'll take care of the rest of the container now we're starting to add advanced functionality so the first three things are the basic things to make storage available but we're now going beyond that to enable use cases like snapshots so a snapshot is a point in time copy of your storage and the reason it's useful for application developers is you are running a database for example and you want to do some sort of risky operation that you may or may not regret later so what you do is first take a snapshot of what you have do your risky operation and if something goes wrong you can actually undo that mistake cloning of volume if you want to fork the existing data that you have and use it for another instance of an application CSI now supports that as well and then the use case that we're adding right now is resizing which is hey I started off with the volume that's 50 gigabytes I actually need more storage how do I actually expand that volume to be able to use more storage the interface itself is a PC interface and it's defined as protobuf three protobuf services controller node and identity the reason we segmented the calls in these three ways is that we realized that these calls can basically be executed in two different ways either as a controller or as a node if you think about the way that well let's think about a cluster a cluster is a bunch of different nodes and the storage system has probably two components one is the back end which is able to accept calls to for example provision of volume and then a second is some agent running on each one of those nodes that says how do I actually make that piece of storage available on that node so you're going to have a piece of the storage driver running on each one of the nodes as well as a piece of code running somewhere on your cluster that knows how to talk to the storage back end so controller can basically be any call that can be executed anywhere on the cluster because it talks to the storage back end it's not specific to any given node node are the set of calls which can only be executed on a given node so once we've scheduled a workload to let's say node B we can only there are a set of calls like mount this volume to this specific path that can only be executed on that machine that allows us to basically deploy the same volume plugin in two different ways we segment today interface into these separate services and then finally the third piece is called identity which is a common set of calls just to figure out what this driver is and information about the driver and I'll give you a little bit more detail about all of this in a minute next let's talk about naming naming is hard so initially I talked about attach, detach, mount on mount but the reality is that the names of these calls in CSI are a little bit longer and more complicated and the reason for that is that what we realize when we're developing the interface is that attach and detach are very specific operations that may or may not apply same with mount on out there are specific operations that may or may not apply to every single storage system instead what we want to provide is generic life cycle hooks so the key into specific basically points in the life cycle of making a volume available to a container and then it's up to the storage system what they want to do within those life cycle hooks attaches an option but they can do whatever they like whatever makes sense for their storage system we want this to be generic and reusable for all storage systems this is a list of all the methods that are exposed in CSI that they belong to it's a lot of methods let's focus on three first the capability method capability is a way to be able to advertise for a volume plugin what it actually supports CSI basically allows a lot of different use cases but every single storage system doesn't have to support everything a storage system can be very very minimal and support a very small subset of things and the way that the given storage plugin advertises what it's capable of doing are is this these capabilities calls so there's a capability call for each service the first one to start with is the identity service so for the identity service this is kind of like plugin level what are my capabilities the first capability here is controller service so that's basically saying the entire controller service is optional so as a volume plugin author I don't even have to implement the controller service if I don't want to say volume plugin I don't have to implement it I just don't advertise that capability then there is volume accessibility constraints this is advanced functionality if your storage system has volumes that are available to some subset of nodes but not all nodes within a cluster this mechanism allows you to be able to express that in a way that is generic so that the orchestration system can intelligently schedule workloads so let's say your volume is accessible to let's say your cluster is segmented into zones or racks and your volume is only accessible to a single zone or a single rack this mechanism allows you to express that if you don't have any such limitation for your volume plugin you don't have to advertise that next let's take a look at controller so controller let's imagine that you implemented what are the sub capabilities of controller that you can implement the first and most common is the ability to create and delete a volume you might be wondering why the ability to create and delete a volume is optional shouldn't it be mandatory like why is the controller service optional why is the create delete within the controller service optional and answer for that is that again this interface is for the programmatic interaction of kubernetes or another orchestration system with a storage back end it allows kubernetes to programmatically provision new volumes if that capability exists but if it doesn't an end user may be able to go around kubernetes or the orchestration system and manually provision volumes on the storage back end and create objects on kubernetes manually to represent those objects so both those cases are supported we don't want to limit to only storage systems that are capable of dynamic provisioning and then the other capabilities here are things like the ability to create snapshots the ability to clone a volume some of the things that we've already talked about and then on the node side I'm not going to go into the details of any of these if you're interested talk to me afterwards so here's what a sample response looks like so if I'm a container orchestration system I discover a new csi volume plugin I'm going to issue a get plugin info on it the get plugin info response then contains the name of the volume plugin so that I can expose that up to my users to say hey I have this volume plugin that's available to you and some vendor version potentially exposed another interesting call on the node side is node get info so I talked about topology a little bit the idea that a volume may or may not be equally accessible everywhere in a cluster one of the things that we didn't want to do is make assumptions about what the storage system knows about the container orchestration system we wanted to make a strict abstraction between the cluster and the underlying storage so get node info node get info is a way for us to be able to get information from the storage system about a given node what does this storage system how does the storage system identify a particular node the container orchestration system might call this node b but that's just what the container orchestration system calls it the storage system might call it node foo and we need some way to be able to translate that so the first thing that happens when a new volume plugin shows up on a given node is Kubernetes calls this call node get info and finds out what does the storage system know about the container orchestration here is the accessible topology so we talked about the racks versus zones thing this is a way for the storage system to share a set of labels with the container orchestrator to say hey for this given node it belongs to what I call zone foo or you know rack X and Kubernetes doesn't need to understand the concept of racks or zones or anything like that but it still gives the storage system a way to be able to express these concepts in a generic way. So this is getting information about that node once it has that now it's able to actually provision volumes and so yes so the question is if the if a container moves to a location where it the storage can't be moved how does Kubernetes handle that so there's two things there one is if that happens Kubernetes basically makes that pod keeps that pod in pending state saying hey I can't attach the storage sorry but taking a step back if you implement this functionality correctly you won't get into that case because Kubernetes looks at the volume that your workload is using and uses that information as part of scheduling so when it figures out what node to schedule a particular pod it says what are the constraints on the volume that this pod is using and let me use that to figure out where to place this workload so ideally you never get into that case so create volume is fairly straightforward it's the things that you would expect a container orchestration system saying create some storage I would like you to call it this the name is a suggestion the storage system can call it whatever it wants and in the response it gives you the actual handle the capacity is how much storage I want 5 gigs 100 gigs a terabyte parameters is what I kind of alluded at earlier we want to make an interface that enables our customers to be able to use any kind of storage that they want in a standard way but what we don't want to do is make that interface the bottleneck the lowest common denominator for for storage systems there is a lot of differentiation between different storage systems and a lot of knobs and cool features that they expose that don't necessarily need to be standardized but we also don't want to hinder so this opaque parameters blob here is basically a way for a given storage system to expose any number of opaque attributes for a given volume so if I have some notion of let's say encryption or some notion of some performance for a given volume like the ability to control IOPS or locality for a volume or any number of attributes for a given storage system I as a storage system can basically expose those parameters opaquely through the parameters call and that gets exposed out to Kubernetes in the storage class object we'll look at the storage class object a little bit later but basically it has a way for a Kubernetes cluster administrator to set those parameters and define this is the kind of storage I want to expose out to my application developers the other attributes and parameters here are for passing in secret information I'm going to encrypt this volume here I'm going to use the option key the ability to specify a data source you know I want a provision of volume but instead of it being empty I want to provision a volume using a snapshot as a source a provision of volume that's pre-populated and then the topology requirements that we talked about earlier so this is not just intelligent scheduling which we just talked about but also the ability to intelligently provision meaning if I know that somebody is going to want a workload with some constraints influence where that storage gets provisioned which rack or which zone to actually make that decision even earlier so once I have a volume provision the next step is making that volume available to a given node once the scheduler for the orchestration system runs and figures out where the workload should be placed the orchestration system then issues a controller publish call to the storage system to basically say for this given volume the volume ID that you return to me please place that volume available to this given node and that node ID is the node ID that we discovered through the get node info call so this is not like some random node ID that they're getting from Kubernetes this is something that they told us that they self identified with and then finally the last mile is okay I know that this volume is attached or available on this node let me actually make it available inside the container and the way that that's done for mountable file systems is we provide a directory and tell the storage system to actually make that volume available at that path and then we bind mount that path into the container we pass it to the container runtime and say please expose it at this subdirectory and then anything that is written to that subdirectory from the application is then passed out to the storage system and gets basically persisted beyond the life cycle of that container now I talked about that attach or controller publish step but again that is optional so if you're a storage system for whom you don't have a concept of make this storage available to a given node maybe you're running NFS where that concept doesn't exist you don't have to implement it alright so that's CSI now let's talk about how CSI works with Kubernetes this is what it looks like I know it's pretty scary so let's take a step back Kubernetes does a lot of the heavy lifting for you for making Kubernetes work with CSI for the three audiences that we talked about storage vendors focus on how to develop a CSI driver to expose their storage system to Kubernetes Kubernetes cluster administrators focus on hey I want I have this storage let's say NetApp or whatever and I want to be able to make my Kubernetes cluster users have access to this storage so they take the CSI driver and deploy it on this Kubernetes cluster and now users of this cluster have access to that storage and then finally the application developers have the easiest job which is just consume the storage with very portable abstract APIs through Kubernetes let's take a little bit of a deeper look at it and the take away I want from you for here is basically that it's easy for all three of these audiences it's easy because Kubernetes does the heavy lifting so as a storage vendor the three steps for you are one you write an application implementing the CSI specification this is a GRPC interface you go and implement it as an application you containerize this application to make it easy to deploy and then you pair this application with a set of sidecar containers that we wrote the Kubernetes development team and you basically have a set of yamls that you're going to give to the cluster administrators to deploy this is fairly straightforward the asterisk here is that it can get complicated because you have to look at some of the advanced Kubernetes functionalities things like mount propagations and RBAC rules for creating non namespaced objects and things like that so you probably have the most difficult job out of the three audiences but that's okay because this is the value add that you're giving to your users if you are a storage vendor that does kernel modules that makes people's lives harder but kernel modules work with Kubernetes just make sure that you're deploying them through demon sets so that it's repeatable if a cluster orchestration system spins up and you node make sure that the environment there has the kernel module that you want and the way that you make it consistent is by deploying a demon set that will install the kernel module next is Kubernetes cluster administrators you work with your storage provider your storage provider gives you a set of yamls to deploy on your Kubernetes cluster the assumption we're going to make here is that your storage system is up and running and Kubernetes has access to it if you have problems with that work with your storage vendor to figure out how to solve that second is deploying the CSI driver deploying the CSI driver should be as simple as just kubectl apply or kubectl create the set of yaml that your storage vendor provides to you if that fails you let your storage vendor and finally setting up the storage class this is the way that you expose a given storage system out to your users this is what I alluded to earlier with the opaque parameters so if you look at a kubernetes storage class object you have a provisioner field which basically says this is the driver to use and then you have a set of opaque parameters which are the drivers that are specific to this given storage system so in this case I have a driver that exposes a type that allows me to specify the type of volume as an SD that's going to get provision instead of a spinning disk to allow me to control the speed of this particular storage system and if a storage system has a bunch of different parameters and you want to expose different sets of parameters to your application developers you can actually create multiple storage classes for the same provisioner with a different set of parameters and simply give it a different easy name for your application developers to look at so I could create a storage class called fast storage and slow storage both of these storage classes are backed by the same storage system the same provisioner the same CSI driver but they have different parameters that the cluster administrator specified in one case PDSSD another case spinning disk or something else and then that makes the life of the storage or the application developer easier but overall point here is that deploying a CSI driver on kubernetes is fairly easy as well now as an application developer you have the easiest job all the apis that you work with pvc pod all the deployment apis are portable they're not specific to a given storage system they're not specific to any given implementation of a cluster they're portable across all kubernetes clusters so ideally any pvc object that you could use to request storage you can move to any cluster as long as that cluster administrator has set up their storage classes properly that pvc the creation of that pvc object should result in the creation of a new piece of storage for you automatically and then once that storage is created you create a pod or you create a replica set or a stateful set or something using that pvc and kubernetes knows how to basically wire all of that together and wherever your pod gets scheduled the storage will follow and you write to it using standard linux read write file commands and block operations the objects look like this for system volume claim i define how much storage i want the access modes and then optionally the storage class name so i can specify if my cluster administrator exposed two storage classes fast and slow i can select between them or if i really really want to be lazy i can leave this empty and my storage administrator has the option of marking one of the storage classes as default and whatever the default storage classes is the one that's going to get used to provision or create a new volume for you this is going to result in that create volume call that we saw earlier getting issued by kubernetes against the storage system to provision a new piece of storage and then once that new piece of storage is provisioned it's ready for use within kubernetes and the way that you use it is anywhere you have a pod you define a volume you say i want to use a persistent volume claim and the name of your claim has to match the name of the claim object that you have and if you were to take this pod object and persistent volume claim object i want to re-emphasize that they're portable so if you started on let's say kubernetes running on google and you moved it to amazon this would still work you as an application developer don't have to worry about the underlying implementation of the storage system or the cluster it's portable across all of these environments you could even move it to an on-premise environment as long as it's running kubernetes and has the underlying storage class and csi drivers configured properly this i believe is the power and beauty of kubernetes it's the portability that it allows for application developers and i remember your comments earlier about the focus on application developers but it does make application developers lives easier so finally what is the status of this project the csi specification itself was published as a early draft in may of 2017 we released a 0.1 version in december of 2017 the reason for this was very deliberate that we didn't go to 1.0 because what we realized is that it is impossible for anybody to define an interface and have it be perfect as soon as it's released in order for an interface to actually be practical and working you have to have implementations on both ends in our case driver implementations and container orchestration implementations so what we did is release a alpha or beta implementation of the specification have both container orchestrators and drivers implement that revise make breaking changes so for example between 0.1 and 0.2 there was no backwards compatibility we broke the api we made changes and then get to a point where we stabilized the interface and then finally promote it to 1.0 which happened this past december with 1.0 we're offering a guarantee that the specification, the interface itself is stable and we're not going to make any more breaking changes so if you're a story developer and you go and write to this interface you don't have to worry about shit I gotta rewrite my driver in three months because you know something broke your driver is going to continue to work at least until CSI 2.0 comes out so we just launched 1.1 yesterday and 1.1 added functionality resizing is new functionality but it's backwards compatible meaning if you have a CSI 1.0 driver it'll continue to work with newer versions of Kubernetes you just won't have access to this new functionality with an older driver in order to get access to resizing functionality you'll need to get an update for your driver on the Kubernetes side we kind of mirrored the releases alpha support for CSI was released back in end of 2017 as alpha it got bumped at the beginning of last year to beta we added a bunch of functionality since then including remote ephemeral volumes and we finally promoted it to GA in the Kubernetes 1.13 release at the end of last year and we're continuing to add more functionality in the releases that are coming out at the end of this month and all of that is going to be backwards compatible now that it's GA on the storage vendor side there are already over 35 drivers implemented for CSI all the major cloud providers have drivers for their storage systems and a bunch of on-prem vendors including the big names NetApp and such have drivers as well and then other orchestrators who worked with us on the specification Mesos they have an experimental implementation of CSI and they're moving towards CSI soon Docker and Cloud Foundry work very closely with us and they continue to do so they're part of the CSI specification Prover governance body but they haven't actually implemented it in their products yet my understanding is that they're leaning on Kubernetes since they're running Kubernetes as part of their products now in order to to use CSI and then about a week ago I heard about Hadoop Yarn actually implementing Hadoop is not a container orchestrator but the interface is generic enough that they're able to use it to expose storage to their workloads as well that is all that I have any questions and actually before I take questions if you want to get involved there are two communities that you need to be aware of one is the Kubernetes Storage Special Interest Group we have meetings every two weeks if you go to that link at the bottom there you'll find all the information about how to get involved and then the other meeting is the container storage interface meeting to discuss the specification itself and that meeting this is an outer dig it's not once a week anymore it's about two weeks or four weeks we made it less frequent after 1.0 because we weren't moving the design as quickly so I need to update that slide all right Hi I was wondering do you have some generic drivers for iSCSI, NVMe local storage the question is are there any generic drivers for iSCSI, local storage and NVMe so iSCSI there is a generic driver for there's not an NVMe version of the driver yet they're hosted in Kubernetes-CSI and right now there are alpha releases with iSCSI, Fiber Channel and NFS they're kind of community maintained and it's a little bit difficult to get somebody to take ownership for it recently the SIG has been able to find a few people to take ownership of it if you're interested please join the SIG and help us kind of nurture those drivers what we also did was break the driver and the underlying library for those drivers apart such that you could either use the generic driver or you could use the common underlying mount library for iSCSI or the common underlying mount library for NFS to build your own driver so a lot of storage systems are based on iSCSI or NFS and instead of having to write their own mount logic they could basically import that library and build the rest of the CSI driver on top of that instead of using the generic CSI driver for local storage we have an Intree Kubernetes Volume Plugin, local PV right now we don't have any plans to move that to CSI take a look at that implementation but CSI is designed in a way that you could add another implementation of local persistent volume if you wanted to upstreaming upstreaming the drivers to the kernel so these drivers are independent of the Linux kernel are you talking about the Linux kernel these drivers are completely independent of the Linux kernel so they are a user space processes essentially and they're just I have two questions one is simple one again I get confused why this container storage interface is not part of the CIG API projects Kubernetes CIG API so the question is why is CSI not part of the Kubernetes CIGs the reason is CSI is actually not a Kubernetes sub project it is a standalone project so it is it was built out of a collaboration between Kubernetes, Meizos, Docker and Cloud Foundry and was founded as an independent project between those four organizations right now there is the governance of it is that there is an agreement between those four to eventually incubate into the CNCF so it will become a standalone project of the CNCF okay thank you the other one is hard one you skipped over the architecture of the CSI right is it possible to just quickly explain this left that slide in case anybody asked that question you need a GRPC let me jump into that after everybody else's questions and then I can just go over it with the people that actually care about it you said it is not going to be implemented with object storage type but I saw CepFS and Gluster and other things so are we getting there somewhere a little bit so Cep and Gluster expose file block and object file and block are the APIs that are going to be standardized for Cep and Gluster object are just going to have to use it directly more questions? alright so for the rest of you if anybody is interested in a deeper dive of how the Kubernetes system works with CSI feel free to stake around otherwise feel free to head off and thank you very much for your time alright so the green box here is the CSI driver so I talked about if you're a storage vendor you implement the CSI interface and you containerize it so that green box actually is a container that is the CSI driver the next step is deploying in Kubernetes so Kubernetes has a master and Kubernetes has a number of nodes actually it has potentially multiple masters as well but we'll ignore that detail on the nodes you're going to have a demon set pod running that demon set pod is going to be the CSI driver written by the cluster orchestrator and a sidecar container called the node driver registrar which knows how to interact with Kubernetes and then the creation of this demon set object what happens is that this node driver registrar drops a unix domain socket onto the host file system into a specific directory which is a Kubelet plug-in something and the Kubelet which is the node agent running for Kubernetes and that's what's responsible for starting pods and things like that for unix domain sockets when it discovers that unix domain socket there's a special interface that is between Kubelet and this node driver registrar it's not standard CSI this is generic Kubernetes device plug-in registration protocol that basically negotiates between the driver and Kubelet to register it with Kubernetes to say hey I'm a CSI driver and please register me and make sure that if anybody wants to use me this is how they do it there's a second unix domain socket which is dropped from the actual driver onto the host volume the path for that is passed through this registration mechanism to Kubelet and the Kubelet CSI component takes that path and says okay any future calls that I need to make out to you I know where you live and I will issue those calls through that unix domain socket and then this node driver registrar kind of gets sits out of the way it's not a proxy or anything like that Kubelet will issue calls directly to the CSI driver through that unix domain socket so that's for the node component the second component is the master sorry the controller component the way that you deploy this is a replica set or a stateful set size one that means that there's only one instance of this thing running on the entire cluster again it's your driver the same container that you implemented over here bundled with another site car container actually a set of site car containers you have the external attacher, the external provisioner external snapshotter and the interesting thing here is Kubernetes does not directly reach out to the CSI driver directly the reason for that is because what we wanted to do was often times the master is not a schedulable node right what that means is that it's not a schedulable schedulable node so we didn't want to be able we wanted to be able to allow users to install new storage systems on their system without having access to the master let's put it that way so what that means is a user only has access to deploy workloads on their nodes they don't have access to the master so what they can do is deploy this stateful set onto one of the nodes that they have access to and still have all of this work but because they deploy this controller on one of the nodes it becomes a challenge for Kubernetes to reach out and figure out where the controller container is and talk to it one approach would have been to expose this as a service that the master could then discover and then call out to it comes with a bunch of challenges including how do you secure that how do you ensure that that service is only accessible by the Kubernetes master and not by anybody else so instead of introducing all of these discovery and kind of security issues what we decided is to flip the script if you look at all the Kubernetes components and how they work, how kubelet works how the kube controller manager works all of those components they don't have a central Kubernetes entity reaching out to them to tell them what to do instead they monitor the Kubernetes API server and they automatically react and do things so the controller basically has a set of sidecar containers which are essentially Kubernetes controllers they monitor the Kubernetes API server and then they poke the CSI sidecar container to invoke certain functionality so if you look at the external provisioner sidecar container it has a loop that is monitoring the Kubernetes API server and it's looking for persistent volume claim objects that point to a storage class and then the CSI container is the same name as whatever the CSI driver is named when it finds one of those PVCs that's a signal for this external provisioner to say alright I have to provision a new volume so it's going to internally issue a create volume call through a Unix domain socket which is inside this pod to the CSI container to say please create a new volume and then the CSI driver container is going to return a handle the CSI driver is going to return that handle to the sidecar container it's going to be the response to the create volume call and then this external provisioner container is going to take that information and generate a Kubernetes persistent volume object to represent that new volume in Kubernetes and then Kubernetes says ah okay I see that you provisioned a PV object that's pointing back to this PVC that the user requested let me bind them together now you have a PVC repair that is usable within Kubernetes so now you can the user can create a pod that points to that PVC object and Kubernetes says okay I have this pod I need to go and schedule it somewhere okay I've scheduled it to node B I need to make sure that it's attached to node B so what is it going to do? Kubernetes is just going to create another API object and it's going to create an API object called volume attachment on that volume attachment object it says make PVC such and such available to Kubernetes node whatever and then the external attacher sidecar container is monitoring the Kubernetes API it's going to see this volume attachment object it's going to look at the PVC object and it's going to de-reference the persistent volume and say ah that PV belongs to me I know how to handle that attach operation let me go ahead and issue a controller publish call it's this CSI driver through this internal UNIX domain socket and then the CSI driver says okay I know what this volume handle is I gave it to you I know what this node ID is I gave it to you let me make sure I do whatever I need to do to make this volume available on this node so it'll issue some back end call if you're running a cloud provider for example to attach a volume to a given node and then it'll send that response back to the external attacher the external attacher then updates the volume attachment object and then it's going to say I'm done this thing is attached and that becomes a signal for the cubelet the cubelet like every other component in Kubernetes is just watching the API server so it's saying okay I've got a pod that's scheduled to me that pod is referencing a PVC that PVC is of type such and such and I know I need to wait for this volume attachment object that's associated with that PVC to be ready so it's monitoring that volume attachment object as soon as that gets attached that allows the cubelet to say okay I know that this volume is now attached to this node now I just need to make sure that this disk or this volume is available inside this container so it's going to issue a call out directly to the CSI driver that's running locally on that node to say hey this disk this volume ID that you understand please make it available to this given path it could be var lib, cubelet, blah blah blah and then the CSI driver says okay I know what I need to do in order to take a volume and make it available to a given path in my system I will go ahead and do that it does it inside of the container there's mount propagation so that any mounts that happen inside that container are propagated down to the host and so the path that Kubernetes passed in is now actually backed by the volume so then Kubernetes takes that path passes it to the container runtime and says please go ahead and start this container but also make sure that this path is exposed at this path inside the container and then the workload starts and the user's application starts and whatever this path is if the user goes and writes to it it will automatically write out to whatever persistent volume this storage system is backed by and then if the pod dies for some reason either the user deleted it or there was not enough resources on the node for other reasons the pod is terminated, Kubernetes says ah ok I know that this pod should not be running I see that it's currently running let me make sure that I safely unmount the storage for this pod so it's going to issue a call out to the CSI container to say hey I don't need this anymore node unpublished such and such volume the CSI driver says ok let me clean that up and remove those mounts and then the kube controller manager says this volume is attached to the new node so it's going to issue a detach operation the way that it issues a detach operation is it takes that old volume attachment object that it had and it just deletes that the kubernetes external attacher sidecar container notices that deletion and says oh ok I guess you don't need this anymore let me issue a controller unpublished operation against the sidecar container and that controller unpublished then results in a detach of that volume from that given node and then the kubernetes controller manager says ok I need to make this available to the new node that this pod is scheduled to and then it's a rinse and repeat of attach, controller publish, mount and node node publish so that's kind of how it works so CRDs are so the question is can these sidecar containers external attacher external provisioners be implemented as CRDs CRDs are custom resource definitions they're a way to extend the kubernetes API so kubernetes ships with a set of API objects like pod, node and a bunch of other objects CRDs are a way for you to basically specify your own objects and more and more the project internally is relying on CRDs to extend the kubernetes API instead of introducing new built-in types but the API objects alone don't do anything they're just a object something needs to operate on them so in this case the external attacher and provisioners are the controllers and the controllers basically operate on these objects the witch controller? Sure yeah I mean you could define a CRD and a custom controller that does the same functionality absolutely but you don't have to because they're built in so now instead of if you're a storage vendor instead of having to create a new CRD to represent what an attach is what all of these things is and figure out how to make kubernetes emit that and then write a controller to be able to read that and monitor that and then issue the CSI call you don't have to do any of that you just write this interface, the gRPC interface focus on the things that you know which is calling out to your backend and you pair it with these sidecar containers which are basically the controllers that operate on the objects that are already built into kubernetes so the question is how you make sure these containers the sidecar containers and the driver are deployed and how they're running and making sure that they're running so the cool thing is that the CSI driver is deployed just like any other workload on kubernetes so the answer to that is how do you make any workload in kubernetes that's made up of multiple containers ensure that it's deployed ensure that it's running and all of these things in this specific case we use a stateful set which is a way a kubernetes object that lets you say I want to have this a certain number of these pods running all the time so in this case if we create a stateful set of size one that means that kubernetes will start a pod which is made up of multiple containers a pod is a basic unit of scheduling on kubernetes so it ensures that all of these containers will be scheduled together if a given node only has room for one container it's not going to get scheduled we make sure that there's room for all the containers we schedule them together we start them together and if something happens that the pod crashes the containers crash the stateful set kubernetes has controllers that are monitoring the stateful set object it realizes that the container crashed or the pod got killed and it'll create another instance of that pod to make sure that this is always running so the cool thing about all of this is that we're using kubernetes to extend kubernetes these CSI drivers are just like any other workload that you're deploying on kubernetes and this is an interesting point because there's another extension mechanism an older extension mechanism called flex volumes and that was a exact based extension mechanism it implemented mount, unmount, attach, detach but those were implemented as executable binaries and in order to deploy those systems you actually would have to SSH into the node or the master and deploy those binaries or the script onto specific directories within that system so this is problematic for multiple reasons one being just because it's so damn difficult to deploy you actually need SSH access to all of these machines you can't just use kubernetes to deploy it and then of course if you bring up a new node you now need to make sure you have some mechanism to be able to deploy the driver on that node whereas with kubernetes if you deploy it as a demon set as soon as a new node gets added it automatically gets the CSI driver deployed on that node and for some reason there was some repair operation or something that happened that took the node down and brought it back up so with the flex you would have to detect that and make sure you fix your driver whereas with kubernetes you have the demon set pod it will get started, it will do the test to make sure everything is okay and deploy the correct bits in the right places sure and then we'll go back there sure so the question is did you use gRPC for performance reasons or what is the reason behind it it wasn't performance reason it was just convenience it makes it dead simple to be able to implement new interfaces it's relatively stable that was the main driver so what kind of attributes are you thinking about yeah sure so it sounds like the question is given the ability for a storage system to basically advertise how to segment a given cluster let's say zones or racks or whatever is there some mechanism to allow the workloads to basically use that information in scheduling to have some subset of pods go towards a specific wrapper zone and the answer is absolutely yes the mechanism that kubernetes uses to express this topology information is labels just labels on nodes and so what ends up happening is when we get a set of labels from the storage system we apply those as labels to the node so if you want to operate on those for your workloads you simply create selectors on your deployments using those nodes to do whatever you want okay and we are out of time so thank you so much we do have one more session in this track Sonia is going to be coming here in half an hour to talk about container security so come back for that I will not be taking the full hour yeah yeah hello hello hello hello okay we are on we are not on yet hello everyone for our last talk of the day but not the last talk of the track we have three more talks tomorrow we are going to have Sonia Kaptev who is going to talk to us about container security Sonia works for twist lock we have been doing container security stuff as long as Docker has been a thing so they have a lot to say about container and infrastructure security and she will have a lot to say about it couple of things just before we get started, number one is we will be having a Kubernetes birds of a feather in this room immediately after her talk you are all welcome to stay for that we will be talking about asking people to discuss what they are doing with Kubernetes what their problems are how they contribute that sort of thing and the second thing is because we have a lot of die hard people with the track just you are aware of the last talk in this track the 430 talk tomorrow at Sunday we have had to cancel because the speaker got sick so and sadly that one won't be happening we will be announcing that later in the meantime let's learn how to secure our container infrastructure so thank you thank you thanks so much I am really excited to be here so super excited this is going to be a basic talk but just for a show of hands so I can engage the audience a little bit who is everyone is doing something with Kubernetes these days but who is doing something experimental who is full bore into production so for the folks that are in production hopefully you have gotten all of the things that I have mentioned down and they are not a problem so this will be a review for you apologies for that but for those of you who are experimenting thinking about deploying containers cloud-native technologies hopefully this will be a good run-through of things to think about when deploying Kubernetes and containers into production and we will talk about a couple of things so what we will cover today we will do a quick overview of securing a host and what that means we will talk about secure deployment we will talk about the integration of Kubernetes itself and then we will talk about shift left container security anybody familiar with shift left or the concept of shifting left and then we will do a Q&A so the one thing I am not going to cover extensively is Docker security although if you are doing Kubernetes then obviously it is a follow-on but I feel like each one of these topics could be a talk in and of itself and I am hoping there is something covering Docker security as well at the conference maybe not okay well next year maybe I will be back and cover Docker security or I can point you to a couple of resources that we have as well that covers that so let's dive in so we are literally just going to dive right in securing the host hopefully these are just some of the best practices and ways to ensure that what you are doing is deploying Kubernetes in a secure way how many are building their own so you are basically public cloud and spinning up a VM setting up the OS deploying Kubernetes just a couple alright and then how many are using managed so we are more on the managed stuff that is fine the thing to note is that I think all of these still apply and so please don't take for granted that just because you are a managed Kubernetes cloud provider has done all of the good things that they should have done because obviously it is going to very much depend on what it is that you are deploying and what you are doing and how you are doing it so these very much still apply even with managed Kubernetes you are not physically installing Kubernetes itself on your VM so minimal install of the OS there is definitely ways and things that you could put in there that would bloat it and also open up additional attack vectors that you don't want so the first thing is making sure that you are using the very minimal set of services that you absolutely need in order for your app to function and nothing else the next one down is update your packages so the packages that you are using making sure that they are completely up to date and staying up to date on them one of the biggest things that we see in the cloud native security field is that packages get updates or there are new vulnerabilities that are released in the cloud I should say sorry and folks don't update their packages they don't update the version that they are using of those dependencies and that is one of the major attack vectors that we see another big one is development teams will deploy various cloud native technologies and then leave them running and forget about them and you are just begging for somebody to come and infiltrate and then you are going to have to have to be there and who has access to it as well as I mentioned remove any unneeded services so anything that is running that doesn't need to be making sure that that is shut down and not open close any open ports that you don't need another massive one that we see and we will dive into it a little bit more in detail when I cover some of the other best practices but locking down the ports that are used if you just leave it open which is very much the default then you are just asking somebody to start scanning all of your network ports that are open to the wild and start plugging away at your host secure your logins by certificates so having a separate certificate store that then allows login access is ideal and then definitely making sure that you use least privilege and so unnecessary logins aren't available to folks and then intrusion detection systems are great definitely alerting you to when something is noticeable or when someone does attempt to access something on any of your hosts and that will give you some preemptive knowledge hopefully before they are able to access anything within your host so then securing Kubernetes again if you are already doing something with it then these are pretty much review things that should be things that you are doing all the time anyway using RBAC RBAC is great it's great but it's also complex so again back to the whole least privilege concept which is a best practice in any type of development but especially cloud native making sure that you know exactly what roles you have enabled what cluster roles you have enabled what users are then bound to those roles and that no one has access to anything so that's definitely something that you want to sit down and plan out and we'll dive into that a little bit more later when we talk about shifting security left but this is not a trivial task right this isn't just oh well I have my administrators group and I'll plop some of the key folks in there and then I have my developers and they'll have access to the development environment like there's definitely a lot of very intricate design that needs to be thought out in advance especially for name spaces and how those roles and cluster roles relate to your name spaces if you've got a dev stage prod environments throughout then definitely making sure that ideally just service accounts have access to your prod environment and developers have access to just development environments so take RBAC seriously Layer 3 protection is also crucial so Layer 3 is your subnets it also determines how different pieces of your cluster the different pods and nodes all interact with each other and how they communicate together this is another thing I'll show you a little bit more detail on but when we get into some network policy things but Layer 3 protection is absolutely crucial as well because if somebody can infiltrate your subnet and I have a great example of that in a second then they can absolutely dive in on your network and then potentially find anything else that's exposed and so this is kind of like a rolling up type of scenario which is why all of the best practices mentioned on the previous slide really relate because as soon as they infiltrate your subnet then they can do a whole host of other things if you haven't done everything else upstream we'll get into how to configure the master nodes and the worker nodes and some best security practices there I'll also talk about at CD and secrets injection so with RBAC I think we covered a lot of this wireless security model that's additive so there is no deny at all so it's not like you can deny someone from doing things you can only give them access to do things so keeping that in mind as well as you're going through your RBAC strategy is key to making sure that you only add in the things that are necessary and we covered that roles and role binding so obviously you're going to define the role itself such as a reader role or an auditor role and then secure the role binding to particular users and within that role binding you also indicate the namespace that's used and the specific user so making sure that all of that is aligned with what you want it to be and again back to the basic denominator of least privilege across the board same thing with cluster and cluster role bindings this is another super key point no you shouldn't open up cluster access to just anyone it should be absolutely mission critical that they have access to the entire cluster we'll get into some federations stuff as well but this becomes even more so important if you do if you are in a multi-cloud, multi-environment HA type of scenario where you have multiple instances of your cluster running that's just asking for trouble limit access to your nodes we covered this already and then access to containers in the cluster should be able to do that that's just a general best practice that we've seen once you enable access through other methods or APIs and especially in the right scenario then again you're just asking for people to be able to hack into that so network security I touched on this a little bit it's a layer 3 segmentation making sure that only the services that need to talk to each other talk to each other on particular subnets and then employing Kubernetes network policy which is actually super intricate and I want to pop out of here real quick and show you some of the docs if you haven't seen this or for those that are again playing around so this is what that API would look like for the network policies and what you see is you've got the ability to set the namespace the pods and the ingress in the egress which is super important so this limits how any of the pods talk to each other and if this isn't employed I believe it's all just open and we'll get to how to set it to not be open by default but this indicates exactly what IP blocks to can communicate with which pods on which ports again getting back to the basic principle of locking down all the other ports unless absolutely necessary and then validating this with the network policy using only the ports that you need to in order for the containers to communicate between themselves so ingress is traffic coming into and egress is traffic coming out of this is also key because you want to make sure it's expected that certain things that are inputs to particular containers are only allowed in and you don't have files randomly being copied out where you don't want them to be so definitely check this out the network policies is pretty cool real time network profiling so this is something like network scanning or even port scanning there are tons of tools out there that will do this for you and it can effectively show you any kind of vulnerabilities that exist any ports that are randomly open for whatever reason again kind of onion layering right if you have to see them potentially it's even more important to have network profiling of some sort so that you know exactly what's going on with your network and when and then automatic rule creation this can be something that's slip streamed into your CICD process that we'll get into or worked into your pipeline for example where as you add new components to your app there's some sort of automatic rule creation that happens to make sure that you've got your RBAC configured and it's doing exactly what you want it to be doing from a network perspective so CIS benchmarks are folks familiar with these yes I've got one head nod and one hand raised awesome so CIS benchmarks are a very good thing they apply to the Docker engine itself they apply to master nodes and worker nodes and I've got more in here so I'm just gonna pop out again sorry and show you this so there's CIS Kubernetes benchmark and they're effectively if you're not familiar there's scripts that you can run to make sure that you're implementing best practices from a it's an industry just standard basically so there's scripts that you can run in your environment to make sure that you are applying to those benchmarks there's one for Kubernetes it's here on GitHub and then there's one for Docker same thing just scripts they're just best practices you can download them and then just run them in your environment so for the master node configuration any configuration changes to the master node can be done via the apiserver.yml file some basic examples of things that you want to make sure are turned off is anonymous access so it will default to anonymous access if you don't indicate where you're looking for traffic from and so if it can't authenticate the source of the traffic or there's no like multi-factor auth or an auth provider indicated it will default to anonymous if you've got the set to on so turn it off and then insecure bind address this is going back to that network policy example that I showed you in the docs if you don't set the bind address then it will basically be open to insecure bind addresses and you want to make sure that that is disabled otherwise on the ingress you can have folks coming in and potentially doing things to your containers that you don't let them doing the recommendations also to apply the cluster to apply this to the cluster Federation API server as well so what the cluster federation service will do is basically allow you to federate your cluster across multi-cloud environments and so this is great if you're using it for HA high availability or disaster recovery for example or things to lower latency if you've got a global user base which at this point I think everybody does so federation is fantastic just making sure that it's locked down and this again goes back to that cluster role and role binding that we talked about if you're open at the cluster level to a particular thing well then it's not just in one cloud of that cluster instance that you're susceptible to attacks but now that's your entire cluster across all of your multi-cloud environments so this is something to definitely watch out for and lock down at the federation level for the worker nodes some best practices is making sure that the kubelet or the kubernetes agent is configured properly some of the basic examples here and again there are a ton so as you're going through and figuring out what your security policy and approach is going to be depending on what your app needs to function a couple very quick examples of things to make sure that are configured is allowed privilege is set to zero so allowed privilege will basically allow privileged containers to run in your environment so containers that potentially need root this is bad don't do it so set it to zero and this way privileged containers cannot run on your cluster disable seed visor so seed visor gives you a ton of information about your entire kubernetes deployment like what pods you have running what their namespaces are what your RBAC configuration is what your network policy is so it's basically like a map to your entire kubernetes deployment turn it off I mean in development it might be useful in fact it probably is very useful in development as your developers are getting up to speed on exactly how to deploy the app how to configure, how to optimize it but as you move through your deployment pipeline making sure that it's set to off and we'll get I'll cover this a little bit in the shift left as well as it relates to the different stages of your deployment pipelines and some ways to make sure that you're doing you're implementing best practices throughout that process and then finally ensuring proper permissions and ownership to configuration files this is one step further than just the RBAC conversation so the configuration files basically are the heart of exactly what's happening and how it's happening to your cluster and all of your nodes and pods so making sure that only the people that need access or only the services that need access to it for whatever reason have access to it really as you get into production that should be extremely minimal if at all locking down at CD are folks familiar with that CD? yes good okay so obviously it holds a ton of sensitive cluster information so and it's unauthenticated by default that's awesome so making sure that no one it is? do we know when that's changing? tomorrow yeah go home and work on at CD yeah okay the comment was that the unauthenticated status of at CD by default so keep an eye out and know when that's happening because it will likely break you it's also recommended that if you are and I'm assuming everybody is using at CD that you put it behind its own firewall again generally best practice in making sure that all of the good network ingress egress policies are set for at CD as well yeah kind of basic secrets injection don't include secrets in your images this is another massive one that we see from a lot of enterprise companies people just leave secrets there either from the development process or because there's not good hygiene for checking exactly the structure of the images that are deployed into production and so the secrets are left in there they're not encrypted bad things happen use a secret store if at all possible there are plenty of options here you can use the secrets injection at run time or at startup and then as files are in very environment variables that will then dynamically get injected in so you don't have to worry about just leaving them there rotate secrets often this is another good practice we've also had a bunch of customers that have left secrets there for months and then close to years now so make sure that they are rotated and that you have new secrets and then entering that secrets are encrypted at rest is important as well I don't think we've had anyone hacked who had their secrets encrypted at rest so that's good news and then encrypt them from view when a pod is inspected another good reason to turn off the advisor but not just the advisor whenever you're doing any kind of pod inspection and there's plenty of tools that will do that for you to be able to read them if they're not encrypted some basic practices around secrets on that note so who saw this article one, two, okay couple of hands here we go yeah so this goes back to a bunch of topics that I mentioned definitely are back right definitely least privilege encrypting secrets even though this wasn't necessarily secrets but the theory at this point is that they just did a password spray approach and took two to three very very common passwords and were able to try across a large number of usernames and hit bingo so another good thing to note another way that this kind of cements a lot of the information covered is making sure that you have the right gene practices just on basic usernames and passwords because once you get one infiltration or one attack vector that's open this happens so don't do that okay so transitioning a bit from best practices to what else can be done apart from very specific kubernetes security best practices is obviously securing the images and as I mentioned I'm not going to dive in into any of the other best practices pieces but a lot of the same concepts apply and securing the containers as well and then also securing the entire CICD pipeline and so that then brings in the concept of shifting security left and so what that means for those that aren't familiar is moving security practices left so earlier in the software development life cycle with the goal of shifting from reactive to proactive security posture and so what we've seen is that traditionally security is something that's thought of after it's something that the security team gets pulled in once the app is built and ready to be deployed and it's potentially just inspected for some basics and then either shipped or given back to the development team to fix those couple of basics and then deployed and so what this ends up doing is it doesn't allow for the developers to be part of that entire process but also what it does is it leaves the app open to even more attack vectors or attack potential because security doesn't necessarily understand all of the intricacies of the app and what we're also finding with a lot of our customers is that the security teams aren't necessarily as well versed in cloud native as the developers are who are actually building the app in these new emerging technologies and so what's worked really well is and the security folks start that conversation for the development process and making sure that the security folks understand exactly what the app is going to do how the different components of the app need to talk to each other and helping the developers actually build that into the design of the app also a lot of the security tooling that we found doesn't actually surface any of the basic security information in a very dev friendly fashion so it's effectively like the two teams that are speaking in very different languages and very well so what we've seen work very, very well with a lot of our customers is actually getting both of the teams in the room early on another great way that this has functioned is that security is then actually able to work with the development teams to set control gates so basically thresholds that allow the app to move from dev to test to staging to production and then they're also able to monitor in real time and apply any updates or changes that need to happen as well as address any new vulnerabilities that come out and so that way everyone's on the same page as far as what needs to happen to make sure that everything's functioning well in production and nobody's open to any kind of attacks like every lot of this so security by design is really important and it's also important to make sure that all of the things that you have here at front automation's been a crucial key as well whereas much as you can automate these things the better and the less room you leave for human error as much as you can automate your network policy as much as you can automate your RBAC configurations and keep those up to date as folks come in and out of the teams as I mentioned control gates are your friend so making sure that those thresholds are established early on in the process and they're known to everybody on the team that generally helps to make sure that everything's moving along smoothly and you don't have for example some regression fixes that need to happen right when you're ready to deploy and potentially extending your timelines share the tooling don't silo information the more information that both teams can share with each other the better and I think the more understanding it fosters across the board and it makes it easier for everybody to do the job that they're meant to do again getting everybody in on the process early is really helpful and then using common expectations for better production protection so what do we expect to happen if we find something in production what is that process going to look like and not just saying well we did all this good work up front really invincible once we get to production everything's going to be great but really setting up that process in advance to know how it's going to be handled and potentially if there are any intrusions that are detected who gets to do what and how that's root cause analysis and all of those standard processes that would happen anyway and making sure that everyone's on the same page there so containers are great and they definitely empower security teams to shift left successfully than traditional architectures we find that it is easier to get security on board and everybody deployed now into production if we get security involved earlier on in the process and containers inherently are more secure than traditional software development so that's always great but definitely some education is necessary and making sure that the in-house security team is up to date on exactly why that is and how that is and they know how to apply security policies that they've been applying for years now to cloud native I think I would be preaching to the choir here if I covered this but I'm sure you all know the benefits of containers and then security for the entire CI CD pipelines is some things to consider if you are doing this manually or if you're looking at a tool for example to do some of this for you is definitely throughout your build ship run process there's certain things that are going to be crucial to deploying a secure app so things like vulnerability management compliance should be very early on in the discussion, the design of the app compliance is things like PCI or GDPR all of those famous acronyms that we know and love making sure that the app not only follows best practices that we've covered for some basic security things but then when you layer the complexity of compliance on top of that as well some shifts that could happen along the way so making sure that's very early on in the design process vulnerability management is just general knowledge of and mitigation of common vulnerabilities that are out there and making sure that you're staying up to date on those as well there's over 30 close to 40 now industry standard sources for vulnerabilities if you're doing it manually there's definitely a ton of information to comb through and understand and they're getting updated every day it was just two weeks ago now that the big run C bug was found that affects all containers everywhere so things like that are really important to be sure that you're up to date on and then as you're moving into shipping and then obviously when the app is running in production cloud native firewalling so firewalling as it relates to cloud native type technologies taking things into account like the network policy stuff that we covered so firewalling off at cd and things like that that can become quite complicated to manage runtime defense so back to that intrusion detection type of thing and what happens when someone tries to hack into your containers and obviously access control is layered on top of that with our back being one thing but then also authentication as well so just some things to think about from the security perspective so it's a little bit shorter but happy to take any questions question why is it called cloud native firewalling why is it called cloud native firewalling well because it's a little bit it's the same similar concept as basic firewalls that have existed for decades but now it's much more concentrated I would say concentrated right because you're not doing like a vast perimeter firewall that we used to do basically firewall your entire server now you're firewalling individual services for example more questions why not considering that Kubernetes was born in the cloud during the time of security why wasn't security first our back is a thing you add if you'd like it but Kubernetes is meant to be run locally it didn't come local until later it started in the cloud no security Josh do you want to take that one I can take that one first of all let me tell you right now at this point our back in Kubernetes is mandatory so you can't actually not use it you can obviously deploy an our back profile that gives everyone permission on everything but you have to actually do that let's just say that early in the history of container applications security like general infrastructure security was a little bit of an afterthought early versions of Docker had no security anything either yeah I don't know what Docker's doing these days and the other influence for what came to Kubernetes was Borg from Google which was behind several layers of corporate network security so the initial thinking of the people who were developing it was to look at Kubernetes security as a network security problem as people moved into production or towards production with our first versions of Kubernetes I both Google and Red Hat Red Hat had been pushing for some of this stuff for a while because SE Linux so basically you had Google looking at it as a network security problem Red Hat looking at it as yet another opportunity to deploy SE Linux and nobody looking at building the security controls into Kubernetes itself as we got people running into production it immediately became obvious that that was not a good approach and that we needed to include our back we needed to include actual encryption of secrets we needed to include mandatory authentication and a lot of services the you know for that matter I think it was only in 110 that encryption of connections to the API became mandatory before then it was optional and honestly it was optional because in early hackish versions of deploying Kubernetes it was actually very hard to set up right and so we didn't want to make it mandatory if it would be a total roadblock to anyone deploying Kubernetes at all now there's tools to set that up which means we can make it mandatory and it is it's also a fairly common design model to identify what a project is not going to address early on especially in the cloud often times things like security or back end storage are clumped into something somebody else will do I think too just so cloud native came out of the desire to get things out the door faster right and Josh feel free to jump in and correct me where it was developers wanting to get things out as quickly as possible and so it was very much an afterthought it was the how do we get out into the cloud quicker faster spoken like a true infer guy it was piggybacked off of the pretty much the same the same question of why do we have another question I'll be right with you you talked earlier about the difficulties or challenges around cluster role binding and role binding do you have any tips for designing that or what are some concerns you think of when you're starting that project concept number one is lease privilege lease privilege everywhere so just as a best practice just say no say no to everything except what's absolutely crucial to keep things running and we're working our way up from there so if Mary Jane doesn't need access to your cluster don't give her access to your cluster same thing with service accounts unless it's doing something very mission critical it should be a very specific task and it should be defined very explicitly I would say start there any other questions as you said even though we implement this all vulnerability management in the build phase it is still possible that vulnerability or exploits can be found after all the project is done is there any like IDS native to Kubernetes that we can keep monitoring our Kubernetes in containers so we can react to the malicious activities I don't think there's one native as in built in do you mean built in that can or both malicious activity or also the at vulnerability I don't I mean there probably is stuff for malicious activities I don't actually work in the security field myself I don't believe there's one built in there's nothing built in and earlier this week I was at RSA and there was an entire show floor of various cybersecurity solutions but I would say that the market is saturated with IDS solutions is to say the least there's a ton of different options out there depending on what you're looking for the tool to do for you I can't recommend one in particular mainly because I'm biased but I would say definitely take a look and see what it is that you need and what you require and what level of coverage you're expecting things to your point about vulnerabilities that come out post do we critical to keep an eye on new vulnerabilities if nothing else that runs the vulnerability taught us that two weeks ago but in general a good tool will give you insights into when new vulnerabilities do come out it will also be able to tell you which components of your cloud native environment are vulnerable or susceptible to that vulnerability and exactly how to mitigate it so if it's something like hey go update this version of library X to this particular version where that vulnerability doesn't exist if it's something where the vendors already patched it and released the next version that's ideal often times you'll get vulnerabilities that are fresh and so the vendor hasn't even had time although you generally won't get details for about a week to two weeks that's the general grace period for vulnerabilities to have to give the vendor time specifically to release an update to that package ideally not opening up all of the customers that use that package to that vulnerability if everything's public is it 90? depends on who you ask one thing I will mention one of the areas that there has been some focus on vulnerabilities is on the container registry level so both the quay and harbor container registries include image scanning that will scan those registries for known security vulnerabilities usually by matching them against CVEs known package profiles and that sort of thing I mean that's obviously only one piece of a much larger security picture but it's one of the tools another quick just add on to that while Josh is walking another thing that we've seen and as silly as it sounds is we've seen folks download random images off of public registries like Docker Hub make sure you know what's in the image before you deploy it right again it sounds silly but we've had a few customers caught by that where my favorite story to tell is actually there was a legitimate library that somebody was using open source the maintainer couldn't maintain it anymore so we handed it off to a new maintainer that new maintainer had other malicious intent for it and so published a minor update with the malicious code and then published a major update and so everybody went for the major update which now then included the malicious code and here we are so just no and that's why it's just one line item that says you know check your packages make sure you know what's in them but check your packages make sure you know what's in them Hi similar area of question like so can you give us any idea about the runtime defenses like is there any provision for Kubernetes or something baked in that allows you to react as well as require from attack again this is back to the ideas question I don't believe there's anything baked into Kubernetes to do there is oh okay you're saying yes to there's nothing baked in okay there's nothing baked in to do any kind of runtime defense so again that would be a whole host of various tools that'll give you different levels of coverage any of them will give you alerting so they will let you know post attack that hey you've been attacked which is not very useful because now you're attacked there are some that will do preventive or they will do proactive blocking and prevention of those attacks so as soon as they I mean ideally what they'll do is they'll know the exact landscape of your environment and exactly you know based on some of that network policy goodness of the ingresses right and knowing exactly what portion of your app talks to what other portions on which ports what's normal behavior what's not any time that there's not normal behavior it'll be able to prevent and then also let you know hey by the way I just prevented this thing and it looks like it was really bad because somebody was trying to like open a bash on your host so yeah the ultimate solution was yeah to direct and I was wondering if there was something built in okay thank you yeah okay any other questions okay well thank you very much thanks guys for any for anybody here who is using Kubernetes who is looking to use Kubernetes is looking to contribute to Kubernetes we will have a Kubernetes buff in this room starting in about 10 minutes