 Thank you very much. All right, so we're going to do this sort of, I guess, stand up comedy because I have a handheld microphone right now. So who am I? As was mentioned, I'm a technology evangelist at Docker, so I work for Docker Incorporated, which is the company behind the Docker project. I spent the first half of my career doing IT networking and SysAdmin stuff, but that was a long time ago. So I was a certified Banyan engineer. There's always like one or two people that are like, wow, that's old. And then I was a Microsoft Certified Systems Engineer on Windows NT 3.1, so that's how long ago that was. So then I moved in, second half of my career, I've been doing a lot of product management and technical marketing for Docker. Most recently, before that, I was at Puppet Labs. So that was sort of my open source exposure, coming into Puppet Labs. Before that, I spent about six years at VMware, sort of from 98 to right, 2008, 2015. And then before that, I spent six years at Microsoft, where I did a bunch of stuff on IT admin features and functionality for versions of Windows, starting with Windows 2000 or NT5, whatever it was called. If you've ever installed Windows XP and you've ever read the text that says, Windows XP will be faster and more reliable, I wrote that text. So that's my claim. Well, my other claim to play with Microsoft is I was on stage with Bill Gates in an earthquake. So there's that, too. I come from an IT pro background. I'm not a developer. When I pretend to be a developer, you'll see painfully, obviously, how I am not a developer. But if we get to a demo, then we'll see that. You can follow me on Twitter. I tweet mostly about soccer and the Portland Timbers, occasionally about work. So why are we sitting here today? So first, I'm going to start with the informal poll question. We'll start at the high end and move down to the more beginner end. How many of you are deploying containers in production today in your company or in your organization? That's actually really good. Normally, I just did Container World, and I had a room of about 100 people, and I had six people in production. How many of you have downloaded Docker and have done a POC with it? So you're planning to go to production. How many of you have downloaded Docker and run something like a Hello World app or a demo app? How many of you are brand new to containers and are here because you just want to learn? It seems like everybody, the same people, all raised their hands. It looked like it was about 25, 25, 25. So we're going to spend some time on what is Docker to get people up to speed. So if you're familiar with it, bear with me. We'll get into a little bit deeper stuff. But as you get familiar with containers, as you start moving into that journey, questions come up, right? Especially if you come from how many of you are developers? How many of you are sysadmins? So this talk is really aimed at people coming from the sysadmin perspective. What are these things I need to know about? So when do I use a container? When do I use a virtual machine? If I'm going to deploy containers, do I put them on a physical box or a virtual box? Do they go in the cloud? Do they go in the data center? All of those different things. So we're going to talk a little bit about how you answer those questions. Actually, I'm going to give you the questions to ask, to answer the questions you wish that I would answer for you, because I can't answer your questions. And you'll understand why as I go through the presentation. So we're going to talk about what are containers and what containers are not. Containers are not virtual machines. When I first started at Docker, the summer of 2015, everybody would be like, oh, it's like we're virtualization. Oh, I've heard Docker. It's like we're virtualization. They're lightweight VMs. They're not VMs. They're totally different. They have different architecture. They have different benefits. So if you think about a virtual machine, a virtual machine is a house, right? It's fully self-contained. It has its own heating, its own plumbing, its own air conditioning. It has a front door, so you can't just bust into it, right? It's got a little bit of security there, a little isolation. Think about a virtual machine. It's a self-contained operating system. The entire OS is in there, right? And if I'm running, let's say I'm running a couple of servers and their web servers, and I need to have a third for capacity, I spin up a virtual machine. I spin up a whole new operating system. I build a whole new house. There's nothing shared between those houses, right? I would say maybe the sewer, and that'd be the network, but that might offend the network guys, so I won't say that. So VMs, monolithic, fully self-contained. If you need more capacity, you expand them out, right? Houses can be different. They can kind of look different, right? Like you can have one house that's kind of a big two story house, next to a one story house, and they're different sizes. But houses only get so small. First house I ever bought was about 800 square feet. It was two bedrooms, a bathroom, a living area, and a kitchen, right? That's the smallest houses I could find when you talk about proper houses. I could talk about micro houses, but they break my analogy, and some of them are made out of containers, and they really break my analogy. So we'll stick on normal houses. And that's like VMs, right? Even if you go and you shrink them down, you're usually looking at a couple hundred megabytes to get in a VM, right? And sometimes you're talking 10, 20, 30, 40 gigabytes, right? They get really big. So containers, how are they different? So think about containers as apartments inside of an apartment building. So an apartment building is the Docker host. The Docker host is any computer, physical or virtual, cloud, data center, whatever, that runs the Docker engine. The Docker engine, you know what I'm talking about? There's a lot of different container runtimes. I'm going to talk about Docker, right? This whole idea of container runtimes, pretty much this is exactly the same in all of them, right? So you've got a system running the Docker engine or container runtime. And that's like the doorman at the apartment. Now what makes an apartment different than a house? Well, one of the fundamental things is it shares the infrastructure. It shares the heating, the plumbing, the air conditioning, right? It shares the elevator. Containers are processes inside an operating system. They share the underlying kernel in the operating system, right? So unlike a virtual machine which has its own kernel, containers share that resource. That means there's a couple of big benefits. It means there's also a couple of limitations with respect to virtual machines. The big limitation is that every container running on that host has to be from that host family, right? So you cannot run a Windows container on a Linux host. You cannot run a Linux container on a Windows host, right? Because they share the kernel. But you can have a host that's running CentOS and you can run an Alpine Linux container on it, or a Red Hat Linux container on it, or a Ubuntu Linux container on it, right? As long as the architectures are the same of the underlying hardware, those things are just going to move along. But like houses, we have a front door, right? So a container is isolated, can't just bust into it. And then the containers can be quite small. Someone, they're doing a kind of informal, if you want to get on a little informal competition on Twitter, one of the docker, I don't know if he's a captain or not, but he's an aficionado at least, a community member, hacker, pose the question, what's the smallest hello world container you can write? So what's the smallest container you can write that says hello docker, right? And so anybody want to guess what the size is, the smallest I've seen? 68 bytes for that container. So that's a full running container, 68 bytes. I have an Alpine Linux container that's 2.6 megabytes, that's a full operating system, right? And so containers are much, much smaller than virtual machines. The other thing, or they can be, right? The other thing is that in a container, because it's booting up a process, it's not booting up an operating system, they start really fast. They start in about three-eighths of a second compared to a virtual machine taking several minutes to boot. So fundamental differences, shared resources versus dedicated resources, a little bit faster, a little bit smaller, but sort of very homogeneous in the network, or excuse me, in the, with respect to the operating systems you run. But just because they're different doesn't mean they're mutually exclusive, right? What you see is most people, you see what's on the, my left, your right there, where people are taking containers and they're running them inside of virtual machines, right? They go and they spin up a Linux VM or a Windows VM. How many of you knew that there was containers on Windows now? A handful of you. OK, I don't have that in my slides. I know this is the Linux Expo, but how many of you run environments that are 100% Linux with no Windows at all in them? So like 20%, like, you know. So for the rest of you, containers are also available on Windows Server 2016. I won't get into it in a lot of depth, but just know that there are such a thing as Windows containers. They use the exact same API, the exact same CLI as the Linux containers do, right? But they're based on Windows operating systems, not Linux operating systems. So you could, in this example here, one of these blocks could be a Windows Server 2016 machine with Windows containers. The other could be a Linux virtual machine with Linux containers. And you could just have a traditional VM over there doing whatever. Or you can run them on bare metal. You can have a host OS running on a piece of hardware. And you can install the Docker engine. The thing is, it doesn't matter, right? If these are Linux containers, they can move from the left to the right seamlessly, regardless of the distribution underneath, right? So in tests I'm running, sent to us in production, I'm running Red Hat, whatever. It doesn't matter. We can move across there. So like I said, there's really good reasons for each of these architectures. And I'll get into them in future slides. So one thing I want to talk about today with regards to the Docker project, we just announced this today. So I have this in here. And so this might be of interest to everybody. So today we announced we're actually releasing two versions. So we're splitting the project, not splitting the project, but we're splitting it up into two specific releases, right? So Docker is a commercial company, right? I work for a software company called Docker that happens to manage the open source project Docker. And we wanted to make sure people understood that there were a couple of different ways to get Docker. So the first is Enterprise Edition. And I won't spend a lot of time talking about this because it's a commercial offering. It's stuff that we sell. But basically, it takes the Docker technologies. It bundles them up into releases for people who want support and they want some advanced features. Then we have Community Edition, which is the same Docker that people have been using forever under open source, right? So it's great for do-it-yourself organizations or hobbyists or all that kind of folks. We're going to be on a quarterly release with those. With every month, you'll get an edge, what's called an edge release, right? So it used to be experimental or beta. You're going to be able to see, OK, I'm on 17.3 edge, right? And that's the sort of beta upcoming releases. And then we do maintenance and support on the Community Edition for four months for each version. We do it for a year on the Enterprise Edition. Enterprise Edition also comes with a whole bunch of certified certification stuff. And I think I have a slide on that, where we've certified a bunch of ISPs, a bunch of hardware. So this is new, and you're going to hear more about it, but I just wanted to bring it up since we were here this morning. We're also going to talk a little bit about editions, right? And I'll talk about editions at the end, but I wanted to explain what an edition is. An edition is any way you get Docker running on your machine. So I have a whole Getting Started section, but we release Docker for Windows, Docker for Mac, Docker for Azure, Docker for AWS, right? And those are ways for you to get up and running and integrate with whatever platform you're running on. So when you think of something like Amazon, it's going to take into account load balancers, it's going to take into account S3 storage, it's going to take into account the VPCs and the Natting and everything you need to run, right? It'll stand up Docker, build all that infrastructure out for you. That's something you can get with Community Edition, thoroughly free if you're running on AWS, it's the fastest way to get up and running. We'll talk about that a little bit more later. Skip this slide. So the idea basically is with the Docker platform, regardless of whether you want to pay for your software or not, we're going to give you one platform, right? And the idea here is that the promise of Docker and at the end of the day, why people deploy it is that they want to be able to write an application, have their developer write an application on their workstation and have that seamlessly flow through the environment and move from developers to a QA into production, right? And they want to be able to do that on Linux and they want to be able to do that on Windows and they want to do it on premise and they want to do it in the cloud, right? And any type of app, when I joined Docker, everybody was like, oh yeah, it's containers, it's only for microservices. And what we're seeing is most of our customers today are not exclusively using Docker for microservices and I'll give an example later, companies are taking traditional applications out of VMs, containerizing them and putting them on to Docker for a number of different reasons which we'll go through. So if you've got homegrown applications, ISV applications, microservices, it doesn't matter, right? All of that, moving that. And if you're on Community Edition today and you want to move to Enterprise Edition, great, it'll just work. Okay, so now that I've been speaking for like 15 minutes, some of you were in there like just answer the question. Like the question I came here to get answered. But the thing is like one does not simply answer the question without doing a bunch of marketing stuff. Besides, you wouldn't like the answer, right? So this picture, so when I was at VMware, I would get this question and actually who thinks, what, I want to hear what you guys would say. What is the number one question I answered when I was at VMware? Yeah, well that was number two, but, because I didn't actually, people asked me like how much it costs, I'd be like, I have no idea, I don't sell software. Okay, so the number one question I got was how many VMs can I run on my server? What's the answer? It depends. But if I was feeling kind of nice, I would say it depends. If I was feeling surly, I would say, how many plants can I get in my car, right? Because it depends on your car. It depends on the plant, right? So my car is a, I used to have a suburban, it was a better analogy when I talked about, when I had my suburban and my wife has a Honda Fit, right? All right, so I could clearly carry more plants than her, well no, because I'm carrying ficus trees and she's carrying pansies, right? So it's, there isn't an easy answer to this, right? There's a lot of variables to consider. So when you think about, and you say, okay, well I get, I understand what a container is, but I still don't know what I'm gonna do with it. There's all these things you have to think about. So I went back and I, and we go back a couple of slides, and I talked about this idea of, you know, mixing and matching your workloads. This is all about capacity planning and, you know, somewhat about licensing, licensing costs, but if you think about where VMware came from, like, so 2008, VMware was really picking up steam and the number one use case was, I stand here all day. Legacy I'm sure, but the problem we were trying to solve was resource utilization, right? Because you had all these servers out there and they were all running at 30% utilized, but you couldn't mix the workloads. So then we said we put the database and the exchange server and the IIS server into VMs and we can now put them on one VM, on one physical server and I now have a machine running at 80% utilization instead of 30% utilization. So I'm spending a lot less money on hardware. That's where we came from. If you're sitting there today on your first stages of your container journey and you don't have enough capacity to drive the server utilization, you'd be foolish, you know, if your number one concern was capacity utilization to put that on bare metal, right? Because you would be wasting money on that hardware. So capacity planning and mixing workloads. But now if you go, wait a minute though, I have a security implication and there are security implications that say I cannot run this workload on a shared kernel. Like there's some, you know, there's a law or there's a security policy or there's a regulation or something. Then you're like, well, what do I do? Well, I put it in a VM then, right? Because I have to because in a lot of play, a lot of people still consider, well, I mean, it is a shared kernel technology, right? And so they are like, we can't do that. It's just a policy, we can't do it. So if you have multi-tenancy requirements, right? And especially in the Linux space, in the Windows space, there's this thing called a Windows container with Hyper-V isolation, which we're not gonna get into, but if you're interested in that, find me afterwards and I can tell you about it. So you're like, well, I have to go in a VM even though I have enough capacity, but so I'm gonna go VM. But wait a minute, I really care about performance and latency. Well, okay, then you're probably gonna go bare metal because containers always run faster on bare metal. They run faster than virtual machines. They run faster than containers in virtual machines. They are fastest on bare metal. So you're sitting there and you work for a financial trader in New York City, a company who just paid $70 million to relocate their data center two miles closer to reduce their latency by three milliseconds because they do that kind of thing when you're dealing with billions of dollars in trades. So then you're like, well, I don't care if that server's 15% utilized if it's 3% faster, I'm gonna go ahead and spend that money. So there's that. So anyway, if you think about it, there's a lot of different variables in there. The only one I haven't really touched on is, and some of you were sitting out there and you're like, well, I'm a vSphere admin, right? All my tooling is vSphere, all my tooling is KVM, all my tooling is Hyper-V, all my tooling is Puppet, all my tooling is Chef, all my tooling is Ansible, right? So you've got these existing automation frameworks and you've got these existing skill sets. So you look at that and you say, well, it just doesn't make sense for me to go 100% into containers right now. Like I wanna keep what I've done with vRealize automation or vCake or whatever it is, right? So that's another thing where you look at it and you say, how am I gonna make this transition with my skill set and my staff and the people who have to do the work? Because a lot of people come in, especially on this admin side, they come into containers because someone walked into their office one day and said, hey, the new app is ready, it's in a container, go ahead and put it in production. And you're left with this slide and you're left with, well, I don't know how to capacity plan for that. I don't know how to secure that. I don't know how to do that. So we're gonna take a look at two examples and we're gonna talk about first consolidation and compute consolidation and then we'll talk a little bit about security. So VM consolidation, so this slide is actually based on some testing we did with Hewlett Packard. So HP and then we had an independent consultant named Brett Fisher went out and they took suspension, they ran suspension. What they did is they took suspension, they said, okay, on this physical server, we're gonna create eight virtual machines and we're going to run suspension them and then we're gonna measure the output. So let's just say for Grins that each machine or each VM could do 1,000 transactions per second, so 8,000 total transactions, right? So then they said, well, what happens if we put that in containers? So let's just use one big giant virtual machine. So they took the eight VMs that were using 16 cores and 16 VCPUs and 32 gigs of RAM and they made one big VM that was 16 VCPUs and 32 gigs of RAM. Just again, kind of just making up these details really, kind of making this whole thing up. Actually, that's not true. This is actually a true study. And what they found was if you move, just by moving from eight smaller VMs into a single VM, you actually got 27% greater output. So where you were running and getting 8,000 transactions total per second, you were now getting 10,000 transactions per second, which means if you had a server farm out there, you could take 25% of the servers out of the farm and still do the same amount of work by moving from a bunch of smaller VMs into a larger VM and containerizing those workloads. So if you had each container represented a different work group or some other different workload, by bringing those together, the other thing that you've done there is you're gonna use about 30% less storage. You're gonna use about 7% less RAM. You were going to move from managing eight guest operating systems in eight virtual machines to a single guest operating system in a single machine. And you're gonna reduce your data center footprint, right? So less electricity, less cooling, less real estate. I will say so for those of you in the room wanting the technical details on this, one of the things we did have to do was it wasn't a straight just move over. We did some stuff with pinning and affinity to make sure that the workloads went to the right CPUs. And there is a 50-page white paper on this available from Humboldt Packard. So if you follow me on Twitter, I will tweet the link out to that paper after this talk. And it goes into all the detail of how we did it, right? So if you're sitting back there, you're like, well, geez, okay, there's a benefit. So if you do that exact same experiment and you go, okay, now I'm gonna take eight VMs and I'm gonna move into eight containers running on bare metal. So no virtualization at all, you get a 47% output increase. So you go from doing 8,000 transactions per second to about 12,000 transactions per second. Again, I'm just using those numbers to make them easy to, you know, because they're even whole numbers. But what it basically means is if you were doing a server farm, you could take two of those servers out, right? It's a little trickier than, it's a little trickier if you say, well, I have one server doing the work because of the affinity and the pinning. So that sort of example works better when you talk about using server farms. But if you were just on a single server, you would still get more output from that box than you were before. So again, storage savings, operating system management savings. You go from managing eight guest OSes and eight virtual machines to running a single host OS running on bare metal, right? So you have the bare metal server, your Linux operating system and the Docker engine and eight containers out there running. So that is, you know, the VM to bare metal consolidation. So things to think about then. So you're like, all right, so that's interesting. So what do you gotta do, right? Like, so higher density. Like, containers are going to give you higher density than you're used to having. So just think about that, right? They, because they're lightweight processes, you know, you're going to just get more of them in there. They're gonna be more responsive. Bare metal or bigger VMs, right? So do you want to now look at getting rid of virtualization all together in some use cases? Well, if you don't have to worry about multi-tenancy, right? That's a, you know, if your existing tool set lets you do that, right? Then you can get performance gains there. So you can, you have to balance those things out. And then tuning to optimize. So I mentioned this before, right? We didn't just slam those VMs and just move them over, right? We had to do some work with the affinity and the pending on the CPUs. The white paper details exactly what we did. But by, you know, by tuning it a little bit, right? And it took the consultant, you know, a couple of days to figure that out, right? He made an investment about, you know, I think most of us would invest 16 hours to get 47% performance gains. That's a pretty big return on your investment there. So that is, you know, the idea of like capacity planning and some real quick overview on that. But that's an indication of the things you need to start thinking about. We're not, you know, you don't want to just go into it on this mindset of, oh, they're exactly the same. We can just sort of do this one to one mapping because there are things are going to change for you. So what about security, right? I put this in here because that probably the number one, well, it's not really the number one, but one of the top five questions I get about Docker containers is what about security, right? Most people, when they say Docker container security, they worry about like exploits that allow you to break out of the container, which have happened. They've been there. We work very hard to protect against those as do all the members of the community. But security is more than that, right? When we look at security, we think about end to end security and how you can do that. And we look at it now from this perspective of, like there's three things that we really care about with security. The first is that it's usable because if you make security unusable, if you make it hard, they're going to circumvent it, they're going to write that 15 character password on a post-it note and they're going to slap it on the side of their monitor. They're going to use the same password for every system. Like I'm guilty of that. I'm not guilty of that. Do not try to hack me. Anyway, so we know this, this is a common edict, right? And you want to have a way to move things around in a secure manner, right? It's not just about like what's happening when the container is running, but what happens to that application through the entire life cycle? How do I guarantee that the application that I'm putting in production is what I think it is? How do you know when you deploy a VM that it's actually the VM you think it is? How do you know that somebody didn't come in and swap that VM out? And you're looking at it and it says, Master Golden Image 216, 216, 2017.7. And you think you've got the right version. How do you know that I didn't come in last night and change that and just rename it? You don't, I don't think, maybe you do. And then whatever you do with that container, whatever you do to define that security policy, it needs to follow that container regardless of where that container exists. Infrastructure independence, right? So my security works on my MacBook, my security works in the cloud, it works on my data center, right? All of that equals safer apps. So how do we, how do we do that, right? So things to think about when you're looking at deploying containers in production around security. One is application secrets. How many of you pass, okay, we're gonna do it like they used to do in Sunday school. Everybody close your eyes so nobody has to be embarrassed. How many of you pass passwords using environment variables or in static config files? Come on, all right, there's a few brave souls. I think more people than raise their hand do it, right? We all know that there's an inherent risk in that, right? So how do you pass that stuff securely? How do you maintain a workflow that works for your developers and your operations team? And then how do you set up a system that out of the box is already secure? So we're gonna talk about those things in a little more detail. I think, hold on, we're gonna talk a lot about secrets. So in here, the image signing and image scanning and verification. So these are functions available through Docker. Image signing and verification is fully open source. It uses something called Notary, which is based on the update framework. The image scanning is actually a paid service that you can have added to your private repos on Docker Hub or you can have it added to your Docker data center enterprise edition thing. But what they do is they work in concert. So the idea is with image signing, I as a operations person provide a base image to you. So you're going to write a Java application. So I hand you a Java app, a Java image and I've signed it digitally. So as a developer, as an operations person, you can configure Docker to only run signed images. So I'm a developer, I pull down the Java image and I try to run it. Docker verifies the signature that runs. I add my code, I sign it. So now it's been signed that it has the right base image. It's been signed that the application you're about to deploy has come from one of your developers, right? And then it goes to QA, right? And it goes through the testing process or whatever and then the CICD system or whoever signs it. Before I can put it in production, I can verify that it's been signed based on a secure base image, that my developer wrote it and then it's been through our testing cycles. And that's what we mean by the secure software supply chain, right? That's the marketing term for it, as we move through each step in the process, we can verify that it's been working, right? So the other thing is image scanning. So again, this is a commercial thing, but just for completeness of the story. Once that container is running, we can do a, when containers get stored on something called a registry, right? It's a server to store your Docker images, your container images. When you push an image, when you send it up to the registry, when it hits the registry, we can scan it. We do a bit level scan on it. We can break it down. We know everything running in that image and we can list all the vulnerabilities out for you. So we can say, look, this image has these known vulnerabilities based on this CVE database and the CVE databases will be updated regularly and will rescan for you. So once you put a container in production, we will tell you if anything is ever discovered against that container so that you can go out and then remediate it, right? It's this ongoing security vulnerability scanning. Couple of other things, we talked about secure by default. The idea of full TLS encryption across all the communication between the containers or the host in the system and that all information is passed over secure channel by default. And then as you deploy in production and you start getting into a little bit of the, some of the, again, a commercial feature, users and role-based access control, right? So syncing with Active Directory and LDAP for user information and then using that to apply granular permission so that when you put a container in production, if it's a production version of the container, only the ops team can modify it, right? The development team can see it or maybe they can't, right? But they can't modify it or any level of permissions in there based on these labels that you apply. So being able to go through and ensure that only the people who should have access to those containers do it and regardless of where it's running, right? Regardless of the system that it's on, we should be able to do that. So we already know what this is. I wanna make sure there's anything. I didn't really talk about the secrets, but let me, I'm gonna use this slide because I wanna get done here in the next 10 or 15 minutes. So a couple of things on secrets management. This is actually new in the latest release of Docker. So what we basically do is we use Docker swarm mode. So we're gonna come up next, we're gonna talk about Kubernetes. Kubernetes is a clustering and orchestration system. Docker has one called swarm mode and in the cluster managers, right? So the cluster is a bunch of managers managing a bunch of workers. In those managers, we store, we can store the secrets. Now they're encrypted by default at rest in the store, right? And so, and they're encrypted while they're sent across the network. And those secrets will only be sent to systems that actually are running containers that need them. So if you've got 10 servers and only two of them are running the database, we're only gonna send the secrets of those two machines. They are never stored on the disk on those worker nodes. They're mounted in an in-memory file system. So you don't have to shred the disks when those machines go away because nothing was ever written to them in that regard. And the other thing we have is we have something called cryptographic node identity. So those worker nodes can't be spoofed. You cannot go into and say, oh, I'm worker seven because worker seven was cryptographically signed by the manager. And so we can verify that identity, right? So there's a lot of protection built into that. So if you kind of look at this right here, right? So the RAF consensus group is all those managers. That's the data store. We put the secrets up there and you can see in the example below we have three workers but the secrets are only put down under workers one and three because that's where the containers are running, right? Every, all the communication between those workers and the managers is encrypted. So that's what we, when you think about security, don't just say, think about, oh, did I patch this or did I, did I, you know, is the container isolated? Think about the entire end-to-end story. Think about protecting stuff in ways you actually maybe hadn't thought about before. Being able to adopt different ways of managing sensitive information. So we talk about passwords here but you can send anything through a secret, right? You can send directory environment variables, directory structures, whatever you wanna send, like any piece of information. All right, so why do we even start, right? So why do we even start? And I kinda, sometimes I start with this and sometimes I end with it because I think it's hard to understand it if you don't kinda have a little bit more information about what containers do. So we start by, you know, just being faster, just being quicker, right? Being able to run CI CD pipelines on containers and you can cut the time to do a test cycle by orders of magnitude simply because if you're booting up 100 VMs to do scaling testing, that's going to take you a couple hours potentially and I can boot 100 VMs or 100 containers in 15 seconds. So I go from hours to seconds to get my infrastructure and if you think again like CI CD, I'm not managing a Java build server and a Python build server and a whatever build server. I have build servers and they run Docker, right? Or they run some container runtime and everything comes into the container because everything my application needs to run is in the container. I don't have to do anything with it, right? That makes all the dependencies much simpler, right? The dependency to now run your application on the infrastructure side is an operating system and the Docker engine, right? You don't ever have to worry about what version of Java is installed on that machine or I'm a Java developer and you wanna send me Python, I have to install Python, I don't have to worry about any of that anymore. Unified tool chain, yep, unified tool chain. So the idea that the CLI and the API that you use is the same on your MacBook where you're using for development. It's the same as on your Linux hosts running up in Azure. It's the same as the hosts running on vSphere. It's the same APIs and the same CLIs. Oh, and by the way, it's the same APIs and CLIs from Linux to Windows, right? So Docker run is Docker run wherever you type it. Docker pushes Docker push wherever you type it. We talked about that API automation. I'm going to move forward to... Actually, I wanna talk about this stuff. So you're like, okay, cool, I don't know where to start. I heard that I have to do microservices. The reality is that you can do any number of different application paths into Docker, into containers, right? So we talked about already the benefits of just lifting and shifting applications. Just taking an application and moving it into a container is going to be more performant. Even if you keep it in the VM, you're going to get better performance that way, right? You're going to get better portability. And a lot of people are doing that. We have customers who, we just did an announcement with a big financial company back in New York, like a Fortune 5 company, Fortune 40, sorry, that's doing lift and shift on Windows. And they're moving a bunch of their applications into Docker containers, just straight up moving them in there. And sometimes what people do is they move them into containers and they start breaking them down, right? So they take a container and they take an application and they say, all right, I'm moving that into a container. Well, now I've got these three applications that are all running in containers. And I notice they all have the same authorization, sort of authorization function. I'm going to put that in a microservice and I'm going to point those applications out of the VM into the microservice and they start breaking those things down, right? There's this whole continuum between monoliths and a monolith being a traditional standalone application and microservices and you can be anywhere on that journey. You can put it in the container and never worry about it or you can start breaking it down or you can just, you know, we've had some customers that have just said, you know what, I'm just going to blow it all up and they blow the whole thing up and they don't do, they just immediately go to microservices. The things you have to worry about with containers as you start looking at these monolithic applications are things like fixed ports and static configs, right? When you have to define the application for different environments and you're talking about a container where everything is sort of the same, that can be problematic. But the beauty is, and this is my favorite, absolute favorite thing about Docker. And this is, this could be my last question I ask you guys. How many of you have ever deployed an extremely complex project exactly 100% correct the first time? Oh, I saw a guy trying to raise his hand over there. You guys get his number quick, hire him. You guys want that perfection, right? No, and that's the thing about Docker is that, you know, we hear a lot of people that are like, oh, you know, we were in the cloud and we realized the economics of the cloud didn't work. And so now we want to move back to the data center. Super example, you know, common example, super easy to grasp. The problem is that if you've done that and they're in EC2 instances and they need to end up over here in a virtual machine, how do you do that? Like that is a serious engineering effort. If they're in Docker containers, it's trivial, right? And what I mean by start somewhere and end somewhere else is you are able to recover from decisions that you've later come to realize we're not the right one. I don't want to call them mistakes because we don't make mistakes. Well, at least, I thought I was wrong once, but I was mistaken. But you start somewhere and you do something and you're like, oh, this isn't what I want. Well, it's easy to recover from that and adjust to it and move around on the fly. I'm gonna skip that. So getting started, for those of you who are new to Docker and don't have it, if you have your Linux machine, you can just do curl, SSL, HTTPS, getdocker.com, pipe that to shell and you'll have Docker in about three minutes. Now, for those of you who do not want to pipe to shell, we have full manual installation instructions up on our documentation. If you want to do beta builds, you could do test.docker.com. If you want to do bleeding edge, you could do experimental.docker.com. Any of those will get you up and running. All you need on the Linux side is a kernel greater than 3.1. That's the requirement, right? A kernel greater than 3.1. We've got a whole bunch of, any Raspberry Pi enthusiasts in here? Anybody got Docker running on the Raspberry Pi yet? There's a company out there called Hyper, I call them Hyper, I don't know how they say their name, but they're basically, they're all about Docker on Raspberry Pi. They're actually doing a DockerCon workshop on getting Docker running on Raspberry Pi at DockerCon this year. But if you, that's something you can geek out on. So even on Raspberry Pi, if you are running Windows or Mac, you want to get Docker for Mac, Docker for Windows, easy to download. It uses the integrated virtualization on the platform. It is running to be able to get Docker up and going in a matter of minutes. So I'm running on Docker for Mac. On Windows it uses Hyper-V, on Mac it uses Xhive, super fast. Again, the same API, the same CLI, it doesn't matter. It's going to work exactly the same way. We used to have something called Docker Toolbox. How many of you have used Docker Toolbox? How many of you love VirtualBox? Some people liked VirtualBox. We moved away from it because there were some limitations on it. And that's what we mean by OS integration for speed and stability. So if you're in the cloud, right? If you're using Azure or AWS, we actually have the ability to go out and run a cloud formation template or the Azure equivalent, an ARM template, I think is where they are. And you can basically stand up a Docker cluster in about, well, you can kick it off in 30 seconds. You actually answer like five questions and it will build you a full Docker cluster up in AWS. It'll integrate with the load balancers and everything. So if you start a service on the cluster that needs a firewall port opened, it will open the port for you, like that sort of integration. And I was geeking around the other day and I just said, like, what could I do with this? And I deployed a thousand node cluster up in AWS. Like I just said, give me a thousand nodes. And like it took an hour to provision all the instances, but they all came up and I had a thousand node cluster up and running. So that's pretty nice. We do that for Azure and AWS. So, RockJog run. So you're new to Docker, go out and install Docker for Mac, Docker for Windows, put it on your Linux box and just do Docker run hello-dash world, right? Super easy, you could run Mike G. Coleman Catweb and you could get a really cool website that shows random catgifs in a Docker container. And then so for me, when I came to Docker, I knew nothing about containers. So I went on and I said, what are the things that I could do? And someone said, why don't you build a CI CD pipeline? So I went and found Jenkins and I got a Jenkins container and I built a Jenkins CI CD pipeline with Docker that basically runs the test jobs, pulls the Docker file off a GitHub, builds the container, tests the container, if the container is good, if the test pass, pushes it back up to Docker Hub, right? Do something like that or WordPress because WordPress has multiple components, right? Build something like that. There's a thousand and one blogs on all these things. Go check those out. And then once you get that done, you can say, well WordPress is cool, but I need multiple databases on the back end. How do I do a replicated database or a sharded database or a scaled database? Or that Jenkins thing was cool, but I really want a full CI pipeline, right? I want more than just getting it to Docker Hub. How do I build this so that it pushes it into production? And what does that look like? Build your own Docker files. We didn't talk about that. Build your own compose files. Get into swarm mode. Talked a little bit about swarm mode. Didn't talk much about compose. These are some of the more advanced features of Docker you can play with, right? So at the end of the day, I'm not here. So I made the comment. I'm here to get you the questions that you need to answer to answer the question that you wish I would answer for you. I can't answer it for you. I can't tell you how to run your application in a container because only you should understand at the level necessary to make that informed decision. I will say that putting something in a container does not abdicate you of the responsibility of understanding how your application works and what it does. You still need to understand your applications. You still need to performance profile them, all those sort of things. And then pick the right tool for the job. Containers are not a panacea. They are not the right tool for every job. I think there's a lot of problems they solve and I think they're obviously the motion and stuff that we're getting shows that a lot of people are happy with them. But it's up to you to decide what the right tool for the job is. And my job today was to helpfully give you some more things to think about as you go down that journey. So what's next? Go ahead and pick a project. Get your hands dirty. You will make mistakes. You will do things like inadvertently delete every image on your system or corrupt an entire cluster and have to rebuild it from scratch at 11 o'clock on a Thursday. At least I did. But join the community. So if you're not involved in the Docker community, we have thousands of contributors. We have a bunch of ways for you to get involved, everything from just downloading it, playing with it and opening issues and letting us know what doesn't work to joining a meetup. Like I was mentioning, I'm the organizer, one of the organizers for the Portland Meetup. Join our community. Share what you've learned. And with that, I have about three or four minutes for questions before we need to bring up the next slide. So thank you very much. And if you have a question, let me know and we will do our best. I don't know if we have another microphone or if I should just run around. I think I have to run around. Okay, any questions? Perfect presentation. No questions. No, this guy's got a question, he's gonna ruin it. I was born 300. So the gentleman's question was, we talked about Docker for Azure, Docker for AWS. Is there Docker for Google Cloud? I'm not sure that I can tell you that we're working on that. But I think I can. And I think, so what we did with, what we did had the way that Azure and AWS came out is we did a private beta where you would send in a request and we would approve you and let you in. Then we did a public beta. I'm not sure where we're at in that process, but we are working on Google Cloud, most definitely. Yep, for sure. Yeah, so the question is Docker data center. Docker data center is one of the tiers of the enterprise edition. So Docker data center is a commercial offering based on a annual subscription that gives you a graphical front end for managing containers. If you're a vCenter admin, think about vCenter for containers is essentially what Docker data center is. But yes, it's a paid offering. It's part of enterprise edition. You can get a through today free trial up on Docker.com. But I'm not here to sell you anything. If you're just using it Docker at home, you probably don't need it, honestly. But outside of the 30 day trial edition, I don't think there's anything like that. There's some open source projects out there and some emerging things. I think Portainer's one of them that kind of gives similar functionality, not quite as enterprise ready. If you just want a graphical front end, you might want to check one of those out for more of a hobbyist thing. I'm going to come over here and then I'll come back over here. There was a way in the back there. How does the AWS integration compare to ECS? So ECS is each C2 container service. It is basically the Amazon offering for sort of managing your containers up on Amazon. So a couple of big differences. One of the big differences is that the ECS integration uses, is always going to be behind as far as the Docker engine goes because they have to integrate it back up. So I think they're still like on 112. So with Docker for AWS, that release is part of the core release that we do. So when we release a new version of the engine, we automatically update that cloud formation template and all the omis behind it. The other part of it is that EC2 uses its own proprietary sort of chain, like a tool set or whatever commands. And so if your developers are over here developing on Docker on their MacBook and then it goes into EC2 container services, it's a completely different set of APIs and a completely different CLI. So you kind of break that workflow. Essentially they basically do the same sort of thing, but EC2 is a little more proprietary than say Docker for AWS. Like Docker for AWS is more open and leverages all the AWS stuff underneath it. Yeah, it's a really, really good question. So gentlemen's question was how do you relate or what's the practices around config management, puppet chef ansible and containers? And I think, I had a long conversation with the chef guys last week in container world about this. Most of those vendors are not gonna ever recommend that you put agents inside the container, right? That you wanna treat those containers as sort of immutable and they don't change. And so if you do an update to a container, that's a bad way to do it. You wanna update the stuff outside and do the upgrade. Puppet in particular has done a lot of work to help set up and maintain Docker environments. So that's sort of how I see the delineation, right? How do you build, they use them to build the infrastructure to maybe track the base images. Puppet has some tooling to do like manifest into Docker files. And then chef is doing this thing called Habitat, which is kind of cool because it's actually like a super set of everything, right? So it understands that containers are one way to put out applications, but there's also VMs, there's also physical applications. So they're kind of taking a more holistic approach, but most people, most of those companies, like when they get to the actual container, managing that, they leave that to Docker, right? They leave that to the Dockerfile and the images. So any last questions? All right, well, thank you so much for, oh, one more back here. Yep, in the Oregon sweatshirt. Yeah, gentlemen's question was what about the performance characteristics of different workloads and how would that impact the consolidation ratio? That example that I gave today is actually what we consider to be a fairly conservative estimate because that was a CPU bound application, right? And so applications that are more memory bound, you're gonna get better consolidation ratios than you will. So, but you won't know until you test the individual application. CPU bound application is probably gonna get less of an increase because you're just like, some applications are gonna eat as much CPU as they possibly can, right? And you're not gonna be able to do much about that. We find that like memory applications that are more memory bound seem to do a better job of consolidating. And applications that use a lot of storage, you can get really great storage savings because things, we essentially have something that's akin to deduplication or thin provisioning built into containers. Like containers are by their definition thin provisioned. So, if you scale them out, the more you scale out, the more you save in terms of disk space. Okay, all right, with that I have to stop because, not time for the next speaker, but thank you so much for coming out and letting me kick off scale. I was here last year, I loved it, and I'm glad to be back this year. Testing, testing, testing, testing. Hello. Testing, testing, testing, testing, testing. Testing. Yeah. But they were having trouble with it earlier than the sound guy disappeared. Hello, hello? I mean, if you have a chance to actually run and get somebody, yeah. Hello? Testing, testing, testing. Testing. Hello? Testing, testing, testing. Aha, there we go. Okay, now that's a different problem. Testing. Okay, no, just the one. Okay, wait, is this, it's still working? Yeah. Okay, awesome. Hello. I think somebody's here too. Okay, yeah? Yeah, okay. How are you here coming from? Yeah, yeah, he was just fixing that, so, okay. So let me actually do something else. Yeah. So I'm gonna ask you a favor, since I'm up here on stage, can you fill Donnie here with this when it comes to Q&A? Who's filled Donnie here? He was a talk show host and he would run around with a mic. Rocker. I need to type. Okay. Welcome everybody. This is Kubernetes 101. Let me apologize in advance for the static. The sound guys are working on it. But we are going to talk a little bit about how to use Kubernetes. The first thing I wanna say, this is scale's first container day as a scale thing. And I actually wanted to thank the conference committee who helped pick the talks for today and the scale 15 crew for actually making hosting this possible and that sort of thing. And next year we'll have sponsors and other stuff to thank, but not this year. So we'll keep this short. So this talk coming up is for people who actually know something about Docker. Maybe you've used Docker. Maybe you've actually developed some stuff on containers. But you don't necessarily know that much about orchestration or deployment or Kubernetes or whatever. I mean, really, I'm trying to address the question of, okay, so now I've got all my app is now in a container. So now what? Right? It's on the container, it runs on my laptop, but now I actually have to get it into production. So what happens next? So just briefly, the big background for Kubernetes where it came from and that sort of thing. Google is this thing called Borg. They decided to recreate the concepts of Borg as open source code. They made that compatible with Docker instead of their own internal container system. Then they contributed to the cloud native computing foundation as a project. So it would be non-profit independent. My employer Red Hat is a big contributor to that. Also CoroS and a bunch of other companies and that sort of thing. And individuals. And the big thing is one of the reasons why I deal with Kubernetes so much is that I work for Red Hat. Red Hat is this thing called OpenShift which is a complete CICD pipeline for development to production and containerized infrastructure. And OpenShift is basically a distribution of Kubernetes. It's Kubernetes plus a bunch of other stuff in order to make it a complete system. So now whenever you talk about containers, one of the things you always end up talking about is microservices, right? And the idea of a microservice is that for each service defined as some widget that does something, whether it is a web server listening on a port or a database server handling a data stream or a backup service or whatever, that service is going to be isolated to its own thing in the case of containerized infrastructure is going to be a container. But that means that for real production application, you are going to actually have a lot of containers. So for example, I've containerized Django applications for a while, right? And this is actually a reasonable stack for a Django application. You're going to have some PostgreSQL containers, right? So master replicas, et cetera. A database connection proxy. Oh, storage, PostgreSQL needs actually some network storage so it can be reliable for fail-downs, that sort of thing. So we've got Gluster running containers. Then we have the actual Django running containers under Whiskey, and then we have web servers front-ending that so it can actually be performant and do a little bit of caching. And then we have varnish or the reverse proxy up in front of that for performance reasons. Oh, and then we have to do some scheduled jobs like database backups and that sort of thing and pushing from get in that. And then a CI container to actually handle the pushes of the application every time we make a commit to the application. So excluding duplicates, that's eight different types of containers actually. Oh, wait a minute, nobody actually said anything here. Here I've been going on. I can't actually see these screens. Yeah, so I was dealing so much with the sound. The, yeah, how about some slides? Okay, so there we go. There are all of our things. So like eight different containers, the eight different types of containers and just this is actually sort of a simplified stack. So this starts to become a real coordination problem. It's like, oh my God, I've got 180 different containers and I have to coordinate them all and some of them over here and some of them there and some of them need external access and some of them don't. And you know what? I think I'm just gonna go back to the monolith, right? Just give me a VM or a VM and I'm just gonna go back to the monolith and I just, I understand that structure. So, and that was what we encountered a lot early on. There was actually why when we first saw like in the first year of Docker and that sort of thing, you know, you would ask you and say, how many people are using Docker for development and lots of hands will go up? How many people are using a production and like one hand would go up and that guy was working for Docker? And part of it was that there wasn't any way to coordinate all this stuff. And so we would talk about the microservices and everything, but nobody could actually deploy it because you're using, oh, bash scripts. Actually, I did multi-container deployment by bash script or using your configuration management system, you know, Puppet or Ansible or whatever it is or using some custom code or just typing real fast. These were all, you know, sort of popular systems but not really productionizable. So, but now with orchestration, we actually have a productionizable system. So I'm just gonna start at the basic sort of unit, right? Before we get up into doing the whole infrastructure, let's actually start with a small level coordination which is that sometimes you need a couple of containers to stick together. So an example of that would be, you know, something like, you know, a CDN and it's accompanying storage container running a network storage system or a database and a database proxy. You know, where you want those containers to deploy together, you want them to upgrade together and you want them to scale together. And for that reason, you wanna make sure that they always go out together. And in the world of Kubernetes, this concept is called pods. Far as I know, this is something that only Kubernetes does among the orchestration systems. I could be wrong. I know that Docker Swarm doesn't have a sort of pod concept and neither does MazeUs. But, you know, they're called pods and a pod is a group of one or more containers. They deploy together. These containers are their own sort of machine in that they share the same IP address on the network overlay that you have for your whole orchestration system. And a set of ports. So for example, if I've got a database, Postgres database and a database proxy together in a pod, the port 543.2 is gonna be allocated to the database proxy and you can't connect to the database directly. You have to connect to the proxy which within the pod connects to the database. And, you know, they'll share storage and other things that are connected. So let's actually take a look at that. So I've got this which is in my repo and I'll give you the address later on which are just some examples. So here's an example of a very basic pod because it only contains one container, right? We've got the AscentOS base image and it's running Apache HDB on top. So this is a very basic pod and you see it listens on port 80 as you'd expect from an unsecured web server. And that's it. Now, what I'm showing you here by the way is what I regard as the, there's a variety of different ways and I'll talk to them in a minute about how you can configure and tell Kubernetes to deploy things. And one of those ways and the one that makes best in my opinion for sort of long-term maintenance because you can check it into a Git repo is to use YAML files, which is what this is, YAML or JSON files, and load those into Kubernetes because it makes it very easy to track. It also makes it easy for me to show you things as opposed to doing interactive commands or other stuff. So this is just an example of a one container pod. Now, obviously the whole point of having pods though is that you might want more than one container. So this is an example from Kelsey Hightower where we're taking the Ghost Node.js based blogging platform and we're fronting it with an Nginx web server because it needs a web server to front it because otherwise it can only support one connection. So this is a case where you actually want two containers in a pod, right? We've got the Nginx container, we've got the Ghost container and if we're gonna have multiple ones of those, we're going to want multiple copies of that set for whatever reason and then they would share a database that would not be multiple copies. Yes, so that's a simple pod concept. Not only a two hung up on pods because you just get used to this because one of the things is you start out with containers and you say, oh, okay, well I want to deploy a container in Kubernetes, well no, you only need to deploy a pod. I mean actually, honestly, most of the time I end up deploying one container pods. The other reason why we need pods and containers is stuff that is transparent to you but when Kubernetes is setting up a lot of things like configuration and storage and other things that are built into Kubernetes, it actually launches utility containers within the pods that you don't have to manually specify but even though, as far as you're concerned, it's a one container pod, it is often not actually a one container pod. So, okay, so I've got this pod thing, so where am I sending it? Where am I sending everything? Well, our next sort of concept in Kubernetes word is nodes, right, and a node is an individual machine, either a real hardware or a virtual machine. You're gonna have several nodes in your cluster because this is the whole point of it, right, we're building a cloud unless you're in development in which case you only have one machine and so let me actually show you the easiest way to do that if you just wanna try this out. So, I installed something called MiniCube and MiniCube is an all-in-one VM that runs one instance of all the pieces of a Kubernetes stack so that you can try it out for development and you download it and it's available for a variety of different platforms and it's good for testing your Kubernetes stuff and that's how we're gonna actually do some of this other stuff and download it from there and if you'd rather do OpenShift, we also have one called MiniShift from the OpenShift team. So, but each one of those individual nodes represents a machine, again, hardware, virtual machine. The one thing that needs to happen in advance of Kubernetes accessing them is that each one of these machines needs to run a program called kubelet which is basically the Kubernetes daemon and then if you're running kubelet, kubelet can actually run everything else that's needed for kubernetes. On your various machines and those are your nodes. So, let's see if we are, we're not deployed yet so I'll show you that in a minute. Starting with VM, it's slow. Which is why I start containers for everything else. Now, obviously if you're doing clouds and you're doing orchestration, part of this is that you actually want to have a scalable infrastructure. So you want to take your application or whatever it is, right? I'm gonna have my web application, that sort of thing and I want to deploy one web server back by that code. I'm expecting a bunch of traffic on here. This is a plugin for Pokemon or something and so I need to actually deploy 150 of these and that's one of the other things that Kubernetes is going to handle for you. Now, the core concept in scaling out is what's called appropriately enough replicas, right? Which is you have a specification for, this is my pod right with a web server container with NGX or something in there and then I'm just going to tell you to scale out, deploy a whole bunch of these and possibly scale up and down. And all you're doing is sort of duplicating that and then you give Kubernetes instruction and Kubernetes runs a scheduler that decides where to put those. The default scheduler uses sort of a round robin algorithm where it basically looks at all of your nodes available, sees what the load on those nodes is and then starts deploying things to nodes that aren't heavily loaded. What that usually ends up meaning is that say if I'm deploying 16 replicas of something and I have 20 machines, the 16 tend to end up on different machines. But in the default scheduler that's not enforced unless you add extra information. So let's see if we've actually got, yes. Okay, so we've got Minikube up. So again, for development, Minikube will start up and it will configure, you also have to install the Kubernetes client, kubectl and it will configure kubectl to talk to the VM. So here's our node. We have one node because it's a VM running Minikube. So we're going to see our one node and then we want to go ahead and deploy some stuff to that one node. So let's actually take a look at what we're looking at in terms of replicas. So okay, so I wanted to show you my presentation. I do slides as JavaScript. So the slides are basically a web application. So we're going to actually redeploy the slides onto Kubernetes. So here we have, I've got something called a deployment and I'll explain what that is later on. This is not important right now. But here we have some information. You have to give everything a name to tell it, hey, this is the name for this thing so that later on I can delete it or scale it or do something else. And then here's the specification, right? I want three instances of it running, which would actually make a lot more sense if we had multiple machines, but we can test it here. We've got three instances running and then I'm going to give it some labels that allow me to find it later, which gets important when we get into services and I'll tell you that in a minute. And then I'm going to tell you what container to grab, right? So we're going to grab the container, jberkis, kube101, version 0.3. And then I'm going to have it listen on port 8000 because it's actually running the Python mini web server. So that's pretty simple. So let's go ahead and deploy that. So we've got deployments spinning up here and we've got pods and we're up and running. Now, obviously if I was actually deploying this for the first time, that would have taken a little bit longer because it would have had to download copies of the containers. But I didn't want you to wait through that, particularly not off of conference Wi-Fi. So that's sort of it for replicas and this allows you to actually sort of scale up or down and we can actually do things like, did I call it that? Yes. And so obviously we've just started up another one, so we've got four, right? Because you say, oh, these are all busy, I need to do more. Now, there is CPU based auto scaling built into Kubernetes. So you can also tell it, hey, anytime these are over 80% CPUs start deploying more of them. I'm not going to actually demo that, but just know that it exists. So that's pretty good, but there's a problem here. So I've got that listening on port 8000 Hold on, let me get the address. Okay, so I'm going to go ahead and try to listen on port 8000. Oh, that's a problem. Because what's happening actually with that port is that port is only available inside the Kubernetes system. Because in any microservice architecture, you're going to have a lot more services connecting to each other than you have available from the outside. So I actually need to tell Kubernetes how do I make this available from the outside? Or available specifically to other applications by name. And this is called services. So services are our wrapper that give you a way to access microservices by name and sort of a contained way from each other or access them from external networks from the web. There's also a second thing for sharing called ingress, which I'm not going to demo because it requires configuration with external DNS server, which I don't have set up. Which is designed specifically for web applications because it gives you path based access, HDB path based access where you have things like, well hey, the main application is at my app, but then I've got this other application at my app slash admin. For people who are familiar with Flask and some other things, it's actually fairly intuitive on that. So let's actually show. So this is the service. So again, another YAML file here, right? And I have to give everything a name. Names are scoped to whatever type of object it is. So you can actually reuse the same name. So this is, I have a Kube 101 preso deployment and a Kube 101 preso service. And this won't conflict because I have to tell Kubernetes what kind of object I'm asking for. And again, I add some labels and that sort of thing. You see, I'm not really making much use of the labels, part of the reason, in a real application you would because one of the things you can do with labels is you can use them as sort of search selectors for things. And then I'm going to give it a specification. Now, in a real production environment, I would have some kind of a load balancer set up in an external DNS server and that sort of thing. And then the service would hook into that. But I don't have that set up. So what I'm actually displaying this on is a type of service known as a node port. And a node port says, I'm going to allocate a port for my whole cluster that that port will be available on all the nodes of my cluster. And if anyone connects to that port, they will automatically connect to the service. It'll get redirected to the service. And then once I actually push this, Kubernetes is going to update a bunch of rules for the internal software defined network and go ahead and route that stuff. Now, if I were in a production environment, I would probably use a type of load balancer. I would have a load balancer plugin loaded for whatever load balancer I was using, whether that was Google Cloud's load balancer, AWS's elastic IP or whatever, or a Cisco thing if I'm doing internal sort of stuff. There's plugins for a bunch of things. And then that would actually hook up to the load balancer so services would get directed appropriately. But node port you can always play around with and it's a good way to test things. So let's go ahead and deploy that. So now, so I need to actually use that. So by the way, for node port, the IP address range it uses in the 31,000 to 32,000 range just for, you know, because it doesn't want to get restricted by secure system rules that require, you know, that reserve lower ranges, et cetera. And that's just the range that uses for node port. And there we have a presentation. Ta-da. Except there's going to be more to show you. But let me go off and do sort of a side thing. Now, this presentation doesn't require any configuration. But there are other things that do. So one of the things you need all the time is, for example, okay, you're going to have a different set of configuration for development than you have for production. Like for example, you're going to have for development, you obviously are going to have a whole different set of passwords than you do for production. At least hopefully you do. You're going to have maybe some other different, you know, sort of specs and that sort of thing in test and configuring the application and number of nodes and that sort of thing. The only thing you might be doing is if you're hosting an application for somebody else, you might want to actually create multiple instances that application tailored to each one of your users, right? So this is Joe's web app and this is Mary's web app and this is Rashi's web app. And in that case, you're going to have a different configuration for each one of those. Plus you have to do things like set passwords. Sometimes you want to tweak performance settings and that sort of thing. And so we need some configuration methods. Now there's three main methods in Kubernetes. One is you can pass environment variables to the inside of the container. A second one is something called config map, which can be passed into the container either as environment variables or as a file. As long as you can support the, you know, the variable, as long as your application will support the simple INI file sort of format. And then there's secrets and the built-in secrets in Kubernetes is an obfuscated password facility for a real production quality secrets. You would actually use a plugin utility like Cache and Corpse Vault that actually has robust encryption and automated rotation of passwords, et cetera. So let's actually show you a couple of those. So I'm not going to actually demo these. I will actually be demoing them tomorrow for anybody who's at the Postgres Day tomorrow. So this is for a scalable Postgres on Kubernetes thing. So this is an example of using the secrets facility, which just has hashed passwords. And then this gets passed in and then those passwords I get allocated as a tiny tempfs on the local machine and, you know, and get available to the individual, the individual containers. And then if we actually look at, is this the wrong copy? This is probably the wrong copy. Okay, hold on. Ah, there we go. Okay, we're just having scrolling problems. Okay, so ignore the rest of this, but then we have, you know, this sort of thing where, again, because of how I've set up the Petroni application, it still wants to read these things as environment variables. And so I'm passing a number of things. I'm passing some things as raw environment variables like what is the name of the cluster that I'm joining? This is cluster of Postgres nodes. And then we have other things that are gonna be pulled from somewhere else. Like for example, these passwords are getting pulled from the secrets facility. And so I'm referencing them by name and then they actually get passed along to the main servers. And then all of this gets loaded in by a startup script within the Petroni node so that the individual Petroni node gets configured correctly. So, now I did show you some database stuff and the whole reason I got involved with Kubernetes to begin with was that, frankly, I wanna run databases on the container cloud. This is what I do. So, stateful services are a big thing for me. So there's a couple of different, a couple of different facilities that you need to talk about in Kubernetes. So one of the Kubernetes that's out of the box is the ability to actually manage storage. And in the Kubernetes dictionary, there are two terms for this. One is volumes. And volumes refer to local storage on the node so you can actually have a container write some files individually to the node in a directory allocated to the container, usually in Docker's storage space on the system. And the second thing is persistent volumes, which is what you're gonna use for more production thing. And persistent volumes attached to a Kubernetes plug-in for whatever you've loaded that will write to shared network storage of some kind. So that could be Gluster, that could be SAP, that could be EBS, that could be the Google storage engine, that could be whatever you have that is providing storage to your cloud instances or the shared storage that you built. And the reason that persistent volumes focus on shared network storage is that, of course, one of the things that we need to be able to do with pods and services and everything else is relocate them across our cluster. And if we're storing only things in local storage, then it becomes up to the application to handle moving that data instead of it being available via Kubernetes. Now, there's a whole facility sort of beyond this called Stateful Set. And Stateful Set makes use of a whole bunch of other things in order to handle services that have a lot of Stateful data like databases, including persistent identity and routing and associating storage with particular instances of running pods and that sort of thing. And I will be talking about that tomorrow, specifically focused on Postgres and on Sunday, which I'll be talking about Stateful Set in general and the theory behind this design and what it does and what it doesn't do. So that's gonna be all for that for right now. So let me actually, I'll show you something briefly for that. Oh, that's right. So here's a more easy example. This is just setting up a persistent volume as an object to make it available for pods to use that is for Amazon EBS. You know, and you're basically allocating a particular volume on EBS to be available for storage for that. And then you can actually reference that within the individual, you can reference that within the individual pod definitions as this is where my storage is for this pod. So, okay, so that's great. We've got everything set up. We've got it deployed. We have initial application deployed. But now, well, let me show you an example of this problem. Does anybody see what's wrong with that presentation? Yeah, this is the version of this I gave at Faustum. Well, it's not what I want for here. This is scale. I need the new version of this. I need version 0.4. So what am I gonna do about that? Well, so in Kubernetes 1.3, I think, we introduced deployments. And a deployment basically represents as an object an instance of an application. And it has some sort of continuity of identities that, hey, I deployed this application that's got these six replicas of this and these four replicas of this other thing and these services, et cetera. And so that is an instance of my application and then I can actually modify that at runtime and update it. And one of the things that does out of the box is it supports rolling updates. Like automatically, if I tell it, okay, I want a new container for X, it will by default replace one at a time instead of bringing the whole thing down and then bring it back up, which is generally what you want. And if you want to modify the behavior of rolling updates, you can actually specify a bunch of variables about how many to replace set of time and how long to wait and that sort of thing. But let's actually just do a simple rolling update thing here. So, okay. So, in this case, I've got, I had the wrong version here. So I actually want version 0.4. That did not take long at all. So you can see right there, it already killed off the three old versions of the pod and is bringing up three new ones. Actually has brought up three new ones. So let's see if that worked. Oh yeah, there we go. Yay. One of the other things that I actually did because I didn't go back, I didn't reset the number to four is it actually killed off four instances and brought up three because that's what I had in the file. Okay. So let's get back to where we were. Okay. And that's your basic functionality of Kubernetes in a nutshell. So there is a lot more things that you can do with Kubernetes and a lot of special features and a lot of plugins and a lot of other things. But I mean, frankly, if you just actually figure out pods, replicas, services and deployments, then you're actually already ready to use Kubernetes for something real. And it doesn't take long to actually learn that. Now, so you can talk a little bit about how some of that stuff works. Just in case, because early on, when Mike was asking questions, we've got a lot of sysadmins in the room. And so for your sysadmins, you wanna know what's actually gonna be running here. So for Kubernetes, you're going to have one or more master nodes. And the master node actually runs a whole bunch of different processes. There's going to be a connection to an ETCD cluster. ETCD is what's known as a distributed consensus store. It's a way to actually have consistent data that is nevertheless available across machine failures. And so you'll have three, five, seven nodes of ETCZ that you need to connect to. You're gonna have an API server, which is what everything else in the system connects to. Is that API server? So again, I'm using kubectl. It is connecting to the API server to do all of its stuff. You're gonna have a scheduler at schedule services. You're gonna have a controller manager that actually makes things go up and down, et cetera. And then that connects to all of your nodes and your nodes have the kubelet running on it. Docker, at least at the present time. In the last, actually right now, you can do Rocket also. If you're into CoralS Rocket, you could actually use Rocket instead of Docker. And a proxy service that plugs into the software defined network to make that node part of the internal Kubernetes cluster network that's there. And by the way, people say it's like software defined network and network overlay and that sort of thing. It's really just a bunch of IP tables rules. A bunch of IP tables rules and then a small network widget that keeps track of where stuff is supposed to be. It's not actually that complicated. So, EGCD, Distributed Consensus Store holds metadata. It allows you, for example, to switch where if you lose your Kubernetes master, you can bring up a new one quickly and automatically because it just needs to connect to EGCD because that's where all the data is. The, now, what I was showing you in terms of how to modify objects and manage Kubernetes was using YAML files, which I said, I support because I find it's the most sustainable way to actually work with it because you check all those and to get your application is version controlled, everything else. But that's not the only way to do it. For example, if you have a valid connection to the EGCD cluster, you can just update the JSON in EGCD itself. And if you update the JSON, those changes become real on the Kubernetes cluster because the kubelets are polling for changes all the time. You can write a lot of kubectl one-liners if you just need to deploy a basic container image. You can just do that, kubectl, create whatever. You can do the YAML file things there. The only thing you can do is that it is an API and so some programming languages like Python have drivers for the Kubernetes API. So if you would rather write your deployments as Python code, you can do that. And there are reasons why you would want to go ahead and do that. And there will probably be, as Kubernetes gets more mature, people will probably develop even more editing utilities. And of course, you can use a more sophisticated system on top of Kubernetes like OpenShift that lets you manage a whole cluster through a more user-friendly system. So there's a whole bunch of these other, because of course we believe firmly in the microservice architecture in the world of Kubernetes. And so for this microservice architecture, we have everything within Kubernetes runs on microservices and there's other things that can go over like the DNS servers and because we have to have internal DNS so that the services can connect to each other and all these other things. So to recap, the things that Kubernetes does, right, is we group stuff through pods, we deploy stuff under nodes, we scale out using replicas, we share stuff, all of our microservices using services, we have a variety of configuration services and we have volumes for storage. The, there are some other, whoop, you know what? I apparently uploaded that too soon. We're missing a couple of bullet points there. There are a few other Kubernetes things. Federation for doing multiple data centers. There's now something called Helm, which is a different way to configure multi-container app to template multi-container apps. I, CoreS, one of the things I've been excited about CoreS has been working on this thing called the operator pattern for expressing application semantics through Cube, like you wanna say, hey, I wanna actually call the Postgres object through Cube and tell it to backup, you know, through the Cube API. And so that's actually pretty cool for me because of what I work on. There's ways to run Kubernetes without Docker if you have some reason to do that using like RKT and this very alpha thing called cryo and that sort of thing. We don't need to, you can look those up later online. You don't actually need to know any of those things to run Kubernetes. I mean, the big thing that Kubernetes does for you is it lets you turn a set of containers into an application that you can deploy and that's what it's for. So, questions? Yep. Okay, right over there. One of your slides, I noticed you had, the one that you said for like CIS admins, I noticed that you had your control plane and your workers are two separate entities. So, assuming that there is some kind of network connectivity between the control plane and the work plane, there is no reason, there's no requirement to co-locate those services, correct? Right. Correct, there's no requirement to co-locate them. Generally, your Cube master or masters will be running on their own machines or virtual machines just because they have a lot of load running the Kubernetes system itself. Do you follow me? So you don't necessarily want them running an application workload that might use up all of the CPU or the RAM on the machine. And that's caused the whole Kubernetes system to slow down. That also ties into another question that people get a lot where they say, okay, we're running ETCD for Kubernetes. Can I use the same ETCD cluster for applications? And the answer is yes, you can, but it's a bad idea because if that ETCD gets so busy with applications that it can't answer Kubernetes requests, then you as an administrator can't do anything about that because you can't shut down the services because Kubernetes is not responding. And so you can actually, like in Minicube, I am running the applications in the Kubernetes master stuff all in the same VM, but you wouldn't necessarily want to do that in production just for reliability reasons. So we've got somebody else over here. How does it connect in with a large physical load balancer? Do the services add sort of network hops and latency, where does it go directly to the containers themselves? So the first question was, how does it access a large load balancer? And that's through the load balancer plugins. So you'll need to load a plugin for whatever load balancer you're using. Generally, if you use like the prescribed installers on AWS or GCE, well, GCS Kubernetes is built in. That automatically plugs into their load balancers in their cloud. If you're using a hardware load balancer, then you're going to need the plugin for that. I don't know the whole universe of plugins and what's supported and what's not. You just look. There is going to be, there is necessarily going to be some extra latency because generally you're adding an extra hop or two in connecting to the services, right? Because you're going to connect from the load balancer to one of the Kubernetes nodes. And then the service, then the pods you want to connect to may not be on that Kubernetes node and so you have an extra hop. And most of this runs through IP tables, rules, et cetera. So it's pretty fast on the individual machines, but you are adding extra hops. And so if sub millisecond latency is your problem, then yeah, you're going to notice. Most people don't notice. Okay. So you mentioned that there's an API for the Kubernetes platform. If we're already using something like Salt, per se, would we be looking at using maybe a state or maybe a formula in Salt to connect to it? Or would we be, or is there something in Kubernetes that you can use to connect to a configuration management system? It would go the other way. You'd be looking for a module for Salt that actually has Kubernetes support. And one of the questions actually we had for the last speaker was, how does configuration management fit into this? And different people have different ways that they deal with this, but from my perspective, I do a lot of Ansible, and I use Ansible to manage Kubernetes. So what happens is the configuration moves to a more centralized role, where you use it to build, you know, build my templates for my applications and that sort of thing and manage Kubernetes. I know Ansible is a plugin for Kubernetes. I don't know if Salt has a module. So just look on Salt for that. Yeah, okay. How would I interface with LogStash or other logging infrastructures? How would you interface with LogStash or other logging infrastructures? So that's an example of exactly where pods are a useful concept, because what you would want to do is in each of your pods, you run LogStash as a pod and actually have it collect logs from the other containers in there. I mean, there's one way to do it. The other way to do it is obviously all of these containers, if they have been built correctly, are dumping stuff to standard out, which is getting collected centrally on the machine. So the other thing you can do is you can run something like LogStash as what's known as a demon set. And a demon set is simply a special type of pod that runs one per physical node. And so you would run LogStash as a demon set, so there'd be one in each node and those would forward to whatever your central log application is. I had a question. Do you see Kubernetes and Docker kind of solving similar problems? And if so, what advantages does Kubernetes have over Docker? Okay. So Docker engine operates at a different level than Kubernetes. And like I said, in the example I'm showing you, Docker engine is what's running on the individual nodes. Docker swarm is the thing that is the direct alternative to Kubernetes. And there are some different architectural decisions there, like I mentioned, Docker swarm doesn't do pods. And Kubernetes has pods. There's some different architectural decisions. I'm not actually all that familiar with the architecture of swarm inside. And even less familiar with some of the other alternatives like Mezos and the thing from AshleyCorp. In a lot of cases, with any of these distributed systems you have to make a lot of trade-offs. And so having all of these different teams working on different versions is really helpful because it lets people try different trade-offs and see how they work out. But other than that, the nice thing about it all being open source is give it a try. If you're not sure about Kubernetes versus swarm, try them both. In your example where you updated the slides, I'm wondering about like say, would you run Kubernetes in local development where say you had a front-end that's dynamically compiled like watching files and compiling JavaScript and you have a server that you're working on that code and it's restarting the process, would you have to like rescale up those nodes in Kubernetes or is there a way for you to have live updating while you're developing locally or is that just not a use case that you would use it for? So, I mean first of all, if I'm doing full CI CD, that would be a reason for me to use OpenShift because that's one of the things that OpenShift adds to Kubernetes. And the second thing is it doesn't really affect scaling because like say if I have 20 instances of this application and I do a commit and that's a new version of the application and the CI CD is going to deploy it, when it pushes the deployment, it'll do a rolling update. So it'll replace those 21 at a time. Was that your question? I'm not, ah, okay. So question if you're developing locally and you want to have it update as you're coding. I've actually seen that. I know Dusty did that as a demo with OpenShift which is based on Kubernetes and some code. I don't personally know how to do that though. So if you want to ping me by email or something like that, I can actually connect to you later on. So from creating a cluster point of view, would you rather have, so if my application, like different applications have different resource requirements, would you create a huge cluster of same type of instances or would you rather have different clusters for different type of instances? Okay, so I don't know if you were in the first talk but I'm gonna give you the same answer that Mike gave earlier, which is it depends. Because really the question is what, how different are those different requirements, right? If different applications just require different amounts of CPU and memory, there is a whole resource allocation thing built into Kubernetes that operates through Linux C groups where it'll, you know, where you can tell it limit this to X amount of memory, et cetera. But if they're different in, hey, this application actually needs GPUs, then you're probably gonna wanna handle that in some special way. Now there's two ways in Kubernetes to handle that. One is you can add labels to the nodes. Like I said, these are the GPU nodes, add a label and then run a custom scheduler, which sounds scary but there's actually a whole toolkit in Kubernetes to customize your scheduler that actually pays attention to those labels when it sends applications out. The second way that you can do that is federation. That is in general Kubernetes federations intended for people who have like say distributed data centers, but if you have like this set of machines that have super fast, large SSD storage and this set of machines that are more generic, then you can actually, even within the same data center, have two different Kubernetes clusters and federate them. So, and which of those makes more sense for you is gonna depend on sort of finer grain things. And this will be the last question. What would you say is an easy way to expose this as like a self-service tool for say like development? Okay, what would you say is an easy way to expose this self-service tool for development? Well, again, I'm gonna bring up OpenShift because that's what it's designed for. OpenShift is designed to be a multi-tenant system among other things so that you can actually give people application access and they have access only to their applications. If you wanna come by the Red Hat booth, probably tomorrow afternoon, certainly on Saturday, I'm actually gonna have a little tiny OpenShift cluster running and I can actually show you more of that. Okay, well thank you. And we have another speaker coming up next. Thanks. Okay, if everyone would take their seats, we're going to start up here. So our next speaker is Murley from Rancher Labs, who is a network guru there for RancherOS. Is that what you were looking for? Rancher. Yeah, for Rancher, okay. And he is going to give us a practical use case for refactoring or how to refactor to microservices. So welcome. Thank you. Thanks for the introduction. Thank you everybody for joining me today. So here is the high level overview of my talk. So we'll start with the introduction and we will go over what microservices are and why we need microservices and how do we build or refactor microservices. And I will share some of my insights, lessons and tips that I've learned so far in the past, year and a half that I've been working with microservices. After that, let's containerize one of the applications and open source application called FreeCodeCamp and let's see if we can run it. A little bit about myself. So I am the principal software engineer at Rancher Labs. So what I do is I build pipes. So I build pipes between various containers so that you can send data across. Just like Mario goes inside a pipe and then he shows up in a different world, you could send data from one container and it shows up magically on the other container. That's what I do. I take care of the plumbing. And you can see my contributions on GitHub. I work on a bunch of networking things. So one of the recent works that I worked on is I built a new microservice which takes care of securing the overlay networks. So if you're a good guy and you're not supposed to enter a pipe, I'll tell you nicely, hey, you cannot do that. And if you're a bad guy, I'll just make you disappear. And some of the other works that I worked on is I refactored IPsec and created a new microservice using VXLAN overlay. So just to give a quick overview, IPsec is an overlay technology, networking technology that is used for public cloud. So if you are sending data from one cloud provider to another cloud provider over the internet, you would want to use IPsec. So it's like Mario on Superstar. And if you want to use networking within your data center and you know it's secure, you would want to use VXLAN. So it's like you just zoom through it. And I am talking more about networking this Saturday. So please feel free to join. Okay, I hate marketing team for making me do this. You know, I'll just not go through this. Okay, so what are microservices? You know, I went online when I first started working on microservices, trying to understand, okay, what microservices are. Okay, here are the different definitions. A microservice is a specialized SOA architecture, okay? And what exactly is a service? A service is something that's providing a need or a value or you know it's doing some kind of, I don't know. And what exactly is an application? An application is a group of services, okay? And what is a monolithic application? An application that is big, that's just one. That's simple, right? That's what I felt when I started reading. Well, do you get it? This was my reaction when I first started jumping into microservices. So I wanted a more practical example. You know, I like things, you know, head first way. So, okay, sure. So let me explain my way of explaining microservices. Where is Mario, by the way? So he found his co-founder and they are on to a much difficult quest next. So let's begin the next quest. So the quest is, you know, the same dream that we all share, you know, we want to build a multi-billion dollar company. You know, of course, we want to change the world on the way. So Mario wants to, you know, start a company. So he looked around, okay, I have a castle and let me bootstrap with it. So I want to build a bed and breakfast company. So I have, you know, different services that I can offer. I can offer the bedrooms in the castle. I can offer kitchen, the dining room, living room, et cetera. So he fit in a bunch of beds in the bedrooms and started this B&B business. It was a hit. You know, people loved it. They wanted to visit the castle and, you know, people, the demand started flowing in. And then Mario hit a scale problem. You know, he had only so many bedrooms that he could, you know, put it out. So he wanted to scale, you know. What was one of the options that he had? He used the magical mushroom. Just, he made the castle much bigger. So he just threw in more beds in that and he was able to scale. So this is an example of how some of the services or some of the applications, which have been built in the past, scaled up. You know, you throw in more CPU. You throw in more storage. You throw in faster networking. That's how you were able to scale. And after this, you again hit, you know, element. How do you scale further? So the next solution is, okay, I want to buy more castles. Just throw in more servers. But here, the problem is Mario's business needs more bedrooms. So that he could accommodate more people, more guests, more customers. But if you buy a castle, you have an additional overhead of kitchens, of dining rooms, which are not so useful. So the next, you know, solution, you know, a good solution for Mario would be, take the same castle, just scale up only the bedrooms, which are the most needed. So this is one of the ways I could understand what a microservice is. You know, there are like, a lot of services in an application. And, you know, if there is one service that I would like to scale up, and if I could detach it from the rest of the application, you know, it's something called a microservice. So that's how I explain it to myself and I explain it to others. So, you know, with this model, with this microservices architecture, what you could do is, you can just scale only the services that you need on demand. You know, you can scale up, you can scale down. So to summarize what we have seen so far, a microservice is something independent. It can exist on its own. It does, it needs help of other things, but you know, it can live on its own. And it's flexible. You know, you can scale it up, you can scale it down, and you know, you can move it around like you can move one of the microservices from one operating system to a different one, things like that. It's very flexible, and it's very replaceable. You know, when I say replaceable, it means, okay, today, you know, the Mario's bedroom are built out of brick. You know, tomorrow he comes up with, oh, there is a new building material that I could use to build the bedroom so he could just replace it. So in the same way, microservices, if they are implemented today with Python or some other language, you could just replace them with a different programming language, a different implementation. You can refactor them easily. And microservices are upgradable. So what it means is, if you have a certain version of microservice and you want to add a new functionality and you want to, you know, make it run faster, you can put it in a new microservice and swap it out. So that's an upgradable services. So these are some of the good things about a microservice. So why exactly do we need microservices? So from the discussion so far, one thing is quite obvious, you know, when companies or when teams hit scale issues, that's when they start looking for solutions and microservices provide a good way to scale up. So the other way we could look at of, like, you know, answering this question, why do we in microservices is look at the problems of a monolithic architecture. So what happens in a monolithic architecture? You know, one of the telltale signs of a monolithic application is you have a large code base and you have a lot of developers hacking, developing stuff on the same code base. What happens when you have, like, a huge code base? You know, I'm sure you must have used IDs in the past. If you have a big, giant code base, it takes a long time to load the ID. And when you are trying to make changes, when you are trying to run, it becomes very difficult. And also, it's kind of difficult to manage different developers to work on the same thing. You know, oh, I need to get in this feature. The other developer wants to get in his feature. So how do you coordinate? So you need management teams. So you have a lot of overhead. And if one of the developers decides to upgrade one of the packages, it just, oh, I want this new feature, I'm gonna upgrade this package. But what happens? You know, the other developer is using the older version and it just breaks. So these are like some of the problems during the development process. And once you have this application deployed in production, if there is a problem, what happens? You know, there could be teams who would be blamed or who would be brought in to analyze, okay, what has broken? And most often, it happens that the teams who are not responsible for the breakage end up debugging the whole thing. And those people may not have the expertise of where the problem lies. So what could take a day for the right team? You know, it could take a week for the wrong team to debug it. They would definitely debug it. But you know, from a developer perspective, it's great. You know, I'm learning, you know, other modules in my product, and I'm expanding my breadth, but that's not good for business. You know, think of if your service is down for a day, that's really bad. And when things go wrong and you fix them, and finally, you know, all the XX come back and then they ask, hey, what happened? Why did we have an outage for an hour? Then, you know, nobody wants to take responsibility. And you know, nobody wants to stand up and say, hey, it was me who caused, you know, millions of dollars of loss for the company. Everybody starts pointing around, oh, it's that team. It's this team. But on the other hand, if you had like an outage for five, 10 minutes, you know, you just fix it, and then if it exists as, oh, it was just a down time for five, 10 minutes. So why microservices? If we look at the opposites of the problems, you know, these are some of the good things of microservices. So you have small code bases, they are much easier to manage, and you have a small team, and the appropriate size of the teams working on these code bases. And generally, it's proven from, you know, research and experience across the industry that, you know, the right team size is about, you know, if you can feed your team with two pizzas, that's about the right team size, you know. It's perfect. It works, you know, as a well-oiled machine, you know. There's not too much free space, and there is not too much friction as well. It's perfect. And the teams of this size tend to deliver stuff much faster, with much high quality, and, you know, a lot of good things. With microservices, the advantages is, you know, each of these microservice can be built, tested, and, you know, scale independently. You don't have to worry about, how do I, you know, coordinate with the project managers, or how do I deal with other teams? How do I communicate? How do I get these approvals? How do I deal with the conflicts? You don't have to worry about it. You just focus on your microservice, get it up, and running. With microservices, you know, we have clear separations. So when there is a problem in production, it is very easy to debug. You know, you look at the logs of the different microservice, you can just easily, very easily tell, oh, it's the DB which is having problem, or it's the networking which is having problem. In the past, in my experience, you know, when something doesn't work, the first thing people used to do was call the networking team. I was like, okay, everything is broken is because of networking. I jump in, I start looking at the logs, start debugging, and then finally figure out it was not a networking issue. It was a problem with something else. So these kind of things can be avoided when you are using microservices. And the other good thing which a lot of developers love about microservices is they can bring whatever language they are comfortable with. You know, if you have one giant application which is built in Java, and if you happen to be a person who likes Ruby, or you like Go, you are forced to work on Java. But with microservices, you can bring in your own language. You can bring in your own tools. So it's more empowering. With microservices, the advantage, the other important advantage is refactoring becomes much easier. Because each microservice is independent, you have tests which are specifically written for this microservices and you can run them. So when you refactor, you will get an immediate feedback loop whether the tests pass or fail. So developers are more confident when they want to improve an existing microservice. With all these things, the CI CD pipelines are easier to build and use in companies. So are microservices are the best thing in the world? Are they free? Of course not. Everything comes with a cost. So the first cost is the complexity that comes with running a large number of microservices. Earlier, if you had a monolithic application running on one server, all you had to deal with was like, okay, I have this four or five different processes, four or five different logs. Can I do PS-EF and then figure out what's going on? Things like that. But now imagine you have 10 different microservices running on 10 different servers. That means you are looking at 100 different things running in production and they are very hard, very complex. They need more resources to be managed. So you have a lot of overhead when you deploy microservices. And for taking care of these microservices in production, you need ninjas. You need to bring in the experts. You need your team to be well-versed with the best practices. And they should be provided with all the right tools to be able to debug and enable logging, security, a lot of things. There's an additional overhead. And the important thing with microservices, with microservices architecture is the microservices talk to each other based on contracts. So if one of the contract is broken, if one of the API changes, it can have a cascading effect, just like dominoes. If one thing fails, it can just create havoc. It's very similar to monolithic application, but at least in microservices, you can protect yourself. You design microservices to handle failure. The other disadvantage is because you have developers working independently. Sometimes it happens that you end up with duplicate efforts. Two developers working on the same similar feature, they create similar libraries, and there's a duplication of effort. Testing challenges, this is very interesting. Although microservices can be tested standalone, but when they go into production, you do not know how many of them are running. It could be 10 or it could be 100, or when there is no load, there could be only one. And a certain problem can happen only when you are scaling about 10. So to recreate this kind of topologies, the kind of setups in-house during testing is a challenge. You cannot predict everything. So these are some of the costs of running microservices. So it's not free. To summarize whatever we have seen so far, we have the monolithic where everything is packaged together. It scales together, but microservices are individual. You can just scale only the services that are needed. And the release cycle in a monolithic app is larger, whereas with microservices, you tend to be more agile. You can release stuff faster. So it's easier to build scale with minimal infrastructure and computing resources. So now that we have seen what are microservices and why some of the people are motivated to use microservices, let's see what are the factors you have to consider when you build or refactor some of the microservices. If you are building from scratch, if you are a new company or if you are a new team, if you have a chance to start working on microservices, the first and foremost thing that you have to do is you have to plan. You have to come up with a design. You have to identify what are the responsibilities of this particular microservice are. What other microservice does it depend on and what other services depend on me? So you have to identify the interfaces much ahead of time and you have to write your unit test first before you write code. This is one of the most important things. Like I mentioned earlier, different microservices talk to each other based on contracts and you cannot break those contracts. And sometimes you have to break them so that you could provide new functionality and you can come up with a totally different one. So you need to come up with deprecation policies and you should have proper documentation for all this stuff what each microservice is offering, how it's going to deprecate and how you need to update and things like that. And if you are a big fan of microservices, don't just go overboard. If you have to start chopping up you could end up with very small microservices for nothing. Like you can end up with like 100 microservices. It would be overwhelming. So just find the sweet spot where you can divide your application into the right functions. It need not be like, okay, I'm gonna have logging as one microservice, I'm gonna have UI as one microservice. Think of it as a microservice is providing some kind of functionality. It can come up with a UI, it can have its own logging but it does one thing and one thing well. That's how you identify what to build as a microservice. And just because you have an option to use whatever language you want, and let's say you are building like five different microservices in your company, you can start using five different languages. Each developer can bring in their own language but it comes with its own drawbacks. If one of the developer moves on to a different project or a different company, the other developer taking up that microservice for sustenance and maintenance he will have a tough time. So probably if you could stick with two or three languages that would be good. But if you have 10 different languages it would be very difficult. And the most important thing when you are building a microservice is be prepared for failure. Think of all possible failure scenarios during your design phase. What if this happens? What if I can't connect to the database? What if my dependency microservice doesn't start or it doesn't show up? How do I handle it? How do I handle failures gracefully? It is one of the most important things. A lot of developers just move the normal application into microservice and they forget about failures and when they take this microservice into production it fails very badly. So please be prepared for failures. It's not about why failures will happen or will they happen. Most of us who have experience they know that failures are definitely going to happen. It's just a matter of when they will happen. And at this point this might seem a little off topic but I would like to talk about Conway's law. What this law talks about is in an organization whatever is the hierarchy of the structure is generally the design of the software follows the same pattern. So let's say you have one architect who has like six or seven developers and if all of the developers are talking to the same architect the architecture of the software that they are building tends to be the same. So this law is very important. The reason I am bringing it up is when you are building microservices you would want to empower your teams who are building microservices. You want them to be independent. You want them to learn from their mistakes. You want them to experiment. You want them to fail and learn. So that way they can iterate faster and they can build stuff in a much better fashion and a much better way. So if your organization is not encouraging the teams to take their own decisions even if you start using microservices the design of the overall product will represent your organization structure. So if one of the teams is responsible for getting all of the releases your design will follow something like that. There will be one bottleneck. You would want to give freedom to all the developers and make the best use of their talent. Now if you are not building from scratch and let's say you are already having a legacy application and you are having scale problems or you are trying to refactor this application at this point you are trying to evaluate if microservices architecture is the right one for me or not. The first question that you have to ask is are microservices the right thing for me? You could be having scale issues. So if it is something that you can solve by having the mushroom like just throwing in more resources probably you don't need a microservice. You could just live with the same monolithic architecture. But if you have really the scale issues that's when you have to decide okay microservices is the way for me to go. The first step is when you are refacting an application is you would want to repackage your existing stuff into smaller pieces. That's the low hanging fruit. Identify the different components in your monolithic application and start packaging them separately. That's the first step you could do to move towards a microservice architecture. The thing with the existing apps is you cannot just change them overnight. It takes time to evolve to microservices architecture. So you have to support the old one as well as the new one while you are trying to switch over to the new one. So it takes time. So start with the low hanging fruits and then iterate. So after you are done with repackaging the next step would be analyze your code. Where exactly are the bottlenecks? Which are the functional areas that I can separate it out as a microservice. And after that think of scaling it. Okay now that we have a microservice architecture which of them needs scale. And with scale we need more resources like you need to have logging infrastructure. You need to have monitoring infrastructure. You need load balances. So those things come much later. And once again even when you are refactoring please be prepared for failure. Implementation details. So how do we deploy microservices? How many of you have not heard about containers? Wow, nobody, everybody has heard about containers. How many of you have not heard about virtual machines? So to deploy microservices you don't need to use containers. You can deploy microservices with virtual machines. But it's just that containers are a good way to build microservices and to deliver them and to scale the microservices it's much faster. So this is my slide to understand what microservice or what containers are. They're like virtual environments that share the same host operating system. And because all of you know about containers I'm just moving on to the next one. So this is a slide that I had to explain what VMs are and what containers are. How you have an additional overhead of the guest operating systems when you're running VMs. Whereas in the container virtualization you don't need the operating system in each of the guest OSs. What are the different runtimes that you would choose when you are trying to run your microservices using containers? You have RunC, you have LXC, you have OpenVZ, you have Rocket, you have Docker. You have a lot of options. Pick what works for you. One of the most popular runtimes that all of us are familiar with is Docker. The Docker's mantra is you build once and you can run it everywhere. So this mantra has what made it so popular. You could just build on your local laptop and you could use the same Docker image to run on cloud as well. So how do we run containers? So the first thing is you have to pick a runtime for the stock I picked Docker. And the first cycle, the first step in running containers, you have to build a container image. A container image is like a blueprint. You know, it's, think of it like as a cookie cutter. You build a cookie cutter first and then out of this you make cookies and you run containers. And when you have built this cookie cutter, when you would want to use it in different places you have to store it somewhere where you could use it. So that's the Docker Hub. And once it is on Docker Hub, you could pull it in private or public clouds and you could run containers. So this is the kind of the life cycle of how you take a spec of how you want to build the microservice, how do you build the image, how do you push it to the Docker registry. You can run it in public and the private cloud. There are so many options available today in the market. These are some of the open source tools. Some of them are closed source solutions. So you have a variety of choices for various things. So I want to share some of the insights and other things that I've learned along the way. So the first thing that we have realized is converse law is not just theoretical. It is true. We have really experienced it in our own company. And I have these slides. I have put in all the lessons. If I just talk about each of this, it's like there's no context to it. Just like the way if I talk about what a microservice is or what a monolithic application without giving you a practical example, it's kind of difficult to get the context. I had a debate yesterday with my marketing team if I should have the next couple of slides or not because it's like talking about the product, about what I have worked on. But my take on this is I'm not pitching the product, but as an open source developer, I want to share what I worked on. And when I build a context, that's when other developers can appreciate what I am talking about. And they can learn from the experiences and because it's open source, you can look at it. And it could be useful for you when you are trying to build your applications. So a high level overview of the product that I worked on is it has a management server and the main responsibility of this product is to run containers at massive scale in production. So we have ability to run containers on different clusters. They are like isolated clusters. So if I zoom in a little bit, this is one particular cluster which is managed by the management server. So here, the thing that I wanted to highlight was, you see the blue container running in the management server? It has multiple processes. So these are like different modules or different services running in one monolithic application. And if you look at the compute node, we deploy services of our product on the different nodes inside containers. So this was like the first low hanging fruit that we attempted at, like, you know, just put everything in containers. So we started running different stuff in different containers. We started with two containers. So here, what happened was, even though we had multiple microservices, the architecture of this was all of these small components they were communicating to the management server. This one. So when we look back now, after we have evolved into a newer architecture, we realized that that was because of the way the hierarchy was working in our organization. So we had our architect, chief architect, working on this management server and the different teams were communicating with him. So when we developed the microservices, all we had to do was use the APIs exposed by this central piece of software and run on top of it. So there was no independent nature. I mean, we were independent to a certain extent, but it was not scalable. As all the microservices running in the different hosts or the different compute nodes, they were talking to the same central entity we couldn't scale. So we came up, that's when we realized, oh, sure, the converse law is really true. So we need to change the hierarchy of the organization. So we made some changes. We empowered the small teams that are working on different pieces of the software. Okay, you develop, you take care of the stuff and we have to change this architecture. So what happened later once we changed was we broke down the different pieces or the different components into different microservices. And here one key thing that really enabled us to be independent was the introduction of the locally available distributed data. So what it means is the reason why the different pieces of the software were talking to the central management server was you want to get data. So what we did is we, instead of pulling from the central management server, we changed, we reversed the roles, we pushed the data into something called as meta data. So these services, instead of talking to the central entity, we started talking locally. So that really enabled us to move to the microservices architecture. So because of that, the teams became really agile. We were like releasing much faster and we came up with all the best practices, come up with common libraries so that other microservices could share and we fixed upon like one or two languages instead of having like 10 different stuff, things like that. And we did learn this lesson the hardware, like don't build too many microservices. One of the things we did was let's break up this monolithic application into smaller pieces and run in microservices. We did that, but we ran into a lot of problems. That's when we decided, okay, for this piece of software, it's a bad idea. So we switched back. And not every problem is a nail. So just because we all love microservices, we cannot use them everywhere. It's not the solution for everything. You really have to analyze whether it's the right thing for you or not. So let's containerize one of the open source applications. So I picked up the FreeCodeCamp on GitHub. It's an open source heaven for the current century. And what FreeCodeCamp is, it's a free website which helps developers to learn JavaScript programming. And the technologies used to build this software is Node.js and MongoDB. So here are some of the steps used by FreeCodeCamp to run the software. So what you have to do is you have to install all the dependencies. You have to install Node.js. You have to install NPM. You have to install Bower and all the stuff. You have to run through the different setup steps to be able to run this application. So here, what's happening is when you have all these dependency steps, they're not the same for different environments. If you are running it on the Dev machine, if it's on your Mac, that's different. When you are pushing it out to protection, you could be using different operating systems. So it's not a deterministic way. The other thing that we are doing is we are polluting the host. Like you are installing all the dependencies onto the same host. So if one of the package needs an upgrade, it could potentially break other components. To have an analogy of the FreeCodeCamp application is you could think of it like the bedroom is the web application. And you could think of the database and the library as something that is not something that you are interested to scale. The application is what I am interested in scaling. So how do we break this apart? So the first thing is, remember the low hanging fruit? That's the first step you can do to containerize or move your monolithic stuff into microservices architecture. So two of the big pieces that I can see in this application are the database and the Node.js application. So I would like to repackage them separately and run them as different containers. So I had a few slides explaining about what the Docker file of MongoDB is doing, what the Docker file of Node.js is doing, how do you build and things like that. So this is the source code of FreeCodeCamp application. So here what I did is I introduced a new Docker file and here we are using the Node.js existing image and building on top of it. So I'm moving the source code of the FreeCodeCamp inside the container and I'm running the scripts inside them so that I could start them. So this is an example of the Docker file that I used for this and the different lines in this Docker file are going to end up as different layers in the Docker image. Let me just build this. So I have a script which does the building of the Docker image so here you can see I am building an image on my local laptop and I think I'm having some networking issues. Okay, so I'm not sure, I cannot debug it right now. So what I wanted to show here was I wanted to build the FreeCodeCamp application as two different containers on my laptop and then take these two images, push them to the Docker hub and then use the same images to run on two different servers in the cloud and one of them servers which I had set up was running Ubuntu operating system and the other one was running Red Hat operating system. So I wanted to start this application in the cloud to show you that the same image for the containers that we have built locally is the same thing that we are running in the cloud. And it's the same environment in development as well as in protection. That was my intention with this containerizing the FreeCodeCamp application and running it, pretty much it. Thank you, thank you for joining. If you have any questions, I think he has a question. If you were trying to work with developers to have them containerize their own applications, like what advice or guidance would you give them? I couldn't hear you, could you please repeat? If you were working with developers to help them understand containerization and to guide them, what advice or policies would you give them? So some of the, so the question is, what advice would I give to developers who are trying to containerize applications? The first thing that I would tell developers is like, you know, when you're trying to containerize something, you would want to use something existing, you know, something, try to leverage what is already existing. So let's say you want to package your Node.js application, instead of trying to build from scratch, you could use Node.js image. That's one of the things. And the other things is like knowing the details of what the different Docker commands do. That's the second thing that I can tell. And the other thing that I have learned running containers and building images is if you have multiple microservices trying to run on the same host, you would be able to save some time by using the same base image. So these are like some of the tips I could give for developers. There's another question. So I was curious, when you first start moving from like a monolithic application to containers, how did you tackle the service discovery at that initial step? So in our product, we built our own service discovery. So we have our own small little microservice which provides the DNA service within the cluster. So if you are using different technologies like probably Kubernetes or Docker's form, they have some solutions to provide service discovery. I had a few questions regarding microservices. And one of the first one is at which point do you think microservice, like there's too many microservices because it's very easy to go to extremes. So you can have too many microservices. How do you manage all of them? It's, I see this. Yeah, I guess the true question is how you, I have a question about testing it. So you can have a bunch of microservices and they all kind of- I can't hear you, could you please hold on? Oh, sorry. All right, so my question is that how do you test microservices that are overly decoupled? That are over? Overly decoupled? Overly decoupled, I see. So from my experience, the testing is in phases. So the first thing that I do when I build a microservice is write unit tests. So that makes sure I'm making sure the code that I have written is doing what it is intended to do. That's the first step. So I write unit tests. The second thing I do is can I spin up this microservice locally and then run tests against it? So that way you know, let's say most of the microservices provide services over REST API. So when you start the container, you get a REST API. So then I test it. That's the second step. The third step is I deploy the whole application together and test it again. When you test it again, do you, so for the tests, the unit tests, are you combining, are you grouping it based on that microservices or do you like, let's say you have three, okay? You have the identification service, like a database and application microservice, right? Together as a whole, do you test them for the test to test the whole? Do you maintain it in a separate package or do you kind of have each microservice kind of being aware of like a central testing thing? So the question is like, so when we have these microservices, how do we test that? Like do we test the same microservice or do we test it along with other stuff, right? So like, so the next step that we were doing in my experience is once we deploy the whole microservice, the first step in that would be, I would test the specific functionality that I'm expecting out of this microservice. Then I would test the whole application together, like usually we have test suites, we have automation suites. So we just run through them and make sure the rest of the application is working fine. So when you have a new microservice, it's hard to come up with this automation test suite because you don't know the functionality yet. So at that time it would be more manual. Sure. From your experience in terms of you're deploying microservices and also containers, it doesn't make sense for you for each microservice, say for each image to have its own specific Docker registry endpoints or would you actually try to do different like images in the one registry or each microservice on its own registry endpoints? So when you say registry, you mean like the repository and the tag? The repository. Okay, okay. So we prefer to keep them separate. Most of the microservices that we have, we keep them separate. The way we optimize the pool time, like you saw we are running 10 different microservices. So if we have 10 different images, when we were bringing up all of them, it was taking a long time. So the way we optimized it was we have one base image which has all the needed stuff in the base one and then each microservice, whatever is specific to that, we added as like a couple of other layers. So we start with one base image and then kind of extend it. So thank you very much. We're actually gonna finish up now. You can ask the question in person, like, because we're gonna finish up now and take a break for lunch and we will be back in this room at 1.15 for patterns and anti-patterns in Docker image lifecycle. So thank you. Thank you. Thanks, everybody. One, one, one, this is loud. All right. Yes. Okay, well, I guess it doesn't matter then. Howdy, everybody. I hope you had a good lunch. How many people here went out for tacos? Tacos? There we go. How many people went out for lunch? Same people. Same people. Yeah. Sushi, sushi. Oh, there we go. Awesome. Okay. Anyway, our next speaker is Baruch, who is a developer advocate for JFrog and he's going to tell us all about how to screw up our Docker images and shut everything down. Big time. Yes. Take it away, Baruch. Thank you. Hello, everybody. Thank you for coming. That's my first time at scale and it looks like a great show. So thank you for having me, the organizers. I know why it makes noise. I hope it's not me. Yeah. Okay, now it's better. And today we're going to talk about patterns and anti-patterns in Docker image lifecycle. So as I was already introduced, this is me, Baruch Sadogurski, developer advocate at JFrog. This is how I usually describe myself. It's a little bit too early, but I guess soon enough I will get to the first part of the statement. Show notes. As of now, you can go to jeffrog.com slash show notes and under scale 15 you'll find the slides and all the links, reference material for this talk, feedback form and a small raffle just to say thank you for being here. The video will be uploaded, I guess, later today. Somewhere between this talk and the part when I am already drunk. Someone in between it will upload the video. So come back later if you want to use it as a reference or something. So, yeah, I'm with JFrog. We sponsor this great event. So tomorrow we will have a booth in the expo hall where we will give away those amazing t-shirts. So, please don't forget to stop by and also hear about some software that we do which is Artifactory, Bintray, Mission Control X-ray. I won't get, of course, into details on any of those. I will use Artifactory as an example here for the stuff that I speak about for docker images for obvious reasons. But other than that, please stop by our booth to hear more about JFrog and its products. And now it's the poll time. You're all after lunch, so a little bit of hand exercise won't hurt. And show your hands, and when you vote, you keep your hand up until the option is not relevant for you anymore, right? Who heard about docker? Just for the reference, all the hands, all the hands? Yeah, all the hands are up. Who can do the tutorial? Tried or think they can do the tutorial? Well, a couple of hands were down, but pretty much all the hands are still up. Who did some proof of concept for work, played with it for work? All the hands are up, almost. And now, who has it in production? Yeah, take it down, that's okay, that's okay. You can take your head down now. Okay, so we left with, I guess, in what, like 20% of what we started with and of course it wasn't real poor, it was just manipulation. I knew exactly what was going to happen because it happens every time, right? And the real question is why? There are a lot of reasons, of course, why people hesitate to take this amazing technology to production and I'm going to speak about one of them and this is the lack of trust. And it's natural because a docker is kind of another layer of very obscure wrapper around what we already do. So when we have docker-containing running in production, how sure you are that what's run in it is what you intended for it to run. There is some process, continuous integration, continuous delivery of the docker image that we create the container out of it and the question is, is there enough integrity into this process so you can be absolutely sure that what you have in production is what you intended to have there when the developer first crafted the docker file, right? So we're going to talk about that. Any employees of docker-ink here? Anyone? Okay, I can run my jokes. Okay, whatever you see here is a hug. Of course, that's frog hugging the whale and nothing else and that's because we really love docker and that answers the question why I'm here speaking about the docker stuff and that's just because of very long history with it. We in JFrog do binary management or binary files for more than 10 years and when docker first come out with their layers and images and all this stuff, we of course were very interested and we actually implemented the first docker registry before it was docker registry, right? So we know at least this part of docker pretty well and this is why I'm speaking with you now. Another interesting aspect of what we do compared to what docker does, this is of course, Mr. Solomon Hikes, the creator of docker is speaking at the keynote at dockercon 2015, two and a half years already by now in San Francisco asking this question, who is using docker and nothing else? And the question is, okay, now you have a bunch of technology in your company, is there someone that the only technology they use in their company is docker? Anyone? No. Thousands of people in dockercon the same answer exactly and that's because the analogy of containers works here as well. No one, no company in the world are in business for shipping empty containers. It just doesn't make any sense. It's the same as here. We always have something in docker inside the container. We have Java, we have Node.js, we have .NET, we have applications that we ship in docker containers but there is other technology inside. And this technology need to be managed as well and when you manage everything on top of one universal platform or artifact repository like artifactory of course, here is your benefit. So let's now talk about patterns and 90 patterns in managing docker images. We are all software engineers and this is how we operate. When we have a new technology, first we look for a pattern that we already know and we try to apply it and sometimes we also think about are there any changes that we need to do? But usually we just try to apply it, right? And continuous integration delivery, those are all news. We do those pipelines for years, if not for decades. Next couple of slides are a very brief recap and I really hope I won't tell anything here to any of you. So this is a promotion pyramid and here what you can see is in builds that are passing through tests. At first we have a lot of builds and very fast tests, unit tests. Next we have less builds because some of them failed but longer tests integration. And this is how it goes. In some stage we have manual QA which is like long process but we have very little amount of builds to test and eventually we have couple of them that go to production and everything is great. Same picture, just put it on the side, is this. I really love this diagram. It's for a great book called Agile M by Michael Hutterman and it describes exactly the same. You have here binaries that are goal from one repository to another, that's your promotion, by passing what are called the quality gates. Quality gates are very important because they guarantee that only the right quality of builds reside in specific area in your pipeline which means in specific repository in your pipeline. And we promote here the binaries that we build from repository to repository all the way from development to production. Again, I really hope this is no news for you. Now let's talk about what's happening in Docker and why Docker forces you to do the wrong thing and stands in a way when you try to do the right thing. So this is why. Docker build, as you all know, is extremely powerful thing. Docker file as a build descriptor is more powerful than any build descriptor that we knew previously. We can do everything in there. We can type in any command and then we just run and we get an operational image that we can instantiate a Docker container out of it. That's absolutely awesome stuff. The problem is that once we have this great hammer in our hands, everything look like nails. So instead, that's almost the same picture. That's almost the same picture, but look at the difference. Instead of promoting binaries now, we trend to promote the build file. Promoting a text file, comparing to promoting binaries is very easy. All you need to do is re-tag it differently in your version control, or maybe just move it to another branch, or even change the name. It's very easy to deal with one source file. And then all you need to do is you promote this source file to another stage, from dev to integration, from integration to staging. And what do you do in each stage? You run Docker build, and it works. Everything's great, well, not so much. That's a home that was built and then felt. Because fast and cheap builds are not always the way to go. What do I mean by that? Why, where is the problem? The problem is that. Now, when I prepared this slide, I wanted to craft something ridiculously wrong. I said, I will do something that no reasonable man will do. And then I will explain that, you know, it's just to make my point. And then I went to the internet. And I didn't need to craft anything. In the show notes, there is a link to this file published on GitHub, extremely popular, tons of stars, tons of clones, and this stuff even is full of that. Now, you understand what the problem is, right? If I run this build file with Docker build more than once, chances are, I will end up with different image, right? So if I run this one in development, I will get one result. When I run it in integration, staging production, I might end up with different results. And that's exactly the root of the problem. This is why I am not sure. Do I have the same in production that I intended in development? Ah, I'm not sure. And now all of you guys are, there is a good number of people here that have their hand up when I asked about Docker in production. And this is where you should tell me, come on, Baruch. Okay, someone did something stupid over the internet. It's the internet's full of stupid people. We can actually do better. We can fix that, right? How do we fix that? How do we fix that? Version. Version, thank you very much. Let's fix it. Here we go. Let's start from the first statement, from from. And now we fixed it. Now it's Ubuntu 14.04, and we know exactly what it refers to. Do we? Why not? Thank you very much. Now, here is the thing. Ubuntu 14.04 was released when? 14.04. Apple 14, right? It was three years ago, and since then something very bad happened. Remember? What happened? How did it, thank you very much. How did it happen between 14.04 and now? And here is Canonical, who are great company sponsors of this event. Anyone from Canonical? Here? No? Okay, there are some here. Well, this is annoying. There are some of you, and they had a dilemma. Their dilemma is, from one side release versions, like 14.04, means it's an ironclad, immutable version. This is what release version means. It means that every time in the future, when we try to take this 14.04, we will get the same file. But, many of you guys come from the Ops background. Who comes from the Ops background? Okay, half of you, that's great. And what do you hate? You hate upgrading servers. Why you hate upgrading servers? Because things break. And because they screw with your uptime. Right? We don't like that. So, here is the dilemma. From one side, we cannot let people stay on the same version that was released on April 14. On the other side, there is no way to convince people to upgrade to the next Ubuntu version. So what Canonical did, and that's a perfectly good choice. This is the right thing to do. They incorporate severe security updates into existing version. Right? Any Windows users or maintainers here? Couple of hands. All right, you don't need to be shy, that's fine. But you definitely experienced Windows update that ended up in blue screen of death? Right? Exactly. Here we go, and I will raise my hand. Not lately, but there were times. What it means is, we can definitely can get an update. Incorporated into this, not immutable, but mutable version, and still end up with a different docker image when you run docker build multiple times on this docker file. Agreed? Can we fix that? How can we fix that? We will get to that, don't spoil my talk. Use a shot, thank you very much. I didn't pay this gentleman, but I should. Here we go. Now, this is great. This is ironclad. SHA2 refers to a set of bytes in the file. Every time I use that, I know for sure that I get the same version of Ubuntu. By the way, which version is that? I just sat on the keyboard when I did that. It's not a real SHA. Right? This is great, but completely useless. Okay, now what about those? Yes. You want more? All right, yeah. Well, let's say I have update got, but that's a good point. For the next version, I will do an update, but okay, let's say I do the update. No, I don't, I can fix it. How do I fix it? Easy, come on, guys, huh? Version numbers, of course. Of course, I can refer to Python with certain version and Node.js for certain version, and that will work. That's great. How do we know that we can put a versions there? Because we know how app get works. This is scale conference, of course we do. How many of you have Java background knowledge? How many of you know Maven? Maven? Okay, a couple of hands, good, good, good, good. Most of you don't. So how about now? What do you say? Is it reproducible? Every time I will run that, will it get the same results? You have no idea. The guys that know Java, what do you say? Will it produce the same results? Well, it depends on the form. Not only depends on the form, it depends on all the transitive forms that this form uses, which are, of course, outside of our control, and basically the answer is we have no idea. If you know Maven really well, like unfortunately I do, I know exactly how to mail the versions. Well, I know how to fill the build if there are some kind of non-final versions. But that's knowledge I don't wish anyone to gain because there is no normal way to gain it, but through extreme pain. So you don't know, and that's fine. How about that? Well, you got my point. There is no way we can escape that when we build multiple time the same build file. There is no way. Every time we run a Docker promotion, this is our nightmare. We cannot be sure that this container and this container are the same. And this is why we don't trust Docker. So what we should do? We should look at another pattern. Inputable server pattern? Anyone? Okay, not enough. So here you go, useful talk. You will learn immutable server pattern. Mr. Martin Fowler, the one and only, came with a great name for something we know how to do for many years. So if you remember when all of us were younger and the servers were actually hardware, if we needed to reconfigure a server, we actually logged into it like a physical machine. It was a box. Not many of you remember, but it was like a box. And we actually logged into it and made changes, right? And it made sense because this box, it has a value X thousands of dollars. So, you know, if we need to be reconfigured, it will be reconfigured. But those days are on gun. Now our servers are virtual, which means they don't cost us any money. And if we want to reconfigure a server, what do we do? We throw it away and create a new one. And that's because managing state is painful. There is absolutely no way to reconfigure a hundred servers in exactly the same way. You will end up screwing somewhere. But now our servers are code, infrastructure, right? Infrastructure is code. So we code this configuration and then we instantiate as many servers as we like. If the requirements changed and now we need to reconfigure, we change the code that configures the servers, new old ones, create new ones, and new ones are all the same in exactly the same way. This is the immutable server pattern. We never change the existing server. Instead, we kill it and create a new one. So this notion of immutability comes from a lot of different sides in computer science today, right? A lot of languages put a great emphasis on immutability, especially in functional ones. Immutability is good. And we should take this immutability to our CI CD pipeline as well, right? Instead of rebuilding this image on every given stage, although doger file kind of invites you to do it, you need to release this urge and instead promote a file that will, binary file that will build only once. So that's exactly the quality gates image from Michael's Wutterman book. You build the image once and then you promote your image through different quality gates that are right here from development to staging to production, right? And now the gates. I keep talking about the quality gates and you might say like, what is this problem with gates? Well, I do have a good reason. And those gates are very important because those gates give us the guarantee that a right image won't get to the wrong place. They guarantee that the QA won't waste their time testing what they shouldn't test or that will stage something that is not, shouldn't be staged. And of course, that we won't end up in production with images that are not ready to be in production, in terms of quality. This is why there are quality gates. So they are extremely important and picking a tool that knows how to provide you with the most in terms of quality gates is important. And this is where we go to the part when we now know what not to do. Let's talk about what we can do and why it's not that easy with Docker. So as I mentioned promotion, the hardest quality gates we can get are separate Docker registries, right? Because they have unique URLs. And if our provisioning tools only see registry that it allows to take the artifacts from, we are guaranteed that there won't be any glitches in terms of our quality gates. So what we actually want is implement a continuous integration pipeline when every stage is a different Docker registry. Because, you know, let's take a moment to speak about, I just saw just a couple of skeptical faces here. And this is good, you know, your Docker. So let's talk about what are the other options. We can use different tags. Can we set permissions on tags? No. We can use different repositories in Docker terminology. Docker repositories are directories inside your registry. Can we set permissions on repositories? We can. But how do you manage repositories? We create new set of repositories, development staging production for every image. And then for every image, we need to go through those repositories and set up new permissions. What are the changes that we won't forget to do that? Not strong enough. When we are talking about strong quality gates. So having isolated space, isolated registry, per stage of your pipeline is the hard guarantee. Did I convince you? Did I convince you? Good. All right. Now let's talk about why we cannot do that. Or it's not easy to do. You only remember that, right? I mean, it's fun to make fun of Bill Gates and Linux conference. What can be better than that? We have something similar in Docker. Our Trump implementation in Docker is Docker tag. Now, as you all know, Docker tag includes the registry host inside the name of the tag. It's kind of innovative decision. It has pros and cons. The biggest advantage of doing that is that it's very clear where this image belongs to. If we just see Ubuntu as a tag name, we know it's the official real deal. If you see here JFrog slash Ubuntu, you know that it's our flavor of Ubuntu or whatever reason we have our own image. Right? But you definitely know that that's not the official one. So it's very in your face where this comes from. And that's a good thing. The downside of having the registry host in this format, in this syntax, in the tag is the slash that they use as a separator between different types of, between different parts of tag. And the problem is that is, well, slash is already taken by the URL format and standard. So when you see that, you realize that how do you implement multiple registers per host? If you have host and then the tag name, how do you do that? So this is what we would do with any other technology inside your artifact repository. You will maintain more than one repository or registry. So here, that's one for development. That's one for QA, one for staging and one for production. But you cannot do, you cannot express that with the syntax of Docker tag. You cannot, because after the host, what need to come directly after the slash is the tag name. So what do you do? That's our trump up limitation. Well, there is a workaround. It's a heavy, it's not a very pretty one, but that's the only way you can do it. And interestingly enough, this standard of Docker tag being there for four years now since Docker first emerged, but the understanding of we need multiple registers per host hit Docker guys much later. And if now you will ask their support of how to do that, they will point you in exactly that direction that we are going to show now. Because there is no magic here. There is only one reasonable way to do it. That. No. All right, so let's talk about what we need to achieve. This is the tag. We have host port and then slash the name of the image. This is how the request gets out of the Docker demon to the registry. Right, so it's pretty much the same. We have the host port and then the protocol of Docker, current protocol of Docker, and then the name of the tag. What we want to get is from here to here, because that's the real URL that we want to achieve. And it has, of course, the host, the port, the context name, and then the virtual, the repository name that the repository that we want to get. And then eventually the tag name. How do we do that? It's right here. So not very intriguing question. We can do that with virtual hosts and virtual ports. There are a lot of tools that allow you to do that, and basically any HTTP server. That's an example of NGNX, and Apache HTTPD is another option, HAProxy, I think there are a bunch of others, but they all can do the same. They can listen to a request that comes to a virtual port or virtual host. That's example of virtual port, 5001, and then they can say whatever comes to here redirected to here. Right? And we use here this port as a unique identifier for this repository. So Docker staging, for example, will have another port, 5002, and we will have the same block for the new repository. And then for production 5003, et cetera, et cetera. Right? So, and that's the way to do it. It's not pretty, it demands babysitting. We do our best to make it easier for you. We have generator for all this stuff that actually updates NGNX automatically. So it's kind of behind the scenes. But there is another layer of what the hell just happened which is never nice. But I don't think there is another way around it. Now there is another problem. Since we have the host and the port inside the tag, the question is how we promote the Docker image from one repository to another? How we take it from localhost 5001 and get it to localhost 5002? We know that behind the scenes, it's the same server because all these virtual ports is just solving the problem of the tag. But from Docker's perspective, those are different hosts. And that means that we need to pull the image, retag it locally, and then push it back. Which is not fun. Especially considering the size of Docker images. They are huge, right? So what we actually need is some way to minimize all these dancing around different registries, but still keep the quality gates up. So our solution to that problem is what we call, and again, this is example of artifactory, obviously, but there are other tools that give you the same, maybe in different names, different terminology, but the idea is the same. Minimize the interfaces between your repository manager and Docker and pretend there is only one registry. When actually underneath, keep as many registries as you need to build continuous integration pipeline that respects quality gates. So this is what we did, right? We have a developer here that does Docker push to one URL. That does Docker push to one URL. This URL is what we call a virtual repository. There is no local storage of files there, and all the files are actually backed up by a local repository, first in our continuous integration pipeline. In our example, this is Docker Dev Local. Now, all those are fully pledged Docker registries in terms of the APIs as they expose, and each one of them use different virtual host or different port, and tools that are in charge of doing the testing can pull the images directly from each endeavor stage by using the virtual host and port. And of course, they only see their registry. This is our quality gates. They don't know about the existing of any other. But when eventually this image is promoted all the way to production, it can be again retrieved by using this public virtual repository, which looks like the only registry for the rest of the world. Does it make sense? I hope it does. Now, just an example of building one of those, and how you implement the pipeline on top of what we spoke about. So let's talk about a typical container or typical image, a typical stack, and you will do the adjustments for your stack, of course, appropriately. So this is our layers. We will use here CentOS as example, and our framework that we run, our code is Java EE, just to show you that even in Java EE, we can do good job, right? So we have JDK8 and JBoss application server, Wildfly, whatever. And then we have some raw file on top of it that actually runs in this application server. Those are the layers of our Docker image. Now, we can separate it to two pieces based on how frequently they change. We have those two, which are the framework. They change very rarely, only when we have a new version of CentOS that we want to upgrade. As I mentioned, we don't, if we have a new version of Java that we want to upgrade or a new version of application server. And we have our raw file, which is constantly in motion because this is what developers write. So those are the two, and we need two different builds for them. One will be the framework build and the other will be the application build. Framework build used a verified base image, the one that we played with, and we are sure that it's good, some version of operating system that we can trust or even a base image that includes Java and maybe includes this application server. And then we can add additional system dependencies from the source that we trust. And it should be in house repository that gives you those dependencies. So your up-get server source or your RPM source or whatever. Now, for example, all those can be implemented in artifactory, of course. Stuff like JDK, like Tomcat, you install in this image. And the most important part is you own it and that's back to the answer that the gentleman gave me. You own the base image. Once you own the base image, the problem is how can I guarantee that my image won't change under my hands? Of course, go away. This is a minimal framework build Docker file. It does absolutely nothing. It just nails down the version of CentOS and guarantees that until I build this again, I don't have a base image that I can trust and it won't change without my permission or without my action for this regard. Application build. In our example, it's a Java build that we use our framework as our base and then we run this Java build and we end up with a world file. And again, it can be anything, right? It can be Python build, it can be a node build. It can be whatever your software is and then you end up with the file that you need to end to your base, to add to your framework, to deploy to your framework. Right, so this is our example and here we use our own trusted base and we add one file. We take the world file from our factory and we deploy it to wherever place, oh, this is annoying, to wherever place the deployments go in your framework. And here when you look at this file and you think about all what I told you through these 45 minutes, you are like, well, everything is wrong about this Docker file. First, why the hell do you use a release repository for your file? This is an interesting question that can be answered differently depending on your process. And there are two types of processes. Sometimes you check your wherever goes in the container always with the container. And then you have matching steps. When your application is in dev, your container will be in dev. When your application is staging, your container will be staging, et cetera, et cetera. But sometimes you can run a continuous integration pipeline only for the contents of your Docker container without actually bringing the container all along. So if you have a continuous integration pipeline that brings this world file separately from development to release, you can take it from release. It will save you additional testing. And now you should go like, well, you just told us not to rely on the latest version and this is what you do. The difference is, this is my latest version. It is guaranteed to be good because it is in release repository. And we just spoke about how strong our quality gates are and everything that is in release repository is actually good. So I can take the latest one with confidence. And then you go like, okay, but why the hell do you rely on latest in your base? And the answer is for the same reason. This is verified base image. I built it. I want the latest because I know that it's good. There won't be any magic in the world that will make the latest framework bad. It might be broken. That can happen. And we are going to talk about how we can we find this problem very soon, but it's definitely not malicious. It comes from a verified source and I can definitely rely on the latest framework. And now let's talk about testing. Who heard about sandwich testing? This is a very useful talk. Here you learn something new every slide. Sandwich testing is a flavor of integration testing when you actually do bottom up and top bottom testing in the same time. This applies perfectly to our two builds. This is our two pipelines. This is the framework and this is application. And when you want to test your application, you take the latest good known framework to run your application on. That guarantees that you only test the application without testing the framework. And the other way around, when you want to test your new framework because you updated your Java version or your base image, you test it against the latest known good application. And if it works, that means that you didn't break anything. What can be the problem? That. When actually the changes are mutually breaking and you did some change that breaks in application that breaks it and some change in framework that breaks it as well. And what you can do to prevent that is setting up a good triggering system for your continuous integration. For example, you make changes separately for application and for framework. And by the time the CI server kicks in, it will only run on what's changed. So if you change the application, you will end up with good known latest application build before you change the framework. And if you change the framework, the framework build will end up and test it against the latest known application. So as long as you don't have non-backwards compatible, dependent breaking changes in both frameworks, you're good. And you shouldn't do that anyhow. So this is all. And what I want to end up is the reason why we actually need that. We spoke about trust. We spoke about this feeling that can I be sure that what I have in production is what I have in development? And how can we live with that? Except of raising the level of stress, it also slows us down. Because when we are not sure, we will add more tests. Those tests will probably be ad hoc, will probably be manual, will probably take a lot of time. When we build this trust, it allows us to go faster. It allows us to release faster. It allows us to do our work better because you can go faster when you are not afraid. Now it's time for Q&A. We have good five minutes at least. I'm at J Baloch on Twitter. That's good time to follow me. Scale 15x, that's the hashtag. Use that. Show notes, video by tomorrow or by today even. Slides, links, feedback, and raffle should come early. You can double or triple your chances for coming early. Thank you very much. Yeah, questions. You should have tons of questions by now. Like for example, what the hell did I just see? That's a good question. I'm not sure I can answer it, but it's a good question. Here you go. That promotion model that you showed? Yep. Is that only in Artifactory? No, no. So generally what we spoke today is not some kind of unique feature in Artifactory. If you want to do it, you can just set up multiple Docker trusted registries in your host and do that. It will be more painful, but you can do it. There are other alternatives, competitors of ours, that provide kind of similar functionality. I can now explain you why we do it better, but the more appropriate place for that is tomorrow at the booth. So for the promotion, why don't we use the Docker manifest retail scheme material, multiple tagging of the image to do the promotion? Oh, that's a great question. Now, Docker have a very neat feature which is different layers of metadata. One of the metadata, and then there is not the only one, is the ability to add properties to Docker image. And then we can say, you know what, screw that, I don't need that. All I need to do is annotate those images with their current state, right? I can say this image now is in dev, this image now is in staging, and this image now in the production, which we can do, but as you probably remember, I am very much obsessed with security gates and their quality. When you cannot add permissions on this metadata, there is no way to guarantee that a test that should only take image which is in development, which is in staging, won't take an image which is in development. The only way to strongly guarantee it is completely different registry as long as Docker concerned. At least for now, maybe it will change. Maybe there will be some other models, but now this is the strongest way to guarantee you want and test what you shouldn't test, but more importantly, want to end up with production with something that shouldn't be in production. And that's scary. You say there's no way of actually guaranteeing. Yeah. You could, for example, restrict what you're allowed, use some other layer to restrict, say, pushing to the Docker repo or something like that, so your only your CI host is actually able to push and tag to your Docker repository. Yes, you can add permissions on registries, on repositories, which are directories in Docker registry, and you can do it on the level of Docker itself. You can do it on level of NGX, even, right, or Apache. But the problem is that you create new set of repositories for every tag, sorry, for every image you work with. Repositories in Docker are tied to an image. Yes, then, yes, what you can do, yeah. All right, so what you can do is, for example, you know what you say, screw it. I will use one set of repositories. I will have repository for them, for staging, for production, and then all my images will go into this one repository. Right? That will work, but that's the opposite of Docker way of this concept of repository, which means you can have multiple repositories per image. I don't say there is no other way to do it. There is, it's just one, you know, you will fight Docker to achieve that even more. Yeah. Sure. You mentioned pinning the base image. You did an example with CentOS 7. What guarantees that the CentOS 7 image is not updated in the public repo? What guarantees that this CentOS 7 not updated in the public repo is, of course, nothing. But what guarantees that I won't get this update is I don't run this Docker build until I wanna run it. Thank you very much. Okay, thank you very much, guys. Thank you. Thank you for every other support aspect that we have. Yeah. So, yeah, see what I mean, comment. That's not you. Oh, okay. You just have to live with it, if it's on your body. But I'm gonna go ahead and introduce you and I'll be back in a Q&A. Would you want me to use that instead? It would really help if I turned this on, wouldn't it? Good afternoon, everybody. Our next speaker is Lee Calcote. Calcote? Yeah, that's right. Lee Calcote of SolarWinds, who's going to talk to us about container networking. So, welcome. Very good. Thank you, Josh. Excellent. It's been a while since I've been in a room of this many like-minded folks. I can see it. I've been to some other conferences and there's a variety of folks there, but I'm getting a good vibe today other than the microphone. So, yeah. So, a little bit of a play on words about the over-under, the overlay underlay on container networking. So, we're going to dig in today. Networking is a complicated thing. People make entire careers out of it. Actually, that's kind of an interesting question, maybe, is how many of you guys and gals consider yourselves to be sort of all of the network, more of a network engineer, network systems administrator, network-oriented folks? Nice. Very good. Boy, you guys have just learned to tune that out, haven't you? Okay, I'll get there. Quick note, we will no doubt get through all the slides today, but to the extent that I skip over something or there's something you guys want to look back up, take note of the URL, the slides are there. A little bit about me. Feel free to harass me online if you don't do it today or tomorrow the next day. Reach out, poke me. I like to chat, nerd shop whenever I get a chance. I'm into a couple of things. By the way, I'm coming to you guys from Austin, Texas as of this morning. It was an early rise for me. Word of advice. I've just got some issue with time zone calculation and then the 24 hour time that's listed on the schedule. I thought I was speaking about two hours from now, so I'm nice and alert right now. Anyway, I make myself a nuisance in a couple of different areas. I'm a cloud native ambassador with the CNCF, so I spend a lot of time hanging out with in that general area. I'm an advisor to container security startup, Twistlock. Anybody know of Twistlock? They're kind of, yeah, very good. Excellent, yes, I'm sure you do, Mr. J, for God's sake, yes. Fanta, I do some writing for various places. I'm an O'Reilly ambassador. I organize a couple of meetups at Austin. So anybody coming out to DockerCon for Austin? Anybody come out to OzCon for Austin? There's a ton of great tech conferences coming there, so that and barbecue, so. We don't need to talk about my favorite subject, me anymore, but like I said, reach out. I organize a couple of local meetups, the Docker Austin one, another one that's affiliated with the CNCF, so it's on microservices and containers, and so we just cram the two buzzwords into one title. Currently, in sort of full-time, I'm with SolarWinds. I'm gonna do another raise of hands. Who's familiar with SolarWinds software? All right, so I'll be honest and say that's more hands than I've seen the last time I asked that question. That's good, and since you guys are here, you're obviously interested in containers, doing container-y things, no doubt. How many of you guys would like to see SolarWinds do, brings the monitoring software for containers? Would that be helpful to you, or is that, are you guys sick and tired of SolarWinds software? You're looking to get off of it and run away fast? Yes, everyone, yep, okay. Nice, okay, well, like I said, so this doesn't, this goes without me saying, but container networking, well, networking in general is complicated, right? And I think I've made a friend, Justin, if you're out there, I don't know if, if container networking, if everyone agrees that container networking is a thing, I think most of us can agree that virtual networking is a thing, right? When VMs came around, and we used to just have things that we could touch and put our hands on, physical networks with cables, and we got into some VLANs, we brought in VMs, and we got virtual switches, and to use, you know, VMware parlance, what distributed V-switches, and SSX came around, and things like VXLAN got, you know, established and more popular. Anyway, there's, there's sort of virtual networking as a thing, and if it's a thing, then I'll contend that it's always container networking. Like, hey, it works a little bit different than the container land, and that's actually one of the things I wanna try to point out today is that for those of you who are familiar with physical networking, and I think you're gonna find yourself maybe leaning into different types of your, different types of choices you can make in terms of how you set up your container networks. So, we'll see. I think the other aspect of the fact that container networking is complicated is that things have only gotten worse for us in this sense, in the sense that physical networking is tough with just, you know, routing and spanning tree and layer two stuff, and just, like I said, people have got expert certifications in this, but it's gotten worse than that. We used to just have, you know, one or two physical network interfaces on a server, right, or maybe a couple of virtual sub-interfaces, and we might bond them together, or team them, or depending upon whose server you bought, you'd use a different, you know, vernacular for that. But then, you know, then we loaded up VMs on there. We got to, what, you know, 10, 15, 20 VMs on a host, and each of those VMs might have had an interface or a couple of interfaces, and so, what, you know, 15 or 20 times two, we're starting to get up into 40-ish interfaces per host, and this is just like, just interfaces, right? So, if we keep going on that count, and we say, well, now a host, that same host, might be able to hold 50, maybe 100 containers, just depends on, you know, how large your containers are. If you consider that they've got an interface, or maybe a couple of interfaces per container, if you take 100 containers times two, anyway, we're just getting into, it's just getting worse, right? And then we bring in microservices, and we make the network front and center to how an application functions. As if that wasn't already true with, you know, enterprise-architected applications, the, you know, your classic three-tier apps, like, hey, networking was fairly front and center to how the health of those applications, but as you move into microservices, and you do things in a distributed way, you guys, some of you've heard of CAP Theorem, and anyway, the point being is like, a lot of times for a given transaction, for a given function behavior of your application to execute, that, you know, microservices land in a distributed systems land that doesn't necessarily always happen on a singular host. A lot of times that's happening, you know, at execution across hosts. Anyway, point being is, not only has it gotten more complicated, but network has got a little more front and center. So for those of you that raised your hands earlier, I sort of identifying as being a network nerd, and I use that in the best way. Hey, you know, your job is secure. In the case is, as people move into this area of networking and using containers, there's some preset expectations. People already have an understanding of how networking works, how virtual networking works. They're probably gonna go in expecting the same amount of, particularly if they're gonna do things in production, they're certainly gonna want these to be, the networking there to be reliable, to be performant. They're gonna have a lot of the same expectations that they've had in the past. And they're gonna want that experience, that performance, that reliability to be on par with what they've been using today. Moreover, they're probably gonna inch up, or begin to expect a little bit more as they move into this more, you know, SDD, this more software-defined networking, software-defined data centerware. Now you can just express things as code and you define your infrastructure as such. You begin to get into, you know, another excellent, I don't know if this is necessarily a buzzword, but if it isn't yet, I suspect it'll hit you guys at some point, what is intent-based networking? It's, you know, in a nutshell, it's a lot like being declarative with the network. So to the extent that you've seen a Docker compose file or Kubernetes pod specification and you're declaring your infrastructure state, that that type of a, you know, that essentially is what intent-based networking is. There's some declarativeness to it. There's some reconciliation loop type of a concept to what intent-based networking is. It's the notion that you can go and define that in code. Let SDN controllers go and go and affect and implement the network as you've described it. So anyway, point being is not only do people expect to network to continue to function as it has in the past, they're gonna have the same expectations, they're likely to expect more. They're gonna want the ability to define how your network works in a developer-friendly way, in an application-oriented way. However you guys have heard of mode two and mode one, kind of Gartner's bimodal IT definition. Okay, good. So this is not, that goes back to I think what I said earlier which is there's some like-minded folks in here. This is not the Gartner crowd. So we won't go there. This probably used the crowd hopefully that where this image resonates with you, this is probably my favorite image of the deck. So we'll just share that. There are a couple of container network specifications. We're gonna walk through the, really the two that are there. And I just wanna disclaim this by saying it's some very interesting kind of thought-provoking things in here, some models that we can walk through and then as it goes to application, unless you're actually implementing a network driver as a plug-in to one of the container orchestrators, it's probably not necessarily a key that you digest all of this but I do think it's pertinent background for understanding really the, how container networking has been shaped and it's kind of between these two models. The one on the left-hand side is the container network model at C&M. It's really not necessarily a specification as much as it is an object model for how it is that Docker, the project has defined how networking functions within the Docker game and within the Docker environment. There's a singular canonical implementation of this model. It's called libnetwork. There are various vendors of written plugins for or vendors or projects have written plugins to libnetwork. We may touch on some of those today. The other specification, this is probably really is a specification is the container network interface. So specification proposed by CoreOS. It's been, the specification's been implemented a few times over and there's a list up there. Not only has that spec been implemented a few times over, but so have some plugins been written, that plugin to C&I. So the container network model, Docker's libnetwork. What does that model look like? They're both relatively simple in terms of how you diagram them out. In essence, what this describes is the hexagram, I think, if I've got my geometry right, is your Docker daemon, your runtime. The libnetwork or this container network model is really just an API specification that sits between the Docker daemon, the Docker engine, and network drivers. Out of the box, Docker daemon comes with three network drivers. That said, actually, I think that that's true as of Docker 111. I think with Docker 113, I believe it was, there's a fourth network driver. We're gonna talk about that one. Anyway, the point being is you get some, you get various types of container networking out of the box. Those are fully supported by Docker. Of the vendors that I've listed out before that have written those plugins, those are the remote drivers that plug into this. If we look at the actual topology of the container network model, we're seeing a few things. We're seeing that the model allows for more than one container to be connected to the same network. I think that's probably a fairly obvious thing already. One network driver, it really is responsible for managing a singular network, so one network can't be managed by multiple drivers. Multiple drivers can be running concurrently, so you could be running bridge networking on one network controlled by one driver. You could be running host networking on a different network controlled by a different network driver, so there's separation of concerns there. There's a lot of, some of you might have heard the term micro segmentation with respect to network, so there's some fairly granular ways in which you can interconnect these containers in this model. One of the things maybe to note is that even though the container in the middle is conceptually, I should say, the container on one of the ends, even though it's conceptually interconnected all the way to the container on the other side, by default it can't necessarily use that middle container as a bridge, so there is that micro segmentation. The CNI, the container network interface, the one that's more in Kubernetes land and was proposed by CoreOS, this is what that looks like. It is simple as well and it has a lot of the same characteristics that we just saw for CNM. So the flow of how it is that when you go to provision a container and it spins up and that container engine needs to assign it in a network and needs to assign it an IP address, this is the process that happens, so the container engine allocates, puts that container into a network namespace, it ends up passing along the config information to the network driver. The network driver then picks up that request, ends up attaching the container to a network as well as then assigning an IP address. So the network driver, the network plugin, not only is it responsible for the network and the services that are there but also, I guess I should say, that includes services like IPAM, IP address manager. This is just a quick JSON schema for what a network looks like as you describe it in CNI. So really straightforward. Both of these models are similar in that they're both driver based so we saw the plugins along the bottom there. That democratizes some of the selection of what types of container network you wanna run. Like I said, you can run multiple concurrently. Containers, like I said before, can join one or more networks. And the nice thing about the way that these models are written is that some of the application logic or the business logic as to how those networks are managed, that's dedicated to the network driver so vendors can come in and differentiate a little bit. These two are different in that. CNM really only supports the Docker daemon, the Docker engine. CNI supports the engines that I'd listed up there before, Docker, Rocket, Kerma. So I'd said that CNI is a little more of a specification because it's been adopted beyond its creator. All right, so good. So let's get into, I guess, some of the basics of how these different types of networks function. It's unfortunate that the screen is cut off there. But I wanna walk you through each of these types. I wanna start off with none and see if I can't give you guys quick examples as well. Okay, so as you go to, and we're gonna use Docker, the Docker CLI and the Docker engine to walk through this. So the, as you go to spin up a container, so a little off-screen, okay. Okay, so we're talking about what's the behavior when you go to spin up a container and you tell it that it has a network type of none. Any guesses? Let me see if I would get this correct, I'll go around. Thank you. Turns out you actually do get some networking. You get local host. You just don't get external connectivity. So the container when you spin it up, it gets assigned to a network that doesn't have a driver. Or rather, I'm sorry, it does have a driver. The driver is called none or null. It does get a full network stack. It just gets loopback connectivity. Could be potentially useful for maybe batch jobs that don't need connectivity to the network. Maybe it's just writing things to the local file system when it's done. You can spin up a container on a network like this that isn't connected. And while it's still running, come back to it and take it and attach it to other networks. So just because it starts out with nothing doesn't mean you can't come back and reassign it to networks. So speaking of that, may be good to just do a quick review. So in terms of just how Docker works, I'd said before that out of the box you get three types of networks. Bridge, host, and null. Null is what we just looked at. Okay. So another type of container networking is links. And links were often used with the ambassador pattern. Really, this is a type of container tech networking that has come and gone. This was, I think if Jerome were here, I think he would agree this is probably a poor man's attempt at facilitating service discovery before there was kind of full-born multi-host networking inside of Docker. And so as such, we'll skip past it because it's just not necessarily that interesting. I mean, it's interesting to note that even this space has evolved just in the short time that it's been around. So another type of networking is bridge networking. And this is probably the one that most of you are familiar with. So if you go to spin up a Docker container and you don't specify the type of networking, this is the default network that you'll get. A lot of you have seen maybe Docker zero. So if we go out and just take a, do a quick run of that. Okay. So we see in this case, we do get the loop back zero. We also get E zero. It's attached to a Linux bridge, the bridge that we see out there. How many is it? How many of you is this familiar? This Docker zero bridge, the Linux bridge? So what's happening here is netting, right? And the use of IP tables. There's some, even though there's some cost to how Linux bridges work, there's some cost to the translation of that. We're gonna see if I can't show you guys some of that. Another type of container networking is container mapped. So this is, this type of networking is used very frequently in, it's essentially the default type of networking that gets used within Rocket and within Kubernetes pods. In Docker land, this doesn't necessarily get used that much. But let's take a look at some of its behaviors. So the, I guess I should explain what it is. It's container mapped is when you've got a container up and running and it has networking, maybe you're wanting to spin up a second container, have it attached to the same network name space as that first container. This is what you can use container map to do. So maybe you're having an issue with this container that's running. You're wanting to diagnose what that issue is. Maybe it doesn't have the right tools. So you spin up a separate container, drop it into that first container's network name space, and they both share the same IP address, the same MAC address. So let's see if we can show that. So if we start up a container, or just maybe this is what you were saying. Hey, hey, hey, hey, hey, hey, hey, hey, hey, hey, hey. Yes. Good, well I'm glad you guys were seeing all of that before, so yes. Okay, so we just spun up a container. Take a look, we're gonna have some fun with the formatting here I suspect. Here's our container. If we just take a look at what's in this container, so we didn't explicitly tell it what type of network to use, so it chose the default, the fridge networking. If we'd spin up another container, and in this case tell it to be container mapped or to use that type of networking, the syntax looks like this. So we're just saying the network type is, use the network of another container, and here's the ID of that other container. So normally as we would spin up additional containers on that same bridge, we would get an E0 and it'd get 172, 17, 0, 3, 0, 4, 0, 5 as we just kept spinning up containers on that network. If this container is gonna drop in to the network namespace of that first one, we should see that it really has the same MAC address, I'm sorry, same IP address and even same MAC address. So I think that's what we're seeing. Kind of interesting. So the two are actually sharing the same network namespace, kind of an interesting thing. Okay, host, there's host networking. So that's one of the three network types that supported out of the box with Docker Damage. And there's some pros and cons to using this. This essentially says that the container that you spin up is dropped into your node, your host's physical network namespace and it begins to share that same network namespace. So for some of the costs that you would have with NAT or IP tables, you'll avoid some of that because that container will be using the same IP address, same MAC address as your host. You'll get some better performance for, I think for many of us that either scratch our heads around network or are accustomed to physical networking, you may find this type of networking a lot easier. There are some question marks around whether or not it's more or less secure. Another thing you kind of suffer from here is potential port conflicts with the host. Since you're using the same IP address, you'll be sharing the same port range with the host. So let's take another quick look. Docker run. This time it's network equals host, alpine. And so what's happened here, so okay. Let me describe this a little bit. What's happened here is the, okay. So I'm running, I have a Mac. I'm running Docker for Mac. Docker for Mac uses a virtual machine running an XIV. The name of that virtual machine is called MOBI. That's just its host name. So if I were to go into that VM and take a look at that VM's networking. So by the way, so in order to get into that VM, what I did here was just spin up a privileged container and now that that privileged container has access to that VM. So if we type host name, we're actually in that VM and that MOBI VM. We take a look at our container hosts networking. We're seeing Docker zero, E zero, loop back zero. As we look at that container that we spun up and that we told to use host based networking, it in fact is sharing the same Docker zero, E zero, back zero. So kind of interesting. And like I said, can resonate with network engineers a little bit more. It's the default networking style used in Mesos at least last time I checked. Can avoid some of the complexities and kind of the costs of convenience that you might have with an overlay. So overlays are really convenient, pretty popular, particularly with developers who may have gotten a couple of hosts from their IT department or a couple of hosts out in a public cloud and now they need for those two hosts to talk to one another. One of the ways for those, well I should say they need for the containers running on those hosts to talk to one another. And a convenient way for them to do that and maybe do some shadow IT without actually having to go to IT and ask IT to make sure that these two VMs are in communication and give me another network so that I can use that network. Instead they might just set up an overlay between the two hosts. So very useful there. Very popular, a lot of the vendor plugins use overlays. They do oftentimes, are they almost always require a key value store? Whether or not that key value store is built into the solution or whether or not you've got to bring your own key value store has been changing over time. So over time what's happened is, it's bad, isn't it? There we are. Yes, here I am. All right. Okay. Boy I thought it was because maybe I didn't know what I was talking about. Okay, we're gonna have a much different conversation. It's gonna be much more personal. We're gonna, okay, good. Fantastic, so the train of thought that I was on was this notion of some of the way in which networking is working is getting easier with multi-host networking. As you go to track, oh boy, neighbor tables and names of hosts across different hosts, like hey, you need a centralized key value store to be able to keep track of that stuff. And it wasn't too long ago, a year or more ago, that essentially all of these offerings required a key value store, you to bring your own key value store. Some of these offerings now build it in. And many of you are familiar with Docker 112, the inclusion of Swarmkit inside of the Docker daemon itself inside of the engine. Part of that inclusion and part of that massive leap forward that they took was not just the built-in orchestration, but what was a lot of built-in, a lot of advancements on their networking as well. They have leveraged the RAF implementation from Ed CD and the GoMDB library from HashiCorp as well as the SERF project from HashiCorp, a gossip protocol. Just kind of a neat tool. But anyway, some of the other, essentially all of the other offerings out there do require a key value store. It's just how easy is it to spin up? Do they bring it or do you need to bring it? All right, so I just talked about overlays and the notion that they're convenient, the notion that developers can set them up. They don't necessarily have to go to their IT to ask them to set up a new network that the developer can do it themselves. But there's also a cost to that overlay, just in terms of adding on packets, tearing off packets. There's a kind of a category of container networking referred to as underlays. This is much more, you know, I think folks that are of the physical networking world will probably just resonate with those a lot better because that's in essence how these work. And by the way, these are not necessarily cloud-friendly either, so these are probably much more oriented towards, you know, running things in your own data center. The first on the list, and actually I'm recognizing that we probably blew past the time here, so I'm just gonna skip past this, like I said, the slides are available. This is Rocket's default networking, and it's fairly simple, it's point-to-point. MacVlan is, I mentioned before, like the latest addition to the Docker networking suite, sort of what were the types of networking that you get out of the box. There's actually two types of upcoming networking, of network types, I guess we just said it. One is MacVlan, the other one is IPVlan. They both end up, they're both physical networking oriented. They both end up utilizing, essentially, your host's network interface. Both end up creating sub-interfaces off that host network interface. The difference between the two is whether or not, and so as such, as we just saw with host-based networking a second ago, whenever you deploy a container onto like a MacVlan network, or an IPVlan network, you'll, that container will use the same network namespace as the host. They'll get the same IP address, and really the difference between MacVlan and IPVlan is whether or not they use the same Mac address. So with MacVlan, and I would try to demo this if I didn't have to hold the mic, maybe, but the, so MacVlan assigns a unique IP address and a unique Mac address for each of those containers that you put into that network. The issue can become in a physical network environment, a lot of times you'll have security configuration on your switches that will shut down a port if it sees too many Mac addresses pop up on that port. It's just kind of a common security best practice. We'd mentioned before about how many new types of container network, or how many new, or how many container interfaces you could have on a host, way more than a regular switch would necessarily expect. Anyway, the difference being with IPVlan, it doesn't use, it doesn't assign a unique Mac address to each individual container that you spin up on the network. It reuses the same Mac address. Can be good and bad. It's like, yeah, so there's some differences, some pros and cons of using IPVlan, MacVlan. IPVlan works in either Layer 2 or Layer 3 mode. In the Layer 3 mode, it doesn't allow any broadcast traffic in, but with Layer 3, you might head towards that if you're looking towards massive scale. And actually speaking of Layer 3 and massive scale, this is where you might also get into a project like Calico, which does direct routing, it uses Layer 3, it uses BGP. It gives you more granular control over the filtering, the firewalling of where traffic goes, kind of the isolating of that network traffic. It's also just kind of resonates with network engineers that have been doing Layer 3 routing. Okay, this last one, this last one actually, I'm not gonna describe. Who's heard of fan networking before from Canonical? Fantastic. Has anyone used it just out in the cloud, kind of for its most common use case? The use case here being you've got a VM that you've provisioned in a public cloud. It's got a single IP address that's been assigned to it, but in fact, you need a lot more of those IP addresses and you want those exposed. Fan networking is a clever way of doing some subnetting in order to essentially assign yourself a whole lot more IP addresses on the host and have those netted out or subnetted out. Anyway, it's an interesting one. I haven't seen it necessarily broadly adopted. Okay, so for the most part, what we kind of walked through today were just the different types of network drivers, the different types of the way that they behave, how it is that whether you get an IP address or a MAC address that's the same as your host or what network names they basically go into, but there's a whole genre of a class of other capabilities and services here as well. So we were looking at IPv4 addresses. IPv6 is still kind of an issue for us. As many of the top clouds don't support IPv6, has this changed in the time that I've written these slides? Has any, I don't think so. Okay, yeah, so none of the public clouds have announced IPv6 support just yet then. So we were saying that as we were spinning up new containers, at least for the Docker engine, that we were getting new IP addresses assigned, right? And those were, so IPAM is kind of in the box there for the Docker engine. IPAM and how that works is pluggable in Docker engine. If you plug in a different vendor's network driver, you may get, you'll get different behavior actually. One of the drivers is from info blocks. So this is where you shouldn't try to edit your slides on the plane. I guess what I was just gonna call out here is that the other thing that we've been talking about mostly is how it is that these different types of, different types of container networks function in their most basic form. And I said that Docker 112 took a massive leap forward. They did, and unfortunately, you know, I'm gonna have to try to describe that to you. Some of that's been around the leveraging of IPvS. So it had been about 10 years since I'd used IPvS. People familiar with that just in the kernel build balancing, highly efficient. Pretty cool things that the Docker networking team has done here around creating a routing mesh. It was in 112 that the Docker service construct came out and that service lets you essentially assign a virtual IP address to that service. The service may represent multiple, really kind of begins to represent your application and your application being comprised of multiple containers. Some pretty neat things that they've done with mesh routing. And so that's the ability to spin up a new Docker service, a new application offering. Maybe you're exposing a given port. And for your swarm kit or your swarm mode environment, for you to deploy a Docker service, it to come up and essentially for an external client to make a request of that application, it can send in that request to any node in the cluster. That client doesn't necessarily have to know which node to send it to. Because of the way that mesh routing works, the client can send it to any one of those nodes in the cluster and each of those nodes will reroute that traffic to the appropriate node where that container is running. Okay, another kind of high strain slide, but just some research around the various container networking solutions that are out there. Most of these, I think actually, well, all of these except for NSX are open source. Again, slide's spilling off there, but very popular, the use of VXLAN and overlays. If you're interested in more research there, these are links, so pull them back up. As we wind down here, I thought I would go back to the original question that I'd ask of the group. That was whether or not you'd use SolarWind software and if it would be helpful if SolarWinds kind of did some things around container networking. Well, we're beginning to, certainly giving some thought toward it and to learn more, go to the GitHub link there. I was gonna try to show you guys a demo, a preview of what that looks like. A tool, really, it's a free tool that gives visibility into your multi-host network. The containers that are running, how they're interconnected, who's talking to who, how much bandwidth they're using, what's the direction of some of that traffic. Right now, it's a very nascent capability. It's an early free tool. The corporate VPN that I'm running to connect back to this thing is not functioning right now, so this is kind of, this is not, this is disconnected from the environment, but this environment was a Kubernetes, three node Kubernetes cluster running VXLAN as its network overlay. Anyway, a horrific demo, mostly because we're disconnected. The point of the tool is really to help give you visibility into what's happening on your network. It's really an educational tool. It's intended to help you make some kind of day zero choices to just better understand what's happening in your growing environment. And then to even let you do things like prove out what I was saying before about there being a cost to the convenience of an overlay, so the tool will ultimately let you select a given container, select another container, and do a bandwidth throughput test. So if you're familiar with IPERF, this is kind of basically the same concept. IPERF isn't being used here. This is on the back end is written in Go and is using a Go library to just generate traffic and slam as much network traffic as it can through there to just perform a test. So what ends up happening is you can run this in multiple environments and this will facilitate really just flexing that network driver or that type of network, the ones that we've stepped through before. So you can really begin to see the difference in performance. So unfortunately the tool is separated from the back end but I did get a screenshot of kind of what one of those results look like. And I guess this is a different environment. I believe this one is a two node Docker swarm environment. Anyway, just, yeah, you know, it's a horrifically generic name. It's container network performance tool. It's extremely descriptive of what it does. It is, the link to it is just GitHub under SolarWinds. There's a container section. So yeah, bad demo. I don't know that the tool isn't clearly not at a V1 but I did want to take the opportunity to expose it to you guys because ultimately it would be a free thing and hopefully you'd be able to take and learn and maybe self-instruct better than I have for you today. So if you do try it out, let me know. Let me know too if you're running SolarWinds software and you've got use cases for us. Always open to feedback. So questions. I know that all of what I said was crystal clear. Both, half of that's probably just my issue with talking. Yes sir. Yeah, great question. So the question is, hey, where does the 172.17.20, I think it was .20, or .00, you know, address space come from and that's a great question. So in this case, we were walking through the how Docker networking works. We were taking a look at, right then, we were taking a look at how does bridge networking, the default network type inside of the Docker daemon work and whenever you, whenever you spin up a new container on a bridge network in Docker, it will have assigned a subnet that it, so it has built in IPAM and it will have assigned a subnet to that network and it by default starts with 172.Jerome17 or is it 16? 17 and then just kind of keeps going from there. So you can create new networks as well, new Docker network, new Docker networks of type bridge and each time you create one, it will assign a new subnet and it just has its own internal IPAM so it'll go 172.18, 172.19 and it'll have a, it'll track the IP addresses for that subnet. Every time that you spin up a new container and assign it to that network, it'll just dole out a new IP. Oh yeah, sorry. So the second part of your question was, hey, where does that come from and can I change it? So it comes from RFC1918 if I recollect correctly which is private, yeah, which is extremely helpful. It comes from, it's a private network. Oh yeah, yeah. So there's, I think, I don't know if it's an IEEE standard that says, hey, that particular network space, 172.16 slash 22, help me out, slash 16, I guess. Anyway, there's a private, sort of globally worldwide. Oh, okay, yeah, sorry. You know, I don't know. There may be others in the audience that know, sorry. Oh, okay. Yep, so it's part of the, so I'm slaughtering the question. It's part of the question. Hey, I don't want to use bridge networking and I don't want to use, no, okay. It's, hey, I want to use bridge networking but I don't want to use that address range. I want to use a different address range. Fantastic question. It is configurable. I forget specifically where it is, that was the problem. So yeah, there. Fifth time's a charm. Yes, sir. Ooh, say the last part, the first part was if you're using shared networking, oh, I see. Maybe I'll take the example of the, like the container maps networking that we looked at where one container joined the network namespace of the first one. They literally are sharing the same, yes, that second container does suffer as the first one does. I think two use cases come to mind for me. One is that, and I don't know how often it's necessarily used in Docker land, but one is, hey, you've got a container running. It's performing a particular function. Turns out you're having an issue with it. You want to diagnose it before you kill it off, but it doesn't have the right tools maybe that you'd want to have in that container. So rather you bring one with the right each tool and other, or whatever that tooling is that you need, map that new one into that existing environment, take a look, do some diagnosis. I think the other probably common use case is like when you're in, if you're in Kubernetes land, if you will, it's not just, if you're using Rocket as your runtime, it's default networking is such that, let me describe a Kubernetes pod really quick. Inside of a Kubernetes pod, you may have one or more containers. And part of how that pod works is they all share the same network namespace. So actually by default, whenever you deploy a Kubernetes pod, they're all of the containers in that pod are all doing container mapped networking. So that's just like the style of networking that happened. And so why is that beneficial and why would you use it? So why is a lot of times you've got, maybe you've got a logging capability. You've got two containers that won't have co-scheduled next to each other and they've just got high frequency of traffic between them. And so you might want to map that second one into the first containers network because you just want them to talk over loopback zero rather than having them go out, having that traffic go out somewhere and come back in. Good, excellent, yeah. Thank you guys. Okay, we now have a 15, actually 20 minute break and we will resume at 3.30. So, see you then. Test, test, test, one, two, test, test. Two, two, two, two, hey, hey, one, two. So what's it like that? Yeah? It wasn't even on. Okay, okay. Two, two, hey, hey, one, two. Two, two, two, hey, hey, one, two, two, two. Yeah, who knows? Yeah, I think we're good. And this is the same country, right? Same one? Yep, yep, all right, we'll try it. All right, cool, cool. Try it. You want me to hang out? Yeah, it's okay. Where's comes to where, so I'll just change it. Just get it forward. We're just going to ask about the monitors for the exhibitors, so I'm right here. So if you need me on the back end. Thank you. No problem. Test, test, test. I'll keep the volume down there until you're ready to start. Make sure I don't bring this to the bathroom. Please. And talk something in there, I should not be saying anything. Don't drop it in. That's even worse. Is it good enough? Or I need to point to my mouth. That should be good. Right, that's something, it's a problem. So it's actually pulling, there we go. Okay, here we go. So. I do see security to be an important aspect. And I think this is why my session got accepted too, because it's popularity. Yeah, well it's something, because I think it's like, I wanted a lot of basics in here, right? So I'm actually, you know, I'm running this thing. So I wanted a lot of basics in there, right? So I did like the basics of Kubernetes, and then we had somebody else talking about the, you know, overview of Container Network and so on. So I'm going to start with security. I mean mind you, my basic attitude is that containers and security are largely orthogonal. Those containers don't really give you any more security than you would with different processes running on the same machine under different users, right? So, but people think that because stuff is not immediately visible from inside the container that somehow in security. And people are always saying, oh, namespace, you group, they can help you out and the thing is running on a sandbox. For sense of security. It's a picket fence. This has to be willing to jump over. We have to be right all the time and the hackers can only be right one time and it's already good enough. For 40, 45 minutes and then take questions, right? And then we take questions, I'm going to run around with this mic so that people can ask you questions. And also, yeah, 35 minutes, I'll hold up my hand until you get 10 minutes. And you will be somewhere back? I'm going to be right here. Oh, thank you. I'll hold up my hand until you get 10 minutes left and then at 40 minutes I'll hold up the time. But I never timed my presentation. I don't know how long it goes. It may be shorter. Right, you might have to cut it off. Yeah, yeah. That's why you get the 10. Yeah, that's good, yes, yes. Yeah, thank you for giving me this opportunity. Oh, with some kind of thing, not security, so. Are you local? Are you out of the Bay Area? I'm in Los Angeles. Los Angeles. I was surprised people start walking out. Why? Don't be fooled on a previous session and so I asked one of the people why people are leaving and said, oh, the mic was terrible. And with revealed JS, you have to tell a year's screen resolution before you start, which you hadn't done. I run Ubuntu on the old laptop. It's on. Oh, it is on, OK. It has the volume to be viewed. Oh, it's the volume to be viewed. OK, hold on a minute. I mean, I can plug this in. Welcome back, everybody. We're going to get started again this time with container security. Our speaker is Anthony Chow, who programs networking devices for Alcatel Lucent, and he's going to give you a general overview of container security. So welcome. Thank you. Welcome to scale. I feel better now because anytime when I see people walk out of the room, I can just blame on the mic. It's not my content or something I see. Actually, with a little minor correction, I don't work for Alcatel Lucent anymore. Alcatel Lucent got bought by Nokia. But that's another story. My name is Anthony Chow. I am a software developer. I work on writing firmware for networking equipment. But I'm also interested in security. And that's why I always look into different aspect of security. One of the part of some of the features I work on is also on network access control of the switch. I also help on DXCB snooping, where we do some consider as a security add-on feature to the switch. Last time I worked on is to help our switch to pass the common criteria certification. So there's a lot of things that we need to look at. We need to sear out the password. All these things, those are. Security to me is a fun, very fun subject. But it's also a very important subject. I'm on Twitter and just to give you your interest, you follow me, I follow you back. I also occasionally, I write some blogs. I'm also a contract blogger for a car company. I once, I heard there was a conversation saying, in recent days when there is a technical conversation, container will always come up as the subject. So I think it is very true because this slide I put up to just put together as an introduction. I'm sure you guys can think of a lot more advantages of using container in your production. I don't see that we need to go very deep on that. But again, when there is an advantage, everything there is also some kind of disadvantages. And again, this is also an introductory slide that if you can just, I'm sure some of you can always think of some more other disadvantages of using container, but this is not the concern. The concern today is that we want to have a general overview about container security because this is a popular subject. But it seems like container is very useful, but security is not something that people are looking. Although this is becoming a topic that people really look, trying to look into. We, the different types of threats to a container, and I listed some of it, I think the biggest thing I think is people are afraid of being escaped. Escape means when we're running a container, the hacker can escape from the container because we think of a container, it'd be nice if we can run it as a sandbox. But actually, it may not be dependent on your requirement of how we look at the container because there is an article back in 2016 about, I'm sure you guys can heard of it, is that container do not contain. Somehow it was circulate among Twitter or the social media for a very long time at the end of 2016, even though the blog posts, maybe we'll take a quick look at that. This is a very, it's a good, even though it's an old article, but then it is, this is very relevant. I think that the containers do not contain, it lists out some of the things that what container can do and cannot do. I encourage you when you go back, you can take a look at this, just look for this title is from Red Hat, the containers do not contain, and the exact title of this are Dockers Container Really Secure. Also, since I look at Docker, I also have to make a little disclaimer. This topic is more on container in general, it's not specific to Docker, although some of the examples I use is on Docker because it's more popular. It's similar to the Xerox machine. Actually, it is a photocopy machine and it's not a Xerox machine, but Xerox makes it so popular that people intermix the company name to the product, and it's the same thing as Docker container. And for Docker, I like the way because before Docker, containers are already being used, but it's not that popular because it is not that easy to spin up, although Google has been using it for a long time. They also have the project, not the project, the feature board, BOR-RG I think now becomes Kubernetes, which is a very useful container orchestration tool. And also, there is a BOR-MAR. I read this from the book, something to do with... SRG, I forgot the name of the book, but it's talk about how people do DevOps in Google. I think this is a very interesting book too, but back to the subject. Today, we're talking about container in general. There is LXC, there's also Rocket, that different flavor of a container. So this is talk about how things... Because most people, when they think of a container to be a sandbox, they think, oh, the immediate thing of, oh, there is namespaces, there is C-group, and these are the two major things. But in this article, it points out not everything in Linux world is namespace aware. And something like, as I just mentioned here, some of the, let's say, the PROC FS or the SysFS, they're not namespace aware. So these are the things that they may not be able to contain a container using just using as namespace. Let's get back to the slide. The XCIP is one of the things we want to look at and also for container, they're running on a network. In the previous session, we talked about networking. And networking is a dedicated, a very, I'm thinking of trying to think of a great word to explain. It's a dedicated matter to make the network right. Before all these CCID, how to configure a good Cisco network, just try to make things pumping through. Although there's one thing as well, we want things to be able to communicate easily. Of course, when these things happen, security is also a concern. So the cross-container at text is also something we need to look at. I always said security and user friendliness is too spectrum, too side of a spectrum. If you want security, it may not be that useful. If you want usefulness, security may not be there, so we need to strike a balance, tailor to your application. So this is something, one part of the threat is that the cross-container at text, if I be able to set, do, let's say for example, for cross-container at text for app spoofing. We spoof, and then we'll be able to look at other container, look at the database within the network. Well, although for easiness, the bridge network is always an easy way. Even for VxLan, it's also considered, from the container's point of view, it's also a bridge because underneath is, I layer three networking, but then it will just bridge over to the other side of the whole. So to the container, it's still a bridge. Whenever there's a bridge, there's a broadcast domain, and there's a broadcast domain, it's very easy for container to look at other containers. And of course, other types is not, it's very obvious, it's the application vulnerability of the container. Let's say we run a NGINX application. The natural vulnerability of that application exists in that container. We still have to deal with it. It doesn't, if any run on a host machine, physical machine or running container, those vulnerability is always there, so we cannot forget about it. Even if we sandbox it, it's still there. And also, there's also the denial service or attack on the host. We might use up all the Inox on the host, or we might just need some kind of denial service on the host. These are some of the threats. When we look at the threats, then we'll try today. In this session, we try to look at container security from this different perspective with the host base, container base. I can point out some of the third party security offering and there is something I don't know how to categorize, so I put them into miscellaneous, which is a clever way of saying they are miscellaneous. For host space, there is always, I guess people mention about namespace as C group, but also, we've also found out there are also the root capability and also the Linux security modules. We will look at it. There's a general thing about host-based container security. Something about this presentation is that this is a general overview, it's more on the broad side. Anyone is a CISSP certified or GSAC certified? You know, the question itself is not too difficult, but if you have that many aspects of it, you have 10 domains, you have to know everything of it before you pass the test, that's more difficult. And also, same thing with this presentation. There are so many things we can go on or we can go deep into that I don't have enough time to present everything, but this is just to let you know, this will be a survey of all the two things that we need to look at so that you have a general understanding of what security concern we have with containers. I borrowed this slide from the internet. Of course, you can see from the end it's a presentation by IBM. I always said a picture is worth a thousand work or in a developer's point of view, a picture is worth a 1K, 32 bits. Anyway, this is a lousy joke, but just to see if you guys are awake after so many sessions and after a heavy lunch. So this is the problem with the namespace is for isolation. Mainly, namespace is used for isolation and there are a few things. As we look and see, there is the mount point, namespace, network, IPC, UTS. These are the very common. And the last one is the user namespace. This is something new. And I think for Docker container, it's got introduced in release 1.10 for the user namespace. But then, even though it's out of experimental mode in the Docker container, it is, by default, it's turned off. So I believe when I look into it, this is a very nice feature because it helps to mitigate lots of the security concern about container. But somehow, Docker is still making it the default as turned off because there are certain things that still need to be worked on. This is the namespace. And to turn it on is that we use this user, user namespace remap. Actually, is this big enough? Do I need to turn on the fonts bigger? A color, what's the good color? Black on white, let's see. Great on black. This one better? And how about this font? Do I need to make it bigger? Oh, thank you. How's this? Okay. I need to get to some of the cheat sheet. Let's say I have my cheat sheet. I'm not a professional presenter, so I need this to help me out. This is a Docker command that we, it's a simple Docker command that we use to spin up a Ubuntu container. And then it is to map slash, the host slash sc to root sc in Ubuntu. That's what it means. This is when username space is turned off by default. So I have this in here. And then let's see, I'm running as root. Let's say there's something I need to show. Let's do this. Actually, if you are, you can read fast, you will see what are some of the things I will try to do later on. So on this file system, everything is named under root. And then let's say if I want to touch slash root sc, let's say test one, I create a file. And let's see in here. See, within the container, without username space, I'm able to create a file in this directory where the container is running as root and I'm also running as root. And that is not a good thing because if I'm hacker, actually, this is not a good example. If I'm hacker, I have physical access of your machine, you'll be dead. I don't need this kind of thing to get into your network or into your servers. But this is not a good example. Or in some case, in case this is something once someone break into this container and try to do something or maybe do something with the password or to create some account, then this is not a good thing. One thing is that one day I was trying to prepare this demo, I was trying to say, oh, yeah, it's right. We have to set the flag when we start Docker. But where is it? Is this a 16, Ubuntu 16.04? Where do I make the changes? I spend the whole morning and then eventually I find an easier way try to how to reset it with the flag. It's not changed something to do with changing system D changing system D but let's say, let's see, hold on, let me get out my cheat sheet. I forgot, let's see, it should be, it should be here. We have a file called JSON file. But if we can copy, I already have it prepared home. If I can type, let's get out of this. DAE, this is a simple JSON file. I already have a user created as Docker doc remap. Let's see, a user. So I try to remap to this. So now I need to restart Docker to make it to work. I have a slow machine, actually it's not too bad. So I restart, let's try to do this again. Try to do the similar thing as we did before. We start, we try to start a Ubuntu container and then we also remap the two directories. This is expected because anytime we restart the container and then we create a new, even though I ran this command before, we have to download Ubuntu again because in a short way is that, let's see, it's varlib thing Docker. I hope this is it. Because now we see we have a new directory and this is because we turn on username space. So now this, now I am in the container. Let's see if we can do the similar thing as before by, let's say we do this ls again. This is inside the container. Before the file is owned by root and root, if you remember. Let's see if I can go up and see this whole bunch of file. Before the file is under this. Now we have username space enabled. If we start the container, it's changed to a different, it's nobody and no group. And then let's see if we can do a similar thing. Let's touch, let's see, say test two. We create another file in this directory. Now with username space enabled. Now we cannot do this. So username space is in a way is a very effective way to do some of the things, but I have to do more research as to why there are some other problems that prevent this to be the default to enable username space. But you can see username space with this because what it does is the root inside the container is not root in the host. And this make a very big difference. We remaps to a different user. So it will not be running as root. And this helps mitigate lots of security concern. And now gone in security term, we can only mitigate as much as possible. We can, we cannot totally eliminate security concerns. We just make it as low as possible. And this is something, let's me get out of this container and then move on. So this is just to show you with name space, this is a useful way of doing, to prevent some of the things from current slides. And let's move on. There's also the, yes, if I can answer, yes. I am not so sure, I don't know. I do not know the second, but also, but it depends on how you spin up the container. The file, you have different ways. One of the quick way I do the demonstration is just to map, but you don't really have to map that. You can attach a volume to the container. That's, there's also one way of doing that. I think people, if they are doing in production, these things is, they can maybe help answer this question. But for me, I do not know how that works. And we move on to C group. And this is something, name space is to limit the view of the system. And the C group is to limit the resources. To follow, again, I borrowed this from IBM to prioritize control and obtain metrics of a group. So this help, this is something. And then there is one more thing is called the root capability. For the first line, this root capability is a fine-grained control over root privileges. If you go to the user on a Ubuntu system, I think Fedora also may have this tool in the same directory. And user include, it will have more, a lot more, let's see, let's maybe take a look at, this is the other one is root, this is better. So before that, I think we can show you the capability of Aspin, okay, SH print. This print now all the capabilities that is divided, and let's see, forgot where it is. There's too many things I need to use to include Linux. This is to show different ways, this is good. For me, because I am a developer, always like this .h files, it helps to, usually it helps to clarify different things because you're supposed to have a good comments on this .h files. If you're interested, just take a quick look. This capability, there are a few things we need to take a look at. This is one of the web page that shows all the different capability, if you're interested, just go in. And also, when we run a container, we can also specify. If we want this, we can do a cap drop, do a, this flag will help us to drop one of the capability. You want, I'm not sure if Ubuntu has the same thing, this article that I read is that Red Hat has something in this way, let's see if we can go there. It has a script to look at the capability of a container, so you will know what kind of capability this container has. Do they have, because we always want to follow the principle of least privilege, we don't want the container to run more than it should or necessary. So, if we can find what are the capability that a specific container that you are spun up, then maybe you can decrease the capability of that container. And this one, for Docker, has a good way of using set-com to control. To control, I can go to the website and take a quick look. This one, this is a good way. Maybe I will have this included in the end of slide, because my slide will be attached to the presentation. This, no, I already passed. It's difficult to think and it's the wrong way. This is not the right page. Let's see, let me jump ahead a little bit and go here. Wrong one again, this one, it's different. It should show me the, let's see, let's see, let's see. Thing is this one. It shows you how to use set-com to control because it has the default profile that you can specify with a JSON file to create or to limit what kind of system call because for this capability, it is trying to exclude the CAPE system in. What I have found during the research is that this is like a catch or capability, some of the things you can do and with the container to break out, to escape from the container is using this capability. So it'd be nice if you can find out what root capability a specific container has and how we should be able to control and to follow the principle of at least privilege. Then so we get back to root capabilities. And this is, another thing is access control. In securities, there are always a saying called, I forgot exactly, there's defense in death. Just like when you have a castle, you have a thick and tall wall, but before that, I'm assuming the European castle type that in front, there is also some gap with water, maybe you have some hungry kogledows running so that people cannot swim over and, because this is called defense in step. With all these namespace group are also root capabilities. We can also use this type of access control to help to mitigate some of the things. Usually when we look at the Linux system, we are looking at the discretionary access control, the file privileges, user, the group, let's say if I'm just a regular user, I'm not root, I have to ask you to root so that they are able to go to slash ETC to do different things. This is a discretionary access control, but there is also something called a mandatory access control where the system specify the subject can access the specific type. Everything is a label in the mandatory control. I think if someone is involved with the military, they're more familiar with mandatory access control. At home, I have two machine, one is running Ubuntu, not this one, this is just an old laptop, I have an NUC running Ubuntu and I have another system running Fedora, just for the purpose so that I can learn SC Linux, which is one of the flavor of mandatory access control. This is something to do with two of the things because Linux security model, there are two flavors. Actually more than that, there is also the GR security which are mentioned under, I think, miscellaneous. GR security is a specific hardened Linux system that you can use before it was free, but now I think you have to pay to get the operating system but it's more fine-tuned and it's more supposed to be hardened, more secure than the general Linux distribution. For SC Linux, there's three most enforcing permissive and disabled. Some people I know, just some of the developer I know, they will use Red Hat or CentOS, but they will, the first thing they do when they spin up an operating system, CentOS or Red Hat, they will disable SC Linux because there's too much trouble, they don't want to deal with it. But in production, it's a different story, we do want to know, but I think it's something nice that because we have three most, one is the enforcing permissive and disabled. Permissive is that it will flag you with the concern or the violation so that you'll be able to tune instead of really enforcing. Because this, again, tuning security in a container environment is dedicated as a fetch out issue. It's not a silver bullet that, oh, follow this rule and you can do it. But then this is before you have to move SC Linux to be the default, we have to put it, we can put it in permissive mode so that we know all these other things that's not so we can tune it. And then there is another way as the app or more, if I can say this correctly, the two mode is enforce and complain. I think it's also a default in, you've been to, this is through PPA more status. These are the things, it's a little bit different than SC Linux, this is a file base of a path based security. It's the way the implementation is a little bit different but it's a default way for if you look at what is that called Docker info. You will see in, if I can find it in, believe me, it's there. It's spotted. Yeah, you can see it there, it's good. Anyway, maybe it's crap. Oh, it's, anyway, if you see it, it just be this good, it's there. I also have a machine that I can show you more about SC Linux, but I don't have anything, we have time to go through that. But I do have to want to point out a one of this article. Again, this is from the same author that says, the talks about the container cannot contain. It talks about more things about the file system and it also talks about capability. These are the things that, I think these are the default. See, dockers remove these things by default. So for sure you will not, but if you need it, because sometimes depending on the application that your container is running, these capability might be necessary for you then you will use the cap add to add it on to bring that. But what I want to point out is that there is a good way, this talks about cap add and cap drop. It talks about namespace and them, but the very big thing is about the description of SC Linux. I think this is, SC Linux is a labeling system. Every process has a label and then it's again, but what I like it is that with the picture, again, the picture says it all. This is supposed to be, I think they make it as a colouring book. If you look at it, it says, this is the kernel. If you are a doc, you cannot eat the cat food. This is something to call type enforcement in SC Linux, but that's not enough. There is also something you have to look at is the multi-category security enforcement. If you look at this, the food itself is for the doc, but specifically for the doc name spot and this doc is called Fido. So the kernel doesn't allow it to eat the food. This is a very simplified way of how SC Linux works. This captures the essence of how SC Linux is implemented to control user access to different resources in the host. So I think this is worth something going to because, again, this is an overview session so we're not going to get into more detail of that. So for any question, comments, because you might know this subject a lot better than me, but again, this is a general overview, so to prompt you to see if you can look into these specific areas. If no, we move on. We're done with the whole space security. Now we talk about container security. There are three different ways of looking at the container-based security. I look at it as two different categories. The first is the digital digest and then another one is the container scanning. Digital digest is an important aspect in security. Again, I get all this picture from the internet. At this time, I don't have the skill to draw my presentations yet, but this is a very good nice way of putting things, how digital signature works. Basically, you have a good container. It has a hashing algorithm. One thing you have to remind is hashing is only one way. You can only hash one way, but for other encryption, there are also the symmetric encryption where both sides have the same key. You encrypt and decrypt with the same key and also there's an asymmetric encryption which I think is necessary for this type of discussion. Asymmetric means that there are two key, a private key and a public key. You can use a generally available PKI. You can buy the certificate from some of the vendors or you can create it with the open SSL, Kamel, or Linux to create your private PKI. But the most important concept about that, about asymmetric encryption is that there's a public key and there's a private key. You encrypt something with your private key and you decrypt with the public key. If you encrypt with a public key, you decrypt with the private key. It's always the other way, so you cannot encrypt with the private key and decrypt with the private key. Even you yourself cannot do it. You have to use the other way to make sure that, and how this kind of digital certificate works with the container is that they hash the cold container. It's not like a CRC in networking because they just, CRC the whole packet and then if the two bits are flipped, then your CRC is the same. But this is different from digital certificate. It won't happen this way. It will just hash it and then use to see whether it's the sender's private key to sign the message. They use it to say, oh, this is the hashing value and send it to the receiver. And then receiver use the public key of the sender to decrypt it and then they do a same hashing algorithm to see if the number matches. If it is matched, the container content has not been altered or changed. So you are safe to use that container. This is one way to, I think this is how, in a nutshell, how Docker container trust is how it suits them. I think there's more in depth to that, but then this is in general, this is the general understanding. If you use something, you can decrypt and you can extract 509 certificate. This is two sessions. The beginning part is the certificate, which has all the values, the versions and the different contents. And then, again, if you look down here, it's the private key to sign this certificate to ensure this content is from the sender. So this is one thing to look at. And what I point out is that CoreOS has an interesting implementation of how to verify or to ensure the integrity of a container image is that it is doing it during runtime. I don't have time to go into detail. If you look at just Google DM, Verify and CoreOS and during the keyword runtime, then you'll be able to find out this interesting article to see how they do it in the runtime. And then for container scanning, there are different vendors that is providing different ways of container scanning to make sure you're not vulnerable. Because again, it's one of the threats to container is that generic vulnerability of the application that is running on as the container. So these are some of the offerings that is available for us so that we can be able to scan the container to make sure their security patches are up to date. Then these are some of the third-party security offerings that we can look at. I did not sort it with alphabetically. Our core is one thing. Encore, I'm not sure if any one of you is going to the Los Angeles Docker Meetup. I think they had a presentation last time when they meet a presenting this. And then if you're interested, just go to... I did not attend, I cannot make it to that meetup, but I know last presentation was about this product. Before we talk about TwistLock, the previous presenter asked people if anyone knows about TwistLock, I think. If this session is before him, then every one of us can raise our hand and say we heard about that. And then these are the things that third-party offering that you can go into and see. This is what I call the miscellaneous. OCI, the Open Container Initiative. When I Google this, when I put together this presentation, I did a lot of Googling. Open Container, I cannot find any specific content to see how they are trying, using the initiative to improve. But one of the interesting things is that one of the objective, one of the goals within OCI is that they want to improve the security of container. I'm not sure how far they have gone, but that's why I want to mention that OCI, plus also I think with the ecosystem, people are moving toward the OCI spec. Even CoroS, Dockers, they're all moving toward, and I think Dockers donated RunC to OCI to make it open source or to make it available to everybody. So I think for container security, OCI is definitely a thing to look into. Another thing is interesting is that the hardware assists. Intel has the clear container. It has a very interesting concept. What we talk about from the beginning until now is about running a container as a container. I don't know if this sentence makes sense, but Intel does have an interesting concept of running a container as a VM, because what they said is that a container running on software cannot take advantage of the hardware-assisted isolation, in which case we can all agree is that the isolation of a virtual machine is much better than a container, and this is where Intel is trying to put itself, is that it will try to change the KVM, I think it's the KVM tool and some other things to make the container take advantage of the Intel-assisted isolation, so make it more secure. The last time I had was in a seminar, it's some kind of workshop, and Intel clear container make a presentation. Lots of questions talk about, so are you a container? Are you a VM? I think they are both. It depends how you look at it. If you look from the application point of view, it does work like a container, but if you look from beneath, it does have the hypervisor and it takes advantage of the hardware assist, so depending on which you're looking down or looking up, it all depends, but the general idea is that Intel is moving this direction is to use the Intel hardware to help the isolation. And again, under miscellaneous, thing Docker 1.13 is just being released, and I have just to mention the secret management. Secret management is a very important aspect of hiding the secrets of the password or these certificates. I think this is also worth looking into. Let me take a look at the time. I almost run out of time, let's see, because for security, this is the, if you look at my description on the talk, there are three parts. The last part is supposed to be a demo. I'm not sure if I have time for the demo, but let's go on to see. I choose to use the Ansible container. I'm not saying, it is supposed to replace, I try not to say the politically correct word, not to replace, but the augment Docker compose. It's supposed to be able to use the same, because they use the same JSON file, they'd be able to, what I like Ansible, before container, well, I already like Ansible, because I look at puppet, I look at chef, but all I like Ansible a lot, and I have done some work on Ansible, so I think it is very nice. Mix of things, not only can this Ansible container use your existing Docker compose file and spin up, it will also add on different roles to help your security. To, let's say, let me see if I have time to make a quick demo on this. Let's see, for the Ansible community, they have something called the Ansible Galaxy, where people write already existing roles to different things. When I search, I search this. This may not be a good way to do it in production, but this is a good thing to show examples. Let's say, I download this, I copy this from Ansible Galaxy. Let's see, there are a few things. One thing I want to look at is that, SSD, oh, one thing I want to show is that Ansible is very, container Ansible is this, let's say, make can be Ansible container. This command is very neat. It's already, we can provide you with a structure. See all this is, oh, I did it on the wrong. This is a general mistake I always make, is that I make it directly, but instead of going into the directory, I did not do and do the command in here. But anyway, let's do it in just because of time. Let's do it again, Ansible container. It's not Ansible, but it's Ansible container, init, initialize. So it gives you the directory, and it also gives you the different files. A sample file, based on YAML, main dot, this is a sample file, that's easier. Then container, this is supposed to be the replacement of Docker Composer, how you spin up a container. But what it does is I show you here is that same thing. For this download, there is Ansible file, and then there are this container. So this is a simple, able to see, yes. This is to spin up a Ubuntu container with SSH using port 22. If you don't want port 22, you can change it to something else. And this is some of the things specific for container. But the good thing about what I meant by security is that when we can do this, I mean, well, I'm not sure why this is the case, but people said it's not an advisable thing to turn on SSH on a container. I don't know, although I picked this example, because it has the ingredient that I want to show how Ansible can be used as a security tool for container. Because if you look at this, we have all this, this is a very bad way of thing, because we are coding the password to be password inside a file. Maybe this is what the secret management can, I think Ansible also has something called a vault to help to hide the password, but this is not that important. So these are the things that we try to do. One of the things that's there for Ansible, we can configure the container or the VM that we want or the target. To say in Ansible's turn is the target. We look at the target and then it will be able to, let's say if you attend servers, then there will be 10 targets. There will do be the doing the same thing on these two. You can pick and choose. If you do want to have some security hardening of a Linux machine, you can do it on all 10 or you can selectively do some of it. And this is something you can change. One of the replace, this is to replace the destination. This will the target with something and this is how you can, this is a good way to also, just to orchestrate a container, but also can add on to make it more secure in a way, so some way to harden your Linux system. Let's see if I have time for a little demo for 18. So I still have a little time. Let's see if Ansible or my machine can work it. Oh, I forgot to show you the SC status I get and force on the. It's very simple. Just if in that directory, let's say in this directory I do this command. It will build. Are you in it or are I should be doing this in the Ansible directory? Let's do it again. So I pulling in the files, building. While it's building, is there any question and comments about security? I think this is a hot topic. Everyone wants to talk about it. And if you have some, I may not be answered the question, but if you throw it out, we can answer it together. Yeah. Will the slides that you're posting and all the URLs you've visited be in the slides because they weren't. I will put it at the very end. What area within a container security do you think presents the most serious vulnerabilities? I think the answer to, everyone can anyone hear about the question? It's okay. I think it depends because security contacts within different organization is different. It depends on the application you're building in different environment. Let's say for me, I used to write software for embedded system. So my security concern is one thing. If you're writing a web application, then your security concern is a different thing. That's why I think for this presentation, we try to get a much border perspective of different ways of looking into container security so that we'll be able to look into it that more in depth and to make the proper adjustment tailored to the requirement or the needs of the organization. I think closer, Mike. So one of the things that I see in how-tos all over the place, everything runs as route. How much of what you're talking about right here is able to kind of lock that down so that we can follow the how-tos without compromising security and keeping things from being able to break out of those containers? Do you mean the running Docker statement as a route? A good example, like the Ruby on Rails, everything runs as route. And we've been pounded in our heads for our entire adult lifetime, never run anything as route. I myself do not have to answer. Is anyone can answer that question? Will someone from Docker can answer that too, maybe? I think you get an answer. So eventually, user namespaces are going to solve that because they will allow to map user route in the container to something non-route outside of the container. So for now, if you want to be secure, you have to put user something in the beginning of the Docker file to say this is going to run as non-route. But eventually, once all the little problems that you mentioned in the beginning, like why doesn't Docker engine enable user namespaces by default because there are mini-little things to figure out mini-little features that break with your namespaces. But once this is figured out, then user namespaces will let you run stuff as route, except it will not be route, it will be mapped to a non-route user, so that'd be fine. Under Project Atomic, there's also a little project called Bubblewrap that basically simulates that until user namespaces are available and standard. Could you please address the use of TLS certificates to lock down the daemons that are running? So whether it's in Kubernetes or it's the Docker daemon itself that are running, it's system daemons, what's the value of that? And clearly it's insufficient for the container itself, but does it help at all in locking down the container? What do you mean by to lock down the container in what sense? So that authentication access to say Docker-D, Docker-Engine, the Kube daemons, Kube-CTL, Kube-API server, all that stuff is authenticated only with TLS. And does that help or does that hurt? I'm not too familiar with the tools, but usually in the security world, using a certificate is always better than using a password-based authentication system because with the certificate, you get more elements being put. You have the PKI, you can authenticate the user or to creation, you can have more sense of security with the digital certificate and use a simple password. I think this is my answer to your question, although I do not know too much about the tools security. Anyone can address that too because this is a beauty of the community. I do not know everything and then if someone can answer the question, then the community will grow. We'll have to follow up on that because we're actually out of time at this point. So thank you very much. Thank you. And don't go anywhere because we're gonna have a couple of speakers up here in 10 minutes so we're going to show you building Minecraft servers on Docker. Oh, just in case, I'm not working with Akital Lucent, I'm looking for a full-time job so if you have some job offering, ping me. Rex opened in Potato Technologies in Red Hat. Oh. See you in a moment. What do you buy me somehow? I'm not gonna make you think you are one of those people. So if you're interested, absolutely. Let me give you, I know you know where to find me. Is it easy to see the screen? Of course. If you don't see the particular point on the slide. You're just gonna have to like stand here for the video and like look at that. See right on. Okay everybody, take your seats. You're gonna learn how to deploy Minecraft on Docker so that you can not get any work done for a month. This is Proud Heng and Akira Wong, both students at different UCs, I believe yes, who are going to show you how to do that. So welcome. Thank you so much. Good evening. Like you said, if you're here for Docker and Minecraft, you're definitely in the right place. Thank you so much for coming out tonight or today. We're gonna get started right away. He already introduced us, but my name is Akira Wong. I'm a computer science senior at the University of California, Irvine. And this is my co-presenter, Proud. Hi, my name is Proud and I'm a junior at UC San Diego studying math and computer science. So let's get some audience participation going to start off this presentation. How many of you have heard of or played Minecraft before? Please raise your right hand. Yeah, that's a lot of you. Keep your hands up. How many of us have used Docker in production or development before? Please raise your left hand. Now some of us look like we're going down a roller coaster. Some of us look like we're asking a teacher a question, but we all have something in common today. Please put your hands down. Thank you so much. We all have something to learn from this presentation and that's that Minecraft is awesome. It's an excellent entry point into Docker. We can show you an excellent demo to show at your local meetup group, to show to your boss, to get you into Docker and to show what's possible with Docker in containers. And for those of us who didn't raise their hands, you guys are in for a treat. This is going to be really cool. So without further ado, let's talk about Minecraft. Minecraft is gigantic. Some would say huge. With over 23 million PC downloads and 100 million players over all platforms, Minecraft is a gaming phenomenon. Part of the Minecraft experience is exploring these huge sandbox open worlds with your friends and building complex megastructures like castles, farms, townships, who knows, right? The only problem with the entire Minecraft experience is the Minecraft server setup. It's a little bit of an ordeal. You have to jump through way more hoops than you probably should to do the thing that you care about, which is ultimately play Minecraft with your friends. You have to configure ports, get your Java correctly, all your dependencies, all that stuff is keeping you from what you actually want to do, which is play. So how do we solve all these problems, all the difficulties with setting up a multiplayer Minecraft server on the PC? The answer is Docker and containers. For those of us who weren't very familiar with Docker, we're going to give you a very brief overview. Docker is an isolated application environment for us to host our Minecraft server in. It conveniently encapsulates all the dependencies we need, but is more lightweight and scalable compared to a virtual machine. This is because the kernel is shared between container instances on the same host compared to a virtual machine. Therefore, it's more lightweight and scalable and you can fit more containers per capita. In addition, containers are also portable. This is similar to the idea of the Java virtual machine where you can take any computer capable of running Docker and run any container on that computer. This means that our Minecraft server solution is completely portable to any platform and it's very general. This combined with the next facet configuration as code in the form of Docker files allows us to encapsulate everything we need to set up this Minecraft server and have it be compiled into what are called Docker images which are then hosted on Docker Hub, which is kind of like GitHub for Docker. We can pull down these Docker images and then Docker stamps these out into individual Docker containers. This allows us to automate our Minecraft server setup and get all that nonsense out of the way so we can do what we actually care about, which is, again, Minecraft. It's all about Minecraft, guys. This all sounds great, right? But it comes at a price and that is the price of statelessness. Statelessness is a core tenant of Docker. It's the idea that if you spin up a container and you write some data inside of it, then later when you kill the container, bringing up the container again, there won't be any data retained. And, you know, if you're following along, you're thinking maybe this has some implications as to the long-term sustainability of our Minecraft server. You're going statelessness. What was that? That's double entendre. It's pretty good, I'm sorry. Okay, so continuing on. Yes, there may or may not be a problem with our Minecraft server. If any time a janitor trips over a cord, they'll knock out everything, all our progress we made. But to get everyone on the same page, I'm gonna tell you a story just to make it more real to you. So imagine you watch this presentation. You were super enthused about it. You were like, man, it's so easy to set up Minecraft in a container. That guy who was talking, you did it in 30 minutes. It was so fun. You've been playing with your friends and family for two weeks. You've set up this massive castle. You've explored and created a very nice farm with tons of watermelons. You've established friendly relations with the local villagers. All this time, you're thinking in the back of your head, what was that thing that that guy said about containers being stateless? Stateless. What would ever happen if this container were to ever shut down? As soon as this thought occurs to you, like a curse, AWS goes down like two days ago. There's a combination, Hurricane Zombie Apocalypse Firestorm. Again, a janitor could trip over a court, who knows? Your server goes down and two days later, AWS comes back online. Your lips tremble as you start up your container and your blood runs cold as you log in to see the empty expanse of a plane's biome. All your progress is gone. Your castle disappeared. Your watermelons never to be eaten. That villager that you named villager McVillageface, you'll never see him again. This is a disaster. We wanna play Minecraft with our friends. We want our state to be retained. How can we keep all the benefits of Docker, the easy deployment, the universal portability of a Minecraft container, but still retain our state between container instances? Proud will tell us exactly how we're gonna do that. So the burning question we now probably have is, how do we make sure that our Minecraft server's world data is going to be safe? And we figured out that our containers need to be stateful in order to save our Minecraft server world data. And the way that we can do this is similar to how we save our files when we're working on a public computer. So if we're working on a public computer, we'd want to not just save our files only on that public computer because as soon as you leave that computer, you're probably not going to see your data ever again, right? So you'd want to save your data somewhere else like a USB drive or the cloud or something. So you can still have access to it. Now think of that public computer as your Docker container where if you store your data only in the Docker container and if something bad happens like your Docker container does get killed, then you won't be able to access your data ever again. So think of the USB drive as some space that's outside of your Docker container. And we want the space to be ideally a place where we can read and write this data from inside of our Docker container but also be able to access it from outside of our container. So the way that we can do this is with a volume out, with our Docker containers. So here's an example of a typical volume out. So let's say that we want the space to be on our local machine. So like the Minecraft folder on our local machine in the home directory. So what we can do in this volume out in our Docker container is we can map this Minecraft folder to a volume inside of our Docker container. Let's call this volume data. So we get our Docker container up and running. We get our server up and our world is generated. We started putting in some data into this volume and that data will also be stored in the Minecraft folder on our local machine. So as soon as this Docker container gets killed for some reason, then our data will still be safe in the Minecraft folder on our local machine. Everything's okay. And all we have to do is spin up a new Docker container with that same volume out, that same mapping and our Minecraft world will be the same as we left it. Our state is retained and we can continue to store even more world data in our Docker container and keep it on our local machine. And that does what we want, right? But before we get too excited, let's take a second look at where exactly we're storing our world data. We said that we're gonna store it on our local machine. And for those of you who worked with the computer, you know that that's not the best choice, right? Like something's bad is gonna happen. Your computer crashes, your hard drive fails, you accidentally spill some lava on the keyboard or laptop, just something bad's gonna happen. So where else could we host our Minecraft server? That's going to be more reliable than our local machine in terms of uptime and storage. And well, the answer is the cloud because that's the answer to everything these days, right? So ideally we'd want to set up our Minecraft server in the cloud, so let's set up a virtual machine in the cloud, get a Docker container running in it, and we want our Minecraft server world data to be stored in some block storage in the cloud. So that sounds good, right? And it turns out we can do this if we introduce our final element in our Minecraft server setup, which is the open source storage driver, Rexray. And what Rexray allows us to do is to get our block storage volumes to be mounted directly to our Docker containers, so we can store our Minecraft server world data directly into block storage in the cloud. So we'll be showing a demo on how this is set up. We have a specific example with using AWS and EC2 and EBS Docker containers and Rexray. So if you've seen AWS before, this is gonna be pretty boring. We're gonna be accepting defaults pretty much all the way through this, but it's gonna go super fast. So those of you who haven't seen AWS before, you probably wanna set up an account and learn it yourself. It only takes like 10 minutes to figure everything out that you need to know. We're gonna be setting up a virtual machine, and the way that we're gonna do it is by hitting this blue launch instance button and choosing a Ubuntu image just to make sure that all our commands are consistent. Just so you guys know, all these steps from now on are going to be covered in a GitHub page that we're gonna show you at the very end of the presentation. So don't worry about following along with the minutia. We've got you covered. It's all in the GitHub page. So from here on, we're just choosing the size of our image and everything's good. We can accept defaults pretty much all the way through this. The one thing that we really care about here, oh, here's subnets. So brief thing about subnets and Rexray. Rexray allows you to share volumes between your containers, and basically it's what's led you, sorry. It lets you upgrade your Minecraft server afterwards simply by sharing the slash data directory that we create for our Minecraft server. So in order to do this, we have to make sure that that subnet is exactly the same for all container instances that we spin up or for all virtual machine instances we spin up. But aside from that, we're all good. We can accept defaults and the one thing that we care about, we'll get to in a little bit and that's the security group. We have to make sure that the Minecraft port is open. Port 25565. And it's very simple to do this with the UI. You just hit some buttons, hit 25565 and then make it accessible from anywhere and then you're good to go. You can configure SSH keys and launch the instance. So from now on, we're gonna be in the command line showing you how to install Rexray and Docker and how to pull down our Docker image, which is a compiled Docker file that we created for this Minecraft server. And Proud's gonna run you through this. Okay, so SSH into our virtual machine that we just set up at any time now. And then as soon as we get that up, we'll do our basic pseudo-appget updates and upgrades that you Linux enthusiasts all know and love and we'll also install Rexray with this curl command. And after we install Rexray, all we had to do is set up this configuration file in this config.aml file. That's from the GitHub repository. So we'll just dump all of that into this config.aml file and you'll see that you will have to put in your own access keys and secret keys. We obviously have not put them in because we don't want 50 virtual machines to magically appear under our account. Okay, so after that, we'll go ahead and install Docker. Docker has this nifty script at get.docker.com. All you have to do is run this curl command and pipe it to the shell. You also sped this up with some video magic so you don't have to wait through all of that. And what we're doing here is we're adding our user to the Docker group. So we don't have to use sudo in front of all of our Docker commands. And we'll have to exit SSH back in for that change to take effect. We'll do a Rexray version to make sure Rexray's installed correctly. Docker PS to show Docker's running correctly. And we can start up Rexray. After that, we'll do a Rexray volume list to get Rexray to talk with our AWS account. So it should list, or no, we didn't. Okay, so we'll do a Rexray volume create to create a volume for us to store our micro server world data. And what we'll do is we'll map our world data. There's the Rexray volume list. Okay, so this shows that we created our EBS volume through Rexray. And it also shows up in the EC2 dashboard. So we know that Rexray is indeed talking with our AWS account. There we go. There's our volume, it's available. And then we'll go ahead and pull this Docker image from Docker Hub. We'll show you the Docker file in a little bit. But this Docker image will have what it takes to set up the Minecraft server. And so this Docker file allows us to automatically set up our Minecraft server. So whatever commands are in here, we'll get shown through, yeah, that is the downside to having videos. But at least our Wi-Fi or we don't have to depend on the conference Wi-Fi. Anyhow, going back to the Docker file, these commands that you'll see in the Docker file are what are going to be run whenever you have a Docker container that's based on this Docker image that we pulled from Docker Hub. So first we're going to start from Ubuntu 16.04 and we set a variable for our Minecraft version. We'll also get the basic essentials. Oh, thank you. We'll also get the basic essentials, like updating and also getting Java 7. Then we'll be getting our jar file for our Minecraft server. That is what will set up our Minecraft server. We'll also expose the port 25565. And this won't set up the whole port configuration. We'll also have to expose our port later on in our Docker run command, which we'll see later. And we'll have to accept the terms and conditions to start up our Minecraft server because that is what we have to do. And also we'll run our Minecraft server jar file to get our Minecraft server up and running. So Akira will go through the Docker run command. So we pulled down this Docker image from Docker Hub, which is that compiled Docker file that you saw earlier. Now the last step is to, oops, sorry. So the last step is to perform a Docker run. And this is going to encapsulate everything that we've set up so far. We're going to do a port map. Sorry, that's hard to see from podium. Anyway, we're going to do a port map from port 25565 inside of the container to 25565 outside in the virtual machine, part of that security group that you saw set up earlier. In addition, we're going to set up a volume mount to MC server volume and call out our volume driver as Rex Ray. And you'll notice that this is being mapped to the data directory inside of our Minecraft container. At the very last step, we call out the name of the Minecraft, or sorry, the Docker image that we just pulled down and we give it a run. Once it spits out a container ID, we're good to go. And we can sign into our Minecraft server container. Excellent. So you'll remember that the IP address that we used to SSH into this container, sorry, this virtual machine, and we'll just take that and go directly to Minecraft, add a colon 25565, and we can sign in right into this Minecraft container. And just to show you guys that we're not liars, we're going to make some changes in this server, maybe some modern art. I'm thinking like a backwards L for Linux or something, just to be fancy. So this world has been generated right now, and if we didn't have that volume mount, it wouldn't stick around when we shut it down and brought it up again. So you'll see it's a pretty nice day. It's a great day today too, anyway, outside. We're gonna build a nice backwards L to show you guys that the server is in fact persistent. And this is me messing with the controls. For some reason, proud bound like Q to placing blocks. If you played Minecraft before, that's really strange. Most people don't do that. She didn't have a mouse at the time. So yeah, we're gonna kill this container to bring it down. And we're gonna bother it with the Minecraft client one more time to show you that it's actually down. And once it spits out a container ID, we know that the container's down. We go bother it. That virtual machine is still up, but there's no container there. So it says like, what the heck are you doing? So we have to spin up the container one more time using that same Docker run command. And then once it spits out a container ID, we're free to log in and suspense. Boom. Backwards L right there. Persistent Docker container. Excellent. So what have we done here? We've created a Docker container that encapsulates all the dependencies, all the ports that we need to set up. It sets up Minecraft for us and does it all automatically. It's great. Plus, it's stateful and we don't have to worry about AWS going down or any disasters in the future. We've created something awesome. So proud is gonna bring us home and wrap it up for us and sort of talk about the basics of this presentation one more time. All right, so we saw in the demo that setting up a Minecraft server or at least a persistent one isn't too bad after all, right? If you use Docker containers and AWS and Rexray. So just as a recap, we'll go over the questions that we had earlier in this presentation. So like figuring out the dependencies and what we do if we need to add more updates as well as how do we configure the ports? That's handled by putting our Minecraft server in a Docker container. So for that, we take advantage of the Docker file. If we need to add a dependency, then all we need to do is take another line in that Docker file, build a new Docker image and just bring up a new Docker container using that Docker image. And we don't have to type a single thing after that well, after the Docker run command, you don't have to manually set up the change as well as if we want to configure the IP addresses, making sure our server stays online and our data is safe, that's handled by putting our Minecraft server on the cloud. So when we put our virtual machine on the cloud, it already comes configured with an IP address, which is good, we don't have to deal with that as for making sure that our server's going to be reliable in terms of uptime and storage, that's basically what the cloud's business is, ideally in general, it's going to be more stable than your local machine. To get our Docker containers to work with the cloud, we use the open source storage driver Rexray. So now what can you do? You can set up a virtual machine in AWS and install Docker Rexray on it, get a Minecraft server up and running and be able to play multiplayer Minecraft. And as a bonus, if you ever want to upgrade your Minecraft server, like throw some more RAM in there, then you can create a new instance, like with a different instance type or something, then you can reuse the same volume that you had for your earlier AWS instance and use the same volume out command, the same Docker run command. And that would be how you would change out your Minecraft server instance. And here's a few bullet points on Docker Rexray and AWS. If you want to recap, these will also hopefully be in the slides that will be available to you. Also, here's our contact information and here's all the steps that we took in the demonstration, which is in a walkthrough in our GitHub repository. So if you want to follow along at home and get a Minecraft server up and running, you can do that. We also have a Minecraft server that we set up for one day that you can connect to. It's in creative mode. You can get crazy, but not too crazy on it, if you would like. Well, thank you all for your time and we can answer any questions you have. Questions? So your Docker image has the Minecraft and Java in it. Why isn't Rexray part of your Docker file? So Rexray is a volume driver for Docker and it's part of the setup process. It's actually a separate thing from our Docker file. So to be honest, you can accomplish this entire project without Rexray. Rexray just allows you to throw your storage onto the cloud in EBS. It separates your storage. Instead of it being a local volume out, you can put it on EBS and it's separate and you can consolidate it with data from other servers theoretically and it's what allows you to upgrade your server later into a bigger image or yeah, bigger instance. Yeah, I guess to clarify, we didn't actually use Rexray inside of the Docker container but we set up Rexray on our virtual machine in AWS. So there's that. Did you guys experiment at all with adding mods to your Minecraft server? Yeah, there are definitely mods that you can put in on this. All you have to do is, usually the mod hopefully will package it in a jar file and they'll host it somewhere for you. So all you have to do is pull down a jar file in your Docker file. Right now we're using a default jar file from Minecraft themselves but as long as you can find a jar file hosted from somewhere, you can pull it down and run it yourself and it'll work completely fine. Also, if you want to tinker with the files, you can also view them no problem by just doing a plain old volume mount and you can look around and see what actually got written into that data directory if you ever want to. But yeah, to answer your question, you can definitely mod these container servers. Thank you. Quick question. So when you create the shared data directory and you upgrade Minecraft versions, does that work? I mean, is the data directory guaranteed to be compatible between versions? Yeah, generally that's just, that's handled on Minecraft side, right? Generally when you upgrade your version of Minecraft, usually the world will look the same. You might have like new abilities or new things you can make but it should be, the world data is made out of the same blocks. It should be compatible. Thank you guys so much for coming out today. If you don't have any more questions, we'll stay up here. Please play on our Minecraft server if you like. We know it's gonna be chaotic if everyone goes in there but it's only gonna be up for one day so make some friends, have some fun. Thank you so much. And thank you everyone for joining us for the container day. Don't forget that we have some keynote sessions tonight from six until eight which are in the main ballroom like around the corner. We get in there ourselves. Well it's kind of strange to think of it that way because Rek's way is just mounting, it's like Docker itself is providing that functionality of the volume mount and it's not really built into the, the container is meant to encapsulate like the application itself and not like, Rek's way is more like a cable or a wire that goes from one place to another rather than something you would configure and install. Sorry, if that's difficult to understand, the guys that write Rek's way, they're right over there. Their EMC code is gonna be around for all of scale and we can talk to them directly. I think I just need to go home. Yeah, no problem. Thank you so much. Thank you. Well thank you. The slides were really nice. She put a lot of work into making the transitions and, oh yeah. Yeah but this is so excellent because you can learn some Docker while you're doing this and that's like, it's super cool. You have something to show for all your work. Sorry, JVM's. It's running on Docker. So Docker is the one that's hosting your Docker container. Sorry. Okay. So yeah, you can deploy this Docker container. No, your virtual machine is running Docker which handles the container for you. Yeah, we're sorry if this is confusing but, oh yeah, but it doesn't work very well on Windows. So I guess it's better to remove Java from this whole thing and it's confusing a little bit. But the container just has all the dependencies that you need for your solution. Java is running inside of our container. And that's, yeah, yeah, sorry. It turtles all the way down like Docker on top of Docker on top of Docker. I guess you could do that. Oh, sorry. You can run Docker on Linux. Sorry, is it configurable you want to know? You can actually change that but if you go look in your Clash Data Directory, you can see it'll say like server.properties and it's like X file. And then in there it lists up for 255.65. So I think you can actually change that yourself if you want to do. Like if you just go look at that X file and mess with it, I'm sure there is an R. I think, yeah, I would just Google search it and hope that that can go with something. Yeah, it's Minecraft, or the Minecraft server jar is running inside of that Docker container. It's not a Minecraft client. The client is different from the server. Yeah. So, yes, exactly. Yeah, it works just like that. Because that server is vulnerable but the idea that we created Statement because we have this container data and we have the virtual machine data. And when this container dies, when we bring it up again, it'll still have the data from inside the virtual machine. Yeah, it's still in the same server. What on a single host anymore? Yeah, and that allows you to create multiple Minecraft servers and they can all throw data into the S and you can back it up in one convenient location and we can even switch out individual servers for as long as we keep referencing this data back in EDX, that's what we were getting at. Hopefully that answered your question. Yeah, thank you. Yeah, thank you so much. This is the third time we've done it once for EMC, once for Linux LA. Yeah, I was just like, you look so hilarious. Thanks for coming out again. Yeah, thank you. Oh, more Minecrafts. Minecraft changed versions. We updated the GitHub for info but basically the instructions are the same. But thanks for coming out again. We appreciate it, it's pretty profitable because it's pretty easy to get started, right? Worth exploring, right? It's fun.