 Yeah, I'm going to DigitalOcean and I'm going to spin up three droplets. Now I tell you to go with the cheap five or ten dollar, this is if you're going to run it for an extended period of time and you don't want to spend a lot of money. I will be honest with you, these smaller droplets struggle. They do. The five dollar droplets, they're quite tiny, they actually struggle with us. But I'm cognizant that sometimes they talk to students and they don't have a lot of money and they just got fifteen dollars to run up the cluster and they're learning this thing at university so that's the only option they have. So if you can go with the larger ones and if you seriously want response time, go with the twenty dollar ones. So for this one I'm going to spin this up, I'm going to deploy the workload, I'm going to tear it down so I'm going to go with the twenty dollar droplets. So incidentally this method that we're doing the install with, thanks to Vincent, is going to become the default installer method for Kubernetes. Is this Kube ADM? No, thanks to me. No, well the great thing about Vincent is everything I think I discovered and I come into the channel I'm going, guys guess what happened Vincent? Yeah, five minutes ago I already posted that. Okay, so this method that we're doing the installer will become the default installation method for Kubernetes. You can do this on CentOS but there's more steps which is why I'm going with Ubuntu for now. So I'm going to go with these twenty dollar images. If you want to run this for a longer period of time, go with the smaller ones so it's cheaper for you. But I will tell you that if you have response time problems it's because they're struggling and if you install Htop and you just have a look at it, you'll see that like one of the worker nodes is maxed out at 100% and it's just really struggling with the workload. Singapore, it's so great to select Singapore as my data centre because I'm always in Australia and there's like nothing near me. Okay, so if you're following, just let me know. If you need me to, yeah. Doing quite a lot of work with it from the media side. Anybody know DigitalOcean here in Singapore? Or know anyone from DigitalOcean? They'll be cool to talk. I don't think they have a lot of stuff here. I think they use equinics in the centres. I want your business cards. Can you start out with doing loud stuff? I mean like before you have like the complicated Amazon requirements having DigitalOcean is pretty awesome. Just five seconds. Okay, a little more than 10 seconds. And who changed the world? I mean, love me. I always feel better after working in DigitalOcean. They make me... Yeah. My master doesn't want to... Come on, you can do it. I'm going to blame Honest Bee's network. Did anyone not get a sticker while I'm playing for time before I kill this guy and... Everyone get stickers? Yeah. Did you just say Cloud Foundry? Cloud Foundry... Too many opinions tonight. How do you get your workload out of Cloud Foundry? That's on top of... Yeah. So our friends at Cloud Foundry have... They developed their whole solution before... There was Kubernetes or Docker, right? And they built an elegant solution and even Red Hat with OpenShift were developing their own thing called Gears before Kubernetes came along. But Red Hat changed gears when... Gears changed gears. They switched over to Kubernetes. They ripped it out and replaced their orchestration engine. Now, Cloud Foundry, if you are... It's walled garden. Like as a pass, it's perfect if you're inside here. But again, my question is like... Once you've developed the apps in here, how do you get them out into the next one and move them? They actually had a very good talk on the next 17 about the service broker with Cloud Foundry. So there's a very good talk about the development that the Kubernetes team brings in. Yes. The idea there is that Cloud Foundry service broker is much more developed in terms of building up a catalog of all the other services that are not really containers. This could be, for example, a database as a service hole somewhere in your enterprise network. So by letting the container, Yaml, attach it, attach to a service published by a broker from a Cloud Foundry perspective, they somehow close these two worlds. There is no more gap. So they allow you to basically specify in your deployment profile, as part of the deployment, I want this system environment and I want this pod to attach to this MySQL or Postgres SQL service published by the service broker. So that's how they kind of start to come together. So it would be interesting, but it's the beginning of a big journey right now. It's not yet kind of like a solid answer there. Two months old. It's an interesting project, and I think it's got a lot of potential, especially if they're legacy environments, but environments that have to be able to abstract not only just connection strings for a database or anything else, but to be able to generate temporary login credentials or anything else. And in Bloomix, for example, it's vice versa, so they run actually Kubernetes underneath Cloud Foundry, so you have to install IBM Container extensions, ICEs, and then under your Cloud Foundry as a plugin, and then you're able to deploy onto, your build pack could become a container. So instead of like publishing into the native environment of Cloud Foundry, you can say, whatever, this is the build this is the build profile, but deploy it into this instance of a container. So it's also coming together in their world, but again, how it will transpire over the years. Cloud Foundry is getting fairly friendly with Kubernetes as a base layer because it's kind of become that utility service that other people can build on top of, and there's already open shared, there's already days, and so, you know, there is a pushing in some ways, but whether or not that eventually adds to the question. Yeah, it's like one year from here. Okay, so my master has gone somewhere nobody knows where. I've spun up four, and all I'm going to do is I'm going to use number four as mine, so if you're following along you've just got three, and they're numbered usually one, two, and three, and you've got the public IPs, and you've got your party session ready to connect in and start cutting and pasting from the gist page. Okay, everyone good? Okay, so I use Bobby Exterm, so these are the three that I'm going to use. This is going to be my master and my two worker nodes. Okay, so we've basically done this. We've basically just spun up SSH keys, the link for me, and I'm going with Ubuntu. You can do this on CentOS if you want. I've included the links up here, so this gist is basically a summary of some other documentation that's got better documentation that's official. So I've just pulled out the Ubuntu pieces for this. If you want to go see how this works, go grab this link and if you want to do this on CentOS or something else, who likes Raspberry Pi's? Kubernetes runs on Raspberry Pi's on and you can this very same build step that I'm busy showing you now, you can put Hippriot OS on your Raspberry Pi's and use Kubernetes and have a cluster going. Kubernetes cluster running on Raspberry Pi's. I don't know what workload we're going to put on it, but it will be cool. Okay. The next step is basically just setting up the operating system. So in this particular case, you just grab each one of these and I'm just going to do the multi-paced in all of them because it's common across all of them to prepare them. So it's the same set of steps to set it up. Okay. So basically I'm just grabbing these commands and I'm putting them across onto each of them. So it's the same set of steps. We may still get hot pizza. Yeah. Any Kubernetes one where they will try to support the more latest version of the docker. So they stack it pretty much at 1.12. And do you know why? Because docker isn't playing nicely. Yeah, I know. They introduced two armors in builds at one point, but actually my problem is that the standard was AP tables changes in the drop. And why did they do that? Did they tell you? No, they just did it. But if you look at what like right now docker.com is happening in America, right? And I always forget his full name. I know his Twitter handle. Yes. He just, he was doing, they were doing kind of a quick keynote at docker and they were like completely reversed. Like right now they're saying like docker is like being playing very nice for Kubernetes. They donated their docker daemon. Continuity. Continuity to the cloud native foundation and this Continuity is practically written to run Kubernetes. So they have this whole abstraction of the container runtime interface. So that like as James explained, you can run Rocket or you can run docker as the container runtime. And Kubernetes is just talking to the container runtime. And container D just now donated by docker is like perfectly written to that, towards that. So I don't know what's the status like. But I'm pretty sure that they will come their own binaries for No, no docker is split up. So they just see the community edition enterprise edition. No, no, no, more than that split off. I mean the docker solution itself has been split between a container daemon, a run c, a container runtime that purely run the container. And what's the other thing like the swarm, right? The swarm is basically additional API endpoints built in. But, yeah, they completely split off the parts. So you're able to just run container D and then hook it up with run c. So you're able to just like, you need this Kubler talk to that. Yeah. Okay. So just one of my opinion does work with docker 113. If you're using a later version of it. So you just need to be careful. So there are certified versions and there are versions that will work with you. Okay. So the operating system steps, these are just prep steps that you do on all of them. Okay. So just cut and paste these multi, multi paste them and do them on all the nodes. It's common on all of them. The very next thing you do is on the master. So just one of the nodes select one of them. I selected zero. You do the kube admin in it. And this is the initialization command. Now, while they were talking, I kicked it off because it takes a bit of time to do. But at the end of it, what it's going to do is it's going to complete and it's going to spit out this token join command. This join is what you run on the worker nodes to find the master. And that is the IP address that it's going to find it on. Okay. So don't don't run this on the worker nodes yet. We've got to do some overlay networking and some other stuff on the master before you do this. So just grab this command and put it somewhere. We'll come back to it. So we've done that. We've got that join command. So grab that. But first we've got to finish doing these things. So this was interesting. To do this last week, I said, this took me two weeks to clean this up because 1.6 RBAC is now default turned on in Kube ADM. And I couldn't log in. Because RBAC is turned on. So I did some work for you and you've got to do these three commands which you won't do in production to log in later. And this must be done as root on the master. Only the master. Okay. I don't know if I should talk while you're cleaning the commands. Is it interesting? I think it is. If you've been looking at Kube ADM with the security lead of Docker, there's been a lot of tweets about toning Docker clusters. Basically where people are able to get the token that's published by Kubernetes by default inside a container, inside a pod. There's a default token published by Kubernetes. And using that, you can take full control of the cluster. So prior to RBAC, prior to 1.6, it's actually to get control of an API server of a Kubernetes cluster and spin off your own workloads and totally take control of the cluster. So it's very important this RBAC thing. It's very important that it's coming to it. Right on time. So these three commands were basically because of RBAC that came in 1.6. The last thing that we're going to do is we're going to install the overlay network. Now there's a couple of options here for you. I go with weave works because there's an optional category I give you down there. So because we all work for managers and managers need to see things to believe them. I believe in UIs. As many UIs as you can throw at management we'll get them across the line because seeing is believing. So I installed the weave works because there's an optional category further on where weave works will give you a UI which is really quite nice for seeing the microservices application. It actually breaks it apart quite nicely for you and gives you the links between the nodes. So if you're a fan of Flannel what do you guys use? Are you Flannel? No, which network? Which overlay? So there's a couple. Calico, Canal, Flannel Weave Works. There's a couple of overlay networks but I'm just going with. So this uses a primitive I'll talk about now. It's called a demon set. A demon set is now you've got your cluster ring and you want one particular container. It can be a logging container it can be a monitoring container. In this case it's the overlay network container. You want one container to run on every single node. This is an example of a demon set. It's where as an operations person you've got a requirement to use case to just run one container everywhere. You use a demon set and this is using a demon set to put the Weave Works container out on everything. So cut and paste this guy and this is why I said you don't do the join yet. Because we've got to set up the networking we've got to set up the security This new coming up which is Romana or VIPs which is virtual IPs that you're going to run directly. So which is high performance basically low latency. Romana, I heard about it. It's also a way of managing stuff that thing. So we just give you like more granular on the MAC address. And that's where that name jumps out. Demon set. So if you run any of these commands and you see demon set pop out. What they were asking you to do is they want one container and only one type of container to run everywhere within the cluster. And this is great because if you add a node later on, Kubernetes will see this node. It'll go it doesn't have that particular container and it'll add it to you. So you can just carry on adding nodes and Kube will go and add these demon sets to it. So in this case the network will go is there anything else I need to know there? No. Okay. So the master's done. It's all the prereq steps and the RBAC steps and we've set up the overlay networking. The next thing quickly do the dashboard because if you can't see what you're doing. Okay. Is there anybody with with you? Like anybody stuck somewhere? Nope. Everybody's gone. Okay. I'm going to quickly I'm going to skip the dashboard and I'm going to go to the join because we've got the join command over here. So you grab this command you grab this and what you do now is you go to the two worker nodes and you just join. Okay. You go back to the master and your first command Kube CTL get nodes. We're done. You can. You can. Okay. So one of the worker nodes has come up. Okay. Now we've built one master and two worker nodes in Kube ADM by default it puts what they call a taint on the master and a taint basically means that no work will ever be provisioned to the master. You don't want the master doing application work. That's what the workers exist for. Okay. So by default if for some reason you deploy your application and you can't for your life figure out I've got three nodes. One of them is a master but I wanted to do work. The master is not going to do work. It's currently got labels set up and they call them taints which basically excludes it from any workload. Okay. So whenever you look if once you go pull this apart on the weekend or whenever you do it and you actually go in and you do do it and you go through. So also another thing that I had to get my head around is that Docker still exists in this whole ecosystem that we're doing now. Right. All that we've done is we've put this lifecycle management on top of Docker. All your Docker commands that you know the logs, Docker start, Docker stop. All the Docker commands will still work. So in this particular case I'm just doing a PS. I'm listing all the containers that are running and Docker is responding to me with everything that's running on it. Okay. So application workload won't run on the master. It actually tells you on the main page if you want to provision work on to this there's several commands to remove the taints or remove the labels and work will come to it. This took me a day to figure out. I had another client who wanted to use his master as a worker and clients are always right. So I had to make it do that. You see this form basically doesn't have a in theory to create the master but the master is basically this form you can run is the form has the concept of a master or they all Yeah. So the same thing when you created Docker swarm cluster it had managers multiple managers and basically they took the XCD you know implementation of raft in the managers and I think by default the master is also not schedulable and you can also just set up a label that you can schedule workloads on top of it. It's exactly the same concept. You can set the labels on it because I remember running a demo I think maybe by default is enabled to run because I was running a demo of Docker swarm 1.13 and where I was showing some of the features and one of the things was it didn't work because the the master was sorry I'm going to go back again the thing was we had the boat app and the boat app has a database and the database schedule had a requisite that it had to run on the master and when I run it by default it didn't spin up the database wasn't getting scheduled so I had to actually go to the master and enable scheduling to make the whole thing work so I'm pretty sure that actually by default it doesn't run. No because I run it by default 3 and I didn't put any workers I put all the masters and between them they just schedule if you have a small swarm everybody is a master basically you can do that if you want you can schedule it but this is the resiliency in my case I created a cluster with one master tool to workers and I was running the boat app which runs Redis and the database and the database has a volume on the node and the master takes care of the database on the master but by default it wasn't running like it was not getting scheduled and I had to actually make the master but anyway it's working exactly the same way it doesn't matter if it's the default or not but why is there any way to run the console instead of making CDs? Yes as a back-end for Kubernetes? No, not... okay so the other thing I want to tell you about Kubernetes is it's super-pluggable so the API is like the only mandatory piece of the Kubernetes solution so if you're a fan of HashiCorp and you want to use console service discovery you can actually replace the service discovery that comes with Kubernetes with HashiCorp same with Vault so Kubernetes has got its own way of managing secrets and configs but if you're a HashiCorp user and you like Vault you can use that it's supposed to be very pluggable even the scheduler can be replaced with another third-party I'm going to use HashiCorp again you can figure out I'm a fan of HashiCorp's stuff so you can use the scheduler I don't know if it's called Nomad or whatever the HashiCorp is so if you've got those components already in your organization and you're using them you should be able to plug them in to use it the only thing, as Vincent said the only thing which is completely mandatory is the master, the API service the only mandatory component that you must have and there's an HCD cluster behind that that's doing the key value store is there any reason why they chose HCD because I remember two or three years ago the SWAR was using the console the SWAR was using the HCD version one at a time so HCDs come quite far compared to GRPC as they protocol internally now so it scales and it's much quicker than it was before version three so I don't know why not console but I remember SWARM Docker SWARM version one before SWARM mode was using console actually was using live KV like an abstraction of a key value store so you could talk to a library key value store and then behind it could be zookeeper or console or you could store it directly remember I was using it directly with the console sorry did you get past your problem now no rebuilding so actually at that point I couldn't connect to an API server so it was just for some reason not responding the ADE port and master are you on digital ocean yeah it was basically not responding there was no service deployed like what that get was so I'm just okay is everybody else roughly at the same point those that were able to follow so all I've done is I've installed htop on the two worker nodes to show the load and I'm back on the master so if you've run your first kubinities command kubectl get nodes kubectl can be pronounced in three ways I've found out in the universe it can be kubectl I call it kubectl and you can call it kubectl so whichever one you want to call this main way of interfacing with kubinities choose whichever way you like and you can make it yours so you've done kubectl get nodes you should have the master they're all running 161 which is quite recent and we've got the two worker nodes so the next thing I'm going to do is I'm going to do the dashboard and the reason I'm going to do the dashboard is because of the mind trickery that I had to figure out in how to get the UI out so grab this this is exactly an example of the manifest file that I was explaining kubectl create and input it to a file and this kubinities dashboard if you actually go in and edit it you'll see it is actually an edit it so what I'm doing now is I'm actually taking that manifest file I'm grabbing it from the origin page and I'm basically giving it to my cluster I'm saying install this for me please so on the master only okay it's done so remember I talked to you about that primitive the service this is the service this kubinities dashboard so the containers running the dashboard are running in the background now but there's a service that's fronting it so if you want to interact with the dashboard actually talk to the service the service will proximity to the correct container that's running it and kub is keeping the whole thing alive this was also a command I had to put in for obac 1.6 or I couldn't log into the dashboard it talks to the kubinities API again so I actually put this in here for you so before this you just would do this and then get in but I found out you have to actually create this user for obac changes now the way to figure out where this is running to get your first UI up is basically we take a worker node and we take the service port now you'll only do this in this lab exercise you will never do this in production you know the worker nodes that we deployed we've got the master and the two worker nodes and this is the master the worker nodes have got an IP address go get the IP address of any one of these worker nodes we need the IP address of this then we're going to execute a kub command over here and it's going to throw out a high range port number above 32,000 to get to the UI I need that port and I need an IP from here so it's going to be the IP address of a worker plus this high port range and when you put that together into your browser the UI is going to come back don't ask me to explain why or how yeah but this is okay if there's an easier way tell me I know the swarm was using the internal the proxy right the routing you can apply any port any IP address of the cluster for the machine that you put internally yeah you don't use this so we're doing this for this lab exercise you would actually use something like an Ingress Controller or a load balancer but assume they're just students and they just want to get the UI to come up okay and because we're exposing we're exposing the service on one of the node ports so a node port is the worker that's why I asked you to get the IP address of one of the workers so the first thing I'm going to do I'm going to go to one of my workers and I'm just going to grep for the Ethernet so on one of the worker nodes I'm grabbing the IP of one of the workers and I'm doing kube control get service in that namespace I'll quickly talk about namespaces oh that's a worker got to do it on the okay that high range port number it's going to be different for everybody okay so mine will not be the same as yours so you grab that high port range okay so see what I did there so now I grab that high port range node port and I throw it into a browser this is dynamic this is dynamic without modifying any of this I want them to be able to get the latest version that the projects are working on pull them in and maybe the pain of going through this will help them understand some of the stuff so did everybody get a UI yes I'm not such a bad teacher after all okay let's quickly talk thanks Hunter you've just volunteered for the next talk thanks champ that's a very valuable lesson thanks for volunteering there was another question I was going to ask what kind of frequency makes sense for this kind of user group once a month twice a month how much of this can you there's a lot of user groups in Singapore a lot of interesting user groups precious a lot of content that's out there so I'm just trying to get a feel for once a month by monthly by monthly based on VMAX doing VMAX for Benjamin Torch you've probably heard about this guy you heard about Benjamin yeah I've heard his name we were doing VMAX a couple of years ago thanks on the third or fourth occasion if you do it monthly so it's rather six weeks or eight weeks okay so thanks Hunter you can start getting ready and I heard it's being hosted at Google haha okay so your UI has come up so if you've got this give yourselves a hand well done if you've got them up I've given you the instructions to go tear this down build it up on your own on your own time and get into the UI these are the most important things you've got it up, you've prepared it you've deployed it and you've got the UI if you quickly go through this name spaces name spaces are this awesome cool function inside Kubernetes for doing soft isolation between applications okay you can run on one cluster, you can run three environments dev, test and prod using name spaces to isolate them so that the applications running in the name spaces can't see the other name spaces okay so from a utilization point of view I've given you X amount of money for hardware but now I'm using name spaces to carve up hardware and because I'm using manifest files to drive the applications I do test test test test move to the next environment on the same cluster okay you'll never do this right you'll actually only have your test dev user acceptance testing and your name spaces there but your prod will be its own set of hardware okay but name spaces are basically isolation that you can use so there's nothing viewed here because I'm in the default name space so I would have to do so you do the click down here and you see these default name spaces that Kubernetes created for system so what we're going to do now is almost there I can see Vincent's pitching we're going to deploy a microservices application onto the search you can deploy your first app and see what it looks like inside the UI okay so it's called Sockshop it's a microservices it's an example of a microservices application so what you do is on the master and the master only so I just explained to you what name spaces are so in this case I'm creating a name space called Sockshop and this microservices app is going to go and nobody's taking pictures of me and put it on Twitter I've just realized the camera came out I get I get paid for Twitter I do I like the up links okay so we got to create a name space for Sockshop so back on the master so again this is grabbing the file and inside the name space of Sockshop I'm applying the file which I'm going to get over here demo okay all the primitives I spoke about on the whiteboard services so here you see how they do a deployment but then they create a service in front of it so the pod that wants to run the app is sitting behind a service that users will interact with the service before getting to the pod and if you're going to do upgrades you upgrade the pod and you do rolling updates to the back end but the service is stable in this particular case I think they actually do some volumes some of these have volumes that they need to be created okay no these are just they're just doing like empty volumes scratch spaces for the containers but thanks for volunteering to do a talk on persistent volumes I think once we talk about persistence who? DLMD that's very nice I would just talk about that I'll do it over in the past okay so here's the next most important command you're ever going to learn is get pods okay so get pods in the name space so when you talk to the cluster you actually tell it inside which name space you want to look if I gave if I did this without the name space without sock shop I understand it's a name space right? yeah it is a short version for name space this is defined in YAML this one is created when you import it in YAML file or you don't create it separately I created it separately yeah but you can create it in the manifest file you can so by the way you can build up your whole micro services app with your name space definition your pod definitions of services into this giant file the sock shop is a really good example of like a really complex deployment that has been put into a single file except the name space wasn't there on the YAML side if you want to have a clean structural code instead of having one huge file like file formation or whatever can you structure the YAML file to read from another YAML file to structure the code you can split up YAML files into multiple files and then just record including the directory if you want to do nested YAML files not in default you can use things like Helm to match the deployment configurations which can do equal okay I quickly want to show you so I'm doing kube control get pods and inside kube system these are all system related pods that are running kubernetes if I do sock shop I'm looking in a different one and you can see these are all the micro services that I've asked to run so if we go back to our UI kubernetes dashboard and I change my name space to refresh okay so what happens now is we've deployed the micro services app and inside the UI I've changed my name space to sock shop and I can see all the deployments that have run you can see the images there's no demon sets there's no stateful there's no jobs there's no pods the services which front end this are over here so to get to those pods you come through these services that are over here how much time have we got because I can show I can getting into the front end of this getting into the the port you basically you basically do the same thing you get the IP address you do sock shop front end and you take the IP and the I'm just gonna do it you already have an IP thank you Vincent can you edit out sarcasm from this video or not it was my sarcasm okay here's my high port this is an example where they hard coded it it will always be 3001 so in this manifest file remember this now they've actually hard coded the port that it'll be on so if you take that guy and you just do that I'm not a fan of edge so Devin I'm gonna tell you this you make chrome and you make kubernetes but you don't have the kubernetes UI inside chrome I have to run firefox or edge and I have created an application okay you've got your app up and this works you can go in I've left the user IDs these are the user IDs that are valid for the app to login as different users this app you can you can so it's fully functional you can play around with this the whole point here is I want you to build a cluster on coop understand the primitives that it gives you take an application and deploy it on top of it and then run it and this is yours to pull apart now in your own time go and have a look at how the master is use the UIs to basically understand go figure out what these primitives are that kubernetes is offering you and what each one is doing and why is it doing it so hopefully I'm wrapping down you learned something today you know more about kubernetes now than when you came into this session and my south african accent is now I have a question on auto scale on google or on azure because they integrated the kubernetes let's say if I'm running open stack or amazon amazon has the auto scale in module you just scale these parameters you'll memory whatever so and as well the kubernetes have this you can specify parameters how do you scale it in how do you marry them both any way to marry them so I can only tell you I've watched people give talks about using prometheus to decide when to actually do the scaling just looking at the utilization of the node so we're getting into monitoring across in a cloud native stack is a bit more complicated than in the old world where you just watched the vm and if the vm got hot you knew that you were facing a problem here so something that you need to understand about this is your workload can live anywhere in the stack so traditional monitoring kind of goes out the window because it's not pinned to a specific host right if I the services available but if I redeploy the container it can deploy anywhere that kubernetes decides the scheduler decides where to put it so monitoring becomes a problem which is why prometheus is interesting because prometheus is designed to actually monitor these cloud natives so from a scaling point of view I'm going to tell you that probably watching the host node is not a good indicator that you need to scale because the workload on top of it is transient it could be there now it could be not there tomorrow when you do a rolling update basically you integrate from a deal so long that functions which is in instances and I think the if you're talking about nanda you talk about amazon you actually have autoscalers for people that are integrating with amazon with amazon api so you can autoscaler itself yes elastic scaling we're not using it at the moment but we're looking at it but yes it's possible to deploy a kubernetes pod that talks to the kubernetes api to find out the utilization of the nodes and looks and autoscales to group to have more work with I think they actually alpha there was just an alpha release for like the ability for replica controller to add pods I think this is the very very fresh one like two month old you also have a pod autoscaler so they would do the pod autoscaling based on cpu ram and external events so this is something that they pushed like very very there's a there's a few atoms with horizontal and scaling as well horizontal and scaling I was talking about the actual amazon autoscaling group scaling out so adding more resources to the cluster to the node nodes adding more nodes and the other option is obviously to like scale out the pods which who are we I mean the company that with the philips we they have these lights they have their internet of things who okay I can't see it so they actually build their own autoscaler that looks at the number of connections to a pod and then scales out the number of the pods so they have like custom autoscalers I don't think if they I don't know if they put that open source but they definitely talk about it so yeah but this autoscaling of pods is very big as well and on the nodes yeah it's a platform specific thing it's I think a little bit easier just from the regular AWS or Google platform standpoint just to look at the nodes actual utilization in Google Kubernetes engine it doesn't automatically right when you create the cluster you can say autoscale automatically from A add more nodes when there needs to be more you define like from 2 so it's all really cool I think when you were saying play with the cluster I thought about killing a node but maybe the data is no no no I will stop you from having pizza because I can okay that guy's gone so what what Vincent's basically talking about is I told you this is life cycle management for your containers so one of the most important things is the ability to recover and keep running so that you don't have to get called in to do work so what we're going to do now is very carefully destroy one of these whole nodes and then watch the UI to make sure that the sock shop continues to run okay let's kill it let's see what happens we have a single master come on take the sticker away from that guy that's when your question is too intelligent yes this is a you probably can not test it if if I kill it completely and it continues to run that is a better show than rebooting it because rebooting is like kind of cheating but we'll find out but you're connecting to the work node directly right? yes so if you kill that work node it's okay you just connect to the other work node you just grab the so what's going to happen is the guy's gone okay so one of them is completely gone I have a recruit print on the server for example if I run Kubernetes if I would run just a container and in the cluster with Kubernetes all these dependencies what would be the memory footprint of the master? of the work node yes you can run it on a Raspberry Pi on IOS and networking any additional stress? very much it shouldn't be taking up a significant amount because you want your applications to be using all that so it's a like from IOS it's a copy on the right so it continues like this snapshots are from Docker right so it's a copy on the right it's not redirecting so it's continuously kind of puts together the new thing that it's in persistence state right in persistence so if you're writing something inside the container it will be a contiguous very very spindle aligned right so no need to worry about it and this is the beauty this is what separates containers from virtualization you don't need like any kind of shared hybrid mode stand just run it within Pi 3 so we killed one of the worker nodes completely but the app is still up it's struggling trust me I built a Raspberry Pi tower that I'm running Kubon and then I do engine X containers across them and then I actually pull out the power to them and you actually just watch it gracefully move the workload to the next one so I do know it works one of the next talks I want to do is actually bring it in and then do the live fire we pull out the power and do it any questions? everything be very careful kill the namespace you kill everything in it it's like rm minus r recursive namespace kills everything yep yeah on the other way it's a great way to clean out your system namespace gone docker registry server is not connected most are not connecting to the docker registry when the pods don't get populated with containers the first run or subsequent runs it pulls and then it runs it doesn't need the registry for anything unless you want to deploy a new manifest that does an update of the container and if it can't reach it it's not going to do anything because it couldn't pull anything across it continues to run the old version it continues to run the old version but tell you that it couldn't connect if you add new host to the cluster that's an expression it won't so each node is dependent on its own docker local cache but if you run a caching docker registry in the cluster then you can cut off your external connectivity and have fully replicated the docker container to modify it we can make a simple caching so it works it's like a squeed proxy yeah, like a squeed proxy or an architecture which is basically caching the data anyway so it's an architecture did your thing work this time? yeah, I've seen the socks so right now you brought down one of the workers so let's say you have a new deploy additional like two or three workers service so once this workers get back up we do the rest of the thing automatically the master's got a scheduler that tries to figure out where's the best place and because the new node has come up and it's empty the scheduler will basically go this guy's empty and it'll automatically start putting workload onto it so it's something that you don't worry about anymore because kube will figure out where to put the stuff for you that's why I said you won't actually even know which host your app is running on anymore because one day it could be here the next day when you do an update it could be running over there because kube is deciding an operating system for your data center the more you start to think about like that the more it starts to make sense what is the limit? limit of? 300 containers 500 containers is it any limit? it's at 5,000 so this current version of kube is either sitting at 5,000 or 10,000 nodes nodes 5,000 nodes which will basically happen for itself so it's like almost 50% of what this super company does you don't build cells more than 10,000 so one cluster well that's dependent on the amount of RAM or memory across 5,000 nodes but they're not they're all the same but per machine the machine it depends on which memory the machine has because each... did you detect any memory for example limit? if for example it's a machine that's outside the limit and the kube is still trying to oh no, no so it has some monitoring on the memory there is no memory over commit it's not like VMware you can't have memory pressure because you containerize you basically are forced to present a segment of RAM to the application inside a container this is the 30 year old containerization concept right? so there is no thing like memory mapping or memory sharing or memory compression or whatever like we got used to for VMware there's nothing like that so it's hardcore you basically leak 10 megs or like 20 megs out of like RAM you cannot put another container so it's very very precise but at the same time you will have very very different containers right? so some of them consume small amount of RAM this is where Kubernetes will place all those it's like dynamic resource scheduler in VMware but again there is no thing like memory pressure or CPU like CPU aside there is no thing like memory compression or memory sharing basically what you're saying is as long as you have the capacity on the server it'll run correct that's why the auto-scaler design should be like similar nodes you cannot add instances of different types if you want to do like an auto-scaler for the nodes you have to pick up okay what I'm able to pay for what will my boss not fire me for basically if you add like 10 superpower nodes on the spike of the capacity and then you don't destroy them right? so you cannot have like small small large large large because it's not the best practice you have to have like a single layer 8 core 96 gigs nodes or something like that and then it will be very homogenous you can actually do you can actually have special nodes with labels like if you need to have a couple of nodes that have GPUs for like AI of course you can have nodes within your cluster that are labels that they have and then when you schedule your workload you tag it that it needs to have a node with GPUs I'm not saying that right now the support of GPUs is maybe still an alpha on Kubernetes but it's possible to have different types of nodes I would just deploy a separate cluster for that it's basically designed to one enterprise right one of the problems with companies in the time you acquire some hardware once you want to switch to the Kubernetes you deal with the different types of the hardware so you don't have a choice buy R6 David you can have mixed nodes in that environment we do for we do without being able to deploy it to Kubernetes we have you know blade servers so if anyone's familiar with the HGP Mercury I'm sure there's a few people here for the NEC servers but we also have so we have microblades and then we have large scale storage servers which we have in our full cluster and we tag particular nodes that will handle certain workloads can you let the target to the namespace for example say you add these nodes to the namespace and then you know that your application in this namespace will run all these things yeah so you're talking about labels so I had to do that but you can do it in a box like this sorry I forgot I have a call no worries thanks yeah so that's a whole other talk is labels and how everything in kube is labeled so well that's something else we got to talk about sorry Devin you had a question yeah actually the current problems we are facing is that we are having a lot of end-of-the-stripes and we are moving towards to other Kubernetes so what would be the best option because we are also thinking to integrate with azure block storage so storage is is a talk completely on its own so I'd save that for the pizza seriously yes sir so what if the master node goes down does it auto recover or so you deploy your masters in HA and you put them in front of a service so that if they go down that the service will select the next master in production you never do single master this is purely for play in production you always deploy your masters in HA and usually three so what are you running threes five because we did some manual operations on the masters actually it's not there's two separate components here one is the fcd cluster and two is the API servers first case because we wanted to save resources we put our fcd together with our master which means that we are running a highly available fcd cluster of five and that's actually the only component that we want to have like highly available well actually the master actually we could do like a three node fcd cluster two node master cluster and then a couple of like 10 workers but we are running five fcd nodes because we were doing manual operations on it and if we by mistake if you have five you can support up to two node failure you still have like three majority along the cluster so if you do three you can have one node failure and we had three and we brought down one node and then by mistake we took out another node and consensus was gone and we didn't do an automatic recovery from snap up or anything like that so we lost our consistent state that way so that's why we were running five because we were late we just went five so for example let's say even for three nodes if one node go down you can one of the boards one of the decalmy master basically to program it so the masters like the api servers they actually run multiple components the api server itself is completely stateless and you can have like it doesn't matter where it's running like if you have two masters the api server will run on both and you can load balance across there are controller like the scheduler maintaining or changing state so when the scheduler needs to make a scheduling decision he's changing the state of the cluster so you can only have one and they can do master election using scb so if you have two nodes running master services then you can make sure that the scheduler if that one node stops responding then the other scheduler will take the lead and starts making scheduling decisions that's automatic you don't need to like program that in the controller looks are also the controller manager is also a single the tricky part then just becomes how to identify where my api server is gone so you usually have to have a lot of process well yeah if you consistent api yes but I mean if you have two nodes of api servers if you put the node balancer you always talk to your api servers your controller scheduler and your scheduler you never talk to they just switch between them but yeah you do need to have a load balancer to make sure that you're talking to the master api server that's alive yeah I think one more question anyway there is none it's all goodness your life will become infinitely better your LinkedIn profile will get so many hits you won't believe it and now we're talking right you've got nothing to if you if you're an ops person and you've come from a history of ops and struggled with life is developers going like this direction and ops is going that direction okay go slow go fast this thing is the first thing that this is the the most perfect perfect operations platform I've come across in my life okay it allows you to move at the same speed as the developers really because there's this layer of abstraction and whether they hurt themselves you can roll back and you can move forward really quickly with your apps so all of a sudden instead of this two speed economy running inside your company it's like one speed and you're just running with them the whole time and it works yeah because the separation with Google is not yet maybe a logging team right a little security team it's every point of those teams can run independent of one another push their updates as they need versus everyone you can even be on that same page but you it's a very legacy system you're handling as well as everything coming in because there's huge database and different matching servers well you saw Honest Bee was hosting this tonight because they're looking specifically for these skills okay this is this is what they're looking at what is the legacy system it's like 30 years ago like you look a ball okay it says whatever you write you think it's legacy tomorrow yeah okay so this pizza next door thank you everyone for coming and probably see you in 6 weeks thanks guys