 Good evening, everyone. We're very happy that you chose this hour session. It's last session of last day, but thank you for coming. So our topic is using CFCR to manage Kubernetes cluster. If you come here for using Kubo to manage Kubernetes clusters, don't freak out. Project has been renamed this September after we submit talks, so now the new name is Cloud Foundry Container Runtime. And a little bit about me. My name is Alexander Stinko. I work at Pivotal. I work on this project since it started. So last year in December, we started it, and we named it Kubo that time. And then this September, we got renamed, and it got split to open source, only part, Cloud Foundry Container Runtime and commercial part, Pivotal Container Service, which I'm working right now. Hi, I'm Brandon Nolan. I'm a principal software engineer at Pivotal. I joined Pivotal about six months ago and started working on the Kubo project. And I've currently transferred with Alexandra onto the PKS team. So here's our agenda for today. The demo is quite involved a lot, so I'm just going to get straight into it. So for those of you who do not know what CFCR is, it's the Cloud Foundry Container Runtime. It's an open source software project, which is part of Cloud Foundry, and allows the deployment of vanilla Kubernetes clusters using Bosch. The major, the initial founders of the project were Pivotal, Google, and VMware. And we've since been joined with a dedicated developer from Swisscom, but it is an open source project, and we are looking for contributors, people to commit issues, and PRs. So some of you may be familiar with the Cloud Foundry application runtime. And I think initially, when in Pivotal and within Cloud Foundry, people started to talk about supporting containers. There was a decision, well, is this the end of the application runtime? And it's not. The application runtime, we don't see it as an or, it's an and situation. So application runtime and container runtime still fit together. So we have a lot of Cloud Foundry customers who've been using the application runtime to deploy cloud native applications. But it's a very highly opinionated framework. They're very happy working within that framework. But it's not suited for all types of workloads. You have a lot of legacy applications that people don't want to port to new code. And then you have third-party applications that are delivered by containers, and you want to be able to run them in applications that have very complex networking or persistence requirements. And they don't fit within the highly opinionated framework of the application runtime. Why Kubernetes? So for the types of workloads that we're looking at, Kubernetes is the obvious choice. It's the de facto container orchestration engine. And then why part of Cloud Foundry? So we have customers that love the Cloud Foundry operator experience. They're very happy with how they can manage their elastic runtime application runtime instances within Cloud Foundry. And they want the same experience operating Cloud Foundry app or container runtime for Kubernetes. So they want a simple way to deploy, simple way to upgrade, a simple way to apply security patches, and a simple way to maintain the health of their VM infrastructure. So the problems we're trying to solve with CFCR. So I think we've all seen that the installation of Kubernetes has become pretty straightforward. We saw Kelsey the other day create a cluster from this phone. But day two is not so easy, particularly if you're managing your own Kubernetes clusters, that how do you upgrade those clusters? How do you deal with CVs on the underlying operating system? And what Kubernetes keeps your applications up and running? What keeps Kubernetes running? What's monitoring the VMs that your master's, nodes, and etcd is running on? And high availability doesn't come out of the box still with Kubernetes. It's left up to yourself under a lot of patterns. So as I said, we are powered by Bosch. So Bosch is a Cloud Foundry project. It's quite a mature project. I think it's been around for five years. It's based on the Google Borg idea of something that manages infrastructure. So we've got Borg, Kubernetes together. They want to be mates. And that's where we get the Cloud Foundry container runtime. So as I said, Bosch is a tool for release engineering, deployment, lifecycle management, and monitoring of these distributed systems. So teams can create their own software releases that are deployable in a very repeatable fashion. And the operators can deploy these software releases very consistently in a reproducible manager. They're going to do so very, very quickly. Via the CPI, the Cloud Provider Interface in Bosch, it has multi-IaaS support. For CFCR, we have support at the moment for four IaaS. It's GCP, AWS, OpenStack, and vSphere. And then it uses this OSM outsell infrastructure, which is a very base-level operating system, which I'll talk a little bit more later on, which helps to standardize deployments across all of these Cloud platforms. So operation teams find they're incredibly efficient when they're using Bosch. So for CFCR, CFCR is actually just a Bosch release of Kubernetes. So it's made up of three particular component parts, the NETCD release, the standard Docker release, which is already part of Cloud Foundry and OpenSource Docker Bosch release, and then the Kubernetes Bosch release, Kubo release. Each release is for a specific version of Kubernetes. The CFCR plan to keep up to date with each of the Kubernetes releases were currently on 184. It plans to track the Kubernetes version in use by GKE. One of the ideas when Google came on board was that they wanted something where people could run containers in a platform that was very, very similar to what was running in GKE. It has high availability in VM healing through Bosch and ED scaling of the VMs. And there's a very defined release upgrade pattern from Bosch. So we're going to jump into the demo now. And what I had intended to show was an initial installation. But as I said, initial installation is very easy. So I've actually pre-installed a Bosch deployment here. And I'll run through how I actually did that Bosch deployment. As you can see, it took about nine minutes to do. So we thought it might be kind of a little bit of a waste of time for the demo. So it's a bit awkward. So this is using the Bosch CLI where we're deploying a deployment called KubeCon. We're using an initial YAML, which is a deployment manifest which describes the Kubernetes cluster we want to deploy. In our particular instance, this Kubernetes cluster is a single SD node, two master nodes, and three worker nodes. It also takes a variable, which is the Kubernetes master host, which is a value that is just put on the SSL certificate that is serving the API server on both the master nodes. Bosch will try to deploy this. Since there's actually no change in deployment, it'll just say, oh, everything is the same. Your VM infrastructure looks like the deployment as it already is. So just a second. Now I have a script here where when I created the cluster, we use an internal thing within Bosch called credit, which, again, I'll talk a little bit more later on, to store the SSL certificate, the credentials. And this is just a script that extracts those bad out and uses kubectl config to set them. So we set a Kubernetes configuration. We see we've got our cluster here. So I've just split the screen here at the top. We've got a Kubernetes cluster. And then at the bottom, we just have something that's showing the version and the node. So three nodes, as I said. And I'm going to attempt now. No. Oh, gee, sorry. Yeah, could you just grab that for a second? For some reason, it's hard to type. It's hard to type. I'll put one hand here. So I'm just going to deploy an application here. Thank you. So here's a simple EngineX application in Kubernetes. We're going to deploy this. So we're going to kubectl apply. We're going to hope it rolls out, depending on the network now. Please stop. It may not roll out. Stop all Wi-Fi. But while that's rolling out, I'm going to start an upgrade of the version of Kubernetes. So a new version of Kubernetes has come out. And this particular reason, I'm going to upgrade to 1.8.2. So here's the operation file. So an operation file is just a patch mechanism that Bosch uses to patch a particular deployment manifest. You could go in and just edit the manifest, but I'm just selling this operation file for this particular reason. It's easier. Don't have to type and go into VI. So what I'm doing here is I'm taking the Kubo version, and I'm changing the Kubo version from 0.8.0, which was the release that had Kubernetes 1.801 to Kubo 0.8.1, which is the release that has Kubernetes 1.8.2. I'm running the same deploy command we had earlier on, where I'm targeting the same deployment, the same initial deployment YAML. But I have an operations file, which is my upgrade operations file. Bosch will now tell me that the difference between my manifests or what's going to happen is that you're going to upgrade the Kubo version from 0.8.0 to 0.8.1. So I'm going to start that off. And if the demo gods are with us, we're going to see that the timeout did fail, but maybe in the background it's actually. So it's container creating still. What I'm going to do is I'm going to run this command, which won't start at the moment, but in a few seconds after it's downloaded, the nginx container should start. And what that's going to do is it's going to pull the nginx server every five seconds and just print out the HTTP status and the date. Alex is going to take over now and show the capacity, how we scale a cluster. So he's a cluster in GCP. What we'll look for when we come back here is that the server version will upgrade to 1.8.2, and the node versions will start to roll to 1.8.2 as well. Yes, so let's see. Now on GCP, on BystronBox, and let's see what do I have. I have two deployments. So one I call production, Kubo, and one is staging, Kubo staging. So one production runs with four workers, staging with three workers, and this is fine. And let's see what happens. So I have QCTL running, checking something. And actually, I get one bonus demo for you because I will also delete a node on staging. So I choose this node, OK, gcloud delete, but gcloud GCP deletes VMs for three minutes. So it will start deleting. And also, I want to scale production because why not? Everyone wants to. I anticipate a huge load. I won't show any applications because each one's application is going to stop. It's not our job, but it's from 4 to 10. And then I do Bosch with deployment and Kubo, deploy production. And you see similar output at branch out, but I scale working assets from 4 to 10. Yeah, scale. And we'll be able to see it. I will think how to show the QCTL with nodes brought later. And for now, it will just run. So let's go back to Brandus node. So I can see that application is running. And you can see it's getting every second. It's getting new request, HTP OK. Everything is fine. Master is updating. And while master is updating, I will talk a little bit more about Bosch and all primitives that parts of the Bosch. So one thing that Bosch is based on. The summer's Kubernetes is based on work. And they are two very close systems. But Kubernetes manages containers. Bosch manages VMs. And they have similar primitives. And some names are the same. So like Kubernetes has deployments, Bosch has deployments. By trying in node containers, it's trying VMs. And so it's running system. The same as in Kubernetes. Deployment is running system. But here it's VM with this, some network configuration, some releases that's running on top of it, and some process running on top of VMs. And it's all mounted by Bosch. So Bosch checks if it's running. If it's not running, it will try to restart. If it's not restarting, it will try to restart VM. And does its magic. And while I was talking, I could see that master got upgraded to 1.8.2. And deployment is defined by deployment manifest that we saw. It's declarative. The same as in Kubernetes, it's declarative. You have to specify what you want to achieve. I want to achieve 10 nodes in my production cluster and some specific versions. It also has some VM layout. But it's cloud agnostic. So you don't have to say, for example, in AWS, it has a variability zone. And it's all principles across every IS. And bare metal as well. So you just say some Bosch primitives, network, availability zone. And it will do everything for you. And it runs version packages called releases. And it runs on top of base images called stem cells. And some parameters that you can configure how the series will run. For example, authorization mode for Kubernetes, RBAC, or AIBAC. And let's talk about releases. So releases, it's versioned packages which contains software binaries. And everything, actually, every binaries is required to start class. And it's not only software binaries, but also Docker releases, for example, for Kubernetes, like KubeDNS. And software packages, like Kubernetes itself. Then definition of properties for all these jobs. And then it self-contains. So at any point, you can download this big turbo and upload to any cloud. And if it doesn't have access to internet, it will still run correctly. Because it all has everything with it. And one more important thing is that you can actually rebuild it at any time. It has some definitions. And all those software packages blobs uploaded to the S3 or GCS. And you can rebuild at any time and reproduce this deployment. And a while was talking. No started to be updated. And schedule disabled. Not got codonned. So now it's starting. It started 1.0.1.8.2. And other nodes are getting updated. And let's take a quick look here. So we see that this VM died. Work is starting. And I will try to close this and see if it's coding. I will focus later on the keepsitl context part that will start VMs. And one more thing that I forgot to mention that, as you can see on production, I run 1.0.2. And here I run 1.0.8.4 on staging. And it's important because everything is self-contained. So you can run multiple versions with Bosch. No problems. It's Bosch keeps all the software packages. So if it has a system running and you scale it, it will use existing packages that you already uploaded. Let's go back to this. Come on. Still running. Still running. And jobs are part of the release. It describes single running service, like Kubelet, or Kubernetes API, or Docker. And it has some life cycle scripts, like pre-start script, and start-stop script, obviously. But Bosch expects that you can just kill minus nine any service and it will restart it. And drain scripts. So for example, for Kubelet, it's Cardon and Drain. Also, it's monitors using Monit. So most of the releases just use PID files. But you can use something more complicated with Monit. And it has some configuration, again, defined by property. So like kubeconfig, something else. And stem cell. So you use the same release to upload to any cloud or bare metal. And stem cell, the images are defined by the cloud, but impose it as a version. And each version has the same software on each cloud. So for example, you have stem cell 3, 4, 4, 5, and it has the same software, like base software core, on each cloud. And Bosch team manages it. And in case of CVE, they release update version in 48 hours, in up to 48 hours, which is they released quicker. So you got this new stem cell update immediately. And this leads to same manifest. As I said, manifest can be the same for each cloud, except some cloud-specific things like cloud provider for Kubernetes, but for other releases, it can be the same. And credit hub, which Brandon mentioned. So you need to stock some credentials. And some credentials, you don't even care. Some passwords, they're generated for a CD password. You don't care about this password for a CD. You just care that it's there. What are the certificates for Kubernetes for KubeStl? You don't care what is there. And what is the certificate for Kubernetes API? You care about CSR, but not about certificates. So credit hub manages all of it. It generates credentials for each new deployment. So you deploy new deployment, it will generate credentials. If you update deployment, it has the same credentials, which are defined in manifest. You can set credentials manually. And not only credentials, you can store everything. It's config storage. And you can easily rotate credentials. So you, for example, generate a certificate, and it will restart KubeStl certificate. It will reset only KubeNet and not Master, because Master doesn't care about certificate. It cares about CI. And one more thing that it's actually provided by a community face called Bosch. I don't know exact name, but it's Bosch config server. And it's possible to use your own implementation of this config server. So someone uses Svolt, but credit hub serves our things perfectly. And updating is still going. I see that it's scheduled and disabled. We'll wait for one minute. OK. And yeah, maybe I'll check quickly my GCP. Oh, no. But, yeah. Yeah, something went wrong. The important part is that you can always restart it, and the system is still running. It's some problem with scripts. I, some time out. Come on. I. Yes, quota. I forgot about quotas. Shouldn't have scales that much. So, sorry, sorry. But let's see. It's, class is still running. And, yeah, it still runs. It was able to run nine, but not 10. What a mistake. Sad, sad, but, I mean, in real life, you can. So as I said, you can proceed with it. Maybe delete some instances, request quotas, run in different zone, like, multiple ways. Yeah, demo guts, demo guts. I forgot to mention them. That was my mistake. OK, let's proceed. So, vision for class foundry container runtime. We focused on making operators happy. Kubernetes makes developers happy. They can develop their workloads, and no problems with it. But operators, they have some problems, and we want to make them happy. They want to, they don't need to think too much about day two. It's Bosch will match it, and it will be very simple. And it's simple. And we wanted to keep developers happy. We wanted vanilla Kubernetes. So one way was, maintain compatibility with GKE. If it can run on CFCR, it should be able to run on GKE. Unless you don't have load balancers on your ISN, or maybe change volume type, but in general, it should work. And then, when it's conformals was announced, we passed conformals test. We found one bug in conformals test, and there was some issue. It has been fixed since then. And Steam, right now, working on RISD-R10. It's not 1.0. They still have some issues in self-automation. And this is going to be RISD today, or maybe next week. I don't know. Exactly. So they worked on search catalog integration. And the important part was that they need to configure. Something was missing in Kubernetes configuration, maybe. And they just run in pipeline. So in pipeline, we deployed each startup of the cluster. We deployed search catalog, and see that everything deployed successfully. Then guarantees RISD upgrades. Again, add this to pipeline. So we can upgrade from 0.9 to 0.10. No problems. It's tested in pipeline. And test resurrection in pipeline as well. Bosch does it for you, but does it for everyone. But we want to test it and see that it's every commit passes and we hand-broke resurrection in this configured cluster. Should we come back to? You're back, maybe, June. Hey. Explore it. So we can see our upgrade of Kubernetes has completed. Our applications continued running. We're now at version 1.8.2. So we're going to do a very, very similar process. And we'll probably come up with where we're going to upgrade the stem cell this time. So as Alexander mentioned, the stem cell is kind of a base OS image. We have a 48-hour turnaround agreement, generally managed by Pivotal, where any CV will be patched within the 48-hour period. So our stem cell, we have an operations file here again, which, again, is just changing the version number stem cell. I've got the same upgrade operations file. And I'm going to apply both of these to the initial deployment YAML. So normally what would happen is when you make a change to a deployment, you'll save that old deployment locally. But just for the demo purposes here and for scripting purposes, I'm just applying the same operation files again and again and again. But when we look at what happens in the deployment, the only change between this deployment now is that the stem cell version has been upgraded. Now, that's going to go and roll all the bits of the components of the system with that new version of the OS image. There's nothing to really see in terms of the actual demonstration here, but it's just to show that that's the functionality you get. It will shut down node, then start node again with new OS image. Okay, let's go back to the future. So what CFCR focuses after 0.10, they focus on security, availability, so multi-AZ. I know that some customers run in multi-AZ setup, but the thing is that we want to run in pipeline again. We want to test it verify it. It works fine and all the affinity and affinity rules work, so we want to test it. And then define process about 48 hours CVE patches. So if some CVE happens in Kubernetes, in Docker, in HCD, we can patch it and update it and maybe how we can automate. And then define process about one week, when it's upgrade, so we can process, maybe patch it after mate, we'll see. And now commercial part, so if you want to pay our bills, commercial part and I have stickers with this. So peer-to-container service, when you're doing like in bigger organization, often small, you don't have one cluster, you have multiple clusters. And imagine them using YAML, it's not interesting. No one wants to be enterprise YAML developer. So we create PKS so you can manage multiple clusters. It gives on-demand provisioning, so we can provision new cluster with simple select amount and it just starts cluster and you can upgrade and do all the same that CFCR can do because it runs on top of open source Kubernetes, package in open source CFCR. And it's multi-cloud, for now we support GCP and vSphere and it's enterprise-raised, it has multiple VM integrations like SXT, Harbor, Verobs, Webfront and more, which didn't fit in one slide. You should have split multiple slides. And everything's still managed by Bosch, so no additional things, it still manages CFCR managed by Bosch and control access to, control beta access and no hots. And it will be available mid-December, so it was announced on spring one this week. So it will be available starting 15th of December. Ask VMware folks who are here or contact Piotl if you want to have access. And these are links, so documentation, code with CFCR, it's still called Kubo, we haven't renamed it. We know what should be the correct name for CFCR. And then I use Kubo deployment to deploy my cluster on GCP. I just modified it a little bit to change amount of instances. Then Slack, team is using this Slack, open source Slack and demo for the stats cluster on VirtualBox so you can start and see how it's working. And by the way, it's still upgrading because it needs to shut down VM start container, even if it's inside the VirtualBox. And outside, it's green, it's green. I mean, it's always green. So this is just something very important in terms of what Alexander was mentioning about particular things that are coming in the future. So particularly security, operation alerts, or multi AZ that unless we have this in a pipeline where it's fully automated and tested for every single release, we don't guarantee or don't say it's supported. So while people are running multi AZ, it's not in the pipeline yet. So that's why we're not saying that it's there. We have GCP, V-Sphere, AWS and OpenStack, some custom configurations for, oh no, okay, no one saw that. Yeah, demo gods, forgot to say it, does get demo gods. Okay, that's it. We open for questions and we leave in tomorrow so we will hand out after this talks. Yes, this is a question. Question about, you can, Diego, for people who don't know, it's container run times that might just call for the application runtime. Call for the application runtime can run together with call for the container runtime, but it's not required. So it's Diego still stays in application runtime. Container runtime runs Kubernetes and it can run separately. It can run together and reuse some features from call for the application. So no, so not exactly. Container runtime is completely independent. For the initial release container runtime we're operating a bring your own load balancer. So initially internally within the clusters we were deploying HA proxy load balancing in the demo that I've shown here. I'm running a HA proxy load balancer on my local machine and we're rooting. And then in GCP or V-Sphere, particularly with the latest version of NSXT, there'll be load balancer support there as well. So while you can, if you are running Cloud Foundry or Pivotal Cloud Foundry, you can use the TCP router or the Go router, but it's not a requirement. And there's no interdependencies. There's just additional hooks that allow you to use it if you want to use it. If you want to use it, you can use it. You don't have to use it. Yes? It's a separate add-on and it's so right now it's only for PQS and I'm not sure. Maybe you can get it somehow for, I used this Cloud Foundry container runtime, but right now it's only... So NSXT is a commercial product from VMware. The VMware folks might be able to give some more information about it. You could run it using container runtime, you could integrate it yourself. The default is Flannel. Flannel is a very open, flat network for containers. NSXT has a lot more enterprise-y type features around security, firewalling, and controlling that network of your containers internally. So that's where there's a lot of value out for the enterprise with NSXT. And NSXT integration part is not open source, I think. I don't know, but I think yes. Yes? Yes, VMware images, so it's it. So stem cells are built by... In general, you want to use stem cells built by Bosch team and for Amazon they use AMIs, for GCPs they use Google images, for Azure images, I don't know how they call it. For vSphere, like big turbos, I forgot. And for Opus, like again, big turbos. So they build it. If you really want, you can build your own stem cell, but if you really, really want, I don't know. I don't want to manage it and keep against CVs. So... What are the lessons there? Ubuntu and CentOS are supported by Bosch team. Then SUSE supports SUSE stem cell and Windows stem cell. Also, we have Windows stem cell, but we don't have Kubernetes for Windows yet because it's not like, I think it's in bad condition Windows. Yes? Okay. So that's good question. So let's, I will show my manifest. So I will show my manifest, so production, and I have this az in YAML, which is, for now, it's Z1. But I can do, if I would be really sure I would use Z2, Z3, Z4, and it's all defined in specific configuration that called CloudConfig. So I will show CloudConfig. I hope it doesn't have any secrets. It has my source account. Oh, no. So we have this types of VM. So it's like N1 standard, and I have az, which is az. It's US West 1. So I can have US West 1b, it will be Z2, and for example, Formazont will be different, for VMware, it will be different, and it's still called Z1 in my manifest, but in CloudConfig it will be different. Does that answer your question? So usually you just specify the az in the manifest that you specify, okay, I deploy it on multiple azs, and I don't care how they separate against it. Bosch does it for me, I don't care. It separates them, it checks, okay, can I deploy here? Usually it separates them evenly, but if it can deploy in some az, like Lemiso, it will deploy in different az more nodes. Does this answer your question? Good. Any more? Thank you, thank you. We have stickers, and we'll hang around for some time.