 We're gonna start on time just because I have a talk right after this so I have to make sure I don't go over because I've Got to run to another floor. You can use a clicker. Okay So this is us I'm Steve long I live in Los Angeles I Have been active in Kubernetes since 2015 at the beginning I worked mostly on storage but Right now. I'm a co-chair with Fabio on the VMware SIG which is charged with maintaining the cloud provider for running Kubernetes on top of the vSphere hypervisor I'm also lead of the Kubernetes IoT and Edge working group And my name is Fabio Rapacelli. I work for VMware as well. I'm an engineer working Kubernetes upstream so my day job is basically to make sure that Kubernetes is a health project and I am you know, we're able to contribute new features and so on so I'm mostly active in SIG cloud provider obviously SIG VMware. I'm a co-chair with with Steve and SIG class lifecycle and don't forget you took a fence at me saying Italy and not your city So tell everyone what city you're from so I live in Italy But I live in a very small town on the east coast very small. So it's far from everything. It's called Chervia So I'll just give a Steven hard time because he says Los Angeles instead of United States, but you know Well, if I said Chervia wasn't they wouldn't know where that was. Yeah, exactly So today we're going to talk about Why the Kubernetes cloud provider? Not just the VMware one But all of them are moving out of tree and what that means then we're going to go into a deep dive of Installing and configuring the new out of tree cloud provider along with CSI storage And Fabio is going to give a demo of that and address the migration options available for people Who are already using the entry version? So the original architecture of Kubernetes Started with what are called the cloud providers built right into Kubernetes itself this stuff was like built into the fabric as a type mesh and I think that You know in the early days Maybe they didn't see the success of Kubernetes and the the ultimate number of platforms that people would go to What a cloud provider does if you don't know this already is that it's the abstraction layer That makes Kubernetes portable so that you can take Kubernetes and have the identical experience Whether you're running on an Amazon public cloud a Google public cloud Azure or on your own on-prem hardware with or without a hypervisor and This cloud provider sits there and makes it contains the code that makes everything uniform So that when you write or deploy apps the apps Aren't even able to tell where they're running Unless you do something stupid and intentionally try to pierce these layers, but the premise is that in order to be cloud native the app should run identically regardless of where they're hosted And the cloud provider layer is what does this mapping to whatever the underlying infrastructure actually is So these eight cloud providers are the ones that are built into Kubernetes Automatically whether you like it or not as a version 114 now It turns out that having these cloud providers built into Kubernetes It's easy if you're on one of those eight, but there are some problems these These cloud providers if code in the cloud provider has a bug Or needs a security patch They can't be they can't actually be published independent of a full Kubernetes release so and For any given user you're only going to be using one of these cloud providers at a time Yet the binary distribution of Kubernetes includes them all so it's bloated unnecessarily The code in these cloud providers runs as a privileged component of Kubernetes So this can present security and stability of risks because if something goes wrong in there it can affect key it can bleed over into the key components of Kubernetes itself and Finally the steering committee and the architects have come to the conclusion that really Kubernetes should just be the standardized orchestration kernel or code and These drivers or cloud providers should be completely independently Maintainable by people familiar with that platform Even free to declare independent release cycles so that if we on the vSphere cloud provider wanted to get a release out Two weeks after Kubernetes itself we'd have the freedom to do that if The code isn't tightly coupled so this move of The cloud writers out of the tree Isn't really urgent meaning it's The standard Kubernetes deprecation policy applies meaning users get a minimum of a one-year notice that this is happening and aren't forced to March overnight to the new one So there will be a period when both are available and that we're in that period right now where the Entry cloud provider for vSphere is still alive and well and present and supported But the auditory version is also there There hasn't been a firm decision on when the cutoff date is going to be but you will by Kubernetes policy get at least a year's notice if right now we're looking at 1.20 as the release where everything will be pulled from entry. So by the release 120 which will happen approximately around late 2020 we'll have All the Kubernetes all the older Kubernetes entry provider removed from the Kubernetes core So the way this move is structured It's based on a number of these caps these Kubernetes enhancement proposals I'm not going to go into the details, but the link. I'll give you a Shorten link to this whole deck at the end and you can go look at those caps that have Detailed designs as to what's going on with this move out of tree one thing that is happening in conjunction with the cloud provider is for exactly the same reasons there are storage drivers that were built into kubernetes and at one time they were monolithic and those are moving out of tree as well and If you've got a cloud provider that's integrated with storage and that that's the case with the vSphere cloud provider where We need an abstraction layer not just to the compute infrastructure And things that go along with it, but also to storage The storage provider Is architecture is changing to use something called CSI the container storage interface that you can get detailed information on CSI from Presentations done by the kubernetes storage sake, but the just of it is that CSI is an orchestrator independent industry standard where you write a driver and it is Capable of supporting not just kubernetes, but Apache mazos docker and cloud foundry and it enabled Storage vendors to write one driver that supports more than Just kubernetes, but kubernetes put in place a Storage interface to use these out of tree storage implementations and When you move to the out of tree cloud provider, you have to move to the out of tree storage plug-in as well They're there a couple that move is coupled together. So I Guess with with that I'll move on so the Schedule here is that the CSI the storage driver for vSphere is it is in the first release now We've declared it alpha Version zero two zero and anticipate a GA in August a couple months from now It does have some enhanced features. So One of the things that's happened with both the out of tree cloud provider and the out of tree storage plug-ins is that The old versions are no longer having feature enhancements and new features are going strictly into the out of tree and Those new feature ads have been happening since about the beginning of the year so already we're in a situation where you can do things with the out of tree that are possible with the entry and What's gone on with the entry is bug fix only so no new feature ads are being admitted into those So if you want these new features You might have a motivation to move to the out of tree versions right now even though you're not forced to What we're going to have going forward is VM independent volume management what the so-called first-class disk That is something that came in the later editions of vSphere where it was possible to define storage volumes that aren't Strictly tied to one VM. They they have their own existence independently So that these volumes can be created and handed off from one VM to another We also have support for straddling Kubernetes clusters across multiple vCenters multiple data centers and Fabio is going to give a demo of that later in this presentation and We can also handle both Conventional and what are called raw volume mounts the difference is that a conventional mount Attaches in the file system a raw one appears as strictly a device and there are certain types of apps It's might be somewhat unusual But there are applications that are prepared to deal with a raw disk device that doesn't have a file system installed upon it And then we also have zone support, which is a feature of Kubernetes scheduling that Can make the scheduler aware of things like failure domains to have better workload placement behavior so This interaction with pod scheduling and zones. I think this is where I'm going to hand it off to my colleague. Yep And this is going to be right before the demo. So as Steve mentioned club provider and And CSI are actually getting new features compared to what we have currently in tree So one of these new features is that we're able to schedule pods that have Storage that is assigned to a specific zone So what we have is we can define Arbitrally in vSphere the type of zones that we want to have so we can specify with using tags at data center and the zone Where the data center lives and using those tags We can schedule pods with a specific storage the lives in that specific region or that specific zone Without further ado, I'm just gonna go on with a demo I had to record a demo because I wasn't sure if we're able to we're able to Connect to our system So it's recorded demo has a lot lots of pauses in the middle. It's gonna step Ahead and just highlight the the major things in the video So what we have is we have our our environment here is made up with 2v centers So we have to be centered think of them as two independent control planes the first one we have here as a single data center We have a cluster has a bunch of hosts and it has part of our Kubernetes clusters So we have a master and we have a bunch of workers here and this is tagged as Region Let me just go closer get closer. So this is region EU So the European one and we have a zone that says Case region EU all because we don't have a specific region for this. We just have one the other vSphere The other vCenter is connected to two data centers One data center is in the region US and in the zone US West So this this data center is on the West Coast of the US while the other one is On US East, so it's still in the region US, but it's it's in a different zone and Here we have the rest of the nodes for our Kubernetes cluster that are spread across the two vCenters so the first thing we want to do here is we want to deploy our CCM which is our our Cloud controlling manager, so the the the auto tree cloud provider And apologies for this, but you know, this is basically a bunch of commands We were gonna just deploy the manifest that we have here And this is gonna deploy this is the manifest that are coming from directly from the repo where the CCM lives If we look at the pods, so we're gonna see that we have a clock controller manager being deployed There's only one controller manager deployed because there's really one that needs to be running or It has to run on every master node if you have a multi master cluster The second step is to deploy the CSI driver So what we're gonna do is we're gonna get the manifest for the CSI driver and Deploy it on top of our cluster that we have Maybe it's not, you know, visible here, but we were coming we having the same Configuration file for both the CCM and CSI's called vSphere.conf is the same file that is deployed Both as a configuration for a CCM and for CSI because we're sharing that configuration file And what we're doing here, we're basically just creating and applying those those YAML files Now that we applied them, we make sure that they're running correctly We're gonna see a demon set deployed for CSI because CSI requires to have Pod running on every single node because this pod here is responsible for attaching and detaching the disks So we see there are controlling managers still running. We have a CSI attached here We have all the nodes. We have the provision in running. So at this point, we're all good We have a working environment with CSI. We're able to provision storage So the next thing we want to do is we want to create a storage class and a persistent volume claim and I'm gonna show that in a second in the demo and This is highlighting the fact that we have storage class the data store cluster So this is another object in in vSphere So we basically have iSCSI one cluster, which is a data store cluster that Contains data data stores that are shared in the specific data center So there's a bunch of data stores and we want to target that specific cluster So we want to target this iSCSI one cluster on region US US East So we're gonna open up And apologies if it's not there's a little light here, but this is a storage class It's a type vSphere fcd, which is first-class disk It's gonna have a parent type of data store cluster and the name is iSCSI one cluster Which is the one that we just saw on vCenter We're gonna mark that with two labels We're gonna say that the failure domain the zone for the failure domain is Kate's zone us East in the region is Kate's region us So we want to have this storage class attached to this specific zone Now we're gonna create it just apply the YAML file that we just saw And now we are ready to deploy our simple application The simple application is just really just a busy box container that sleeps So the application itself is not doing anything But the important part is that we are mounting a slash data here Which is coming from my fc volume My fcd volume is a persistent wall persistent volume claim that we're gonna see after this That is a five gig virtual disk that is going to get attached to this application Also this application will be We'll have some labels that are will that will deploy and schedule this application This pod specifically in those us east zone in the us region And in a second we're gonna see these persistent volume claim And this is the one that we just declared up there This is the persistent volume claim This is the persistent volume claim we have vCirc si pvc it's a type rate right once And it's a five gig disk So we create our we apply our test pod YAML so this is our application We just go get pod we're gonna see that we have a my csi application up here We have my csi app, which is our application that is being created Now it is running so if we describe that application If we describe the pod We're gonna see that it's being it's mounting the data from my fcd volume, which is our pvc And it's being scheduled on kates worker nine So if we go back to tv center now We see that we're in kates zone us east region us we have kates worker nine And if we open up kates worker nine and we edit settings to look at the vm itself We're gonna see that we have a secondary disk here attached with a five gig virtual disk Which is the one that we just applied through the YAML file And this is an application scheduled on a pretty complex vSphere environment That has regions that has labels attached to it And we're able to just define the region and the zone where we wanted that application to be deployed and The storage will follow the same rules that we have for compute Let me just go back up here And this is something that is only available in the out-of-tree cloud provider and NCSI driver If you want to know more about this if you want to try it yourself Here there's a bunch of references obviously, you know, you don't have to write them down You can look at the slides after we're done here There is There's decent documentation for all this It will be it will get better. We're actually working on a new documentation website With all this and you know much better Walkthrough and some workflow and you know some other type of configuration right now. There's just a bunch of Deployment models that we support. Well that we document So if you don't find what's in there, or if you don't find this useful for your setup Feel free to open an issue and get up or just reach it reach out to us on slack on the kubernetes slack There's a channel called sick the emmer. There's a channel called there will be a channel called provider vSphere There's going to be a open up soon There's also a bunch of other presentation that we've done in the past. So if you want to Learn more about that if you want to watch other demos just make sure to check them out And with that We are done. Thank you very much. If there are any questions, we're happy to take them One thing I'd like to point out I guess we should have put it in that slide, but it was so crowded there wasn't room, but We did a presentation specifically on the csi autotree storage just a month ago in barcelona and that Got videoed and the deck is published at the kubecon barcelona website So you can find it there and that was a whole hour strictly on the storage part. So it's a it's a deeper dive So you might be interested in that Any question For csi if we want to mount a pvc to a container We should first mount a disk to the vm, right? No, this is automatic. This is done directly by if the container migrate to an restarting another node the disk will Mount from the walker and mounted to another walker. Yes It will be unmounted from the worker and mounted to the other worker Do you know, uh, do you know what the For example, uh, set step or cluster Is is is that necessary to mount the the the disk to the vm first before container use that? No, in general, it's that's a bad idea It is possible with the kubernetes architecture just because they did it a different way at the beginning To do things like create a volume first And then specifically indicate that you want that exact volume in The pod spec or the container spec, but that's generally deemed to be a bad idea Uh, so there's something called storage classes where you define Classes that you should label with meaningful names Kind of indicating an outcome if you're familiar with vSphere It would be along the lines of vVol where you divide storage into classes of service like ssd might be fast class of storage and something that's slow rotating drives might be called a cheap class and the pods are Defined to take advantage of those labels for the classes of service And that's the kind of thing that makes them portable where if you had a database that used cheap storage for log files But fast storage for indexes If it was based on those labels You could run it on vSphere and and admin maps the storage classes to physical data stores or classes And the users consume based on the outcome specifications You could take that to a public cloud like amazon And have a mapping defined to use those same class names and it just works It gives you the you know maps the expensive Fast storage for your database indexes and the cheap storage for your log files, and it is portable If if I take the same example that I shown in the demo with what you were asking You would see basically we had three workers in that specific region So if you were to drain a node let's say drain kates worker nine Which is the one that the pod was scheduled on if you drain that node and you remove that node The pod will be rescheduled the disk will be moved to reattached to another worker inside that region So kates worker eight, which is what's in the in the same region It will get the disk attached and it will get the pod schedule on So under the covers kubernetes as an orchestrator along with the csi is handling these detach and reattach Now they they can't defeat things like Um, if you have data plane communication limitations where certain volumes aren't simply aren't detachable in some zones or regions They aren't going to create some miracle where It's going to hook those up, but in general if you intelligently define these things will just work and you don't have to manually deal with attach Uh detach as long as you have the underlying infrastructure the underlying v-sphere design done correctly where you have multiple works That are able to access the same vmdk So they have a path to access that vmdk then it's going to be automatically dealt with by kubernetes And csi obviously that's csi is doing the work really so a couple of like operational questions, I guess so what's involved in If i've got the current entry implementation in moving is it just an upgrade and That's a very good question Right now Like as in today, we don't have a migration path Uh, the migration path is actually pretty convoluted because the entry Club provider well the entry storage provider really It's using a different api in v-sphere to deal with disks While csi is using the newer api, which is first-class disk And between the two api's inside v-sphere. There's no clear upgrade path So we are trying to work around this limitation and try to figure out a path to move people that are that have like an extensive set of Disk and persistent volume created with the old in tree and move them to the outer tree Uh, but right now we don't have an answer for that like not today Yeah, the storage saying now this only applies to the storage aspects and not the cloud provider aspects, but they have a plan to eventually uh replace the existing entry with stubs that would simply call the csi implementation But that isn't implemented yet. So you can't get that today but that is That's the closest thing to a long-range plan that has been stated and this isn't unique to vm To the v-sphere implementation the plan is to pull this off for all forms of storage The second one was more You said the admin defines Is it oh, so if it's vsan are they expected to be writing the yaml for the storage class? It's an operational thing, but you want to take this? Well For now, yes, there will be changes coming in the v-sphere platform that will probably Will limit the amount of work you have to do But right now this is what you have Yeah, and when it comes to admin, there's a concept of both kubernetes admins as well as admins of the underlying Platform and it depends on your organization how fragmented they are, you know, I've seen somewhere you have admins for network admins for storage admins for Compute and others that that's one role where they have hundred percent visibility To define these storage classes in kubernetes. You have to be a kubernetes admin Now you could have another layer where Storage admins would provision the underlying infrastructure and simply Give enough detail to the kubernetes admin to enable them to do that or it could be one person And that's kind of going to be organizational dependent on How many how complex they want to make their own internal Organization chart Anybody else got a question? going once twice Okay, thanks for coming. Uh, it looks like this is recorded if you missed anything and do you want to bring it back up with the link to the deck Yeah, absolutely. This should be The right link to the disk deck So if you want to if you want to open that up Uh, it's a short link. It will get you all the links that we have on this deck Uh, it will also be available on the kubecon website And just check it out. And by the way, we have had a SIG going for um, I think over a year at this point Uh, where we have meetings every couple of weeks There actually was one due this week that we're going to cancel just because fabio and I will be traveling but You can join this vmware SIG and if you're a user or I think the people who could give value out of that would be People who want to help us develop this Uh, if there are are any that's a rare niche But if you're a user of kubernetes running on top of vmware hypervisors This could be of use to you So we have users showing up with questions support requests feature requests um And the other class of people who could get value out of this is if you work for somebody doing a kubernetes distribution Examples might be red hat open shift. Uh, I don't want to plug vmware's own solution because I don't want to be viewed as commercial here, but pretty much everybody with a A kubernetes distro that can run on-prem will support running it on the vmware hypervisor and if you're Uh, working on one of those you you're welcome to come to these meetings ask questions or ask for features or discussions um The kubernetes project is reforming these sigs that are related to cloud providers. So by the end of the year the plan is that there will be a user group and then the group doing of developers doing the vmware cloud provider are going to be folded into A group underneath the cloud provider sig so expect this to change but for now those links are good join the group and Join our meetings if the time zone is bad because you're in china the meetings are recorded both notes and youtube videos And That's pretty much it. Thank you very much for coming. Yeah, thank you You