 Welcome everybody. Can you hear me? Loud and clear. I'm Brad Topol from IBM. This is Ton No from IBM. And Yin Ding from Hawaii. And we're here to talk to you about Kubernetes on OpenStack. What's available and where are the gaps? So just a quick overview of why you're going to want to integrate OpenStack and Kubernetes. That's always fun to talk about. We'll then talk about how it's done. What are the integration plug points? And then we'll talk about the different specific pieces of OpenStack that we're leveraging. The identity access management component of OpenStack. Also notice Keystone, Networking. Many of you know it as Neutron. And the storage components, Cinder and Manila. So this is really one of my favorite charts. So why are we going to integrate Kubernetes with OpenStack? Well, let's take a look at this from both sides. From the Kubernetes side, why are we getting interested in Kubernetes? Well, I've had the pleasure of going to KubaCon in Seattle and in Berlin. And it's an incredibly hot open source community. A lot of energy, a lot of participation, a lot of vendors. Kubernetes provides a nice usage model with a constraint-based approach with controlling the number of replicas, auto-scale up, scale down, lots of other features organizing the pods. So it's really becoming a popular approach for managing and scheduling containers across clusters. But if you're going to deploy Kubernetes, if you can go to a public cloud, well you can go to that. But if you're going to deploy it on-prem, private cloud, well, you're going to need some type of underlying infrastructure to support that. And, you know, you're going to figure out how to do it. So one of the things we'll talk about is why OpenStack is great for that. Ideally an automated process, which again we'll also talk about and what you'll need. Now let's get to the other part. So we've talked about it, you know, put my Kube hat on. Kube's getting really hot. Now let me put my OpenStack hat on. That's how most of you all know me. OpenStack's a really great platform for deploying Kubernetes. So what does OpenStack have to offer? Well, OpenStack's got some major components. Identity, storage, networking that they've been working on for a long time and they're very mature. And let me give you an example. Cinder. Anybody know how many different drivers are supported by Cinder? Anybody? It is right at 100. Thank you, sir. You've got 100 drivers in Cinder already there. And, you know, one of the critical things about having drivers for those third-party vendors is, well, how good are they? And how much CI is done? And on those hundreds, do you know how many all have third-party CI set up for? Anybody? Anybody? All of them! So you've got this component sitting there with 100 incredibly well-tested drivers for probably about around 25 vendors. That's the things that folks need for, you know, stability, make their lives well, sleep at night. And we'll take another quick example. If you look at the identity component of Keystone, former PTL over there can't answer. But one of the nice things about Keystone, it started off with some basic identity authentication support and very quickly added LDAP support, MSAD support, multiple LDAPs. If an insurance company had 25 LDAPs, it could support that as well. And then it moved on to federated identity support. So all of that smarts with SAML and OpenID Connect, all of that already available in Keystone. And so why wouldn't it be great to be able to reuse all that? And finally, you're going to see areas where folks already have OpenStack deployed. They've already had it connected up to their LDAP and they've created their groups. And now you come along with Kubernetes. Guess what? Those customers don't want to reset up those groups and reset up everything yet again. So anytime you have one of these situations, whether it's a... We learned this when we were putting Cloud Foundry and OpenStack together. If you're going to put Kubernetes and OpenStack together and they're having the same environment, those folks want to reuse as much as possible. And so that's another reason to want to put those things together. So how's it done? Well, Kubernetes has a nice plug-in model. They've got this Cloud provider interface and it can work with OpenStack, GCE, AWS, Azure. They're doing a really nice job there. And so there is a place as the different components of Kubernetes want to access storage or networking or authentication-type components, there's a nice place where that's encapsulated and where it can be called. And so you'll see there's work done there in that Cloud provider. And that plug-in will, again, support stuff for identity networking storage. And these guys on stage who are much smarter than me through those technical details. And all this code is available. You can go find that Cloud provider interface in the Kubernetes repo. It also uses a nice SDK, the Gopher Cloud, for integrating into OpenStack, so you can look at that. And then there's a repo where you can see where it's all provided. And how do we pull this all together in OpenStack to make this easy and automated for you to leverage? And that's what these guys work on. The Magnum Project delivers all that. So why don't we get into the ROP stuff? Thank you, Brad. So let's talk about the identity management. So in any system, you can expect the identity management to be the key component of that system. So in OpenStack, Keystone provides that service. So you can expect that Keystone is pretty robust. It's pretty well integrated and very well established in the enterprise. And in any system, it will also be populated with the user base already. You can expect to be integrated with a product like LDAP or Microsoft Active Directory. And in some cases, it might be even federated across a different organization. So in terms of the code, the interface between OpenStack and Kubernetes is in the Google Cloud package. So what you have to do before Kubernetes can talk to OpenStack is that the Kubernetes has to authenticate with Keystone to get a session. And to do this, you need a user credential. So what you have to do is, in the config file for the plugin, you have to put in your user credential. And then once Kubernetes authenticate with OpenStack, it also gets along with the authentication or the authorization for the user account. So it gets all the code or what service you can access to. And in fact, this is where multi-tenancy come into the picture. Now, the config file is a plain text file. So if you are concerned about security, because you might be running container on the same host, then a better approach is to use Keystone trust. So with a Keystone trust, then you can limit the scope of that trust to a much smaller scope. For instance, you can define a scope to work only for the Kubernetes endpoint. That way, if the trust is compromised for some reason, then the security exposure is not too bad. So when we use Magnum to build a Kubernetes cluster, then all this is automated by Magnum, and it just works. Looking to the future, then you can start in the last talk that multi-tenancy is coming to Kubernetes, probably later this year. So there's still a concern about whether we have enough isolation between containers for different tenets. But anyway, if when we have that multi-tenancy support in Kubernetes, then it would be good, it would be natural to ask whether it makes that sense to use Keystone in Kubernetes to manage the identity. All right. So next, we'll talk about networking. Networking is pretty complicated. So there are many ways to implement networking. So what I show here is one way to do this. It's not the only way, so I'm for sure. But when you run Keep Control to, let's say, launch a pod, you know, that pod can go through the networking layer to talk to any other pod in the cluster. So here you can see what I have here is the pod on the left side being connected to a VETH interface to a bridge, and then from the bridge it goes to the network interface of the host, and then from there it goes out to the network and then it gets routed to another host and the other pod resides, and then it goes, you know, that's how it gets to the other pod. Right. If you run Keep Control to, say, expose a service, right, what happens there is that the Kubernetes will talk to the cloud layer to get the load balancer and float the IP so that, you know, your service is now accessible from outside the cluster. So what I've done here is that I kind of grouped them into three groups, roughly based on how they are implemented, and I'll go into each of those in details in the next couple slides. So first the network interface, right, so what you have to do here when you create a pod or delete a pod is to set up the network interface so that pod can talk to the rest of the network. This is done by Kubernetes running on each of the nodes, and you can tell Kubernetes the network driver, the network driver that you want to use to do all those setup. Right now there are two ways to do this. One option is to keep net. So keep net give you the very basic network configuration where the pod are connected to a bridge, and then from the bridge you go out to the network. So that's the simplest basic implementation. Another way is to use a CNI driver. CNI driver is basically a piece of code to implement the container network interface spec. There may be many of these that are available. So for OpenStack, the career team built a CNI driver that you can plug in, right, and what you get with that driver is that you get a native neutron support in your Kubernetes. So what happened there is that the node that the VM is running would have a neutron transport, and when the pod is created, then the career CNI driver would create a VIF on the pod and connect it directly to the transport. So you have a pretty simple interface. So the advantage here is that being all neutron, you can now leverage all the network driver that comes with neutron that comes from the network provider. So that's a nice thing to leverage. The other advantage is that, again, since it's a straight neutron private network, you can put VM on it, and that VM would be able to access all the pod. You have VM and pod living on the same network. Looking into the future, the career team is working on improving the performance for all these setups, and they can come up faster and perform better. Now, one thing I want to note is that this kind of network setup is still in alpha phase, so we can expect a lot of change in the new future. So that takes care of the setup in the network interface. The next step is to take care of the routing. So once you get out of the network, how do you get to the host? So the requirement here is that any pod has to be able to reach any other pod. So this routing is set up when you build the cluster. Do you take care of all the configuration and so on to set up the routing? It also happens when you add a new node to your cluster, so that's kind of like in the infrastructure layer of the cluster. So to get this kind of routing, one simple way is to use an overlay network, like Flannel or Calico and so on. So with that overlay network, all you have to do is attach the pod onto the overlay network and then the overlay takes care of all the routing for you no matter what the nodes are. The disadvantage, of course, is the overhead for the encapsulation. In a talk in Tokyo, we show some performance measurement of what the overlay network would cost you, and it is pretty significant. Now, the good news is that for Neutron, for our cluster here, if you have all your nodes running on the same L2 network, then Flannel has a mode called host gateway where it actually avoids the encapsulation and just creates an IP rule to do the routing for you. So with that, the performance is quite good. It's good enough to use Flannel in production. So looking at the future, we know that with the Google Cloud Engine, they have native routing in their fabric so that all the routing happen automatically. You don't need overlay or anything like that. So we do have Neutron routing also, so it will be natural to look into whether we can do the same thing as a Google Cloud Engine. I think the crew team is working on this as well. Third, we look at the external access. So why do we need this? So we know that when you have an app running on a pod, you don't want to use the pod IP to access the app because the pod is not stable. It might die and come back with a different IP. What you have to do is create a service so that you can get a stable IP in front of that pod. Now, the service does have a load balancing mechanism inside it so that it can load balance against all the pod behind it. But the problem is that the IP for the service is still local to your cluster network. It's not visible outside. So if you want to get in from outside and then get to your app, then you need more support. And this is where the load balancer from the cloud comes in. The plugin for the Kubernetes would talk to the stack and request the load balancer for you so that it gets automated. The plugin code right now supports both Elbas version 1 and 2. And again, if you use Magnum to build your Kubernetes cluster, then all this is automated also. It just works. So for the future consideration, the code for the plugin is fairly complete in terms of supporting the load balancer. But there's a few things that we can add to around our support. We get add support for region. Right now, it only supports the current region. We can also add automation for full time piece. So that helps you a little bit also. You can see that in the demo. So with that, let me pause and show you a quick demo. This is a recorded demo. So what I have here is I use Magnum to build a small Kubernetes cluster with just one master and two nodes. So it builds pretty quickly. It's about just take about a minute. This is just DevSack from code from master about a week ago. So do a cluster list, and you see the cluster that Magnum created. You have one master and two nodes here. If you look at the Nova instance, we have three Nova instances here. So now we don't have any load balancer yet. It's all empty. We look at the load balancer. And if you look at the floating IP, right now I have three floating IP, but for each of the Nova instances. So that's the initial setup when you create the cluster. So what I do now is I'll do key control to start up two instances of the Internet two-part in a deployment. So those parts are coming up. So this is a detail of those two parts. And then what I do next is to run another key control command to create a service. And this is a service where I would request the load balancer. So you can see that here the service come up, but you get the cluster IP, which is internal to the network. And then you have the external IP. And that's still pending, because if you go back to the open stack side, you see that the load balancer is being created. Take a little while for that to come up. And then eventually you get the load balancer running. And now you have the external IP, which is the IP for the load balancer. That's a 10.0.0.11. So once this come up on the open stack side, you can also see a whole bunch of resources that get created alongside with the load balancer. You get the pool. And you get the member or the pool, which are the two nodes of your cluster. You get your listener, and you get your health monitor for the load balancer. So all those get created by the plug-in code. So now we're not quite done yet, because the IP for that load balancer is still an internal IP. It's live on the network cluster. So now what I have to do to get it outside is to associate a floating IP to it. What I just did in the previous slide is to create a floating IP. Now I go back to the load balancer, and I'm going to associate my floating IP to the load balancer. So I look for the VIP port of the load balancer, and I do the floating IP associate. So this is part that right now we still have to do manually, but we can imagine adding the support to the plug-in so that this happens automatically. So now you see that the floating IP that I just created is associated with the load balancer, and then we can do a curve on that IP, and we get the default page for the Internet that comes straight from the pod. All right, so that's a quick short demo to show you how Kubernetes is talking to Overstock and working together. So with that, let me pass to my colleague, In. He will talk about volume. Hi. Now it's my turn talking about storage. So last year in Barcelona, I talked about how we use Fushi plug-in to connect the Kubernetes to Cinder and other stuff. So here I gave a more detailed chart so you can see the client talk to Kubernetes master. Then Kubernetes master go to the agent and talk to Kubelet, and Kubelet talk to our driver. Then we can pick what to connect next. Either Cinder in the bare metal case, in the VM case a little more complicated. You need to connect both Cinder and Nova. I can show you what's next. In the next slide, a little bit more detail. So with this, using the Cinder volume, let's say, we support multiple different storage drivers and from different vendors. So it solves a lot of problems. You don't need to write individual drivers for your container cluster. And also, you can pick your choice, say, I only want to connect to individual drivers directly. It's also working. And also with this Fushi plug-in, you can connect to the raw device directly. So then you do the formatting and whatever you want to do with this storage. And I'm talking about here, as Brad mentioned in the opening talk, integrating with Cinder, giving the users a lot of convenience and a wide range of storage options. So you don't need to write your own drivers to get this benefit automatically. So now I'm going a little bit of detail, say, what's the performance? After last year, we talked about the Fushi, how they connect to the volume. We solved the problem, say, connect storage from OpenStack to the orchestration of a container. However, we observed a little bit of performance issues, say, OK, is that slow? So here I gave a detailed call chart, say, what's happening behind the scene. So you can see when you call, say, a creative volume and attach it, so what's the secret of the operation? First, you call the Cinder volume, say, RESTful call, say, create volume. And behind the scene, Cinder talked to Noah, say, and fetched this volume information. And Noah got to the connector information, then Noah called Cinder, say, OK, you need this connection, and call storage RESTful API to do the volume mapping. And then Noah used this LeapVirtual iSCSI volume driver, and this method called connect volume. And let's hold to scan this disk and do the initial formatting, then attach this volume to the host. And then Noah called the Cinder API to attach this volume to finish the volume status. So that's our observation. So I set up a small environment about 8-core, 8GB. Everyone stack is about 24 cores and 24 GB of memory. And the host is about 8 cores and 8 GB each. So the command I run is volume driver fushi and to connect this. In this test case, I already pulled the image back. So the time span, it doesn't include any docker pool from pull the image from the docker hop. It's pure analysis about this plug-in performance. So you can see the 1, 2, 3, 4, 5, 6. I highlight some numbers in this chart. So the longest time we span is back to the previous slide. It's number five. The host attached the volume. So this one is related to the dot volume size. The bigger the volume is, the longer time you spend on this operation. And the second one is in the crit volume call. It's about 1.5 seconds. You can see the speed in here. And the third longest time we span is this volume mapping in the number four. It's about 0.7 second average. So this raises the question, say, can this plug-in operation keep up the speed of the container cluster? Because we average, I think average container life span is less than 20 seconds. So we spend about 80 seconds here to the volume. And it apparently slowed down the performance. So here we say how we improve this performance. So we can see the Kubernetes introduced a PV and a PVC mechanism. So basically it allows you to create a persistent volume in your head. So you don't need to spend this time to waiting for the crit volume call. So it's save about three quarters of the time it's spent here. But it's still about two seconds to spend among this volume to the container. But in the bare metal case, we don't need to know what we can use in there. Directly you talk to the volume. Actually, this performance got improved a lot. So in the bare metal case, much better performance. Then I don't have demo here, but I can show we also tried to use Manila to do a file system. So basically it's a file stage can help you. So in the share storage, we're using Manila to create a share mount point. So then the PVC, the performance is in the VM case, the machine case, makes it much faster. So probably next time if I have more time, I will show a demo to show this. And let's do a quick demo. So here this first demo is to show the sender plug-in cases. You can see initially there's a no sender volume, then we create one. Now we can see we create a new sender volume. And we use this ID we put in the Kubernetes YAML file and we replace here. Then we create a Kubernetes cluster, use the kubo control. Now you can see the pod is creating and created. And we can see this sender volume is attached as in use. So now we just use normal to find out the host that we are using to host this container. Now here we are trying to connect to the host container and show what exactly, I mean, actually the volume is attached to the container. Here is the container ID. We log into the container, use the Docker command. Here. So yeah, here. Yeah, we can see here the container mounted this volume in use in this directory as we specified. Now we delete this pod. We show the PV and the PVC cases. Now you can see the sender volume is available. So this is the PVC cases. So we are going to mount a volume in that directory inside the container. Now we create a pod, the pod. Yeah, it takes about 30 seconds. So the pod is bringing up the container. It's running. So we can see this is the PVC volume. And there's one gig as we specified. Here is the sender volume. It's in use. So that's the case. Let me show. Let me show another one is to support my cases. Oops. So here I show you the performance of what's going on for this Fushi plugin as I mentioned how many times. It's about one minute. I just want to show you the feeling you can see. Here we create volume. It takes about a few seconds. So we create one as a device here. And then we try to mount this to the VM. Here you can see as the mount takes about less than a second. However, the create takes about two or three seconds. So I just say we can operate on this volume. Now you can see even the detach takes some time because we need to give the information to no one. This communication takes some time. Now it's deleted. So it's got deleted. But as you can see, it takes a few seconds. So now I'm talking about a little bit more advanced topics. So initially when the container introduced, we were talking about, say, okay, the state is micro services. But now and now when the containers are well-used, people are talking about, oh, we need some stateful container and we need to protect the data. So here I'm talking about the potential work we are investigating to integrate our effort with COBA. COBA is the OpenStack project. It's focusing on protect data and metadata for the application. So it provides a standard framework API and services so we can integrate our vendor mechanism and to protect and unify the workflow for the users. So with this, we can provide a snapshot and the replication. The snapshot in the Kubernetes Storage SIG we were talking about by next week, we are going to release an alpha version of a snapshot API. So snapshot is really useful for rolling back or upgrade cases. So in the stateless cases, for the container, when we do the upgrade, we just upgrade directly. If there's something going wrong, we just roll back and kill the new version and roll back to the old version. However, in the stateful cases, we cannot do this because the data may be changed and metadata may be changed. So it's not compatible with the previous version. So what we do is say, okay, we need to take a snapshot first and we do the upgrade. If there's something wrong, we need to roll back to the previous version of the containers or container clusters. We attach the... We roll back to the snapshot we took and attach them back to the container to ensure that the data is compatible with your services. For the replication, it's basically trying to protect from any hardware failure or network failures. So when you disconnect it with your storage, in the stateful cases, it's a disaster. So that's why we try to integrate the cardboard to using this replication data mechanism to replicate your data. So we are able to recover, even if we have a hardware failure or a storage failure. So the fushi we are talking about putting another effort we are working on with OpenSCS. So OpenSCS is focusing on how Kubernetes with OpenStack and other infrastructure layers. So we can do storage discovery. So basically you can bring your new storage. This OpenSCS controller tries to do the scan and discover and add your storage to the existing storage cluster and do some lifecycle management for your storage and provide the data protection as we mentioned. So fushi basically will become the first layer as a plugin layer for the OpenSCS project. So what in blue is OpenSCS. The top layer is the plugin basically talking to... connect to the container orchestration layer. Then the controller is the core. So it's do the storage discovery, backup snapshot, something like that. Then we have the driver. We can connect to Cinder driver, Manila driver, and also other storage. So here is some information. That's the links. Let's get back to Brad. Thank you. Clothing. Sure. So what have we done? We presented how OpenStack. It's an excellent IaaS option for Kubernetes. It provides you multi-tenancy, good efficient utilization of resources. Gives you lots of storage driver options and lots of authentication federated identity options. Lots of things going on. All of this out of the box. Really what you're looking at and a theme here is looking at great open source technologies, open source communities, and combining them to give folks like you the best-to-breed option. That's what we're focused on. It's active development. We've looked at the different code and where the different pieces are and where they fit together. There's more improvements that are possible in the near future. So on that note, we're happy to take questions. Don't be afraid. I think it's scary out there. Try being up here in front of all you. People know we are. Oh, yeah. Thank you. There's a mic right behind you, sir. Hi. Thanks for the presentation. I have a question with regards to load balancing. I was kind of curious. What was the reason to use Elbas and Kubernetes already providing you load balancing so it's kind of redundant? It seems much simpler solution just to pass the floating IP to the Kubernetes so it would use it as external and perfectly do the load balancing. That's a good question. Thank you. This is something we... There are two part of it. The first thing is that the service, as I mentioned, does do some load balancing with it. So that happened within the internal network of Kubernetes. So what happens then? If you have a part that access that service and behind that service is a bunch of other part, then that service would do the load balancing for you. But that's only internal. There's no way to get into... to see that service. But I guess your question is whether to just attach the floating IP to the service. So we can't quite do that. There's a long... If you look at the discussion on the Kubernetes discussion, there has been discussion on that. And when we start doing this, we also ask that question to the Kubernetes sign and it's not quite... You can't quite do that. Thank you. Anything else? Going once? All right. Thank you, everyone. Thank you.