 Good morning everybody We very much appreciate all of you making it to a 9 a.m. Session the day after all the parties So we appreciate your dedication there The title of the stock is a base. Oh, it's an open stack the perfect tag team for containers If you look at the world the way we look at the world You've got open stack which obviously huge amount of interest and lots of wonderful IIS capabilities and If you also see what's happening in the world these container things popped up and a huge amount of interest there And to make things even more interesting, you know the container story is Is one that has different facets? Swarm kubernetes mesos So so what we're talking about in this presentation is how we're taking that world with those pieces and Combining it with open stack and our strategies for doing that to get the best of both worlds and So we're gonna look at you know how we're providing an environment that's optimized for Allowing you to run swarm and kubernetes workloads We're gonna talk about how we leverage all the benefits from open stack including Magnum and If you know Paulo here, he's gonna talk about our combined mesos kubernetes and swarm architecture And then after that Tom's gonna show us some demos of where we integrate with open stack And then we'll do a scaling demo someone hand over the Apollo. Thank you So let's talk first about the swarm How many people are familiar with the swarm here? Okay, what about kubernetes? Very good. And what about mesos? Cool, so anyway, I'm gonna talk a bit about this this frameworks I think it's probably good to to mention what they do what are the difference between them first Let's talk about swarm what is swarm essentially swarm is a way when you have A number of hosts where you are running docker the docker engine to expose all of them as a single Virtual engine it provides a standard docker API and that is important because you can use the standard docker tools For example docker compose Docker CLI Jenkins all these tools will work out of the box and will provide essentially the same familiar experience to docker developers So if you're really using docker for development in your laptop or in your desktop You essentially have the same experience, right? But docker swarm provides the way to schedule containers in in different hosts and Looking at the we have here a very simple Architectural chart looking here what we have is there is a swarm manager component that connects to a number of swarm nodes where each one of them is running the docker engine and The manager essentially is two components the scheduler that is essentially what it does is to schedule containers on the different host based on Different criterias for example scheduling constraints and then there is also a discovery component that provides the The membership for the different nodes the membership formation to the swarm manager which nodes belong to to this particular cluster So again, it's relatively straightforward Very simple very lightweight Let's talk a bit about Kubernetes Kubernetes it's essentially an orchestration open source for docker containers was promoted by Open source by Google more than a year ago and It's getting some traction in the open source community It has much more sophistication if you compare with swarm in terms of capabilities Essentially what it does it provides also the ability to schedule containers on different nodes, but it provides More concepts compared to just form it provides the concept of pods What are pods? Pods are essentially a Set of containers that are collocated in the same host they work together. They share resources like a network namespace for example and They work together imagine a pod can be for example a web server and a database working together For for building an application and you can imagine also adding the same pod other containers for things like for example logging monitoring and Additional capabilities that are part of your application. They can be packaged together in the same pod There is also a notion of Replica set formally known as replication controllers Simply these are kind of scaling groups. It's really a set of pods that homogenous set of pods that can be Scaled up and down and also provides capability like Auto recovery and auto scaling And another notion that we have in Kubernetes is also services. You can imagine a service let just a way to essentially associate with the law balancer endpoints That are essentially the pods or the replica sets that you can essentially expose using labels that can be associated with the pods for example and the selectors So this pretty much the main concept that we have in Kubernetes There is much more of course to talk about Kubernetes probably alone. It will take some time But I won't just to point out to that the API for Kubernetes is actually quite different from swarm essentially because there are these additional capabilities compared to containers just Docker containers for example, you can Create pods you can create replica sets you can create a manage Services, so the the API is definitely much more sophisticated and many more capabilities So let's talk a bit now about Misos this is another component that we're gonna show in this demo Misos is also Open source cluster management system and the focus here is really on managing resources And one of the main use case for Misos is really the ability to Consolidate workloads imagine that I have for example a workload for running ad-up applications one for maybe spark one for maybe Storm here is at least in this example and normally we run them all in different clusters Imagine that now I want to share a single cluster around all these different Workloads together this pretty much what misos allow me to do because it provides the ability to manage resources and by resources I mean compute disk storage Essentially and CPU so Once basically I deploy misos on on a cluster misos will provide the resource softwares to the different Frameworks framework is also terminology using misos for example ad-up work your misos is called a framework and The result then the offer essentially the resource offer will provide information like the nodes that are available What is the CPU available? What is the memory available on these nodes and the disk available? Etc. And typically the framework will have a scheduler to choose which particular nodes will want to run this Resources so there is some separation of concern between the actual management of the resources and the actual scheduling that is done by the that framework and There are many frameworks in the open source communities that they they keep on adding more and The example you're gonna show today is essentially running as a framework Kubernetes and swarm in this environment Finally, let's put all together. We talked about Swarm Kubernetes and misos. Let's see how these things work together And also with open stack because that's what we really want to show here as how we leverage open stack as well as misos Swarm and Kubernetes in this in this environment So open stack is essentially providing the resources compute storage network Essentially the the machines where we're gonna run Misos and then we're gonna transform and Kubernetes on top of misos. So the bottom layer you see we have open stack and We also have a misos agents in the bottom layer each node as a misos agent installed and then there is a misos master Okay, the misos agents are essentially getting information on the available Resources on each host and they provide information to the misos master that aggregates them and Creates what I mentioned earlier as resource offers. So again a resource offer is It's a list of hosts essentially with the list of available resources like CPU Memory and disk on each host and you can have other resources as well as extensions for example ports if you are using port mapping or they can be Potentially GPUs or other type of resources that can be deploying those host the misos master will send these offers to each framework That is registered with misos. For example, he's gonna send the The offer to swarm is gonna send to Kubernetes as well And now each framework knows pretty much what is available to deploy To deploy basically the workload on these resources So for example, if we start from swarm suppose that there is a request for swarm to start a Container for example a Docker run command to start a Docker container on swarm or misos it's gonna go first to a queue here and And then the the scheduler is gonna pick up the request from the queue is gonna compare with the offer that was received by Miso's master and the offer will contain information like oh I have like three notes available and this are the CPU memory That I can give you and the schedule will decide in which particular nodes these containers will be started it's gonna send the request to Miso's master and miso's master is gonna forward that to the agents in the nodes where this particular containers should start and It's gonna start as a task as a miso's task This docker containers with essentially the docker run and there is a particular component called the miso's docker executor That is gonna take care of actually starting that container is gonna also Remove that particular resource that has been used memory CPU and disk by that container from the list of up a liberal resources The same thing happened pretty much on Kubernetes and Kubernetes. You can imagine this time We're starting pods so there will be a request coming to the Kubernetes API to create for example a new pod and The request will go again to the scheduler that has also the available resources from Miso's and It's gonna decide where to schedule the particular pod in which host it should be started and then it's gonna send the request to the To the miso's master Miso's master this time is gonna forward to one of the miso's agents where if it's not already available is gonna fetch a Kublet that is essentially the component in Kubernetes that manage the Pods in each individual host and the Kublet will take care of actually starting the pod in the particular selected host So this pretty much what happens and we're gonna actually demonstrate to this Tony's gonna actually show how we built essentially this this environment and Demonstration how this all works together Thank you powder. All right. So what I'll do next is to show how this whole environment run in Opelstack Right. So first, how do you deploy it? There are two ways to do this one is by Magnum. This is an Opelstack project So with Magnum what you this is a few prep work to have to do manually we have to prepare to keep here the image and What you do next is create a Bay model and so in this case, it's gonna be a miso's Bay model All right, and with that then we can invoke Magnum to go and create the Bay So right now we don't have the support yet for Deploying the Kubernetes and swarm framework, but I'm working on a couple patch to do this So we should have that capability pretty soon Second way is to do this by Ansible script and shade So this is actually what we have developed in an IBM and for this We also have to prepare the key pair and the image and the private network to deploy the cluster into So to describe the cluster to build we Create a little config file for a simple key value pair to see to describe what we want to build and then We run the script your script would Use shade to talk to Opelstack and it would go off and deploy the Nova instance and so on Then it's going to log into one each of this instance install the software pre-reg and launch all the container that would provide the service for Kubernetes, miso's, swarm and all the supporting service like HCD and ZooKeeper Right. So the nice thing is that once this cluster is up and running in Opelstack, you interact with Kubernetes and swarm through the normal interface like Qctrl or Docker command line So there's no new interface to be used there Okay So let's take a look at the networking side are the way the implementation is right now the Supportful networking is too pretty simple. It's pretty basic. We are using the basic host and IP and port mapping So what it means is that the the container would take would not have its own IP it would take the IP of the host is running on and The port from the container is going to be mapped to one of the port on that host We but we are in the process of integrating courier There's a pair of talk on courier right now So what courier does is that For now is supporting swarm. So what it does is that it it mapped the interface from Docker lab network to Opelstack so when you run a Docker command to create a network a courier would handle the mapping and Create the corresponding Network in neutron for you and what you get back is an ID for your Docker network And then you would when you do a Docker run to create your container you just specify that you know put this darker Container on this network that I just created right so So with that on then what you can do is that you can achieve this kind of a Any kind of network to branch you want to do you can isolate the container as you wish and If you can even if you want to have a neutron router between them to if you want to to connect connecting some network to have this working on the community and And and mesos we do need a couple patch in mesos itself and these patches are right now under review by the Mesos community So that's for swarm, right Now for Kubernetes right now, we don't have support yet for for that kind of networking But the crew team does have a spec to implement the CNI and networking for Kubernetes and That's under review right now, and we expect that the implementation will be Be done in the Newton cycle Right Now let's take a look at the milk the storage side again similar to networking right now the Implementation is too pretty basic when we come up The default is to use a whole source, right? So when we create the nova instance, it's going to come with a certain amount of storage Depending on the in the flavor that we choose for the instance, right? So the document and demand is going to just use that source to host a container But to to manage a source a little bit more in a more flexible way then we would want to create Cinder block and and you know With the Cinder block then we can use the source in two ways First we can just use the normal we can configure the rock as a logical volume attach them to the host and provide the storage to the document to Host your container, right? So that's the easiest way and that's this is what my name is doing right now So that's that's this picture here a Second way to use the Cinder block is to use it as persistent storage for the container. So for example if we want to hold a database and make it survive the container then you would want to to Have some kind of persistent storage And there's a couple way to do to do this Kubernetes actually provide a plugin for for this purpose So the way the work is that you would create your Docker Your Cinder block and then when you write up your YAML file to describe your part You will just code that ID for the Cinder block in your YAML and what happens is that when you deploy the part and Kubernetes would talk to Cinder and tell Cinder to mount that blockstores to the host where the part is running and then you know the Blockstores will be visible to your part. So that's how to work For our case here where we have Kubernetes running on Mesos that's not quite working yet and we're working with the Kubernetes community to try to Support this. So that would give you this picture here where the container is able to use blockstores So looking forward a little bit the career team has expanded their mission so that they they will Support storage for container as well. And we just found out that there's a new project called Fuji That want to provide support for storage for container So so between the Magnum team the career team and the Fuji team We're meeting to I just submit here to to kind of hash out how we can collaborate But this is a pretty active area of work here. So on this chart we List a number of operation that are not well supported yet for Clustering in particular the Mesos cluster but we think that When you scale when you talk about scaling to a very large number of hosts in your cluster There's and this is a kind of operation that you really need to be able to manage your cluster So first you'll be able to have to be able to add remove note to scale and scale out. So that's a given Adding notes not too bad, but remove removing note might take requires some more care We might have to think about having to migrate container and so on So that depends on the the orchestration layer for community. Maybe it's easier because The the party community is a ephemeral so it can die You can kill it anytime and it would be recreated somewhere else But in the other case we have mad to give some more thought about how to vacate a host before we remove it Networking whether we we saw that we the user can you know create private network for their container So we have to be able to manage those Private network in some some nice way and we want to be able to connect to existing network as well So if a user come along and then they run some container and they want to connect to the existing network We had to provide that capability Same with storage and we saw about Block storage and you know user might come along with some Center block that they want to mount and make available to their container We will be want to be able to support that from the the cloud provider climate kind of review How we are it's going to fail so you know note is going to die and and so on So we have to be able to manage that we have to be able to see the the health of the cluster very large scale and When that one you know something have some wrong happen We have to be able to recover from the error and then software software will change also and they might show up as a patch hard fix upgrade and so on and They could impact and everything from the OS of a denote to the mesos base to the framework themselves or even the supporting Service like at CD or Zuki's keeper. So we have to be able to to handle this kind of a change to the System, okay So what I'll do next is show you a quick demo of how this is running on opens up You can get kind of feel of how what it looks like We do have this environment running in our lab and I could do a live demo, but we do have a lot of shows So to make sure that don't lose time but tapping command and time making type over someone. So what I do is a Show a recorded video and I'll pause and explain what's going on. Okay. Okay, so on this screen here. I have a Browser on the left side and two terminal screen on the right side where I would run command, right? So first we take a look at the I can a simple config file, you know So so here we're running the deployment by using the answer but script that we developed So here the is the config file to describe the cluster. You want to build right? So pretty straightforward You have the side of the cluster How many worker how many master and someone so pretty straightforward there? so what I'll do now is a Run the answer but script step-by-step, you know, playbook by people so that you can see what's going on But in a normal case, you just you know group all these playbook into one your master playbook and run it with one command so first we run the playbook to create the VM right so what is what happened here is that the answer but script you shade to talk to almost that and It's going to launch a number of VM here and you're going to see it showing up on horizon here Okay, so here are the three VM that we just launched For this new cluster So next we'll run another script to set up the Ansible inventory This is to tell Ansible how you want to configure your your cluster So once it's done, we can take a look at the inventory here and We see this is how we are going to configure the cluster so we have Out of the three node that we just create One of them is going to be the master and the other two is going to be the Work of the slave so the script also set up the host file in the Know themself because we don't have DNS running in our environment But if we do it to have DNS then then we would need that so next we'll run another Ansible script and This is going to log into each of the at the node to install software pre-req for For the node before we install missiles We could have done what we could have done also is to use this image builder and build a custom image with all those Package pre-installing to the image if we do that then we don't need to run this step So just so the way we will set right now okay once it's done now we can deploy missiles and This is a you can see it's very simple. We use a we'll run the Ansible Playbook and we just launch a Ansible script to go out and deploy missiles So the way missiles is packaged here is is a set of container So so what the Ansible script is doing is pretty simple. It's just go out and Launch those container for missiles So it runs should run pretty quick Right so it's done So with the missiles framework up and running what we do next is to deploy the framework Or so we start with the Kubernetes framework. So again, it's a very pretty straightforward. It's a running a playbook with the script for the key framework and again The framework here is such a set of containers. So mostly we're just launching container. Okay, so that's the key framework and We do the same for the swamp framework and like I said the this is just another set of container and up and running Okay, right. So Now everything is up and running in them the In the cluster So what I do now is I show you what the the resource looked like for this cluster in Opelstar, right? So here is a the security group So what I want to show here is a number port that we have open especially for this cluster so that the the Odyssey the service can talk to each other And here's the key pair This one is a little bit special because of the way Ansible work as well has to be able to log into all the node They managed as route. So the key pair here is prepare using the the route ID RSA of The Ansible host that you run from Here's the bit of the the network and this network we had created the manually before we launched the cluster and it's a private network and here we have the three node that we had created along with the Test node that I created earlier Right. So we have a router here that connect the private network to the public network And this is so that they view the host and the container can get to the external internet We do have floating IP also for all the holes so that you know, we can log in and when I access the Endpoint from the container Right. So this is what the mesos UI look like So at this point here, I had already run a number of tasks so you can see that there's history of those tasks here and If we go to the tab for framework You see that there are two framework running right now there that we had deployed the kubernetes framework and the swarm framework Okay, if we go to So if we go to the slave tab, we see that for this cluster, we have two slaves and then and then these are shown up here and Then we if we go to the tab for offer, right? so the way of missus workers that it make resource offer to the framework and So here we have a we see a number of resource offer that's been already made to the kubernetes and swarm framework And these are for running the tasks that start previously. So right now. We don't have any Active tasks. Let's go ahead and create a couple Container in the cluster. So we start by creating a Part in kubernetes So going to kubernetes, we do a kubernetes command to look at a node So this is what the kubernetes thing is seeing as the node for for its cluster So we see two nodes here kubernetes, one of them is not ready yet, but that's just the we find that There's heartbeat synchronization between the kubernetes in the node and the kubernetes API server in the master and Sometimes when you wake up it takes a little time to to become ready If we do a get part, there's nothing right now. We're running right now. If you do a get service, we see two service Running these are the internal service for kubernetes so let's create a part There are some running and so this is what the yaml for that part looks like. It's a very simple part so all it does is that is Launching this image engine x here, and it's saying that I want to map the port 80 from that part out to the outside Okay So we see on the side here that the task just pop up on missiles Now we for kubernetes we can't talk to the part directly So to expose the endpoint from the part from engine x we have to launch a service Right, so this is a service that get launched. It represents the part and here It's looking for this part here co-op internet And it's gonna map the port 80 from that part out to to the external sign so now The node the part is running and we look at the detail for the part We can see that it's running here and it's running on this host 10.0.100.121 And if we look at the detail for the service We see that it's running it's showing that on this host 10.100 121 is Showing up on port 5503 Now we need to go back So so the node he does have the phone point the defaulting IP of a 9.30.107.181 So now we have to open that port 5503 on this node You do that So once we open that port we can now go to that host with that port and We see and the next shown up there. Okay, so that's coming from the container that we just started in kubernetes So let's now do the same thing for swarm So same thing with foot swarm. We run use a darker command line here We do with a darker ps. Do you see what's going on? Nothing going on yet? So we're gonna do a darker run and Here again, we're going to run The internet image and we're going to tell it to map for 80 same thing So the doctor swarm Container the swarm container get launched and we do ps again And we see a detail for that container up there And now we can go back to the mesos UI. We can see that Both of the container are running. This one is from the swarm that we start from swarm and this one we started from kubernetes kubernetes both are running on the mesos bay Okay, so going back to the swarm container We see that it's running on this host one to one on this port one five zero zero zero Okay So we do have to map open that port in the security group of scope I'm not just not showing that step right now and then I do I'm going to do occur So what I'm doing is a little bit different from last time last time we accessed the endpoint from external So we use the phone the IP but here I'm going to accept it from the internal private Neutral network, so I'm going to use the the internal IP instead, but with the same host of one five zero zero Nice and we get back the the page from the internet container. This one is from swarm Okay So without you see that, you know, we we have a workload Running on the same cluster from both from kubernetes and swarm So what I do next is a pass back to Pablo you'll talk about how this scale Yeah, so we go fast on this one because we are almost out of time. So essentially what we show here is a demo where we Will we've actually deployed a larger cluster with about 15 nodes in soft layer using the same automation We saw before at them on open stack. We've done the same thing on the cloud in soft layer And we are starting you see there are two windows crawling down there. We actually started two Clients that Okay, for example on this side on the left side. We have a window showing Starting swarm containers on the right side. We are starting kubernetes pods and they are represented here You see these boxes. They are essentially nodes in the cluster. There are 50 of them and the green Hikons are essentially swarm Docker containers the blue ones are kubernetes container. This is a speed-up demo, of course It took about couple of hours to actually run 10,000 containers and About a thousand pods in there in the same cluster. This is just to show actually how can we share the same cluster? When we run at the same time both swarm and kubernetes and essentially mesos is managing as I mentioned earlier using the result software mechanism Then the ability to provide the now resources to both swarm and kubernetes to run this In this particular demo will say to do some optimization. This didn't really work completely out of the box and A lot of the things actually have done. We also pushed them back to community for kubernetes and Especially swarm where to do some work in particular on swarm to make this work the way we show here and Essentially at the end of this we pretty much get 10,000 swarm containers and 1,000 pods we didn't start a lot of pods because There are currently some restriction and the number of pods you can run today for each individual hosting kubernetes to do the way The polling is done. So this pretty much at the end of the demo and I think we have a last chart To finish So just to wrap it up. These are the community that we work with certainly Magnum is we're very productive in Magnum career to handle in a networking and Upcoming we will handle these support for stores as well Measles we do have a lot of contribution there and the game missiles Kubernetes framework for missiles and as well as a swarm framework, right? So these are pretty a lot of community that we work with you to build this kind of environment if there's any question we'll be glad to Ask them please come up to the to the mic, please Is this on yeah, oh there we go I have a question about if I'm I understand what mazes buys me if I'm using swarm and kubernetes if I'm only using one What does it buy me having these us there on top of open stack? Well indeed, that's why we actually show two frameworks It's it's a way essentially for me if I want to share the same large cluster To have different type of workloads as I mentioned we could have run also spark For example together So the idea is you know, I have this very large cluster and then I can deploy different type of workloads If I'm just running one particular work like swarm kubernetes by itself probably missiles doesn't buy you a lot Right because that that's pretty much the use we are doing sharing the resources and managing this resources For the workload Yes, this is where Magnum give you the option if you Just run kubernetes and nothing else at all in your cursor then then you might you would just create a cursor for kubernetes Okay, can I get another one well on here? Could I do without open stack? What is open stack? Absolutely, they give me that I that I can't get anywhere else. Ah, sure sure So open stack, you know give you the is layer. All right, so Compute storage and networking and so on so so it give you that that support You can certainly you know do all this without open stack in this No, not quite at all. In fact, this is one of the reason why we develop the as well script so that we can run In other cloud not the opposite cloud Excellent. Thank you For the use case of your service provider to a bunch of different groups Can this help us? run different workloads that Different containers from two different groups and you know assigned coders and things like that multi tenancy Actually, this is a good question We actually have done work in community, especially in swarm To enable there is not really multi tenancy today, but there is a new framework called out of Z for authorization That actually we have contributed in the previous page There was actually some of the contribution that we are doing that area and we have some way today to sort of have some kind of Multitenacy, there is some additional pieces that is needed, but at least some of the core components are there For Kubernetes is a slightly different story. They actually have already some way to enable that using Namespaces, so it's not really multi tenacy, but it is there are ways to essentially do some of that But also Kubernetes provides this a mission controllers that allow you to set up things like Quota, but you have to do additional work It's not out of the box like an open stack and swarm also is not really part of swarm If you want things like quota and stuff like that, this is all sudden needs to be added So we're actually working in this direction with the community to add some of these capabilities to these frameworks Okay, the question about Some we have a kind of very high level quarter concept like a resource limit You know pre-defined somewhere through the open stack or something else, but some of workloads Sometimes requiring very specific, you know performance guarantee Not only like okay, not only I'm going to use a 10 gig disk space But actually I need it to be guaranteed have like let's say 200 megabyte per second the writing performance Or something like that. So when there is a kind of, you know, the storage performance guaranteed or networking side bandits Guaranteed or latency stuff if you combine the oldest possible different variety of workload or in this one frame way and you know the question is that right now is there any way to Control such a sophisticated workload Through the Mesa's a framework or it's not still available some you know communities working on that So I think yeah This is a very good question asking about really not just assigning results But also guarantee that that resource is available and this actually requires additional pieces is not something that the Mesa's provide you out of the box We're not actually doing that here. We're not showing that but there is work in terms of using even linux C groups capability or stuff like that to actually be able to to actually ensure some of the Result, but this is all additional work. It's not something that is Already there parts out of the box today So we actually have some work in the direction, but we're not sure in that here But you know to add a little bit to that the Mesa's give you that capability in the sense that It it gives you resource offer and it's up to the scheduler to to expect that resource offer and accept it Another so if you see a resource offer that you don't like, you know something that doesn't meet your your need Then you just reject it and then wait for another one to come along Yeah, but the thing is that that resource metadata information whether or not including such a performance guarantee You know, that's a very tricky because if you run Some VM on one host at the somehow happen to be on the same host They already Contend each other for getting the story G performance then there is no Thank you. Okay. Thank you So, I think we're out of time, right. Thank you. All right. Thank you