 OK. Hi, this is Jaisal Gan. Welcome to our talk. So today, we are talking about putting OpenStack on Kubernetes. So we are three. We present together. This is Jaisal Gan from SK Telecom. And we have the Shingyuan from the SK Telecom and Will from Solinia. So what did happen today's talk? So I'll just give an introduction about the background and our motivations. And then there will be some more detailed explanation about how to set the Kubernetes and OpenStack together. Then we will just start demo. Demo is about actually deploying full OpenStack on Kubernetes and then showing if one node fails, what happens. And after start demo, since it will take some time to deploy OpenStack about 708 minutes, Will will talk about our CI system leveraging containers and Kubernetes all together. So let's start. So SKT is number one mobile service provider in Korea with 50% market share. And I mean, the responsibility of stuff we are doing in telco area, especially in Korea. Recently, we just opened 4G LTE 5NCA with max 700 and VPS. So I think we are like a front runner when deploying like a new network infrastructures. Also recently, we are really focusing on AI and media. So because of that, OpenStack becomes a more important and narrow company to support not only network, but also media platforms or the AI platforms and other IT services. And also SKT is very active in various open source community, including OpenCompete project, and Telco Impala project, and Honest, NSAP, and OpenStacks. And also the Serenia, the company we are working with, they are like a professional service partner that accelerates enterprise cloud adoption. I'm leading this through. And they're very technical and agonistic. And they have various clients with Global Fortune 1,000 organizations. And for the personal experience, the Serenia has very good engineers and developers to work with. And before going into more detail, our work is totally community effort. And we have like 11 people team, a small team. And the wheel provides a very good value to us. And to make our work possible, we get lots of support and help from OpenStack Heron project and COLA, and also the large contributing OpenStack operator working group. So if you don't know about OpenStack Heron project, this is like a new project in OpenStack. It started by AT&T developers. And it's using a technique called the Heron to provision the containerized OpenStack on Kubernetes. So I think there is a bunch of talks throughout this summit about this Heron project. So if you are interested, you can find other talks about this one. So what makes us to work on this Kubernetes stuff? The current or soon-to-be-previous way of doing OpenStack was that, as a technical, we don't have enough resources. We don't have enough developers or engineers to do our own OpenStack. So we decided to work with vendors as a user. And that means vendor will provide OpenStack package and a way to do the configuration management and deploy automation. And our job is to put the requirements and working with vendors to set up the deployment architectures. And we have to purchase hardware and appliance. And once we deploy OpenStack, then operation will happen in the collaboration between us and vendors. And in this way, SKT just need to put small number of very capable people, working with vendors, to manage OpenStack services. But so we considered building a fail-safe environment, I mean, good OpenStack environment. We focused only in the deployment page. But we realized that's not the answer and that's not the effect to make a fail-safe environment with OpenStack. So problems we had. Well, vendors provide a very good way of automating deployment. But the upgrade is like a no, what I can say about upgrade. That was like a very difficult. And even updating a patch, a small patch, was very challenging for us. So without any way to apply patches very easily and without interrupting any running services, we had problems to apply most up-to-date the security patches or the patches to solve the problems in our environment. So that becomes our operational problems. And also, in our company, we don't do a single huge scale OpenStack, but we do a lot of the small or medium scale OpenStacks. So we ended up with having a siloed version, or saying snowflake environment, with various vendors, with various versions of OpenStack in our company. And it becomes very challenging for us to manage those all the different OpenStacks in our company. So we wanted to have the better way to manage that. And also, we found out that there is lack of flexible configuration management capability from the vendors, meaning that if our requirement shoots their reference architecture, what they provide, we have no problem. But if our requirement is deeper, so we have to set up very differently from what they offer, then even if it is a small changes, it becomes a huge problem. So we really needed a more flexible way to do more configuration based on our needs. And also, it was very difficult to integrate with our own staffs. So we have our own team doing the SEP, and oil price-based SEP appliance. And we have our own team to develop as standard control based on the owner's technologies. And we do have our data center operation platforms, and our monitoring platforms. And sometimes, integrating with the vendor package that was very difficult for us. It was very challenging for us. So we really wanted to have solved these problems. And then we realized that not only focusing on the setting of requirements and setting up deployment architecture and trying to do the better operations, we actually need to make our own configuration management tools and better deployment operations. And more flexible configuration. And patch upgrade should be more easy. And scale out should be more simple. And we realized that to do that, we need everything. So at the end, everything forms continuous loop to accomplish your fail-safe environment, like from active community environment, and doing the CI pipelines, and setting up a CD pipelines, and deployment automations, and like OpenStack lifecycle management, upgrade, update, rollbacks, and everything, that has to come all together. Otherwise, it will be very difficult for us to solve this problem we are having. So we decided to look at some platforms to help that. And we used Kubernetes and put OpenStack on Kubernetes. So what we're trying to do is we need to find a better way to deliver OpenStack and manage its lifecycle. So we want to reduce the overhead, like a dependent management. And one have its fast multiple deployment in a standardized way. And upgrade, update, rollback should be easy, and easy scaling, and healing. And finally, we decided we need some underlying platform to do those OpenStack lifecycle management for us. So that was Kubernetes. And key technology we are using is also Kubernetes for control plane orchestration, and Helm for the application lifecycle management automations on Kubernetes, and CI CD pipelines leveraging Jenkins, and OpenStack Helm project, and Kola, and Onos, and SAP. That was our key technology for us to use and to develop this OpenStack on Kubernetes. And our plan, our goal is very big. So we don't want to do the red POC. And we want to do the production ready by end of 2017. And we'll do the first production within 2017, targeting the private cloud in IT infrastructures. And we will expand those to the more deployment, including media platforms and NFP deployment based on OpenStack on the Kubernetes. And once we have those very stable team line to provision OpenStack on Kubernetes, we want to put more than OpenStack. We want to put other platforms, other apps on top of Kubernetes. So we can have those Kubernetes as a common underlying platform to deliver cloud apps. So that's our plans. And I put some diagram for our overall architectures, but I don't think I can have time to describe everything in detail in this talk. So I mean, we are happy to share everything we are doing. So if you have more questions, you can contact us after talk, or you can send emails. So the upper one is our CI pipeline. We thought having a very good CI pipeline is like a key success point for us to do this OpenStack on Kubernetes. So we use bunch of community upstream code. And we sync up with our own repository and trying to override what we need to set up. Then we use Jenkins to set up the containers and have them which is automation code and doing all the unit tests and doing the integration test and promote and have a stable version of package. And it should happen automatically via Jenkins. So we have been putting most of our resources to set up these pipelines. And then we have this deployment in the downside diagram, saying that we want to put the OpenStack and onus and compute node and steps stories and some of the network gateway on top of Kubernetes. That's based over architectures. And for the specific demo system we are using today, there is Jenkins. And Jenkins lab has those CubeCitter and Helm CLI. And through the Jenkins, we deploy OpenStack on the Kubernetes work node. There will be three node configuration for the controller and one compute node. And we are actually dynamically pulling all the containers and Helm code from our repository. So from this moment, Sunggyuan will describe more details about how we set OpenStack and Kubernetes all together. So I'm going to explain a little bit Kubernetes and OpenStack. So first, let's talk about what will be targets for high availability. Starting from Kubernetes, first, the Kubernetes stores the necessary metadata in the key value store, SCD. So SCD consists of three nodes in stable domain. And the Kubernetes API handles requests from external client or internal parts. And load balancer, as you know, is used to properly distribute the request and operate very safe. And the scheduler is responsible for properly distributing the part to the node. If it is configured as three, one of them will be elected as a master using the read election. And the read reaction method is the same as a controller manager. Let's talk about the OpenStack. If you have built and run OpenStack, you know what parts of OpenStack you need to achieve. The openStack project, SKT, we are using Keystone, Glance, Nova, Cinder, and Neutron. The circuit is not used. Like Kubernetes, the OpenStack configures the HA mainly for the controller and the things API server first. The API is solved by load balancing multiple servers or openStack projects. Both have an API server and two or one more API server. Both Nova and Cinder have a scheduler. And these servers communicate with the queue so that they can be populated in multiple ways. MariaDB used the glada to configure the cluster and select the master through the column board to synchronize. And Web thank you used the auto cluster plugin and used the SID as a backend for node configuration. Neutron network node HA is always the most sticky one. And in SKT, we use the onus-based SDN controller called Zona to handle this problem. Onus is an open source software to build carrier-grade SDN controller. I will describe more detail in the following slides. When you put OpenStack on Kubernetes, things will be a bit different. Since Kubernetes natively has auto healing capability, you know, the OpenStack on Kubernetes can be run without specifically configuring HA and regarding user VM availability. Kubernetes will do it for you. That means the Kubernetes will automatically recognize a process failure in OpenStack control planes and regenerate the container with that process to heal itself. Since running VM is not affected by a process failure in a control plane, a user will not notice any problem. Kubernetes usually auto-detects node failure depending on the setting. By default, it could take up to five to six minutes. If you want to make sure our OpenStack API is also not interrupted from any possible failure, then you have to consider apply a specific HA setting like an Azure setting Galata cluster. It requires both specific container Kubernetes and computer for the proposed OK. I will look at the each one in more details. This diagram shows how to deploy the Kubernetes nature rate. The green color, kubelet, and flannel are displayed as a system process. And blue color control measure, and scheduler, and APS server, SCD, kublux are executed as a path. The flannel is very simple to install, but there is an issue that network performance is degraded because the data is encapsulated in VXLand to enable network communication between nodes. However, there is no big issues in network performance because it is used for communication between control plane. There are several options of kublux networks, but it has its own pros and cons. So we are to decide to use the flannel issues for our needs, since we are only using it for network controllers. And so now we'll take care of the VM network. That's why we are using just the Flannel network. You have to find out what type of Kubernetes network flagging is good for your requirements. And we have another set of building separately in Kubernetes. Set for a set storage, the volume is created by utilized dynamically by using secret and storage classes. In order to easily build kublux into multi-master, you need to load the module as a path. This is about the kublux. In the absence of APS server, kublux takes charge of it. So kublux is the model that allows the system to process in load and manage load information and creates a managed path. This option is more detailed. The host override options can specify the name of the node. In this case, it's a kubemaster01. And you can also use the path manifest path option to specify the directory, where the YAML file to create the path is located. For example, the SCD and APS server, control major, scheduler, kublux is all created in YAML file and run as a path. Service when created, it is registered in DNS. You can commit with the path using service IP. Specified IP of the DNS server to be created rather with the cluster DNS options. SCD is a key value store with three cluster paths for stability. In this example, we cluster the three kubemasters. All information of a kubemaster such as node information and pod information is stored in SCD. The initial cluster option specifies the host IP, as you see, to configure the initial SCD cluster. API server is used to store or call information in SCD, so set the location of the SCD server. And also, the important option here is the automation controller, which is covers many policies. If the secret context deny value is included here, you cannot deploy a demo set with a regular user account. For example, it's a Nova compute, a report, and a neutral agent specified as a demo set among open-to-components, but they cannot be distributed. So the automation controller is very important. And since we are using kubemasters only for managing open-to-process pitch means of the user, we don't access any container at all. And therefore, we only utilize authentication with token auth file. And that authorizes such as rule-based access control. Furthermore, since enabling authorization complicated our CI system to dynamically setting a kubemaster open stack, we decided to use authorization. In these specific cases, using kubemasters only for managing open-to-processes simplify the wins over security. And when setting up as cluster, cluster manager and scheduler must select master server or through the leader election. The Kubernetes control manager is a demo that embeds the core control loops to shift with Kubernetes. The example of the controls that shift with Kubernetes today are the implication controller, endpoint controller, namespace controller, and so on, which control is created as a go routine. And described in the previous slide, the scheduler is also master as the leader election. Cube proxy is a response for managing IP tables. The two handle IP tables, you must set the previous value of the security context to true here and read the characters. And the cluster IP used by the service is a virtual IP. That is not the IP recognize the physical switch, but the IP table is through the cube proxy. To access the part from the host via service IP, you can access by specifying the IP CIDR and cluster CIDR options on the cube proxy. Now let's talk about how you deploy open-to-processes on Kubernetes. You set every node as a Kubernetes worker, then label them either as a controller or a compute. In this picture, we label the four nodes as a controller and others as a compute. Then when deploying through Kubernetes, you can specify the way to deploy through node selector. If you install open-to-processing, the usual way you will use three servers to configure the control node as HL and so on. So for the stateless processes, with the pre-cut settings, Kubernetes will take care of its failure with auto-healing and managing scaling out and process. The keystone services, you just need to decide how many of them you want to run, then let Kubernetes manage the rest of them. Of course, you can always change how many of them you want to leave. In other words, you can increase or decrease the API server more easily in terms of the capabilities. So if you have four nodes labeled as a controller and have a Kubernetes deployment with three parts in the resource set, Kubernetes will deploy three Nova API or any open-to-process like here. And when a node fails, the Kubernetes will move part to another available node. Therefore, it always keeps the same number of the Nova API. I will show this like a diagram of how working in demo. So basically, we use a galeric cluster for MariaDBHA. The first time we create a job called MariaDB seed, this step runs MariaDB Galeric Part and does initial setting to the configure Galeric cluster. Then you run some of the MariaDB Galeric Part and dynamically configures and joins it into a cluster. So first, the MariaDB seed. And second, the MariaDB 012 is a sequence. So it's running. So once all three MariaDB parts are set up successfully, MariaDB seed job finishes. So we can no more see the MariaDB seed, the part. Since it requires to do dynamically joining a new node, it uses the joiner. For example, when MariaDB 2 is initialized, it's a third MariaDB, it sees three cluster addresses here. Like one is a seed and MariaDB 0 and MariaDB 1. The diagram is now showing is one way to avoid problems if you have just only one physical network port. So when you install the neutral with the open tech harm, you have to do some trick like this. The neutral has a network interface to BREX OBS bridge to communicate with the external network. In the case, the host can no longer communicate with the outside because the port is using an OBS BREX. So the VR data at the top creates the RIMS bridge and the RIMS virtual interface in yellow color in advance. So after creating a pair of virtual interface in advance, virtual redundant one adds an interface to the VR data bridge. And the other virtual redundant zero is reserved for VR EX to add a port. So like this, you can then use one physical network port. This is about the open tech ham neutral chart. We are the override that some of the values. So upstream, which is updated quickly due to the content of the month ago. So may have been slightly changed in content. Even so, understanding the neutral chart will help you a lot. This is the Sona is an implementation of open flow based neutral ML2 mechanism driver and entry program with an open flow controller. The major features are the no agent. The no neutral agent is required. And beyond traffic, the ARP and DHCP gets handed by the controller. And with West traffic is handed a compute node. And they provide the load distribution toward North-South traffic. And the state-preferable gateway node. Basically, Sona manages the virtual network state to receive from neutral ML2 driver and set flow rules to OBS on compute and gateway nodes based on the state. That indicates any kind of data plane. Failure is recoverable as long as the controller is active. If I have any chance, I will talk about the Sona. Now I distribute the hand chart and the different open state on Kubernetes. I have created one Jenkins Java and it can be deployed at one time. I will run it and deploy it takes maybe eight minutes. So during the process, we will continue to explain the CI. This is our Kubernetes that there is no open state part. I'll create the demo in our Kubernetes cluster. This job is just to install the hand chart. And we can see it like this. And we'll just keep going to the presentation. All right, so that's going to take about 10 minutes to run. So in the meanwhile, I'm going to entertain you with our CI configuration. So on the left here, we have Jenga. Jenga lives in your living room. It's probably less than an hour old. Otherwise, it would have been knocked down already. On the right here, we have the Volote World Tower that just opened up in Seoul about a month ago. It's 123 stories, half a kilometer tall. It's pretty impressive. During when they opened it, they actually shot fireworks off the sides of it. It was pretty, pretty neat. Why is this relevant? I guess it's all kind of the foundation metaphors. So obviously, you want to build a really strong foundation. And I think this is especially true when you're layering your virtualization like we are here. Anytime you have something built directly on something else, you now have two somethings in there that breaks. So that's sort of the takeaway there. So inherently, something like this is going to involve a lot of moving parts. So it's kind of fun to actually visualize those numbers exactly. So right now, we have 34 local Git repositories, Give or Take. Those are mostly internal projects and wrappers that we use. In addition to those, we have 10 upstream repositories that we actively sync. Eight of those are OpenStack projects, Keystone, Neutronova, all that. They're pretty stable, consistent, not a lot of change. The other two are Kola, which is also fairly stable. And then there's OpenStack Helm, which, just given its age, is not what I would call stable. It's actively developed. They fix stuff when it breaks, but it does break for sure. So then deployment configurations, profiles, the number of ways we deploy stuff. Right now, it's four. That's going to get bigger in the future as we look to support Onos and other configurations there. Right now, we have three hardware-based environments and one or more virtual environment at any given time. So that's probably not going to change too much. Charts for the deployment, 12. That one's interesting because that's the bare minimum of charts you need to deploy, bare bones, open stack. So that's no heat. That's no metering, nothing interesting. That's just the bare bones stuff. So that's 12 charts. Docker images for every deployment, 23. Obviously, the deployment here is OpenStack deployment. So that's actually 23 images that we were building and maintaining to support that installation. Actually, 22, because we don't actually use the Kola rabbit image right now. We're still using the Mirantis one, I believe. Pods and jobs in a single deployment. One single open stack deployment is 34 pods and or jobs. I think the split is like 10, 15 jobs and the rest are pods. For the HA, it's almost, but not quite, strip all that because, obviously, you're not going to just multiply everything by three. There's going to be some jobs that are just going to be run once per clustered environment. The other interesting thing is for stuff like LibVert, one LibVert is three pods. So that's something when you think about it. It's pod per service. So you have that as well. So what kind of workflows are we running? A workflow, by our definition, is one or more jobs and or physical activity interactions that are all chained together. So we, like I said, we build all our Kola images. On a good day, we're in the green for both Ubuntu and the CentOS ones. We build and test our own charts. With the charts and the Kola stuff, we try to track upstream as much as possible. We would actually have many changes that we're interjecting just yet. At least with the Kola images, it's mostly just metadata, couple SK specifics, repos, that kind of stuff. We have testing workflows, obviously test things. And we have our upgrade and deployment workflows as well. How am I doing on time? When did you start? So kinds of jobs. So those workflows would then reference the various commits. And that's obviously going to be your basic validations for any sort of changes that you're pushing. There's going to be live system deploys, which is going to test the update functionality. Our nightly builds are mostly just a CI system validation as well as catch anything that might have slipped through the cracks there. And we have a nightly rebuild as well so we can get the clean slate. And additionally, we try to vet out the upstream builds before we merge them. That's not always the case, and that's something we need to work on. But it's something we try to do anyway. And yeah, so interestingly enough, the single biggest problem we've had recurring has been the Docker storage drivers conflicting with Kubernetes config maps. And they either deadlock or they do really bad things. But if you look at the Jenga picture we had, that'd be like the second block up on the architecture there. So once that falls down, everything else kind of goes tumbling over it. We started out using CentOS 7 with the 3.10 kernel on top of XFS with the overlay driver. Probably the worst possible combination. We've tried a bunch of other stuff, and it's better now. So nothing here is specific to this project. It's just generally good DevOps-y CI stuff. People underestimate how much a fork costs. So you should really not fork unless you're willing to accept the commitment of a full-time project. Isolate all your builds. That was another big one that we hit. We naively tried to run everything where we deployed it and discovered that you cannot run more than one instance of Okupin stack on a Kubernetes cluster without special attention, which we were not giving it. So it's best to keep those builds isolated. Incremental upgrades don't change one thing more than once, and I know this, you know this, and I'm saying it once again, because I'm gonna screw this up. Like the morning when I decided we should upgrade to Kubernetes 1.6, use our back and upgrade to at CD3. That was not a good day. It didn't work, by the way. All those things individually work, but with our system not so much. And just rebuilding all the time. You can never rebuild more than you need to, for sure. So just a quick testing on what we test, how we test it. We're trying to look at the Kubernetes end-to-end stuff. Obviously, if Kubernetes isn't working, then nothing else is gonna work, so you wanna make sure that's stable. Keeps sure it's not a test. I get it. That said, we do use it to validate things, and we have our own custom scripts there. For the cold container builds, we inject some batch tests just to validate users, processes, executables, and then we also use Clare for the static testing analysis. And that's, Clare's a nice idea, but the actual value I would argue is minimal, but that's, I could explain that later if anybody cares to argue. Testing, so Helm has this wonderful command called Helm test. It's a little bit rudimentary, but it will allow you to actually execute tests. This is especially important in the case of, if you're doing OpenStack Helm and Kola, because OpenStack Helm injects all of its configurations, so you can't actually really test a Kola container in a vacuum there. A lot of stuff has their own built-in tests, like Ryzen, so that makes it super easy there. And ultimately, once everything stood up, then you got your full tempest runs and rally as well for validations. And once stuff is actually successfully deployed, you know it works, there's still more stuff that has to happen there. Just as far as, you know, does it work? Can you log in and click? And is the user experience there? Is it performing enough? All that stuff. And then the failure testing is actually kind of the interesting part, because once you have that at the double stack there, it's, the failure cases is like almost exponentially bigger just due to the fact that stuff can fail on a physical or on a virtual level. You have multiple networks, you got, it's even better than a triple O because you actually have the different virtualization technologies there. So the rules definitely change as far as like how you test how stuff breaks. And ultimately kind of the end goal is here is to make upgrades as boring as possible. Once again, you know, this is just DevOps stuff, you know, test, test, test. In this case, you can make testing a lot easier. So that should have been about 10 minutes, hopefully. No. The update is already running. Maybe it takes the six minutes so we already spent 11 minutes. And now I'm the create the VMs. You can see that? This is the, we are creating the VMs before we VMs resetting the router when it's created like this. And we are now we are... Due to the time limit, I mean, we are not going to show when we failed one node what happens, but the process authority will be created. So just summing up, but still we have lots of challenges in here. There is like operation burden. You have to do the Kubernetes OpenStack and OpenStack is not cloud native app. So there will be lots of customized way to you have to do on the Kubernetes. That's still be the problem for the operation and just making this technology better. So we probably need to discuss how we can make OpenStack more feasible cloud native way. I know there is some discussion going on that part. And also we need to very focusing on Kubernetes stability and the security problem. So there are lots of challenges we have to overcome, but we believe this is a very feasible technology we can put on our production. So this is the end of our presentation. And thank you. And if you have any question, you can come to us. Thank you.