 How's everybody doing? Enjoying the summit so far? Cool. It's about to get better. We're here to talk about OpenStack control planes and how we can make them a little bit more resilient and self-healing with Kubernetes. So before we go into the intro, I just want to say that we're all co-workers that we and friends in the container community. And we decided to come together to work on a proof of concept in our spare time to help solve a common community problem, which can we make a self-healing control plane. That being said, I'm Derek Chamorro. I'm a cloud security architect at eBay where we try to containerize everything we deploy. There's my Twitter. I blog primarily about lab stuff and a lot of virtualization. I've been an OpenStacker since the Folsom release. And for those who know me, I'm a container in Kubernetes. I'm the founder and CEO of a startup company called Cloud Perception in RTP North Carolina. We provide application-centric solution through analysis and interpretation of sensory information with actionable outcome. We're also the organizers of a triangle Kubernetes made up. So today, we have over 120 people join us. We also have a great help from OpenShifter team and also Automatic team from Red Hat. And as a local community from the different region, such as Charlotte, container orchestration made up, and also New York City Kubernetes made up. And I'm Randy Tuttle. I'm a co-founder and CTO of Cloud Perceptions. I've been in software development in the early part of my career, switching over to systems and solution engineering regarding capacity and high-availability systems, particularly around a service provider of VoIP and a service provider of video systems. I've been playing around with OpenStack since Folsom and also along with Shishon, also a co-founder of the triangle Kubernetes Meetup Group. So moving to our agenda, we've basically broken this into three parts conveniently so we can all speak. The first part basically sets a bit of context about why we're doing what we're doing. The second part is going to talk a bit about the approaches we took. And finally, the third part will be the demo and is what we actually did. And then we'll wrap up with a bit of a summary and then some time for Q&A at the end. So what is our vision? What do we think would be the perfect environment for an OpenStack control plane? Well, first of all, we believe it should be easily deployed. So with minimal steps, we can have a functional OpenStack environment, spending fewer man-hours and requiring minimal expertise, especially around configuration and troubleshooting. We'd like for it to be consistent and repeatable. So any deployment should be repeatable and easily replicated across organizations with consistent outcomes. We feel that with consistent outcomes, we can spend less time troubleshooting those deployments. We'd really like for it to be quickly operationalized. So with OpenStack deployments, we need to overlay operation stacks to effectively monitor those deployments. This takes a lot of engineering skills. But ideally, we could have a fully functioning stack in five to 10 minutes. I mean, that'd be perfect, including the ability to monitor and do straightforward in-service upgrades. It should be easily and quickly. We should be able to drop in some value-added features. It should be simple to scale. We want to be able to do some hands-off. You don't want to touch it. We want to have it expand to a maximum scale to meet demand and then retract when that demand has subsided back to a guaranteed scale level. Surely it should be able to self-heal upon failure. In other words, we want our primary focus to be on maximizing availability. And secondarily, we'll address what the cause of failure was. So then with more uptime and confirmation that the service is performing at the level expected, we expect it would see improved SLAs. So for today's focus, though, we're going to focus on these main areas. So what are some of the OpenSec pain points that we sort of deal with? Well, around deployment, we want it fast. When we talk about fast, it's just synonymous with automation. And furthermore, I think most people would agree that automation is table stakes and DevOps. Some automation tools are currently available, such as Ansible, Chef, Puppet, et cetera. And these have really emerged to aid and speed deployment, some of which a few OpenSec projects actually utilize. But these have an appearance of being somewhat adjunct to the core of OpenSec and still are time-consuming to utilize. Despite improvements in OpenSec documentation, they're still targeted more to a generalized deployment approach with indications of what knobs and things to tweak to add flexibility. So while OpenSec is quite flexible, this is flexibility that organizations sometimes struggle to overcome when adapting to their own deployment architectures. And so companies are sort of left with this do-it-yourself approach, or even turning to many of the turnkey solutions that are out there. And of course, all of them require their own skill sets. So then we run into these inconsistencies between our development environment. Last week doesn't work this week for some reason. The different dependencies between development, staging, production environments can lead to breakages all along the points along the way. And we could argue that some of this is probably attributable to complicated organizational processes, which tend to have many moving parts and many points of failure and great opportunities for miscommunications all the time. And then another area we see is operalization. First off, what is the appropriate engineering? Typically our systems are overengineered for worst case demand, which can result in a wasted or underutilized capacity. We have the traditional monitoring tools, which really not enough anymore. They sort of give us this reactive rather than a proactive approach. So we need to bring in more advanced tooling to collect these metrics, these use metrics, and what I mean by use. I mean utilization, saturation, and error rates. These kinds of metrics can help us better trigger failovers and adapt to scale of demands. And then, of course, we want to have some level of upgrades. At some point, we're going to have to upgrade our application. And ideally, that operation should be non-disruptive or have very minimal downtime. But when we bring in the advanced tooling, of course, it can require a specialized skill. So going back to our vision, in terms of these three main points, we'd like for our solution to be easily deployed, quickly operationalized, consistent and repeatable across DevOps, and finally, self-heal the primary focus on availability. OK, so since the focus is this presentation around open-stack control plane, we'd like to touch on a couple of those pain points we consider problematic. So up front, we have this investment in pre-deployment planning and engineering. First, we've got to determine the anticipated demand and sizing requirements prior to the instantiation of our control plane. This capacity planning skill testing is significant, and it requires a lot of upfront man hours, which can delay our time to market. Systems tend to be over-provisioned because no one really, really knows what the demand will be, and that adds cost. And then we have this inadequate post-deployment problem. We can't really monitor that well. So again, touching on DevOps, we know that DevOps organizations must utilize sound monitoring practices. These provide teams with solid feedback, where adjustments may be needed. As we said in our vision, the primary focus is on service availability. Failure causes are secondary. Nevertheless, the insufficient use metrics, the utilization, saturation error rates, makes it very difficult to diagnose and troubleshoot service failures. And then we also have the lack of these built-in API health checks, specifically application or service-centric health checking. And then we lack service healing. So a lot of systems don't really have these built-in service healing capabilities. Sometimes we have to just drop in an overlay of automation to achieve that. Again, self-healing should be our end game. Again, with our focus on service availability, and the cause of the problem being secondary to that. Then we have lack of elasticity. So as we mentioned earlier, the systems tend to be a little over-provisioned. And so when we have to add additional capacity, expanded the system may result in more capacity than we really, really needed. And then we have to go, of course, stitch all that in. The other side of that coin is that all this added capacity that we just added in is not really removed when it's no longer needed in a lot of cases. We just leave it stranded there, and it's not really ever utilized. So by having some level elasticity, we can spend less time up front with the engineering and planning and move more quickly to a service deployment, and then we allow our auto-scaling system to handle elasticity around the predetermined policies. So just quickly, we'll drill down on what really makes HA control plane, and where do our problems manifest themselves. So with any HA control plane, we know that our goal is to eliminate a single point of failure. So let's look over some of those elements and discuss some of the previous invention pain points. So at the top, we have our load balancing services. Generally, when we add a service or expand it or contract it, we have to go in and manipulate our load balancing service. And this can be a tedious, error-prone manual process, or maybe we're lucky enough we have some sort of overlay orchestration to do that for us. But again, that still requires specialized skills to drop in. So we'd prefer it if our load balancing services could update behind the scenes without us being involved in it. So then we talk about what about control plane failover. So all of our control plane components are HA, as we've noted. And a lot of the services in OpenStack we distribute them across the various controller instances. So this works pretty good under optimal conditions. But so if we're near or at our engineered levels or a single fault occurs, then we might have some problems. And we'll lose all those services kind of get lost on that one node, for example, because we tend to bunch all those redundant services over on that one controller. So we're sort of in a weird air condition where we've got services sort of at some level of outage going on. So what we really like is some sort of self-healing control plane that we could know with confidence the service would function at maximum availability regardless of any failures that may have occurred. And so what about upgrades? This is some of the same characteristics of a failure, but in a bit of a more controlled fashion. In other words, we can plan for it. So nevertheless, upgrades tend still to be disruptive, which can make the HA service partially unavailable. So you can work around these to avoid these disruptions. But again, we have to put temporary measures in place. And that requires, again, additional man hours to do all that. So it'd be better if we could just sort of selectively pick what services get upgraded and when we want to do it. So then, again, just touching a bit on the topic of scaling, when we build out our control plane, we sort of take, again, our best guess estimate. And eventually, or maybe we might even use some empirical data that we have handy, it can help us predict what the anticipated demand might be. So while that's imperfect, it can lead to over or under engineered systems. This can then manifest in poor user experience, for example, high latencies, or because you're under engineered the system. Or you might end up with some underutilized capacity. So it becomes a real challenge to find the right balance. And operationally, it becomes onerous to make all these changes. So we'd prefer if our system could sort of scale up and down on demand whenever it needed it and operate within a minimum max range. And then it would place the workloads when and where they were needed. And then finally, on the topic of service troubleshooting, it's sort of difficult to isolate all the instances while trying to maintain that maximum service availability. So next, I'm going to hand it over to my colleague, Derek, who's going to talk a bit about Kubernetes. And while we believe the attributes of Kubernetes and the attributes of containers in general can accommodate many of our needs and help us with our pain points. Thanks, Randy. So before I go into our proposed solution, I want to do a brief primer on Kubernetes, covering some of the core concepts as well as some of the overall architecture that we've used. So first, Kubernetes, what is it? It's an open source automation framework for deploying, managing, and scaling applications via Docker across the cluster of hosts. It was started about over a decade ago as an internal container management system at Google. It was codenamed Borg. It evolved over time, infused with the lessons learned from a decade of their experience and managing applications and scaling them out. It was open sourced in 2014, and its current version is 1.2. So what's the advantage? Well, first, it's declarative. It allows your applications to be deployed quickly and predictably. Kubernetes works to monitor the current state of services and synchronizes it with the desired state as defined by you, the administrator. What does that mean? That means that if an application container temporarily goes down and you declare that you need three copies of that application, it's the responsibility of Kubernetes to start up another container, scale. Applications are scaled on demand instead of requiring resources to be allocated statically. The current version of Kubernetes can scale up to 1,000 nodes and up to 30,000 containers per cluster. It's easy to build. It's easy to update. You can build a fully functional self-healing control plane that you'll see in our demo later in as little as five minutes. You can seamlessly roll out new features to existing deployments without the need for added downtime. And it's efficient. So you're using only the resources you need. Thus, you avoid over or under provisioning and increasing your average server utilization rate. So why does it benefit the OpenStack community? Well, it's what this talk is pretty much based on, it's self-healing, auto-restart, auto-scale, auto-replication, those are features that are needed now and not tomorrow. It's consistent. It's consistent with your build as well as your resources. You pragmatically and efficiently only use the resources you need on demand. As well as improved SLAs. Kubernetes comes with built-in API health checks, specifically application and service-centric health checks, as well as self-healing services that lead to greater uptime of services, and that's always a good thing, right? So let's discuss the architecture a little bit more in detail in the next slide. So the controlling unit within the Kubernetes cluster is called a master node. It serves as the main management contact point for administrators, and it also provides many cluster-wide functions for its worker nodes. And when I say cluster, a cluster is a set of compute, storage, and network resources where applications are deployed. Some of the core components of the master node are at CD. At CD is a distributed key value store for shared configuration and service discovery. And that CD provides the ability to watch for changes and represents the state of a cluster that each component can reference to configure or reconfigure themselves. We have an API server. The API server exposes a restful interface that processes operations, such as creating pods and services, and updating the corresponding objects in that CD. We have a scheduler, and what it does is it schedules things. The scheduler watches the API server for unscheduled pods and schedules them onto healthy nodes based off of resource requirements, as well as tracks resource utilization on each node. And then we have a controller manager, and it manages a few different controllers, primarily the endpoints controller, which manage service endpoints, and the replication controller, which manages the ability to scale pods across the fleet of machines to ensure the desired number of pods are always running. Then we have a worker node. The worker node runs all the components necessary for running application containers and load balancing service endpoints. Nodes are also responsible for reporting resource utilization and status information back to the API server. So the components of a worker node are as follows. We have a kubelet. The kubelet service communicates with a master node to receive commands, and these commands are received in a form of a manifest, which defines the workload to be performed. The kubelet also is responsible for checking the health of the service. We have Docker, which hopefully everybody in the audience is aware of and familiar with. It's the container runtime engine. It runs on every node and handles downloading and running containers. Dockers control local EVS API by the kubelet with pods. Pod is the basic unit that Kubernetes deals with. Containers themselves are not assigned to hosts, but instead are grouped together as pods. A pod generally represents one or more containers that should be controlled as a single application. And finally, we have kubroxy, and kubroxy is responsible for forwarding requests to the correct containers. It can do primitive load balancing as generally responsible for making sure the networking environment is both predictable and accessible, but at the same time, isolated. So how does this compare to our previous control plane architecture? If you look at the previous control, HA architecture that we displayed before and then update it with a proposed architecture, we've replaced HA proxy instances with kubroxy. Each service is built out separately and managed and monitored via Kubernetes, which you fail over by having multiple copies of each service that we scale up or down based off of need. Load balancing that we found to be tedious to update with HA proxy is updated in unison with each service without the need to reconfigure manually or through a separate layer of automation. The architecture looks similar from a service perspective, but in the next slide, you'll see how we've built, scaled, and monitored our control plane with Kubernetes. So first, we build each open stack service, in this case Glance API from a corresponding Docker file matching our defined standard. Standard version, packages, volumes, and config. Containers are built as replicated services, grouped together as pods, and exposed via kubroxy as an open stack service port. Health check probes, we'll check the, which are native to the build process and managed by the kubelet are added to each pod configuration to perform pre-build readiness checks, as well as check the health of the pod post build. In the instance of a service failure, and provided that you declared more than one replica in your open stack, in your build, then a copy is rebuilt, making the service, quote, unquote, self-heal. And then we leverage Kubernetes built in monitoring to detect service disruption repair, as well as customizing our resource utilization. So why did we select Kubernetes over other container, automated container platforms? From a provisioning orchestration perspective, it's a complete automated container platform, where that we can have a fully functional self-healed control plane as little as five minutes, thus spending less time on the installation and configuration and more time customizing it to fit our needs. It's easier to provision and orchestrate new applications as pod can be started in seconds versus the minutes it takes a VM to instantiate. It also allows you to eliminate the need to worry about under-over provisioning, reducing the time needed for additional capacity planning. From a CI CD consistency standpoint, it's easy to deploy and maintain applications. Kubernetes leverages Docker as the build and deployment artifact, and you essentially gain a portable and shareable package in which a company can deploy their software to almost any infrastructure stack. So pretty much anyone who can build a Docker file can build a replicated service application in Kubernetes. Kubernetes also provides fully integrated deployment options like rolling updates for version or security purposes, an example, green, brownfield deployments, and AB testing. We also found upgrading to be no longer a disruptive procedure, but actually a productive one. Scaling, the benefit of the CUBE service system is its ability to maintain connectivity to the underlying service aggregation, even when it's rescheduled to a different node after failing or if the pod's IP address changes. The key benefit here is that there is no reconfiguration required from a load balancer perspective. Kubernetes does it for you. There's also significant cost reduction in increasing your average server utilization rate, therefore reducing your server footprint. A lot of environments that run their VMs run them with a low utilization rate, sometimes as low as 10%. While applications in a Kubernetes cluster can see a much higher utilization rate, sometimes as high as 70%. It self-heals. Kubernetes has built-in self-healing mechanisms such as auto-restarting, rescheduling, and replicating containers. As a user, you define the state, and Kubernetes ensures that the state is met at all times in the cluster. We've all, hopefully, a lot of us have here who experienced a lot of issues troubleshooting open stack services, whether it's in the pre-build architecture or in a post-build service failure state. With the ability to maintain service uptime, you replace what happened to my service with the well it corrected itself. And I also like to reiterate the point that service availability should always take priority to finding the cause of failure, not to ignore the cause, but uptime is king to maintaining any guarantees on your SLA. So you're all ready to see the demo? I'll hand it over to Sean. Thank you, Derek. Today's demo have three parts with four video clips. It's going to last for about 10 minutes. In the first part, we're going to show you how easy and how fast you can use our proposed solution to deploy an open stack liberty control plan. Since today's topic is self-healing, we're going to demonstrate the self-healing capability of Kubernetes in part two. And we dedicate the last part for our friends in the operation team, during which we're going to showcase how easy you can use to use Kubernetes building elastic search engine and Kibana for the monitoring purpose. The entire demo system is built upon three Ubuntu VMs. One of them act as the Kubernetes master and the worker nodes. Another two act as just the worker nodes. In this context, Kubernetes is going to orchestrate open stack liberty release running inside a Docker container. We also offer the live demo, but unfortunately, it's not going to be feeding to today's 40-minute time window. So if you are interested, let me know offline. And I can run it for you. All right, so in the first video clip, I start from a working system. I already have existing pods. So here, in this case, I run a clean up script to try to clean it up. Here, I said just go grab a cup of coffee. It's actually a little bit exaggerating because by the time when your coffee is ready, the work is already done. And then we can use a Kubernetes control command, control get pod command, and verify the status of the system. And you can see that there's no pod exist at this moment. Then as the next step, I kick off another script just to try to deploy the open stack liberty control plan. As you can imagine, it starts from our typical stuff like a Mexico database, the rapid MQ cluster. And these two steps actually take a little bit longer and followed by the keystone, which is one of the most important component in the control plan. Then afterwards, as the next step, as you can imagine, I start deploying other core services. For example, Glance, Nova, and Neutron. As the last step, we actually deployed the horizon dashboard. We also added the Neutron agent. Actually, we don't want a Neutron agent. We want to eventually come out with a solution to replace it. I just added here for fun. It's still running. But as you can imagine, I actually fast forward a little bit because the entire deploy moment is going to take about five minutes, about five minutes. So by the end, I want to highlight two things. The first one is the cleanup process take about one minute and 18 seconds. And surely, it also have a lot of room for us to improve. But the deployment itself only take five minutes and one second. In the second video clip, we're going to start from where we left. And in this case, let's first take a look at the Kubernetes cluster. And here, you can see for all of those OpenStack key components, they are all running. And each of them have three replicas. And thank the god of the demo, they're all running fine. There's no restarts. And then I said, well, let's verify the OpenStack control plan functionalities. I kick off another script. As you can see, there's nothing fancy. I just go there, check the Keystone API. And you can see the list of services. And then the OpenStack glance image is actually written nothing because it's a brand new system. And then afterwards, all of those normal computer just come by fine. They're all up and running. And the neutral agent, they all come back with a smiley face. And then I said, well, I need to entertain my audience. So here, let's say, just upload an image to the Glance image store. And you can see it's just running fine. It will come back. And the OpenStack image list command returns this entry. The next step, let's take a look at the self-healing capability of Kubernetes. This is one of the most exciting stuff. I write a Python script. It's actually do nothing, just go to the Glance API, constantly grab the list of image. And here, it returns how fast the Glance API can react to the API call, together with the HTTP error code or the return code. So let's keep this Python script running. On the different terminal window, we also let's review the status of those parts. They're still running fine. I'll just pick one of them. I use the Kubernetes delete part command and just remove them. I remember, at this moment, that Python script is still running in the background, is doing its job. As you can tell, Kubernetes immediately figured out that part is being terminated. And in parallel, it launched a new one. The new one is actually in the painting state. In the Python script, it's actually reporting no outage. There's no outage. And the use in operation, you can see, it's not about deletion. At a time when you actually add a service back, it's cost more outage. So for the extreme cautious reason, let's take one more look of the part. I want to wait until the part is in the fully running states, joining back to the repetition controller. And then by that time, let's verify the outage. There's still no outage at this moment. So this demo gives you some idea how fast Kubernetes can detect the termination of a part and how fast it can bring up another one based on our pre-determined, pre-designed template. In the next one, I want to show you the Kubernetes dashboard. This is the native Kubernetes dashboard. As you can see, all of those OpenStack components were deployed in the previous steps. They all organized as a separate section. I label them as a dash RC, means replication controller. Here, if you clip those view details, you can see the details of those parts. In this case, Glance API, I have three of them. The couple things I want to highlight here. The first one is, as you can see, those parts, they actually consume very little resources in terms of the CPU utilizations and also the memory consumptions. The second thing I want to highlight here is, as you can see, those parts, they also have their own unique IP address. They also spread across those three virtual machines with IP address 7.11, 7.12, and 7.13. And do you see that the blue, the log icon, if you click it, you can see the log. This log entry is actually generated based on my Python script. It gives some ideas of how your service is running. If you see that edited icon on the left, on the left panel, here, this is the place you can use to scale out or scale in your cluster. And surely, we're not going to demo that today. I just try to... All right, so this is the Kubernetes dashboard. It's nothing fancy. And in the next step, I'm going to show you the Kibana dashboard. Here, I laid out four charts. The first one actually gives some ideas of how fast your Glance API is going to react to your HTTP call in the term of milliseconds. The next one shows you just the traditional CPU utilizations. In this case, I actually timed it by 1,000 because 0.001, which is kind of boring. The next one gives you some ideas of the memory consumption in terms of megabytes. And the last one, that big green pie, that is just the HTTP return code. And in this case, you can see the old HTTP 200 OK is a good indication our system is healthy. So that concludes our demo here today. Like I said, it was over the live demo. You roughly run about 40 minutes. If you are interested, let me know. I'll be more than happy to run it for you. So here, based on our years of experience, I witnessed a lot of issues in the open-set cloud, both public and private. And for so long, we have been trying to tackle them, mitigate them, or fully address them. But surely our efforts are not complete yet. And I hope today's demo convinced you the order can be brought out of the chaos and your business can be successful with open-set cloud if the right approach is taken to get it closer to Nalana. So let's take a look at what we've learned out of this proper concept. The first thing is, we firmly believe that open-stack deployment can be very easy and fast. With the manual installation and the configurations, we'll talk about days or even weeks of work followed by the months of chaos. With the rise of automation tools like Ansible, we can brought out hours. As a matter of fact, we love Ansible. We have Ansible solution playbook, which can help with deploy open-stack liberty in about 30, 40 minutes. But still, that's not enough. Today, we demonstrate a way to give you a fully automated approach, and then give you a fully functional, self-healed open-stack control plan in five minutes. We would love to hand over to our friends in the security team and also in the service team so you can spend less time on the installation and the configurations, and more time to make sure you're customer happy or execute test suite to make sure your code is always in the deployable state. And I threw out an entire course of CACD where I anticipate a small batch size of work flow through the dev team, query team, and the operation team. So original thinking has significantly improved the productivity by creating an environment on demand, limiting the working progress, and also build a system which is very safe to change. But in reality, what we realized from our customers was not the case. This great concept actually introduced a more delay due to the inconsistency between the diving environment, which is built and developed for the unit and function test, and then the staging environment driven by the query team to find the corner cases and test the business logic, and also the production environment. So given the nature of a container, you can package your application with its dependencies and we can create a virtual isolation on top of your physical box. And don't forget, Kubernetes as a very typical orchestrator is also provided lifecycle management capability. So when a dev team need an environment, we can burn up in five minutes. When they don't need any more, we can shut it down in one minute. It has never been so easy before. With those tons of good news, go to the dev team and also the service team and also the query team. For our friends in the operation team, I have a bad news for you. With the introduction of those replicas, the complexity level of running an open stack is going to be timed at least by three. And also make the matters worse. The name of the container and also the number of the container can be quite dynamic over the time. Take those 10 plus deployment into the production every day, actually, the complexity level can be exponential. But don't feel frustrated because we still formally believe a solid operational solution is still achievable and we're actually working on it. Just in case we have any decision maker here in our audience, we can summarize today's session with one word, which is efficiency. By the end of the day, it's all about improving efficiency. So today, we demonstrate a approach which we can apply the technology developed outside of the open stack community and onto the open stack cloud itself. And the business value is quite clear. It can help you cut on the cost, for sure. And it also can help you increase the usability of your open stack services by a significant factor. And sure, this approach would require a lot of rethinking. Now, the question is, what are we going to do next? It's very simple. Just make it better. The top priority on our roadmap is scale. Today, this system is running fine in the small-scale environment. And tomorrow, we're going to go above and beyond by continuously integrating other open-source tools. And eventually, it's going to become a great product for everybody. So the next one is the flexibility. Today, we actually hard-code several parameters in our code, including user credentials, due to the time constraint. But tomorrow, we're going to make it more flexible so our customer will have more knobs to tweak for your own design and the use cases. The last one is reliability. Today, we're going to share with you a subset of what we already developed. Tomorrow, we're going to continue expanding our efforts. So eventually, the auto-skating decision can be smarter and it can be more reliable given the retro set of metrics. But what's going to happen next, I don't know. But I guarantee you it must be something more exciting. I think we can close for Q&A, if you have any questions for our presentations. Sure. How extensive is the quality of service support within Kubernetes, if any? Actually, Kubernetes have a couple of building features. It gives you some basic level of color service even performs the measuring. For example, one of those metrics I displayed here on the Kibana is actually what we're going to do next. It is actually provided by Kubernetes by using Kibster. But with that being said, like I said, it's very basic. We're still kind of exploring other alternatives. For example, integration with Datadog. Those can give you a retro set of metrics, in our opinion. But that work has not been done so far yet. I have two questions. So one is, where and how do you do the initial seed, for example, of the database and registering the service endpoints? And the second question would be, what do you do with OpenStack services that are not horizontally scalable, like Cinder Volume? Actually, we're at time. But if you want, we can just continue a little bit offline. Yeah. The quick answer to your question is, we actually give up our own algorithm to tackle those issues. Very troublesome is the metric and everything that is kind of the reference.