 I'd like to thank everyone who's joining us today. Welcome to today's CNCF webinar, Achieving True Reliability and Disaster Recovery for Mission Critical Acts. My name's Ariel Jitib. I'm a business development manager for Cloud Native at NetApp. I'm also a CNCF ambassador. I'll be moderating today's webinar where we'd like to welcome our presenter today, Oleg Genitkin from CTO at Kubler. A couple of housekeeping items before we get started during the webinar, you're not gonna be able to talk as an attendee. There's a Q&A box over at the bottom of your screen. Please feel free to drop in your questions there and we'll get to as many as possible at the end of the presentation. This is also an official CNCF webinar and is subject to the CNCF's code of conduct. So please don't add anything in chat or in questions that would be a violation of that code. Basically, be respectful of all your fellow participants and presenters. And with that, I'll hand it over to Oleg. Yes, Ariel, thank you for the introduction and welcome everyone. Thank you for joining today's webinar. I'm CTO at Kubler working there for the last four years focusing on, well, Kubler development and just a couple of words about what we do and why we decided to talk about this subject. Before we proceed, so essentially Kubler is an enterprise Kubernetes management product which allows a company to set up a Kubernetes as a service essentially inside of their organization. And as part of our business, we see how different enterprises address various concerns and reliability as well as security and governance are two of the most important ones. So here in this presentation, I tried to summarize what we see in this field in these areas and address essentially some of the areas of confusion that people still see in figuring out what Kubernetes does for them already in terms of reliability of their application, what Kubernetes can do but doesn't do out of the box and what you need to take care of yourself. And essentially usually, so this journey for a company who just plans a cloud native strategy starts with, well, learning how this field looks, finding out that there are great technologies as container containerization and container orchestration. And so when your operations teams learns and starts learning about Kubernetes, usually the first steps follow sort of excitement route. So Kubernetes is great. It solves a lot of problems out of the box. And so it will be a breeze to build a reliable system with Kubernetes. So then you start realizing that, well, some things needs to be set up correctly. Some other things Kubernetes doesn't handle because it's out of its scope and you need to look at the full stack to make sure that your system is reliable and can recover automatically. So I'll first talk about what Kubernetes does out of the box and I'll start that with a quick refresher with Kubernetes architecture. Just first, so how Kubernetes looks under the hood and some of you or many of you probably know that already. So two main categories of components in Kubernetes are sort of master side and worker side. So there is a brain of your Kubernetes cluster running on master nodes usually and running master components. And one of the most important is an API server which stores all the information about cluster state and future state or desired state and current state and provides an API for all other components to work with that information. Then controller manager, scheduler and potentially other components that help to implement various strategies to handle various processes inside of that Kubernetes cluster. They all work through API server. API server stores data in ETCD cluster and that data must be stored persistently. On the client side or on the worker side, there is an agent called kubelet which handles containers, runs them, stops them, reconfigures them through container runtime and then components responsible for setting up intercontainer network connectivity. It's usually an overlay network and kube proxy. There is also an important component called in cluster DNS that helps containers and pods resolve each other. So that's a Kubernetes under the hood. From the client perspective, Kubernetes is seen through its API and through master API. You see a number of objects, most important of which are shown on this diagram. So you see all the worker nodes that comprise Kubernetes cluster as node objects. You see all the containers as running inside pods. Each pod is assigned or scheduled onto a specific node. Each pod is assigned a unique IP address in cluster IP address. And you have also objects called services that allow you to discover groups of pods and load balance between those pods and essentially connect to applications running in those pods. There are also a number of objects related to persistence, most important of which are persistent volume and persistent volume claims that abstract out data storage for your containerized application. So what Kubernetes does out of the box for application reliability? So first of all, it relates to the lowest level of execution in Kubernetes cluster to pods. So pods can report their status, their liveness and their readiness to Kubernetes and Kubernetes can act upon that information. So if pod is not alive, according to those pod probes, Kubernetes will destroy this pod. If pod does not report readiness, Kubernetes will not route traffic to that pod. So and those probes can be of 3D types out of the box. It can be TCP, HTTP connections or exact comments inside of one of the containers in that pod. What happens if pod is terminated? Well, that is managed by controllers. So almost never you run your application just as a pod. It may be done for test purposes, but the deal about the pod is that when you start it and it dies for any reason, it is not restarted automatically. So pod is sort of immutable object. It runs, it dies, it disappears. Kubernetes has a number of other objects that are called controllers all together that manage how pods are recovered. And first of them is a replica set. So essentially it's an object that tells Kubernetes that certain pod type must exist in the specified number of replicas. So then if one of those replicas dies, Kubernetes will restart a pod from a template. Deployment is an extension of the strategy. In addition to just maintaining a specific number of replicas, deployment can perform rolling update of this set of pods. Stateful set adds to that persistent volume stability. It can essentially provide certain information to each pod in the set that identifies it uniquely like a number in the set. And it guarantees that the pod with a given number will get connected to a specific persistent volume. And the last, but not least is demo set. It's essentially a controller that maintains a number of identical replicas on each node irrespective of how many nodes and when nodes are added, those replicas will be started there. More advanced strategies may be implemented through operators. So essentially operator is an application running usually inside of Kubernetes cluster as well, which implements more complicated strategies of managing your applications in the pod. And operators can do anything. We'll talk about an example of operators when we talk about cloud native storage solution called Rook and Ceph. Next set of Kubernetes capabilities that help manage and set up reliability is resource and scheduling constraints and controls to say so. First of all, well, you cannot have reliable application if you don't manage resources for that application. So if your container can uncontrollably it up a memory on the node, it puts on the risk all other containers running on the same node. So Kubernetes together with container runtime provides means to control to manage that. So you can specify resource requirements for certain pod and you can specify limits that Kubernetes and container runtime will enforce for those containers. You can make reservations for system components. Like for example, if you have a 32 gigabyte RAM on the node, you'll reserve two gigabytes for operating system buffers, et cetera, et cetera. And then only 30 will be used by Kubernetes to execute your pods. Ability to manage resource overall location is also part of Kubernetes based on capabilities called eviction and disruption budget. So if Kubernetes sees that certain node is constrained in terms of some resources, disk, CPU, RAM, it may automatically evict pods based on priorities. So you can specify higher priority pods that will be evicted last or priority pods that will be evicted first. So eviction allows you to dynamically rearrange your applications to move them to a different node if this node is constrained. And then tools like affinity and affinity node selectors and matchers allow you to specify which pods can run on which node. So you can base it on node labels. Trivial example would be this container that has ARM binary since I can only run on ARM nodes. More complicated example may include, for example, pods that require certain node characteristics or just can run on nodes labeled by administrators, by operators correspondingly. Affinity, anti-affinity rules allow you to say that certain pods should try to run together or must run together or must avoid to be placed on the same node or availability zone, et cetera. So it may be important for clusters running inside of your Kubernetes clusters. So for example, if you run a cluster let's say each CD cluster inside of Kubernetes, you may decide to specify an anti-affinity rule so that different TCD pairs will run in different availability zones. And auto scaling is last but not least feature that I will mention here as a reliability related. So anti-scaler comes in different flavors. So most familiar is probably horizontal pod at a scaling. So it's when Kubernetes can increase or reduce number of replicas in a certain set of pods based on custom metrics or standard metrics, usually CPU load for example, CPU usage by these pods. Less known is vertical pod at a scaling and currently it's implemented by restarting pods with different parameters and limits. So for example, Kube Kubernetes DNS uses this strategy. So to scale up based on CPU load, on request load, Kubernetes DNS can restart with higher RAM limits, for example, but Kubernetes community also works on enabling in-place updates for pod resource limits. And I hope it will be available sometime soon. And the last flavor is cluster at a scaling. So because pod at a scaling won't help you if your cluster is not available your cluster doesn't have any more resources to put more pods or larger pods into it. This is where cluster at a scaler comes into play. But because it is related to infrastructure management, so you cannot get new nodes out of thin air. So it usually infrastructures dependent how you configure it. So it's possible to configure Kubernetes cluster so that if it sees that it's not enough RAM to run all the pods that need to be run. So it will add nodes to the cluster automatically. So this is what Kubernetes provides you in terms of managing Kubernetes application reliability and controlling how your application will recover, how it will scale, et cetera, et cetera. But you need to keep in mind the whole stack. And so this slides summarizes different layers that you may want to think about when planning for a reliable system. It starts from infrastructure which can mean anything from bare metal instances and your NAS data center solution to your cloud account with, let's say in AWS with EC2 instances, EBS, disks, et cetera. Operating system because Kubernetes runs on those nodes. It doesn't run just as a hypervisor. It runs on top of some OS, usually Linux and that layer requires certain measures to make sure that the whole stack is reliable. Well, we talked a little bit about Kubernetes itself so how Kubernetes ensures application reliability for applications running inside Kubernetes cluster. But then you need to think about how Kubernetes components themselves made reliable and recoverable. Next layer, so I separated it although some may consider it part of the middleware quote unquote container persistence. So that's the whole subject of the whole separator of interest and a set of problems where it comes to reliability. Well, actually to any other concern, security, operation, et cetera, et cetera. We'll talk about that quickly as well. Then middleware, so it's not yet your applications because it's cross-system aspects, operational aspects or application life cycle aspects. So by these components, but from Kubernetes standpoint, they can be considered applications. You can run them as a part of your Kubernetes cluster as applications on top of your Kubernetes cluster. But this is also an important layer in your stack and in your approach to making everything run with reliability. And only then comes the layer with your business specific application, mission critical application. So what's important when you are thinking about reliability of this system is the whole stack. So first of all, well, architecture one-on-one, don't forget to make sure that not only your application, not only Kubernetes layer, but the whole stack is well architected. Layers are separate and independent. This helps with flexibility. This helps with future readiness components on every layer if possible are well disposable, restartable. So well factor up principles can actually be very useful, not only on the application layer, but on every layer of the whole stack. This also includes ability to reattach dependencies easily irrespective of which part, which component fails and restarts, making sure that you at least conceptually separate persistent state from disposable processes that helps with managing application persistence, containers persistence, make sure that you can recover components separately and you don't add extra interdependencies. So, and here it helps to think about different components in terms of pets and cattle. So pets are things that you want to care about individually and recover individually, those that have individuality. Cattle is something you manage as a cattle essentially, you can change the number, but you don't care about individual properties of each specific components. And if you only allow your data management component to be pets, it simplifies the architecture in general. So what can we say about infrastructure and the trading system in reliability in context of Kubernetes? Again, so here comes the separation between pets and cattle. So if you use that node or that infrastructure as part of your pet-like components, for example, if you run a cloud native storage system on top of that or just storage system on top of that, well, the only way of managing that is trying to fix it. This is usually more complicated than replacing or resetting it. So which works with cattle-like components, so to say. So if you can, it's better to minimize the number of parts that have to be treated like pets. So even in data center with bare metal nodes or with VMware virtual machines, you can say that my virtual machines or my physical machines are replaceable and only my data drives that I connect to them are pets, essentially. So that helps with, again, simplicity and in general reliability of the system. But again, whatever method you use for a certain node or component, there are, there are these tools that can help you both with fixed quote unquote strategy as well as replace and reset strategy. So for example, certain tools are available for Kubernetes that help you identify and try to resolve problems on the nodes like tools running in cluster, NPD from Kubernetes with work security, et cetera, et cetera. So they watch for hardware kernel services in issues and can reboot an instance. If necessary, infrastructure provider automation tools, of course, help a lot. Like if you are running in the cloud, there are convenient tools like AWS auto scaling groups or scale sets in Azure that you just have to set up, have to use, even if you are running specific number of nodes, even if you just run in one master, it's better to run it as a part of a single instance ASG after scaling auto scaling group, then a separate instance because then you can just terminate it if you want to replace it and it will recover automatically using infrastructure automation rather than you would have to recreate the virtual machine by yourself. And then, of course, there are tools that allow you to recover nodes, watching them externally, not from inside of infrastructure provider tool set. And here you have most, of course, customization capabilities, just like a script watching your cluster and changing its parameters based on certain signals, recovery, things, et cetera, but at the same time it's as being most flexible, it's more complicated to implement and test. Kubernetes itself, so you need to make sure that Kubernetes components can be recovered, can survive down times and issues. And the most important ones are, of course, related to master, it's a TCD, master API server, controller manager scheduler, but also agents running on worker nodes and container runtime can have their own share of problems which needs to be detected and resolved if necessary. So, and in part it may be handled by the operating system layers, layer in particular container runtime, for example, most issues related to container runtime can be detected by tools that can also detect kernel problems, for example. But for other components, you need to make sure that they can at least restart if they fail and so you also need to be able to monitor liveliness of those components and make sure that they can reconnect to their dependencies. And in most cases, this is done by either running them as services, creating specific services or running them as static pods managed by Kubelet, which we just have to run as an operating system service. Both can handle restarts and liveliness and readiness detection. Both have their own pros and cons in different situations. So, we over time in Kubler tend to failure running components as static pods because it makes the whole set and the whole deployment more portable. So, it doesn't depend on operating system match, for example. But services may be a preferred solution, for example, in situation where you know for sure which operating systems you need to support and you have deployment requirements and restrictions. Dependences, the most important dependencies that you need to care about and remember are of course, in that case, ETCD data because it's the memory of your cluster, everything about the cluster is there essentially. If you want to recover a cluster, ETCD data is essential and important. There is a certain number of secrets, keys and certificates that are not part of that ETCD data that need to be present for cluster to recover. So, they also become part of the information that you need to remember about. Well, Kubernetes did a great job of minimizing this set, but still bootstrapping cluster requires some initial secrets to exist. And the configuration of those components in some deployments and in some distributions of Kubernetes may also have to be persisted for recovery. Again, Kubernetes did a great job and made a great progress over a couple of last years in making it smaller like ability for QBlets, for example, to be configured from API servers, et cetera, et cetera. Still configuration on even on the worker nodes may sometime be important to persist. Another component and consideration for Kubernetes reliability is setting up multi-master. And again, so decision about multi-master versus single master is not as simple as it may seem. Again, both approaches have pros and cons. One important fact that not everybody realizes is that even if you have a single master, it doesn't mean that your cluster will immediately fail in case of master downtime. So applications running on your Kubernetes cluster will survive at least for some time, even if master disappears for short period of time. So it's a balance between ability to recover over longer unavailability periods and the complexity of managing multi-master and certain other considerations. Like, for example, if you have multi-component ETCD cluster under your API server, benefits is of course is high availability and data replication. So it's easier to recover potentially. At the same time, it's more complex to operate and potentially higher latency. So you need to be more careful when you scale it up. As a part of that multi-master decision, again, it affects also how you persist cluster data, ETCD data, because when you run multi-master, it becomes probably less important to have fully persisted data for ETCD cluster because you have lower probabilities that you will lose all the replicas of this data. At the same time, if you persist all of the replicas, then recovery processes, at least full disaster recovery, full downtime recovery processes become more complicated like just which replica you will use to recover from and questions like that. So it's again a balance. So you need to consider your specific requirements and downtime limits and tolerances to make the right decision there. In some cases, single master with reliable persistent storage of ETCD data may be just enough. Next player in our stack is container persistence. And again, Kubernetes provides a lot of great abstractions and objects to simplify work with persistent persistent data for applications. That includes persistent volumes, persistent volume claims, ability to provision volumes automatically when an application needs one. And this is very flexible abstraction actually because there are a number of types of storage that this abstraction allows you to work with from cloud native block storage, types like AWS, EBS, Azure disks, sphere volumes, et cetera, et cetera, to local storage on the hosts, your applications are running to distributed managed network storage available through NFS, iSCSI, et cetera, et cetera. What's important to realize is that while it's very flexible and very rich, you still need to be aware of certain idiosyncrasies and specifics of underlying storage you are using. And very often when you're using one of these types of storage, what you see is topology sensitivity, for example, like EBS volumes in AWS are availability zone local, so when you create an EBS in availability zone A, it can only be attached to virtual machines in the same zone. It may not be then suitable for a cluster which runs across multiple zones. There may be cloud provider limitations like in Azure, certain types of virtual machine can only attach up to certain number of Azure disks, et cetera, et cetera. So what solving those idiosyncrasies and problems called for certain class of solutions and cloud native storage is essentially a category of solution that help you solve those issues and provide a convenient way of persisting data with for Kubernetes-based applications. Cloud native storage is usually a storage system that integrates well with Kubernetes, works with flex volume, CSI, container storage interface, standards who provides volume provisioners and snapshot support in Kubernetes so that Kubernetes abstractions can be used to manage these features in storage. And normally those cloud native storage solutions run in cluster, but you can also run them externally. And what there are also different categories of solutions for cloud native storage. One is a layer on top of some baking storage or pretty much any baking storage, like for example, CF is a distributed file storage system, block storage system that runs on top of any block devices. So you can use your native block devices to back this CF cluster, which in turn will provide cloud native storage to your applications. There are also solutions that augment or extend that baking storage capabilities. So for example, it may be a solution which adds quote management or snapshot support and integrate snapshot integration to certain non-cloud native storage solution. And examples of solutions like that are cloud native storage solutions, Rook CF is a great, probably the best open source solution out there. Then Portworx, Novalozo, Blaster, FS, Limster, OpenBS, so there are a lot of solutions. They're available for this layer. This needs to be vived into your reliability and recovery strategy as well. So I wanted to talk about Rook and CF as an example of such a cloud native storage, and I won't talk about that in much of details due to time constraints. Switch to the next slide, which talks about middleware, and a consideration for middleware in terms of reliability is essentially how you run it. So first of all, middleware is various components that are not essentially end user facing applications but are important for your operational capabilities for your application lifecycle, et cetera, et cetera. This may include your CI CD pipelines, source control management system, monitoring, clock collection subsystem, registry, image registry, scanning components, et cetera, et cetera. In most cases, there are several options for each of those components. Pretty much every cloud provider has managed services in almost every category, which is probably the best choice if you can limit yourself to a specific cloud provider. So operationally managed services is the easiest solution, but if it's not possible, which is often the case for enterprises for different reasons, either because they want to achieve multi-cloud, have multi-cloud strategy, want to be able to deploy across clouds or because of security and governance constraints. So two options that you have is either deploy it separately and manages it separately and consume it from Kubernetes and Kubernetes applications, or, which is an approach we advocate, most is to run those components inside Kubernetes themselves because in this situation, in this case, you get a number of operational constraints, concerns resolved for these components as well if you have a reliable Kubernetes layer already. So this essentially covers most of the stack layers. What I want to mention in addition to that is multi-site. So we see that as a requirement and the request more and more over time. So just recently at the last QubeCon, we also announced a direction for Kubler and a technology preview for Kubler 2.0, which covers a number of those concerns for multi-site deployment. So it's not possible to achieve essentially full reliability if you don't consider a site blockout. So it may be allowed in some situations, but for a global company, it may be a significant issue. And in most cases, you would look into various multi-site deployment strategies and various strategies to recover from a whole data center or region blockout. And what's important to remember to consider when you are also thinking about Kubernetes deployment is well, various layers inside the Kubernetes stack that may need to be integrated in multi-site scenarios. So first of all, physical network connectivity is clearly necessary, but for Kubernetes, it's also often a requirement to provide overlay network connectivity. So that either pods can talk to each other or at least your microservices or services running in different Kubernetes clusters can talk to each other. And this is achievable through various overlay network solutions. For example, KaliCobbGP and Kalaos, you can connect clusters in this manner. But then cross-cluster DNS, so you need to be able to make those services, not only talk to each other through IP, but also discover each other through DNS and it can be done with Kubernetes core DNS integration where different in-cluster DNS components will delegate resolution to each other. But then there are also questions of how you deploy into those different clusters running in different sites, how you would load balance incoming traffic, which can be achieved with Kubernetes federation and various Kubernetes external DNS integration features. And the last, but probably the most difficult question is ensure cross-cluster data replication and recovery, which is fortunately achieved on different levels for different categories of storage through cloud native storage capabilities. So for example, for block devices, CIF allows you to set up geo-replication for object storage as well. CIF allows to do that. You can use native approaches like EBS replication for AWS through snapshot, intelligent transfer. Database level replication is possible through cloud native databases that become more prominent recently like UGBITE or VTS, which is cloud native database examples, allow you to replicate across regions. And of course, application-specific approaches for disaster recovery and replication that may be implemented by your application layer. So to summarize what I hope to demonstrate today is that Kubernetes by itself provides a number of robust tools for application reliability, but at the same time, you need to remember that underlying infrastructure also needs attention and its responsibility of cluster operator. There are a number of layers that you need to take into account and make a part of integrated strategy for disaster recovery, architecting it well and simply and in a independent manner is important. Important points are that middleware and various components, so it makes sense to consider running them on top of Kubernetes and benefiting from Kubernetes reliability features and also multi-site high availability balancing and failover while still complex task is made easier with Kubernetes and allows you to achieve these reliability requirements essentially in much simpler ways than it was even five years ago. So hope this, you find this presentation useful and I think we are ready for questions. Great, thanks for that, awesome presentation. Does anybody have any questions? If so, again, there is the Q&A button down at the bottom of the screen. Go ahead and feel free to add them there. Give them a minute and if not, we'll go ahead and close it up. How about, well, here's one question for me. Are there any kind of pitfalls that you've found on engagements or kind of the things that folks should really, I think you highlighted a lot of them, but is there anything that you often see in engagement with the community? Do you see any kind of redundancy in engagements that folks try to avoid when planning for hybrid cloud or redundancy? Well, again, so probably it's not a pitfalls, but more of a source of confusion. So often people who start on this journey don't yet realize where and what Kubernetes can do and where it's essentially scope of responsibilities ends, or at least not ready yet to clearly delineate those responsibilities. And what we see often is that, well, sometimes people deploy frameworks that are more than just Kubernetes that enforce, like, for example, certain application life cycles and maybe just run Kubernetes as a path engine inside and then have difficulty to follow this quickly growing and developing cloud native ecosystem because, well, if you use a product which provides you certain high-level features and just uses Kubernetes as an engine, so it has an advantage in terms of as an engine, so it has an incentive to develop those features but not stay compatible with the ecosystem. But the ecosystem is much wider than just, for example, CI CD or just monitoring, for example, and new tools appear there every day. So what we found works better in the long run is to make sure that your Kubernetes deployment is stays essentially Kubernetes and container orchestration provides good operational tools to your ops teams, IT teams, but stays compatible with open source upstream Kubernetes from developer's perspective. So this not only helps with, again, your applications deployment, but it also helps your operations teams because now they also have an option to run certain operations-level components like enterprise-wide cloud native storage on top of Kubernetes and benefit from Kubernetes features. Cool, and to dig a little bit more into that, Arthur, have you seen teams kind of embrace this flexibility and what kind of tooling have they switched over monitoring solutions? Are you seeing activities around service meshes? Like, in order to, how have you seen folks leverage this flexibility that you propose and champion kind of in real-world scenarios? Yeah, good question. So for example, if we are talking about, for example, a use case that comes into mind, a team of a large financial services provider who uses Kubernetes as a test bed for their applications what's it called, break testing or something like that. So they run essentially applications, containerized applications in the Kubernetes cluster and they use a service mesh capabilities to introduce errors and problems into intercomponent connections and make sure that those applications can survive and work reliably in this situation. So it wouldn't be possible probably if it's not sort of a standard scenario for how you would use Kubernetes, right? So you usually think about Kubernetes as your platform for to run applications, your end users hit. But in this scenario, you need to be able to use service mesh in a creative and a little bit non-standard manner and running that on top of an open Kubernetes platform that fully complies with Kubernetes rather than focuses on a certain specific scenario and delivering applications helps a lot. Gotcha, gotcha. So to read it back, it sounds like by adhering to the upstream version of the project, it makes it easier to integrate as you kind of progress along your journey and different tools like a service mesh start to come kind of into the purview of where it maps more directly to value back to the organization. Exactly, exactly, yes. So, and again, so this may be just a temporary state because maybe in five years, this whole field matured enough so that there is a clear winner or winners and architecture and the ecosystem doesn't change that quickly anymore. But I don't think it's even a short term perspective or middle term perspective. It's probably several years to last at least. Yeah, yeah. Well, cool, we're at the top of the hour. So thank you so much for your time today on this great presentation. Thank you all for, in the audience who joined us today, this webinar recording and the slides will be available later today. And we look forward to seeing you at a future SCNC app webinar, have a great day. Thank you.