 Taylor, I'd like to thank everyone who's joining us today. Welcome to today's CNCF webinar, Kubernetes as Infrastructure Abstraction. I'm George Castro, Community Manager at VMware and a Cloud Native Ambassador. I'll be moderating today's webinar. We would like to welcome our presenter today, Oleg Chunikin, CTO at Kuebler. A few housekeeping items before we get started. During the webinar, you're not able to actually talk as an attendee. There's a Q&A box at the bottom of your screen if you hover over the Zoom UI at the center bottom. There's a little Q icon, click on that, and that will take you to the Q&A box. So please feel free to drop your questions in there and we'll get to as many as we can at the end. And of course, if you're in chat, just remember to put your question into the Q&A section and not just the chat because that will get overwhelmed with the amount of people. All right, and with that, we'll hand it over to Oleg to kick off today's presentation. Take it away. Thank you, George. Thanks for the introduction and welcome to the presentation. I'm going to talk about a specific view at the Kubernetes, which might prove fruitful when planning cloud native implementation strategy in enterprises and at scale. And so, I'm a CTO at Kubler, as George mentioned. So we are working with Kubernetes since its release and Kubler was born as a productization essentially of our practice for application delivery in different environments. So the main subject of presentation today, so as I said, is bringing together Kubernetes and cloud native in enterprise environment. So I tried to summarize some ideas, which we go to, and we are working on Kubernetes and deploying Kubernetes and cloud native applications in different environments and some ideas around which we are trying to build the product. And the first thing to note is, well, let's go through cloud native attributes. First, so to mention, Quikler. So cloud native stack and approach in general is based on using lightweight containers, language agnostic, deployment, microservices in general and API strict stateless and stateful separation that's on the application architecture side and on the deployment infrastructure side, important components of cloud native stack are self-service infrastructure, isolation from operating system and server dependencies, making sure that your components are portable. Agile develops processes highly automotive and based on declarative resource management. So all that is aimed at making sure that your delivery processes are agile, quick, allow fast turnaround and allow deployment in different environments to make sure that you deploy the same artifacts in different target environments, whether it's DF, QA, for example, and production, whether it's different clouds or different target environments in general. We see that those cloud native architectures are applied and used more and more today. They are used in different types of applications from more or less classic digital web mobile applications, digital transformation, strategic implementations to data science and machine learning and internet of things. They are useful in a variety of environments from classical cloud or data center environments to hybrid and edge computing environments. So they bring clear benefits to operations and development teams and they promise a huge improvement in huge business value. The more business values, the higher scale of the application and the higher scale of your development and deployment and DevOps processes, the more potential benefits this stack promises. Unfortunately, what we also can see is that larger enterprises who stand to benefit most from this stack often lack and most in implementation of this stack and one of the reasons, at least, is that those enterprises have what you can call restrictive environments due to scale and nature of their business, often both. There are various restrictions that either prevent directly or limit or make it more difficult to plan and actually implement various components of cloud native stack. So and two main categories of those restrictions include governance and access and option limitations for various deployments. Just to mention a few, what we can think about is strong requirements in some situations to deploy into multiple and complex environments when you cannot rely, for example, on just a single cloud for security reasons or for legacy reasons when some of your applications are in the data center, but you still want to benefit from the cloud. You need to think about a way to provide a hybrid stack for your applications. Requirements for centralized and unified management and governance is another one. So when implementing a cloud native stack, the first thing your operations team think about is how do we make sure that we have a view of all of our, let's say, Kubernetes clusters in that we create for different teams and that we maintain. And how we centralize security, secret management, for example, things like that. Security infrastructure often already existing when you, for example, have an identity management system deployed already, you have certain operating system, security technologies in place and hard-coded essentially as a policy to be used in all applications. So all that, if not prevents, then at least makes more complex planning and architecture planning and strategy planning for cloud native technology deployments. So just a few other examples include often existing separation of responsibilities when you have different infrastructure management operations teams, security teams, legal developers standing quite aside and working complex workflows together with all those teams, network access security tools, pre-existing security tools and processes that you need to somehow integrate with. And of course, last but not least is the complexity of the cloud native landscape by itself. So I think some of you saw that picture already well-known CNCF landscape, which structures various projects and components existing in this space. Still it's a huge number of tools that you need to pick from, select from and make sure that they work together, integrate, et cetera, et cetera. So what I wanted to talk about today is to talk about a way of thinking about cloud native technologies and specifically Kubernetes, about place of Kubernetes in this architecture that we think is productive in terms of planning for cloud native implementation strategy and helps you build an architecture which will survive the next few years at least in the future, ready, et cetera, et cetera. So when you are planning for Kubernetes deployment, so how do you think about Kubernetes? Is it just a container orchestration? Is it just a step in the evolution from mainframe to serverless? I saw this nice evolution chart on one of the cube cons in the past where we have mainframe in the bottom left corner of the diagram and the serverless computing in the top right. So and Kubernetes is somewhere in the middle. It makes us just a step between those. Is that the case or should we be ready? For serverless platforms? Or maybe we should just focus on using Kubernetes as a basis for microservices platform. So what's the best way of thinking about that? And our answer as a main idea around which we build Kubler, I think, is that Kubernetes is actually great as an infrastructure cloud abstraction and platform layer. So and what that means. So the idea is, if you think about evolution of technologies for applications delivery and development, you see that we went from, well, at some point from virtual machines, from containers to pass as an integration of both essentially delivery and deployment technologies and CI CD pipelines and application lifecycle management tools. Kubernetes surprisingly bites into the value of pass by providing a lot of pass functions out of the box. And at the same time, relaxing the strict application lifecycle, how to say, makes it essentially less opinionated. So when you have your Kubernetes cluster, you're free to use pretty much any application CI CD, build pipeline, deployment pipeline, while still keeping it very convenient and flexible. And most importantly, unified and uniform way of deploying actual binary artifacts into different environments. So as a result, so if you look at Kubernetes and container management platform in general as your key component of your future application delivery platform, you can think of a structure and architecture similar to the one shown here. So you essentially have at the core of your platform certain container runtime and container orchestration technology, for example, Docker and Kubernetes or you use them in different environments that are under your management and you use them to make sure that you are ready for any future changes, infrastructure management brings to you so you can deploy the same Kubernetes application onto a Kubernetes cluster in Amazon, in Azure, in your own data center, whether it's sphere or open stack private cloud. So Kubernetes has an infrastructure abstraction layer in that sense gives you portability and future readiness. But at the same time, you have to have certain operational harness around your container orchestration platform, which is shown here. There's two columns on the left and on the right. So you have certain components responsible for infrastructure management for automation. You need to have API to manage and work with your Kubernetes clusters, certain level of centralized preferably log collection and monitoring systems, components to add observability into your applications because when you use container orchestration, natural direction is to from application architectures standpoint is to split it into smaller components, to split on a list into microservices, serverless functions. And that significantly increases requirements for your observability and monitoring tool. You need to make sure that you can trace your requests through this increased complexity of your application. On the other hand, certain components responsible for security and governance are, have to be in place, rule-based access control management, both for clusters and at the same time, you need to some way of integrating your general organizational access control policies with Kubernetes rule-based access control, integration with identity access management and essential ability to deploy, not only in this open world of open public repositories, but also in environments that are much more restrictive, like air-gapped deployments, quote-unquote, security, audit and image management that essentially ensures that you can enforce common policies for which container images are used and where they can run or cannot run. So having this cradle of container orchestration, management platform and operations and security governance around it, now you have essentially a convenient place to start building your application architecture. And another aspect of this view of Kubernetes as an infrastructure management platform or infrastructure abstraction is that you can actually deploy your middleware services like data storage components, databases, Ingress management, logic and tools and base them and run them actually on top of Kubernetes themselves. On one hand, in many situations, people prefer managed services and don't get me wrong, in most cases, if you can afford using a managed service for the database like RDS and AWS, it's better to do so, but it is not always possible and not always desired and in those cases, having this portable infrastructure abstraction layer like Kubernetes and being able to run a data storage platform on top of that can again make you much more flexible and future-ready and platform like that already exists. There are both open source and commercial products like Portworx or Losa or Ceph and Druck, for example, as an example of open source tool for data storage and management on top of Kubernetes. It's just making them a part of your cloud native and container management platform, which is important. In addition to that, you can also run application lifecycle related components on top of Kubernetes themselves and that's where you convert your container orchestration and management platform into full-blown pass when you run your favorite or integrate your favorite CI-CD pipeline tools. It may be Hosted Jenkins or Plop or you can integrate with cloud services like CircleCI. The main idea that those pipeline tools will use Kubernetes as target environment, of course, but in some situations they can also use it as a host environment. Important moment and an important thing here is to remember that having Kubernetes as an infrastructure obstruction means to keep it somewhat separate from higher level components. So essentially remember this cradle picture and make sure that your higher level components can run on pretty much any Kubernetes and your Kubernetes is pretty much standard, compatible Kubernetes layer, which does not integrate, for example, your CI-CD and does not lock you into a specific implementation of Kubernetes in the past like fashion, so to say. Okay, so having talked about that in general, in principle, I am going to talk a little bit about how such a platform can be architected, how it can be organized and about some specific technical decisions that you will face when designing this platform and that we faced when we designed Kubler essentially and decisions that we made on the way. And I'll start with Kubernetes management versus container management, don't mix those layers up. So essentially when we are talking about Kubernetes itself, it's a container manager and Kubernetes is great at managing containers. It orchestrates them well, it can recover them when infrastructure fails and gets reconnected to the cluster. What it lacks is being able to actually run and deploy multiple clusters and multiple environments. And many tools existing out there are still, we think, underestimate the value of centralized management and orchestration of Kubernetes clusters themselves. So I think that the key component of any future-ready cloud native platform is essentially a Kubernetes orchestrator, a component that is responsible for cross-cutting concerns related to Kubernetes cluster management, which includes, well, single pane of glass that allows you to see all your clusters ability to deploy, manage, change, delete your clusters through API and UI, of course, centralizing infrastructure management and operations. Cost management may be part of that. Very important components are centralized log collection and monitoring. Image management may be such a cross-cutting concern that refers to or required by pretty much any organization of their disaster recovery is something that also can be part of the platform like that. And, of course, integration with identity and access management, so which that's one of the first thing that we faced and a solution here that is quite frequently implemented is to have a identity broker running as a part of this Kubernetes management platform. So in our case, we use key clock and great open source general purposes identity management tool, which by itself can serve as OADC endpoint for clusters that get deployed out of the box, it implements user group management, authentication, authorization, single sign-on functions or OADC sum of what you call it. But most importantly, it can work also as identity broker where you integrate with your enterprise identity management and your enterprise users immediately get a way to authenticate a single sign-on with the platform itself and through that with Kubernetes clusters managed by the platform. So I mentioned already binary repositories and image management that can be part of the platform or can be a standalone services. And there are fortunately many tools already implementing this functionality and image management and binary repositories not only include hosting those images and artifacts, but also advanced functionality like security scanning or metadata management, et cetera, et cetera. So we can think about sonotype nexus here with their commercial products and extensions for image management, artifacts, of course, I'm sorry, Oleg, could you speak into your mic a little bit better? It sounds like your audio got really quiet there for a second. I'm sorry, okay. Is it better? How's that sound for everybody? They say no. I'll try to speak louder. That sounds better. Okay. Does that sound better for everybody else? I'm sorry, everybody. Yes, yeah, I'll try to speak louder. Perfect, thank you. So we were talking about binary repositories and image management and there are a lot of tools that can be integrated from standalone hosted tools that you can run on top of Kubernetes as a part of your own setup. And of course, manage it. Sorry again, your audio cut down low again. Could you perhaps speak a little bit louder and we'll see how that sounds? I will, yeah, give me a second. I'll try to maybe switch on and back. That sounds hotter. Could you talk for a second? How does that sound? That sounds better to me. Okay, okay. Yeah, I'm sorry for this technical issue. And please let me know again if it doesn't go well. Okay, so the next slide and that's what next thing I wanted to talk about is actual deployed clusters architecture. So, and again, on one hand, we have pretty much everybody deploying Kubernetes nowadays. Every cloud has their own managed service. Every private cloud is either have it or working on it. Think about AWS EKS, Azure AKS, POTL PKS, et cetera, et cetera, et cetera, et cetera. So the first site, you can just take those different Kubernetes clusters and maybe add some centralized API and UI management tool on top. Downside of this approach is that your clusters are actually while Kubernetes is a great unifier, but even then it may be difficult to work with different Kubernetes versions that different providers deploy in different environments or with different limitations they implement in different environments like some providers may not allow you to adjust master configuration, for example, others may be limiting the choice of iterating systems for working nodes, et cetera, et cetera. So, and what we found is important is to be able to deploy at least as an option, a sort of uniform Kubernetes cluster architecture in different environment, just relying on the basic capabilities of each target environment as an infrastructure as a service. And so building such an architecture. So what we tried to implement is ability for a cluster to self-heal without relying on that centralized Kubernetes orchestrator. And that ability is based on configuring cluster in such a way that it relies on infrastructure provider as much as possible for infrastructure recovery and then takes the rest of the path in terms of recovery using standalone agents running on each instance. So this diagram shows a typical cluster architecture that we eventually built that we deploy. So it's self-recovery and self-archestration relies on a certain shared security store through which agents running on each and every instance of the cluster, of which specific cluster can communicate and orchestrate exchange secrets like foundational secrets. So to say those secrets that are required to bring the cluster up, Kubernetes cluster up first. And then so essentially as soon as Kubernetes cluster is up with all of its components running as Docker containers which improves portability, et cetera, et cetera, then those agents can report their status, et cetera, through Kubernetes conditions and objects and be controlled through the same media. But the baseline and bootstrap essentially stage is relying on this orchestration store which may have different nature and different infrastructure providers in AWS. It's, for example, an S3 bucket accessible only from that cluster and worker and master nodes in Azure storage, et cetera, et cetera. While also master hosted, self-hosted orchestration is also possible. Couple more things that I wanted to mention are actually monitoring and log collection. So both are available in various Kubernetes implementations including open source Kubernetes. When you deploy it, you have a standard add-ons that deploy Grafana and Prometheus, for example, or Elasticsearch and Kibana. Problem with that is often that you have to deploy it in each and every cluster if you rely on those add-ons and from enterprise perspective, from future ready container management platform perspective, what's important to have is to have centralized view of all your applications and Kubernetes clusters in different environments. And that's, you can build that by using any number of managed services again or existing tools like CSD for example, or even CloudWatch in AWS. But in some situations when you have to rely or have to host your data in more closed fashion so to say, you can host also your own centralized monitoring and log collection system running as a part of that Kubernetes management component. And what we build is a essentially Prometheus-based solution when in each target and managed child cluster, so to say, we run a Prometheus collector, which is a relatively lightweight Prometheus instance, which only retains metrics for let's say a few minutes. And then we run a centralized Prometheus under Kubernetes management platform, that figure eights all those metrics in a single database and uses Grafana to visualize them, et cetera, et cetera. The same is true for log collection. So hosting a full log collection stack in each Kubernetes cluster under your management may be wasteful and difficult. So one of the possible solutions is centralizing that and running only agent components in your child Kubernetes clusters. Of course, both of these items require certain additional consideration, let's say, because when you start collecting metrics from different applications and metrics and log entries from different applications and different clusters, one of the question is how you distinguish different clusters and that's where you need to implement essentially augmenting raw metrics you collect from target clusters with additional meta information, which cluster it was taken from, which space, et cetera, et cetera. Another question is making sure that people who have different level of access to different clusters have that access reflected in how they can access a log records and metrics. So that's where, again, central control plane makes it easier to do because you have all that information in your database and you can use various access control tools specific to Elasticsearch, for example, to make sure that you either synchronize or dynamically pass permissions essentially that are configured for clusters onto log collection and monitoring subsystems. So, but all in all, as a result, you can get to this essentially future idea, general purpose container orchestration platform, which has all essential components, that an enterprise, especially larger enterprises require, and at the same time, don't limit you to any specific opinionated, let's say CI CD approaches, for example, or tools and make it possible to grow together with the ecosystem and together with the technologies that appear in the market. That completes my presentation. Now I think we're open for the questions. Yes, we have quite a few questions, so we're gonna try to get through them here. Vasya asked, it can run multiple clusters. Does that mean we can run and manage another cluster using one Kubernetes? So I think it refers to, so the idea is that if you use, let's say, as an example, Cops or Hubbub them when you deploy or even OpenShift, for example, when you deploy Kubernetes cluster there, what you get is you get a single Kubernetes cluster. What we are trying to argue for is that it is not enough for a larger enterprise. For a larger enterprise, it is important that you can not only deploy separate Kubernetes clusters in different places, but also as a result of that deployment, have them accounted in some system and have them centrally managed through some sort of universal API, for example, or at least method. And so answering your specific questions of what we do. So there is this centralized Kubernetes management platform. It provides you API UI and you can call that API and ask it to say deploy a Kubernetes cluster in that AWS account. Next thing you ask that API to deploy another Kubernetes cluster in another Azure account, for example. And now you can see all that cluster on the same pane of glass and then manage them using the same API. So I hope I answered the question. If not please let me know. And I think. Sorry, I lost my windows. Devendra asks why Kubernetes, why not Docker swarm? I think, yeah, well, a couple of years ago, I think there was still an argument whether it's Miasas, Kubernetes, or Docker swarm. But nowadays, I think we see that Kubernetes won by a large margin. I can say that nothing else is used, but Kubernetes is used much more frequently from where I stand. And I think there is a good reason for that because when Kubernetes appeared, so we essentially compared several orchestration technologies when we implemented first steps on our journey to standard delivery platform. And what we saw was that Kubernetes has, I think, the best abstraction of, the best balanced abstraction of infrastructure as it is presented to developers. So I think that's most importantly what most important thing that made Kubernetes so successful. On one hand, it's not too low level. At the same time, it is quite rich. It can represent pretty much any cloud, any infrastructure, any operating system out there. It is very flexible and extensible through plugins and various integrations you can essentially deploy and integrate with various infrastructure providers and without sacrificing actually what you already learned about those abstractions as a developer. It's difficult to quantify, to be honest, but yeah, so that's what I think is the main reason. So somehow it's the most convenient infrastructure abstraction out of all others. Okay, moving on, Piotr asks, this Prometheus exporter for central monitoring is your proprietary software or is it open source? Could you point to a good exporter? Good question. So let me switch back to that slide to maybe explain it a little bit. It's actually a blog collection. So there is no special exporter per se. What happens there? So we run centralized Prometheus using its standard metrics federation capabilities to federate metrics from children, Prometheus. Interesting solutions, I think technically interesting solution here is that, first of all, we do that through Kubernetes API using Kubernetes proxy API. And secondly, so we essentially dynamically reconfigure this central Prometheus when we add and remove Kubernetes cluster. So there is no sort of special exporter per se. It's just standard Prometheus metrics federation. What's proprietary here is essentially a way we transform our set database. So the database that contains all the clusters into Prometheus configuration. But yeah, I don't think that's something that can be easily open source, mainly because again, so you can have those assets accounted in a different manner in different fashion. You can have for example, a Kubernetes cluster API to be responsible for managing meta information about all of your clusters. Okay, moving on, Peter also asks, how do you monitor Kubernetes control plane processes? That Kubernetes control plane runs itself in a Kubernetes cluster. So it's essentially a home package that gets deployed into a Kubernetes cluster. And that Kubernetes cluster meta information is registered in the control plane itself. So in a sense, it monitors itself, which is clearly a, maybe a weak point. So what we, because if it goes down, it cannot alert you about the fact that it went down. So clearly you need to have some sort of external monitoring to make sure that at least most important functions of your platform are alive. So, and it's not an unusual thing to have it, to have a system like that designed to have multiple components monitoring each other to make sure that you can see. So you can have, for example, deploy it across different regions and monitoring each region, monitoring others to make sure that they survive or you can rely on a cloud-managed service that monitors that your platform is alive and make sure that that monitoring information is not critical and not sensitive, for example. Because again, as I said already, it's probably beneficial to still use managed services where you can. And one reason why you don't want to use managed services and you want to use a centralized self-hosted solution is when even your monitoring blocks are sensitive in one way or another. Okay, Devendra asks with Kubler, can we manage multiple clusters? Yes, so that's the idea essentially. So the idea that you have this central component that has an insight into all your clusters and accounts for keep track of all your clusters. Okay, this one sounds hard. I'm new to Kubernetes. Can I set up Kubernetes the hard way in VMware Workstation locally? If so, can you provide some info? I think nobody created a better guide than Halsey Hightower in his Kubernetes the hard way. So I need to look it up, but I think you can quickly Google it, Halsey Hightower, Halsey Hightower, Kubernetes the hard way and you'll find it on GitHub. Mohammed wants to know, are there any best references to start adopting Kubernetes? Maybe one or two, I don't think we have like a list. I'm not sure what references mean. I think they're just looking for lists. I have an idea. I'll post that in chat for them. Here we go. How many Docker container engines can I add in Kubernetes? Can you provide maximums and minimums of Kubernetes? Is there any advisor for installing and updating Kubernetes? All good questions. Container agents. Let me, so okay. I'll first address it as an abstract. Kubernetes question, and then I'll switch to what Kubler does specifically. So first of all, Kubernetes in general, I think supports container D and Docker at the moment. And in general, if I'm not mistaken, OCI API that is implemented by... Sorry, your voice cut down again. Okay. There you are. Is it better? Okay, sorry. So you pretty much can configure Kubernetes with any of the supported container engines. What we tried to do in Kubler is, first of all, to make sure while Kubler takes care of all those platform related issues, it adjusts Kubernetes to different environments and infrastructures, et cetera, and provides operational harness around it. At the same time, the Kubernetes itself is as close to upstream as possible. We literally run Kubernetes components out of GCR.io hypercube container. So it's exactly the version of Kubernetes build by Google. And we make sure that we provide as much configurability to that as possible. So while Kubler sets up certain configurational options based on the structure of the cluster you want to deploy, it still gives you an option to customize those configuration. And part of that customization is actually what container engine you deploy and integrate with Kubernetes. We have about four minutes left. Just giving you a heads up. Yeah. And the idea is to, yes, to provide some sensible default for which we use Docker and we support that. But then allow people to customize it any way they like. All right. And the next question was what Kubernetes version release strategy you follow? I believe you said you just always follow the latest upstream? Not really. So yes, theoretically it's possible with the latest Kubler you can again in this customization screen, you can use the latest Kubernetes version. But default is usually one or two versions behind mainly again because then community and has time to iron out the critical issues out of Kubernetes. Okay. Do you provide auto failsafe for Kubernetes cluster for control plane? I think this will be your last question. I'll take it as out of failover maybe. And at the moment, so what happens we do support failover across availability zones, if that's what you meant, but not across regions yet. Okay. How does Kubler help with security of application endpoints? Okay. This will be the last one. Let's make it in there. Again, by providing some sensible default which is future-rich enough. So to say out of the box, Kubler allows you to choose one of four overlay network providers with Calico, Canal, or Flannel and three of those providers with Calico and Canal support security policies. Calico I think is the most feature-rich. It also supports endpoint network policies and et cetera, et cetera. Veef on the other hand, for example, you can configure for transparent transport layer security for all container communication. So you can pick and choose what fits best in a specific situation. Okay. And with that, we're out of time. Great. Thanks Oleg for a great presentation. Thanks everyone for the question. Sorry if we didn't get to all of them. Thanks for joining us today. This webinar, the recording and the slides will be available online later. Taylor posted a link that's cncf.io slash webinars and we look forward to seeing all of you at a future CNCF webinar. Thank you and have a great day. Thank you. Bye.