 Hey people, good morning or whatever the time is it and my name is Dario and I'm here with max. Hi max. How do you do? Hello, everyone. Yeah, I'm fine. Thanks. How are you? Well, well a little bit excited because it's my first time at cubecom as speaker and also as attendee And I'm really happy to be here with you because we are going to shoot with it with capsule and it's really Astonishing I'm really free to be here and I'd like to thank all the people that is joining this session Because we did a great job together as developers and also as architect of the software that we built so Yeah, I'd like to show you how this presentation is made of so this is obviously the Introduction and we are going to have to session because this is a presentation between me and max and I'm going to introduce you what is capsule because capsule it's an open source project and It's able to provide a multi-tenancy also. It's the topic of this session And after that max is going to share what they were able to achieve at work gaming net with capsule And he's going to share some really nice insights regarding the abilities of capsule and the platform that they built So I don't want to provide so much spoilers So I can just suggest you to enjoy it and see you later for the last session. Bye. Bye Goodbye, and I hope you will be interested Hi people, my name is Dario Tranquitella, and I'm an open source software engineer at Classics an Italian startup Based in Italy with the mission of bringing Kubernetes everywhere in this section I'd like to introduce you to capsule the multi-tenancy operator But first I think I should give you a brief introduction about the multi-tenancy Dailinma and the current state of multi-tenancy in Kubernetes itself and the ecosystem Multi-tenancy is nothing new if you're already familiar with the Kubernetes The basic resources you're using is namespace Allowing you to group application into an application logical group all the polls Deployments services ingresses and so forth of each application or maybe team in big corporations is Collect in namespaces. It's pretty easy to understand This is common and best practice Dividing components and applications in several namespaces, but doing so doesn't mean you're adopting multi-tenancy There are some aspects you have to consider since it's like the short hosting in the mid to Thousands and I'm referring to the noise-enabled effects We all know Kubernetes is great at crafting distributed systems and provides out of the box Some resources that help you to tackle the downside of dealing with a multi-tenant environment The resources I'm referring to are limit range resource quota and the well-known and famous network policy I'm not going to do a one-on-one introduction about this because documentation is there for this kind of stuff But I'm going to provide a brief overview since with the combination of these resources You can achieve complex multi-tenant strategies Said some limit range is the resource you have to use to set a minimum and or a maximum amount of resource requests Also known as CPU memory and storage These policies are useful to set the minimum amount apart as to use really useful when dealing with Unconstrained containers that can lead to CPU froggling or out of memory of node as well for the upper limit since few things are infinite And resources pretty sure are not except you are using a cloud provider and it's not your business Anyway, following we got the resource quota This API allows you to specify some constrained policies for a specific set of resources Let's imagine that you have to limit the amount of PVC requested by a namespace Precision volume claims This is the perfect use case as well as limiting the amount of pods to avoid the container sprawl and Finally the well-known network policy I guess we all know them because it's used as a sort of firewall for the applications that we deploy in our clusters So sometimes we are biased to consider them as a sort of security thing and this is absolutely true though However, a network policy must be used to deny communication between services that don't belong to the same tenant for obvious reason I'd say security in first place Okay, great Now we all know all the tools to build up your tenant platform besides the downside that all these Resources are namespaces scope and it means that the Enforcement of these policies is applied just to the namespace where the resources living and this doesn't apply well in a multi-tenant environment But exactly how a tenant can be defined Let's try to summarize how it can be interpreted properly in the Kubernetes ecosystem and I think the most basic definition could be a Collection of namespaces where each namespace is collecting the resources requested for a specific component or application owned by a team or a group of developers and In the end if you start thinking about if you are giving developers access to your Kubernetes clusters You are just granting administrative capabilities to the namespace. They have to manage and sometimes just in read-only mode And working with multi-tenancy with just a team is pretty easy straightforward You prepare the environments creating the namespaces set the Kubernetes resources We presented early and I'm referring to the limit range and through policy resource quota and giving assets to them But dealing with this at scale could be a nightmare and the automation must kick in for several reasons Capsu is our non-opinionated solution to solve this sweetation You got a custom resource definition named tenant that assigns a namespace creation right to their owners and of the story What you just need to do is specifying all business logic in the tenant manifest and the tenant owners can create on their own The namespaces that will be treated as a sort of single namespace the tenant abstraction But before proceeding with capsule is capable of I think it's better to explain how capsule was born and Yeah, I still recall the first call I had with Adriano the classics the classics founder and he asked me directly Hey, do you want to deploy and develop a Kubernetes operator for the multi-tenancy and my reply was even more direct Why do we need yet another project to deal with this if there is read-out open shift and range are doing this stuff? Because multi-tenancy in Kubernetes is broken was it sounds were a bold statement. Yeah But in the end, that's true And it's the reason why we got several other projects that are trying to tackle this goal And just being just thinking about the your a quick LMS based controller by Google or the virtual cluster initiative All these projects are living in the multi-tenancy special interest group And I said earlier that Kubernetes is great at design distribute system since it allows to just use the requested Resources and components to build your infrastructure. It doesn't mean it's out of the box you have to put in place on your own and The same applies also for the multi-tenancy and that's the reason why we started these lovely at momentum together at that time classics was doing scouting for one of its customer to provide a Public container platform a sort of hero cool, but built on top of Kubernetes If you prefer fancy namings and acronyms, I think that NAAS is the right one namespace as a service and jokes apart The prerequisite of the project was delivering a true and native Kubernetes experience to developers No need to add further APIs binaries to install YAML definitions and any fixtures that called puzzle turbines the goal was If you keep CTL from your terminal and there you go and we started the project with this in mind and After some time realized that we were building something really disrupting disrupting because I was I was working on it I did not but being non-opinionated on the multi-tenancy strategy provided Showed us that we were crafting a framework Allowing any or almost level of multi-tenancy Obviously starting from the public-facing scenario the namespace as a service One I've asked shaping in a smarter way the specification, but what changed our project and the project itself was the community And I'm really honored to be on this stage with max although virtually Since together we went further from the initial plan and this was the demonstration that the community is the real engine that helps you to delivering software solving people's problem and That has been a crazy and emotional ride with all the people showing interest and the filing bug or feature request But let's try to step down from the emotional talk and let's try to drill down the Capsule capabilities. I don't want to let this talk look like a safe speech So, um, okay. What are the capsule super capabilities? I'm trying to summarize in small and short sentences to leave enough room to max presentation that is absolutely stunning Okay, you can limit the namespaces that can be created by a tenant owner really useful for the pay-per-use scenario You can specify which ingress or storage classes a tenant can use as providing a tiring for the backing storage used by applications or forcing to announce applications using a specific load balancer and Since we are we are talking about networking at ingress You can put in place a policy for the allowed ingress source names as using a specific domain for the exposed applications Or you could mitigate the CVE 20-20-85-54 and forcing a specific set of IP addresses or ranges that can be used by the Service in the tenant and also from the security Perspective the container registry used by pods running in the tenant can be enforced as well as assigning additional role bindings to use policy security policies or maybe other CRDs and Last but not least the automation for the most important resources in a multi tenant environment and enforcement of limit range network policy and resource quota across all the namespaces of the tenant and Before proceeding to the final consideration I'd like to talk a bit about the bring your own device capability as with other resources Capsule as a reinvented wheel since it's using the pod node selector admission controller available in Kubernetes since the 1.5 release very correctly However, when all these features kick in together, you can achieve for real bring your own device Scenario because What you just to need to do is to spin up some worker nodes provisioned with a specific label Let them join the cluster and you can manage your reserved partitions of the cluster and it's absolutely Amazing since you are totally decreasing any side effect of a multi tenant cluster No noisy neighbors your own resources. It's like being on a managed Kubernetes cluster, but owned by your organization Anyway, I know what some of you started mumbling about. Mmm. I can already do that on my own I just need to write down the dynamic admission controls and Yeah, that's true, but there is something you have to consider the declarative way also know as the YAML The tenant YAML definition as you can see is easy to understand and easy to grow as any any other Resource in Kubernetes, but more it allows to combine all these policies together That will require otherwise a big effort on wiring all the webbooks together and building a way to group together The namespaces in abstraction the tenant one And keep a look to the node selector key That's the keystone enabling the bring your own device scenario beside the provisioning process Though that is still required to be configured at no startup I'm trying to recap all the features I'm pretty confident to say that capsule can be considered as a sort of framework to build your own multi tenancy solution Without disrupting the developer experience that will enjoy a fully compatible API server managing their resources using kubectl and without dealing with any complex CRDCs There is just the tenant one and it's up to the customer administrator to do to deal with it and Said so yeah I think I can pass the word max that is going to show us what they were able to achieve with capsule at work gaming net during the day of journey on using kubectl as a scale to avoid the famous cluster sprawl. So thank you so much and cheers Hello everyone as You may be know my name is Max Fedotov and I'm working as an infrastructure engineer in a maintenance department of wargaming In my part of talk I would like to share with you how we were able to redesign our internal Kubernetes clusters with the help of the capsule Why we decided to adopt multi tenancy and which benefits we got from this decision But before we start I would like to tell a few words about wargaming for those who are maybe not a big fans of a video games So who we are? We are a global online game dev and publisher with over 220 millions of players who are enjoying our games on all major gaming platforms And I think everyone had hit or maybe even played our flagship products like world of tongues or world of portions And in my department, we are in charge of supporting all the infrastructure which is used in order to run these games So what do we had before adopting capsule? I think this is the this was the most classic on-premise setup that you can imagine so you can find it at any Kubernetes guides As we are operating in multiple regions We provisioned a single Kubernetes cluster per region because it is much easier to manage And we were able to provide additional in cluster services and integrations with our internal system like monitoring login Authentication and so on we used shared nodes a mix of hardware and VMs and we created a separate namespace for customer Which are our internal development teams in order to provide type of isolation We also used Kaliqa global network policies for network isolation and At the beginning when we hadn't got a lot of customers this setup was quite good for us But things changed when clusters start to grow and we got more customers on board. So what was wrong? The first problem was that namespaces are too granular Most of our customers needed more than a single namespace. They needed a namespace per project or namespace per application So we had to manage a namespace provisioning for them and we also managed access and permissions Not provisioning was also on us So we had to do some type of capacity planning and management and another problem was billing So when you had shared nodes you need to implement complex billing rules and that's not easy at all really and Also due to shared resources It was very easy to get a noisy neighbor effect that Dario already mentioned in his talk when customer workloads began to interfere with each other So you will ask me an obvious questions if the shared model was not so good for us why not to use a cluster per customer and That was one of our initial thoughts too, but in this scenario There were also some disadvantages for us. The first one is that we have six regions of presence So in reality for each customer we need to provision not a single cluster, but six clusters so we multiply the number of customers by number of regions and We understood that it will be too much really for us So the maintenance cost for this solution will be quite high for Another problem was that we had to support different version of Kubernetes. Why because you know like it's always in a big companies There is a customer A who wants to have the newest version and there is a customer B Who wants to have the most stable solution and that's a problem for us because we had different tools Some of them were our internal tools Some were different open source operators that we installed in order to provide additional in cluster services and Sometimes we even modified fork them so they fit best for our environment So we had to build and maintain a separate version for each tool for each version of Kubernetes and It may seems like not a big deal, but still it adds something to maintenance cost too Also, we were a bit afraid that when everyone will be sitting in their separate clusters It will be hard to share some knowledge of services or ideas So what we wanted to is that if one of our customers had adopted some new technology or had brought or implemented some operator He should be able to provide these as a service to all other customers So as a result this separate cluster pro customer approach seemed more like a Kubernetes as a service and What we wanted to provide for our customers was a managed Kubernetes solution So we understood that having a single cluster per region is the best model for us And what we needed was to brought some advanced multi-tenancy capabilities to Kubernetes So we wrote down some critical points that we wanted to have in our new cluster design So the first one was that we wanted to have a bring your own device approach So customer can bring his own hardware nodes or he can provision a VM in some of the cloud provider and His workloads will run only on these dedicated nodes This also helped us to simplify billing as there is direct mapping with nodes and customer We also wanted to have more self-service for customers So they can provision their own namespace if they manage access and permissions and them So if it possible the best option for us would be that Customers think that they're living in their own private cluster while in reality They're located in a big shared Kubernetes cluster and also important was to keep vanilla Kubernetes experience So when our users are reading Kubernetes dogs or they are reading some articles in the internet They don't need to think if they think they're reading about they reading about will work on our internal Kubernetes cluster And as a good engineer, I've started googling and reading about current situation with multi-tenancy in Kubernetes And as Dario already mentioned in his talk, there are problems with multi-tenancy and I will definitely agree with him on this But then I found capsule and it was really a perfect fit for our demands So we became one of the early capsule adopters and soon it's became a central component of our internal Kubernetes clusters And what I really love about the capsule Not saying about some technical details or technical part. It is its community and maintainers So when you are adopting some new open-source technology, you always found out some use cases in your environment that are not Already covered by it. So you need to add some functions Or maybe you even don't understand how better to use this new product and Dario and other maintainers were always so helpful and they helped us not only with Adopting it, but they also helped us to contribute different features and I'm really very proud that now I am also one of the capsule maintainers too So with the help of capsule we were able to formalize two core concepts that we are now using in our internal Kubernetes clusters first one is tenant. It's our customer So it has some name a group of owners who had an admin privileges in the tenant and a cost center Which is a label that we applied to Tenants node as a tech and then later we use it for billing so we can directly send bills for a for our customers Another one is a node group It is a set of nodes with the common flavor that are created in one of the cloud providers And they're always bound to some for some tenant so you can think about it like a cloud agnostic obstruction of AWS after scaling groups or a Google cloud after scaling non groups and not group also had a name Which we apply as a label to all nodes so our customers can use it later in label selectors It is created in some of the cloud providers and all nodes in the non group had the same common flavor For node group. We also introduced a concept called role. It's some type of a predefined template So for example for ingress role, we install and configure an ingress controller on each of their nodes of the node group And the last one is min node in max nodes These are used in order to support auto scaling And we have a set of core components that we are using to in order to implement this concept First one is called provisioner. It's our internal component which is in charge of provisioning everything So it subscribes using web sockets to our internal configuration management databases Database where information about tenants and non groups is stored and then it's managed tenants ingress and storage classes provisioning as well as Nodes group provisioning in different cloud providers. So this is the core component that communicates with all cloud providers Next one is capsule and The first thing that capsule helps us is with implementing bring your own device approach So when a node is registered in Kubernetes, we add a label with the name of the tenant on it And then we use a capsule node selector to ensure that tenants workloads will run only on dedicated tenant node groups We were also able to get rid of a shared ingresses When a customer creates a node group with ingress role provisioning creates a dedicated ingress class for it and then Capsule forces the application in the tenants to be published only by the assigned ingress controllers another feature that we get I Was additional service service abilities So our customers are always mapped to a dedicated LDAP groups and when a customer creates a tenant We assign his group as a tenant owner so they can manage everything inside their tenant without any need without any help of the cluster administrators and The last one is isolation. So capsule had an ability to assign specific labels or annotations on tenant namespaces and tenant services So namespace labels are used in order to achieve network isolation with Coleco global network policies and Our tenants are always isolated from each other and with the help of labels on services We are able to make not boards to be opened only on the tenant nodes and I will tell you about How we will be able to do this on the next So a few words about not board and multi tenancy we are using Couproxen IP tables mode and by default it watches services and creates IP tables rules on all cluster nodes and That is okay for a cluster IP services, but how to deal with the not boards in a multi-tenant environment So we forked Couproxy and added additional configuration flag with the name of the tenant on whose node Couproxy is running And then we added additional label selector for the watch where this configuration flag is passed So our Couproxy watches only for services located in the same tenant as Couproxy is and now Not boards are opened only on the tenant nodes Couproxy is another member of Coupsul ecosystem. It is a simple reverse proxy that intercepts specific requests to the IP service So for now nodes and namespaces and points are already in open source version of Coupsul proxy And the last two ingress classes and storage classes. I know a private fork, but we are going to contribute it soon So Coupsul proxy modifies users requests So they only see resources available in their tenant and it provides users an experience that they're working in their own private cluster And The last company we have is a tenant after scaler It is a forked version of cluster after scaler which watches for unschedulable pods only in the tenant namespaces it has also We also had an implementation a custom implementation of cloud provider interface for it So instead of directly interacting with specific cloud providers It talks to our provisioner and that is the way that we are able to support nodes from different cloud providers in a single tenant If we take a look from a user perspective, they're working with us the same way that they will work with any other cloud provider So we created a dedicated mean portal for them where they can create and manage their tenants and known groups So instead of creating a bare cluster in one of the major cloud providers They can create a tenant and non-groups in our internal cluster where they had a lot of additional services and Integrations with our internal systems out of the box And of course they got a premium support from our engineers who are close to them and who know Their opinion who know how they work So of course, it's very hard to compete with big cloud guys But with the help of capsule we can be a cloud provider of their choice due to additional value that they will get from working with us That's all that I wanted to Share thank you for your time and giving work back to Daria Okay, welcome back people and yeah, it has been really Enjoyful. I really enjoyed the presentation by Max because what they achieve that work gaming is absolutely stunning I'm really happy that Max help us also shaping the capsule capabilities and features and What do you think Max? Did we do a great job? I? think yes, and thank you Daria for making capsule available for the world because mostly it is Your big achievement and I was just a guest who joined in a time Yeah, yeah, probably I had also to share our first Meetings because I recall when you open the first PR Regarding feature requests or also little optics. I recall the post support one and it was the broken Binary search into the capsule user groups. Do you recall that? Yeah? Yeah? Yeah? Yeah? That was awkward. But anyway, I'm really happy because we are shaping a great product And as I said also in my presentation of capsule is able to be a framework to build your own Multitenancy environment on top of Kubernetes. So it's highly non-opinionated You can do whatever you want and obviously we are planning further improvements Regarding also the ecosystem Max because you shared also the capsule proxy and the capsule proxy is playing a big role into the multi tenancy because in the end We are using the capsule proxy just to provide the list of the namespaces owned by the tenant But we got also another project and that we had a time to share and it's the lens extension so lens it's an idea developed by Mirantis and it's really useful because with this idea you can you can see all the stuff that is running in your Kubernetes cluster and There is also the ability to add some extensions and we at classics we developed that with our partner Nikolai and Adriano and It's quite interesting and I'd say that it's really useful. Well, it's not quite but it's absolutely useful and take a look to that and I would like also to take this chance to do To ask for people if you like capsule if you are also considering the multi tenancy Topic really interesting Please drop us a call on Twitter on LinkedIn or github because we are looking for contributors And also if you have feature requests or you are facing some bugs Don't hesitate to open the issues because as I said We are really happy to get feedback from people So don't be shy anything else to add max or if you just need some help about how to build multi tenancy Watch options you have you can also join us on the github and we will be glad to help anyone Yeah, yeah, absolutely. So don't be shy. We are looking for you and thanks for all the time Attening this call and it has been amazing and thank you so much. Thank you. Good. Bye