 So here we go again I must say when I don't I work on Kubernetes like I think a couple of you here and when I when I read the outline of this talk I must say I absolutely don't care about Kubernetes. I study physics So for me, sir, and is the mecca or something close to it So when I was particularly excited to to to lead to hear what Jack our friend Jack ancient engineer Cern has to tell us so please give it up for Jack Hi everyone, thanks for the nice introduction Yes, so I'll be talking about how we're using Kubernetes and OpenShift at CERN And I was getting a little bit worried before When we had to talk from ing from from Robin about how they're having their namespace as a service product But I promise you they will simply enough new things in this talk here So first a little bit about me so I studied it security and cloud computing at Alte University in Finland and in my free time I like cycling and also hiking in nature and I am then I'm also a engineer at CERN Which Is actually quite nice because this is the landscape of CERN So we have the Lake Geneva here and at the tip there is Geneva you can see in the background we have the Alps close by so that's really good if you're into cycling and hiking and Yeah, here on the right side. There's the main campus and there's a red ring here That's the LHC the large head on collider That's this huge underground proton accelerator that we have where lots of experiments are being performed You might also know that this guy was also working at CERN sir Tim Berners-Lee and he invented the worldwide web there and and What that also means is that we have a long legacy of running web services at CERN because the web was invented there and And most of these web services. Well, actually all of them are running in this data center. So this is the CERN data center So basically everything we do is on-premise But actually 80% of the workload that's running in this data center is just what we call physics workload So it's just running simulations computations analytics machine learning et cetera et cetera But more and more of our users of course also want modern ways of deploying applications Doing it in a cloud native way and we as the CERN IT department offered two ways to do that we have a Kubernetes service that is based on OpenShift Magnum and or sorry on OpenStack Magnum and We have a managed Kubernetes service that is based on OKD and We frequently get the question. Why do you offer both like I mean Managing like doing a Kubernetes management is already enough of a hassle. Why do you duplicate the effort and This is one of the things that I want to explain in this talk because both of them have really different use cases and In the end I will also highlight the benefits of kind of splitting these use cases So let's first have a look at the the benefits of an unmanaged Kubernetes cluster So it's basically like a VM. You create the thing We give you an SSH key and you are rude on that thing. You can do whatever you like You have the full control if you want to install packages or in the case of Kubernetes You want to deploy a service mesh or you want a different CNI or whatever? You can go ahead, but you also need to know what you're doing so it requires more skill more knowledge from the user and Further on you also need to Understand how to scale your infrastructure But this is then one of the nice things because it's just your cluster You can for example have dedicated nodes in the cluster that let's say have GPUs in them So you can run your machine learning workloads Or you can take Kubernetes nodes that have more memory if you have very memory intensive workloads and all of these things are Aspects that you can customize when you have your own cluster And finally one of the benefits that we also see is that with a dedicated cluster You can really decide which type of dependencies you want to take on so let's say for example you have a stateless application and you basically just want HTTP traffic in and Then you're connecting to a database somewhere else in that case your cluster doesn't need any storage So it doesn't need a dependency on this additional service and the lower the the lower depend now number of dependencies you have the lower the Risk is that you're gonna have some kind of outage because one of your dependencies is not available on the other hand the benefit of a managed Kubernetes service where you only have access to the API and not the underlying infrastructure is That we can optimize the resource usage usage much better This is also what Robin mentioned in his talk earlier today Because by putting multiple different tenants on the same cluster we can utilize The the inefficiencies between the different workloads much better or we can kind of smoothen out Traffic peaks and bursts in whichever workload you have In general also we can it's better whether if we have smaller and medium-sized workloads So if you have just a small Python application for example or just a standard Java application That doesn't need like a lot of resources, then it's it's much more efficient to run this in a shared cluster And finally the the most important point is that you have a managed infrastructure So you as a user you just come to us and you basically have a have a container as a service platform So you you give us a pod and we just run it for you And that's all you need to take care of and we take care of handling security upgrades for the for the Kubernetes infrastructure for the underlying VM infrastructure for Anything else that needs to be done replacing nodes if they are unhealthy all of these things And Finally especially with with okd or open shift you also get a nice user friendly web UI And I really want to to highlight how important this is and how many of our users for this is This is actually their main entry point to our platform because yeah We as experts we know how to use cube CTL or OC or helm or whatever is our the command line tool of the day but Users are really happy when they can just see their pot that is running and they can click on it and they're gonna see the logs and This is just really nicely prepared and it it's a very smooth entry to the kubernetes cloud native world now at CERN CERN is an academic institution a research institution and that means our users also have a lot of academic freedom To put it lightly so we don't have a golden path for deploying containers Some people are just using the web dashboard like I just said others are using cube CTL Maybe from a from a CI CD pipeline others again are using customized manifest or helm charts And then we see that the most advanced users have already adopted github's approaches such as flux or Argo CD But we also see that there's not really a one-size-fits-all solution because we have a very very wide Range of users we have people who are from inside it to who are absolute experts We have time to understand all of the details of kubernetes yaml But we have also people who maybe just wrote a small dashboard so they can monitor what they are what their Sensor is doing or maybe they just want a small application that checks if their job that they have submitted to the batch service is still running and They don't care about all of these details. They just have their python app or or JavaScript app or whatever and they just want to put it somewhere and just let it run there and In this case of course just using a web dashboard to create this deployment once and just let it run there and updating the Container image whenever I need to is perfectly fine One thing that is very important for us at CERN is the topic of resource management so all of our projects have a well-defined owner and Usually also an administrator group and this is necessary because again as a research institution we have a lot of people coming and going people who are maybe only temporarily associated with CERN because they have a let's say They are doing a PhD thesis or or maybe just a student thesis So it's really important to keep track of the owner for us So that resources don't get orphaned and then something has been sitting there for five years and we don't know if we can turn it off and If we turn it off if it's if something is going to blow up somewhere that we don't know about And finally also associated to these projects are also always resource quotas That kind of restrict how much you are allowed to use because we don't do internal accounting So we don't account for how many resources you're using But that that's why we need a restriction a limit on by default how many resources you're allowed to use Of course people are allowed to request more resources if they need to but by default we have low quotas so let's get into the topic of how people can get a Kubernetes cluster at CERN so this is a service that is managed by AT and as I already mentioned before it's based on OpenStack Magnum and which completely automates the initial provisioning of the cluster and Our team has introduced feature toggles So that you can enable or disable very common integration. So this is things like That your that your Kubernetes cluster will report to the central monitoring service that it will log And that it will also send the logs to a central location Which kind of storage endpoints you want to have All of this is very flexible and and this is what I mentioned before is what people appreciate about the service that they can Really choose which integrations they have and also then of course, which dependencies you have and The team is using ARGO workflows since very recently to do continuous testing So spinning up a new cluster running the entire test suite over the cluster and then deprovisioning it again In practice it looks something like this So you just run an OpenStack COE cluster template list and then you can see which kind of flavors are currently available Then you run a cluster create command and you specify which flavor you want to have you just need to specify also then an SSH key so that you can access your nodes and Finally, you can specify additional labels. So in this case, I'm enabling the monitoring component and a couple of minutes later After running this command your cluster will be ready Pretty simple for users who know what they're doing Behind the scenes it looks something like this. So we have at the bottom here We have the OpenStack layer. So where we have compute so that means VMs And then Magnum is the OpenStack component that takes care of the provisioning together with the heat component and this kind of spits out some resources so the virtual machines etc and then the actual cluster configuration gets applied with a Helm chart and This is where these feature toggles feed into into this Helm chart, which will then Kind of configure your cluster the way you initially requested it But I should note that this only happens during the initial provisioning So afterwards you can really do whatever you want if you want to take out the Prometheus and put something else there That's totally your thing. You're free to go on the other hand. We're also using okD so that is the community edition of Red Hat's OpenShift container platform Which by now since a couple of years is really the foundation of the web services infrastructure at CERN and that is because it provides a multi-tenant highly available and secure base It's it's really amazing if you compare a vanilla Kubernetes deployment to an OpenShift deployment How much you get out of the box? We then go ahead and enhance this kind of out-of-the-box okD With additional features that are either requested by our users or that are just necessary Integrate well in the CERN computing environment. So these are things like registering your hostname for you the ingress in the DNS servers handling automatic certificate renewal and requests Very important also for us and for our users is the fact that we provide lots of different storage integrations So our main persistent storage is CFFS But we also have this thing at CERN that is called EOS Which is a giant shared files system which basically houses all of the physics data that it gets produced at CERN And is partially backed by by tape drives actually and then there's also things like CBMFS Which is a which is a basically a software repository that gets served via the network and so these these things are Something that you don't get out of the box even with okD We also handle custom ingress router sharding and something that I will come later to is we also add our own operators on top of okD So this then what I just described is then kind of the shared base of okD and Based on this we then have different cluster flavors that are specialized in a particular use case So our our most generic and kind of lowest complexity use case is the platform as a service Which currently houses around one thousand four hundred different projects and it's actually our largest production okD cluster And this is really just container as a service So users just put pods inside or well deployments to be precise And and we run it for them and we take care of handling all of the infrastructure But basically the user sees a completely normal Kubernetes API We also have what we call the app catalog that gives the users Much less control over the application and they only have access to predefined application templates so these are things like Grafana instance nexus wordpress and You don't actually have a deployment in there But instead you'd only deal with custom resources that are backed by operators that we have developed Which will then deploy the application as necessary for you Then there's also the the web yours cluster, which was actually our first okD cluster Which is at the moment housing around four thousand projects and this is basically just a shared web hosting cluster So it's serving files from the eos file system that I that I mentioned before But at the same time It's highly multi tenants. So it yeah, like I said, it's housing for 4000 websites, but as you can see from the numbers it's actually only 20 nodes so it's very very efficient at what it's doing and Sometimes also serving terabytes of data in very short time when one of our experiments for example releases new data on the web and people start downloading it and Finally, there's the Drupal use case where our colleagues from the Drupal team have developed an extremely advanced operator For running Drupal websites because Drupal is the most widely used content management system at CERN so all of the main websites are currently using it and this Drupal operator provides a really simple interface for users to say I want a Drupal website with that version and they don't need to take care of anything else and then the operator goes to the cluster or is running inside the cluster and Creates deployments as necessary Provisions databases make sure that database migrations are applied and so on and so forth Now we don't of course want to bore our users then with having to write YAML manifest So they that our Kubernetes operators know what they need to do Instead we have this thing called the web services portal which exposes all of these features in a really user-friendly UI So for example to create a website on this web use cluster that I mentioned before you basically just need to fill out this form Where you say what the hostname should be and what the description is and from which path it should be served and you click on create But what happens behind the scenes is that this portal actually does not have any state itself It just takes the permissions that the user has and creates a resource in the cluster In this case a custom resource definition or a custom resource With the parameters that you put in here The same happens for the Drupal site So again, I'm just I just need to say what the hostname should be and then kind of which version of Drupal I want and I click on create and behind the scenes something like this gets put into the cluster and We entirely rely on the role-based access control that Kubernetes already implements and that the multi-tenant Okd gives us out of the box So if the user is in principle to allow to create such an object against the API Then our web services portal will do the same on behalf of the user and You can see that the same fields kind of show up here again So we can see the site URL we can see that the Drupal version has been specified And then we can see what the the operator has been doing. So the deployment has been rolled out as expected It checked if the any database Migrations need to be applied and which backups are available and all of this is completely automated. I Also want to mention how we actually manage these clusters so we have these four clusters that I showed on the previous slide in in production plus additional ones in staging and deaf environments and Contrary to to the current trend. We are actually treating these clusters as pets because they are stateful in our case Users are putting their workloads directly into our clusters be it a Drupal site or a regular Kubernetes deployment And we need to make sure that we don't lose it because we cannot just reprovision the cluster because then all of this state would be gone Furthermore we have we have implemented this architecture that each cluster that we create is completely self-sufficient and Isolated so it has a separate open stack project. It uses separate application registrations everything is isolated from each other and we don't For example connect them with different Argo CD deployments or something like this and this is very important for us so that when we make a mistake We reduce the blast radius as much as possible Finally, also, we don't really have a need to like other people to throw clusters away and create new ones and move the workloads Because okd4 makes in place cluster upgrades completely seamless I don't know who of you ever tried to upgrade an open shift cluster But it's really amazing open shift by itself We'll just go through the entire cluster and update each component that it needs to in the right order and highly available And as a user you don't even notice that it's happening So it will first update your it CD nodes and then it will update your API server and so on and so forth And it goes through each one of them Now as I mentioned before we deploy a lot of things on top of okd actually and for this we're using Argo CD Which is I'll come to in in in a future slide which is a very nice integration into this way of managing an okd cluster and Finally we recently also developed and what we call okd CTL tool that allows us to facilitate common operations Such as creating a cluster deleting a cluster Performing maintenance tasks such as replacing nodes or maybe freezing a work node so that we can debug something Overall the deployment then looks something like this. So at the at the bottom. We still have our open stack cloud Then we put okd on top of it and our CD is kind of the brain of the entire operation So it's managing both the okd configuration because if you're not familiar with open shift Everything in open shift is managed through an operator itself So if you want to change the configuration of an API server, you just edit the API server CRD And you need to change some parameters in there and then the operator will go ahead and actually put those changes onto the nodes and Similarly for all other components such as image registry, ingress controllers, etc and In addition to managing okd Argo CD also manages all of the other things and this is actually just a small sample of the things that we deploy such as Manila and Cephaphase integrations for the storage. I previously mentioned third manager open policy agent Then also integration with the certain single sign-on, etc now Using Argo CD for this is really great because it's a natural extension of the of the of what we are already used to in Kubernetes It's this continuous reconciliation of resources If a resource is not in the state that we declared it to then Argo CD will just make it that state and then this is very nice because The operators either our own operators or the okd operators will then pick up this change again So eventually this the cluster always converges into the desired state that has been described by a kid repository and Well, what in in addition what we also like about this approach is that if something cannot be converged into the desired state We automatically get an alert about it because the Argo CD application is out of sync then So it might not be that something is necessarily broken in the cluster So that the cluster is completely offline in which case of course we would also notice But maybe there is just a small misconfiguration. Let's say in the DNS setup This would then also get picked up by Argo CD And if it cannot ensure that the resources in the desired state, we will get an alarm about it So it combines really well with this operator driven management that is already built into okd clusters and finally We are Argo CD both the the command line and the web UI also gives us a great Overview of the resources that we have manually deployed into the cluster and by now they're really a lot like we're deploying hundreds of Resources into the into a cluster and of course you cannot keep all of them in in your head or always understand where each resource is coming from But this is why the Argo UI is really nice to to kind of understand Where is this resource coming from which stage should it be in and why is it there in the first place? I also want to give a special Spotlight to the open policy agent, which is a component that we're really using for many many different use cases Including helping us as the administrators, but also helping our users with some tasks So for example helping us the administrators is Ensuring that each host name because we have this shared namespace of or shared DNS zone of Of host names that users can use is only used once across each cluster so that you cannot steal someone else's DNS name It's also used for for the ingress up setup for volumes For networking visibility so that an application for example is only exposed on our internet or to the internet or on the Technical network which is then another internal network that we have But also for more tedious tasks like automation of mount points So for this when when a user wants to mount this use file system We need to make sure that it has valid Kerberos credentials and then these credentials expire after 24 hours so we help users by actually Injecting an internet container and a side guard container automatically into their deployment that already has Sets up everything as it needs to and the only thing the user needs to do is set a small annotation on their deployment and Open policy agent will take care of the rest Like I mentioned operators ok DS operators We're using upstream operators such as developer backup operator or a third manager for requesting certificates But we have also developed a lot of operators ourselves and these are what is powering these other clusters that are not the generic platform as a service So we have the web use operator that takes care of handling this shared site hosting We have a githl app pages site operator that integrates with our githl app deployment We have the authentication operator that takes care of the entire project lifecycle and tying it back into the CERN single sign-on We have a land to be operator land to be is CERN's internal Kind of DHP management Solution so that needs to be integrated with DNS if you want to request a new hostname and Finally also for each of these application templates We are using helm operators, which are a very thin layer if you're not familiar. So it's you basically The operator SDK allows you to build these helm operators just by giving it a helm chart And then based on the input values of this helm chart, it will automatically generate a CRD for you And this is nice for simple use cases such as Grafana, for example, where the user just needs to Needs to say I want this version of Grafana with that and that hostname Then the operator basically just runs a helm template once and installs the necessary resources in the cluster and again It continuously reconciles those Now what did we learn during this time of operating these services? Users are very happy about the internal documentation that we have This might be something if you're if you're building such a service you might be Tempted to just skip because well, there is so much great upstream documentation and that's true But still your users like to have a friendly entry point into your specific service and then from there You can link to external resources We're also very happy that Operators can help us in so many different tasks both for us as admins, but also for the users but we also really need to pay attention how we use them because they are very sharp tools and We just recently two months ago had a pretty big incident where one of the operators was actually Misconfigured and started deleting a bunch of resources in the cluster This is where I want to point out that wherever possible you should try to use soft deletion So market resource as deleted market is unavailable and then just clean it up 30 days later At the same time not everything has to be automated We Frequent relatively frequently have steps that we could automate But actually it would not be worth the effort because in the end we just need to run one or two commands for five minutes And honestly in five minutes. You cannot reliably script it Finally splitting the Kubernetes as a service offering at CERN into both this kind of unmanaged flavor and the more managed flavor Is really beneficial for both services because we can share expertise knowledge and Experiences but at the same time we can serve very very different use cases that we could never satisfy with either one of them If we just did unmanaged Kubernetes because then most of our users would just be completely Overloaded but at the same time if we only had the managed service We also could not serve all of our users because some of them just have really special requests Some of them have different Compliancy or auditing requirements. Some of them have different compute requirements Others just want a service mesh that maybe we cannot offer to all of our users because it's too complex to implement So having these two separate services overall is very beneficial for us With that I would like to thank you for your attention and thank the organizers of this awesome meetup for putting this all together, I think it's really well organized and You can find the slides under this link. Thank you very much. Thank you very much Jack I Just have one question for everyone before I open up for questions. How many shiver when he said tape There you go. I Impressive. Thank you very much. Do you have any questions for Jack? We have time for three questions Yes, we do So it's for us. It's it's internal users So well if they want to run their own cluster they can we don't forbid them from doing it This is again something that we really try to adjust to whatever the whatever the people need They are also unmanaged completely unmanaged Kubernetes clusters. It's really coming back to that academic freedom If you want to do something You can I've noticed that you got like thousands of different projects and I'm curious how do you handle kind of API updates in terms of Workhold specifications that change like from One version of gates to another How do you handle that for so many years? That's that's a good point. So as I would say As a starting point, okay Do you really helps with this because well you have all of this support from from red hat open shift? That they already make sure that you get warnings early enough Hey, this thing is gonna be deprecated into Kubernetes versions like you should really look into Fixing that now like what we recently had with Kubernetes the cron jobs that the one beta one was deprecated So that is one thing and for our internal self self-developed crd's there we can actually Kind of collaborate with our users and do the patches on the fly as we need to Let's say if something changes in the spec we can say hey, you need to modify this or that It's gonna the behavior is gonna stay the same. Just add that field there. I do have one question. Jack. Yes, please How do you guarantee a good security posture of those clusters that you end totally over to your users? I Would say we don't guarantee it You can definitely mess that up. We try to provide a good base as a starting point and the sufficient references then There's an additional check that Unlike unlike these okd clusters You basically need if you want to expose it to the internet for example You need additional approval from the security team and in that case They will also check that what you're doing in that Kubernetes cluster is actually somewhat secure So those unmanaged clusters by default. They are not exposed at least to the internet Thank you very much Jack So CERN Impresses as usual We we are going to have the next talk at five o'clock Just before the next talk we're gonna have two very short pitches from our very beloved sponsors and we are So you guys have ten minutes to stretch your legs if you like so should you want to But this is the best room so come back here, please Thank you