 Hello everyone, and welcome to our talk, Kubernetes from Scratch for Neuroscientific Research. So a little bit about the agenda, so we will first tell you a bit about us and the target audience for this talk, and then how did our interesting Kubernetes start, and then a bit about prototyping Kubernetes and our organization's infrastructure overview, and then how it is to run a Kubernetes cluster and do maintenance, and then the use cases that we have within the organization, and the process of migrating to Kubernetes and how it is from a Kubernetes user's point of view. So about us. My name is Karolina Lindquist, and I work as an SRE at EPFL in the Blue Brain project, and I'm in the team for Neuroinformatics Software Engineering, and my team is developing a storage application for Neuroscientific data. Hello everyone, my name is Daniel Fernandez, I'm also an SRE. I've been in the Blue Brain project for the last three years as an SRE as well, and I'm in the core services team, and we can think of us as a small IT department for the organization. We take care of different IT services for the users of the Blue Brain, and we are a small team of six SREs. So a little bit about our organization, the Blue Brain project. The ultimate goal is to digitally reconstruct and simulate the mouse brain, and also pioneer simulation neuroscience, and we hope that it will better help the understanding of the human brain also with some potential applications in health and disease research. And as you see from the organization chart, we are coming from different parts of the organization, and this was done as a collaboration. And about our presentation. So we want to give you some ideas for how to approach an on-premise cluster setup, and these ideas can also be applied to a cloud environment, and it's more intended as giving you some pointers to start, and also some of our slides might have a lot of text, so we should also be able to use them as a reference after this talk. And some useful prerequisites, since this is the 101 track, is basic knowledge about Kubernetes architecture and basic knowledge about some resource type, and some knowledge of Helm and Helm chart is a plus. But again, if you haven't learned about this yet, you can always come back to our presentation later as a recording. So how did our interest in Kubernetes start? So within my team, we were using OpenShift, which was the organization's container platform, and in the setup that we had, there were some features that were missing, for example, flexible language configuration as NGINX was doing it, and using Helm charts was something that we wanted to do, but at that point it was not possible. And then we started realizing together with the developers that Kubernetes, vanilla Kubernetes might be a good solution for us from the developer point of view. So we started a bit of prototyping and figuring out how to deploy Kubernetes on OpenStack, and started creating an Ubuntu-based Kubernetes cluster for internal testing. And then we did some documenting of the installation and the deployment process, and then we presented our test results to Daniel's team and started discussing the next steps. And as you can see from the timeline, this internal testing already took a few months, but we had a prototype and some use cases that we wanted to do that we could hand over to Daniel's team. Okay, so I would like to start my part of the presentation talking about the BlueBrain project infrastructure, and as you can see on the slide, we have three different data centers. First one in Lugano, where we have our supercomputer. We also have an 8-petabytes spectrum scale for scientific data storage, and a TSN system for back up on tapes. We have another one in Lausanne, where we have our disaster recovery storage based on NetApp. And finally, we have another one in Geneva, where we have our private cloud based on OpenStack. And this is where we run most of our services from, including Kubernetes. Also in Geneva, we have a little bit under 4-petabytes of enterprise storage based on NetApp. And this is the catalog of services that we offer to our users in the organization. So as you can see, we have things like OpenStack. We have Puppet. Puppet is our configuration management tool that we use to configure and install all of our services. We have things like GitLab, Jira, Confluence. We offer several different databases technologies to our users, authentication via key clock and all that monitoring stack of tools. And all of this I just mentioned is all virtualized and all on-premises. So with this in mind, let's talk now about the situation we had before Kubernetes. So before Kubernetes, we were running the open source version of OpenShift. So OpenShift Origin 3.11 based on Kubernetes 1.11. And by the release date, you can already tell that it was getting quite old at the end of 2020. One problem that we had is that people who work on this deployment had left the team already. And also it was relying on an Ansible Playbook. This is not a problem per se, but as I mentioned before, we used Puppet for configuring all of our services and OpenShift was just the exception. Because of the way it was deployed, day-to-day operations were tedious and cumbersome. So things like adding new worker nodes or changing the guad of a project required several different steps from the operators and it was quite error prone. And finally, it didn't have support for some of the features that our users were starting to request. Things like HelmCharts, GitOps, or a more modern Ingress. So this is why a change was needed. So at the beginning of 2021, this is when we started this proof of concept based on vanilla Kubernetes. In this case we went for Kubernetes 120 with container D and run C. And since the beginning of the POC we had a few things clear. We wanted to be virtualized running on our existing private cloud OpenStack. We wanted to use Red Hat Enterprise Linux system and of course with SC Linux enabled and set to enforcing. And we would use Pappet for the installation and initial cluster configuration. So no difference with any of our other services. And also we would like to use some kind of GitOps approach to automate the configuration of the cluster. As you can see there, there is a little timeline. It took us around three months to get the first version up and running. And then starting in April this is when we tested it internally for around six months. And then in October we decided to invite some external users to this proof of concept Kubernetes cluster. And then in December after some testing we decided that it was production ready. And after that this is when we started the commissioning of OpenShift. So this is some of the features that we offered back in 2021 with that first Kubernetes production cluster. As you can see it was based on Kubernetes 1.21. For the CNI we went for Calico support for Ingress NGNX controller, authentication via key clock. For the CSI we went for Trident since we were using NetApp as the backend. Kiverno, support for Helm charts GitLab registry, since we were already using or installing having a GitLab in-house. Support for VN spectrum scale, VNFS mounts, flags encrypted secrets with Mozilla SOPs and SERT manager. So it's the Q3 of 2021 and we had this new Kubernetes cluster up and running but we were also running the old OpenShift in parallel. So at that time what we did is we audited everything running on OpenShift and we contacted the responsible for those deployments and luckily a majority of those users those developers they managed to migrate their applications by themselves and in most of the cases it was just a matter of exporting the resource as a Jamel file and then re-importing it back into Kubernetes. Unfortunately for us there was no way to automatically migrate the volumes from OpenShift to Kubernetes although in both cases we were using the same backend NetApp in this case the driver was different so an automatic migration was not possible. So what users needed to do is basically copied the contents from one volume running in one system into the other. And finally we also needed to migrate the container images and this was an easy migration what users needed to do is push the already existing images into the new registry in this case GitLab. So about our cluster deployment as I said is 100% on premises 100% virtualized running on open stack for the operating system we are running Red Hat Enterprise Linux 8 although at the beginning we started with 7 and around 1 year ago we made an operating system upgrade the current version of Kubernetes we are using in production is 125 so that means we did 5 upgrades in the last 2 years and we tried to run n-1 version of Kubernetes in production being n the latest stable of Kubernetes. Currently we have 2 different clusters brought for production and depth for development architecture wise they are both the same we have 3 control planes of both of them where we also run HCD and n amount of worker nodes yeah both clusters they have the same configuration this is enforced by Puppet and then by Flux but of course in depth we have fewer and smaller worker nodes so we don't waste resources there since config wise they are both clusters identical we encourage our users to run some of their test workloads on our depth cluster so every time we change some configuration file or we upgrade the Kubernetes version they realize if they need to change something on their side first there so about the usage with this slide you can get a pretty good idea of how big our production cluster is we have around 500 allocatable cores at the moment a little bit under 1TB of allocatable memory and a little bit over 1TB of requested storage it's not here on this slide but if you're curious we have around 450 pots so let's talk now about multi tenancy because this was a very important concept for us because this comes out of the box in OpenShift via OpenShift projects and we wanted to have something similar in Kubernetes since we were using Flags to configure our Kubernetes cluster we based this multi tenancy configuration on that repo the Flags 2 multi tenancy repo although we ended up changing it so much it's basically a different thing but it was a good starting point and the way it works is it's team in the organization gets their own namespace and then we define some resource quotas per namespace so we can limit the resources they can use and then for authorization there is a role binding referencing an LDAP group and then each member of this LDAP group gets admin access within that namespace and when the user is logging the user the groups the user is member of are passed in the IDS token as a claim another thing we implemented is this isolation between pots so by default we have a very restrictive network policy that basically blocks all ingress and egress traffic and we wanted to have it like this so our developers need to make a conscious decision if they want to open access from and to their pots and finally we offer optional Flags integration for those teams that they would like to start using Flags let's talk now about the installation so as I mentioned before we did it with Puppet so for this we used the Kubernetes Puppet module developed by Puppet Labs and if you are familiar with Puppet this is not the typical Puppet module it comes alongside with a container image that basically generates the certificates and some of the HERA configuration that Puppet will apply and yeah this tool is Qtool and you need to keep this in mind in case you want to reinstall the cluster in the future because this certificates the HCD and the Kubernetes certificates are generated before the installation with the IPs and host names that you specify so if you want to reinstall the cluster you may need to also regenerate the certificates but as you can see the usage is quite simple you just need to specify some variables things like the operating system you would like to install Kubernetes on in our case Red Hat the version of Kubernetes and the container runtime your CNI of choice in our case Calico with the Tigger operator and its version and things like the HCD initial cluster HCD IP and the advertised address you run this command Jamel files its one per control plane node and one common one the common one, this Red Hat of Jamel is basically what we specified before as you can see is this HERA variables format and its the values we specified before and this file is truncated but as you can see it also generates the certificates and stores them as variables and just to finish with the installation part here is divided into two different parts so there is one first part done by Puppet and then there is a second part done by Flux with the help of Helm so the first part done by Puppet is also divided into two parts the first four steps are done by the module so the module makes sure that the certificates are created on the nodes it bootstraps an initial HCD cluster and installs Kubernetes in this case it uses QBADM under the hood and it does the init and join worker node or a control plane and finally installs and configures your CNI of choice as I said in our case Calico and then we wrote some custom Puppet code that basically does the Flux bootstrapping and enables OIDC authentication so up to this point this is where you have a minimum working Kubernetes cluster and this is when Flux picks up and starts installing and configuring things like Qverno plus all the cluster policies third manager Trident Kubernetes dashboard etc etc so how is it running Kubernetes clustering in production so I must say that for us the operators the overhead is quite minimum nowadays things like water changes are all managed with Flux so for us it's just a matter of changing a file and then pushing the changes and then Flux will take care of the rest creating new worker nodes as simple as creating a new VM put the VM in the right host group and then Puppet runs and that VM will be a new worker node part of the cluster in a matter of minutes for cluster upgrades we do them in place so that means that we upgrade every single element of the cluster and we do this of course with Puppet and some helper scripts that we wrote and it's as simple as changing one of those in that file that I showed you before pushing the changes then Puppet runs and things like QBDM QCTL and QBLET they get upgraded it's just slightly different if you are upgrading a control plane node or a worker node for control planes we also need to run that command so we use QBDM upgrade command and for the worker nodes we use one of those scripts that I mentioned before to drain the node first so we make sure there are no pods running there then we run Puppet so the upgrade actually happens and then when everything is fine we encode on the node so how is my team migration into Kubernetes and the future plans we have for the cluster so it's true that we have this cluster and we offer it to the organization but we run most of our services still nowadays in VMs on open stack but these are just some of the examples of applications that we are already running in Kubernetes so we have things like the give blood runners we have some dummy applications to verify the cluster status some slackbots open search and various permitted export and for the future plans we would like to have we would like to have some kind of cluster autoscaler so ideally VMs on open stack will be deleted or created depending on the cluster node and we would like to have service mesh probably with link rd we are interested in having an empty ls in between pods so the overall experience is very good we are very very happy with Kubernetes it's true that at the beginning you need to get used to looking at huge jammal files and some of the flex core concepts but it's definitely worth it one of the fears we used to have before moving into Kubernetes is that our users would miss the fantastic open shift web UI but in the end it was not the case and our developers they are happy interacting with the cluster via kubctl lens kai9s or even the kubernetes dashboard and if you are in a similar situation we were a couple of years ago and you are thinking bringing Kubernetes into your organization one thing that help us a lot or help our users a lot is to write some easy to follow how to guides with things that your users would need to do on a day to day basis things like how to connect to the cluster how to use the Kubernetes dashboard or a quick start with flex and I will continue the presentation with the users point of view so as I mentioned before I work in the team developing BlueBrain Nexus which is the main data management platform for BlueBrain project to organize all of the Neuroscience data and it has a backend for knowledge graph management and then a web interface for more user friendly access for the scientist and it's fully deployed in Kubernetes and there is a link to the application if you are interested in learning more about it so the BlueBrain Nexus infrastructure consists of several different components there is the delta backend and fusion frontend and then there is the storage part and other backend components like elastic search, blaze graph as the knowledge graph and post-crescule for data storage and then key cloak with its own post-crescule instance and we collect metrics with the permit use and then visualize them with Grafana and also visualize some of them with Kibana and then we use alert manager for alerting and sending alerts to Slack and all of this is deployed in Kubernetes so before in our old setup when we had to make a change we first had to use the OZ apply F command to apply the change to our OpenShift cluster and then there was the second step, the manual to commit your changes to Git so this was a quite error-prone process that we wanted to improve because somebody changed something in the cluster, forget or not have time to commit it to Git so this is something we wanted to improve and now we are still storing our configuration in Git but we are using Flux for a more GitOps approach so it will pick up the changes and then apply them in the Kubernetes cluster for us and some other new use cases that are now possible in the new Kubernetes setup is to have Git as an actual source of truth and then use Flux as for the automation part and we can also start using Helm charts and some flexible ingress configurations with NGINX and we are using Mozilla subs to encrypt secrets and storing in Git and for the daily operations again from a user point of view so we have two different Kubernetes cluster in the organization there is the dev cluster and then there is production cluster so in the dev cluster we have a replica of our dev environment where we can destroy it completely delete backups and see what happens or test some other new features or test Kubernetes upgrades and how it influences us and then there is the Kubernetes production cluster which is more rainbows and unicorns so it has a dev staging and production environment but it's more for the software developers to test their new features in dev and then for external users to test the features in staging and production is for production and as an example if you want to move into Kubernetes what you would need to do to port your applications to Kubernetes so first you would need to decide how you want to deploy your application some alternatives are to find a Helm chart or create a Helm chart if nobody has done it yet or you just create the necessary Kubernetes manifest like if you just need a deployment and service an ingress configuration if you have something very simple to deploy and then you need to go through the steps configuring and testing the Helm charts or the Kubernetes manifest so you deploy them in your dev cluster or dev environment and then keep resolving any errors that might appear because if you take a Helm chart it might not work like right away and the Helm chart quality also varies between one Helm chart to another so you just need to test that it works for exactly what you want to do and of course in storage ensure that the storage and the network configuration works and add some test data in your application and finally you can apply your working configuration to some other environments if you store your manifest and charge ingit it makes it easier and then you can use Helm chart and flux to to scale your applications to other environments and the migration process for a user if you are migrating into Kubernetes from somewhere else so first you need to learn about Kubernetes and it can be quite a lot to take in but don't worry it's doable so you can learn about Kubernetes and the overall architecture and then some research type for example Royal based access control and network policies and then you go into the testing and development phase which is more what I was presenting in the previous slide so finding or creating a Helm chart or the manifest to deploy your application and then actually deploying and testing your application and then deciding also how to use your available namespaces and Kubernetes cluster and then looking on how your organization is setting up Kubernetes and using it and then you need to plan and test and execute the data migration so do a backup and then migrate the existing data to a new environment and depending on how your application is set up it may require some downtime or not and of course keep a backup of the old data in case it's needed at some point after and then finally you switch to the new environment at some point so you verify that your application can connect properly, notify users and then finally start routing traffic to your new deployment so some things to keep in mind throughout the migration process so quite a lot of time was spent on prototyping and testing and also Kubernetes version has about one year of support so make sure that you also test operating Kubernetes and include that as part of your process and also consider and test existing use cases so talk to the user so this is something we did a lot my team with the developers talking to Daniel's team core services who are the operators and making sure that everything works and then allocate some time to train your teams and ensure that it's easy to use rollback changes if needed so if you use something like githops you can just revert the commit if you committed something that was wrong and also something like network policies and rollbase access control and the ingress configuration might need some adjustment to each other so it's good to understand how this component works in Kubernetes and I'll link to the documentation here if you want to read more about it and also beware of internal networks in a company, if you have a company proxy you may need to set this HTTP proxy or HTTPS proxy or the lowercase version this environment variable in your virtual machines or where you're running Kubernetes and maybe also in your applications or help charts so this depends on a strange connection issue like you're not able to clone a git repository to your pod for example then this might be something to look into and help chart quality varies a lot so again it's a matter of really testing and trying it out and see if it works for you and of course set up monitoring and alerting and dashboard to keep an eye on your applications to make sure they work so as a summary start port typing as soon as possible because that's the only way you can start figuring out what problems you will hit and how to fix them and set up a collaboration between the users and operations team so that you can understand what use cases you have and how to solve them and also evaluate the decisions against the needs of your team and organization because Kubernetes has a quite big ecosystem so you may start to make a lot of choices upfront like Kubernetes clusters or namespaces for the users or what container network interface should you have or what permissions do the users have in the cluster and usually a good thing is to use a standard alternative unless you have a reason not to for example if you want a specific feature from your networking plugin or if you do some benchmarking that's a good reason to not use standard alternative and ask for help in the Kubernetes and CNCF Slack instances because there is thousands of people there who are happy to help you with any question and benefit from all the automation possibilities that the Kubernetes ecosystem is offering and finally become a Kubernetes and present what you did at KubeCon and CloudNativeCon so yeah that's it for us if you're interested in this we're actually hiring and if you have any questions come talk to us thank you