 Hello everyone. So today me and my colleague Francisco, so we're going to talk about managing a fleet of Drupal sites at CERN. So my colleague is Francisco and I'm Rajula. So we work as infrastructure engineers at CERN. So we manage all the Drupal infrastructure, all the clusters and manage upgrades, all the other operations. So before I start, like, how many of you know what CERN is? Okay, quite a good number. So we are the world's largest physics research laboratory. So we're not like, we're a non-profit. So we work with the largest physics equipments and stuff. And just to give an overview. So this is our, this is called the large hydrant collider. This is the biggest ring. This is a, yeah. So we have a couple of experiments on this ring. So this is one of them. This is called Atlas. Just to give a new slight view of this, the weight of this experiment is 7,000 tons, which is equal to the Eiffel Tower weight of the Eiffel Tower. So given all this, yeah. So what are we going to talk about today? We're going to start with how we use Drupal at CERN, give a bit of overview. And then we're going to have a demo. It's going to be more of a half of the presentation, half the demo, and then we'll have a Q&A. So given all the physics, where does Drupal fit in? So every public website that runs is on Drupal. So we have almost 1000 plus websites. Of those, we have 600 plus official websites and 400 are on clones and test websites. We have a community of more than 100 plus creators and site builders that we support. And most all of this is hosted on-premise in our data center. We run it, we create the machines using OpenStack and then we run OpenShift on top of it and then we manage all of it. So just to give an idea of the scale at which Drupal runs at CERN. So these are the recent achievements. These are like blog posts that were posted on home.CERN, which is the official CERN homepage. So we had the start of the RUN3, which is the experiment started after the shutdown. And then one of the experiments also discovered three new particles. So these were posted in a span of three to four days. And all of this generated a traffic of almost one million unique visitors in less than a week. And we were able to handle all of this without any performance degradation. This was the record for us this year. So talking about a fleet of Drupal sites, like I said, we have 1000 and we have a mix of things. We have the simplest of simplest websites and to the complex of all. So there are also something similar among these sites. So let's say each of our website is like a vehicle. So each of it has its own features and stuff. They have their own modules, own themes, but there's something common among all of them. So we try to jot down the common stuff and then we call it the CERN Drupal distribution. So like an engine, which is common among all the cars. So we bring down all the common libraries, common modules, and then we package it into our own distribution, which is on top of the official Drupal version. So we have three parts on it. One is composite adjacent. That is the Docker images that we build to deploy. And then we have CI and testing for each of this version that we build. So going through each of this. So this is our repo. And now composite JSON is quite simple. So we just take the official Drupal one. On top of it, we add all our CERN. We have our own custom modules and themes that we add on it. And then all the other libraries that are used by most of these sites. And then images. We build a couple of images. So we try to copy all this composite, the Drupal code that we built along with Nginx and PHP in the same image. And we use the single image to deploy as a PHP server and also an Nginx server. So all of this lies in the same repo where we do the build. Yeah. Coming to CI and testing. So every time, every commit that's pushed to this repo, we first build. So we have two builds here. One is a composite builder and the other side builder. To make build faster, whenever we change something in the composite adjacent, only then we trigger a composite builder. For other smaller stuff that doesn't relate to composite, we just do side builder. So these two build an image that has Drupal code with a specific version, PHP and Nginx. And then we also test it. So how do we test it? We create a test project on the production cluster. So every time the pipeline runs, a new website is created on the production. And there are a couple of tests that we run. See if the website is working, if SSO is working. And then if it's working, then we use this version image and then we update the website. So yeah. So what if the module is not installed? Let's say the user wants a specific module that he wants. There are two options. One is they would have to install it by themselves. The other is if it's a module that can be used by every other website, we just add it to the common distribution that we just showed. The second part would be demonstrated in the demo. So I'll go to the first one. How do we do the first one? So for the first one, we use something called source to image. This is an OpenShift workflow. We provide a Git repo with a Composer.json. And then we just take this and then we build a Docker image on the fly with this Composer.json, merging it into our original Composer.json, and then run and get a new Composer.log with both the modules in it. Yeah. So it would be a Drupal distribution Composer and then the custom one. So we use the Composer merge plugin, and then we merge the new module into it. And then we this thing runs on the fly, builds a new image, and this image would be used to deploy. So this is an example of how a custom Composer.json would look like. There's just one line here, line number 12, where we just add the new module that we new module that we want to install in it. Yeah, always that infrastructure. Now my colleague is going to talk about Intra and then the demo. Okay, so my colleague just showed how we build images, how we develop and add new modules to our images. But the question now is, how do we deploy all this? So our deployment, like my colleague said, okay, no, you can sit. It's fine. We run OpenShift, which is basically Kubernetes with some extra features like security and multi-tenancy. And on our Kubernetes cluster, we have two major components that make everything run really smooth. The first is an operator. It's called, it's known as operator in the Kubernetes world. It's a Kubernetes operator that works with the Kubernetes API. And we have a second thing, which is called the Drupal site resource. So this is a custom resource that works with the Kubernetes API and basically allows our operator to understand what we are requesting and create all the necessary stuff in order to make our Drupal website run. So here we have basically this example of the structure. The Drupal site is a YAML file. I'm going to show an example of this Drupal YAML site in a little bit. And after we have this YAML site, we push it to our cluster. And then our operator will interpret this and will do the creation of a new deployment, of a new storage, of the grown jobs if necessary, ingress, secrets, all the necessary stuff. So here's the next example. So imagine I want to create and I'm going to do this in a little bit. I want to create a new website and I want the URL to be Drupalcon. And I want a specific version. And then I have a few configuration values. All I have to do is give this to our Kubernetes cluster. And the Kubernetes cluster will be able to create a new deployment running nginx, php, and some extra stuff. So without further ado, let's go to a demo. Hopefully this is readable. How readable is this? Readable? Okay, good. Okay, so like I said before, let's go here to my temp folder, make here Drupalcon. And I'm going to call it Drupal site, but YAML, right? And here I'm going to basically add on. I don't know by heart a Drupal site resource, but I'm going to get one actually. So here I have one. I'm going to use this as an example. As you can see, this has a lot of stuff. And like I showed here, don't be too scared. I'm going to explain all of this in a bit. Okay, so our name of the website is going to be called Drupalcon. And this is how Kubernetes will understand our website. We don't specify the namespace. And on the configuration, we can say that we just need, well, one gigabyte should be sufficient. And then we have a few extra configurations that allow the cluster and the Kubernetes to understand how many resources this website will need. So for our use case, we can just use a standard database and the standard quality of service. And we don't need any of this. And for the URL, we can say Drupalcon 2022. Sounds good. And this is the specification of the version. So whenever in our GitLab repository, we want to deploy a new version of Drupal, say Drupal.9.5, all we have to do is create a new branch, call it v9.5, and basically the cluster will be able to fetch it from the container registry and make it run. So this is our specification. And let's say we want to create Drupalcon. Do I have any project with the name Drupalcon? Or do I need to create one? Okay, I have one here. So here, this is a really cool Kubernetes specific, but basically these projects are what allows multiple websites to live alongside in the same cluster with full separation. So within a project, you are basically isolated from all the other projects. So this Kubernetes, this website will live alone and you will not be able to interact with any other project. So like I said here, let's create the website. So website has been created and we can do a few things here. So I'm going to, yes, sorry. So what I just did is the classic operation. So if you ever interacted with Kubernetes before, you just do Ctl, apply in your resource name. And basically what I did here was I requested the cluster to create this resource, which like I said is a resource that the cluster understands because we have created this resource into the cluster and we explain the configuration that the operator reads. So like I said, hopefully we still see some action, otherwise we'll just be looking at all logs. But if we check to the pods, we should see now that our operator has read this configuration here and it will create a bunch of stuff. So we can see here that we already have a pod running. Actually it should be more than one. Okay, so we actually have two pods running here. So this first pod running is let's say a startup and what it's doing is basically it detected that this website is a new website. So what it's going to do and we can actually see what it's doing by checking its logs. What we can see that it's doing is basically enabling themes, certain themes. It's enabling a bunch of models, all the stuff that came from the Git repository. It's basically enabling it and installing it. And well, in a little bit it should have been done with the job. So it's now completed. We have now the serving pod where actually Drupal is actually running. And we actually should have something called the route, which is something similar to Ingress, which will enable us to access the website. And this should be available to all of you in a bit, hopefully if the NS is not too slow. So this is a URL and if I go here and open Firefox URL here. Voila. Okay, so the website has been created. And as we can see, well, if any of you tried to access the website before, you would have seen that it was not created at that point. We can see that it has already all the look and feel of a classic CERN website. This is basically the vanilla thing. You see on top the CERN bar. You see on bottom the contact, etc. So basically this is what we provide out of the box to any experiment at CERN. And even more, we configure some of these modules out of the box. So if I log in, probably I have cookies. So I'll automatically be logged in and be shown as an admin. Well, that's it. But if I were to copy this and open in private, you'll see that I should get redirected into CERN's SSO. So this is the CERN's login system. And with this, we can see here on the extent that we have a bunch of modules that we can enable if you can enable out of the box. And that's it. So for our infrastructure, let's say to create a new website, all we have to do like I showed was to provide a specification of the Drupal site. The operator reads it. The operator creates a deployment. The operator creates a site URL. Actually, I mean, if you want to test it out, we can, for example, come here and we say we want to edit our Drupal site, which was called DrupalCon. And we can go here into the specification site URL. And we can call it something. We can give it a second URL and you want to give a suggestion. So we see that basically the operator will take care of everything for me. No, no suggestions. So no suggestions. Web.CERN.CH. So if I now do a watch on the routes, the operator again will read this description of what I want from the Drupal site. It will detect, oh, he wants a second site URL. I'm going to create the route. I'm going to publish it and I'm going to make it available for everyone. So without further ado, I can copy again, go to the browser, open the URL, and well, it's the same site. It's empty, but it's the same site. Yeah, so like my friend showed you before and I'm showing again, this is what we have with our out-of-the-box package. So what if I want to add a new module into our package? So we had a discussion with our users and we see that the community could really use another module. To do that, we just go to our GitLab repository, which hopefully I have it here. We should have here, like my friend showed you, the composer.json. And now I would like to request a quick help from my audience and get a suggestion of a new module to add quickly to our Drupal distribution. Anyone wants to give a module a suggestion? No? Workspaces, okay. Let me just confirm that I don't have workspaces on composer. So workspaces, just like this? No, I don't have workspaces, so it's a good example. Okay, so I'm going to run Podman. So basically, when we want to add to our certain Drupal distribution anything, I don't rely on my machine. Actually, this machine has been freshly installed, so I'm sorry for all the hiccups. We rely on using Docker or Composer and we run our own machine locally so we can actually build everything consistently. So certain Drupal distribution, hopefully I have it in the chat here. Okay, so Podman, IT, okay, so hopefully this will run, but I don't have Docker. And well, so what I'm doing here quickly is just I'm wanting my folder, work folder, where I have the repository into the image and I have this image here, which hopefully still exists. Okay, so I'm inside the image. This doesn't look good. Yep. Okay, so actually, maybe I should do it manually because I can just edit manually, I guess, because my machine is not working properly. So does anyone know what is the latest version? Actually, we can put any version, right? So workspaces and we can define, sorry, so sorry, could you repeat that? WSE? WSE, like this. Well, I need to give, sorry, and the version we, thanks. And which version should we require to be in? We can just say three. I don't know which version this runs on. Okay, I can put 1.0. Okay, so this is enough. Sorry for the e-cup and hopefully you're still with me. So, well, in the Git repository, we see the difference here of Composer. So we have the new module. We can add this and call it new feature. Now I'm going to create a new branch and call it Drupalcon. Git checkout, sorry, checkout B, Drupalcon, because it doesn't exist. Okay, so now I'm a feature branch and I'm going to push these changes with the new Composer. And well, GitLab will say that this is possibly a merge. So I'm going to go quickly to GitLab where we host all this Drupal distribution. So we can create the merge request. I don't know if you noticed already on the pipeline, there's something already running. So like my colleague has said, every time we create a new change, the pipeline is run, it is executed. A new image is created. This image will be a feature image and we can then push this feature image to our production cluster, dev cluster, or even in our machine to see if everything works as we expect. So we can say new feature workspaces. I'm not going to push too much here, just doing this. And I create a merge request. And well, before merging it, I want to be sure that it doesn't destroy what I already have. So after creating the image, we still have a few extra steps where we would see the same website being created into a specific place. The same way that I just created our website, it will create automatically. It will check a few things. And if everything is all right, then it will say, okay, the test has passed. This means that it's deployable not only on staging or dev, but it will also run on production correctly. So our pipeline is currently still building the image in a bit a little bit. Hopefully it will as finished. Like I said here, I have a Drupal site. I'm going to edit this Drupal site. Also, I don't know how familiar you are with Kubernetes overall, but in the Kubernetes resources usually have two important fields in any resource, not only the custom ones, which are the specification and the status. So this specification is where I can change things and where basically I define how I want the world to be in. And the status will say basically how it's currently in. So we can see here that it's already using that image for debugging purpose. We can always export further. The image has still not been done. So in a bit, I will just change this version here. And instead of being a version V9.4, it will be the branch name. And then we will use the commit from the release. So the branch name was Drupal corn, right? I mean, I can just wait for the pipeline to finish because it should be finalizing hopefully in a minute. And just to export it later. So basically, if this was to be merged and then the new version release has been finalized, then I can just go to all websites in all production versions and just update it with the latest one. So if I ever want to just update, for example, Drupal core to a new version, all I have to do is make these changes on GitLab, run the CI, the image is run. I know that the image will not break anything. Then I just go to all the cluster and say, now this is the new release. And basically, the cluster will take care of everything for me. So as you can see here, the image has been pushed. So the image is this feature one. When it fits a production one, it will take it as a release. I say here, this is the version that I want. So the version name is always the branch name. And the release spec will basically be the tag. So I changed the Drupal site. And again, my operator has noticed that the specification that I gave is different and that it has to do some things. So what it's doing now is it's ramping up. I can do actually a watch. You see that it's trying to ramp up a new pod. This new pod is with a new specification, which is the image that we just created. It will try, really try, but maybe not succeed to actually have zero downtime. So if I get the route here, Drupalcon, if I do a curl and I try to get the status code, HTTP, it's 200, 200. Well, it's already running the new one. So we saw that a new pod got created. So this one is brand new. And this one should theoretically be running the new image. So we can actually test that by going to the website. So if we go again to the website and refresh, sessions should remind the same because Drupal stores the session and it's not within the pod. And now we work for workspaces, right? Workspaces. So there it is. So now, just with this simple change, workspaces. Okay. So which one is it? Is it any? Is it this one? Okay. Sorry. Yeah. So basically it's this one and became available to anyone. So basically now, if you wanted to use this new feature that we just deployed to everyone, you just have to check it and install it. And that's it. I don't know if, by the way, if you have any questions at any point, feel free to interject. But imagine that if we want to go back, I can easily go back into the old version. And we see that if we put the old version, which is this release with this release spec, v9.4-1. And with the release, sorry, with this release spec. And we watch again. We see that the new pod is again rolling because like I said before, the operator again detected, oh, there's a change in the image. I have to update again the deployment. The other one is already terminating. I actually saw a 1503. So you can actually see that it was barely out of and available. It's almost zero seconds. This is one thing that we have to improve. But basically with this, I have just deployed the same website with minimal downtime. And again, with the new version, this is basically a change in the module, but it could be a change in anything. And now I should not be able to find workspaces. Workspaces. Well, now I just find this. Yes. Are these all models, for example, or you always rely on fetching them from there? So basically, you're questioning if we cache our images? No, no. When you're building something new library that has been used already before in another place, but is being added. So if there's a library that you want to add? Yeah. Then, or a model, a contributed model, anything. Do you always rely on the original source? Yes. So basically, each image once created, it's immutable. So it will remain with all the packaging that has been created on. But if at some point you want to update to a module, like you said, from upstream, we push it to the git repository. And only then you are basically fetching from upstream. So we always rely on upstream for fetching the stuff. But once we create the images and we deploy them, then we basically hold these images and we can always redeploy them. Well, basically, the goal of the demo was this, was basically to show that I can easily change our distribution into adding a new theme, a new module, anything based on the community request. And the second point which I find interesting to show is that this is basically Kubernetes native. So none of this is particularly specific, let's say, to CERN. This could be deployed in any Kubernetes cluster. Obviously, there's a few extra features that were added for the CERN specific requirements, but overall the idea is the same. And now I suppose we can go to the questions, which should be on. So feel free to throw questions. So like my colleagues showed, I don't think we have time in terms of the demo to do this. But if you were to, and this I can show you again in the terminal. So the question was what about custom modules? So if you have, you mean personal custom modules, modules that you can create. Okay. So all of this, like as persistent storage, so our pods are, so if I get the pod here, this pod here that is running our website can be, we can get into the pod. Let's go to the PHP FPM container. Did I lose? Okay, I lost the character there. Okay. So once I'm here on one of our containers, this is the one running PHP FPM. We can see that we have something which is the Drupal data, which is actually the one on top. So basically, we can see that our persistent storage is currently empty. At any point, you can go to the Drupal data and here on under modules folder, you can add any custom module. And basically here you just dump the folder like any other module. And there's a link between this and the PHP code. So actually, we can even try that we can actually go to upstream. What will any module that you want to dump directly there? But basically what I would do is you would tell me the link and I would download the zip into this folder here. I would do a Drush Cache Reset basically, and then it would be available to enable to the website. Yes. So we also have web doc so we can also mount the file system of the website into your local machine or have it in the browser for convenience basically. Any other questions? What was the business need that justified this entire infrastructure? How many websites are we talking about? Okay, so okay, you missed the presentation because it's actually one of our first points. So this is the simple graph that was extracted a few days ago. And basically at CERN, CERN has a lot of experiments happening, even though we have like the large idran collider as our main point. There's a lot of small experiments, a lot of different things that are being tested. And all these experiments might be interested in having their own website that they have a full control over to expose to their communities, as well as maybe you can or want to create an internal website within the CERN community to show a few things. Basically, you can rely on the CMS Drupal to create your own websites and have it mostly maintained by us. So if you just use like the repository that I showed you, you don't have to worry about anything. Drupal gets automatically updated to the new core version. The modules also get automatically updated. You as a user, don't worry about anything. You're just a content management provider. Yeah, you said that you're merging the comporter JSON, if you have a custom one. Where I was thinking normally what I would do is to make this share this version of dependency instead of... Okay, there's a reason behind this. So this is the approach that we support. So at CERN, this is an engineering perspective, any custom module as its costs. And in the CERN community, the focus is on physics. We do IT for supporting their experiments, both on providing websites for public hosting, but also for computing needs. So we try as much as possible to demotivate users from creating custom custom modules that you put into a folder and you forget about it. And in five years, they have to be maintained. But if you have a composer, usually it's a package manager. So it will automatically update or at least it will be able to update easier than if you just dump a folder with static content. I'm not sure if that answers the question. Maybe. Yeah, because the way I would think about it would be that if each website has its own repository, but I think maybe it doesn't. No, it doesn't. By default, you just use our repository. So most of maybe some websites have their own repository. Yes. And then they have this composer JSON that is being merged with the other one. But it's not... But I would think if I have my own composer JSON, then I would just add this other thing as a dependency. And then I don't... There's no... You don't need to worry. So basically what you do is on your GitLab repository, you create this small composer with all the modules that you need. And basically you create a webbook, basically. So basically there's a trigger. So every time the GitLab changes, it will tell the cluster, look, this repository has changed and I have a new module. And the operator will fetch that repo. It will merge it with our composer. And this guarantees that you also get the modules that we provide, the themes that we provide. Because like I said, if you look at the website... Sorry, on site five. On this website, it's also of, let's say, your interest as a user to not worry a lot about themes and looks. So like I said, this is one of the themes that we provide out of the box. And it's just simple because you already have the CERN look and feel, and all you have to do is have extra stuff. So it's usually you don't want your full composer. How do you handle configuration? Is the configuration shared across outsides? No, each instance has its own configuration. It's on database. They live completely isolated. Conductions and so on. Do you have some kind of starter kit to get the basic context or something? So like I showed you, this is basically the superstar kit. So like I showed you in YAML, we have a portal that is more fancy. And you just have... I mean, I can show it if you want to. So basically here, this is our web services portal. And if we at some point want to create a Drupal site, which we just come here and just say create a Drupal site, we say if it's official, so it's for the organization, if it's personal, because you might want to have your personal website where you show your paper submissions, your presentation, et cetera, or a test website where you're testing something. And then you just give it a site description, a site name, the type of profile. So the CERN or Easy Start, this is the two basic themes that we provide. And once you click on this in two minutes, you have the website up and running very similar to this one. And this has content types and fields already or not? This doesn't have a lot of content. This has mostly just integration. So if you go here to the extensions, we'll see that CERN, we have quite a few modules that are here and you can enable. There's actually a bunch of them. We also have something called Indico where we... Sorry, Indico, where we polarize our events internally. And basically, this is integration with your website. This comes out of the box. But I'm not sure if I'm answering your question. You can say no if it's... So I assume this events module then has some conflict tools for the content type or something. Sorry? The events module might then have a content type configuration. This is a better question. Actually, I have the web team with me. So basically, I'm infrastructure. To be honest, I'm not really a Drupal... No, wait, hold on. I'm not the Drupal expert whatsoever. I'm an infrastructure engineer and I try as much as possible to automate Drupal. But once we reach this point of actually handling... Sorry, managing modules, etc., actually, I have my two colleagues here that they're from the web team. They're in the first row and they probably can reply to that question better than me. We can also talk later if you want to. Or if you want to, yeah, you can have a discussion later. I don't know. Feel free if you want to reply. Okay, I guess you can discuss later unless anyone else is interested, but they can also ask them. Any other questions? Curiosities. Project browser initiative gonna affect your workflows? Well, we haven't given it too much of a thought. So at least for now, we accepted the fact that we have to basically update the core, update everything. So at least until further notice, we will continue with this workflow where basically we just have to update the composer and the operator does everything for you on updating. If at some point they make improvements on the life cycle, we just integrated to our use case simply without much of a fuss. So is this updating and offering new features the main things that this test that you guys do? It's a bunch of things. So this, as you said, automatics is one very good improvement from our infrastructure. But the second is the deployment itself. So this is basically a deployment that it's easily easy to duplicate through tests. So basically, if at some point you want to test something out, creating a clone with this infrastructure is automatic. I can create a clone within two, three clicks again. And basically Kubernetes will create a separate instance that is fully integrated with CERN ecosystem. So basically, it automatically integrated with our single sign on, but also it will also create for you a database. So you don't have to worry on anything about the Drupal and everything is hosted in-house, all self-hosted. Okay. Any other questions? Hi. In your CI CD pipeline, you mentioned you had automated tests. I'm curious what kind of tests you have, especially around security? That's a good question. It's a very good question. So, well, we first and foremost, we rely a lot on Kubernetes itself on OpenShift and all its features. So the pipeline is finalized, as you can see, and the tests have passed. We actually do two tests. So I'm not sure how visible it is. I'm sorry. The tests are really simple. So what the tests actually do is they create a new website out of the box with the version you just created. And the tests that are made are basically you can run the website and the website as SSO integration. And the other one clones an already existing site, which already has a few modules and basically updates them and see that they run. There's no security tests. So on the application layer, if Drupal is not really secure, we could be exposed. Our advantage here is that since we don't use things like multi-sites, even if a website gets exposed, it's minimal because you cannot get access to any other website or anything else. Basically, you just get access to the website itself and to what that means could have access to. One thing that we try, and it's the advantage of the way that we have with automatic updates is we try to be as possible as soon as possible up to date. So whenever Drupal itself publishes a new patch security, etc., we basically are able to put it to production if we want within the day. We usually don't do it within the day because we want to guarantee as much as possible that nothing will break. If it's usually just a patch upgrade, we can do it within the day. It's a matter of just like I did now, doing a Composer update with the new Drupal version, run the CI. Once the version is released on the CI, I just update all websites in the YAML file that I initially showed with the latest release, and that's it. Within minutes, everyone is up to date again. Any question? Okay, so if there's no more questions, thank you so much for attending.