 Okay, I think we're ready to start. Welcome everybody to the CVMFS tutorial at the Easy Build User Meeting. We'll get started today with covering a couple of aspects to get you started. And also the first part of the CVMFS introduction or getting started itself. We have a couple of slides here to guide you through the tutorial, but the main part is actually gonna be in the CVMFS tutorial website itself. So this one, we will basically cover these topics one by one and switch a bit between the slides and the contents we have prepared here. The slides have not been shared with you yet. We'll do that right after this session. So the sessions we have for this week actually started yesterday already with the introductory presentation by Jakob. The recording of that is available on YouTube already. So you can find it at this link. It's also mentioned in the documentation in the tutorial here. So if you look at the introduction parts, the video, the recording is embedded here in the documentation. Today we'll get started with getting you set up on the Azure resources that are available for this tutorial, which are sponsored by Microsoft. We'll give a very quick introduction to CVMFS. We don't wanna redo Jakob's talk, of course, but we'll highlight the key concepts and the key terminology that you will be seeing throughout the tutorial. And then Bob will explain and demo the setup of the stratum zero and accessing the, your first CVMFS repository from a client system. And then we'll wrap up with a small exercise that you can try things yourself. During this session, we will only be explaining and demoing things we don't expect you to follow along while we do that. So we ask that you take the time to do that after the session in your own time. And we will be available both at the end of the session right now and also through the EasyBuild Slack channel if you have any questions. So for this, please make sure you join, let me fix the typo here. Make sure you join the CVMFS tutorial channel we have in the EasyBuild Slack. Or you can also send us an email at EUMatList.de for any problems or questions related to this tutorial. Also after this week, so if you're watching this in the recording, feel free to jump into the channel on the EasyBuild Slack here and we're happy to help you out. If the need arises this week, we can set up an additional Zoom call to help more hands on or to be a bit more interactive in terms of handling questions. We'll see if the need arises and hopefully that after each live session we'll also have some time for live Q&A. So first of all, we have resources available in Microsoft Azure in the clouds for this tutorial. So you get a couple of very small VMs there and they only have two cores each but that's enough for this CVMFS tutorial. You got credentials for this in an email. So you have a CVMFS user name that ends with three digits and a random password. With this, you can log into the cycle cloud environment that has been set up. So if you go to cycle cloud, you'll see this view. And here you can use the account that was provided for you and the password to log in. The first time you do this, you will get a request to set a proper password. So the password you got is a random one-time password. In this case, that's already done for this account. So it doesn't pop up here. So make sure you pick a password, of course that you will remember throughout the week. So we don't have to do any password resets. This is the view you should get once you are logged in. Then the second thing to do, let me also show this in the tutorial itself. This is all outlined here in part zero, the Azure cloud resources. So logging into cycle cloud, what your account's doing, the password reset. And then the next very important step that you should do before starting the VMs is adding your SSH public key. So for this, you go here to the top right, you go to my profile. Here you do edit profile. So there's no SSH public key assigned yet to this account. You go to edit profile. And then in this box here, you copy paste the SSH public key, which I have here. I should have here, yes, for this account. Just copy paste that. So one very important thing here, this has to be an SSH RSA key. So other formats like the ED25 something, the format will not work. It has to be an RSA key, so please be careful. The system will allow you to add another key, but when you try starting the VMs, it won't actually work. So make sure you use an RSA key here. You click save, that's saved, and that's all you need to do in terms of editing the profile. Then you can go back to the overview of your cluster. So that was step one, basically, after logging in. And then once the SSH public key is in place, you can start the VMs. So here there's a big start button at the top. This will start all the nodes in the VM cluster. So we have four nodes provided for each of the different parts of the CVFS setup that we will be playing with. The central status zero server, the mirror server, the stratum one, and a proxy that acts as a cache. And then the client where, which we'll be using for accessing the CVFS repository. Then you can go ahead and start the cluster and you will see all these nodes firing up. So they will start, it'll take a couple of minutes before they start. Now one thing here is that we were notified yesterday that we only have 100 public IPs shared in between everybody. So we have I think 31 or 32 accounts prepared. If everybody starts all four VMs, we will run out of IPs. That should be fixed by now, but we're not sure. So what I will do is the nodes that I will not be using. This view is a bit annoying. So the proxy will not be using yet. So I will actually shut this one down again. And I'll do the same for stratum one, which we'll only use tomorrow. So here also shut down stratum one. So for now you only need stratum zero and the client. So let these boot up. So just so we're safe in terms of public IPs that we don't run into any problems there. So we'll take a little bit eventually these two will be available. And you can already see if you pick for example, stratum zero, which is the first one you will be logging into. Here below it should be showing the IP for this host. I'm not sure why it now says proxy, but okay. So yeah, when you select something here, you should be seeing the proper IP below. So make sure you use the right IP when you're logging into each of these nodes. One important detail here, the IP addresses are not fixed. So if you restart a VM, you will get a different IP address. For the client, that's not a big issue. You just have to SSH into a different IP address. For the other parts, that's a bit more of a pain because you will be using these IP addresses in some configuration files. So whenever you start a VM and you start using it, we start using the IP and configuration, just leave the VM running until you're done with the whole tutorial. So that's not a big issue. These VMs are quite small. And they are sponsored by Microsoft. So we are not paying for them. Just at the end, so either Thursday afternoon or Friday, make sure you shut down the entire cluster. And if you don't, we will Friday evening, make sure that we get rid of all these resources. Then the introduction itself. So I'll give a bit of a graphical overview here. Again, this is also covered in the tutorial itself in the introduction part. And really the best introduction is Jakob's presentation that he gave yesterday at the EasyBuild user meeting. So if you have the time, please, and if you haven't seen it yet, please take the time to give it a view. The most important parts, aspects of CVMFS are covered in the introduction part of the tutorial here. So just very briefly, CVMFS is a read-only file system over HTTP. So it's read-only for clients who access it, which you can use on Linux and Mac OS, also in Windows, in the Windows subsystem for Linux environment. It's sort of similar to NFS and AFS, the network file systems, but there are very important differences. Any file system that is hosted in CVMFS will be mounted under slash CVMFS. And you will see this popping up in the tutorial. And the CVMFS servers, so the stratum zero, stratum one are basically web servers to the outside world. So all traffic is done over HTTP, which makes it very easy to pass through firewalls and things like this. The clients get a read-only view of the file system, so they cannot make any changes to make changes to the file system. You have to go through the stratum zero or through a publisher notes, and we will explain that in the tutorial. The primary intention of CVMFS is a software distribution system. So it's really focused on distributing software, so not just any files, not huge data files that you wanna share with the world, but it's really focused on sharing software and to be more specific sharing software installations. So it's not about sharing packages like RPMs or container images or things like this, but really the installations themselves, so the individual files that form the installations, that's an important distinction. So there's CVMFS is tuned in various ways towards that, so it's very aware of how software is typically accessed, so involving lots of small files and involving directories or several files in the same directory at once, so it knows about these access patterns and it's tuned in various ways to cover that well. There's also very aggressive caching as will hopefully become clear throughout the tutorial and CVMFS makes sure that the storage that's actually needed for these installations is used as optimally as possible, so it duplicates files that are included in the repository in multiple different locations, but with the same file contents and it also heavily does compression. So the amount of storage you need to host even several hundreds of gigabytes of software installations is pretty limited. CVMFS was designed to be scalable and reliable and there's deployments out there that serve hundreds of millions of files and tens of thousands of clients and they are certainly not the exception. So it was built for the LHC project at CERN and making sure that they can distribute all their software around the world, which is quite a lot of software as Jacob covered yesterday which also has a heavy or a very frequent refresh rate in terms of updating the software. And also the way the CVMFS network of servers is built is it's very robust against network disconnects and hardware failures and things like this and it's tuned to give optimal performance in terms of access to the software installations. So the terminology part here I will actually cover in the slides because that's a bit more visual in here. So this gives an overview of the network of CVMFS servers and the clients that connect to the CVMFS repository that is provided by these servers. So I'll step through these one by one starting with the core of the network so the actual server part and taking out the client for a bit. So the core server, the most important one I would say is a distractum zero. So that's the central server in the CVMFS network. There's only one, so there's only a single central server and this is also where the CVMFS repositories are created and hosted. So whenever you want to add software or make changes to the software in a CVMFS repository this is always done on stratum zero either directly on stratum zero or through some other server that gets the right access through the stratum zero. The actual changes are actually made on the stratum zero. And of course this is a critical component also from a security point of view in the CVMFS network. So usually the access to stratum zero is very restricted, only certain accounts, only certain IP addresses, things like this can actually get access to stratum zero. It depends on how you configure things in terms of firewall but that's very typical to protect the access to this central server. Then the stratum one server in the network there's actually usually several stratum one servers. These can be viewed as mirror servers. So they have a full copy of the repository that is hosted on stratum zero and they're also called replica servers in the CVMFS terminology. Here there's only a read only access to the file system. So you can see all the files that are in the repository but you cannot make any changes here on the stratum one. There's typically multiple stratum ones which are geographically distributed. So they are intended to take away the load from stratum zero. So stratum zero doesn't have to serve the clients directly but the load is spread over multiple stratum one servers. So the more you have the better even though there's limits in terms of how efficient things get. So a rule of thumb is like five or to 10 stratum ones is sort of a maximum range. Yeah, and they are intended to reduce the load on stratum zero and improve the reliability of the network. So even when you lose a stratum one, that's not a problem CVMFS will just go to the next one. And even if the connectivity to stratum zero dies the stratum one servers have a full copy of the repository and they can keep surfing the clients. So even if stratum zero is unavailable for whatever reason the only thing you lose is that you're briefly not available to make any changes or not able to make any changes to the repositories but you still have the read only full copy on each of the stratum ones. And this is also where the clients typically connect or get access to the CVMFS repository through a stratum one either directly but usually through a proxy. So that's the next component in the setup. The squid proxies are reverse proxy for the stratum one servers. So they have only a partial copy of the files that are in the CVMFS repositories based on what has been accessed recently. So they are used as a cache for the stratum one servers. So again, to upload the stratum one servers and upload the whole network. They can also be used for load balancing. So you can have multiple proxies connected to a single stratum one to spread the load a bit and you can even do load balancing and automatically start additional proxies if the load increases and things like this. And this is also typically how clients get access so they connect to a proxy. They ask for the necessary files automatically through CVMFS. And whenever the proxy doesn't have the files yet it pulls them in from stratum one first into its cache and then serves them to the client. And then looking at the client part here. So a client from a CVMFS point of view is any user who accesses the CVMFS repository either directly from his or her personal workstation to through a squid proxy or from a cloud environment where they want to use the software or from a cluster where they want to use the software. So also an HPC cluster or any cloud VMs are seen as clients in the CVMFS network. And clients can also use the local file system cache which is part of CVMFS to again get better performance in terms of accessing the software. So any software you have accessed recently will be available really quickly. If you try to use something new it will have to be pulled into the local cache first typically before you can actually use those files. But that's usually pretty well in terms of latency and delay. Okay, that's what I had in terms of introduction. So all of this is well covered here in the tutorial itself as well and especially in the Jacobs presentation. Hopefully this was enough to give you an idea of what's going on. And I'll now pass the word to Bob who will take over for setting up the stratum zero and the client. So I'll stop my share here and then Bob share. Yes, so good morning. I'm Bob and I'm going to tell you. I'm still muted, Bob. I could hear you just fine. Can you hear me now? Yes. Yeah. Okay, great. So good morning. My name is Bob and I'm going to tell you about setting up the stratum zero server for CVMFS and setting up the client. So basically the stratum zero is just some kind of ordinary HTTP server serving all the files of the repository in a compressed and de-deplicated way over HTTP. So I will show you how to do that, how to set up the initial repository, how to easily add a first file to your repository and then set up the actual client that connects to the stratum zero server so that you can actually browse the repository. One big warning here. So we're going to directly connect from the client to a stratum zero which is something you should never do in production. But as a first step, we will do that. That does work in CVMFS, but it's really not recommended to do it because of safety reasons, scalability reasons, et cetera. But since we don't have a stratum one and proxies yet, which we will cover in later sections, that's how we're going to do it now to just demonstrate how it works to set up a client. So I will go back to the Azure Cloud instance because I first want to connect to the systems in my terminal over here. So I need both the client and the stratum zero which Kenneth has just started. So in order to connect to those, you basically just copy paste the IP address from here and connect with your username and key to the machine. Let's see if that works. Yes, so this is my stratum zero server. So it's probably best to just put all these IP addresses in some text file or something where you can easily copy paste them from since you need them quite a lot throughout the tutorial. So in a different tab, I'm going to do the same for the client machine so that I can quickly demonstrate you how this works. So I'll now go to the tutorial website which Kenneth has also shown you. So I will browse you through the page of the second section. So the stratum zero and client section and quickly show you how that works. And then later on during the exercises you can try to do the same thing again yourself. So again, this warning here that you should not do this in production but for now it's okay. In terms of resources, a stratum zero often even in production systems doesn't need a lot of resources because the scalability of the system gets added by having these replicas, the stratum ones and proxy servers. So there's not a lot of traffic between the stratum zero and the other system. So just a few cores and a couple of gigabytes of memory is often already sufficient. The main thing that you of course need here is a space, this space for hosting the repository. But even that will probably be less than you might first think of because it will compress the files and de-duplicate the files. And as Jacob already mentioned yesterday the duplication really helps for software installations. So in this tutorial, we will mostly be using CentOS 7 on all the machines. So all the cloud resources that you can use in Azure will all be running CentOS 7. Every now and then we do provide some pointers for CentOS 8 as well. There's some small differences for instance in terms of the packages that you need. But in principle, we will base this on CentOS 7 and on x8664. But note that you can basically do this on lots of operating systems or at least Linux there's abuses and even different architectures. So if you want to do this on Ubuntu on an ARM system that should probably just work as well. You just need to use different commands to install the packages. Also note that explained by Jacob already as well yesterday that you can also use for instance as three compatible storage for the backend of your repository. But we will not use that for this tutorial. So we will just use the local disk on the stratum zero to host all the files. So what do you need to install the stratum zero first? Well, basically just a few packages. We need the Apple repository because we have a dependency from CVMFS server on the JQ package, which is not provided by the default CentOS repositories. So we're just going to copy paste this to the stratum zero and install all those packages which might take a few seconds to install everything. So in the meantime, I'll just browse through the page. There are also packages that can be found on the download page of CVMFS website. I have some repository issue already. That looks like a mirror, which is down. Perfect timing. What do I do? Disable this one? Yeah. Okay. So let's try this for now. Always the demo effect kicking in. Well, this seems to work at least. So it's slightly different now. Just copy paste this. So this just adds the CVMFS YUM repository. It already works, I guess. So it adds the CVMFS YUM repository and now I just install the packages that I need. So on the server stratum zero server, I need both the client package or the default package which basically includes the clients and some basic stuff and also the server package. Let's hope this works now. Yeah, there we go. So you can also find individual downloads for a different kind of distros and architectures on the downloads page, but the easiest way is to just use the repository. As I mentioned, CVMFS, at least the stratum zero is just an Apache server running. So for now, we just want to start the server and enable it so that it automatically starts after a reboot. It's done. There we go. So let's start the server and enable it. So now Apache is running. So it can already serve the files, but of course we don't have a repository yet. So most of the server related things in CVMFS can be done with the CVMFS underscore server command. So both on the stratum zero and stratum one, you will often need this commands to do stuff. So the initial thing that you should do on the stratum zero is make your repository. So that works like this, make the file system and one of the options which is useful to use for repository system minus O option that basically sets the owner of the repository. So basically who will become the owner of which Linux account on the stratum zero? So it doesn't affect the clients. It only affects who can basically write into the repository on this particular stratum zero machine. Well, just a question. Do you take questions now or you take the questions at the end or? If it's urgent one for now, then sure, I can do that now. I have a question. So where are you gonna install these? You're making a file system, but where? I quickly skipped over that, I guess. We do explain that here under the storage resources, so basically here. So the default location is slash SRV serve and it uses a kind of scratch area when you're going to ingest new files into the repository and those will temporarily be stored in far school CVMFS. And you can change them, but I think Jacob recommended if you want to change them to just make sim links to another location before you actually set up the repository. So that's good indeed to take into account before I'm going to run this makeFS command. If I want to store the repository somewhere else, I should now set up the sim links to points to some larger storage area, for instance. For now, you don't need a lot of storage for the tutorial. So for now you can basically just use the default location and run this command like this. So again, this minus O makes my user account the CVMFS 999 account makes it owner of the repository, which means that later on I can do changes in the repository using this user account and I don't need root permissions anymore. So you do need them here to set up the repositories. So I have to run this with sudo, but later on that's not required anymore. Also important is that you choose a suitable name here and the recommended way of naming your repositories is some kind of name over here, whatever you want to use software or production or some other name, which basically says something about the contents of the repository and then use some domain naming structure over here, which will always be the same for your project or your organization and that you will reuse for any other repositories that belong to the same project or organization. That makes it much easier to configure your clients if they're all part of the same domain because you can easily add more repositories without having to do lots of changes on the client because you basically can add one domain-specific configuration file to the clients for all the repositories that live under this domain. So when you're going to use this in production, really take some time to come up with a good naming scheme for your repositories. You also don't want to make it too long since you might have to type this also on the clients every time you want to use the repository. So for now, I'm just going to use this, it doesn't really matter. And it's going to do some work here, so it's going to set up the initial structure of the repository, it's going to do an initial commit, make the keys that you need for access to the repository. And if that works, you should see something like this. So here we also explain something about the names again. So now you also should have a bunch of keys which you can find in the ETC CVMFS keys directory. So in CVMFS, you will find lots of files, both for clients and for servers. But in the keys sub-directory, you will now see some keys for this new repository. That's all being explained here, which key is what. But for now, so I'm not going to explain all of them now. The most important one that we need to remember for now is this public key for the repository. That's basically the file that you have to distribute to both clients and to your Stratom-1 servers. The other are quite specific for the Stratom-0 itself and for basically making changes in the repository. So this one is important. So next step, now that we have a repository is to actually add files to the repository. So the repository basically lives under this location, both on the Stratom-0 and on the clients. So you can browse through the repository already on your Stratom-0, but there's nothing, except for this dummy file that gets added automatically. But other than that, there's nothing in the repository yet. So this is the only file that you will see now. Just to show you the other directory that I mentioned where the actual contents is being stored. So that's here under sort of CVMFS. And in here, you will find the data files. And well, that's not really human readable stuff, but this is basically the compressed versions and the de-deplicated versions of whatever files you're going to store in your repository. So it doesn't make a lot of sense to go into here yourself. So how can you now change files in the, or add files in the repository? Well, for that we need a transaction that will be explained in the later section, but I'm just going to quickly show you this so that you know how to easily add just one more file. So you basically again need the CVMFS server command and then do transaction that will open a transaction on your repository, which you can publish when you're done. So we're going to open this by using the name of the repository, which should be like this. And note that I can now do this without sudo. So I can just run this as a regular user because I made my CVMFS999 account owner of this repository. So now I can go to the repository and I can add some file over here. So I can, let me just make a new file, hello.sh, and we'll start writing some scripts here. Let's just say, hello, make it executable. And now I want to add this file to the repository. So it's not really added yet. It basically is added now to some kind of writable overlay on top of my repository, but to really add it to the repository, I have to publish this transaction, which you can not do if you're in the, if your working directory is in the repository now. So I have to go out of the directory and then run the publish command. So that's explained on the website over here. So CVMFS underscore server publish and then the name of the repository, which I think can even leave out because we only have one repository at the moment, but if you're going to add more repositories, you of course have to indicate which one you're opening transactions on and which ones you want to publish. So this will publish it. There's also an abort command, which is also explained somewhere on this page, I think, added some in a later section about publishing. But if you make changes and then you realize I want to undo this, I don't want to publish them. You can just do CVMFS server abort and then it will abort the current transaction. So now I want to add them. So I'm going to run this. As you can see CVMFS automatically assigns a tag to basically a name to this transaction. This is a feature that gets enabled by default. So every transaction that I publish will get an attack name, which looks something like this, so generic and then the timestamp. Then it's going to do some work, basically compressing the files, doing the deduplication. It's going to store some information about which file can be found where, map the compressed ones back to the real location, et cetera, that all that's stored in a catalog. So it needs to update those catalogs. But if that all works, then at the end you will not see any error message and it's going to say that it has remounted and newly created repository version. Then the next step is that you basically have to add a cron job for the stratum zero that will resign a white list every now and then. That takes care of the security. Also explained a little bit by Jacob yesterday, but basically you need to use one of those keys that I showed before to resign a white list and you have to do that every 30 days by default. You can change that if you want to. So it's best to just set up a cron job for this, which will basically just run this command for instance, once a week. It doesn't really matter how often you do it, but it should be more often than the default of 30 days. So once a week should be fine because if you don't do that, then at some point the white list will expire and then clients will also lose access to your repository. So it's important that you do this. If at some point you want to destroy the whole repository, there's also a command for this. So RMFS does the opposite of made FS that will just throw away the entire repository. So now quickly go to the clients. I have to speed up a bit to finish in time. So what do we have to do on the client? Well, basically you need the CVMFS client. You need some configuration for the repository that you want to mount and then you have to actually mount the repository. And the clients will also have a local cache. So basically every file that you need on demand will be pulled in from either the Stratton one or zero or the proxy server and stored locally on your machine. And you can set the cache as I will show later on. So first let's just add the packages that we need. So again, I add the CVMFS repository to my machine. This time I don't need the server package which means I don't need the Apple release. This thing again. So just one package that I need only the common CVMFS package. Then the first thing next to that so the first configuration basically that we need is the public key for the repository. So that's the file that I showed you before on the Stratton zero, this public key. I'm just going to print it here and copy paste the contents and then go back to the client. So I have to add that key to my CVMFS folder and ETC on the client and then in keys. And then it's usually best to make a directory here that contains all the public keys for your organization. So I just named this organization.tld with sudo. So that's just the name of the repository without the first part. So without the repo dot. And then in here, I will make a file which does have the full name. So repo.organization.tld.pub, that's the public key. And then I'll add the public key here. Save it, didn't open it with sudo I guess. Right permission, so let's try again. So that's the key and then, so that's basically explaining this part. And then I need some configuration for my repository that I want to mount. And then I can put in config.d. There's already some dummy files here for a certain related repositories. But I'm now going to add the configuration for my repository. And this, there should not be a lot of variables in here. So you just need a few for now. Basically, this will be enough so that it will tell CVMFS client where the key can be found in which folder. So all the keys for this repository. And I have to fill in the IP address of my stratum zero. So let's copy that from here. And this path should always be the same. So this is basically where all the files are stored on the server, where this add FQRN add is some variable that will automatically be replaced by the repository that you're trying to access. So you can just leave that as it is for now and it will automatically fill in the right value for you. So that's basically already the entire configuration that you need. Let's go back here. Then there's one more file. That's the machine specific CVMFS configuration file that will be used for all the repositories that you're going to access on this machine. That's called like this, etccvmfsdefault.local. And in here you can, for instance, define the size of the cache that you want to use. That's on your local machine. So for now I'm just going to say about five gigabytes. That's enough. Depends a bit, of course, on the size of the repository and how much you want to store locally and how much you can store locally, depending on your disk size. But of course, the larger your cache will be, the more stuff you can pull into your cache and the better the performance will be. So the access time to your software. The other one is the local proxy cache that you want to use. So that's a local squid server that you can set up that will also be covered later on in a different section. So for now we don't have a proxy server yet. So what we do here is say, my proxy server is direct. That means directly connect to the stratum one or even the stratum zero in this case because we don't have a stratum one yet. So direct is just a special keyword that you can use here to tell CVMFS to not use a local cache. Ah, I forgot, pseudo again. This one and the quota. That's already it. So now I can verify my configuration by running this. So that will check if the setup is okay, which should I think be in reverse order. So it's a bit weird, but you basically first have to do the setup, which will also set up auto FS, which will be used by CVMFS. And then you can verify that, for instance, all the connections are okay, the connection from your clients to the stratum zero or one or whatever you're using. So if that's all fine, then you should see okay. And that basically means that your setup has succeeded. So now I'm done. And that means I can now start using the repository, which will always be mounted under CVMFS. But if you do this, then you might think that something went wrong because there's nothing in here yet. That's because CVMFS uses this auto FS, which basically will only mount the repository when you're really accessing it. And it will also automatically unmount it after a while in activity. So I really have to type in the name of the repository now, which can demonstrate that you should choose a very complex or long repository name. So first time it might take maybe a second or so, but then you will see the contents of your repository and I can run my hello script. And that's basically it. So I can now start adding more and more files which should show up on the client's item. And again, after a while, this mount point will disappear, but when you access something, it will automatically be remounted again. So that's it for now. So setting up the stratum zero and the client. And for the exercises, you can try doing this yourself. So you can try to set up a stratum zero server on the stratum zero virtual machine. So the machine that you can find here listed as stratum zero. They can make a repository name and you can try to come up with something yourself. So you can pick whatever you want, but make it a suitable name. They can also try adding some scripts here or a simple file that at least does something. So you have something to play with on the client and then you can install and configure the client on the second virtual machine that you have. So for now, again, you don't need the proxy in the stratum one. We will start using those tomorrow. And then make sure that you can actually access your repository, of course, which is the most important thing. So that's all I wanted to tell you today. So timing is quite nice. So if there's any question, we can look into that now. And of course, you can always ask us later on on Slack or send an email. We have some time for questions now. If people want to follow the other talks at the Easy Build User Meeting, that's a separate Zoom call. So just be aware of that. So that's getting started in about 14 minutes. But in here, we do have time for some questions. Can I ask a question related to stratum zero? Sure. You said that you allow on other servers to have right access to stratum zero so that you can have, I don't know, other servers with all the architectures, for instance, right? But what happens if stratum zero goes bad? How do you recover from stratum zero? You redo these entire steps with a different machine? Yeah, it's probably best to, if you really mess up the machine to set up a new machine, it's a good question actually, but I think you can copy back the files always from a client, for instance, or the stratum one. You can just re-unjust them into the repository. And of course, there are commands, for instance, for verifying the integrity of your repository. So if it's just a small issue with the stratum zero, you can probably recover from that. And well, probably it's also a good idea to make full backups of that machine in case you really want to recover quickly from disasters. But in principle, yeah, all the files should be available on the stratum one as well. So even if the stratum zero goes down, clients can still access the repository. And you can basically just tire up the entire contents of the repository and re-unjust them to the new stratum zero server. Okay, thanks. Any other questions? Was there something in the chat? Taking a look, there were some questions that I answered while it was going, let me check again. I have one. Yes. Sure. And so I'm wondering, in the case of, let's say that we would like to add some software installations and to the stratum zero, what's the procedure to follow? So should we set a stratum zero locally like you showed us right now? And then somehow contribute that to the stratum one? So you mean for- Or should we set a stratum zero locally that cannot add the software that we want to have in the repository to the one that is available centrally? So I'm not sure if I completely understand correctly, but so you can have only one stratum zero, but maybe the issue is more about where do you actually, for instance, build software and how to ingest it then or? Yeah, so for instance, you showed us how to add this hello script. Yeah. Let's say that this is some local installation that we have in our site. So if I want to deploy this hello script and do more production systems, should I go to the central single stratum zero server so to contribute the change there or is there a way to have that somehow added locally to my robot? Yeah, that's a good question. I think it was also covered a bit by Jacob yesterday because it's indeed by default the single point for ingesting files to the repository and then you basically have to, if you want to distribute the load of adding software and building software to different people then they basically all need access to that stratum zero well which might be some kind of security issue for some sites also because you want to well have access to that machine as less possible. So one way to get around this is to at least build the software indeed on different machines and then ship the compiled stuff as for instance, star balls to the stratum zero and then ingest them there but another approach which will be covered in the advanced section later on is to have a gateway running on the stratum zero which will allow publisher machines to be used and then the publisher machine is basically a separate machine so for instance, the build host that has right access to either the full repository or even to just the subpart of the repository. So then you can for instance say this publisher notes should only have access to this sub directory of my repository and then you can set up dedicated build machines that only can build software for particular applications or particular architectures depending on the structure of your repository. So that makes it easier to spread that load of ingesting files to the repository but indeed by default, so how we're going to use it now and during this tutorial, the only way to add files to your repository is by going to your stratum zero opening a transaction then inserting the files that you want to add or removing stuff that you want to remove and then do a publish to make all the changes available to all the different clients. Okay, thank you. You're welcome. There's another question that was raised in Slack which I think is a good one. So CVMFS uses HTTP for traffic. Is there any security conserved but not using HTTPS? I think Jacob briefly covered that yesterday in his talk. So CVMFS has internal security mechanisms that basically there's no point in using HTTPS that would only out overhead. I think that's the short version. Yeah, if you really want to, I think it's possible to use HTTPS but indeed it's recommended to not use it because for instance this catalog that I briefly mentioned already stores all the hash first or stores the name of the hashes of all the files that you have in your repository. So it already does some kind of security things and it makes sure that the integrity is in order. So if you access a file, you can be sure that it's the same file as on the stratum zero so that no one can basically change the network traffic and somehow change your repository. And there was another one as well by some, is there a way to connect to multiple stratum zeros? I think he actually means is there a way to connect to multiple repositories even if they are served by different stratum zeros? And they're the answer is yes. So CDMFS doesn't have a problem with that. So mounting different repositories even if they are served by different stratum zeros is not an issue. Okay, if there are no more questions. I have a question actually. It's a follow up of the HTTPS. Yes, if someone can read, I mean check what is going on through the traffic basically since HTTP. Is that potentially an issue or is encrypted anyway? It's not really an issue I would say because well, most of these repositories are probably accessible anyway. And there are ways to, for instance, use certificates in order if you really want to limit access to repositories. And well, even then the contents will be served in an encrypted way if I'm correct. Thank you. So for instance, if you want to mount commercial software or add commercial software to your repository and only want to have a certain set of clients get access to that repository then there are also ways to do that. So we will not cover that in this tutorial but I think that's also documented on the CDMFS website how to do that. Thank you. All right, so if there are no more questions I think we can wrap up for today. Yes, we'll end the stream and the recording here but actually the next talk at the user meeting is only starting quarter past. So that was a bit too quick there. Yeah, we still have time. But we'll wrap up the session here but we'll stick around a bit longer if anyone has any more questions or any issues with setting things up playing around either ask them now or ask them later in the Slack channel. And I think if needed tomorrow we will be around shortly before the start of the next session so people can then jump in a bit earlier and ask questions as well. Thank you everybody and see you tomorrow or see you in Slack.