 Hello, my name is Anthony Brotodou, I'm from Rennes, in France. Today we will run a tutorial about Galaxy interactive tools. These tools are a special kind of tools in Galaxy, and they allow to run interactive web applications directly from the Galaxy web interface. Using these kind of tools you can launch things like Jupyter, RStudio or other specific web interfaces. So during this tutorial we will see all the technical side of these tools and see how they work within a Galaxy instance. So we will start from a Galaxy instance already running and we will see how to configure it to support these interactive tools using Ansible. In order to run this tutorial you will need first to have followed other tutorials in the Galaxy server administrations. So hopefully you will know how to use Ansible using this tutorial and how to install Galaxy using Ansible. The tutorial we will use today is this one, Galaxy interactive tools. So we just click here, and as you can see the main steps of this tutorial are first to install Docker to actually run the interactive tools. How to install the interactive tool proxy that will redirect the HTTP traffic to the Docker running the interactive tools. Then we will see how to configure NGINX and how to generate wildcard SSO certificates. And finally we will see how to enable interactive tools inside the Galaxy configuration files. Okay so now we will run the tutorial on a specific VM that is provided by the Galaxy admin training session, which is number three. And as you can see Galaxy has already been deployed on this VM and is accessible at this address. And as you can see in the tool list on the left there is not yet any interactive tools installed. There is no interactive tools category. So now we will see how to configure our Galaxy to have these tools. So to do this we need to go using SSH to these VMs, the correct password, that's it. Now on this VM you have a special directory which is Galaxy, which contains the Ansible rows and configuration that were used to install Galaxy on this VM. And now we'll modify these files to install interactive tools. The first step of this tutorial is to modify the requirements.yml file to add two new rows that are needed to install interactive tools. So you paste it from the tutorial page. So the first rule is the docker rule, which will be used to install docker in the VM. And the second one will install a specific interactive tools proxy, which is just a small application that will be used to direct the HTTP traffic into the docker containers. GIE means Galaxy Interactive Environments, which were the ancestors of interactive tools. So we save this file and now we install these new rows using the Ansible command. So as you can see, the first rows are already installed, but the two new ones are being downloaded and installed and it worked, which is a great success. Okay, so now that we have installed these rows, we want to use them actually. And so we'll start with docker. We have to configure it with two variables, which we write into group-vars-galaxy-servos.yml at the end. So you copy it from the tutorial. And these two variables are docker install-compose-at-force, which means we don't want to install docker-compose as it's not used by interactive tools. And the second one is docker-users, which is the list of users that are allowed to run docker containers on this VM. And now as Galaxy will run these docker containers, we want this user Galaxy to be allowed. So we will use the Galaxy user.name variable to fetch the Galaxy username from the whole configuration. So we save this. And the last step is just to edit your playbook, which is named Galaxy.yml. And in the list of rows that need to be run by this playbook, you just add this new role we've installed and configured. Okay, so now we just run the playbook and we need to specify that we're using the window user to run this role. So the docker role is actually running and installing docker inside the VM now. Okay, so now docker is installed. As you can see, I've skipped a part of the process to make this video not too long. It can take a few minutes on your VM. So now the first thing you can do is to check if docker is really installed. So you can run a docker command and see if it works. For example, docker ps. Okay, there are no arrows, so it means docker is installed and can be used on this VM, which is great. Okay, so we can go on to the next step, which is installing the interactive tools proxy. So this proxy, its role is to direct the traffic, the HTTP traffic coming from the web, onto the correct docker containers corresponding to the exact container that the user wants to use from the Galaxy interface. Every time a user launches an interactive tools, it runs a specific container, and this proxy ensures that each user will have access to its own container that he has launched. So to install this proxy, you need has docker to define a few variables. So once again in group files, Galaxy server.tml, you can add a section with these variables coming from the tutorial page. So the first one will tell Ansible where to install this new proxy. Which version to use from Git, how to install Node.js. So use Node.av to make sure we have the correct Node version on this VM. The next one is telling which virtual.av command we can use to create a new virtual.av. Which version of Node.js we want, where to install the virtual environment, and setup services to specify that we want this proxy to be launched and stopped if we want using system D. And finally, we have this sessions path. So this file interactive tools map.esculite is a specific file, which is written by the Galaxy every time a user launches an interactive tools. It is written into this database on the disk, which container was launched running on which port on which machine, and which identifier this container has. And the proxy will use this information so it will read this file to determine from the URL, which is used by the user, which container should receive the HTTP traffic. Okay, so we save this configuration file. And once again into the Galaxy playbook, we will add that we want to run this other role, which is named useGalaxy.UGIeproxy. So now, once again, we run our playbook as the human to user. Okay, good. So it seems like the GIE proxy is now installed. We'll check that. The easiest way is just to look if it was correctly installed into its destination directory, which is a savvy galaxy slash GIE proxy. And there should be a virtual and there. Yes, it's there. And it should also be able to look at the log from this new service which was launched using system D. So if you run this, you can see it. So as you can see it was started, but there was some errors. I'm not sure if it worked. Let's see using system D. Okay, system D is telling me that it's working. See it was started one time and it failed and it was then restarted a second time and it worked. Can always look by looking if there's a not process running. Yes, as you can see, there's a process. There's a not process with the GIE proxy code, which is running and listening on this IP on this port, but 8000 and using this SQLite file containing the correspondence between containers and URLs. Okay, so this is the first step. That is down and we'll see now how to configure engine X on top of this. Now that we have this interactive tools proxy running and we need to tell engine X that to use this proxy when it receives some HTTP traffic directed targeted to interactive tools. So to do this, you just need to come to change the engine is configuration which is already into group vase and galaxy servers dot channel. So if you look into engine X SSL server, you have currently only one server which is galaxy, which means it's, it's for the main interface of galaxy. And there we will add a new configuration which will, we will name galaxy GIE proxy. We save this. And now we will create this configuration as a template into the templates engine X. And there we write the same name we've written in the previous file galaxy GIE proxy dot J2. And now we copy the content from the tutorial and we'll see exactly what it means. So if you look at the other configuration, which is galaxy dot J2. It is directing the traffic from the host name of the VM, which is server name here. And it directs it to a server, a USGI UWSGI running on the port 8080 on this VM. Now, if you look into the new configuration we're adding, it says that every web traffic received by engine X with a server name prefixed with interactive tool. So for example, something dot interactive tool dot GT dash three dot Z dot training dot galaxy project dot EU, which is the host name of our VM today. Every traffic coming to this host name will be directed into the GIE proxy, which is running on the same VM. So the GIE proxy port is 8000, which was configured on this step before. Okay, so we run again the playbook to update the engine X configuration like this. Okay, so you have seen in the logs that the engine X configuration was changed. So that's why it shouldn't change three here. So engine X now should be able to direct the HTTP traffic correctly to this new proxy. The next step of the tutorial is getting a wildcard SSL certificate. So currently your galaxy server is configured to have a certificate generated by let's encrypt for the host name of the galaxy instance itself, not for interactive tools. Interactive tools have specific host name with auto random number at the beginning of the host name. So you need to have certificates which are able to handle this. There are several methods to configure this and it depends a lot on the way DNS was configured for your training. So for this session we will just keep this step of the tutorial, which means you will only get a warning from your web browser when you go and try to access an interactive tools from Galaxy. You will see how this certificate is not valid for this host name. Are you sure? You will see yes, and it should work. For production instance, you will have to configure a wildcard SSL certificate correctly, depending on the configuration of your server. So now we have a galaxy server which is running correctly. We have an engine X server which is configured correctly directing some traffic to the GI proxy which is installed and running and configured properly. And Docker is also available for the Galaxy user. So all we need to do now is to configure Galaxy to enable interactive tools in the web interface. So to do this, the first step would be to write a toolconf file specific to interactive tools. So we will write it into templates, Galaxy config and we will name it toolconf.interactive.xml.j2. So this is a new toolbox where we have one section which is named interactive tools and one specific interactive tools, which is etercac, which is pre-installed inside the Galaxy code. So we save this. And now we need to tell Galaxy what to do when the user executes an interactive tools, which destination to use. So to do this, we modify another file, which is a jobconf file, which already exists, we are lucky. So as you can see, there is already a destination which is running jobs into the local VM, which because our VM is not currently connected to another cluster like slurm or condo. Now we will add a new destination, which is specific to interactive tools. So in the list of destination here, we add this block from the tutorial. Just remember to remove the plus sign at the beginning of each line. Okay, so this is, you just need to modify this to match the configuration pre-existing. So there is one plugin, which is the local plugin that will run jobs locally. Here we want to have the destination that we run docker containers locally on the VM. And for this destination, we say that we want to use docker. We want to mount a few volumes inside the containers. We'll talk about it a bit later. We don't need to be a sudoer to run docker containers because the user galaxy is allowed to route containers. And some other options like the network mode for docker. If the docker should be automatically removed when it's finished. And that's it. So we have this destination. And now we need to tell that the new tool we have added just before, Heterocalc, needs to use this new destination, which is named interactive local. Okay, so this file is ready. And now we need to make sure that these configuration files are properly used by the roles. So we go into group files, galaxy servers dot yaml. There should be a galaxy config tool config files somewhere. No, there's not. Okay. So we just need to add it at the end. Or yeah, maybe. Yes. It will be enough. So this is from the tutorial, just have to copy and paste. And now inside the galaxy configuration itself. You need to tell that there is a job config file to use this was already done before so we don't repeat ourselves. But we need to say that. Here, we want to enable interactive tools into galaxy and to use this escalate file that was configured just in the previous steps. Save this. And we also need to tell that we want to to use a template we have just written, which is name like this. This is the list of interactive tools. So doing this we're ready to tell galaxy that it should now support interactive tools we just need to run the playbook once more like this. Okay, so the playbook has run and did a few changes as you can see here. And so if everything went well, you can just go and see if the interactive tools are appearing in the web interface of galaxy. And oh, it's so magic. I reloaded the page and there's a now a new section which is interactive tools here. With it, which was added that you can click and see like this. So now it's time to test this new interactive tool. To do this, you need to upload a file, a sample file from the tutorial again. And we'll say it's of tabular type. So if you look at the file is just a tabular file. So it's just a tabular file with three columns that can be you. So it's our car keys next cell like web application which which you would see just soon. It's very simple for me just have to select the tabular file you want to to show into this web spreadsheets, and then you click on execute. And that's it. So when you run an interactive tools, you see this message. So as you can see, it was a disaster. It's a calc failed to launch with a big red arrow. I went to the debug icon here. And I had this kind of strange errors. And the first one was forever command not found so forever is one of the common which is run inside the tool script. When it's run inside the containers. The fact that it's not working means that the command was galaxy tried to run the command outside the container just as a normal to which fails because the command is installed into the container is not available outside the container. So to debug this, I just went into the console. My SSH and I looked at all the files that we modified just before. I've seen an error in one file, which is job contact xml.j2. And as you can see, I made a bad copy and paste here. So I correct this and rerun the playbook. Let's wait a little. Okay, so the playbook is now has now run and everything is finished. It's time to test second time hoping that it will work this time. So we go there and we launch again at our car crossing fingers. The first time you run an interactive tool, it can take a while to start because the first step is to download the docker image that galaxy needs to run this tool. So depending on the size of the image and the network where galaxies installed can take a while. So now our interactive tool is running. And you can see it from here that there's a link here. Click here to display. So you just need to open it. I will open it in a new tab. And so here you have an error, which is in French, which means that we have a certificate problem as the certificate for this domain, which was a sign randomly. So you see it's a random number followed by interactive tool entry point dot interactive tool dot the rest of the galaxy URL. And the certificate was also only made for this part of the hostname. That's why you need a wild card. This is a certificate as we don't have one on this VM. We will just go into advanced parameters and accept to use this invalid certificate by clicking here, which is dangerous. Say scroll. And as you can see now, it's a character, the web application, which is running in a docker container is appearing. So it's bootstrapping at the moment. And you should see the data set appearing inside the spreadsheet inside the data set that you selected from your string. So once you hear you can use this web application just as just as you would use it if you had installed it on anywhere outside galaxy. You can share this URL with everyone you like will be accessible. And so of course it's in French, but it will be adapted to your web browser. And technically, if you go and look inside the terminal seconds. So we are inside the VM and we have launched an interactive tools and if we look at docker PS which will list every container running currently inside the VM. And you will see that there is one container which is named like this random number. And that uses this image within which is a network specific image for galaxy. And it's running since two minutes ago and listing on this random port. So when you go and access the interface. So your, your web traffic is passing by the engine x proxy, which sees that there is a longer house name with a random number which directs the traffic to the proxy. The GI proxy from this random number is able to know that it should direct the traffic on to the port. 49154, which is running on the same VM. And, and that's it so you have access like this through to proxy to the containers which is actually running on the VM, but that could be running on any slur or condor cluster if you want. So we have just one problem with using interactive tools like this it's a security problem and it's the last step of our tutorial. So if you look at this command that you need to run as pseudo this. Yes. This is all the, the mount points, the volumes that are mounted inside the containers that is still running for attack. You see, so the syntax is the first part is the directory on the VM, which is mounted into this pass inside the containers. And the last part is specify whether this mountain is done in the read write access or read on the access. And if you look carefully you see that the end of the list concern only the job so it's okay the container needs to access the working directory of the job. It needs to access to the two directory where the XML the file is and things like that, but the first one is very dangerous because stash data is the directory of your VM where every data of your galaxy server, every user data is stored, which means potentially as a car can access any data from galaxy from any user of galaxy and modify it if you want. So this is a matter of trust with interactive tools we know that with that our character is few risk to actually be able to modify this data but for other interactive tools like Jupiter you could as a user do really bad things and it's really bad for security. So there are a few possibilities to to fix this security problem, and we'll see how to do it. And so the best option is to use an embedded to saw destination, which will allow galaxy to run the containers by and mounting only the necessary that data sets that are needed by the containers. Data sets that were selected into the form of the tool. So to do this will modify a few files and and to configure it properly. The first one is a new file. It's templates galaxy config and we will name it pulsar app.tml. And we'll write this content into this file from the tutorial. So it means when we run containers for an interactive tools using pulsar embedded mode. We will use a special directory where per job directories will be created. A directory where some specific pulsar state information will be stored, which will be at this position. A directory which will be used by pulsar to determine where dependencies are installed, even though with containers it's not really needed. And finally, we'll tell how we tell pulsar how to manage job job queues. So here we just run job on the same VM as the galaxy server because it's the embedded pulsar mode. So once you've done that, you just need to change the destination of your tool. And to do this, as usual, it's in the same config file, which is templates galaxy config job com.xml.j2. And here you will have a new plugin. Instead of local, you will need to add this new plugin. So it's named pulsar embedded. It's a runner with a specific class, and it's configured by the file we've just written before, at least the template we've written before. Okay, and now that we've defined this plugin, we just need to modify the interactive destination to say that we want to use this new plugin, which is pulsar embedded. So we need to add just two other options, which are here. Once again, copied from the tutorial. Okay, we need to change some variables to say that we have a new configuration file for pulsar that needs to be copied into the galaxy directory. So here we go. We save this. And also, we need to write a specific, wait, okay, I found it. This mod galaxy needs to have a specific variable, which is named galaxy infrastructure URL so that the containers and pulsar knows exactly at which address the galaxy server is running. So we just write it like this and inventory house name is variable, which was defined, which is defined by Ansible. Okay, so now we just once again run the playbook like this Ansible playbook dash you want to get to the channel and wait a little. Okay, so the playbook has finished and now we'll see if it works by going to our web browser, which says we were disconnected, it's not a problem. So now we can stop the container that is running from before, delete the failed one because I don't want to see read in history. And now we will run again as a clock like this. The container is started and we can go here and see if it's working. Once again, the certificate warning that we ignore, and it seems like it's working. Yes, we see all that data set we selected. So now let's look at the Docker container, which is running and see if it's really using pulsar. So we should still have a Docker running. Yes, since 58 seconds. And now we can run the same command as earlier to see which volumes are mounted inside the container like this. Okay. And here it's much better because as you can see, only the directory related directly to the job are mounted inside the containers. And everything that contains the six means it's a job number six on this instance, which is specific exactly to the job. And the only other directory which is mounted is tool data, which is the reference data of Galaxy, which is fine to be shared to any jobs in read-only mode. The slash data directory is not mounted at all in a container, so the container can't access it and nobody can read or destroy data from other users. So that's a good thing. Okay, so we have finished this tutorial for now. Just remember that interactive tools are still quite young, so there may be some a few bugs or things not working exactly. Ideally, but it should improve progressively in the future. If you look back at the tutorial, I strongly recommend you to feel the feedback form, which is at the end of the tutorial here. It will help us a lot to improve the training material based on what you liked or didn't like in this tutorial. And finally, if you have any problem, you can always join us on the Gitter channel of admins of Galaxy project for the interactive tools or GTN for everything related to the training material here. Thank you for following this tutorial.