 Hello everyone. Thank you for joining today's Bitesize Talk. I'd like to begin by thanking the Chan Zuckerberg Foundation initiative for supporting all NFCO events. So a few, some information about today's Bitesize Talk. So the talk will be recorded and is being recorded at the moment and the video will be uploaded to NFCO's YouTube playlist and also shared on a Slack platform. After the talk, which is a 15-minute talk, we'll have a session for question and answers. So feel free to post your question on the chat box or unmute and directly ask the speaker the question that you have. So for today's talk, we are honored to have Yvonne Floden, who's the CEO and co-founder of Secura Labs, who will be presenting on Nixlo Tower, which is a centralized command post for launching Nixlo Pipelines. So over to you Yvonne. Okay, thank you very much. Thanks for the intro and thanks for having me here again. So just a bit of a clarification. I'm really going to specify that we're going to talk about Nixlo Tower and the CLI in specific, specifically that we've released it towards the end of last year. It's been like a kind of work in progress and something that we've been sort of wanting to do for a while now. It really tries and answers some key problems that people had. You have Tower itself, which has a graphical user interface and also has an API. We really want to solve the problem for people who are used to working with Nixlo from the command line, but also people who want to have infrastructure as code like set up. So maybe as a brief introduction, I can start on a little bit about the kind of background to it and also the background to Secura. So if you don't know, we're the main maintainers of Nixlo Project and the company's about four years old now and really kind of solve the problem primarily around the workflows becoming more complicated in terms of number of steps, but also the larger amounts of data and the kind of cloud and cluster configuration side. This kind of really ties in with what we've been doing with the CLI because it really allows you to define like your whole research environment. Nixlo and you know, particularly NFCore have done a fantastic job on defining like the workflow. And if you think about that workflow being typically a repository, you know, you've got some profiles, you've got some sort of test ways of running those pipelines, but there's almost like a problem outside, which is like a little bit bigger than this, the workflow definition, which is like, where does that workflow run? So how is that infrastructure created? How can I create all of that in a kind of reproducible way? And how can I start to do this when I'm using long running services without having to rely on my laptop being the kind of the key provider of all of that. So that's what we're trying to go into a little bit today and it really should be a little bit of a show and tell session with some examples of how you can do this from the command line. So please forgive me if there isn't any kind of typos as we go through this, but hopefully it can give you a flavor of what's possible and you can start to think how you can apply it in your situation. So I'm going to go into the kind of the meat of this first and then this is really kind of the architecture of Nexplo tower, not something specifically to know tons about, but the point is that it's like a full state web application and it connects in with these computing platforms. So all of your different cloud providers and obviously the data associated with that with the repositories. And then how do you interact with that yourself? I mean, just typically three ways of doing so. The first is obviously through a web browser. So you're connecting and sort of going through cloud or through kind of a user interface like model. Then there's the command line, which I'm going to show you today. And all of that is kind of built around and sort of built on top of this API layer. So towers published the API. It says open 3.0 standard from that API. So you can go into the documentation and do that. And we have a lot of people who are just building purely on top of that API layer, but there was kind of a need to be able to interact with tower also from the command line. Here's a kind of quick example of what we're going to see later on. But the idea is that you can start to then launch pipelines, monitor these pipelines, understand a little bit what's going on and define the infrastructure in this way. So I'm going to go through a couple of things. The first is just the very basics of the installation and the setup and then how that works, show you the repository. And then we're going to also go through and just see how we can launch some pipelines and how we can sit up to infrastructure as code. Now, the way that the CLI is written, it's basically in parity with the API. And this means that you can do a lot of things, and probably many more things than we've even kind of considered for most applications. When you see this, I kind of want you to consider like other use cases. If you've got some, it'd be fantastic. We had people reach out to us the other day who were using this for uploading data sets and triggering pipelines. For example, when we wrote a blog post on that, it was sort of a fantastic piece of work. And we kind of see a whole bunch of other use cases outside this kind of direct setup. So please shout out if you kind of want to see how that works. So the first thing I'm going to point you to is just tower itself and the way that it connects in terms of the CLI. So you can go to cloud.tower.nf. And when you log in here, you've got your credentials that you can log into. So typically people are using GitHub or Google to log in here. And the first thing you have is like the community showcase, which is what we're going to be working in for some of the parts of the day and really a collection of pipelines. There's some free compute so you can jump in and start launching it. And you can interact with the CLI as well. The point that I want to make here is to set this up the way the CLI works. It connects to tower and it connects via your token. So you can create your own, as many as you want, sort of access tokens here. You can see that I've created a token for the NFCore demo token here. And when you create a token, let's just give a demo example here. You can see it just gives you that key there. So we can basically, there's how it sort of links in there for doing that. I can just quickly remove that one in case of demonstration purposes. So I've got one which is linked in, and I'm just going to go and export that into my environment. So how does it all that work? So the linking that works is that we have the tower CLI, which is a standalone open source piece of software, which you can download from GitHub. We have built it in all the different environments. So if you go to tags here for each version, we build this for Linux, for OSX, for Windows as well. And then you can simply download that file and really move it to executable and run it here. So I'm running a Mac today. So I've just downloaded the latest Mac version into my terminal. And then you can start to run it here. There's a little bit of background on how this works and how you can set this up. But I'll kind of leave this for later on and we can jump into a demo specifically and see how that works. So I'm now going to switch over to my terminal, to my command line. And we can run a little bit from there. Okay. So here's my command line. The first thing you can see is that we basically, instead of calling it sort of tower or anything like this, the command that we call it is the tower. And you can see the different commands that you have here. So tower itself can really interact with all of the different aspects of the system. So we have things like actions for automations, organizing collaborators, compute environments, credentials, datasets, as well. The other nice thing you can do here is to write tower info. And this is a kind of like a health check. So it's going to go connect with the tower instance. It's checking that we can connect to that there, the version that we're using, showing which user I am, when I'm authenticated to do this, and checking some details there. And this can be quite useful as well for just kind of setting it up and running that for how you go. So the way that you set this up is you would typically export an access token here. So in this case here, I can got my access token, which I've copied from before. I would then paste it and export that there. And then you're kind of good to go and you're connected. There's a couple of other different options that you want to do when you're considering the setup. The first one is really around which workspace you want to work in. I showed you the community workspace before. But if you have your own workspaces set up for your organization, you can simply change that from environment variable, in this case the tower workspace ID, but you can also change it from the command line when you're running this. So you can just say tower workspace from there. So I'm going to say here and just export my one that I'm essentially the community showcase. And anything I do now, you'll be able to follow along live. So if you log into tower, you'll be able to see all of these actions and essentially pipelines triggered off and things that were created, followed them as we go through. So I'm going to export this tower workspace ID here. Just going to confirm that everything is working and it's all fine. So I've uploaded my credentials connected to the right workspace and we're ready to go. The first thing you want to consider in the first kind of use case for tower is the primary use case of what we all do is to basically run pipelines. So kind of how we can trigger those off. And in tower, we have a special concept of what I call as pipeline is. It's essentially almost like a reserve name that we've created for the system. And a pipeline is a combination of the workflow repository. So you think of this as like you get repository where the kind of source code is hosted, combined with a compute environment, which is really where the execution of that pipeline will take place, plus some parameters, essentially inputs that you want to see default parameters, but also other parameters that you want to use. So those three things come together for what is called a pipeline. And if we go into now here and we see inside of the tower showcase, and you should see the exact same thing. If I was to say here, pipelines list, we'll see that it lists all of the pipelines that we have inside of this community showcase. And this is like the end of core pipelines, but also we have some other ones we're adding in here all the time. And these are kind of preconfigured ones, which are good to go. So again, it's a combination of the repo, some compute environments, and some parameters. If I want to launch one of these pipelines, I've got a couple of different options and come a couple of different ways that we can do this. Well, the first thing is to say, we could say tower pipelines here. And I want to say launch. And when I launch this pipeline here, I can then choose the name that I want to give it. I'll essentially launch that pipeline, or I can choose an ID. Let's just choose like a default one. And we will chase that we want to launch in if core chips. So I say this here. And you can see that I have made a mistake here, pipelines launch, I think we have to give it a name here, typo. No, of course. And just copy paste my example, maybe I've made a typo here somewhere. Okay. Okay, I must have made a typo there somewhere. So this is the interaction that we've basically taken place. And we're submitting now that pipeline was saying launch that pipeline with tower. And it's important to note sort of why is this different from from next flow run. And the primary difference is that when we're launching this pipeline, we're directing tower to launch it. There's nothing running now on my local machine where this is launched. There's no like next flow instance, there's no kind of head job or anything. Everything has been delegated to tower, which in itself will submit that compute into the computer environment where that's working. That's got a lot of advantages. First, again, I can shut down my laptop if I need to run out to go to any work at the end of the day. The other thing is that all of the essentially the record, all of this, all of the logs, all of the information about the execution of the pipeline has been managed now by tower and has a full kind of history of that. That's kind of a very important thing. You don't want to be sort of reliant on your laptop or where you're launching the pipeline to manage that information. It also means it's been collaborated. If you go into the workspace now and click on this URL, you can follow along live. This allows people to interactively work together. Maybe there's an issue that you can fix, and then you can fix that and launch that again. We can work in a much more collaborative manner as well. That was a kind of basic one where we didn't actually change any of the parameters. This was a default launch of that. But what if you wanted to start running some of the more pipeline where you're actually changing things up? You've got a couple of different options for doing that. We can define a little bit like the next floor command line here in terms of the different stuff we can run. I want to say now I want to run the viral recon pipeline from NF core. Let's say that I want to now, instead of just run it by default, I want to use an expert profile. This profile may be some test data that you've got. Maybe it's a profile because of something specific about what you're going to define there. I can just use the exact same profiles here to launch that as well. That's going to trigger off there. It's going to be launched into the community workspace I'm doing as well. Another more common use case is really around not so much the changing of a profile but changing off the input data. What you can do here is very much like what we call the params file. You can create a params file in YAML or JSON here. You can see I've defined some input data. I've got some different options for that for those inputs. Here I've saved this in this file called params.yaml. I can then go say, okay, let's go launch the tower launch of the NF core. Now I've got the choice here of defining some profiles but I could also let's say add in now the params file itself. Let's just say a profile for test. I also want to put in here the params file exactly like we would with the exploit. At this point here I can just specify exactly what the location of that params file is. Obviously this allows you to kind of pre-define a lot of stuff that you're working off like sample sheets that you've got pre-defined there and they can kind of trigger off there as well. Okay, let's enter the params file. I think I've probably made some typos in this much better copy, pasting and the life demos than trying to do this so it can prove it works. Okay, so these are all that and then we can launch that. So this is the kind of basic use case of launching those pipelines. You can of course monitor them. You can kind of follow them along. Maybe you want to do that from the GUI. Maybe you want to do it from the command line. There's a whole bunch of different sort of end points there. I want to kind of switch gears a little bit now and think about how we can define a little bit of the infrastructure around this. So so far we've just kind of launched those pipelines and we've been on models of them. What about if I wanted to now do this whether I want to set up pipelines for other people or define my research environment in a kind of more generalized way? So I'm just going to just switch the change over to a different workspace here. I don't want to kind of build the stuff in the community workspace. I don't want to populate it with different things. I'm first just going to change over to the workspace here and then I'll look at the different pipelines that I have inside this workspace. So this is a private one that you can't see but its principles are exactly the same here. So I'm just going to say tower pipelines list. It's going to show me all of the pipelines in that space and you can see that just whole bunch of stuff that we've been populating and inside of here. You can see the repository associates with see a lot of the core stuff and then the kind of name of the pipeline. What I want to do now is imagine that I wanted to say take this and I had like a test version of this or development version and say I wanted to copy that or I wanted to give that to you. So you could kind of capture the whole thing in your environment or put another way. I wanted to really define exactly what that pipeline is made up of beyond just the workflow itself. So one way that I can do that with tower is actually to take the pipelines. So take the pipeline's command here and then export the particular pipeline itself. And the way that this works is when I give it a name. So let's just take the first one from the top or copy paste this time as opposed to trusting myself too much. When I export this, you'll see that it's exported entirely as this Jason found. And this has got a fantastic functionality because it means I can import and export things using this using this command and really define all of my pipelines as code. So these pipelines may have different configurations, different setups for different environments. And you can kind of define that now entirely inside of the Jason file, but also you have a pretty nice way to interact with it in this regard. So that means I could go create a new pipeline, maybe change one or two things out. And all of that infrastructure is kind of shown there as well. This kind of principle of importing and exporting, defining and kind of having this at a stored location works for all of the resources. So I'm showing you pipelines here, but the same thing applies to if I was going to show you, for example, the credentials. So I want a list inside tower of all my credentials that are inside of this workspace. And you can see I've got credentials here for Google, for GitHub, for Azure, etc. All of that becomes available for me to see. And then I can kind of think about how I can link that into actual computer environments of generating this stuff on the fly. So if I wanted to, for example, export my compute environment here, which is, as I said before, it's going to be like where the compute takes place. So it's going to be my AWS batch. In one example, let's say there's some credentials for that, how that's set up. And you can have a look at what one of those looks like. So this export here, the credentials, you can see that this is the whole definition of that that is required. So this is running on US three, etc. All of the working directory for that and any kind of how that sets up. So there's obviously a whole bunch of these things that you can learn just to kind of give you a full demo and probably makes a little bit more sense now all of the commands that we've seen here. If we do tell command on here just by itself, you can kind of see that you can start to interact with creating the workspaces themselves participants, generating the pipelines, creating data sets. And I just will point out one more thing to kind of maybe to give you some sort of inspiration on what's possible here is that we recently released a blog post around this, which really took many of these ideas and put them all into kind of play. The concept here was that we wanted people to be able to essentially drop a sample sheet or sample comes off a sequencer. And as part of that will trigger the execution of a pipeline all the way through. And the tower CLI is obviously is obviously perfect for this. And we wanted to kind of set this up on on the first case on AWS itself. So the way that we did this means we defined some Lambda function that essentially takes a sample sheet which gets generated when data enters to an S3 bucket. And then it uses the tower CLI to generate a data set deposit that into tower and then to invoke the job with tower itself. So it kind of allows you to then to integrate with many other different services for doing so and then create this whole. This is a kind of a walkthrough if you're really interested in and seeing how this can be done. We provide all the files. And there's also a Git repository here if you want to go through and follow this up yourself. But I'll kind of end with that. We're really excited to see what people build with this. We are sort of expanding this out as well as as we add more functionality to tower, you know, keeping everything really, really kind of aligned in that respect and really excited just to see what to see what people build with it, you know. Thank you, Evan. That was a really interesting talk on next floor tower. I don't know if there's anyone who has any questions to follow up on this. Maybe I can start the questions off. So I haven't used next floor tower before, but are there any more supplementary dependencies you need on your local machine when you're using tower? So when you sit up with tower, there's a couple of ways to do it. One of them is just writing with tower from your next floor command. So you can say next floor run with tower. This is still running it with your kind of the head job is still on your laptop. To connect externally, you should log into tower cloud and then you can essentially create your computer environment like that. So they're kind of two different ways of working. And they're kind of to really get the full power of this. You should kind of log in. As I say, it's running as a service in there and to provide full clarity around this. So this is tower cloud, which you can go and log in and use. And our business model is primarily around deploying this in customers own environment. So customers have their own version of tower. This is the public one I'm showing you, which you're free to use. We have a question from Paul. And then we'll go to other people who have their hands up. So some of Paul asked, I have access to a HTC supercomputer at my institution. What is my use case for NF tower? I've only used NF tower on a limited basis and don't see the need of it. Who are the main users of NF tower? Yeah, we can think of a couple of use cases. I can jump in there and maybe demo a little bit. So one thing in terms of HPC, I'm showing you kind of batch and AWS batch and different cloud environments here. If we go down to that same workspace we were working before, these are the same computer environments that I showed you. And you can connect in here with all the different platforms. So you connect in with your own, in this case, your PBS or your own slim cluster that you have here. And this kind of organizes the infrastructure side of it. So you can connect those bits into there as well for that part. There's kind of three primary use cases around the use of it. So if you are creating pipelines and you want to make them available for anyone who maybe doesn't even have an explore expertise or command line expertise, experimentalists, et cetera, you can create your pipeline and define it in a way which makes it super easy for them to come in. So here you can create your own customized user interfaces here. So as a user, I just need to slick my input data. I would come in and I say, I can run my RNAseq sample sheet against that. I want to go, you know, have some kind of options around this and maybe I want to save some particular files. And then I can kind of trigger the execution of that job often. It kind of simplifies the whole kind of launch process for them. As a buying from a tissue, you probably want to create those pipelines and make them available with compute to your users and to do that. Maybe you want like a long running service, you don't rely on that. You have a kind of full history of your execution. So you can follow those pipelines as they're kind of going through as well. And you can also kind of automate things as well. From the kind of system admin side, you have the computer environments, which can be defined. And really a lot of work around the, obviously, the collaboration side of it. So there's a whole bunch of use cases there. Okay. So apparently everyone else was appreciating your talk. I don't know if anyone else has a question. Okay. Seems like Ron is satisfied. So thank you so much, Yvonne, for the talk. And I'm sure people can catch you on Slack if they have any questions. Absolutely. Thanks so much for the time, Ron. And yeah, reach out if you have any other questions. Always happy to take them. Sure. Thanks, everyone, for joining today's bite-size talk. See you next week. Thanks so much, folks.