 Welcome, everybody, to the NFCore Bite Size Talk series. It's a pleasure for me to welcome today, Ebon Floden. Ebon is the CEO and co-founder of Secura Labs, most of you already know him, and he'll be introducing today Next Float Tower, one of Secura's products, and showing us the latest features. So thanks for joining us today, Ebon, and look forward to hearing about it. Well, thanks a lot, Bezella. I've been following these talks along since they started, and it's really great to have this kind of continuity and continuation of ideas and kind of a nice forum for these places. So what I'm going to talk about today is a little bit about Next Float Tower, and I'm going to go more into the kind of new features that we have coming out, some of the more things that we're excited about, as opposed to kind of doing general overview. But if you are interested in general overview, get in touch with us. We can set up a demo. We can kind of go into your specific use cases and dive into that. Before I start too much into that, I'll just provide a little bit about what is Next Float Tower itself and kind of what is it designed to do for. And then we're going to go into a look at some of the new things that we have coming out around community workspaces, around tower actions, so really automating a lot of your workflows, reporting and outputs, and then finally introducing the new tower datasets functionality. So a little bit of background if you're not aware. So Tower is a full stack web application for the management of your Next Float Pipelines. Really developed it after many years of speaking with Next Float users who required a couple of things. They needed a user interface to interact with the Next Float Pipelines. They also wanted API access to make their Next Float Pipelines be able to be services so that you can have other systems talking to them. They also wanted a database of the history of your execution. So you can go back and say, okay, who ran this pipeline? What's the working directory of this? And then you want to dive into some of the statistics of those workflows. The other thing that was really important was the configuration and particularly in configuration of cloud environments. So being able to set up an AWS batch environment, for example, we really wanted to automate that process and enable the people to use the same way they interact with Next Float, but with a long running service which can stand up and do some of these cool things. So Next Float Tower itself, as I say, is a full stack web application. It can be deployed in your own environment. So our customers deploy it in their own AWS, Azure, Google, et cetera, or you can install it on-premise as well. The application itself is made up of a couple of pieces, but the main thing from a user perspective is that you can interact with it in many different ways. And it has the same philosophy of Next Float in the sense that it can be deployed in all these places, but it also connects in with the different computing platforms. So this means that you can deploy it in a given situation or a given location and then have the compute running in a completely different location. Okay, let's step into now some of the few things that we have coming out. And the first of these is the community showcase. And what we've done here is we really wanted to make it very easy for people who are first using Tower to be able to have quick access to some of the key features around it, including pipelines which have been verified. So I'll give you a quick showcase of what this looks like now inside of Tower. So anyone who signs up gets automatically added to this community showcase. And this showcase is what we call a workspace which can contain pipelines themselves which are preconfigured. You can see there's a couple of NFCOR pipelines that we have here and we're adding to these over time. So over the next couple of weeks there'll be a big release where we'll put all the compatible NFCOR pipelines into here and we'll allow people to run them. So this workspace is connected to a computer environment in AWS so you can run in that environment. And it's a kind of nice easy way to be able to get started with these workflows themselves and sort of kick them off. Kind of some criteria around putting these workspaces sorry putting these workflows into this workspace. One of them was to have make sure there's a kind of valid next flow schema file. So for example, if I click on AC care the fact that it has this next flow schema means that all of the inputs can be rendered in this way. This makes it very easy for a user to go down say select a particular option that they want and then kick off and launch that workflow. And that's kind of the main idea of this is to provide a set of validated pipelines which kind of conform to a particular way of running. Obviously things like the profile test which is in a core NFCOR profile that needs to be valid there. I mean a couple of things around how much resources those things take because we want to have something that's just kind of sensible for running in a way like this. So now you can go for example here you'd be able to go jump into that run if I wish to kind of dive into there see that run but also this is a kind of shared space so I can go see other workflows that people have launched dive in and if you're not familiar with power it kind of provides a very easy kind of monitoring single pane of glass into the workflow so you can see the different tasks processes memory etc. And then jump in to that I'm not going to go too much into that today though we want to sort of keep it to keep it to the new stuff for that. So that is the kind of introduction of those of the community workspace that we have. Another thing that I wanted to show you and sort of describe is around the automation of this and we saw last week in Dazella's talk that she described a little bit about how there was a GitHub action which was triggering an execution of a pipeline and that was essentially using the endpoint there. We've started to see some more sophisticated use cases of this and all this can be seen from inside the actions pane so you can see here that we can create an action and actually at the moment can have a couple of triggers one of that could be in this case a GitHub webhook so when I commit to this repository this execution will fire off this pipeline so in this case there'll be a GTK pipeline which will run and this can be very useful for similar to what we saw last week where it's really triggering the execution of a pipeline in some kind of testing mechanism. What's also become really useful is the creation of a webhook here in terms of creating a customized endpoint in the application which will allow you to trigger the pipeline if I hit this endpoint with some given parameters so I can create a pipeline here set up some action choose the compute environment have it all kind of preconfigured and then as soon as I hit that endpoint that pipeline will start off. Now this has been around for a little bit what we're starting to see now though is the ability to have those triggers executed based on different events so a common one that we see with users is setting up for example a lambda function which is able to detect a CSV file for example a sample sheet entering into an S3 bucket and that will then trigger the execution of this pipeline in this way. This is the kind of whole automation you can imagine stuff coming off your sequencer going through a predefined pipeline and then being notified of that at the end and something that we've seen a lot of success with as well. That's really just the first point of that. We wanted to make it a lot easier for you to have any kind of automations here and also be able to automate different parts of the system maybe not just launching pipelines but other aspects of that. And then for that we've started to be released the full open API for Tower itself so as the application grew to multiple users, multiple organizations, etc. We wanted to expand the API as well and make it kind of fully visible to users that have it. This allows you to then interact with it in sort of many different ways which allows us to build an SDK on top of the API which allows us to really get to the point here which allows us to build the command line interface. This is something we've just released in the last week also and this is the Tower CLI it's still an alpha release but I can hopefully show you a little bit about how it works here. You have an interaction very much similar to if you've used Nextflow Run with Tower and you essentially specify a workspace ID basically a token that you're going to be using and then it's going to submit workflows into that area. There's a couple of things here about how you can set this up follow this tutorial as well if you want to set this up. For the purposes of this I'm just going to give you a short quick demo of what it looks like. So if I was here in the command line interface for example I could then interact with my Tower server by maybe I want to list the credentials which are available to me in Tower so if I list them here you can see I have some AWS credentials set up and you can see when I set them up they've got some IDs associated with them and then very much like you've got Nextflow Run we want to allow to have a similar thing which is like Tower Launch and this is kind of a keeping of the philosophy of people who want to interact with the command line so if I say here Tower Launch I can even specify here even just a repository it doesn't even have to be a predefined pipeline it can just be any Nextflow repository here and if it's set to run I can launch this and if I can quickly I can't quickly share my screen fast enough I'm going to jump over to Tower and then hopefully here we'll see inside of this sorry inside of this workspace here you can see that that triggered the execution of that pipeline so this is kind of another quick way to do this we also see use cases where people want to automate the creation of computer environments for example maybe you want a computer environment for every user or parameterize that somehow and that also allows you to do that with the CLI so again one of these things we're excited to see what people build with it and hopefully you can get a lot of value out of that particularly if you're a biofum addition still loves the command line and wants to interact with these things in that way okay next thing is a kind of small thing but it may be quite useful for people if you're often in a situation where you launch a pipeline you maybe have some inputs you launch a pipeline and then you have some outputs sometimes visualizing that information or accessing that information is not necessarily straightforward those outputs could be in some publisher in some history bucket for example and maybe you don't have you don't want to go to your navigate through that or look up the console on AWS to go find that or even go into your shared storage to access it in that way so what we're coming up with is a way to manage outputs essentially be able to view, share results that come off and explore pipeline so what you can do here is be able to define any given outputs I mean a classic example would be something like multi-QC so you would specify multi-QC in the workflow definition that can actually be predefined in the pipeline itself and then from the user once the pipeline is complete you can then visualize those reports inside of here you can go down select PDFs HTML, images, etc and this will kind of keep a centralized place for those results and it's also nice because when you look at those results that are on you can imagine so taking them going back and understanding exactly what the process was, what the workflow was that created that and you can access them in that way the final thing I'm going to talk about which is a little bit related to this and it's sort of a way of also first attempt to going into sort of data management space and this has kind of been a long time coming because it's been a lot of work actually with folks in NFCore around how we structure this and the kind of the schema that we use for this. I'm very pleased though to announce that we now have and we'll have out this week is the next floor tower datasets functionality. What does datasets functionality do? Well, this allows you a way to define typically input datasets for now that can then be run in a pipeline it sounds a little bit trivial but there's kind of a key thing that it does here one is it enables you to keep track of everything that's gone into a workflow itself but also then you can update these keep versions of them and make it much easier for a user to be able to launch a workflow based on some given input so the datasets functionality here you can see I've got an example here I can create a new dataset so let's just call this RNA seek example dataset just call this sample sheet and I'm going to specify description if I want but I can drag and drop a sample sheet in here CSV CSV files for example and when I drop it into here you can see it gets an output and you can see this is the NFCore mega test that we're using so in this case it has got a row as the header so I can specify that now you can see I've got the samples the actual files themselves the strandedness etc and I could if I wanted to then like reupload a new version of this and it would have a version update the really nice thing though is that this follows a schema so if I create this now you can see I have RNA seek example sample sheet imagine that this could also be automated as well so imagine that sample sheet was created from some automation as a launch pad user so as someone who just wants to kick off a pipeline I can have predefined pipelines here now and when I go to run this my input given that knows the schema of the input I can simply select down here and you see I get a drop down of that workflow to run so I can select the workflow that I have and then launch the pipeline seems kind of trivial but you can imagine connecting this thing up with outputs for example then maybe from the launch of a pipeline you would be able to go see the inputs of that the outputs of that and then kind of reuse these things so we can create full ability to reuse and trace the reports that go into this so those are kind of quick overview of everything got maybe if you can think of ways that you can connect those ideas together I think there's a lot of potential for this we're doing if you do want a more detailed demo or dive into the stuff please reach out to us and we'd be happy to walk you through as well so I think it was 14 minutes must be a record yes that must be a record of yeah the right side story that actually kept in time thanks a lot for this great introduction of Tower's new features Evan please if you have any questions just post them into the chat but if nobody has one then I'll have one first it's great this new CLI that you presented I was really impressed by it and can you also provide parameters for example when you when you trigger like tower launch and how does that work yeah exactly you sure can so when you specify the parameters here you can specify them let me just find the example of this so one case of doing this would be to have on the RNA seek sorry on the basic hello and explore pipeline there's a parameter for the time out here so you can see that you can specify the parameters in this way here this is a way on running it in this way so you've got this dash test params field for doing so obviously that you can structure this in a different way another interesting thing is that from the API way you can actually submit sorry through the CLI but you can provide like a JSON object if you want kind of used to dealing with that and it kind of provides a different way of doing it without having to specify exactly each thing oh yeah that sounds that sounds great um are there any other questions so far don't see any other ones in the chat I was thinking if it's possible to upload datasets via the command as well yeah currently it will be available via the API and that's like this kind of big use case we see so you can imagine that you can even automate the whole thing where you're going to be having a dataset which is kind of dropping into the mystery of some sequencer that's going to you know upload to a dataset so you've kind of got the CSV you know all the you know all the we already read so I know your samples are you know all the metadata associated with them and that in itself triggers the execution of a pipeline you can kind of get a full circle here that would create some output which you can then use for something else so yeah that will certainly be available via the API by the command line I think it should be possible as well it just hasn't been integrated because the datasets is still just a little bit behind the CLI in this case yeah thanks a lot yeah this opens a lot of possibilities for automation triggering pipelines okay are there any other questions I'd like to thank you again for this this nice introduction and yeah close the bite-size talk for today