 Okay, so today I'll be presenting to you the journey I kind of undertook to transition from our backbone based history to view based history. And for me this kind of officially started in January, although I had some preparations or starting last year. When I started looking at it, and we'll have three parts I will go through. First, what were the initial findings when I started looking at it, particularly the most important one. And what was the strategy to resolve it, and then followed by a brief demonstration of it. So we looked at when I started looking at the beta one, which I would call the previous implementation of the new history. One of the first things I noticed when doing this profile analysis was that they were rendering delays. And you can see that here this is a process of how our history was rendered basically from start to end when you reload your page basically. And what you can see is that we at first we have data providers which fire pretty early and seem to be efficient. There's not much going on in these providers at on first site. And then there's a huge rendering delay here until we see the view components actually starting to render on the screen. And this was really mysterious. I didn't know exactly what the reasons were and they're potentially different reasons which contributed to it of course. So one early assumption was that it might be just related to timeouts used, but that only partially explained this behavior. I followed a little bit, but there was still a significant chunk left. And then I followed up by testing stopping the time basically on the client locally, but while using galaxy and trying to law histories of different sizes, pretty interestingly just to see if there is a signal some behavior. And what we see here is the number of data sets in these histories loaded, and you can see the time in milliseconds. And really also see that here you have this linear behavior, which was very, yeah, somewhat fascinating but also scary, to be honest. And when I look further into it and narrowed this behavior down. I figured that the absurd operation, which is basically update or insert into the pouch DB database so the pouch DB database is a client site database we used to collect all the information. And then the approach was to entirely rely on the pouch DB database so the data goes into the database and every access or change happens then through the database interface itself. And what I saw in just tests, which I could do systematically locally by just inserting entries data sets basically those are in this sense they are just objects of data sets into the database I saw the same linear behavior and at very low performance. And it took about 70 seconds to get 20,000 data sets into the database. And that's a huge issue, because it shows that at this point regardless what we would do, as long as we use pouch DB will never overcome this challenge will always have a huge delay a huge chunk of lost time. And that new history, regardless of what we do because we are relying on the pouch DB functionality, unless we would dig into this plugin itself, but that's a whole new different type of story. So the solution I came up with there was a strategy to approaches, as I said started last year these early observations was that looking at the initial structure of how our new history was set up was that we have a huge rx js module set, which uses a pouch DB database and sends in and streams out of it, and huge because it was significant chunk of the code. And the first step, the very, very first step from last year was to separate the rx js data providers and the connected database to it from our actual components and here are two types of components. The rx js components are components which are not directly related to the history but do use history data. And this could be the invocations for example where you can very nicely scroll through your data sets so that would that would fall into this category. First we have the view components of the history itself. So this decoupling step was purely shifting directories apart because previously everything was in one directory, and the nature of it is of course that once you have everything one directory. You'll also have these imports and it gets a little chaotic because you can use it from everywhere and it's local anyway this type of idea so this was a purely a pure refactoring of just separating directories literally without changing code. And then after separating the directories I noticed that that there were some dependence some imports which I removed to. Okay, so in the next step and this was the most significant step and the scariest one probably in this transition was to just toss the entire rx js structure model structure and just replace with simple URL providers. And at this stage we removed 12,000 lines or 11 and a half thousand lines of code. So we basically returned to square one. But before I continued on working on the URL providers so just briefly what the URL provider does is instead of using a database or rx js and streaming data and and whatsoever. It just plainly it just does API calls so it doesn't API call and gets data back very useful though because it's easy to measure the time you know exactly. What's going on how many requests did I make and so on and it's very it's not a one liner but close to it to do use your all providers. And in this stage, after I replaced the got rid of the rx js providers I focused on the view history components. And we replaced pretty much all the components. And here again we reduced the code by about 50% for these. And the goal was to get rid of every kind of unnecessary additional features also to align it much more to the previous history to the legacy because the legacy history had actually a functionality which was established and it's much easier to keep that going, instead of trying to reinvent that to at the same time. So it was also that was the last big transition I would say, and then I returned to the view extors, or to our data providers and replace the URL providers with view extors. These are stores, as shown here so that's our current data structure. What we see is that kind of starts off here I mean your components sit on this end they they import the providers and wait for data or request data. The process for the updating, for example, is that we have, we watched the history object at this current stage. And if there is a change in the history object, we asked for all recently changed history items in this in this history change items module, and then we populate by calling actions. The other stores which are associated to it. So that's the update process. And as I said on this side the components sit, and they can request data as they wish from these three stores. And Marius was extremely helpful with designing this by creating an issue where he presented three possible solutions and reading that helped a lot. In order to pick one, the second one. And when I started implementing it, it also fell in place as anticipated in the issue he created, which was very nice. So here's an overall timeline of the entire process there like of course here the pull request IDs for more details is some interesting conversation going on I think also in general on how to approach things like this. And I think the scariest part was when we switched over here in this region in end of or in February beginning of March, where we replace these components replace the virtual scroller. And there were some concerns, because we changed we removed everything and rewrote the rest so there were some concerns, can we actually make sure that we are not getting stuck again with a significant limitation, giving the new structure and we demonstrated basically with this enabling the infinite scrolling as a as kind of I think a prove that this works already better now than what we had before. And then we should take this route, and then followed up with the view x structure which then really enable the usage when we went to the filtering part. We were already people were already comfortable, because the filtering works fast and, and has a new panel, and so on I will demonstrate this little bit. So one other step which was a delay here was to decouple the legacy model, because it turned out that the interface between galaxy and the new histories now in this case, still use the old history. And there was a huge, huge issue, significant delay I would say of over a week to figure out what it is and entirely decouple us from the legacy history, because it could lead to delays of 20 seconds. Just because of events fired in the legacy history which we actually didn't know entirely that this entire process is triggered, even by the new history. So overall, if we look at the current situation, we have 75% less code, and the upserts. So these insert operations, which I now associate with inserting into our store updating or inserting into our stores 250 times faster. So this whole enabled us to have much more control obviously of the code, it's much more readable, there's much less overhead, way less complicated. So people can, and I noticed this to much easier comment on it or understand make additions and we had several cases in the UX team where Dave Ahmed and others also contributed features very useful features. And I think this the reduction and simplicity here combined with the performance we achieved is was very, very helpful. But it was a, yeah, it was a significant process. So next I just would like to show you a brief demo as a video here of the new history so we still have some issues so there was a fresh reload, as you could see. What we want to work on is this flickering here that the items should never disappear. That's probably a throttling thing that's my assumption because it just scrolls too fast through too many at the same time so there is nothing really to render but you can see that you can scroll between these thousands of data sets fast from the top to the bottom, without any slow down, we have infinite scrolling and pagination for the virtual scroll and infinite scrolling for the collections to at every level. So it doesn't matter how large the collection is here you saw the filtering, the filtering actually relies a lot now on this search field. So it's the one source of truth. And you can add different tags into this filtering field. And at the same time you don't need to know what the tags are, because you can click on this arrow button here, and then you get the history filtering panel. And by clicking on it, it will basically populate this search field and tell you what the what the parameter attribute name is here we see the collection builder now. It's also much faster so the collections do not have to wait anymore on anything. They can directly independently of the history start setting sending their operations to the back end. There is no connection between the builder and the history itself as it was before so all the collection builders in the uploader particularly were decoupled. And here we see some basic functionality adding tags, changing the name, and so on. We added I added this storage dashboard. Which I think Dave contributed for easier access to this to this size storage button here. And then we have these buttons here now on the site to show current data sets are active data sets as we call them right now, the deleted data sets and the hidden data sets. So it's the analogy to the old histories is huge. The learning curve should be low. And this is like, pretty much the main part of the of the history revision, it's all in depth right now, all these commits, and hope everyone will test it and get back to us if you notice something. And we also have already some tickets to follow up. We have bulk operations which I haven't shown you Dave contributed that. And we're working on quite a few things in addition during this process. In one transition when I finally activated the history where we did and what we probably should never do again. I mean I understand that I rushed a little bit and there were like a couple. Selenium tests which which ran only on the old history. And I was fine with the idea of quickly replacing those, but I didn't want to delay the the wait any further. And so I fixed those briefly after and while I was working on that I ran into the tours and the tour tests and the tours are actually very, very powerful instrument to create pop ups over the screen and use it as a tutorial. And while I was doing that. It is I replaced our old backbone tour process. And with a with a new plugin, which only focuses on placement of pop ups itself. So it has no effect the new the proper plugin doesn't have no effect on the tour flow. The tour flow is now entirely controlled by us, and it's much easier you have full control, and it's not complicated and I think it's also modular enough and so now we can also do something like play tours. So, besides the fact that it restored all the tours which are the tours we currently have which didn't work. So now you can just play tours you have full control. They're only still only two operations available this is the insert into a text field and clicking something, but we could easily I think consider to add more, and it works very reliably. And yeah, this is a demo of one of the tours running just by itself on the new history I think this could be a very useful tool to introduce the features to help even in some sense was testing. Because all these tours, the way then and implemented it is they are also tested automatically pretty much as part of the test as part of the existing testing framework. And that summarize my talk. Okay, thank you very much some other questions for some marries are you going to handle the questions that probably more technical than I can help with. I mean if you if you have a question just ask, or you can write it in the chat. I said it but I missed it but it sounds like previously one of the main bottlenecks was that couch DB is there and what was the replacement for that I was a little confused on that. Say the last thing again, what what the bottleneck previously was like, well, at least part of it was the couch DB. So what's the new technology that. So now we use with view X. We basically use simple maps, JavaScript maps or dictionary. Yeah, right. So that is feature rich enough that the support style operation. It's actually pretty eternal pretty nice I think because. So now we have one concern was the policy we had these nice select queries, or they seem to be nice. So this select was different criteria like you was like a regular SQL database. And now there was a main feature which we kind of needed. And for the. So now we have these maps, and we can just have a filter plugin which is specialized on filtering by different attributes on the client, and at the same time produce the query to request to send the request to the database. While we filter exactly what we match what we want to match on the client independently with the data we already have. We also submit a request to the database to send us all the data which fulfills the same filtering criteria. And then when this data comes in we just merge it into it. It works pretty well other than that there is no feature we really needed from power to be other than that select and now it's, it's our filtering module which is much more transparent, I think. Awesome, awesome work. There was a question from beer and whether the new history would be the default in the next release, which the answer is yes, and you can, I think you can already try it on test dot galaxy project or search you go there. It'll automatically use a new history. Yeah, I mean I guess one question is where do you see the history going like what are the next things you want to do like I am. I think the transition to you was something long awaited because it also blocked many other projects that wanted, you know, that didn't necessarily involve rewriting the new history but adding smaller things so what are your thoughts on that. I think I have two ideas or potential ideas, I would say, where I would like to see it go I mean one is already in the, in the description, which was suggested that we have this mini maps and add additional ability to visualize which tools were run in what order, I think all the data is available now so it's a purely, you know, the related thing to just start off fresh and create something but the data is is there which is very nice I think that's very interesting for us. And then another thing I was thinking about it would be really nice if the two forms when they populate their data I know their concerns and it's not entirely clear will it be entirely possible because of the data validation we do, but it would be extremely nice if the two phones would not for every select field for every tool input data field, make a full request and get pretty much all the data from the back end because that's probably the single most slowing factor of the UI at this point so if we have histories with 70,000 or 100,000 data sets, which I think this model now covers, then we will still have this delay when we select a tool until the two form appears and the biggest part of it is as I said the data. So if we can expand this or use these stores to populate at least partially the two form, I think the entire UI is going to be much, much faster and we will not have this dependency on the size of the history anymore which would be very nice. Another question is the overall user interface rewrite in view still ongoing, or is it sort of the last. The last big thing. I think the components or or I think I think in general like how, for instance what what portion of galaxy is still using backbone what portion is still using Marco which are the two previous. Okay okay just in general I see. So yeah this is by far the biggest chunk. So I think I would have to go back to our MVC directory and take a look there. With this we, we are pretty strong now on the view part. I don't think there are any huge critical issues left to transition but there of course, even these little classes and modules which we still have will take a time will take some time but overall this is the biggest chunk. We are pretty much. I can inject another question. Do you think galaxy will become a single page application with a top level router at one point or is this always going to be the way it is now. No I think it will become one. I think we made good progress. I'm pretty sure it will become one and that's big goal. I mean it's difficult to make prognosis is on this but hopefully within a year that be, I think a very reasonable goal to be honest, we could probably do it. But still there might be, there might be, and others might know more about it to, to a certain extent to, there might be this that we still have an analysis entrance point and workflow entrance point and the login entrance point but these three or four entrance points but overall it should be particularly analysis we should be a single app. We could hit the half hour mark. We could keep going I see more questions, or we move to the second part I think we're moving to the second part. So, thanks again Sam. Very nice.