 So, hi everyone. Happy Thursday. Thank you for joining us today. Let me just confirm that we're recording. Yes, we are. It's good to be here to discuss Teal and our shiny framework as part of the our adoption series. I'm Maya Malipati. I'm from Prakagya and I'll be hosting the webinar today. A little bit about me and my fun free like in my fun time, or more so free time. I volunteer for the art consortium and get to do things like this, helping out with the our adoption series. I work full time in Prakagya. Many of you here might have already heard of Prakagya due to our heavy involvement in the art community, where a data science consulting firm based in Canada and the US. We help out companies on projects that like anything are data related, but more specifically really enjoy promoting the adoption of art. Very passionate about that. It's to Prakagya that I'm volunteering here today. This event is reported so you can always come back and refer to it and go through parts if you missed anything, or just want to look at it again. I would like to go ahead and introduce our speakers and panelists today. We have two speakers, Chendi Liao and Donnie Onardi. A little bit about them, Chendi is the principal data scientist and product development data sciences department at Roche Canada. She leads the analytical data sciences team supporting early phase oncology programs. She is currently the product owner of TLR package and it's associated family R products, family of our products. Donnie Onardi is also a principal data scientist in the product development data sciences, enabling platform team at Roche and Genentech, part of both. In the last five years he supported the adoption of R and implemented various R solutions. He's currently the engineering team lead, leading the development of the TL framework. We have an analyst here today, Vincent Chen, Pao Ruki, and Ben, and Sevilla. I'm not able to spot Ben, but to tell me if you are, and if I've missed out on adding him as co-host. I apologize for that. I'm going to try that again after the introductions. This is from Roche and product development data science team and is currently the chief product owner for NEST, which is a set of open sourced R packages that generate, deliver and catalog insights for clinical studies in both static and interactive formats. Pao is a principal data scientist and chief engineer at Roche in Switzerland. He co-authored and now became responsible for one of the largest R based products within projects within the Roche product development team, as well as the pharma industry as a whole for clinical reporting and data analytics. Ben is the director of data science at GSK with about 12 plus years of experience in the public, private and social sector. He has a track record of producing high quality advanced analytical work and engaging with both members and major stakeholders such as C-Suite XX. With regards to the structure of the webinar, we'll have the speakers presenting, and after that the floor is open for a Q&A session. Our panelists, Winston, Pao and Ben will help facilitate that. If you have any questions that arise, please feel free to like add them to the chat box, even during the presentation. All the questions will be answered at the end of the presentation. I will let Chendi and Donnie take the stage from here. Thank you. Thank you, Maya, for the introduction. And I guess I'll start by sharing my screen. Can you see the screen? Okay. Thank you. I'll start. First of all, we'd like to thank our consortium for giving us the opportunity to showcase our product. This is a product we've developed within Roshan Genentech over the past few years. It's our shining based framework called TEAL. This framework aims to really unlock the power of interactive data exploration and ultimately accelerate inside generation from our data. So I'll start by giving a background of the TEAL framework itself, how you started, why you started, what issue we aim to resolve with this product. Then I'll provide some examples of how as data scientists, we can use TEAL with different levels of complexity in our daily work. I will then pass on to Donnie to highlight the anatomy and key features of the TEAL framework, followed by a live demo of how you can actually build one yourself. And finally, I'll provide a overview of the various components and extensions of our TEAL framework and how you can actually get your hands dirty with TEAL and additional resources. So I'll start by setting the scene a bit with some kind of self introduction to our group within Roshan Genentech. All of us are from the Product Development Data Sciences Department and essentially the core of our job is to report on clinical trials. And as one of the largest pharma company in the world, Roshan Genentech conducts hundreds of clinical trials across the multitude of indications all over the world at any one time point. So we need to be able to provide a accurate picture of the safety and effectiveness of our product through the data that we gather in these trials. And then be able to share the outcome to the regulators, to health providers, and ultimately to our patients as soon as possible. And the more efficiently we can get these data analyzed and summarized, the quicker we can get drugs to our patient to help them fight diseases. So historically Roshan and the general pharma industry has been relying on commercial softwares and or privately developed tools to conduct these clinical trial recordings. As you may have heard in another closet webinar back in January, Rosh is bear heading a shift towards open source backbone in our next generation of data sciences platform. So one big consideration of this shift is really about the people, the talent base we can tap into when we look towards the future. New graduates out of universities are much more likely to already have skills in these open source tools such as R or Python, then there are commercial softwares. So open source also offers loads of benefits for how we work. First we can get access to the latest technological development more rapidly. Typically, if there is a new statistical method, you will be implemented in our first. So we also want more flexibility in the way we find solutions. We want to be able to switch between programming languages and pick the one that's most efficient tool for a different type of problem that we have. This is much easier to achieve with the open source product because there are no proprietary formats. And last but not least is the ability to really collaborate with external partners. Up until last year, our product, our Teal product and code was developed mainly entirely in-house at Rosh. But now by making Teal open source, we now have the ability to get inputs from people across the entire industry. And then we gain new insights and creative perspectives on where and how our product can be applied to or expanded to. So the exchange of ideas will also lead to better code, better tools in the long run, and subsequently really improve the efficiency in our line of work across the entire industry. And, you know, how ultimately get the drugs faster to our patients for the entire industry. But what I mean by improve efficiency. So we really need to start by examining the way we work traditionally. And now analyzing clinical trial data requires multiple ways of presenting and interacting with our data. As a data scientist sitting on a clinical trial study team, we may need to produce a Kaplan-Meyer plot on the, for example, overall survival of the study patient population per the pre-specified statistical analysis plan or SAP. But at some point when our study team leads and reviews the result, he or she may ask, well, how does the progression-free survival curve looks like in our patients? Do they follow the same trend as OS? So then after a few email exchanges and forwards, another data scientist would have to track down the code, modify that one line of code to switch the endpoint variable from OS to PFS, then regenerate the plot. So then maybe one week later, another question can come up and to ask about how does PFS data look like in a subset of patients who tested positive for a genetic mutation at baseline. And the same progress of tracking down the programmer and the code needs to be repeated. So for scientists interpreting the result, much time is spent on waiting for someone to attend to their request. For us data scientists, we repeat a lot of redundant work in order to deliver on some very small changes. So wouldn't be really, wouldn't be really wonderful if we can have a tool that can do something like this. And to enable our data scientist, non-data scientist colleagues to easily and reproducibly analyze and explore data by themselves. They can switch out analysis endpoint or apply a subsetting of analysis population with just a few clicks of a button. Roshani is a great solution in that you readily offers best of a world, the ability to create web apps and that comes also with very robust sets of a statistical analysis functionalities. But Roshan Genentech is a very large company. So how do we actually get our 600 plus data scientists to start building all these exploratory Shiny apps, training all these people to become some kind of expert Shiny developers would be very costly and time consuming. So instead, we created a framework that really centers around reusable and stackable Shiny modules. And it really abstracts away a lot of the complexities of building a Shiny app. And that allows the app development to focus on actually developing the analysis content itself. So this is our PO framework. It is designed as a easy platform for interactive exploration of the data, regardless of the data type. The framework offers many modularized and standardized building blocks that that's organized into a collections of our packages, each with its specialized functionalities of course. But together they streamline the creation of our Shiny apps that packs many valuable features including dynamic filtering of the input data, code generation for better reproducibility. So there is also a reporting engine for easy cataloging of the results and sets of really purposefully designed and curated analysis modules that provides data summarization and or data visualization. In comparison to traditional Shiny apps, the app developer will will first need to you know gather and line up all the input data together, then depending on the complexity required you may have to write a very long and massive UI slash server script with some interdependent reactivities. So it's a it's like a huge truck with one single cargo hold and you're just really dumping everything into that one or two script. With the two framework, it comes essentially as a frider cargo ship that is already built for you, then everything else comes as modularized components like the shipping containers. So the input data gets its own container. Each container may have a different data structure making the framework really data agnostic. It's flexible because then you can freely choose which analysis module to load in the app and then easily swap out one summary table for another graphic visualization of your choosing. You can also add as many analysis modules as you like. And it's really cumulative that they, they can provide, you know, a 360 degree view of the data that you're exploring. So visualization is also very important for collaboration and crowdsourcing in an open open source development approach. Different developers can work on different modules at the same time without interfering with each other's progress. So really as an app developer, you do not need to worry about how to build UI components or the server logic or reactivity. You can simply use the framework and then choose the already built modules from a catalog then hit plug in and play. At Roche and Genentech, our data scientists have already been using the TO broadly across many areas of our daily work. Within the traditional clinical trial setting, we've been building the TO app as an alternative method to enable quick quality control and the validation of our trial data analysis outputs. We've been using, we've been providing TO apps directly to our stakeholders and clinical scientists who typically may not have direct access to the raw data and allowing them to perform trial monitoring or ad hoc exploratory analysis themselves. Also with the code reproducibility and the reporting capabilities, with also being actively using the apps to generate contents and the materials that goes to support internal decision meetings and especially at a study milestone events, which greatly reduces the turnaround time and the workload on the data science team. It's also one of the most efficient tool for any poll data analysis due to the dynamic filtering feature and the standardized UIs. One can easily stratify or combine data as desired. And then outside of clinical trial setting, TO has been used to explore high dimensional biomarker data types such as RNA-seq data, as well as off-label use of our pharmaceutical product in terms of real-world data. Recently, we've also seen some very creative usage of our TO apps for data quality monitoring and even operation analytics. So, into these areas that we did not even purposely expand into but was nonetheless picked up by the respective teams due to the valuable features of the TO framework that it offers. So, now that we've seen example use cases as data scientists, depending on your our shiny skill levels, you can really flexibly decide on the level of complexity and the interaction you want to have with TO. To start off simple, you can simply be a happy user to just use the app to explore the data with something that's already built for you. But in most cases, as data scientists, you're probably sitting somewhere here because you're going to be the one test with building the app, in which case you will be acting as the TO app developer. In this role, you don't really need to be an expert shiny developer or even an expert art user. You would really only need to have a reasonable understanding of the TO framework. You simply need to pick out from a selection of existing TO modules from our curated catalog and then be able to just insert them into the app framework. Then ultimately be able to just deploy the app onto some server, either on the cloud or maybe through some enterprise servers. Now stepping up a bit, maybe the app you build that doesn't fully met the need of the required analysis. So perhaps you want to, for example add a customized summary table or new format of visualization of your data. In which case the TO module developer then can create a new reusable TO modules using the various building block offered by the TO framework. And then once the modules is ready, the TO app developer can take these developed modules and then insert them into the framework alongside with other modules. So if you have a reasonably good or shiny skills, you can probably perform in both of these roles as both the TO app developer and as the TO modules developer. Or you can have that role split interchangeably with another programmer or programmer on the team. But further we can expand that even more that we can expand the depth of usage and then by by looking at the examples of cross team scenario with skill. With Roshan Genentech each clinical trial will have its own separate study teams. So each study team will be creating its own R Shiny app, and then perhaps add its own customization to enable data exploration within each study. So although each study will go through its own life cycle, nonetheless, there can be a lot of similarities between the studies. For example, they may be testing the same drug or they may be testing in the same disease area. So consequently, that means the data analysis needed are likely to be very similar between these studies so that we can probably design some modules with shared functionality that can be applied to both studies. And then these general and share the module developers then may need more advanced RCO skill sets and to think more towards the generalizability and robustness when designing and implementing these new modules. So at Rosh, we have a small but dedicated team of our developers who can who actively develop and maintain these type of essentially created till modules. You will see some of that in our demo shortly. And then a few examples of these purposefully designed modules that can be used across the different clinical trials. So now you've heard me talk conceptually about all the great things he offers. I will pass the ball to Donnie our teal development team engineering lead to take you on the journey through a teal app to really see it in action. So I'll stop here and then Donnie, why don't you take over. Thanks so much, Andy. That's a that's a great setup to to my part. Before I start just want to say thank you again for a sponsor for allowing us to just to be here and to present this amazing product that we have. So let me start sharing my screen. You see, you see the whole presentation or do you just you see my screen okay. Okay, we can see the presentation. Okay. Yes. So, before I start doing my live demo, just want to tell everybody that our product teal is available on the public GitHub. But we do recommend you set up your personal access token first. The reason we do this is because we're using a package called remotes, and we use GitHub API to install the dependency though. So the access token will will take care of the rate limiting. Once you do that you can simply run remote install GitHub, and we're going to use the release reference. So this will install the stable version of our products. All right, so we do get. You did get a taste of what the apps looks like and we will go into that a little bit more but before I start I thought it'd be a good idea to just kind of explain to you the anatomy of what makes a teal app. So, you know, on the top we have, we have the header. On the top we have the header this is just to name. Some more context of your or futile application. And then we have this section, the menu with tabs. These are essentially different modules or analysis that you're going to provide for your for the user. So each of this tab will have different analysis. We have the reporter. This is a functionality that we will see in a minute here. This is to enable user to create a report when you're exploring your data and you saw some insights multiple insight that you want to pull together. And coding. These are think of it like a control, a settings that controls the output of your visualization. So that's what encodings are. In short, we will see this on the demo but this is focusing on reproducibility and as you know reproducibility is huge context in pharma industry. If you're doing your analysis, you need to be able to reproduce it. So we look into this functionality but that's the goal of this is allowing our users who are using the teal app to be able to reproduce the analysis that they see. So if you do have some debugging functionality here this this part is a debug info button. It just helps you navigate your experience when developing teal. If you run into some some issue some block. This could be the first thing you want to go to just going to see if all your code running for the teal app running correctly. Of course, then we have this middle part. This is for your main output and visualization. We have the filter panel so the filter panel allows the teal user to subset their data interactively with the module. Then finally we have the footer part this is additional information if you need to add anything more. Then we have a filter panel toggling. So if you click that hamburger button it will just hide the filter panel and makes your output and module a little bit bigger. Lastly we have this three buttons. This is just allowing even further interaction with your with your image we got the download button for for users to just have a local copy. A better view the middle one is if you click that it'll show in a model and then you can adjust the width and height and dimension is to adjust it directly here on the teal app. So let's go to the more to the live demo in this we're going to explore the teal features together and we're going to try to create a teal app and so everybody can see what that looks like. So let me. I have a feeling I'm not sharing my whole screen am I so I'm going to share my whole screen this time. Here we go. Yes, we can see the whole screen. Maybe I need to close this. Okay. So first off, I'm going to. All right, so before I start, I just wanted to share to everybody in for the purpose of this demo. I'm using positive cloud and then this is an area that our team has been using the great thing about this is that we can share the workspace. So I did create a a new content called till playground. And at the end of this presentation we will be sharing the link to access the workspace that I created for everybody attending this presentation, so that you can just come in. You'll see my code. The same code that you see in a minute here, and we can just try start playing with teal. You don't need to install anything local you don't need to pollute your local environment so this is a great, a great feature for you to just jump in right away and play around. So let me open up the playground real quick. And was wait till it's loading. Great. All right, so I got a couple files here actually I'm going to go through and we're going to go through one by one. I'll try to go somewhat slow. I do I do want to market we have an R package call that the till gallery till gallery is an R package they're going to showcase the whole some of our till product. And we do have several example apps that that user can see based on the category of the analysis we have some, you know, efficacy analysis which we're going to showcase here and then we have several other more. If you're interested, I think I have a tab open for for till gallery. Yep, there it is. All of our product is under inside engineering until gallery. And then other inside engineering. All right, so let's get this start. So I'm going to run a lot this app. And before is there, I'm not familiar with my is there a way to hide this, the zoom panel at the top. I don't know if you know this is kind of get get in the way of me picking different tab. Do you know my what I tend to do is just pull down the window a little help myself but far than that. I'll do the same then. That's fine. All right. Okay, great. Now we're seeing, we're seeing an example after year. As you as I mentioned before this different module on the menu tabs, it's displaying different thing based on the analysis that we want to do quickly. What I want to say something about this app is that we're using a lot of module from till modules clinical. And till modules clinical is one of our many to several till modules are packages that we design specific specific for user to be able to just plug and play this different till modules. So, in this one in particular till modules clinical is a collection of curated till modules designed to analyze status data to generate status status output center standard analysis output. So we're going to play around with this several of these examples. Let's start with demographic then the first thing I want to show you is how easy to apply filters. You saw it a little bit on the video that Jenny show. So say if I want to just want to see the female subjects, then I can do that. If I pick a country say I just want to see outside of the United States, I can do that too. Right. So you see specific module you see we only be using one data set but will will shift into another module and you can see that you can bring more than one data set for for specific module. The filter part filter summary you can see here a quick summary of whatever you have left after you filter your data set as opposed to the total observation, you can see the number of subjects affected. So you can follow this, making the module a little bit bigger, bringing them back in. If you don't want the filter, you can simply click the X button here that will remove the filter, or you can click at the data set level, or you can click at the very top to remove all the filters. And as you see, as I interact with the filter panel, you see right away the output in the middle it got, it just react to my to my action so it's very interactive, very, very fast. Let's do, let's do another one, let's do no one will do can plot. So let's load up a little bit. That looks a little bit too big for me so let's make the height a little bit smaller. Yep, that looks much better. As I mentioned before now you see more than one data set to play around with your module. We can do the same similar idea maybe we, we want to see male population this time suppose a female. And then maybe we add a more parameter code, we don't want to see a specific one maybe. Let's do that. Right, so great. Another thing that I want to showcase, and I'll lead this carefully because I don't, I'm going to show the reporter later, but the show our code. Again, this is for the purpose of reproducibility. This is big in pharma industry, and that's what this button is for. If you like what you see here after you apply all your filters you like what you see here. You can simply click to show our code button. And it will give you a snapshot of the code that you need to reproduce the output. So let me demonstrate that to you, I'll copy this clipboard. This is my own. This is a local R studio in my computer. So we'll run this real quick. And because this is a plot. It's going to show on my, on my plot. And if I zoom it in. Amazing. It's the exact same thing so if you look at the median is 1517. If I come back here. 1517. Yeah, that's the exact same thing. So I can reproduce my analysis based on what I see here until now. Okay, that's great. Another thing that I want to showcase is the idea of, if you see multiple inside inside you are exploring and you see couple things that you like. One thing you could copy and paste our code and download image one by one but I'll be too much so we created another feature called the reporter functionality so this is an able user to create a report based on the multiple analysis that they see. And let me demonstrate that to you as well so this is the next level after the show our code. Let's say we add one more filter. Let's say, again, for this purpose, I will, maybe I'll remove USA just want to see out, you know, the rest of the Europe. There's of the world. So I said, click that I'm going to click this plus button. I'm going to say, you know, this is game plot a male and no USA, maybe like that. So that's being added to my report, but I haven't been. I'm not done yet. I still want to explore more things. Let's me go to time to event table. And let's assume this time I don't want any filters I just, I just want to see what it looks like without any, any filters. And let's assume that this is what maybe I'll change the encodings. Maybe I want to see progression free survival, for example. But this is what I like. And I'll click that button again. I'll say progression free survival. And I'll just say this is default. Then I'm going to add the card. Awesome. Now, if the module has reported functionality. Then you will have to report previewer by default. And you see here I have two cards and because that's the only two analysis that I want to build my reporting. So you can expand one of this card, and you can see the, the analysis output tables in this case, and then you saw some of my comments. And finally you see the same our code the one that you let us show earlier, it kind of similar features would show our code but yeah. So we have the reproduced reproducibility aspect we have the tables we are common. And that's the same thing with the came plot as well. But the difference is we apply filters on this one. So you'll see the filters on the top. But the rest of the same we get the same comments, we have our code as well for reproducibility. Now, before we download the report, we do have several more parameters for us to add. And it's up to the user, you can put your own name as the author, you can title, change the title of the report for the date. We do have an option of four different document type for users to download. But for the sake of this representative presentation, I'll stick with HTML. Now, when you download the report, you'll get a zip file. Within the zip file, let me open that you get a lot of assets to make it better. Yeah, you get a lot of assets to for for your producer report so we have the png file so this is pretty much the plot that you saw earlier. So any visualization will become png and then all the tables and if an up summary table those objects become an RDS file. Then we have the HTML file, which is the report itself. So this is the final product of your report. And then we have the RMD file. So the RMD file is also provided to you just in case you want to annotate or add more information to your report. Let me double click that again. Here it is. So if you want to add more annotation to your report and just for the sake of presentation, I'll just add more heading here, new item, and then I'll just say testing. Then you can quickly knit this or mark down again. And it will use the same Assets the one I was provided to you, and you'll get a new report with your new adaptation in there. So this has been very helpful for our internal study team and I hope could be helpful for you guys as well. All right. So let me come back here again. Now if you don't want if you don't want this and you change your mind you want to reset report. We can cancel it and reset it. All right. Perfect. So that's the first part of my presentation. Now let's go to the second part. So for my next couple, you know, minutes here I'm going to show you how to create create a till out and here's the very few lines of code to create a very basic till out. And, and I will I talk you I do want to pull out a very important function from our till framework which is the unit function so let me bring that in, you'll see the documentation here. Now while I speak. Now, this is an example straight from our vineyards by the way. So I do also give the link to the to the vineyards on the top here in case you guys interested. So the unit function is very important and that's what makes till till up. It has several arguments, but the required one is data and modules. Data to specify the data that you're going to be using for the till up and module to specify the till modules or analysis that your till up will be using. Yeah, we have some header here is always a good practice to provide some some context for your till up. Now, the module that we're using right now is called the example module doesn't require any parameters so let's see what that looks like. This is it. Now, right off the bat you see the filter panel provided to you by default, no additional programming needed is just given to you so that's what's great about till all this upsetting activities, no, no extra programming needed. We just need to know what is the module. What is your data. What is the module to display the observation of the data set that we provided to the till up and as always we use almost below the data frames iris and empty cars. So as I switch between these data set, you can see the observation changes right. You can still filter this. You can still, you know, at that's too low. You can still filter this or you can. And still play around and explore your data and as a switch, you can see the empty cars has no similar equal for an iris is between these ranges. Let's move on to the next one and I do talk about this a little bit. I do touch this area a little bit about till module clinicals. And again, this is not the only our package that we have focusing only on till modules. Overall, we have about 50 plus till modules readily available for users to be used for your analysis. So till modules is just something that we use a lot internally and it's also a good example to show the more relevant for your for your study if you're in the pharma industry. I do want to highlight another our package called SCDA and this is what we're going to use for our example today. This is another our package for providing us with the sample data. So SCDA stands for synthetic cities archive data. So this is an our package contains sample data in Adam format so it's, yeah, it's just sample data but it was designed in Adam format. So online and 10 you see I'm pulling two data set here. Nine is for demographic subject level, and then a the 10 for adverse events. So ADSL and ADSE. Now look at the code to make the till app. Same idea, you, you, you run the init function, you provide the data that you're that your till app needs. But this time we're using a different module. So let me let me pull that that modules documentation so you guys can see as I speak. Now, the module does require more arguments. But that's why we have a documentation for there's some several, you know, several required arguments some are optional or have some default. But we have documentation on that but and once fulfilled is just as easy as just running this app again. So let me run this real quick and we'll see what that looks like. There you go. We still got the filter panel again that's that's out of the box, you always get the filter panel, but now this time you get more functionality right we have reporter now we have show our code now we have the debug info now. And that's what's great about this modules right if you're a module developer. It's great for you to design what kind of features that you want to put in your module and that's what's great about till is it very flexible if you don't think you need specific feature, then you don't need to add it. But for for this module we add all this functionality. Again, if if you have the reporter functionality, the preview or just by default there. So no extra programming needed. As soon as till framework notice you have a reporter feature for your module. We, we provide this extra module for you all the functionality here as well. That's great. Now, we, we do have more than 50 modules like I said earlier, that's ready for users, but there's going to be always once, you know, specific case where you just have to make your custom module. And that's what the third code going to be like and the, the most important part of us this is if you need to make a module, a till module, if you need to make a till module, you need to use a function from our till framework called module. Again, I'm going to pull this what I speak so you guys can take a little bit. Now, why is this important because till, because the module function, it embed shiny modules in our till application. So really that our till modules is actually shiny modules. If you're not familiar with shiny module is is a technique in shiny app. When working with a big and complex app by addressing namespacing in your UI and in your server logic, right. So, for this module function, we have several arguments here. The one that, you know, you always want to pay attention is the label server and UI so the label will, it will be become the name of your module so that's what it shows on your till app as you add your module to it. Then you have a server and UI. Again, if you're familiar with the shiny module is the same syntax you wrap it in module server and and then in UI you're going to name space your ID. Right. Um, so this is a very simple module. It does not have any in, like in coding panel. I'm just specifying and you why I've specified some area for my table output. And I pretty much hard code this. Hard code the data to become an Irish data set. All right, so then once I make the module, and then the last thing again is we need to run the init function, specify what the data is, and then call the module. I, we name this my module. So let's run this and see what this looks like. Great. So very simple module knowing coding. No sure code button. You know, but you can see the observation of Irish data set. You can, you can still filter it. It will react with it. And you can see how the number of motivation window down as I, as I just select several categories. Great. So that's, I'm, there's going to be a lot more than this, of course, but I just, I just want to give you guys a little bit of a taste, but what it looks like to create a module. Now, if we recall, the first example that we have on here, I'm using example module, right. And let's look at what the example module looks like it again it's not a very long script. Because we have a lot of helper function in our package to kind of help our user to create their modules. I want to drive your attention to this line 50 here. So if you like the way are all the example apps module, you know, the one with encoding on the left hand side output in the middle. You don't have to create the, the UI tack for that the fluid row fluid columns will provide that for you, you just need to call a function and provide some values to the argument, and we'll take care of it the rest. So here this one spreading saying that what do you want to see on the encoding area. What do you want to see on the output area. So we'll take care of that for you. I guess the main, the main topic is, you know, don't worry if you need to create your own module. We do have, you know, some tools to help you out to help you finish that task. Okay, I hope this has been helpful. I know I really enjoyed showing this to you guys again all of this. We will share the link to the deposit cloud workspace for everybody to just jump in, you will see my code right away you can run it again. Or if you look into our vignette you start reading around and you just want to run some example codes come right here. Don't you know you don't have to go through the installation process. I already did it for you. So I hope this has been helpful. And with this I'll turn it over to Chendi. Thanks, Tony. And then I'll start resharing my screen again. Let me move the zoom thing away. Okay, can everyone see my screen. I think so right. Yep. Okay. Yeah. Yeah, so, yeah, hopefully the demo has wet your appetite about, you know, the vast potential of incorporating teal in your daily work as the data scientist so we really encourage you to get your hands dirty by jumping into the teal sandbox playground we've prepared. So there are two ways as Tony was showing with the positive cloud link, which I can actually show you right now as well. So in when you click on the link you should just be asked to log in even if we don't have account you can simply create one on the spot. So for me I have, I do have one account open so I'm going to log in through my email address. And you should be forward then to be asked to be joining the space so once you click yes. Okay, so now if you click back into the teal in playground we have prepared and then just start the session. So while it's so waiting let's maybe switch back to our presentation. So for the more tech savvy audience that you can also download the public Docker image that we have put up in our GitHub repository. So for both instances, all the required packages and pendency have already been installed as down you mentioned, and really ready for you to play around with. So in addition to the our environment container, you can also access a lot of our example apps through either the teal package called teal gallery, or through a public URL on the shiny apps. So we can, I was meant to click on this one, we can also take a look. And while it's loading let's switch back to the positive cloud, which should be firing up very soon. Yeah, you can see this is exactly our studio and then if you just click on the file side there is a folder called teal weapon or 2023, where you can find all the, the script the dot Donnie just used. Okay, so I'm going to close this and then move back to the example app which should now be almost loaded. Yep. So this is one of the example app but there are quite a few that you can browse through. Essentially, then this is going to be the exact the same app that you have seen Donnie demonstrating. Okay, so we can simply browse through all the different visualizations that we have provided. These are the, you know, the sets of 50 something plus centrally curated the modules that we have been developing and maintaining centrally. Okay, so I'm going to switch back to the presentation. So by now you, you've probably, you've heard about all the modularizing concept of our teal framework, and you've seen Donnie calling various teal packages during the demo so I will start wrapping up the presentation by showing a summarizing product map of our teal and family packages. So we have several foundational component R packages for scalability and the easier maintenance. So we have things like teal slides, teal data, teal transform, teal logger and teal code. Each obviously have its own specialized purpose as the name may suggest, but together with teal, it creates the R Shiny app framework. There are also packages like teal widgets and teal reporter, teal gallery that again together with the framework enables the teal modules development. And as mentioned previously, we have several curated and centrally maintained teal module packages that offers a large catalog of data analysis modules. So to start with, we have the teal modules general that offers, you know, just general visualization for exploration of any data types. We have the teal modules clinical, which were designed to analyze, see this data specifically for clinical trial reporting. And the modules package itself provides the UI and interactive with teal logic, but you also realize on several separate R packages that performs the actual calculation or tabulation that ultimately produce the statistic that you see as output within the Shiny module. So for teal modules clinical, it's that the calculation package is called term. Next we have teal Osprey and coupled with Osprey calculation package. These are modules specifically designed to analyze early phase clinical trial data that focuses on safety results. The teal go shock are modules for analyzing and visualizing laboratory and generic biomarker data. And finally, we have also the teal modules Hermes and the Hermes package itself for standardized approach standardized processing and the analysis of high dimensional RNA seek data that require some more specialized handling and statistical approaches. So this is just a more worthy description of what I just talked about. You know, just time I will not repeat the details, but make the text available for your reference when you when we share out the slide. And again, all these packages has been open source it's alive on throughout GitHub and through cram. Before we conclude the way you want to point you to a few places where you can find additional resources teal product is part of farmer verse, and even if you don't, you don't know what it is, you may have deduced to found the name. It's actually a stack of open source or packages, very similar to tidy verse, but specifically focus on enabling clinical trial reporting for the pharmaceutical industry. So, you know, to get get getting in touch with us through GitHub directly we also have a teal channel open under the farmer verse workspace in Slack, where you can hop in share ideas, find support and maybe even get connected directly to sound for our development team members. Lastly, Russian genetic is also actively preparing for a Coursera syllabus on hands on clinical trial reporting using our aim to be released in Q3 of this year. You will be able to learn more about the application of various our products in the clinical trial reporting space. And of course as mentioned the very beginning now being a open source product we really encourage and look forward to collaborations that can enhance and develop this framework further. As individual data scientists you can contribute directly on GitHub or through Slack channels, but then if your organization wanting to adopt or co-develop teal please reach out to our collaborations lead Thomas through the highlighted email. And last but not least, we'd like to say huge thank you to all of our past and present developers and contributors to the teal framework. We wouldn't have such an amazing product with all of your contribution and dedication so and we really look forward to add more circle icons to the slide in the future. Yeah, so with that thank you very much for your attention and we are very happy to take questions now. Should we just pick through the Q&A or anyone wanting to come on, come into the chat and speak directly. I wonder if, if Maya you need to enable if somebody, you know, you need to enable or unmute them. I don't know if it's because in the beginning we can. All right, so maybe that we can go through the question list. I don't know if there's a way to present that actually. So let's, I'm going to scroll to the very first one I see there are quite a few already answered so. This is a question from Abby. How flexible is this framework that you have set up. Could I move around the tabs to be on the sidebar instead of sidebar instead for examples. Yeah, so, so right now, you can't you talk I'm assuming you're talking about the module you want to move the module on the sidebar as opposed to, you know, a tab set panel. I don't know if you do it and shiny. We don't have that flexibility yet. But you know, maybe something we can talk about and think about but for now. The modules is within the menu tabs. Okay, I'll mark this as done. And then we have from March, Marchion. Have you considered other survival analysis model besides Cox model. There's a huge family of AFT models that has been useful out there in the market, those are parameter parametric models. So I think for this, I guess in the current clinical trial space Cox regression is still sort of the, the go to model that we use. But I think recently we have been considering opening up, expand the further the statistical analysis models that we're incorporating into our modules. If you're interested you could come join us and help us design a new model for his module, for example, that uses AFT model instead of Cox regression. Okay, I'm going to mark this one that's done. And then next question also from Marchion, anyway to interactively trim X axis of K and plot those statistics are after 2000 days might not have much sense due to a small sample that is left. I think this is possible. I don't think this is something we've shown in the demo but it's possible to do this. Next one. Let's see. There's a lot of comments so I'm going to see through this and see if we can find a good question. A lot of people asking about the URL to the workspace and also slack so I believe Vincent has been helping posting them through the chat so hopefully everybody cannot have access now. Okay, and a lot of words of encouragement that I see. Okay, so next question I see is from Sam Parma. Have you considered creating a user manual or book on developing with two. Not a bad idea. On our GitHub web page we already have a, I would say quite robust user web menu that incorporates vignettes and the function parameterization of course. I think internally we have been preparing something so maybe that is something we can look forward to put on to our roadmap and share that publicly with the external audiences as well. Andy before you move forward, I think I'm able to allow people who raise their hand if they want to speak. So let me, I see there's a hand being raised so I'm going to Maya is okay I'm going to click allow Mason to talk. Mason, can you unmute yourself and maybe we can hear you. Hello. Yes, we can hear you. Hey, thank you for the very nice presentation I was just wondering because like I'm a PhD student like coming from an academic background. I was thinking like many of these modules actually can help me in my day to day life like presenting data for example my, and I was wondering like if you actually plan to expand into like those kind of like academic setting as well like a lot of modules that can help like day to day like science of data analysis. Outside pharma you mean. Yes. Yeah, I think. So you saw the first example where I just plug it in Irish and empty cars right I mean that's not. I mean, that's far away from so there is some some some opportunities there and and remember when I talk about the different till modules that we have for our user to plug and play. We actually do have an hour package focusing on all two modules just don't generic analysis. You know, the basic statistics summary. I think we have about a handful of those. Yeah, try to take a look into that if you if you want the name of our packages till modules general so these are more like the generic usage of data that's not, that's not cities that's not, you know, or killer trial oriented. And if you think of there's a new module that could have been useful from, you know, generic perspective, then we would welcome you to kind of join us and try to see we can do this together. Thank you. And one more question I was wondering like, if you actually have plan to develop like the adding the modules can also be like interactive. So it can like just drag different modules and in the in a shiny app. Yeah, there's some actually in some of our and some of our encoding we sorry some of them modules within the encoding. We have this is this draggable, you know, the shiny just draggable buttons that you can you can actively drag categories on and off. Maybe, yeah, if that's if there's a use case for that, if we think there's a there's a desire to do more in the module around that area or or any aspect of it to have. Let's, let's talk about it. It's nice to see it but you know, it'll be great. We can get more detail of what's the expectation would be. Thank you. Let me also jump into the topic. Can you hear me here. All right, so. Rosch, let's say Rosch or our let's say initiative farmer versus initiative is aimed for the collaboration between the pharma companies as well as outside into academia. So if you have any place to or any ideas how we wanted to collaborate with us please reach out to us and we are we will be more than happy to to join our efforts into that and and co create things together. Thanks bubble. Yeah, feel free to raise the hand if you have any question and it'll be great we can hear more interaction from but I do see there's a lot of questions open questions. There's the one that caught my eye is about can the input data be except except PT file. I think. So, the data that comes in until it has to be on on some object in our. So any any data reading that you want to do probably have to do outside but we do have connectors data set connectors that we can use to enable that but I guess in short for now you have to convert that into an object first before you bring it into till data. I don't know probably if I get that right. There's. Yeah, definitely so everything like as you as you can convert it and load it into our then you can load it into to like that that's a very simple rule. So as long as you can load it into our, you're free to go, I would say. And then next we have Eric racing him. Let me allow him. Can you hear me. Yes. All right well first excellent webinar I've been a big supporter of to ever since I first heard the idea back at the first our pharma conference so it's been great to see it's life take shape here. I don't verbalize my question but to is obviously got a lot under the hood so to speak so when you think about collaboration with external life science companies or other areas. Are there areas within the to landscape or ecosystem that could use the most, maybe eyes or assistance with development as far as making this hopefully a very ready capability for many pharma companies to do clinical explorations of data. I can, I can try to take that I think, if, if, if you're external contributor, and wants to the, I think the area that we could be need help on based on the level I guess, I think the till modules if you, if you can try to till module and see how that can be useful in your use case. If there, did we miss something or things that we need to add more in our encoding and how to control the output that could be one area. Then the next level will be the framework, you know, when we talk about, you know, this filter panel this reproducibility. This is a lot deeper than just the till modules then. If you're, if you're more or, you know, hardcore developer and interested in this kind of area what's happening in shiny in shiny modules. And, you know, we do handle a lot of our stuff using up object oriented. So, that could be the next level but I think the more relevant for everybody to help out is just play around with our till modules if you have new ideas if you think of another way to produce the calculation, I think somebody touch about this a little bit that's great I mean that's maybe that's an area that we can take a look at really improving the type of analysis that we're doing within the till modules framework. Yeah, I don't know if you have any other opinion or or Vincent. No, I think that was a great answer. Yeah. Okay. Great. Thank you very much. Yeah. Thank you. Next I have fun, Fanny Kumar. Yes, thank you. Nice presentation. You can hear me well right. We can hear you. Nice presentation and I've been hearing about and I saw your presentations earlier too and it's great improvement over the years and my question I was the expediting this comes from, you know, like the medical writers are all those people I like. I come from the SAS forward so I was just wondering like we can use this one how easily to from that world to this far, because it produces the code and it's such a nice nice thing. My question to you is that can we use something from the deal to patient narratives dynamically producing it I'm trying to think from that angle. Yeah, you have everything information right there you know like we want to just grab grab an object from the tables and then get it into in a word document you know something like you know what I'm saying you know basically it produces a standalone documents I can capture them in between and use it as a line operator I mean something that can be produced as a word by patient or buy buy something of interest. So I think your question has two aspect to it so first is conversion from SAS to our as mentioned at the very beginning of our presentation we have you know 600 plus data scientists, many of them actually are also very expert SAS programmers have no idea what they are so it does require a bit of training of course and then initiative from the user size to learn, but we actually found that process to be quite easy and smooth because of the, a lot of that complex extraction abstraction of the process has been already handled by the framework right so it's a matter of learning the, the parameterization within few function and almost a plug in play. So, we have now these SAS old SAS users, they are now also kind of shifting more and more towards our programming and they have been excellent in building up to modules to apps for their respective studies. And then your second question is related to, I guess there's a presentation in terms of patient narratives data. I think you can find some of what you're looking for perhaps in the teal Osprey or the Osprey package. The Osprey package is geared design specifically towards early phase, early phase development clinical trials. So in early phase you tend to look at a lot of things by patient level right to monitoring to make sure the patient safety is looking good, whereas the teal clinical, sorry, teal modules clinical are probably towards more summarization so looking at like a steady population for late phase studies. So yeah I think you can try to look towards the Osprey package for some, some good starting point. There are quite a few visualizations and, and also I guess like kind of just listening to show various components of data from one patient at a time. Good. Thank you. Okay. I think we can have Andrew Andrew who raised their hands. If you want to talk. Yes. Thank you. I just is kind of related to Eric's question but I just want to volunteer myself, if any as an external collaborator if anybody's external and interested in diving into teal feel free to reach out to me, because I am an external team with the team right now. I would love to help you out if I can. Thanks Andrew. Yes, Andrew has been our, our part of our till journey has been great, great contribution from him. And we also have Agnes, who wanted to ask a question. If you want to come out. Or, I think I see another name Amy young. We allow you guys to speak you guys have to unmute, I guess. Yeah, I hear something though. Yeah, thank you for the presentation. No questions from my side. Oh, okay. Okay, maybe we can take one more question. And then if we don't have live one from the audience. Maybe we can pick one last one from the Q&A. Sure. We have one more function way. Can you hear me? Yeah, we can hear you. Yeah, okay. Thank you. Yeah, great presentation. My question is, in order to develop a new teal module, we will need to learn our table and turn right or can we use other package to develop a module. Maybe you, I guess I can start and then maybe then you can add to it. If you want to create a table. I guess the, the art tables is maybe the easiest way to start because we already have a lot of examples that you can, you know, follow through when building the module. I don't think it only works with our tables, right? So things like, for example, a visualization, a plot or something, you can choose to use grid or use ggplot or anything of your liking. So all that is really flexible. You really just need to provide the content that generates the statistics, right? And in terms of the formatting presentation, it's, it's rapidly HTML. So as long as it looks good to you, I think it works. Yeah, perfect. That was what I was about to say too. You could use ggplot if you don't want to use turn on our table. It's up to you how you want, what engine that you want to use to power your modules, right? But then till we'll take that and make it a, you know, UI server element before we pass it on to the shiny framework. Good question. Yeah, thank you very much. Thank you. Okay. I think we are at our scheduled one and a half hour, so we're going to close the call now. But please, everyone who has questions, reach out to us through Slack channel through GitHub. Yeah, we're really looking to hear more feedbacks and looking forward to collaborating in the future. So I'll just ask back to Maya to close. Thank you, Chandi. Thank you all for your questions. You really appreciate you taking the time and to tune in to your curiosity on teal. Thank you, Chandi and Donnie and Winston and Powell for fielding the questions, the Q&A chat throughout the webinar. I think there's not a person. So we will post the link to the webinar on the website. I posted like the link that's there for the previous recordings. If you don't see it right now, you can just Google our consortium webinars and you'll see it there. That's it. Thank you. I hope you have a wonderful Thursday. Thanks everybody. Have a great day.