 Okay. Thank you so much to everyone who's joined us. My name is Shalmyth. I am based in Nairobi. I am one of the Africa R, part of the Africa R leadership team, and we are so grateful that you've joined us today. We are facilitating this tutorial as Africa R. So if you've been following the user tutorials or workshops, you'll notice that different groups have been facilitating different tutorials. And we are so glad that North Africa will be facilitating this tutorial and it involves Egypt R, some friends from Algeria and our ladies algeas. And again, we are grateful for this opportunity. We are going to be speaking or rather we are going to be hearing about Periscope and Canvas Express, creating an enterprise big data visualization. The Periscope R package is a toolkit built specifically to address the need of developers to produce enterprise grade shiny application. And we are going to be hearing from a team from Aggregate Genius and I'll pass the button over to Dr. Coney who will be leading the team for further introductions. Thank you. Thank you so much, Shalmyth, for having us here. We really appreciate it. I'll just start with the housekeeping items and then we can move on. I just want to remind everybody that there will be a large number of people on the phone lines so keep yourself muted unless you're asking a question directly. We have a whole team of people that are helping us with this and I really appreciate all their help. So Amia is the person to direct technical questions and any questions you have during the session that maybe you would want to have answered and we will try to answer as many of them live as possible. And I am not directly monitoring questions or I wouldn't actually get the presentation done. So please send your questions to Amia. We do have a closed captioner here thanks to use R. And so if you would like to view the captions, you can just turn them on in Zoom with that closed caption button. And just as a reminder, the session is being recorded and will be made available publicly along with the slides. It will be edited so that the live coding pieces are not a part of the recording that is distributed. So if that makes you more comfortable interacting during the live piece, you'll feel free, but you will have this for your reference. And with that said, I'll actually introduce myself. So my name is Connie. I am the CEO of Aggregate Genius. We are a small, completely remote distributed company on the west coast of Canada. And my background is as a computational chemist originally. So I've moved into the R world and we run a small company that provides enterprise grade shiny services as well as full stack analytics development. So we have a couple of our colleagues here on the call today to help us out, Todd Brett, and, and Kailin. So they are also helping to field questions behind the scenes and I appreciate them being here. And again, of course, I appreciate the North Africa team as well for sponsoring this and supporting us. So we'll get started right away. I have a lot of content, but not a lot of slides. So you'll be thrilled, hopefully that most of this is interactive. But we are going to during this session create an enterprise grade big data visualization application in a few hours. So luckily it won't take a whole day. And this has been sponsored by use our 2020 and was going to be a live workshop so although I can't meet you all in person as was intended due to COVID I do hope you all are well. And I look forward to interacting with you through our questions in our session. So with that, I will get started. So I wanted to just touch briefly on the problem domain so enterprise grade shiny application so so why a shiny application to start with. Really it's an invaluable resource, in addition to charts and static reports on analysis and data sets right world is becoming more and more visual. And it's a really nice way to see what you've done when you're doing an analysis and to let your users actually interact with it. From an enterprise perspective, what does that mean. So, you know, when you're in a big company, there's competing interests stakeholders analysts the businesses interests and they want to make everybody happy, they need to keep their employees happy they need to keep their stakeholders happy, and they still need to make money. There's some kind of some issues when you take take something like a shiny application and you bring it into an enterprise. So when you're in an enterprise and a big company you have to have applications that if they're going to be used by a large number of users or your customers, for instance, let's say they're customer facing. They have to be scalable they've got to be perform it they have to be safe. They need to be consistent. This is your face to your customer to your stakeholders to your shareholders. And so they need to be professional. And this is kind of problematic because you know we're used to applications developed in the world by developers by people with computer science degrees with user interface degrees with human user interaction degrees that do studies and know how to build applications that meet a lot of these enterprise needs, but shiny applications are developed by analysts, largely people focused on science and analytics and insights and results. We want to get those numbers out and the charts and show people are the cool stuff we've done with all of the great algorithms we have and we can do that with the shiny app. But, you know, and this works fairly well and we can put some really great applications together but we also have another problem that's kind of crept up along the same time and that is that data sets get bigger and bigger, which is fantastic. We have more and more data. But we're relying more and more on visualizations. And so not just applications but visualizations and big visualization and big data. So we're going to kind of address both of those issues today, to some extent right now I'm not providing you with a panacea to all your problems but hopefully I can get you started. So, one of the things I get asked very often is, so what's an enterprise shiny app look like right what is. I've seen shiny apps and I've seen, you know, apps at our studio conference and I've seen apps on the web and, you know, I see all these shiny apps but what does it mean to be an enterprise app so give me an example. So, you know, what makes it an enterprise app beyond the fact that a company sponsors it. So, usually enterprise apps have some complications beyond normal shiny issue complications. So usually the data set is really big. I'm going to show you an application where the data set is very wide and very tall. And the users want to do things like filter and segregate on both rows and columns so it's a big data set they want to be able to do interactive filtering and segregation, and they want to handle data sets from different sources. Those data sets if they're from different sources in an enterprise they usually have different pipelines results, you know, formats. It's, you know, one of those, you're bringing a lot of fruit together to make a fruit basket they're not bringing oranges together to make a bunch of oranges. So, and then of course, you have the access of non developers using the app. And quality assurance and things like that so we the app I'm going to show you is called single cell viewer it is out there in public. We did release a public version of it. And this application itself needs to visualize a large number of points interactively on the charts. And it has a wide variety of charts and number of charts. It also allows the user to do a small, but very heavy duty analysis so there's something called differential expression in the biological world and this application allows the user to do differential expression analysis, which is a little unusual for such a large shiny app because on the fly analysis on a large data set can be very difficult because the user has to wait for it. You have to balance the time it takes to get your analysis back versus the user experience on the front end. And then of course there's that balance I was just mentioning that performance, not only with something like an interactive analysis but you have to balance performance of your charts. So, you know, the user may be willing to wait a few seconds for a chart. In fact, we will show that today, but they also are not willing to wait five minutes for a chart to show up. So this is kind of, I wanted to show you what an enterprise shiny app looks like before we get started because everybody sees the smaller examples on the left. So this is that single cell viewer and I will include the link to this in the resources slide you can go out and try it yourself. There's a version of it out and hosted. And so you can give it a shot it's hosted on a smaller server but you can see here on this plot this is what you're looking at is a data set that's composed of cells and genes. So, just to kind of give you an overview there's a lot of different charts. You can imagine that this data set has about 20,000 cells in it and each cell has quite a large number of genes. So as you can imagine it's a human cells. And so you can see that these, there's a large number of charts with large number of points on the charts. So that's kind of what I wanted to show you this is the interactive analysis portion of this app. And you can see here the user has selected two clusters of cells and is doing an analysis between the two. So this is what what I mean when I say an enterprise shiny application. Now this is used by internal and external stakeholders. It's used by large numbers of people at a time. Of course there are deployment concerns and whatnot but the idea is to give you a jumpstart on making an application that can visualize large data sets, as well as, I guess I should not get ahead of myself, as well as, you know, give you a framework for building an application that could become this big. I think once you see the framework and how it's used you'll use it for everything. I love it. And I did develop it so I guess that I'm a little biased there. One of the things that this session will do is probably introduce you to a charting package and may not be familiar with. So it's called Canvas Express. And I wanted to provide you with a good option for visualizing large numbers of points. So, Canvas Express was developed in about 2009 by Isaac new house. And it was initially intended to visualize genomics data sets, cells, genes, right, and different other visualizations that were very large. One other visualization packages didn't really scale for that number. So it was released as an open source on, you know, in 2011, and it is an R package, and it's been around for a while. So we've been going on six years on this one, five, five, six years. The package itself, so the core of this package is written in JavaScript. So the package gives you live plots by design. So by design, it wants the user to interact with the plots once you make them. So it's not a charting package where you make a static chart and then show it to the user. It is literally you send the data to the webpage and it builds the chart. And then it can do a lot of things on that chart that you don't have to write as a developer. Things that users expect nowadays, I would say five years ago they didn't really expect this. So there are a wide variety of charts. I just kind of wanted to put this up there. So it's not a limited package. It's a very, very expansive package at this point. And new charts and new chart types are being added to it continually. So I just kind of give you an idea of what is there to look at. And in general, this package provides much better scaling than packages based on SVG graphics. So it utilizes the canvas element. So if you're techie that kind of means something to you. But don't worry too much about it. But you can think of SVG graphics in packages like D3 and Plotly. So it works differently behind the scenes. So for example, one million point heat map in Plotly takes about 50 seconds to render. In Canvas Express it takes about eight seconds. Now, that still is a significant amount of time when you're looking at an application. But you also have to remember you have to build in time not only for that 50 or eight seconds but you have to build in the sending the chart over the internet, the data points and things like that. So it kind of gives you an idea of what does a one million data point heat map look like. Looks like this. So, and we will build a big chart. So listen to some nice size data for this tutorial. So I hope you enjoy learning a new package and have somewhere to go if you have large data sets that you want to visualize. The other reason that Canvas Express was created and has been focused on for the last couple of years to add features in this is the reproducible research. So, as you might imagine at an enterprise and especially in a pharmaceutical company like Bristol Myers Squibb, being able to reproduce your research that scientific principle of reproducibility is really, really important. And so these charts actually build in full audit capability. The data, the configuration and every modification of the chart can be captured so that when somebody saves this chart, somebody else could go back and customize it again. They can see what somebody did to change it and we're going to, I'll show you how to do that. So that's really, really nice in an image. So it's all captured in the PNG. And so, you know, if you are somewhere where you do research where you're doing science, you're doing analysis so chances are you probably want to know, you know, when your user says I got this crazy chart from your data. Where it came from? What did they do? What did they change? How did they get there from what you originally showed in the application? Right? So when you provide users with that ability to customize their charts, you'll also get the questions that are hard to answer. Like, wait, I got this chart and you look at it and you're like, wow, wonder how you got that. And needless to say, this gives you the capability to go back and be like, oh, you changed this and then you set this setting and then you change that and you filtered your data this way. And yes, you can filter it right on the chart. So we're going to get to the interactive portion soon, I promise. I wanted to kind of give you a little bit of background on canvas express visualizations just like any charting package it takes a little bit of getting used to. And since it was designed for very large data sets in the genomics field, it applies to any field, but the data format is tabular but three dimensional. So you can think of a data table but a data table with extra information. So it actually utilizes both the column and row names to cross reference the data sets. And you'll see this when we will make live code. The charts are built using named options which I will show you and the order is your personal preference or style you'll kind of get into it yourself and build your own style as you use these charts. So hopefully there is a ton of extra information at the website canvas express that word there's examples there and since this is a JavaScript package if you write Java applications or Python applications and you want to bring in the JavaScript library separately. So if you have pipelines that are hybrid at your enterprise, for instance, maybe you use our for research and analysis and then also use it for, you know, and then use something different to build your enterprise applications which is still fairly common. You can still build these charts without our, but you can build them in our so that's great but you'll see that this website is super rich it has, you know, full API documentation. And it has a ton of examples and each example has both JavaScript and JSON and our code that goes with it. So you can just grab it right out of the web page drop it right into our studio and create the same chart explore the data. So that's really, really helpful resource that I kind of want you to. So there's a couple of terms since the data format is three dimensional and tabular and uses both row and column names variables are the rows. So if you hear me say variables. So you can think of your own names as the variable names and samples are the columns. So sample names are the column names. And that one takes. If you're not a biologist, it takes a little bit of getting used to annotations. So they, it means just what you think it means it is extra information. So you can add extra information about the rows or the variables or the columns or the samples of the data set. So for instance, maybe your variables have some additional information we'll talk a little bit about why it's done this way when we get into the code. And you can either have annotations or not. So that extra extended information is going to be up to your data. Okay, you guys see the slides perfect there's a little bit of lag there. Okay, so we've seen a basic chart. And, you know, not very exciting at this point bar chart right it's pink. But we've seen the data set. And so what I wanted to do was show you kind of do a quick rundown of the most commonly used options, and that's more for your reference than anything else and you'll see them used we're going to go back and we're going to customize that chart to make it look awesome and make some more because we certainly don't want to just have a bar chart in our application. So you have to get your data into the chart. So that's put in with three different variables data, which is the main data set of our application. And that was what we used to create a bar chart sample annotation and bar annotation that's where that that kind of terminology comes into play. And so you know when I say sample annotation bar notation you know what's going on and when the documentation says that you you understand that that is sample annotation is extra information on the columns bar annotation is extra information on the rose. There's the graph type. This guy is almost the limit. There's a ton of different types of charts you can make from pie charts to word clouds to network diagrams. There are a ton of really cool charts. And we're always adding more the graph orientation right do you want it horizontal or vertical. Things that you always need to put in in my mind for a good chart title subtitles the axes titles tell your user what they're looking at. There's a theme. So there's about 10 predefined themes. So if you don't want to have to set like you know all the different colors and stuff each time you can use one of the predefined themes. Data point size transparency always a big one with big data sets. And then there's a special option that's called scale font factor, and it's a great way to resize some of the text but not all of the text right let's say your title is really big and you want your axis titles to be smaller right but just a little bit smaller we're going to do that. And then of course you can move your legend around. It's always a good idea to have a legend if it's not obvious on your chart what you're viewing. So there's a ton and there's more there's there's more. There's a save file name and when your user downloads your chart it should have a good filing. It will the program and the package will assign a file name, but honestly, you know we were using that. And we will set a custom filing for ourselves but you know the chart we just made. It makes more sense to have that be, you know, frequency underscore age or you know maybe the name of the data set underscore frequency right you know so it can start with something that makes sense to your user so they don't have all these files and their downloads that they don't know what they mean. So this is the canvas express main website if you hit the homepage it looks like this. And there are a ton of examples. So the biggest place that I tend to go even I do right and I maintain the canvas express our package, but I still go out here and I look at these examples. There are tons of examples are by chart type. So if you, you know, are taking a look and you want a contour chart. You want a dot plot a facet chart right you're looking at, you know, these are all of the different chart types and they have a ton of examples. And usually what you can do is find an example that you that is similar maybe you want to label a point like this. So this is called a decoration. So you can decorate your chart. And you would find a chart and then you can view the our code for it here with this button, or you can of course scroll down here and it'll talk about the data set the configuration and here is the our code for this chart. And so you'll see the decorations you'll see how to add, you know, additional things like a citation, how to set the different axes right. So this is a very, very information rich place to go and look for your for for help right when you're going to create a new chart and you'll see here obviously these are same functionalities we have on our charts now the chart that's shown in the chart depends on the chart type so you'll notice that for this chart there's a last two. And that's actually to allow you to select, you know, points without without square right so I just selected these three points. But, you know, you can there are tons of examples like this isn't just one scatter to D, you can, you know, go see other plots that are, you know, richer that are based on different data sets. This is the place where I would send people to go for help and it's where I go for help. I don't remember everything about the package. There's a documentation section as well. And then there's the API's right so this will walk you through some of the documentation. And then there's the API so the API itself is all of the options that are generally exposed. So if you're it for instance looking for hey I need to set that save file name. So I do that what can I set it to which charts is it you know used in. This is this is the option name and then you would set it to a value, a string right. You know, maybe you need to tag your chart maybe you want to change the theme here's the themes. So, and we will do this. So I wanted to show you the website just briefly so that you know where to go on the website for help. The website is the latest and greatest version, the R package trails behind just slightly. I, our team at aggregate genius maintains the R package, and we run tests on it and we make sure everything, you know, is still working and we we put out those updates to cram. So, that is, you know, and you're welcome to check out the package yourself and contribute if you, if you find it. Let's take a look at a scatter plot. I want to look at a more rich scatter plot so let's say we've got this scatter plot. Okay, and this is the typical cars data set and you'll see the cylinders here 64 and eight and you've got your your regression lines on here. You can go ahead on this chart. And if you would like to you can facet it by the. Sorry, switching gears. So you can segregate it by cylinders. What if I want three charts where each cylinders on their own chart. Right, so I can do this right on the chart and you can preset it up and you can you can go ahead and segregate it initially, but your user can also do this, which is really, really awesome if you take a look at that single cell viewer app. You could for instance facet that front chart where it shows all of the genes and all of the cells by gender. So that's one of the, the annotations for that data set is to look at cancer cells for males and females. And so the user can do it, you can start this way and then you can even bring it back to and remove the segregation and bring it back to the original chart. So that is a capability that's on chart and everything that's on chart you can use to set your chart up for your user initially how you want. And that you know that. So we're going to jump right back into a topic that's usually much more development focused than not, but I want to provide you with information about scoping in a shiny application since we talked about enterprise applications I want to talk a little bit about scope. So, in there are three main scopes in a shiny application there's global, it is accessible to all scopes and the UI, and it's only present when a global dot our file exists so if you're used to using a to file application where it's app dot our and server dot our, it won't, you don't actually have a true global scope. But that is global to everything. And then we have something that's called session global. It's accessible to all server sessions and this becomes important when you deploy your app. That means that all of the user sessions which are green here are attached to this session global scope. So that means that those are shared across all the sessions. So there's no differentiation between user sessions. There's outside of that server function in a to file app. If you're used to to file apps. And then there's that session local. And that is inside of the server function in a to file app. And that is the stuff that the user changes so if they change a widget right you want them changing their own widget value not Joe's widget value right so if you have multiple sessions, and you put your widgets for instance in the session global. One user changes that other users will see the change. So session local is generally where most of the code will go for shiny application because it's the part the users interacting. So we talk about it because the periscope package is actually set up to make it very obvious where you are putting things. So the global scope is meant for what right what would you put there so what goes there in a shiny app. Things that go in the global are things like library calls if you use dplyr everywhere put it in the global scope you don't need to put a library call inside your UI you don't need to put a library call inside your server file right things that are static static data sets things that nobody can change that you're just referring to anything that connection pool. So if you have a database that you're connecting to the pool would be global, and then your sessions would grab connections from it. You have to be careful in the scope for environmental pollution if you name a variable X and put it in global, everything will see X, and you have to be careful because you can have data and security exposure issues. So session global is a similar global type of scope and in general you would put common functions there. You might put connections. Again you could put static variables and data sets here, instead of in the overall global if you need to access them in your UI file like to set up a UI you would want to put it in global but if you're not if it's just behind the scenes, it would be in session global. Again you have to be careful with this scope. Session locals where most things go most of your code actually goes there it's a reactive context. So this is one session of the application one browser. A lot of people say one user but a user can have multiple browser sessions open so it's one session literally one window into your application. And you should always use a most restricted scope possible. If you can do it in session local you should be doing it there. I'm going to introduce the package and show you why we just talked about it. So periscope is a, in our package it was built by scratch I am the author of it, and I do the maintenance of it sponsored by BMS it was initially developed for internal use and it has been released specifically and a lot of people really like this. It is a framework for developing these shiny applications and enterprise. So, but it is not the solution for all of your problems. So shiny app development is still shiny app development. So, it's also not a substitute or replacement for properly planning an application, but it helps you out. It helps reduce the amount of coding you have to do we call it boiler plate. For common things how do I style that button how do I make this do that. Those are those boiler plate the wheels on the bus items. It creates a consistent UI experience so you may or may not like the UI. A lot of people like it it gives you that dashboard ish UI that works across a lot of different applications. If you're in an enterprise or you have somebody with multiple shiny applications hundreds, you know, developed by many different people you need that consistent user experience when they go into an app they expect certain things to be there. It'll also help with your scaling and scoping concerns for multi user apps. I've developed shiny apps before. If you have any sort of group size a lot of times you'll find that you make it it works great on your machine. You send it to your friend, whatever your group your PhD cohorts your, your colleagues, and two people try it at once and they're wrong. And that's that scaling and scoping concerns. So this package makes it very obvious where things go and it keeps your scopes as safe as possible. So you know when you're making something global, and you know when you're making it local to the user session. That's the biggest thing it does behind the scenes that really you will use it and you won't care about except you really do care about it. If you've ever spent hours and hours trying to track down why something happens. This will solve a lot of that. So you should give it a try. It's really flexible. It is not prescriptive. It is your content can be as simple or as complicated as you need. The content is what you focus on with this package. It's totally documented. There's our help. There's vignettes. There's an example app which I'm going to show you in just a second. And you can template an app and it gets you up and running really fast. And that's how we're able to do this workshop today and not spend three or four days building a shiny application from scratch. So it is actively maintained and updated. This isn't some older abandoned package. We are constantly adding things to it and improvements to it. So I'll just give you a quick rundown of conveniences. We're going to use the shiny module if we have a chance to put in a file button, but there's a number of modules that are actually included with the package. And each of these modules has a dedicated vignette to show you what it does and how to use it. But the things that are really built in that you don't even have to worry about. You can just start your app and it's there and it will work is the busy indicator. The tool tips. Tool tips are super easy to put on things. We'll add those. There's alerts that show up in the app in standard locations. There's URL parameter fetching. There's a reset button. If your organization requires logging actions, you can log your users actions. A lot of places require that audit capability and for things like financial applications. That's really, really important. And it will work on devices. So it's already responsive. And you can have a left sidebar app, a right sidebar app, both or not. So the sidebars are up to you. That's one of the most more recent features we've added. So it is very similar to programming traditional shiny applications. So we tried to make this as familiar as possible. And what will happen is, is we are going to put all of our content under what's called the program sub directory. And this is critical. So files above that program directory are part of the framework themselves that runs the application. And each framework file has a header saying, hey, use this or don't use this right so that that way you know what is going on. So the content is divided by location. So you'll see files like UI body UI sidebar and UI sidebar right so we originally started with just the left sidebar. So it's just named sidebar. We did not change that I didn't want to break people's applications going forward so you'll see these files and you can imagine where your stuff goes by the name. And then the server content and this is that scope, what we talked about so there is a global dot are, and that is where your global global your real global stuff goes, there's server dash global dot are, and that is for your server global scope. And again we talked about what goes in each and then there's server local which is where the majority of your code will go this is an all reactive context in that file. And so this makes it very obvious where you're putting things in the scope. Usually you won't even tend to think terribly much about it. So, so I wanted to talk briefly about planning a shiny app so I've jumped right on in with this example application but really shiny application development is is less like our programming and more like enterprise like you think of building Facebook or building an app on your phone or game or something like that you actually need to plan it. And that becomes more and more important as your app gets bigger and bigger. Are you guys seeing my share. This sharing is paused. We're seeing planning a shiny application. Okay. Now, yeah. Yes. Okay, fantastic. So, planning a shiny application doesn't have to be complicated. I use this example on the right with my teams and my teams are used to this they call it Connie draw. You know, if you're a whiz at layout packages and PowerPoint maybe you'll do it in PowerPoint or word but the idea is you kind of need to say look I have a screen and I'm going to show the user some things. I'm going to put the controls here and these are the kinds of controls I need. And these are the kinds of data and visualizations I need right I need a table I need us some charts. I want them to kind of look like this that's the idea. So, some basic planning is really necessary and a lot of times my planning looks like this. So you need to think about your data. Where are you going to read it. How is it accessed. Is it static. Is it big. Is it going to change to your users actually change it. How you're going to get it into your application. So you do need to think about these things. You need to think about your user. So that's actually really important. Are they using your app. Are they using it to explore. Are they using it to create reports that are given to their boss. Are they using it to find, you know, efficiencies in marketing results right, you know, campaigns or do a B comparison right. Why is the user using your app. And then, you know that kind of goes in concert with what controls are needed. And then you have tools, right, depending on your app. So, and again, you have a tool kit. Maybe sometimes you need a hammer maybe sometimes you need a screwdriver. And so those are, you know, the tools that that you can pick from right you know the charting libraries are, you know, a whole ton of them there's a whole ton of different table formats there's a whole ton of widgets in shiny like a lot of really awesome things that you can add to your application. Your application is actually so cool and has such interesting charts you need you know TensorFlow in there and you need custom D3 visualizations right so remember the skies the limit on content. But you do need to kind of plan ahead. And that's really what I'm saying here. So plan your layout plan your styling. A couple of notes on reactive programming and shiny modules. So there's a difference in programming shiny versus programming are so ours are functional language and we're all used to going line by line, executing something looking at results executing something else. So that's that functional language. But remember in shiny the user is not going to go in the order you expect the user is going to go in their very own order. And that is tough, right so it works much more like a modern application it's reactive user does something something else happens but the trick is is that not just one thing happens, lots of things happen. And as your apps get bigger and bigger, more things happen you have to keep track of what's happening so keep that in mind. Even though our is a isn't as a functional language the pieces of shiny are going to trigger other things to happen. And then there are context so there are places like that server local file that is reactive so that everything that goes in there can change and trigger something else to happen. So that means if you change a reactive variable, something else will probably happen anything that depends on it, unless you isolate it. So reactive variables include variables you define, and that's very useful, but it's not just input and output in session. So, and there's whole blocks of code that can change. And so just keep in mind that those, it's not just observed, and you can restrict it with isolates, but you do have to plan ahead for, if you change something, and something else depends on it, it will change. And the order of that change is not necessarily what you might expect all the time. And I say that not as a dig or anything wrong with shiny or are or the way that it's done it's just that when you trigger an event, you know when the when the light stops at a road. You know certain things are going to happen the cars are going to stop the pedestrians are going to walk, but pedestrians are all individual and so sometimes the pedestrian might walk faster than a car stops right and get hit in the intersection so you just have to be cognizant of the fact that there's a whole chain of events that are going to happen. And if you need them to happen in a specific order you may have to make them happen in a specific order. So, and I just briefly mentioned shiny modules. They are special little bits of code are not particularly exciting but they're just pieces that you don't have to write over and over and over again. So, in shiny, it gives you a UI and a server function. And you'll use it I mean this is no different than a button right you know you know my action button and then you know you or, you know, HTML output, and then you render the HTML modules are similar now module modules did change in shiny one dot five. And they. So, when you're programming in the latest version of shiny, keep an eye out for packages that have moved to the call server paradigm. So that's a little bit different you'll have to read up on it and are right now, periscope is using the old functionality. It's not. They don't go backwards and forwards. So, at some point we'll have to switch over to the new functionality but then everybody has to use the new functionality. So, that's a fairly new change. And some debugging items that we actually already got to and then deployment you do need to think about where you're going to deploy it. So, that's where the project was mentioned. That is where most enterprises and shiny server pro will deploy their applications because it handles the scaling nicely. Remember if you're using the free shiny server. It is single threaded. So if you have 10 users connecting to one application you are using one thread for 10 users and they all have to wait for each other. As long as you scope it correctly they won't interact with each other's sessions but keep that in mind. So, depends on your platform. But this is not a package that actually deals directly with deployment but it is completely compatible with different types of deployments. It is that inside part this is directly dealing with shiny so there are other tool kits that deal with the deployment specific so I wanted to. I will give you these slides of course and these have those resources on them that we talked about. And then is the primary repository for these packages. You're welcome to go check out the GitHub repositories as well. I have put them on here but remember GitHub until it's actually gone through the process of testing and out to crayon and murk back and tagged. You will notice that there may be some differences in GitHub just like any other packages in development. It may or may not work from GitHub. So please use the crayon package when you're developing shiny apps and you'll find the work the best. That single cell viewer that I showed you it has a pre print publication. If you want to read about the science and why it was created and you can check out the code and take a look at an enterprise apps code and you can try it out on our little hosted server at periscopeapps.org where we have hosted some shiny apps for people to give a try. Canvas Express is also out on crayon and you can check it out at canvas express.org. I have a couple of blog posts that you might be interested in one on scoping one on canvas express visualizations and then welcome to check out our company as well. We're aggregate genius and this would not have been possible without support from Isaac support from Bristol Myers support from us are our great North Africa team. Africa are Egypt are our ladies algears and my team Todd and Kailin and of course all of you. So I really appreciate your coming and staying and hopefully you learn some stuff and want to go give it a try. Because it's the it's the coding part that's the fun part so I hope you've learned a lot and are ready to go create your next shiny application. So I am going to stick around for questions of course but I didn't get the the warning that I was very, very close to the time. So, so much. Thank you so much this has been amazing totally and speaking personally and love your mode of presenting so engaging and the energy from beginning to the end is the same. Like I can't even imagine we are almost going to three hours. Thank you so much.