 Okay. So welcome back everybody. We will be talking now about a couple more, a little bit more advanced, a little bit more kind of complex topics, which are performance and security. We'll start with performance and move on to security. And I'm just going to go through a set of some simple things that everyone should think about when they're building applications, particularly DHS applications, but applications in general. When it comes to performance and security and how to use some of the tools that are available, such as the app platform and the data engine to improve and the rest API for DHS to improve performance without a lot of effort. And what to think about when you're looking at an application to figure out what could be improved or what needs to be improved. I'm going to share that screen now. Let's go back to the top here. Okay, so we're going to talk about performance. There are three topics within this that I'll cover. And the main two, I'll actually mostly just be talking about network bandwidth and bundle size and then I'll describe a little bit what will be coming in the future and about with progressive web apps and what that means for performance. So, right off the bat, network bandwidth is important. That's the first thing to keep in mind and to know. And that's because your application and DHS to in general might be used in poor network conditions. This is actually quite common with users of DHS to across the world, because in a lot of cases, a health facility, for example, might not have a fiber optic connection to the internet. It might have a slow DSL connection or it might more likely have a mobile network connection, either 3G or maybe 4G and some cases even 2G but that's even even more difficult to use DHS to. But if you have a 3G wireless mobile connection, and you're trying to use a DHS to application, it becomes very important for us as application developers to think about how many bytes we're actually sending across the network, so that it won't take seconds or minutes for an application to load or for things to happen in your application when someone is on a low bandwidth connection. It can also be important to minimize the number of requests to improve performance. That's less critical than the bite size, the size of the actual payload in most cases. But if you end up with thousands of requests, that could be problematic because there is a high latency between when you start a request and when it actually finishes or comes back from the server. Even if the network is fast, if it has high latency, even if it's a very small request, it could take a long time to go to the server and come back to the user. And if you then have thousands of requests that need to do that, it could make your application feel very slow. And this becomes particularly important when you have what's called a waterfall of requests, which means that you send a request, you wait for that request to finish and come back to the client, and then after you receive the response, you have to send another one. So in situations where there's high latency in the network, that can take a very long time or make your application very, very slow because you need to send, maybe it takes two seconds to send the request, two seconds for the response to come back, and then you send the second request, which takes another two seconds, another two seconds to come back. That's at least eight seconds of just two requests going across the network. So you're trying to avoid waterfalls where you have, especially when it's more than just two levels deep. So if you have a request that needs to wait for a response and then another request, another request, another question, 10 requests down the line, 10 times four is 40 seconds that you're waiting for those requests to finish. And you have to at a minimum give the user a good understanding of what's happening during that time. And if not, if you, that's if you can't reduce the number of requests or the or the size of the waterfall. And another consideration when talking about network bandwidth is caching. So in a lot of cases, the DSS API will do intelligent network caching for you. So the browser will actually have some cash headers on certain requests and it won't request that data again. But that's not always true and you want to kind of make sure that you're doing a trade off where you're not showing the user stale information because it's in the cash, you want to make sure that it's always as up to date as possible. So to address that right now, there are some ways to do it in applications as they as they exist today. Mostly that's around using a redux store or a service worker in your application. But in the future in the near future, app runtime will do automatically do client side caching. There's actually some pull requests open on app runtime right now to explore different ways to do this. We need to do it in a conscientious way to make sure that the that we're not showing the user stale data and that the application can still request up to date information when it's needed. But in a lot of cases, you make the same request or similar request from different places in the same application, we should be able to utilize client side caching so that we don't send a request across the network every time. For static resources, which are like JavaScript and CSS HTML as well, don't have to worry about the index that HTML. You want to make sure that files have a unique hash in their name. What this means is that if the file is has a actually a hash of that file. And no other file with the same name will ever exist because it, if it does then it will be an identical file, and then it doesn't need to be requested across the network. This allows us to do very long term caching of those static assets, things like see JavaScript files on the browser, because we know that the version on the server server will never be the same. It will never be different than what we have on the browser if the hash is available. The, this does happen automatically for generated files when you're using the platform, and it's uses webpack to do this. And it, it also happens automatically if you're using something like create react app and a lot of modern tooling. And, but if you're using files or putting files in the public folder of your application, and you're not adding hashes manually, then you might run into issues with caching. Yeah, so you would you would want to add a hash to the static files that are stored stored in the public folder. Okay. I think that's all I'm going to say about caching there's a lot more that could be said about caching and how to improve it in your, in your application. But generally, we're going to work on improving the way that caching is done in the runtime. And so soon that shouldn't be an issue to think about too much for data, and then for static resources, so long as you use the default ability to have a hash in the name of the files, and you add a hash to file names in the public folder, you can also it doesn't have to be a hash necessarily but if it's a specific enough name. Like you have the full version number of your jQuery five JavaScript file for instance, in the public folder, then that is a good kind of first step for setting up. Yeah, for avoiding cash misses in the in your application. Another important thing and this is actually has come up a number of times recently in some core DHS to applications as well is that DHS to the DHS to API has the ability to specify pagination so you can say I want 50. I want the first 50 indicators or the first 50 org units or the first 10 te eyes from a specific request or something like that. And there's a, there is also an option in that rest API to turn off pagination so you can say paging equals false in some API requests, but don't do that, please don't do that. It's, it's tempting because then you don't have to deal with the case where there's more than 50 indicators and you need to add pagination controls on in your application. But what happens when all of a sudden there are 50,000 org units or 100,000 users, or 10,000 indicators or dashboards that those users have created. All of a sudden you're downloading 10,000 or more objects across the network, which can be very, very expensive. And so you can't assume in almost any case that it's possible to request all of any collection across the network. And for that reason, it's highly discouraged to use paging equals false. And we've been bitten by this in, in the core team as well in a number of cases where with the roll out of COVID vaccination campaigns that are using DHS to for for implementations that are covering an entire country with 15 million te eyes that have 10s or hundreds of thousands of users, and they're running into issues using the users application because it assumes that there would never be 100,000 users. And, and so it's something that needs to be taken into account. And there is a warning that's shown in the console, when in development mode. If you're using the app runtime, a modern recent version of the app runtime that will tell you when you're using paging false, and as well as if you're not specifying fields correctly which I'll get to in a minute. Don't ignore that warning, and if at all possible so try to look at those, take a critical look at those requests that are being made by the app runtime, or those queries that you're using in order to. Yeah, reduce or make sure that you're using pagination and specifying fields explicitly. There are a number of challenges that happen or symptoms of not using paging in a DHS to instance that has a lot of objects are very big database. And sometimes the browser will freeze. In most cases it will freeze. And that's because if you're not doing this in a service worker which most applications are not today. You are actually need to stop the browser from doing anything while that request is sending. And if you have 100,000 users that you're requesting across the network. It might be 10s or even 100 megabytes of data. Which is takes a very long time, even on a very fast connection. And it can also be unreasonable to and completely useless on mobile, because if you have a one gigabyte per month data plan or something 100 megabytes is for for a single request on a single web page is not something happening. And the browser might even crash if it takes too long because it might run out of memory depending on the browser or the system that the user is using. And what could be even worse is because it's requesting all of this information from the server. It's quite expensive and heavy for the server to do as well. And the entire server might crash. And so the moral of the story is if you don't use paging everyone will be sad. And you don't want everyone to be sad so please use page. Here's how to use page pagination in a query as well as directly through REST API. The example on the right with the red border here is what you should not do. You can do this, but do not do it please everyone will be sad. And that's because we're setting paging is false with the users resource. This can be quite problematic because there might be 100,000 users, and we don't want to request 100,000 users across the network. In most cases, if you're requesting a large set of users for a search or something like that, you want to show maybe the first 10 results only. So just set the page size to 10, and then you can actually add pagination or continuous scrolling those types of things as well. So the example on the left here is what you should be doing. So set the page size to anything you want whatever makes sense for your application the default is usually 50 for most resources, and but you can set it to 10. You can set it to two, you can set it to 100, and just make sure that there is some limit and you know how many users will be returned from this result so there's an upper bound on the size of that request. And if you're doing this in outside of the app at runtime. This is what it would look like in on the bottom here that's what it would look like as an API request across the network as a yeah as just a regular rest API request. So you have the page size equals 50 and page equals one, and don't set paging equals false in that URL. Another thing that's kind of related to pagination but slightly different and maybe a little bit more subtle is the concept of fields. So most of the metadata resources in DHS to include the option to specify which fields you want to respond. And if you don't specify any fields in your request by default many fields are returned by the API so there's basically a default set that might be quite verbose, and you don't usually need all of those. And this, this gets even worse when you're requesting resources from a collection endpoint, because you might request, even if you're paginating to 100 if you have 10 times as much data per object. And then as what you need, then it can take, it can be quite, it will be 10 times as as big, and with 100 objects times 10 times it's quite large and payload that will be going across the network. In particular, you need to pay attention to fields which result in embedded resources, and specifically for embedded collections. What this means is if you're looking at the user group endpoint friends, for example, a single user group might have 10,000 users assigned to it, which if you request the user group endpoint or the user group resource and say a page size of one, you still might end up downloading 10,000 users if you include the users field in that response. And to demonstrate this I'm going to quickly go through a quick example. So let's go ahead and do this in the, in the data query playground. So I can do this quick example. So this is not a huge database, it's a fairly small one. I'm going to go ahead and close these because I don't need these. But here we're going to request user groups. So again, this is not a large, large database so it's not written it shouldn't be a huge problem to do. We won't see a problem in the in the browser here or in the server, because this isn't talking to a large database. But if it were a large database, it could be quite problematic. And even on this database itself, which isn't that large, we can still see significant performance improvement by applying some of the things we just learned about. So I'm going to start by just getting rid of this field so we're just going to get the users groups endpoint without passing any, any parameters. So we do that. We'll see that it is paginated by default. So we have a page size of 50. There are a total of 31 here they all fit on this first page. But if there were 10,000 we would still only get 50 in this response, which is good. In this case, the user groups default fields is quite, quite sparse. So we only get the ID and the display name of the user group when we request it. For some other resources, let's say for instance visualizations. This might be much more. There's there's another one. Actually, I think this has been fixed in 236. So this might actually be the default is not as verbose as you'd want. But let's say that you want a different field here. And so you say fields. Let's say fields is going to be star, which is quite common when you're debugging things to use field star. And now we end up with a whole ton of fields in this response. So for all of these indicators, there are 50 indicators on this page. We have a large number of fields that were that are being returned. And there's information about some embedded objects like a user. There's some other information about sharing things like this, that probably in most cases we don't want. And so in for indicators, this could be could be problematic just in terms of the number of fields that we're returning here. But if we go back to our user groups, user groups and point and execute this, we're now getting a large number of fields back and we're using the star the star fields which we should should never do if we have any option not to. But you'll see that this actually includes the ID of every single user that is in this group. So that's, that's quite large and maybe if we're not listing the users in this group, that's useless to us and it's just wasted bandwidth. Let's go ahead and clear this log here and execute this again let's see how big this request is. So this is a five kilobyte request across the network. It's not huge but it doesn't need to be so if we only want the ID display name. So let's say, so something else that we want from this, let's say, and let's just say a trip. There's some other ones here let's say last update. See those are the only data, that's the only data that we actually want from this. So I make this request again. 80% reduction reduction in the size of this network request, because it went from five kilobytes to four or to one. And so this can be quite, quite useful. And there's also the concept of embedding resources. So we have this if we had this star we get the ID of, we get the ID of each user that's associated with this. But if we actually did this. Again, something I would not recommend this will give us all of this, all of the fields of each of the users that are associated with this with this user group. So this becomes quite a bit more and actually users is a fairly sparse class, at least in 236 in 234 it's quite a bit bigger. And so this actually has now gone up to 14 or 15 kilobytes, even though we're probably not using all of this and if there were 10,000 users instead of 10 assigned to this user group. We get potentially in the megabytes of information going across the network for no for no good reason. If I switch to the switch here to a different server, let's say one that is on 235. Let's say 234, I need to add this to the one moment need to add this to the course list for this, this group, I believe. So let's go ahead and remove this, add this to the course list and make it all not recommended to do this in production environments but for testing it's okay. All HTBS request should be fine. Let's go ahead and log in. Okay, so let's go ahead and try this user groups endpoint here with a star and users star result. Let's see what we get here. So here we end up remember before we had 15 kilobytes now we're at 22. And because we're getting a lot more information about these users. And this was reduced in for security reasons in 236. Because the most of this information about this user shouldn't be exposed to other users. So that is fixed in 235 I think actually. But this also shows that there can be quite a bit of information embedded in these objects and let's see, even even further so we have users, then we actually have user roles. So if we go users star comma user roles star. We can really mess this up. We have some pretty big problems. Actually that didn't work exactly. Maybe user roles only has the ID in this in this object, but there can't you can deeply nest some of the information that you're requesting, maybe organization units. Okay, yeah, so that those ones only have objects organization. So we can't get too deep here but we're still at 23 kilobytes instead of instead of 1.4, which we might get otherwise. Now let's also turn paging off. So if we say paging false and execute that. We're still about the same size because we have a limited number of user groups in this in this with this endpoint. But we you'll see that there's no paging now so if there were only 31 user groups to begin with so we had a paging page size of 50, and we were seeing all of those results, but now we have quite a quite a few. So basically would if there were 100 user groups we would see 100 different responses. If we do users instead there aren't that many users, but we're we get a number of get all of them at the same time, let's say visualizations. And again, user star doesn't make sense in visualizations. Let's go ahead and do this. So that is how long it takes just to get the list of visualizations in a very small demo database with paging false and field star. And that is more than almost a mega almost a megabyte of raw data 100 kilobytes across the network. That's quite problematic. So this is just some examples of both paging and fields and how you have to be very careful when you're doing an even with default default pagination so if we don't pass any fields, but we say paging false. So with default fields. Actually in this case it's pretty, pretty okay. Let's say dashboards. Okay, so we actually backboarded this which is good. Let's say fields star for dashboards. That's also again quite a big request. That was 12 kilobytes. Let's see if we get. Yeah, and that also returns all of the dashboard items. And this can be can get quite expensive quite quickly. So, again, just a demonstration of use using you always use paging and use the smallest paging that you can can get away with so if I say page size instead of page, page size 50 default, then it will be very, very fast and very, very small network request that is 577 bytes instead of 12 kilobytes. Okay, see what the next one is. Yeah, so this is just kind of expansion on exactly what I was saying before. Be careful with embedded resources embedded collections if you try to expand those fields, you can have problems. Avoid using star or all in fields at all costs, if at all possible because that can be very problematic on large databases, and it makes caching more difficult because we don't have an explicit set of the fields that we want so we can't do things like requesting only the field we don't already have available in the network or in the in the app runtime in the future. And this is another just basically the same same thing and talking about caching talking about what to do to test your performance of your applications. The size of your JavaScript bundle will affect how performant your application is, and it can take quite a long time to load on 3G networks so we were just doing this this example with let's say visualizations paging false. We did this one paging false and fields star. If we open this up now and we you'll see that I have the network tab of my Chrome Dev tools open have disabled cache, and I have no throttling. This will show me how long it takes for this network to come this this request to come across the network. It's still fairly fast because I have a quite a fast network connection here. And but it is almost 100 kilobytes and more than a second. If I then change to fast 3G, which is a more realistic representation of what a lot of DHS to users will be using. It will take a bit longer actually that didn't take as long as I thought it was going to be. Let's try slow 3G. So that can take quite a bit longer as well. It's also important to note that for things like JavaScript. If we look at, for instance, this year. Hopefully you can see this. This is an example of a DHS to application in JavaScript and what you can see in the network tab for this application and what you should look at so this is using fast 3G. We've disabled cash. This is quite a large application so it has more than a megabyte of resources, 1.2 megabytes actually, which is quite a bit. It's still down a fair amount. But they're, yeah, they're these are the metrics that you want to look at and we'll talk about how this is split up into different chunks as well in just a moment. So I do that this takes quite a long time to load on a fast 3G connection. It's important to note that even if it takes a very short amount of time to go across the network for a very large chunk of JavaScript for instance, this this large one that's actually two and 2.7 megabytes uncompressed. Even if it takes a very short amount of time because you have a fast network, it still costs time for the browser to parse and process that JavaScript once it's available on the local machine. So, even after the download is complete, it still is worthwhile to reduce the size of your JavaScript bundle. So let's talk about how to do that how to reduce the size of your JavaScript bundle and in. Go ahead and keep this open and one one of the ways to reduce the size of your JavaScript bundle is to make sure that you're using tree shaking throughout your application. So if you're not familiar with what tree shaking is you can Google it if there's quite a quite a fair, fair amount of description of what it is on the internet. And, but I'll give you a quick kind of overview of what it means. Basically, it means that when we import from a dependency, we only want to take the parts of the code that we're actually using, and not the parts that we're not using. So for instance, low dash is a classic example of this, which low dash is a is a large utility library that can be used for a lot of different things. And if we're only using one feature of low dash, let's say the index of function and we shouldn't download or include all of low dash which has 100 different utilities in it. In our code base and to do that we to when we import the default export from low dash, it can include the entire library, rather because at compile time, your application doesn't know what parts you're going to use. In some cases, the compiler can be a bit more intelligent about this. And it depends a bit on the on each different library. And so I think actually low dash maybe has improved this since since this was a real big problem. But basically what what the goal is is to avoid importing the entire library when you only want to use a small part of it. So it's better to use this syntax which is import star as underscore, which will use named imports rather than the default import from low dash. Or you can actually use that named in import explicitly with brackets. And so in both the better and the best case here. The compiler should be able to get rid of all the 99 other functions that low dash exposes and only include index of in your library. And it's also important for for tree shaking. Another important thing to think about is making sure that you don't have multiple copies of libraries in your bundle. And you can use yarn to look at that you use yarn list or yarn why the name of the, the function so I'm actually going to do that really quickly here. So let's see. So I'm in this data store that we were talking about earlier. And if I say yarn list. It will list all of the dependencies that I have. That's a little bit hard to read but I can do yarn, why, let's say at DHS to us UI. So I'm going to yarn list. So there's quite a few dependencies in this project. And this will. Yeah, so this, this isn't the easiest to parse because there's a lot of things going on here. But if we want if we know that there's one potentially problematic library we can do yarn why at DHS to us UI for instance. And this will show us that we have only one version of that DHS to slash UI. Six to six dot two. If there were multiple versions it would list both of them and you would be able to think about ways to work on your dependency tree to be able to only have one version. I'm not going to get into too much how to do that today but if that's something that you're running into please feel free to reach out on slack. Another example is react. So react can be actually react dom is much bigger than react. And so if you have multiple versions of that that's problematic for a number of different reasons. And if you have a large visualization library so actually we have low dash probably so I bet we have several low dash year as well. And we can actually see that this is multiple, multiple copies of low dash and that are hoisted from a bunch of different places. And so it only uses one copy in the end. And we have this found a single version, but if the different dependencies had different version requirements for low dash, you might end up with multiple copies of low dash which could be problematic. Yeah, that's basically basically another thing you can do. You can do yarn build. And this will return some information about the oops. Actually have a problem here. See what our problem is in this example. Oh, we don't have react. We'll get to this in a minute. But one another thing you can do is use a tool called source map explorer. I think this should work. And this will give you any a visualization like this which will tell you exactly what is included in the built bundle of your application and what it what those components are and how big they are. And it'll tell you which dependencies are included in your bundle, how they're, yeah, how they're trees, tree shaken out the parts that you're not using. And maybe if you have multiple copies of the same thing you can visualize it there. And finally, another thing that you can do to reduce the size is one actually reduce the size of your bundle but it will split your bundle up into multiple pieces. It's not official if, for instance, you have multiple pages in your application, and one of them requires a visualization library that's very heavy or very large. But you don't need that on the other routes, the other pages in your application. You probably should, oh, you shouldn't include that library in the download that happens when you first load the app. You can use it when you move to the page that has that visualization component. In order to do this you can use something called dynamic imports, which does bundle splitting in your application. This is how you do it in react with web pack, and this works out of the box with the platform, but there are other ways to do this with other frameworks as well. So basically what this will do is it will start to load the my component. My component is what's called reacts lazy, which will import a another bundle with some of the, like a subset of your total code. And then while it's waiting for that component to load it will show this fallback component. This can be usually it would be better to have this be a loading spinner or something like that. But just I'm just saying loading dot dot dot here for simplicity. And then once it's finished it will render this my component. Component that's loaded from a separate bundle. We saw that in a couple slides back. We saw that he in the data vision application that there are actually four different chunks that are all coming from the application. This one, this 812 kilobytes is the one that has the expensive heavy charting library in it. But that doesn't need to load before the page can render. So the page renders once the first chunk loads 168 kilobytes. And I believe that's actually 174 kilobytes, and then it loads these other ones for doing some actual visualization on the page. So this is how you would do that in react. And it's done automatically between the app shell and the app in the platform, which means that the app shell will load faster and first before your app gets loaded after the fact. But this can also be done within your application to split up different, different routes or different, maybe heavy pieces of processing that you need to do in only certain situations. Okay, that's all that I had today for. It was maybe a little bit longer presentation than I was expecting, but that's all that I had today for performance. And I will now open it up if anyone has any questions. Otherwise, we will take a quick break.