 of the application. Bob's always so hard to follow because he brings in a lot of interesting philosophy plus practicality, which is a really unique skill. I'm just going to be all practical, I think, hopefully. So let's talk about some of the analytics problems of large scale implementations. There's a sequence of events that usually happen. First, we hear that we can't get our data in. Apps are crashing. We can't get the 20 million patients we're trying to enroll into the system. We have 5,000, 10,000 concurrent users. Their apps are crashing. People aren't working. So we run around. We try to fix that. Usually we do. And then what happens next? We can't see our data. Everybody's putting data in, but we can't get the analytics out. Dashboards are crashing. Event reports, line list is loading forever. People are waiting 15 hours for analytics tables to run, these kinds of things. We're going to talk a little bit about what can you do to fix analytics performance. I want to say fix. Most of these are kind of, I hate to say the word band-aids, but they're things, they're implementation things that you can do to mitigate the analytics performance issues. Some of them are more practical than that. But yeah. So what are we seeing? The problems that we're really seeing are slow performance of analytics dashboards, long list tables, charts that take forever to load or never load, analytics tables failings, worst case scenario, this almost never happens. Servers crash because the analytics queries are too big. Again, extreme worst case scenario only happened once or twice, not many times. And that was actually due mainly to poor hardware, than anything else. So it's causing, what's causing the problem? The problems are when we start to see large scale tracker implementations usually, you know, around or bigger than 500,000 to a million tracked entities. So 500,000 to a million people in your tracker instance. And what is this doing? These are causing really heavy analytics requests. So we are, you're trying to build maps that show all 10 million tracked entities, right? You've geo-coordinated all of them. So I want to turn on a map that shows all of them, right? So you're asking DHS to return 10 million points on a map. It's a heavy request. Enrollment or line listing with, you know, several thousand, 20,000, 40,000 tracked entities on it, right? You want to see all the data from all of these 20,000 people. That's a heavy request. It's a big, big amount of data to pull in to analytics. Visualization requests that are for longer than 12 months. So maybe you have tracker programs, some of you know, like COVAX or COVID programs now running for a couple of years. You want to see all of your COVID patients for the last 24, 36 months, all, you know, all their data in a single line list or something like that. Or on a, that's a huge request, you know, so these are, these are just the scale of them is really difficult to process. And then the last one, which I think is probably the most important one and something that's more actionable than most other things is any visualization that has an enrollment type program indicator. So an enrollment type program indicator is essentially you're saying I want to see all the data for all of the, for all of these enrollments, right? We have two types of program indicators. We have event and enrollment. Enrollment type produce massive queries. And that's actually what these charts are showing right up here. So the top one looking at dropout, this dropout indicator from the COVAX package, this is our COVAX package, is obviously you see the time there. All those dropout indicators are massive requests. They're enrollment indicators. There's a lot you can do to improve your performance if you can minimize the use of enrollment indicators. And that's the same thing for this COVAX package here. You see kind of two different charts showing large scale. So what are some solutions? The very first one, and this is what Bob was talking about, make sure you have adequate server resources and support. Period. If you're going to do large scale tracker, you have got to have large scale server support and management. You've got to have people who work, who know how to manage it and to do it. These problems usually happen, you know, when everybody is at work, right? So, you know, in the morning, when people are enrolling a lot of people, they're doing a lot of work, they're putting a lot of data in and project managers are querying the databases, pulling up dashboards, right? All of this is happening simultaneously. This is when things really hit the fan. And you've got to make sure that you have the resources, the server resources to do it, to first of all, and then you've got to make sure that when things do hit the fan, you've got staff who can manage, who can actually respond to it. The staff who can respond to it has been a perpetual issue. The other one is do not run your analytics tables during high traffic time, right? Run your analytics tables in the middle of the night when there's no one using it, hopefully, you know? Don't run your analytics tables at 3pm, right? I mean, running the analytics tables is heavy on the server. It's very taxing. And so, if you run it at 3pm, when you have lots of people reporting data, lots of people trying to pull analytics out, you know, there's only so much strain the server can take. Use event report program indicators whenever possible, mentioned that already. And then the first one is do not have dashboards with program indicators or, sorry, enrollment indicators set as your default landing page after login. So, hopefully, I assume you all know DHS too, that when you log in, usually the first thing you see is a dashboard, right? Well, that loading that dashboard is an analytics request, right? It's happening by default. So, it's immediately, as soon as you log in, first thing fires off analytics request. Well, if that dashboard that they're landing on is showing tracker data that's querying, you know, thousands, millions of tracked entities, of patients, then the very first thing you're doing right after logging in is hitting the server with a massive request. And that can be disruptive not only for you, you can wait forever, or it can be very disruptive to other users who are trying to do other things. So, this is a great best practice. And I think one that we really want to encourage as much as possible is, and this is an example from Sri Lanka, where their login page, they've set a default login page or dashboard that's just text. It's information. It's useful information. It's like, welcome to DHS too. Here's some things that you can do here. Here's what's important information to know. Here's some contact information. These text fields are minimal requests. I mean, they're very, very easy to load. So, this is what we recommend, is set your default landing page or default dashboard for users when they log in to something like a text dashboard, especially for heavy tracker implementations. If you're doing stuff in aggregate, everything is probably fine. But if you're doing big, big tracker implementations, this is a good best practice. And of course, you can set it as your first dashboard that everyone lands on by the dashboards are ordered alphanumerically, right? The order that you see the dashboards on the top. So, you can star it. So, that makes it kind of in the frontal list. And then you can also name the dashboard something like asterisk, asterisk notice or asterisk, asterisk info. And then that by default will put it towards the front. That was actually a bug, the sorting alphanumerically. But it became a very useful bug, so we actually just kept it there. Yeah. Some more analytic solutions. Limit the sharing of dashboards, like large dashboards, to just key users, right, that really need to see that data. Does everyone at every level need to see all 20 million tracked entities on a dashboard? Maybe, maybe not. And if they don't have to, don't do it, right? Just give them the key information that's relevant and useful to them. You don't have to share, make big dashboards for everybody. Well, you may have to, but if you can avoid it, I would recommend it. Restrict the number of org units that people need to see. So, you know, if I'm sitting at district level, do I need to see data representing the entire country? Maybe sometimes yes, maybe sometimes no. And if the answer is no, then please make that change. You know, just show them just their district or just their facilities that they're managing, that kind of thing. And I'm almost done. Yeah, thanks. And then tracker analytics requests. So, yeah, make sure if you can do some of your requests outside of peak hours. So, some more advanced implementations, what they're doing is they're actually sending those requests for program indicators in the night. They're caching that data. And then they're, and then so you're able to see that data in the morning without sending these large requests. Just a couple of more points here. The first one is most instances we look at have not changed their caching. So, you have to make sure you enable caching. So, what happens is I log in to DHIs too. That dashboard caches in the browser, right? So, it loads once. And then if I leave and I go back without closing my browser, it's going to be much faster. And the reason it's much faster is because it caches. One easy way to do to make sure that you can continuously go back to the dashboard without sending, requesting from the server all the data is increase your cache duration. So, we recommend setting your cache expiration to 3600 seconds, which is 6 to 10 hours. And there's some steps to how to do that. And then the last one, which is probably not very popular, turn off continuous analytics. Everybody loves continuous analytics, but continuous analytics are running constantly. It's constantly making requests and that kind of stuff. So, it can be problematic. Last thing, if nothing else is working, last resort solutions here, which hopefully no one in this room has ever gone to experience, but just to put it out there, remove tracker analytics for non-critical users. Just don't share tracker analytics. You can consider using SQL views, HTML reports, our shiny apps. These things can work much, have much better performance. You can push routinely to separate instances. So, a lot of large scale will have an analytics instance and a data entry instance. Two different databases. Data entry is pushing to analytics continuously, but the traffic that's on the analytics is not affecting the traffic on the data entry. So, you can have 20,000 current users on the data entry, but that's not affecting the folks on the analytics instance. And, yeah, I think that's it. We'll just say while Marcus is setting up, we are making significant performance improvements and you're going to see some really big API changes coming in the near future to improve analytics performance. But, you know, as these systems grow to millions and hundreds of millions of track entities, I think these will still apply. Okay, I know it's kind of intense. Yeah, okay. Thanks. The next section here is going to be even more practical than the last one. No, I was being serious. When things go wrong, there's always a compounding reason for that. And when a server falls down, there's normally many factors that contribute. And analytics, heavy analytics is stealing resources and other setup choices are stealing resources, missing indexes, for example. We often get called in when there's already a crisis. And during the year of COVAX, we have been called in many times. Many of the people are sitting in this room. And when we get called in to help, usually there's already a crisis and there is no other way for us to help than to get eyes on the server. So, we need to have monitoring in place and it needs to be in place before the trouble arise to be able to help you properly. So, please think about monitoring. Right now, I'm going to go into six very, very concrete things we have found during this very interesting and turbulent period. And this is six things you should always consider. This is six factors that we have seen in many servers have a big impact. And you might not be able to implement all six, but you should consider all six. It might not be a good idea to implement all six, but you should consider all six. And ideally, you should do it as early as possible if you're running a large scale tracker. The first one is ID generation. We know that the random ID generation is slow. We have been trying to make it better. We have not succeeded as well as we want. So, we would actually give a general recommendation that you should not use random ID generation, but sequential, because sequential ID generation was also slow about a year ago, but we have improved it a lot. Reiterating the recommendation of staying on 225 or newer and on the latest point person, because this is where we have been making improvements. The next one is the working lists. We know that the default behavior in the old Tracker Capture app is that you load every single enrollment when you open the Tracker Capture app. This magnifying glass shows the description of the default list. And this is not often a very useful list. It's not often that you go to this list and click the person you want to edit, is it? Well, we want you to consider this. There is a first aid thing you can do, and that's simply to turn off the standard working list. You can see here where to do it. It's in the maintenance app. It's called display front page list. If you deselect that, then the users will be met with the search form here. They will not see the list by default. They will have to search. There is a drawback, and this will also disable the working list in Android, so you have to search there as well. So, this has been a little bit of a catch with this solution. Just to draw a parallel to one of the things Scott had on his slides, the turning off the front page list and making a dashboard that contains only text is like maybe two of the simplest things you can do for your server when you have problems, or if you don't need this list, turn it off and make an informational dashboard. That's the two best things you can do, I would say. The next thing is that the standard lists, as mentioned, might not be so useful, and it might be better to make custom lists. So, this is a slide in Norwegian from the Norwegian COVID system. So, I'll help you with some translations, but the first list here was indexes due for follow-up today. That was the first list you were met with, and this was not a very long list, but maybe there's a good chance you can just click a name and go directly into your the record you need to work with. Some other lists they had was notification not sent, my assigned tasks, and the assigned tasks. The next big tip we have for you guys is the DB indexes. When you search, this is one of the heavier operations performed on the tracker system. As long as you use the unique ID, this is fairly quick, and one general recommendation is that if you can, if something should be unique, it is a good idea to check off in the maintenance and say this is a unique field, because then you get a different kind of search that's happening, which is much, much faster than what you get when you search, for example, for a last name, and these last name, first name queries are generally heavy. To combat this, what you should do is to add a B3 indexes to the searched fields, or the commonly searched fields. There is a very good description in the document that will be linked in the documentation page for how to add these B3 indexes, and example as well, and a good guide there. This is a very useful step to take. Another thing that we have seen is that there is a functionality in the tracker that's not that commonly used. In many instances, we see that there is no limit to the number of hits you can find if you search. This is the default, so that's why people leave it at default, I guess, and the users get used to that they can search for something very, very general, and they can find a very long list of 200 people, or they can try again and be more specific, and we want to recommend that you set a limit here so that when you search, you will not get any results if your search is too broad. This is an example of a very, very long search result list, and this is the setting in the maintenance app, maximum number of tracked entry instances to return in search. It defaults to zero, which means you can get as many results as, well, there is no limit to the amount of results, but it's much better to set this to something sensible, like 10. If your search produces more than 10 results, then you should try to add something more, be more specific. You need to set this both at the program and at the tracked entity type level. When there is a limit, and the user does a search that is too broad, you'll get a message like this, too many search results, please refine your criteria. The last thing I will go through is the blessing and the curse of the custom API queries. Sometimes this is exactly what you need to utilize your system in the most efficient way, an external app, a script that does something smart. What we have seen many times though is that these are queries that is not so tested by us. There might be something with them that is a little bit different than what we are doing. You might put a parameter, you might put something different, an extra attribute that we didn't use or test with, and it might produce a very slow query. It's a good idea to write custom jobs and apps, but put extra care into making sure that they are performing. A few things to remember is skip paging. If you use the skip paging parameter in this custom query, that is a red flag. These queries might be very, very heavy. If you include the page count, this looks so innocent, but it actually causes the query to have to count how many records would fit. Instead of just returning the first one, you're actually triggering something very heavy if you add the page count to your API query. Another thing is that when you're making these queries, it's much less costly to use equals. Use equals if you can, like comparison of strings is very heavy. Then again, make sure to monitor your system, because that is the way to see if there is any queries that is taking, many times we come in and we see that there is one query that takes most of the resources, maybe 70%. It's not a query we know, so we need to help tweak it or change it somehow. But we need eyes, we need monitoring, if we're going to see this quickly with you. All right, thanks. So you can see we have only 10 minutes left and two presenters, so just warning them. But also in the sketch description, we've put a link to the community of practice where you can add questions there. I think it's unlikely we're going to do a lot of open questions, there's just not that much time. So go to the sketch description and click on the link. We'll continue to answer your questions after the session as well. Thanks. This is working. Yeah, I'm not used to this. Hello, everyone. I'm going to try to be fast, quick. I hope if you had some coffee or tea after lunch, the caffeine is still there and you're going down. In any case, if it's going down, yes, for you to know that what I'm going to be talking about here is very well documented. We have everything that I'm going to be talking about here with examples, with the theory behind it. So in case your attention is not the best now, at least go back to the documentation. Basically, I'm going to talk about Android, that's what I'm doing here at University of Oslo. And those are the elements that I'm going to quickly go through because we don't have much time. And what I'd like to do is, first of all, set an example. What I'd like to think the application was designed to be able to work offline. This means that Android is going to take all the information that might need, and I think that's the key work might need. Because if you think about it, you might need to work on the field and you have to, we're not in the detailed war, we're in the analogical, we have to take papers, books, everything, we will load our trunk, we go out there on the field, and we want to make sure that we're taking everything we might need because we don't want to go back. So this is what Android is going to be doing. So it's very important you keep this concept because if you don't set your server properly, Android is going to take a lot of information that might not need or might need. And it's not bad for C, but the thing is, if you have a lot of devices and you're making all these requests to the server, you're going to end up overloading the server. That's what I like to keep as an image. So when you try to configure your server, take this into account. And if you're going to use a user that is only going to be working in a specific organization unit, only use that organization unit assigned to this user, the same for programs and datasets. Because if not Android will take everything and your phone will be slow, the connection or the download of this metadata will be slow. And on top of this, you will be overloading the server. And if you multiply this for 1,000 devices, you are likely going to kill the server. The same happened for auto-generated values. Mark was already explaining the random and why you should be avoiding them. But also it's important to know that because Android works in this way of offline, if you're dealing with dates or with sequential, you might end up having gaps. For example, if you're using dates and I don't want to go too much into theory because it's explained very well in the documentation. But if you, let's say you're using months and now it's today's 20th of June, imagine we are not on 20th, we're on 29th, your values will be expired on the day afterwards. And then you will need to request more to the server. The moment you perform a synchronization, again, multiply this for 1,000 devices, you're likely going to kill the server. So you need to have this theory in mind. So when you're sitting, it's very, very bad to be using day of the month. If you're using auto-generated value, it's a bit better using months, but it's even better using years because this will only expire after a year. Again, links to documentation and you can find it in the official documentation. If you were there this morning, Mark explained very briefly the Android settings were up. These are one, some of the things you can do with the Android settings are up, but basically what you can do is set or tweak in case your configuration cannot be very, very specific for user. What you can be doing here is limiting the amount of data that you want its device to download per program or per organization unit. So these are the things you can do. I will not go to them because if not, all of them have to talk about everything. But one of the last points here is while device management, even though it's not strictly related to the performance, it's important for the performance of the devices. So we recommend and we know it's quite expensive to have an ATM while device management, but it gives you the power or the ability to control this among other things. You have a guideline there where we explain a bit more and we set examples. I know there are a couple of implementations that have been using it. Again, it's quite expensive. So might not be ideal for your solution. In case you're going to be using it, you probably need to budget it before material and make sure you have the economical resources. And one of the things, well, you can do is this, but for me, the important one is the version update control. Why? Because we from Oslo are controlling when we publish the new release. So if now we're in 261 and tomorrow we decide to publish 262 and you have not trained your users on the new version, you might end up having users that are not very happy because they don't know why the change of the interface or whatever. In case you cannot adapt the mobile, you cannot get a MDM, what we at least recommend you doing and this is free to do is basically sitting there, the automatic updates of the application we'll have a mouse here. Yeah, here basically. So disable this so at least your users will not be getting automatically the new versions of the Android. I think I went quickly enough. This is the checklist that I tried to cover in five minutes. In any case, there's not going to be time for questions, but tomorrow at 4.30, if I'm not mistaken, we're having a two hours expert launch of Android. I'd be happily taking your questions there, trying to answer them. And I think you have two minutes. Speed. So I'll continue at speed. So we actually left in a few slides here from the COVAX webinar that Mike mentioned, which I'll skip. It's just one thing I wanted to talk a little bit about. We're talking here about tracker performance, but I think it's important to keep in mind our overall information system as well. So what we saw with the COVAX was that it was for many countries, not really possible to keep up with real time data entry of individuals. So then just keeping in mind that you might actually have to have in parallel with your large scale tracker program, at least for a transition period, some way of continuing with the form of aggregate reporting. So that's sort of slightly on the side of this topic, but just important enough to mention I think here as well. So what I wanted to mention briefly was this issue of linking the tracker data to aggregate data, because what we've talked about the dashboard performance, analytics performance, based on your tracker data, and what is heavy there are the program indicators. The reason is that they're doing lots of counting basically of the tracker analytics tables. So even though we have the process where we take the raw data from tracker, transforming it into analytics tables in the database meant to speed up the performance. These program indicator calculations are still heavy operations. At the same time, we can't have a system where users aren't able to see their data. So what can then be a way around this is to take those program indicators, do sort of a one-time export of those. So you do these heavy operations, but you only do it once and you save the results as aggregate data values. So you need then to basically take all your program indicators that you've defined and have a way of mapping them to aggregate data elements, exporting the data and then letting the users build dashboards analytics based on your aggregate data values. So the interoperability team has developed the tool for them doing the export of all of this is using the API. It is possible and many have developed sort of custom solutions for doing this. But now in the last, I guess, few months, we have an official tool for doing this. In the slides you see we have tools for the links for the tool for a blog post sort of describing how this tool is meant to be used. There is a link to the guidance on the tracker to aggregate and there's a link to a session we have on Wednesday specifically on this whole tracker aggregate integration. So if you're interested in this, please come on Wednesday. We actually, this is again, if I haven't changed the title, but everything here still applies. We have this guideline which Mike already pointed to. We also made this small sort of checklist tool which is aligned with the guidance helping you sort of go through and make sure you've actually done all the things that are recommended in the guidance for ensuring good performance. We even set up this email address which will reach a group of people working on this kind of performance issues. We set it up half a year ago. I don't think we've gotten a single email, so I don't think it's widely known. So this is basically because we see a lot of times there are individuals getting WhatsApp messages, Telegram messages, and it's becoming a bit sort of hard to coordinate what the global team is doing to address these issues. That was it. I think we're over time. I don't think we'll have time for questions, but I'll leave it to Mike to finish. Yeah, just thank you and apologies to our presenters who all get stuck together. They have a lot of good information. So corner them throughout the week, ask questions. That's fine. We have posted up the slides in the schedule and you can find the links there to the community of practice where you can also do the questions online. So thanks again for the presenters and thanks for joining us.