 Right. So the last session for the day is about analytic tools implementation considerations. So this presentation unfortunately is going to be discussion of a lot of concepts and experiences and best practices. So it will be the slides will be somewhat full of text but we will try to keep it as interactive and interesting as possible. But this is a really important consideration, really important presentation that all of you will have to be mindful of if you are especially representing national level of a Ministry of Health or an organization who will be in charge of implementing data analysis and data use in your context. Okay, so regarding the word of the day. We are having some issues our team is looking into that and they will fix it and let us know. Okay. Right. So, first of all, now when we are implementing DHS to we have to be really mindful of local requirements right so the thing is what I mean by local requirements is now in this economy we mainly present to you about how in a very generic way you can implement your analytic tools. So we have discussed about how you can set up your visualizations, how to put them in dashboards, and how to create data to action frameworks and basically interpretations, right. So all these are conceptual things. And, as I have been always highlighting, when you go back to your country's context, you have to decide out of all these good practices that we have learned in the academy. How can I apply it into my country's context. So, the thing is like, even though we talked about this in very generic way. It really depends on your program managers, because everyone interprets the way they they have to use the data in the country context or in the program context in a different right. So you may feel like maybe your program manager or the medical administrator health administrator who's in charge of the program might decide okay, we will have few dashboards or just one or two of them. Or as we may have many of them. And sometimes you may, they may even think, right, DHS to may not be able to serve some requirements, right, they may have different opinions. So this is when even some countries might decide. There are some limitations in the existing DHS to analytic tools. So what can we do about it. There are so many things that you can do about it. We will talk about what we can do one by one. But one thing that we have noted quite often in recent times is to have custom applications developed on top of DHS. So I'm not talking about a separate website or separate web application. I'm talking about an application which is installed inside DHS to, but it has custom visualizations, not the usual layouts that you are seeing in data visualizer pivot table and maps right so these are very custom visualized custom layouts custom interfaces that you have in a custom web. So this is one extreme way of having the visualizations we required incorporated into DHS to instance. And also, you may have to provide training on using the apps to generate the outputs along with interpreting, right, and using these outputs also has to be catered in a country specific way. So this is again, one of my responses to one question asked about how to set the action parameters in data to action framework. So it is country specific, we have to decide and we have to have a discussion with the relevant stakeholders and decide what are the limits that we are setting right. And in general, what is even more required is to prepare a culture of regular data use in the country. We know the situation in most countries, I personally have worked in several countries in different Ministry of Health scenarios. So it's more or less same in most of the countries it's just that the only the degree of the data use culture changes right. But in general, most of the countries, at least at the field level. The value have the majority of the users, they are mostly collecting data. They are not using the data, they just collect, but they are not looking at how to, I mean, what is the data we have collected. Is there any relationship with between different data that we have collected know this thing is not happening. One most difficult but effective task that you may have to do is to design a culture of regular data use. So there are different ways of doing that, which are like I mean, depending on the socio technical scenario in the country. But you definitely have to create this culture of data use for it to get implemented. Okay. And then, when you initially go to a country, right, and when you assume the role of a analytic tools expert. What you can do. So what maybe good starting point is to outline the different outputs that are currently generated and being reviewed in the, in the country. So, this may be within the DHS to instance, or maybe outside. For example, if you, if you are trying to start something fresh. There may be three possibilities right one is the country doesn't have electronic information system or at least a program doesn't have any electronic information system in general. They are they are just referring to, you know, the charts that are hanging on notice boards and places like that. They may be using an existing third party system, which may have some very specific visualizations, or else they may be using DHS to. Right. So, what you can do is in case it is not in DHS to you, you have to analyze the situation and see what are, I mean out of these visualizations, which are presently there, which are the ones you can implement it in DHS to. Or else, also look at like, if you have a DHS to instance, you can have a look at the existing dashboards and visualizations, and you can think how you can enhance the features that are currently there. Maybe the visualizations are too many or too overwhelming. They don't have proper interpretations, which are embedded so that people actually can't make any proper. So these are the things that you have to initially take into account, right. And then you outline what are the new outputs that are possible with the newer versions of DHS to. Right. And again, you can assess if any outputs may require custom solution. Right, so it may be a custom application within DHS to, or maybe we may have to send data which are there in DHS to outside. So now I'm mentioning few never or high tech approaches which you may not be familiar with, but just know like these things are possible. So for example, we can even send data out of DHS to into another plan. This is possible, if it really requires. And also, you can what you can do is you may be able to provide a demo using in country data. So for example, if you're trying to introduce DHS to to a totally new set of people. If possible, you can try to you know set up a demo instance with their own data. So when you actually see your own data, if any country person sees their own data. They are in a better position to actually appreciate what you have done in the DHS system rather than you try to log into the play demo and show Sierra Leone data. Right, so I mean that's something that you can always think of doing. Maybe a bit difficult if you don't have proper access to data, but if the country things like maybe you can have initial discussion and then followed by that if they want to in detail discussion. At that point you can use this in country data so that you can go for a better discussion with the state. And always this creation of analytic tools and creating this data use entire mechanism. It's a multi sector collaboration right so you need you always need subject matter experts. So you may have seen like when you are doing this discussion at group level. So if you are familiar with how the immunization program works. Some of these things may not make much of a sense right, especially when someone asked what would be the cut off level we should set right. This comes with experience and the knowledge that you have in the program. So definitely you need subject matter experts as well as the DHS to experts right both these people are required otherwise subject matter experts might just go and present to high level stakeholders of things which are out of the scope of DHS to. So you need a good collaboration with both these type of people to achieve a better implementation. And then you should these discussions after these discussions there has to be a proper roadmap in which you will be outlining how various outputs will be implemented in DHS. Okay, so the thing is like you have to discuss what are the requirements that they want in their visualization outputs first. So for example, they may just ask a couple of charts. But then, once you show them these charts they may ask, okay can't you also do something like this. So, I mean, like, what can actually happen in this initial discussion is one thing is they may get too overwhelmed with what you are trying to present. There will be a like a flooding of requirements that are coming from this, the program aspect, where you may really struggle to implement everything. I mean, implementing everything is not customizing DHS to instance right I'm talking about the end to end solution, where you customize your train, and you create a mechanism where you can do a proper monitoring of whether they are using it. So to do all that, trying to overwhelm with all the possible outputs in DHS to and the visualizations is not a good way. So this is where you can create a roadmap and mention okay we will implement these few in this program in phase one then phase two phase three like, right, so maybe if they won't total custom, you can, you know, like, keep it to the last or maybe just try to implement with a smaller set of users, so that you can closely monitor whether it is being useful or whether they actually using it because this custom developments take a lot of resources, a lot of time and effort as well as money. And in case of an existing DHS to instance what you can actually do is you can review the existing output and see whether people are actually using them, or sometimes they may be nobsulate DHS to versions and if you can actually upgrade to a later DHS to instance the visualizations can enhance better. This is something you can do. I mean something very simple most of the time that you can start. And then you can also consider developing new outputs which are not there. So for example, two analytic applications that we tend to discuss in the past which we did not discuss this time as co-card and WHU data quality app because we felt that we cannot cover the entire content within these two weeks we didn't include those two, but like there are many applications which are enhancements that are available in the DHS to app hub you can use right so using them you can actually develop new outputs. Like whatever you develop you have to be mindful that you better test it in a safe environment before actually making them available to the end users otherwise they might get confused. If you tell them that you should be seeing this but then they actually not seeing it. So that is one thing and then again what you can do is maybe you can have an initial pilot phase to get feedback from the end users on the visualizations that you have. So, any questions up to this point. That's mostly being me working. I heard that word of the day should be working now. So you can try it in the meantime. Any questions up to this point. Okay, in that case we can proceed. Right. So, when we are setting up DHS to now I know that this is a analyticals academy and this is not a server administration academy but the thing is, you have to know, what is the infrastructure and the resources that you have in your country setting right because the simplest reason is the analysis that you are trying to develop and visualize and the number of hits the DHS to instance is getting for a given analytic visualization as a major implication on the server resources. So why I said that is, most of the DHS to server resources are consumed by him. So for example I can mention a very fresh scenario that we are going through in Sri Lanka so we are having a COVID immunization tracker a national level immunization tracker in Sri Lanka based on DHS to right. So, and then we have the entire country population pre registered adult population pre registered in the DHS to track. So we are talking about 16 million track entity instances in our vaccination DHS to instance right. So every time we make analysis visualization based on say like number of people registered number of people number of men registered female registered. So all these require, you know the database which is like, which is, which is everything is working in SQL. Right, so all this visualization that we request from the DHS to instance are consuming too much of resources from our servers right. So this is where you have to see you have to have you have to be mindful of like, I mean this particular visualization is it available to all the end users, or if it is a very resource heavy visualization is it only available to few people at National. So all these things you have to keep that in mind. Right, so this is where sometimes in most of the countries they have a production aggregate data instance and sometimes a data instance so these are two separate DHS to instances. And the other thing you should not try out everything on production instances because one thing it can get really cluttered your production instance and nobody will know what how many visualizations favorite items are saved, which are not actually used. Right, so a better practice would be why these two or like sometimes some countries just have one DHS to instance for production, you should always have a development instance for each of the production DHS to instances so in case if you have a separate aggregate data DHS to instance and production DHS to tracker instance it's always good to have separate development instances as well. So you test it in these ones. And once entire thing is I mean you are satisfied with the performance and everything, you can push it to aggregate instance. So that's that's a good practice. So next we will talk about something very important when it comes to implementing analytic tools which is about the country capacity. Okay, so implementation of analytic tools essentially require a wide variety of skills. So this skill set that we are talking about should include understanding of all the different analytic capabilities that are within the DHS. So for this one you need some DHS to expert who knows what is possible in DHS to and more importantly what is not possible in DHS. It's otherwise you might commit to saying that we can do it in DHS to and you find that it is not possible. So you will cause disappointment at your level as well as the program side. And then you need the configuration required to support the creation of these outputs so you need to have access and know how to configure. And you need to have knowledge on how each analytic tools can be used to generate specific outputs based on the different requirements. And how these could potentially be leveraged by custom solutions if required. I'd say for example, if certain output is not possible in DHS to within the DHS to how can you use a custom solution to complement what is already there that knowledge you have to have. And then you need also the program specific knowledge on how to interpret and utilize the data that is produced by these outputs. So that's also required. You may have all the technical knowledge. If you don't have the program knowledge you may not know which visualizations to create and how to use them. And because of that, we need subject matter experts right so we need people from program sites and then we also need the implementation and configuration stuff right. And so there needs to be a discussion between these two parties at least on designing what are the outputs that are required. And it's always because of that it's always a collaboration that happens frequently with these two entities, and that there has to be a very good sync between these two entities to when you are designing the dashboards and the visualization requirements, right. And also you may need support to maintain the infrastructure which we discussed previously, right, because you have to know like if you are having a very large data set, or like you're going to have large number of induces will be accessing system, you will always have to discuss with whoever from the infrastructure side to see whether you have the relevant infrastructure available and the support is there to make it. So, when it comes to training of staff, it's always a challenging business, right. And this is where you have to review the each of the outputs and visualizations, right, and decide which visualization should be made available for in users at which level in your say organizational high right. So, based on that you may also have to evaluate the type of training that you have to provide at different levels. Okay, and then you may also need to revise the existing available outputs and see whether these are being used properly, or maybe you need to focus on feedback group, right. So that end users will be provided some knowledge on how to analyze and how to define outputs that are generated at their level, right. So all these have to be considered when you are designing a training program. And again, something which is also more important is how to provide continuous support. So this training program we may be able to do once. So what are we going to do for people who miss the training. So for example, when we are doing this academy, what can we do for participants who don't join the live session. So we need to have a backup. So similarly, you also have to think what to do if I mean like if someone I mean if they require refresh that way. If we need to you know like plan the trainings in such a way where we will have a couple of levels. So we train them on some basics first, and then let them use a system for a couple of months and then we may have an advanced training likewise. So all these have to be thought about well in advance before you conduct the training program. Otherwise, what might happen is like, even now okay now when we are doing this training. We always define our boundaries right we mentioned up to this point we cover it but these particular components we are not covering in this training program so likewise you will have to define it in advance. Otherwise your training programs will be so cluttered and you may disappoint everyone because like somehow because you may get a mix of participants with different skill sets so that is again something you can think of how to categorize filter out participants that you are getting for a separate training program. So this concept on how to organize a training program maybe not just relevant to analytics but point any type of implementation in general. So with this, another important aspect that we have to think is think about is whether we have a DHS to 40. So this I will mention like in most of the countries, if you have a HMIS, health management information system based on DHS to then we are thinking about DHS to national level coding, but in some countries. So for example in Sri Lanka, we have multiple DHS to instances run by programs. So in that case that program has the total authority over that DHS to instance. So then we are talking about DHS to four teams for the programs. So it really depends on the context that you are trying to apply. So you need to have this core team well in place and established before you starting the DHS to implementation. So this core team does not have to be a formally organized one but what we try to mean is like you need to have this capacity. So they have to I mean this core team can include the program staff who are experts on various on the particular health program or if you are talking about the HMIS on like broad set of programs like someone from hospital systems one from preventive health care systems one from community health system like. And then we need to have implementation staff will be responsible for configuring and conducting the training program things like that. And then you also need the it technical staff right who can support maintenance and upgrade of all different DHS to systems right because they will be the one who will be maintaining the server part right and if you want to have multiple DHS to instances or else if your analytics are consuming too much of resources how to fine tune the server resources so for these things you need a technical IT staff as well. So now you will realize like it's a kind of a mix of people that you require to have a DHS to code it but of course in very resource limited settings as well as in some very context specific scenarios we may have a couple of people who are kind of cross expertise like you may have hybrid people who are experts and program experts as well as who has this implementation and configuration capacity so but in general you need to have a team who has all these capacities. Right and this core team should be involved in training activities together in order to foster a team and where we can have ideas exchange. So because the each aspect that is coming from each of these categories are important so that always the implementation is the one that lead no not really then I mean if you think of it like that then there may be frictions unnecessary frictions which might affect your implementation. And then to solve problems you have to get opinions from different people, right, and then you also need to appreciate each other's role in the system. So you can't run a system without having each of these categories, otherwise it's going to fail. So always respecting what other person is doing. Right. So any questions up to this point. We have a few more slides which I am trying to finish faster but because we talked in last few slides mainly about capacity and DHS to core team any questions we can discuss at this point or as I will proceed. So the thing is like after doing all these things till it may be very difficult to address all the gaps in the capacity because that's how the most I mean that's the scenario in most of the countries when you try to start DHS to implement the DHS at least in the very low resource countries. So in that sense, if this happens then you may have to find some expertise which is available outside of the country, right, or else. I mean, like, but then there is a kind of issue with the resilience of the system when you try to depend totally on outside expertise. So you can obtain expertise from outside the country but then you need to build capacity from using that expertise, right, you should not totally depend on expertise which is outside of the country because that's not going to be that much sustainable. So at least a moderate level capacity has to be there for in your country within your country for you to sustain instance. In the meantime, you have to identify the gaps which are there, right, and, and create a plan of action to expand your country's capacity over a period of couple of years, because capacity building is not something that you can do in like one or two months. So you have to have a yearly plan with the proper budget allocations and things like that so that is trainings and the capacity building takes on. And most of the countries which have successful implementations have gone through this process and that's how they are. I mean, they have very good implementations it's not something that can that you can achieve overnight. So that's one thing you have to realize if you study the situation of all these countries who have successfully implemented as to this is the secret behind their success. Right. And then, as I mentioned before, if you don't have the capacity always make include those steps in how to obtain the capacity from outside in your plans. Okay. So, in case the relevant DHS to relevant requirements from the program, we cannot meet them with the standard analytic tools, which are available in DHS to what are the other options we have. So for these things we have custom tools in DHS to. So for example, we have three main customers. The first one is standard reports. And then the other thing that is a common requirement nowadays, especially with this call it is public reporters. And the next thing is customers to basically what standard reports mean. I think we briefly discuss about it when we are doing the reports up is like in case there are requirements reporting requirements or to produce there's a requirement to produce a document report in a specific layout. So what we cannot just do by ranging various items inside the dashboards, then what we can do is like we can make a make something was standard report. So basically the standard report is a very, very cost, I mean a very specific development. So you require to design something in HTML using sometimes the DHS to baby I. But ultimately by doing that you can design a very custom made report in the DHS to that in case your program requires. The advantages, it will really, you know, please your end users and the program because they get what they really want the proper order. But the downside is of course, you need the expertise to do some additional development and coding. So that you require to have the public web portal sweet what we simply mean is like, most of this DHS to information that we have in the dashboard and everything is only available to a user who has access to DHS to but in case if we want, if our country's context over the program requirement. In their own website, or is to have a separate website, which displays output of DHS to visualizations, then this is what we mean by public report and custom application in general what we mean by that is any, you can think of any third party solution, which is not DHS right so they have custom screens or custom user interfaces, right and reporting generation visualizations things like that. But what we try to do is, we try to do this custom development inside DHS to as a customer. So I will briefly show you a few examples for each of them. One example for standard report that we see for my area so you can see here, the layout and visualizations that we see here are very different with some which we cannot just achieve by having a dashboard. Right. So here, what happens is the data the filtering of data and which data items to pull, as well as how they are displayed and laid out in the report everything we can change. The precise, you know, like very precise locations we can change, but to do that we need capacity on it's at least on handling HTML and JavaScript as well as the web API. So with that you can develop this, this standard report. Right, so this is a possibility you can always mention this is possible, but we need some expertise. So here it's an example from law on a public dashboard so what they have done is this is a website, right, public public web portal, but the data is pulled from their live DHS to instance. So the end users did basically not the end users the general public they can just visit this website. I mean just like any other website that they visit, and they will not ever log into the DHS to instance but they can still access the data. And then again this required some close careful development so that we don't compromise the DHS to instances security, but we also in addition, give the required data to outside, but then this again requires some tweaking and development work to get this public dashboard and then custom applications. So for example, what you are seeing here is one custom visualization we have done for contact mapping contact tracing visualization right so. Now this contact mapping and contact tracing is a major requirement which is coming with COVID pandemic and other communicable diseases. But we don't have this mapping visualization available in the co-app in DHS. So this is one visualization we have in fact a custom web application our team initially designed in history run come but later on, they were like we identified there were generic requirements coming from multiple countries. So we release this as a generic web application so available in DHS to have. So if you go to DHS to have up there are there are a number of DHS to custom web applications which you might which might even fit into your requirements. So please check, check out the app hop and see the apps which are available and some of these might be useful to produce custom analytic outputs. And this is again another custom visualization we have done for ICU bed tracking in Sri Lanka. So these are all examples of custom visualizations and analysis, which are possible in DHS to. Right, so with this we come to an end of the presentation that we have an implementation considerations. You have. Hello. Yes. Question. Yes, please. Yes, the custom tool is very interesting because it seems we can have bought to do anything that is in DHS who cannot. One more thing I'm worried and I also fast through a simple like the data indicator or program indicator that we can catch a lot in consistent. So we need to check back and forth every time. And my, my problem is a solution anyway that we can post a visualized with people, people table or data visualized to the end user because the data is not just first and inconsistent. So when the user look the chart or the custom report so it will string or it will something a problem and need to back and forth again so any DHS who can post when the data not yet ready to public. Sorry, if I understood it correctly, do you mean like whether we have a solution to push the outputs visualizations that we have in DHS to like give a table and visualize maps and things like that to the user level, maybe without getting them logged into the dashboard and see is that your requirement. The question. Yeah, it's post to the dashboard but our data is not consistency so we need to take a time to first or do something before the dashboard. Okay, so is it just that you mean like you have some visualizations in DHS to but you want to put it into a public facing portal and you want to do some modifications before it goes to public facing portal. Is that so. All right okay so I will answer the both the questions to the first I mean the first I understood like is there a way to push the analysis to the end users or like I mean program users without asking them to get to DHS so it's like I mean I receive a daily summary or a daily report of visualizations as a report in email. So this, this is what we mean by push analysis. This was a feature available in DHS to it is still there but it has some some some major deficits. So my answer to that one is like we have this feature in DHS to call push analysis, where you can send the visualizations maybe a dashboard to the end users email, we can set it up so that it automatically gets sent every morning like that, but it needs some develop it needs some fixing some co functionalities with the last few versions. So that is being attended by the DHS to core team right now so let's hope at least maybe in 2.37 we will have a good version of push analysis so I mean that's one thing. And about the next one of course it's a tricky one like if you want to like review the data and push some specific outputs to the public portal then you need to do I believe you need to do some some additional development on top of it because right now what you can do is you can set up this data approval mechanisms and things like that so that we can determine only the approved data is visible. I mean, I mean that kind of measures we can do, but even after that you have to decide like, even out of the data which is available to the DHS within the DHS to we want to refine it further and push it to the public portal then I guess we have to do some custom modification, we are to decide like how at which point we are doing this, I mean approval low pushing that thing. So that's what we can do. So what if you want to add anything to that. Yeah, yeah, yeah, thank you. But I don't know keeping call is easy to implement. Yeah, it's a challenging task but again, it's not impossible but you'll have to really think through what you want to achieve and how to do it, but it's possible. Yep, thank you. Any further questions. So if there are no more questions. Thank you so much for we have taken additional 20 minutes. We expected it because like today we plan to do a lot of engagement so we had to put some additional time for that as well. But I hope that you found today's session and the way we conducted it. It's manageable with regard to engagement with your colleagues as well as the facilitators. So, I guess that's all what we have for today we don't have any graded assignments today, and the word of the day. Hold on I should display the word of the day for participants who are referring to the offline presentation so this is the word of the day for the day descriptions. Please proceed with the ungraded assignments which is available in the edX. And then also, please don't forget to give us the feedback we really appreciate and we go through the feedback every day. So please give us the feedback for today's session and any previous sessions that in case you have forgot. And also mark the attendance the word of the day, because this will expire tomorrow morning. Yes, I think that is all what we have for today and tomorrow is the comprehensive exercise and before that sorrow, you want to brief the participants on next two days. Yes, so we have the comprehensive exercise tomorrow will do that on edX. So it would be set of questions, which you need to perform on the exercise instance and answer the questions and be there to support you in case you need any help with any of the questions. And then on Friday you'll have the final exam which would be set of multiple choice questions. Based on the topics that we've covered in the last 10 days. So, we hope you guys to see you tomorrow and day after because both the comprehensive exercise and the exam are your major components for qualifying for the passing certificates please ensure that you're there and appearing for both the comprehensive exercise as well as the final exam. Thank you for joining today and we look forward to what's full attendance at least next two days so that you're able to pass through the major related part. As you've covered all the topics we intend to cover in the general. Thank you. Hello. Hello. Yeah, I just have one question. Yes, what is what will be the timing for the exam. Yeah, the timings would remain safe would be one to four but the exam would be would be around 40 questions you'll have around one hour 15 minutes to do that so once you're done with that then you could leave. So there's no need to settle for the timings the duration would remain same one to four. So we have to be there on that time only or we can do it later on offline also. You can do it offline but we would suggest you do it together so that if there any computations needed that we can do at that present moment. But I'll also try to figure out till the exam is active or it's open for a certain period I'll update you guys on that. Yeah, because tomorrow day after tomorrow it will not be possible so I was thinking to ask you. Unfortunately, I think the for the exam, we will be able to raise your concern to the, I mean, the academy but usually what we do the exam the final exam, we have it open to everyone at the same time. Due to of course some logistics issues in reading and things like that. So, but but we will still try to raise it but for everyone else for time being just try to be present day after tomorrow because it's going to be online as of now but we will be able to get back to you on this tomorrow but. Just one query if we join late because I'm busy till 2 to 30. I think we can discuss with the team and get back to you on this. Okay, okay. I'm talking about day after tomorrow tomorrow I'm available. Right. Yeah, tomorrow of course it's not a major issue because it's a company to say something similar to what we are doing last couple of days, but exam is the middle of a tricky one because we expect everyone to do it together. But if you just ask if there is any possibility of joining late then I can join by 230 or so. Sure, we'll get back to you. Okay, thank you so much. Thank you. Right so there are no more. Yeah, there are some questions in the chat sorry now just now only I'm seeing it. So many more messages. Any announcements. No promote I answered the questions on the zoom chat, and I'll check the slide with any questions and put the answers to those questions there. Thank you so. Okay then. So thank you everyone for, I mean, being with us, even though it was a kind of a lengthy session. So see you all tomorrow for the comprehensive exercise please don't forget to give us the feedback and mark your attendance, and tomorrow we will do the comprehensive exercise. So thank you very much.