 This is the last one we need to record. What if I go and zoom in my computer? You guys can record it. Connect the HDMI coordinates. I have an USB-C. If I connect to zoom, then you guys can record it. Why do you need to connect to there? You guys are still connected to the zoom, right? I'm here. It's my Allison. I share the account. So I'll be there. I'll be host. No, the host. Well, co-host. Yeah, I'll be co-hosting. Yeah, I'm just co-hosting. So you guys are hosting. Yes. I'm just co-hosting. So I'm just co-hosting. I'm just co-hosting. With the priority. It's recording. That's where I'm. Sharing screen. It's good. You looking at the way. No, no. Sorry. Okay. We're going to change things up a little bit. I know you guys have been patiently kind of sitting absorbing a lot of conceptual information. So what I've done in the Moodle system, I've added two more links. And what we're going to do, I'm just going to demonstrate a couple of the features that I was discussing before. I've given you links so you can follow along. I'll identify what those links are. So you can log in and actually get a better sense, you know, for what some of these things are in the system itself. And then after we'll do an exercise. So no more presentations for today. Okay. And instead we'll, we'll do a bit more. We'll do a bit more hands on. Okay. And, and if they're like, I know it's tough because it's a long day, but please, if there are specific items that you want, or just generally want to see more about how DHS to works. And we can have some Q and a sessions afterwards. And we're happy to show you some specific features or walk through some demonstrations or just give you a better idea for how the system itself functions. I know we're not covering that so much during this academy, but we can talk to you separately about that if you have any specific questions. Okay. So for the first demo, I'm just going to cover, show you how validation rules works and some of the other features related to that works. If you click on this link here, analysis demo inside a Moodle, it'll open up a new or sorry, it'll give you this link, I guess, just click on the link. Okay. And then the login details are at the bottom of the page. Okay. I'm not going to go through a ton of features. I'm going to go kind of slow. You guys can follow along. If you'd like, in order to just, you know, get a better sense of how some of these features operate. Okay. So I'll give everyone a chance. Remember if you, it's on the Moodle page. So it's this link here. Okay. And I just open it up and it'll give me the login details at the bottom of the page. Okay. So the first thing that I just going to kind of explain a little bit, and just so you can see it, how it operates. Okay. Is the validation rules. All right. So first is everyone able to access the link any issues. Access and login. Any problems. No. Okay. I see everyone kind of, okay, so far. All right. So I'm just going to explain how validation rules work, both when you enter data, but also when you kind of want to perform this bulk analysis. Okay. And I'll just focus on that so we can kind of all see together. All right. So the first thing I'm going to do is navigate to our data entry app. Now some of this might be new for some of you. I apologize if I'm moving slowly for those of you who are kind of already experienced in using DHIs too. I just want to make sure no one's left behind. Okay. So if I click on this kind of menu button, this multi dotted menu button in the top right corner here. I can access an app called data entry. This is where we actually enter all of our aggregate data. Yesterday you used it in the exercise to enter data from your tally sheets. Let's make it nice and big. Okay. And we interact with the hierarchy. None of this is different so far. Right. All I'm going to do is select a facility, a data set and a period. I'll select this cardinal hospital. This immunization data set and let's select. September of. Let's like maybe something a bit more earlier February. Maybe. Okay. So you select your data set or sorry, your organization unit. In this case, I selected this hospital. This data set, which was immunization. And the period, which was February. All right. So when you're in this data entry interface as people are entering data aggregate data into your system, right. And you guys had a little opportunity yesterday to check this out and try it on your own. All right. So what this allows you to do one way you can check some of the data quality before, you know, at the facility level or at the point, I should say at the point of data entry in case you're entering data at a higher level. So basically, I mean, we're not going to enter any more data. You've already went through that. Okay. But you can run this. If you click this run validation button, it'll actually run any type of logical rules that you have set up for the particular program area that you're viewing. As they're entering the data. Okay. So if I select this run validation button. It actually pulls up in my example. An issue. Okay. And there's an issue with the stock in this case, right. The number is comparing the number of doses given with the number of vials that I've reported are used and wasted. Right. It's saying that for whatever reason, I've used more vials than I've actually administered stock or administered doses of this particular vaccination. So this is an example of internal consistency. I'm collecting all the data within the same kind of process, same form, same tool. Okay. And I'm comparing variables within that same form based on a rule that I've defined. Okay. So all these rules, like this doesn't come. You can use these tool kits that we will talk about later on to bring in some of this validation. Okay. But you can also, of course, define your own validation as well. All right. And we can run this as I mentioned at two levels. So one is here. You can run it at data entry and I can check and I can have, make sure I implement in all my procedures. Whenever a user enters data, they should click that validation button because they should actually check the data as they're entering it. All right. We have similar types of rules that we can set up at the point of entry for case-based data where you can check, you know, you can hide and show variables based on what they've input. You can indicate if a value is out of an expected range, you can let them know. For example, if they enter a BMI value, that's too high or too low or a weight value for a pregnant mother that perhaps is out of an expected value based on which trimester they're in, you know, other rules like this, where we can create prompts that allow a user to kind of immediately see, you know, if there is an issue with the data that they're entering. This allows in this case, for example, I can, I can kind of see what the error is. And then you would, you know, kind of show the user how to fix this, right? So if I go to the vaccinations I've given, so what is this again? DPT happy doses given. This is adding up all of these. Right. And if I just increase this value a little bit, maybe it was 60 instead. Oh, I can't write. Okay. Sorry. Just let me log in as mine. So you can see. Okay, so I see that change value and then I'll run it again. Hasn't registered change the stock. Okay. Sorry. Can't write the data. Okay. So the whole idea is that you're able to alert the user and provide some information as to how they can alter those values and update those subsequently, right? So you can try to catch some things. That's kind of the point of entry. If you'd like. So what I want to show you is actually, you can also do this. So what I'm going to do, if you go to, if you're following along, if you navigate to this apps area, I'm going to go to this app here, data quality. Okay. And this is where I can run in bulk. Validation rules that I have defined. Yes. So this is the dashboard for the people who are looking at the data as aggregate. So we have a look about like what's the dash and what is the dashboard for the people entering the data. It's would it be that complicit to you. So this is the data entry page. What we're looking at right now. So this is where they enter the data. They enter it as an aggregate, not. Well, it depends. There's multiple models. We're just showing the aggregate data entry right now. They could also enter it as case-based data. But in this particular example, they're entering it as aggregate data. So here's my question. Having this much as a huge matrix. Make the error. Higher. Because you have the PCG, the DPT, all of them in the same high, and all of the slots are the cells are near each other. There's no next option. So that's, we'll be confusing for the people who are entering the data. This is my high. Yep. This is just an example. You can change how it looks. Yeah. But I mean, many forms are kind of made this way in reality. We see many, many forms, much less readable than this, but that being said, you know, we, you base, this is just an example. You base these tools that they're entering data on your actual kind of data collection tools, right? So how they're configured, how they look, how they appear for the user, that's kind of based on your own implementation, your own context, right? So examples we saw yesterday, examples I'm showing now, they're just examples, right? You take everything, you localize it, contextualize it to your own setting to make sure that it works for you. I mean, you probably wouldn't want it all in English either, maybe in Arabic, for example, right? But all of these examples are that I'm showing are in English. We do have other languages, French, Portuguese, but nothing in Arabic configured, but we do have other examples where you could use Arabic instead of English for everything that I'm showing, not just the data entry pages, but all the warnings, everything, right? Okay. Okay. So what I'm going to do is just show you how you can kind of analyze these rules in bulk real quick. All right, so I'll just go to this data quality app. All right. And I'm going to go to this first one here called validation rule analysis. Okay. So for those of you following along, it's just the first box. All right. So I'm actually performing this manually, but you can also schedule this to run on a period of your choosing. In this case, data that I was looking at was collected monthly. Okay. I could have this run, you know, on the first of every month or maybe the 15th of the next month to give people enough time to submit their data. And then I could kind of analyze everything in bulk. And I try to identify any challenges or issues with my data. Okay. So what I'm going to do here, just because I'm doing it manually, there's a couple more steps, but of course, if you're doing it automatically, you would pre-define which of these characteristics that you're selecting. I'm just going to select this unit. It's a fake made up place called training land. Okay. The top one here. It's in green. And what I'm going to do is I'm going to run this for 2022. Okay. Instead of 2023. I'm just going to run it for the year. Maybe in practice, you'd only run it for a month. Maybe you would run it for a year if you're doing an annual review of some kind. Okay. Okay. So I'm just using the date selectors to select from. January 1st to, I guess I could say December. 31st of 2022. Okay. That'll run. Run whatever rules I will select. For the whole year. Okay. So what we do when we create these rules is we can create the rules out that way. We're not running. Maybe you want to perform a check. You just want to run it for immunization instead of malaria immunization TV all at once, right? So the idea is we kind of segment those out. So if I select this box here. It'll have a kind of a list in this case, of a couple of different programs. So I could run just the rules that I've assigned to that particular program based on kind of a grouping that I've made. is run it for this group called Immunization Thresholds. And I was showing you some examples of this predictive analysis in some of those previous slides on data quality, where you can define what your outliers look like. So what I've done is I've created an outlier for some of my immunization data. I'm going to compare the actual data that I've entered with the outlier that I've defined. And I mentioned, or maybe I didn't mention, but you can send a notification as well to a user to let them receive this data. So if I was running this as an automatic process after the process was run, if nothing was found to be incorrect, I wouldn't receive any message. But if something was found to be incorrect, it would send a message to the users that I tell the system basically to send those notifications to. All right, so just to recap, I've input which kind of area, geographical area, I want to analyze, I've input the dates that I want to analyze the data for. I've selected kind of the program logic or rules validation that I want to set it up for. And in this case, it's immunization. And I'm saying send a notification if you find anything wrong with the data, okay? And even though I'm doing this manually, all these same parameters can be configured to run automatically. Let's say every month, for example, all right. So, may I ask something? Yes, sorry. So basically you can set this to run every month for each district or each facility. And the system is doing the automatic check of the data quality without the system manager interfering. Exactly, yeah. So you can set this all up to run automatically for various geographical units in your system. It'll let you know if it finds anything wrong, yeah. But of course you have to set up everything initially, but once it's set up, yeah. So I'll just click on this button validate and give it a second. Okay, so it's output a couple items here and it's given me a description. The amount of rotavirus, if let me just pull up the details, I'm zoomed in. The amount of rotavirus to administer should be less than or equal to, this is not proper language, I apologize, I should really fix that. The mean plus three standard deviations of the last 11 months, that's how the outlier was defined, okay. So this is a particular, you can make formulas to define your outliers or you can just use other ways to bring in outliers into your system, okay. But because the amount of doses administered was greater than my threshold, it has identified some values that aren't quite in line with what I would expect, okay. And it's given me an output of this and to the appropriate people, it has emailed them to me, okay. So this is my email now and it sent a notification of all the different rules that have been violated, okay. So we can set this up to be automated, you can send it to the users that are supposed to receive this, you can provide it with descriptions, you can see here I have specific names, they're made up places, but specific facilities, I have the month in which this violation was created. So I could actually go and check, I could follow up appropriately if I had some procedures in place, if someone receives this of course and doesn't know what to do with this information, that's problematic. But of course, if you set this up and then they have some proper follow up procedure and they know what to do when they receive this information, it allows you to go in and modify values accordingly, follow up with facilities to ensure that the values are reported correctly, et cetera, et cetera. So this, I went through a number of features but just to show you kind of hands-on, you know what this looks like? For those of you who were following along, I think, sorry, where was my, yeah, you would have got here, okay. This is also useful because it allows you to review them within the application itself, right? You can pull up details and you can kind of see which value, as I mentioned in some cases, in this case, I have two values contributing to the total number. I can have as many or as few values contributing to my total, yes? Yes, again, please, how flexible the rule creation again? I'm sorry. How flexible, how much flexibility in the rule creation? Yeah, it's totally up to you. Based on what? Yeah, so it's totally up to you in terms of defining the rules, right? As I mentioned, we have a number of predefined rules that we've taken with the guidance of partners like WHO, UNICEF, UNFPA, for some of the programs, that might be a good start, but you define the rules based on any mathematical formula you might have, any type of internal checks that you want to create. Programmatically or there's a user interface to do that. There's a user interface, yeah. This is also here as an example here. So I'm not gonna go all through all the steps, okay? But if you do have questions, we can go through it, but let's just see the outlier, RV2. Just open up one. So this is an example. You just go into this user interface, you can select which variables you want to be part of the equation, you add them in and you say, okay, this is what my value should be. This is what I'm comparing my values to. This is what the behavior should be between them. And it's all defined by you in this interface and then you can save those rules and apply them to the program. And for most, anything that we're showing, dashboards, any type of analysis output, adding in data collection tools, there's user interfaces for all of these things actually. So there's not so much of a need, depending on the scale or what you're trying to create to kind of do something without an interface. All of these operations that we show you have a user interfaces to add these items. Yes. Could the validation rules be determined centrally, not on to the facility level? Yes. So normally, I would say in a normal case, they are defined maybe nationally and then the facilities use those. So the national level defines them for the facility. That's very normal, I would say. That's not to say if facilities have the capacity, they can't define their own and add them, but it's a typical case where there's a lot of national involvement in defining these rules for the programs, which also helps because there's standards to compare the data across the whole country. Yeah. Yeah, so there's not like you typically only allow a very small amount of users access to these interfaces. Otherwise it can create some havoc in your system. So it's really only people who are, we'll talk about this more later on, but people who have the right training, who are involved in the process who really understand what they're doing. It's not something you would allow just anybody to do typically. Yes. Yeah. It's just a comment on that one because those data elements being used for creating that validation is a data element being used by the whole users. So you can't allow users at some national levels maybe at a facility level for them to create different visualizations, I mean, data violations. In that case, users will be changing what has been created by other users. So it's good to give this feature for system administration and aeroized system administration in most cases they are at central level, but also there are some implementations depending to how countries are organized. For example, we do have some countries which have a kind of regions or islands where they prefer to have system administration at each and every island, but in any case, this I think it's good to be created by system administration at national level. Thank you. All right, so I hope that gives you some idea. Here we can, I could show a little more and then we'll kind of move on. So where am I now? Yeah. So I'm just gonna go back and I head back to the data entry app and we can select the same dataset. Say February, that was the one I was working with. All right, so a couple more kind of features to help the user at the kind of site level. So if I'm here and you know what I've just gone back to the data entry app, I selected my location, my dataset and my period again and open that up. Let's open up the dataset for me. You can select any, if you're following along, you can select any dataset at any period, okay? If I just double click on one of these boxes, it's gonna bring up this history, okay? And this can also be helpful if we just show them kind of the basics of identifying trends in their information. If they see some huge spike again here, for example, you can put this on the dashboard too, but at least at the point of data entry, this also helps if people kind of just have a little suspicion about their value because they'll be able to see, like this is pretty consistent. I mean, there's not, there's some kind of ups and downs but nothing really crazy. But for example, if someone hits zero one too many times, so I can do that, let's see here. For example, it's not registered, okay? Yeah, so if someone hits zero too many times or whatever the value was inflated or too small, whatever the case may be, right? They would be able to see some of the dips and increases in these values as well. You're also able to kind of take note of who entered those values, right? So you can see, for example, this I just entered this value now, has my name now, okay? If there are changes, so this is very useful if you have procedures for modifying the value, right? So if you note that there's an issue with your data and you end up modifying the value, okay? You can see who entered the previous value. You can keep track of then who modified the value as well, right? So if there's any kind of procedures around that, any type of approval or other kind of items that you need to look at, there is some information that allows you to track that information as well. Just really quickly before we move on, I'll show one or two more things that you could try on your own as well if you want. I'm going to be using the second link I've posted in Moodle, the data quality demo, okay? So either you wanna follow along now or you just wanna check it later on. We have a couple different demos for you to access, all right? So I'll open this one up and same thing, you can just use the login details at the bottom, sorry, at the bottom of the page here. Okay, so I'll just log in with those. And this is where you can view the WHO data quality tool amongst other things, okay? Oh, it's in French, that's okay. Excuse me. Yes. Yes, maybe I'm confused, but does it work as EMR, electronic record, health record management set? Does it work? As an EMR? Is it designed to work as an EMR? What is it at data analysis and reporting tool, yeah? Pure, okay, core. Okay, so I would be careful in terms of how we classify it, right? So I would not say that we use DHIS too as a type of patient interaction system replacement. I wouldn't say that it's kind of replacing a full EMR. It does not do direct hospital billing. If I'm consulting with a patient and I have no idea what their diagnosis is and I wanna enter in a bunch of notes, patient notes, I wouldn't use DHIS too for that specific purpose, okay? Yeah, I mean, yeah, the life cycle of registering a patient, registering a visit for the patient, dispensing the medicine, follow up appointment, et cetera. Okay, yeah, for that type of thing, that's much more appropriate. If I have very specific services that I'm delivering, maybe I'm delivering certain delivery, let's say it's an ART patient, for example, okay? And they come, they receive their treatment. I want to know when they're coming back. I want to know what medicine I've given them. It's a very well-defined treatment protocol. Yeah, patient treatment, et cetera, follow up. Yes, yes. Then in that case, we can think about kind of case-based models for that and we can use DHIS too in those scenarios. So it is not a full kind of electronic medical record replacement. Just again, for me to understand, I know there's a core functionality, right? You can build on that. Yes. So you want to build on that, what I'm talking about. It needs a lot of customization or not, something to apply. Okay, yeah, so there's two things, right? How complex, yeah. Right, there's two things. Let's look at it this way. I'm considering implementing DHIS too, okay? When we refer to kind of localization and customization, this is not extra, like what I would consider development. We're not creating a new application. We're not kind of having to hire a bunch of developers, programmers, okay? What we're doing is we're trying to understand for our context what health programs we want to implement, what data we're collecting, what indicators we need, what type of outputs we need. This is all configurable in the system without having to create an application without having to kind of extend the platform. This usually takes us 80 to 90% of the way, but then there might be something missing, some type of custom report, some type of interface you need, some type of other solution. And in those cases, you might create like an application or something like this to extend the kind of features within DHIS too. In top of it or something separate? On top of it, yeah? Same system. Top of it, same system, just on top of it. It's a platform, right? We're gonna discuss this more in the coming days, but the whole idea with customizing DHIS too, it just comes as an empty box. We decide what we put in, right? So if you wanna add stuff for your country, you add your facilities, you add your indicators, you add your data collection forms, you add your reports, okay? And that doesn't require some type of extra development work, I would say, okay? But in every context where DHIS too is set up, you go through that process. Because it doesn't come set up with any of those things specific to any one country, yeah, okay? Okay, so we're just gonna switch over to a little bit. If you have more questions on the data quality features, wanna see more demonstrations, please come grab us, okay? And we're happy to go through that. It's not a presentation, don't worry. We have another exercise for you to work on. And I think we've spoken enough about data triangulation and for you guys to have an idea of what that is. So we're just going to do a little exercise to get you thinking about this a little bit more. Abdulrahman and Hanin will help explain some of this in case I'm not able to convey it to you well enough. I'm just going to try and explain it to everybody. And then if you do have some additional questions, Hanin and Abdulrahman will help me out. Okay, so I'll just open up the exercise, okay? So what we're going to do, and just to go back for this exercise, we're going back to the original demo that we set up. So we're gonna use this demo for the exercise, okay? The DHS2 demo with the RHIS login, okay? And let me go to that. So in this system, we have, in this demo system that we've set up for you guys, we have two dashboards that we've set up that are triangulation dashboards, okay? And what this means is we have data from multiple sources feeding in to all the outputs on the dashboard, okay? So what we're going to have you do in this exercise is just review those dashboards a little bit and kind of think about in your own country and your own context, what you could do to implement something similar potentially. All right, so if I go back, okay? So I just list out all the instructions. I apologize, it's all in English, but like I said, Hanin and Abdulrahman will be around and can explain a little bit more, okay? So there's two triangulation dashboards that are available. One is called immunity gaps, one is called program performance. So you can access these dashboards when you log in to that RHIS login demo that we've set up, okay? So these dashboards take data from three different places, okay, one is routine immunization, one is stock and one is vaccine preventable disease surveillance. Both case-based data as well as aggregated data, all right? So what I would like you to do, I have three questions, okay? You can open up that dashboard, just have a look and you might wanna do this in teams and groups, okay? Because different folks are going to have different input as to how to kind of interpret some of this stuff, all right? So remember we're in this system, okay? The system link is taken from the course page, the DHI's two demo, okay? That's where we're performing this exercise. So you don't have to follow along with me right now, you're gonna get chances to work on your own, okay? But there's a couple of things I'd like you to do, okay? I'd like you to look at the items in the dashboard and try to identify where the data is coming from, okay? Which of these three programs are inputting into the different outputs that are available on that dashboard? So you might need a little bit of help from someone who's familiar with health, the health domain, maybe a doctor on your team, okay? Who understands that process. Again, okay. Okay, so the first thing we're gonna do, that's better actually, okay? We'll just log in to this system, okay? The link is on Moodle, the DHI's two demo with the RHIS login, okay? And I'm going to ask you to look at all the different visualizations on the dashboard called EPI immunity gaps and try to identify where that data is coming from. So for example, I'm looking at a different dashboard program performance. This is the second one. So if I look at this first one here, for example, I can identify that this data is coming from both case-based surveillance as well as aggregated surveillance. That's what this is representing. I have something down here that's coming from a different source. So what I want you to do is go through each visualization and try to identify the source of that information, okay? Okay, then there's a couple, yeah? Okay, I think they can help you a bit more with the specific questions, right? So the next question is thinking of the data in your own context. So in your own setting, whether you have DHI's two or not, okay? Are there programs that you would want to present together in this manner? Okay, so are there kind of program areas that you would want to perform triangulation on, okay? And then once again, thinking through your own context, your own country, at the moment, are you able to do that? Okay, so if you want to do that, are you able to actually set up analyses where you're combining information from multiple sources? Okay, and if not, then what are some steps you could take to kind of make this work in your own setting? Okay, so I would suggest you work in country groups. I think Hanin and Abdul Rahman will help me split up some of the teams. So you're made sure to distribute equally, but for those of you in smaller country teams, you can just stay together, okay? You can have a look at the different components of this exercise. If there are questions, I can try to answer. And then as well, Hanin and Abdul Rahman are assisting me and they will also be able to help you explain how to perform this exercise, all right? So I'll give you guys a little bit of time to work through this in country teams and we'll go from there, okay? Yeah, I can post the answers to you later, but we can also discuss a bit tomorrow, yeah.