 All right, so now we are going to take a quick look at how we are able to within tracker programs perform some data quality checks. So in a quick overview, we've just talked about the tracker to aggregate or actually vice yeah tracker to aggregate approaches. Let's talk about then some common issues with data quality in patient records. Then I'm going to do a really brief overview of how we can build out some program rules. These are some logic rules that you can put into tracker programs for data quality checks. This is not a tracker academy. We have academy specifically on building out tracker programs and goes into a lot of detail on how to build out program rules. I strongly encourage you to go to one of those to get a very detailed overview or skill set of how to do this. I'm going to give you some really basic overview just now. Then we're going to talk a little bit about how we can monitor completeness through program indicators. Again, I'm not going to go into detail on how to build program indicators. There's another academy for that, but I'm just going to give you a brief overview of how they can be applied for data quality. Let's talk about some common issues with data in tracker programs. Most of us are building very complex data entry forms in tracker. We're doing the same thing in aggregate as well. We're not building in a lot of workflow support to these forms. It's a problem in aggregate. It's even a bigger problem in tracker. If we're capturing individual patient data, we have got to put in a lot of workflow support. I'll give you an example of that. There's often a lot of questions. Can we calculate coverages based upon tracker data only? The general rule of thumb is probably not. You probably don't have the entire population coverage using just your tracker program. How do we then get calculate coverage based upon our tracker data or in combination of tracker and aggregate data? Then calculate completeness from individual patient records. Another tricky thing is when is a patient actually complete? All right. Let's first go into that workflow question. How do we address that with program rules? Well, it's actually easier for me to show you than to go through it. I'm going to give you a quick example. You're looking at a DHIS2 database for mass drug administration. What they are actually doing is they are using events to capture mass drug administration, so mass drug campaigns. I'm going to go into the capture app. This is not a demo database. This is actually a database from a real project. You're not able to go into this one. I'm just going to show it to you as an example. Then I'm going to go into an org unit. I'm going to select one of my programs. This is the capture app. This is a different app than data entry. I'm going to select one of my programs. Now this is an event program. I'm going to click new event. Now I'm opened a data entry form for an event program. In this case, it's a mass drug administration for LF. LF is lymphatic filariasis, which is a parasite drug. If you're doing MDAs, mass drug administration, for this particular disease. What I'm going to start doing is putting in some data here, just to show you some of the warnings that pop up as I'm putting this data in. I can just put in a contract to the implementation total persons. Tim, here's the first example of workflow support. In programs, I can put in boxes or I can put in questions that if I answer will prompt additional questions. Again, workflow support. We don't want to build in as much workflow support. As you're filling out questions, you want to be able to hide certain answers or you want to be able to show other questions based upon a certain answer. In this box, I have a tick box that says yes, and if I do that, then it opens another field that I need to fill in. I'm just going to start putting in more information here. This next question is a good one. We say year started. Here, I should be putting in a year value. I'm going to put in a number that's not a year and look what happens. I put in like 200,000. We're not even close to the year 200,000 yet. It says invalid year. This year must be between 1990 and 2020. If I go to 2000, see the warning goes away. This is a great example of a simple program rule. If I put in a number that doesn't make sense, it's checking the value that I'm putting in. It's telling me in real time if this is an appropriate number. I'm going to put in just a couple more. I'm just putting in random numbers right now. You can see I've triggered another data quality check. In this case, what is it saying? Well, I've actually got an error box in the top right corner here. It says persons targeted cannot be greater than the population eligible for PC. PC is a treatment process. Here, they say, okay, look, I've got 10 here, and then it says person target cannot be greater than the population. Then it's telling me, okay, check here, check here, check here, and great. I have 10 here, but I have 10 there, but this isn't saying it's not right. If I just put in 10, or zero, and zero, great, now it goes away. The sum of these could not be greater than this. That's what it was. When I put in the zeros, it went away. All right, now I've got another data quality check. I just put in some more random numbers and saying the sum of male plus female treated cannot be greater than the sum of pre-sac, sac, and adults targeted. These are aged breakdowns, and it's saying the sum of male and female treated cannot be greater than these values here. What did I do? I put in 10 male plus 10 female, that's 20, and then the sum of this is just 10. Let's try to fix that. I put in zero here, great, now that went away. I just put in some more random numbers, and guess what? I got two more errors saying the sum of male and female, this male should be equal to the sum of treated. Okay, so they're saying that this sum male female should also be equal to the sum of these three, and then it's saying also the sum of pre-sac, sac, treated cannot be greater than the sum of pre-sac targeted. So they're saying this sum cannot be greater than this sum. So let me go in and fix that. All right, so now all my data quality checks went away. So I just wanted to give you a quick demonstration of kind of program rules in action. So this is a real project. They're entering data across dozens of countries actually in this particular program. Hundreds of people are entering data using this particular reporting form, and you can see all of the various data quality checks are just an example of some of the data quality checks that are coming up as they're putting the data in. You can also see some of the workflow support. So for example, I ask a question if I say yes, then it gives me another box to fill out. These are done via program rules. So let me get back to my PowerPoint presentation. All those different data quality checks that you just seen are done via program rules in DHIS too. So essentially what is a program rule? A program rule allows you to put in dynamic behaviors in response to user input. So just as you saw, I was ticking boxes and more things were coming up or I was putting in values and it was causing alerts and error warnings. You know, that's the kind of dynamic behaviors we're talking about. It allows you to send a lot of direct user feedback. So as you're putting stuff in, you're telling the user, no, that doesn't make sense or you need to check this or that those numbers can't add up. And you are building these out as expressions. And we will talk a little bit about the expression builder just in a second here. But essentially, these expression builders have a lot of different options for you. It's not just about showing warnings or errors or tick boxes. What I showed you, there's lots of things that this can do. And again, there's a whole academy that goes over this. But just to make the point that you can really tailor these messages, these warnings, this user feedback, this workflow support to using a lot of different ways, a lot of different features. So most actions using program rules will happen immediately when the data values are entered. So that's exactly what you saw me doing. So as I was putting the values in, there was something happening immediately. And that program rule, in terms of how it's configured, it constitutes three things, variables, expressions, and actions. I'm going to go through these very quickly. And again, to get more detailed answers and explanations, we will have to point you to another academy, or certainly you can ask questions, and we have lots of resources for this. So what are program variables? Essentially, program variables are represent attributes and data element values, which can be evaluated as part of an expression. So essentially, these are the data elements and attributes that you actually define. So as part, usually they're data elements, sometimes they're attributes. And you are, once you make them as a variable, you can use them in multiple program rules. These can include data values and attributes and operate these as expressions. Let me kind of skip through that. So here's the screen. If you want to make a program rule variable, this is really the first step. You have to define the variables, then you can put them into your expressions. And then from your expressions, you define the actions. So in this case, we're looking at a program variable called sex. And the sex is a tracked entity attribute. So this is like if you have a patient and the patient is male or female, then this would be sex that they're using here. The expression is the next star. So once you've defined your program rule variables, you have to define your expressions. And your expressions are essentially the logic that is being applied to or is defining the kind of interaction that is happening based upon the value that are being entered. So I'll skip through that. In the expression, you have to build out operators. Again, we don't have time to go over this in a lot of detail. There are clear documentation linked here in the PowerPoint, providing you what these are. You have to do a little bit of programming. And it's not programming in like you have to know JavaScript or HTML or something like that. But there's just syntax in which you have to build out these program rule expressions in. And you can see here, I won't go over each one of these examples, but you can see that they can become actually very complex. There's a lot of flexibility here on how what you can actually do with the program rule expressions. So you can say like, if statements are greater than statements or statements and statements, you can really, really build some complicated things here. Of course, it's better to keep stuff simple. But you can say like, if they enter this, they enter sex as male, if they enter age is greater than 35, if they enter BMI as this, then recommend this action. You can really base it, these program rules are based upon several multiple variables and give specific feedback as the data is coming in. Not useful for just data quality, also useful for just workflow support. Custom functions, I'm not going to go over this, but suffice to say that you can build out various custom functions. So you can count the number of days in a month. You can count the number of days between events. You can calculate, you can prompt actions if there's a specific value. We call these our D2 functions. You can see that on the left side of the column there. That's just an example. All right, so actions. So we define our variables, we define our expression, and then we can define many different actions that have an example that I showed you. The action that I was triggering was show warnings and show an error. So you saw that as I entered values and there was data quality checks happening, it was popping up different errors saying these values don't sum up together. You need to double check this. There are many different actions that could happen. So I was just showing you basically one action. You can check the docs here. If you click on this actions icon, it'll take you to the docs and all the various actions. There's really a lot of different choices here. So this is a brief summary. So you enter the program rule details, then you enter your program rule expression, and then you define your actions. The other thing that I want to point out is that many of you are using Android, the mobile app, to be able to capture tracker data as well now. So you're not just using your PCs, you actually have folks in the field, community health workers, outreach workers, clinicians that are using the application. The point to be made about folks that are using the application is as they're filling out data, make it as easy as possible. Put in pictures, make different colors, make it so that each one of the answers to each one of the questions is clear and distinct. So here you see an example of a malaria program where a community health worker is filling out malaria cases. And you can see that for the answers in the top right corner, they're saying, well, what is, how did you diagnose this malaria test? Do you use microscopy and RDT or there was no test at all? And you can see that there's pictures. There are clearly, there's clear colors. It's very simple even for someone who may be illiterate to answer this question, right? Which one did you use? And again, this is the best practice. The best practice is make it as clear and simple as possible for folks to be able to enter this data. The next question of course is, or a later question is, what are the treatment types? And again, very clear pictures, different colors. And so they would be able to distinguish which treatment type they provided. So program indicators are another way in which we're able to calculate values based upon tracker data. Many of you may be wondering what's different between a standard indicator and a program indicator. The main difference is that a program indicator can have a little bit more logic put into it and it calculates values based within a single program. Whereas a standard indicator cannot support much logic, but it can calculate values across multiple programs or multiple data sets. Okay, so a standard indicator across multiple programs or data sets, a program indicator, you can export more expressions, more logic to it, but it can only calculate within a single program. So here's an example of maybe a program indicators that look at completeness of data that's entered in tracker programs. Two examples here, the first one is count of BCG doses given divided by total number of tracked children. So what is that giving us? That's giving us basically a coverage of BCG doses across all of the children that we have captured data against in our tracker program. A very simple program indicator may be applicable to like an immunization program. And the second example here is like return the percentage of tracked children who have recoded a BCG dose. So tell me the percentage of children that have received a BCG dose. And so this is very simple examples of how we're able to actually look at something like BCG dose coverage within a or completeness within a tracker program. So the question becomes a lot of times is if we have tracker data and we have aggregate data, how do we combine those into a single coverage indicator? What we typically recommend is that you do not use tracker data as your sole denominator for coverage indicators. Why is that? Because typically most tracker programs do not have national coverage, meaning that most of them are still in some stage of scaling or pilot. So maybe you have a couple of districts using tracker and they are capturing individual treatment or service data through tracker. But then you have other districts that are maybe not using tracker or they're not using it as well. And they're still capturing that aggregated data on a monthly basis or something. And so how do you actually then make say like a total coverage, like national coverage indicator? Well you need to potentially combine the two sources of data together into a standard indicator. But we don't recommend that you necessarily solely use tracker data unless you know that you have 100% coverage of your tracker program across the entire country. And let me just kind of skip to the point here. So here we have kind of what's the good practice? The best practice is to have two different indicators potentially. So you have your national coverage that's using your aggregate denominator. And then you have another coverage that's using tracker data. And you're able to see those side by side. And so that's what we're actually able to look at here. We have the coverage indicators on the left that are coming from the tracker program. And then we have the individual data that's coming from the aggregate side. So you can see that in the green is the tracker coverage. The blue is the national coverage coming from the aggregate. So you're able to see those side by side. And then the same thing on the right side, the green is the total number of immunized children. And the blue is coming from your aggregate data. And the blue is the total number of children registered in the immunization program. So you're able to see these side by side. Again, the point being that as it stands right now, the general recommendation is you put your tracker data alongside your aggregate data and view these as somewhat distinct unless you know that you have 100% coverage of your tracker program, which is quite rare. All right, just one other slide here. Sorry. Here we want to talk about how you monitor aggregate data compiled from a tracker program to ensure that errors have not occurred in the data transfer. Essentially, what we're looking for is a one-to-one relationship. So what we are saying is that Yuri just talked through some of the tools that can be used for this. But you also want to put in some some program indicators to make sure that the values that are being transferred. So for example, if I am transferring this month's aggregate or tracker data into aggregate, let's make a program indicator that sums up all of the tracker data for that data element for that month. And then let's see that program indicator. And then when we when we push the data from tracker to aggregate, let's make sure that data element that we see in our aggregate is the same as that program indicator. So we can do a check this. So we make sure that we kind of have a one-to-one relationship. Sometimes it can become quite problematic with the transfer process. If you don't have these data quality checks kind of in place to monitor just the transfer, did everything actually go through correctly? Okay. I think I'm going to skip the last slide. Actually, Yuri covered it quite thoroughly, I think. And so that brings us time now. It's 12.06 Oslo time. So we are going to go ahead and take our break. Well, let's see if there's any questions. Nora, any questions from the community? Martin? Scott, you answered the questions about Excel into tracker. And then there's a question from Derivi again. And if we calculate coverage, if we use event program to do surveys of representative samples, I'm using it for our annual household survey to calculate coverage. I'm not sure. And there's also another question from Rose. So can we use, can we calculate coverage if we're using event data to do surveys of representative samples? Yeah, of course you can. What we were talking about, and I think that seems like maybe a lower level, like a DHS, like a demographic household survey or something like that, you certainly can do that. And that's the methodology that DHS also employs. What we are saying is that maybe for like coverage indicators that are like total immunized or percentage of immunized children or something like that, fully immunized children, typically we don't have tracker programs yet that are at national scale to be able to calculate that one with 100% certainty from just tracker data. But I think that if you're looking at coverage calculations that are just from like maybe survey data and you're using event capture to get that survey data, then I think that's probably fine. It's not so much associated with like clinical service delivery. Then Rose has a question here, would it be possible to do some program notifications and tracker events as bulk collective summary, not a single notification? Yes, I think so. Oh, you want to schedule notifications from tracker. I believe so. So you can also have something called program notifications. That's when you're actually building the program that you can apply. And I think maybe not using a program rule, but a program notification might be the answer to that. But to be honest with you Rose, I am not the world's expert on tracker. And so what we need to do is I need to pass your question along to Yuri, who would have a better answer than me, I hope. All right. So now we're at 1208. If you were wondering if I could share with you access to the database and just a quick demo, I cannot. That is just a, that is a production since developed by a DHIS2 implementer that I asked if I could just do a quick demo of their data capture forms and they said that was fine, but it's not, it's not publicly available instance. Okay. So let us take our 15 minute break and we will be then coming back now at, I guess it would be 1225. So, Nora, would that give us sufficient time to go through your session after the break? Yes, Scott, that's fine. We've just had a request for the uploading of the PowerPoint presentations, please. Right. I believe mine is already uploaded, but we need to grab Yuri's as well. Yes. Then 1225, we will meet again. Okay. Great. We'll see you back here in 15 minutes at 1225 Oslo time.