 Great. Okay, folks, welcome to the best session of the conference. This is the updates on the data quality, what we've been doing in DHS to since we last talked to you last last year. And, as always, we have a lot of new data quality tools, I'll go over those we have some new data quality approaches, and of course we have quite a lot of work that's been done in the field and new data quality use cases. And so I'm Scott Russ Patrick. I'm the DHS to analytics product manager here at the University of Oslo. I'll get get us started with some of those new features. Then we have our very good friends on to and Bob pond coming from the WHO they're going to give us some updates on the WHO guidance and which approaches the data quality which of course we follow very closely in our tools and guidance that we put out as DHS to. And then last but certainly not least we have to zo coming from his Tanzania Tanzania just went through a massive data quality review process. And so he's going to give us some experiences and share with us that process that they did there. Which I think I just went over the program in the first slide so we'll, we'll skip that. All right, on the new data quality features. Essentially what we're focusing around now and core DHS to is, we have had in the past quite a lot of data quality features. And these features have been spread out over probably about six or seven different applications, meaning that if you wanted to do kind of comprehensive data quality analysis, you would have to move around as a user between various applications to do different kinds of analysis. And one of those major applications that was that that is currently used and will continue to be used. Most assuredly is the WHO data quality app. And that's what you see on the left side of the screen there is the WHO data quality app. This was developed a couple of years ago. But it was kind of a radical improvement to DHS to data quality functionality when it was released. It included things like year over year charts to look at consistency of data over time, as well as consistency of the data between multiple variables. It also introduced the scatter plot that help us do kind of outlier detection. And, and as well as a more advanced outlier analysis in the form of kind of pivot tables and and WHO and and University of Oslo in collaboration with all of our his partners. We're trying to implement or supporting the implementation of this app. It is many countries as we can through academies through workshops, various projects and programs. And it's well implemented now there's dozens of countries that are using it. We've actually received quite a lot of feedback on it. And one of the main things that we've received on it is this app is incredible is introduced a lot of new features of data quality. It makes data quality checking much better. It allows us to do more routine data quality checking so not just like big annual data quality reviews but like we can look at it every single month and we can allow our users at the lowest level to look at it. But the big feedback was we want to have these analytics is data quality checks directly on our. Yeah, I'm so sorry to interrupt but I think you do want you are not in prison terminal. I'm looking at my, you're looking at the slides with my notes. Well, I can see like the next slide for instance. So, all my secrets. I will try to share my screen again, but I think I am this one share better Alice. Yes, better. So, no worries. Thanks for the catching that so where was I, we were talking about wanting to be able to see the really cool data quality tools and analytics that are in the WHO data quality app on our standard dashboards. Excuse me, and because that's where people are spending the most of their time to do data analysis that's where they're able to present most of their other data so all of their service delivery data and all of their performance data. All of that is under under standard dashboards. So of course you want to see your data quality analytics and tools right alongside your service delivery, your routine data coming from the facilities right. You want to have those side by side and sometimes you want to quickly move between the two. And that is really the primary role of the roadmap for the last couple of years really is to start to migrate a lot of these well known good tools that were originally developed in the WHO data quality app into our standard analytics apps. And, and get those on the dashboard and I'm very happy to say that we have made a lot of progress on this, and we are now able to show almost all of the data quality tools that are in the WHO data quality app, just on the standard dashboard and I'm going to demo a couple of those for you but essentially, they are the year over year charts, the scatter plots which the scatter plots is fresh off the off the presses right I mean this is brand new stuff. And then we also have a more improved robust outlier analysis on our back end. If I just jump into this need to get out of my presentation mode here, working with too many screens. Okay, let's take a look at DHS to. Alright, so I am. So you see here I have this scatter plot on my dashboard. And I am going to open this in the data visualizer app and just do a quick tour of how this works. Alright, so you see here on the in the data visualizer app. You can find a scatter plot any which way you want you can make some really crazy looking scatter plots in DHS to the really aren't any limits in the WHO data quality app. The scatter plot format and configuration was kind of hard coded. You weren't you did not really have a lot of flexibility there with with having in the data visualizer application you have a huge amount of flexibility. You have a vertical axis of horizontal axis. You see right now we have that for the vertical A and C one visits, the horizontal axis A and C for or more visits. Your points are going to show up as org units so that's locked to the org unit dimension. And then your filter is periods in this case you can add additional dimensions to your filters as well so if you wanted to see just org units that are community health posts or something like that you could turn on the facility type community health posts add that to your filter. Now, in the options tab if we go to outliers, then this is where you actually turn on the outlier detection so I'm going to turn it off just to give you so here's our normal scatter plot. And then if I go to outliers, then I click on the outliers analysis I'm given three different options so interquartile interquartile range excuse me IQR. We have Z score and modified Z score. If I click on one of these, you'll see that we have this threshold factor, the threshold factor defines the number of standard deviations or in for interquartile range number of quartiles away from the average or the median depending upon the outlier methodology to to put my threshold lines. And if this doesn't make a lot of sense. That's okay because I actually have a slide on it because I wanted to take some time to kind of explain it to the community, or at least have it recorded exactly why we have these three different methodologies. Sorry, this gets a little bit Matthew, but all of us work in data so hopefully you can stick with me here. So what are we doing. When we do an outlier detection, what we do is we, we draw a linear best fit line. We only have linear best fit lines we don't have like polynomial or kind of other best fit lines we just have the linear best fit line. So the HS to a draw best fit line amongst all of the facilities and points that you have on your scatter plot. And then what we do is we measure the distance between the best fit line and each one of the points. In this case we're looking at facilities, right, and that's and so you see here we have like a B and C each one of these is a measurement. And that essentially gives us the data set that we then use to which is you know, a value is a value associated with each one of the points, how far it is away from them, the mean or the median. And then that is the data set that we use to perform our outlier analysis on. Again, we have the sorry for taking us all the way back to basis statistics courses, but we have again three different approaches. We have our interquartile range, our modified Z score and our standard Z score. Now, interquartile range and modified Z score are bar are by far the most robust. We all have been using Z score for a long time. The difference really in between modified Z score and regular Z score is modified Z score uses the median value so that value that's in the middle of the data set. And as as as the point at which we we calculate our, our standard deviations from regular Z score uses the mean or the average. And of course that average is being manipulated by the outliers themselves. So that's why Z score is not as robust because the mean the average is actually being manipulated or influenced by the outliers themselves, which can give you kind of an artificial counter an artificial calculation of the outliers. So, interquartile range is maybe one that you guys haven't heard of it's probably considered the most the more robust based upon the statistical experts that we talked to and we talked to folks from the University of Oslo statistics department we talked to some folks in WHO we talked to some folks in Norwegian Institute of statistics. And most of them I mean most of them, the statisticians are using something like interquartile range interquartile range essentially just divides the data, the data set up into four quartiles, or four pieces, and then it applies these ranges or these it blocks off like the first and the third, or sorry the first and the fourth quartile as being acceptable or, and then it has anything that's beyond that as an outline. We can just take a look at this in action. So if I go interquartile range, we have a threshold factor of 1.5 that's considered the standard you can of course increase it or decrease it. I click update. Then it shows this scatter plot, you can see that anything that is it within those quartiles or that that threshold is showing up green anything that is not is showing up as red. See, then we have a another functionality that is to show extreme lines. And so what an extreme line is, is we need an indication on this chart. What is the most pressing, or what are the worst outliers. Right, we see a lot of outliers we see a lot of red spots, but some of them are very very small outliers relatively speaking to the entire country. So what are those outliers that are throwing off the national statistics that if they were not correct will given will manipulate or change the entire national data to be incorrect. Right, they're so big these outliers are so big that they're throwing off the entire national data and if anyone looks at that national data, they will see the data is not accurate is significantly not accurate. So what we're able to do is apply these extreme lines, and these extreme lines will put a line at 1% of the total national figures for both of our variables so for ANC one and for ANC four or more. So if I click update. So you see the extreme lines being applied. And anything that is beyond the extreme line above it, or past it is an outlier that represents greater than 1% of the national total values. So here you know if we look at this one this one's very simple. So we have this one community health center. Right. And their ANC one visits were 4,000 800 4,000 681 their ANC for more visits were just 800 or sorry 788. So that eight that 4,000 681 is a huge outlier huge it's it's it's almost twice it's almost 2% of the entire national figures. And, and so we suspect this to be incorrect data that needs to be followed up. Likewise, if we go way way over here. We say we see that Marie Stopes clinic has an ANC first visit value of 4, 844. And then it has an ANC fourth visit value of 2095. And that's almost 2% of the entire national total of ANC for more visits. So it's, it's a huge outlier it's probably an incorrect value it needs to be followed up. Okay, so that is unscouter plots now we have tested this right now you're looking at our demo database which has about 3000 org units or so being displayed here. You see, there's a lot of clustering here if I click and drag over an area I'm actually able to zoom in. So I'm able to, you know, see each individual point at if I keep doing it to reset zoom I can just click up here. And, but we have tested this in databases that have up to about 65,000 or units. It obviously is a little bit slower, but within, you know, it's measured in seconds not minutes or, or anything so maybe just a few seconds slower to render what you saw here. But we, this is a brand new feature, basically no one's using it yet. We would love we can add a lot to this, we can continue to develop this we can make it much more advanced. If people need it to be. But as it stands right now we it's, it's really quite simple. We would love to get feedback on it please tell us what you like what you don't like and we are very happy to make any changes based upon your feedback. The, or sorry the second to last and I know I'm a little bit over time actually but I'll go quickly here is the year over year line charts. So again in the WHO data quality app you saw that we had year over year lines charts and we've created we've recreated that in the data user app but again with a little bit more flexibility what was in the WHO data quality app is a little bit hard coded what you see here, you can turn on any data variable, and you can turn on as many years as you want essentially. So I'm just going to turn on this year last year. And I'm going to turn on. Let's just turn on. I think we need to turn on months. Okay. Oh yeah so that doesn't actually make a lot of sense with this demo database. Because let's turn on 2019 as well. Actually this data is very close. You actually see they're kind of pilot up on top of each other. This is a bad demo but we see here's 2020 here's 2021 and and we're essentially able to do that same kind of year over year analysis so if you see the values are, are significantly different from year over year, especially if you see a spike in one month. For any of these, any of these total values then you can expect then you know that's easily to identify as an outlier, or something anonymous, anonymous, not a different or in the data that needs to be followed up. Okay, the last thing is in the data quality app so many countries are using the DHS to data quality app which forgive us for our nomenclature but it is different than the WHO data quality app. It's been around for quite a long time. Not a lot has happened to it in quite a long time. But we have recently updated the outlier detection and the outlier detection is significantly more performant and gives you a little bit more functionality so I'm going to turn on outlier detection now. I'm going to just randomly choose some dates. I'm going to leave my algorithm for to Z score right now, just for the sake of demonstration, click start. It's going to take a minute. Actually that was very fast. Again, it's much more performant than it used to be. And it does an outlier analysis for me. That's a little bit more kind of road. And then then what we saw in the scatter plot right so we saw the scatter plot is is very graphical this is a much more kind of horizontal role, row orientation to our outlier analysis. And so we see, you know, for each one of the data elements that is in the data set that we selected we see the data element name we see where the data where the org unit is that has the problem we see the value, we see the Z score. We see the deviation standard D mean men max. And so we can see that this value 650 is greater than the deviation of 543. So that's why it's showing up here. And we can of course market for follow up. We market for follow up what that means is that that data value in the aggregate data set is then flagged. So follow up analysis, which will allow the user to go back and check any of these data values later. So it's kind of just flagging it so that you can come back and and and review it or push it down or send a message to someone saying hey, we've marked these values for follow up you need to you need to check on them. Okay, so again we really appreciate any feedback or communication around these features. Again, we're making lots and lots of improvements to data quality functionalities and core DHS to. So any use cases or feedback you have we would love to hear about it. I am now going to hand it over stop sharing my screen and hand it over to and and Bob who will take us through the updates from WHO. Over to you guys. Can you, can you share this slide or is both are both going to share this slide. I can do that on. Let me. In time just a quick introduction so it's time around for now. So my name is anxious, and I am the working at the vision of beta and analytic and delivery for impacts. So, in Geneva, and it's very nice to be back and sorry that it's not being person. It's just good to be once in once a year to be in this community. So just a very quick session today is on some of the latest update on the data quality assurance and data quality review to kids that has been around the last couple of years. I'm going to do some revisions and put some feedback from the countries and have new more materials, and then both will give some more detail of one of the most sky of a significance in terms of the practicality of the tools that has been developed. So, we also have a new sort of not the new website but the new link to the website, since the WHO website has been restructured. And this is a link to the data quality and for all the materials. What do you go in there you will see all the guidance, but then for the tools itself you will need to contact us for that. Next please. Just a quick over captures of the work between WHO and the University of Oslo link with the BHI, the BHI has two tools. Overall we have developed several different technical tool kits and I'm sure most of you here not all of you are familiar with one of the key part, which is the data quality assurance, which three years ago we started to provide the guidance on the meaning of the use of data, how to measure and they bring the data quality review and then later on it just got evil for others and that's also the foundation for the data quality apps that Scott mentioned earlier. Next please. So what has been new one. In the past we have a two modules for the data quality review. One is the data quality review that is supposed to be conducted independently from for the mostly from the crosscutting HMIS data set. And there is a one features in the BHI's to data quality app and the other features as a tool is on Excel. And then the second tool is also called data quality review but is for the data verification and the system assessment and that's supposed to be done in the methodologies of a survey in the health facility and look at the entire structures in the system for the data, which can influence the data quality. So what is a new feature the new feature is looking into, apart from the routine that done externally and sometimes requires a lot of steps and costing, what can be done. So as a something that make it into a habit of all the data reports and then data managers, and that's what we call the routine data quality review assurance. And this is something that we realize it's the HHS tool with all the new feature can plays a significant impact in changing the changing the practice of our data collectors and report in the facility as well as the data managers of how we can use existing feature to ensure that the data monthly data that is submitted to higher level is completely trusted and accepted by the community. Next please. This is a quick overview of what's going on. In the past we already have a lot of materials we have the data quality review framework, we have the data quality review assessment tool, we also have some of the tools that guidance and facilitator trainings and all of that you can find in the website. The two new, the two new materials are number one is the data quality desk review now is called the discreet review. That is an implementation guide, and the same implementation guide is also for the module three. So what are in the implementation. Next, please. So what is an implementation guide for the model to which is the discreet review. It look at the content is not much different because we already have the data quality app. We already have the Excel tool. What is needed is to to give a very clear guidance in the training materials for for those who go to the country or for the country who wanted to to look at the data quality review as recommended which is supposed to be the end of the year. So what is the basis before you going into the national statistic report or you doing a plan. So it is have a more of a standard approach, which is easier to facilitate the process with partners and with other programs in the field. Currently this tool is available for the HMIS only. Next please. And then this is one of the, the new steps that we were suggested by a lot of partners is to see if we're going to identify the data quality review, help to identify all the issues. But then, then so what, and the data quality improvement plan is a next step that is written down as a guidance to help countries and program to say, we have these issues, and what are you going to do about that. We will look at all the potential intervention to discuss with different stakeholders. And to agrees on a time wise improvement plan that can be included as a part of the health information system action plan of the year of as a for a longer term will be part of the strategy. Next please. The very new one that Bob is going to talk about is district data quality assurance which is. I think it is, I can say that this is making the best out of the new feature that book has presented. It's exclusively at the moment is for the as to users, and it, it bring a lot of the feature from the data quality app into the dashboard switch should be part of the routine data review, and, and part of the data views. Next. The same, the same approach as going to the data verification system assessment. Still there's a loss of question we still recommend that it should be ideally should be included as part of the facility data assessment. It will be costly and it shouldn't be done every year, whenever possible between three and five years. And it identified entire systems that that would be able to make it into a health program planning and strategy. And then this one doesn't use the chairs. There is ACS pro software program has been designed for this use as together with the other modules in the health facility data assessment. Next, please. Next book. So, you know, your implementation. I have to say that we, we've been putting these materials out a lot, but we haven't got a lot of feedback. The feedback so far has been the inconvenience of having a data quality app outside and there's a loss of back and forth. And it's improved in the VHS too. So, what we would like to do next is to get more countries implemented and to get the feedback so that we can improve the materials. Another features in the Excel tools that we haven't heard of, but there are many countries that have that have a boss. The HHS and non the HHS and when they do the data quality review, those who don't use the HHS likely have to move into the Excel tools and we would like to hear about that as well. And see how best to put all the data and the HHS or the other way around so that we can identify issues and address the data quality. Next, please. And the next is my last slide. There is a plan. We know that there is community, community health information system guidance just came out. We have committed to agree to work on the guidance to improve or to ensure the data quality of this new structure of data collection and reporting, but it also should be linked with the rest of the data system in the country. There is also there are lots of interest and especially with the funding from the global fund TV HIV malaria also would like to extend it into their own own program specific date quality assurance and review and malaria has started to work mostly for the for the surveillance of data. The use of the HHS to a district level that Bob's going to say but I think what also important is not just stay with the district but how it blend in as part of the entire national information system. So that the data flow and data quality assurance can be reflected throughout the whole data chain, and there won't be a training for experts who's then going to be able to implement the skyline and we hope to do that by the end of this year, and with the country implementations. So with that, I'm going to end this presentation with special thanks to everybody's and also with the escort and the team in Oslo for making an aligning different tool together, but also with lots of fun and support from Gavi and global fund for all the work in the guidance. So I will pass this on to Bob for going to the district data quality assurance. Thank you. Thank you on. Let me very quickly introduce the training package, then demonstrate it so that we have time for to so to share the Tanzania experience with the training approach. This is a training package that can be downloaded from the website that on has in her slide. It is a bottom up approach to data quality assurance. This slide tries to drive home the point that such data quality assurance needs to be decentralized that is it should take place at the level of the district. At the age of 14 it should be every month. And for various reasons, it's more efficient for data quality issues to be fixed closer to the source, rather than through a review at national level. For one reason, the volume of data to review is smaller at the level of the district, especially if the review takes place every month. Another reason to do it at the district level is that investigation of the suspicious values is practical for district staff. After all, they should have access to the paper records to make sure there's not a transcription error. They should have a good relationship with the facility staff and means of contacting them by meetings or phone. And finally, the data belong to the district so the district are capable, they have the privileges to edit the data and remove the errors. Here's a slide that I'll just point to the bottom bullets noting that the training approach depends upon the WHO data quality instance of DHS to which I'm now going to use to demonstrate to you what I call the 10 minute training. So the training approach focuses on use of two tools. One tool is simply a data quality dashboard. The dashboard starts by, are you are you seeing the, I want to make sure you're shared the right screen are you folks seeing the data quality dashboard. We are. Okay, the dashboard begins by with a series of visualizations showing reporting completeness of multiple data sets, a chart that showing which particular type of health facility is responsible for lower reporting of the malaria data set a table that allows you to identify in the district these specific health facilities that have lower reporting rates. The dashboard then shifts to reviewing the consistency of the data. Now, you probably are used to seeing this type of a month to month chart used at national level to identify the very worst of extreme outliers. A chart like this is viewed at national level it will not pick up important, but smaller outliers that are buried there in the national data set they're averaged out and you, you can't detect them. The virtue of doing this kind of review at district level is that you can pick up relatively modest but still important in erroneous values you see here how the penta three doses have jumped from 585 up to 951. This is a value which, if all the data were aggregated to national level would simply not be apparent so you, you, you see here August 2020 a penta three outlier, and here in March of 2021, a penta one outlier. The data quality dashboard then has a table, making use of the ph is to predictor rule that actually makes it summarizes all of the important outliers at the district level. And it identifies the specific health facility here you see that it's facility 325, which has reported the outlier and penta three, and it's the district hospital itself, which reported that penta one outlier March of 2021 that we saw the tool then shifts to use of the WHO data quality tool, which is available to all of you even if you don't have the latest features of 2.36 you, you all have access to the WHO data quality tool, which I like to say is like a Swiss army knife of tools for reviewing the consistency of data. If this is to be used routinely and used quickly, we need to somehow simplify the use of the tool. And for purposes of routine data quality assurance at district level, we like to in particular jump right to using the outlier tool, which I compared to the magnifying lens and the tweezers of the Swiss army knife to use this for a district review. We've defined a special set of indicators for the district that we've particularly want to focus on and to to do the review for a particular district where we will set the organization unit to district day one of course this would be done automatically it when the district staff logged on but this is a national instance. So if we jump to the outlier tool, rather than playing around with drilling down and training the district staff out to drill down there's no need to do it. When the tool is used at district level, and they should begin by simply filtering for values that have a modified score they're extreme. And when they do that, they are automatically taken to a rather short list of values that need review that month, and you see that the tool has picked up. And again, this outlier and Penta three from August of 2020, and it's picked up the outlier that we saw in Penta one values. So the, the outlier tool has come up with a rather modest set of tasks for the district staff to attend to something that is frequently overlooked with this outlier tool is that it also allows you to identify important missing values. So if we on the options filter for missing data, turning the outlier filter off. We are taken right to some of the values that are most important in missing, and in this case you see right at the top of district hospital it's failed to report it's, it's outpatient form for three months of the last 12. It is useful at district level to be able to do some aspects of review offline and for that the best thing to do is to click on the download button. And that will generate a CSV file that that includes all the data in the outlier tab. Now what I have to do is, is a new share. So here's the CSV file that's downloaded. And as with any. Let me see if I can. If you have any Excel or CSV file, you can then use the sorting function to sort it in this case according to the modified Z score largest, the smallest, and it will again identify the, it will again identify those particular indicators and facilities, which have the, the, the most important outliers. Of course, the Excel tool doesn't like the outlier tab highlight in red so you've got to go to go through and and circle these or highlight them. But you can print this out and then take it for supervision or have it there on your desk for for later reference. So, in summary, that's what I refer to as the 10 minute training, and the essential aspect is to identify the the most efficient tools in the DHIS tool box, the ones in which you can train district staff in a small number of minutes to undertake it. So what I would like to do now is hand over to to zoe, if I can do that. And ask him to share with us the experience from a training workshop in Tanzania with this district data quality assurance approach to zoe. Thank you Bob. I think you have a slide. Can you please share. Yes. Indeed, let me get back to. Hello, while we are waiting. For the slide. Can you actually introduce myself. My name is choosing a bit actually working under his Tanzania, and they are providing support to the ministry of health, actually use heavily the DHS to platform and different different projects. The data quality assurance that my colleague has already presented. We actually configured it in our DHS tool. Minister of Health Bob from the consultant of WTO actually collaborated together to make everything works. And actually, the configuration involved the use of predictor to identify the extreme trial and the Bob was shown that in the demo after the same configuration that we did in our DHS to instance in HMS Tanzania, but also that has to go with the creation of the data quality dashboard that the one like Bob showed you. And after that, we had to create the training material because after we've configured all of them in HMS Tanzania, we actually need to create the training material where we'll be able to capacity the CHMT at the council level, although it was not all the council but we selected a few of them so that we can capacity them how to use those data quality dashboard with the WTO data quality tool. And then it can be scaled out to other to other to other council next. As Bob was showing actually it was the same thing. We had different dashboards that was configured in HMS Tanzania as you can see one was kind of checking the reporting rates of several forms because our database is huge and we have a lot of data set. So we had to pick those from that most we use it. And then so that we can configure the data quality dashboard. And as you can see there are several dashboards that was configured. Even that included the other program like HIV and the other and another side that was the WTO data quality tool that we configured and being identified as an extreme or trial within the variables. So and essentially the use of that because in that context. Next again. Yeah, the use of that. That's it. The use of that is that we have a public portal also where after every three months or after every quarter we have to means to have to upload the data so have to make sure that the data has good quality. So one of the thing is was to create something that can help them even at the console level. Before even sitting doing the data quality review the quarter data quality review, most of them being cleared out by the console staff the HMT was direct access to the to the facility so one of the tool was to create easy dashboard by using the predictors so that those extreme outlier can be alerted and then the HMT can see those outliers and then they may contact directly to the facility for for electrification. For example, can you see one of the table and find that some of the facility was reporting this huge amount of data, which somehow maybe due to error. And maybe, which is somehow implicated to have this kind of visit to that to the specific healthy center that's in the, like a medium facility so we expect to maybe be this large on large number of visit to be in a hospital's regional hospital and even in the river but for medium facilities we don't expect it to have this huge number of visit. So this one was to help them to identify the extreme outlier and the thing was not just to to get the visualization the dashboard but we also configured to send an email having those extreme extreme outliers. Next. Yeah, another thing was to kind of use the WHO data quality tool. So this also was being part of the training manual that was being designed to help them to identify those extreme extreme outliers. As you can see, I find that those are just pulled out to see the extreme outlier from out of 26 region, but to my find that there are few only facilities few region which still have these extreme outlier, which means most of them has been has been cleared. So there are several challenges that we observed while we doing this training and the configuring configuring this data quality. One of the, the challenge that we observed that you know, we have this tennis of changing the tools that used for reporting. So these actually affect the configuration that has been done because you need again to reconfigure applying these new variables that has been changed in the form. So that was one of the problem that we face it that you have already configured the dashboard and then you find out that the MOH, the Minister of Health is changing the reporting tool or so you might need to to reconfigure. But again, we capacitated a few HMT for selected the cancer but we have one and it's 70 cancer so it was somehow impossible to capacitate all of them. So what we did is just to select a few of them and conduct like a TOT so that they can scale up to other to other staffs. Also lack of funds, which lead to somehow not all of them that conduct monthly data review, some decide to conduct quarterly data review, some even do once a year. So you might find that doing the cleaning somehow positive because you might find that the same staff who are doing other activities have also to conduct this monthly data review. So at least having the dashboard put looking for the intensity of the data has to somehow fall on them and then and those email that we are sending it to him helps them also to to to kind of follow up on the data quality issue. But during our training to a few facilities to few councils, some of us explained if you can somehow include the functionality when you are using the WHO data quality and exporting the data, the data has to come with the data. And Boba showed you, you need to kind of have kind of sort because they wanted to that is somehow simple, simple to use, you download and it comes in with the with the car for those extra mattress so that they can carry it to maybe to the person that is responsible for the notification. But also, you know, this data quality makes sense when you are actually doing it at the facility level. So, comparing to our, our database which have almost 80,000 facilities. It is somehow impossible to do it at the national so you have to do it one, maybe a concept at a time because it will take some time to load. So, just somehow emphasize that at the regional maybe they can do it for one facility at a time. Next, I think that's like the end of my presentation. So I will come if you have any issue or concern. Thank you. Great. Thank you so much to those really, really great presentation and really incredible to see a country actually adopting all the various tools and guidance that we that we have been developing globally. Very quickly. And I think it's a good testament to say that all of these, all of these tools that these guides that on and Bob are talking about these functionalities that I was presenting. You can use these they work. And you can make significantly significant strides in addressing your country's data quality issues. All of that being said in the chat, we have been sharing resources materials for you. You know, virtually every country is struggling with data quality. And really, every country is is asking for some kind of guidance there. And we are compiling as much of it as we possibly can. We are making as much of it as publicly available as we possibly can. So there's two links in the chat there I just kind of want to draw your attention towards the first one is North Stoops was asking for guidance or an example of a country that has good standard operating procedures for fixing data quality mistakes they find. It's one thing to find it. It's something entirely different to actually change the value, fix it, get it of, you know, verified and approved. And probably the best one that we know of right now is Rwanda, and Rwanda has an extremely robust data quality standard operating procedures. I'm sure Jade I see in the chat has shared that so please take a look at that. If you want an example of a country that's that's really kind of a best practice when it comes to standard operating procedures to actually fixing data quality mistakes. The other things I want to point your attention to is that we have now posted all of the data quality Academy videos online. We have a week long data quality Academy, lots of sessions we go over in detail, everything that you've just seen here and a whole lot more. All of those videos to all of the sessions are available on our YouTube channel so please take a look at that. You can see lots of use cases there as well, just a tremendous amount of information. So then posted a quick configuration guide, who's always mentioning how they have used predictors and automated messages to start to auto kind of automatically detect outliers and send those messages and people who can do something about it. There is a guide here. We're document quite short step by step, how to actually configure that in DHS to the last thing that I wanted to point out is that we are also trying to make our data quality Academy completely self guided, meaning you don't have to listen, you know, all of the lectures are pre recorded the exercises are pre recorded, and you can go through it at your own time at your own pace whenever you want. And I believe that that we're getting very close to be able to publish that if we haven't already I don't know sure G if you have any updates on that but but hopefully in the very near future if not already, we will have the entire online data quality Academy. So self paced course so you can go through that at any time. All of that, again, being said, I think the important principle here is do not suffer in silence, there is a whole world of countries struggling with this working together on this. So please do reach out to us in the community practice, you can send me an email directly Scott at DHS to dot org. If you need any guidance you need any information. If I'm not able to answer your question. There are certainly someone that we know in our network you can. So, so please do reach out, get as much information you can. And if you run into any problems with any of the functionality or implementing anything DHS to, again, let us know. That's a couple of minutes for any questions I think we've been answering questions in the chat as they have come in. Any questions I don't see anything on the community practice either. Nor I feel like I should give you an honorary moment to say something. Thank you. Nor Stoops, who from his South Africa has been leading the charge and data quality in many countries as well. Scott, thank you. Oh yes, you can hear me. Yeah. Quality is both a good news in a bad news situation. It's a good news that people are looking at it. It's not such a good news because we find that so often the data doesn't get corrected and that's a big issue and I'm really grateful for pointing out that Rwanda has a SOP. And I think that that's that's real good progress. So I can say about data qualities and you to continue. Oh, yes. Scott, I don't know if you mentioned but I noticed in a few countries, DHAs to instances that the data entry clocks do not have access to the data quality app, which means they can run the data validation in the data entry form, but they can also run it outside once they've created for the facility and for the district and run it for a whole district and that I think is something that I was oblivious to until recently and it's I think it's something that countries need to look at and make sure. Yeah, some really some really basic steps to improving data quality is make sure your users can actually use the functionalities that identified the data quality issues seems obvious but it is it is a big stumbling block for a lot of folks. We just posted in the chat that we are just putting some finishing touches on the self paced data quality course. And that and that they on the training team will keep the community posted a tremendous effort on their part to try to get that self paced course up and live and once we do again you'll be able to go through all of the DHS to the data quality is all the trainings that all of the exercises that we have on data quality at your own time at your own pace. And, and there won't be any reason why anyone can't be trained on DHS to data quality. And again all of the data quality materials that we developed follow very closely that the go the WHO guidelines and principles that on and Bob were were sharing. That wraps it up for the data quality course.