 Only the HMIS team or the M&E team, the program managers, who all is involved? If you take the purpose of this report, who needs it and why do they need it? And out of this report, were any actions taken or are any actions going to be taken that result in demonstrable changes? So the bottom line is does this report change anything? Or does it just become a nice report in a glossy folder stored in a glass cupboard? The second aspect of this data quality review is to look at the data and to fix it. And it's about checking for outliers for obvious data errors and then correcting them, checking for missing months of data and filling them in. Specifically, if you know a facility has worked, but no pieces of paper have arrived. For the WHODQ tool, there's something that says annual report. You click that open and the first option you have is data and you can select the group. It could be call, it could be immunization, it could be maternal health. For the period, you then select the year that you want to run the annual report for and how many years before that do you want as reference so that there is some background and then you select the organization unit. This could be the whole country, sub-level, sub-national or sub-national times two. The choice again is always yours. You then get a report that is completed and it will show up in the WHODQ tool. You have the option of printing the report and you will be able to save it as a PDF. So this report, the one I ran was nine pages, looks at completeness, timeliness of facility reporting, completeness of indicator data and a whole of guidance and indicators showing you what's happening with completeness with timeliness of the data. So this is what this report looks like. So the questions are, so here is this report. Is this report helpful? If it's helpful, for who is it helpful? What guidance can you gain from this report? Is this report a guarantee that the data is correct, complete, consistent and timely? So all sorts of things that are looked at with part of the development of an annual report on data quality using the DQ tool. The second section of this presentation looks at the data quality review and data cleaning The use case we are going to talk about is South Sudan. Fortunately or unfortunately it was done using DHIS 1.4, not DHIS 2. We are still looking for use cases with DHIS 2. And it was done for the years 2012, 2013, 2014 data. And here is a map of South Sudan and a flag if you don't recollect it. So the first question is, why do we bother to clean the data? And the problem is that poor quality data makes the data unusable. You can't use it for accurately estimating indicators, coverage of performance, coverage of service delivery and that means that all that effort that you went into to create and to collect and to capture and to clean the data from all the facilities, there is no return. So the data is collected but it's not usable. We also know that if there are a few very obvious data entry errors, people can take this as a sign that all the data is untrustworthy and that we know is not true. Experience has shown that on the whole generally HMIS data is generally reliable. It's not perfect, it will never be perfect, but it should be good enough so that it can be used. If we go about correcting any of the obvious errors that we see, the data then becomes usable and acceptable. And usable data and acceptable data is much better than unusable data which has no value. It just stores up space in the computer and costs money. What we do know is that in spite of everybody saying well we do validation checks and we have quarterly data reviews, data errors do creep in and they frequently remain undetected and uncorrected and stay there to haunt you for years. And as we said South Sudan was using the AirTrace 1.4 at the time of the data cleaning process. So some background to what happened in South Sudan. First of all it was a minimum indicator based data set implemented in 2011. So it was based on indicators and from the indicators we defined the data elements, we put them into data collection tools and it was two pages with one page for EPI. And there was a decision made in 2013 to provide a report for 2012, the annual HMIS reports because we now had a year of data. However on reviewing this data it quickly became apparent that the data was not good enough to be used. We could not use it for the annual report. And so the decision was made that we would go back and we would clean the data so that it would become usable. And so that this was then done at state level, sub-national, and we also then combined some states for convenience just so that in terms of time and in terms of distance and travel. The precinct at these data cleanup workshops was the state M&E officer and the county M&E officer for each county. And they were responsible for creating the data. Because it was the county they knew what was going on in their area, they could easily contact the facility, people responsible and ask questions. They did all of that work about fixing the data. It took approximately 45 days per group of states and it was a very labor intensive process with everybody working early in the morning to late in the afternoon in the heat and dust of South Zealand to get this work done. So we first made some crucial decisions. So the first one was that we were going to concentrate on the head count, knowing how many people access the health services. It's crucial to understanding how much money you have, how many resources you need. So head count. Then we looked at children with diarrhea, pneumonia and malaria because we also defined them as a vulnerable population. We then looked at maternal health in terms of anti-natal visits, deliveries, life births, IPTP and the things relating to maternal health because we also see that those were the crucial things to have correct. We ran validation rules. Absolute and statistical were run and any violations triggered were noted. The same process that we use in DHIS too. Extreme outliers were identified and corrected using tools in 1.4 which include interpolation and regression analysis. Interpolation is making an educated guess based on data before and data after. And regression analysis, we're not going into a mathematical discussion this afternoon on this. The good process was we would concentrate on replacing the missing months of data. In other words, we know that a facility worked, but for some or other reason the piece of paper never arrived. And there are many explanations and reasons for that. Insecurity, unable to travel, no stuff, whatever, but we know the facility worked. We replaced missing data. And we looked at validation rules with compulsory pairing. In other words, if you have one, you must have the other. An example would be diarrhea treated. You must have diarrhea reported. We fixed those up. We looked for extreme outliers and corrected those at facility level. Specifically, if a value materially altered the indicator. It became important enough to correct the data. And in this process we looked both at raw data and at indicators. Because frequently on raw data you may not see that something is wrong. But when you look at it as an indicator that that data element is part of an indicator it becomes apparent that something is not right. So some of the actions and some of the things that we did we developed a decision tree for data outliers. If it was out of range and then there was the services, we first of all looked, did the data elements hang together? Did they make sense when you looked at them? Then we did, using regression we recalculated a value that made no sense. If the data elements did not hang together, in other words they made no sense. We would recalculate the key data elements. The most important ones and provide a comment. Again using data from previous months and months after. And if we had data there was no services that were provided by the health facility. We were just deleted. So this is some of the ways we looked at trying to fix outlier data. Trying to fix missing data. We looked at was the facility actually providing any services? And if it wasn't, then we closed it. If we realised that the facility did not work for one month, or maybe even two months due to security, due to no staffing, whatever, but the services were not provided. We wrote a zero and we gave a comment. There was no longer missing data. Facility did not function. So it's not missing data. And if we said that services were provided by the health facility and the data was missing, we corrected it using interpolation and regression analysis to fill in the gaps. So why was it possible to do this in South Sudan? Well, there were a few reasons. So first of all, it was a very, very small data set. There was only two pages. The local staff, the M&E officers were eager to be responsible and to fix their own data. They wanted good clean data. And they took great pride in having data with no violations. And there was a lot of hands-on support and supervision to guide people through this process. Well, what were some of the results? First of all, we had cleaned data for three years, resulting in an annual report for each year. What countries are doing an annual report each year for your HMIS data? Because we had data for three years, we were able to measure service delivery over time. Were counties and states getting better? Were they staying the same? And one of the most crucial aspects was the development of the capacity and the knowledge of the M&E staff regarding both HMIS and DHHIS. And as we all know, when you start out in a country, one of the biggest issues is an updated facility list. So a lot of work went into fixing a facility list. So in conclusion, has all the missing data, where you know a facility work but there's no report been added in, have you fixed all your arctlias, have all your validation rules been fixed or explained? And then the question is also who takes responsibility for this? Is it important to do this in your situation? And so we are left with two exercises. The first one is about the WHO annual report. So you either use your own country, DHHIS to instance or the demo one. Run the annual report for a selected or unit and data element or indicator group. Print it, which will allow you to download in PDF format. Then review it and discuss it. How can you use this report? We know that there are quite a few of you working in the same country and I think that this would be a very nice exercise for you as a group to have a look. Then in terms of the data reviewing corrections, either your own DHHIS to the demo instance, using the WHO DQ to find the missing data and arctlias. What needs to happen to replace these missing values or correct the obvious data arctlias? And here is a difficult question. How will you decide what the new value should be? Thank you.