 In this session, what we are going to discuss is about data quality practices in Sri Lanka. So what I hope to do in this session is that I will be taking two examples, two implementations in Sri Lanka, one is from the RMNCH program and the other one is from the nutrition program. And I'm going to discuss how these elements of ensuring data quality has been applied in these two implementations. So the first scenario or the first use case that I'm going to talk about is the RMNCH program in Sri Lanka. So this in fact was one of the major DHS implementations in Sri Lanka and it took us a lot of efforts to get this implemented. The main reason being the paper-based data collection system of RMNCH has been a well-established system for more than three decades. So trying to introduce a digital system and also trying to ensure at least the minimum data quality that was already there in the paper-based system was a major challenge. So let us see how we applied these concepts of data quality while we were planning the implementation. So we might feel that ensuring data quality is a very complicated task, but I will show you how simple you can start. So for example, the first thing that we did was we concentrated about data quality while we were designing the data reforms. So one simple step that we ensured was that because we were trying to introduce this digital system to already existing paper-based system, we tried to adhere to the well-established data flows. In simple words, what we tried to do is the data entry interfaces or the data forms in the DHS to the data sets. We tried to make the user interfaces as much as similar to the existing paper-based formats. I mean, we always say that you have to optimize the data collection forms, but while ensuring that we tried to keep the flow of the data, the order in which appear, the same as how it is in data entry forms. So by doing that, we can minimize the random mirrors that data entry people might do because they just have to follow the same workflow that they are doing while entering data into paper formats. So for this instance, what we created was a custom data entry form, not a section or standard DHS to data set, but it was a custom form. So let me show you first of all, now this is the paper format that we had in the RMNCH, the main aggregate data collection form. You can see it's a very complicated forms. There are so many data items or data elements, and of course, most of these columns that you see here are the periods. So the columns, of course, do not come into this our data entry interface, but what I'm talking here is about having the same order as how it is appearing in the paper format. And we had the same workflow while we did the transition to the digital format. So here, what you're seeing is our DHS to custom data set or data entry form. And please excuse me for the language, the forms are in local language, but what I want to highlight is the same order that we have it in the paper format. We have been following in DHS to data set as well, even though we have done some slight modifications. For example, we have set up here the vertical tabs to capture this large amount of data, but the flow of information capture remains the same. And the next thing is sometimes when we are designing forms, we cannot actually replicate the paper forms into the DHS to there are some optimizations that we have to do, especially like there are things like duplicate data collection. And there are so many things that that we don't do in an optimum way in paper based collection, but which we can optimize when we are creating the data sets in DHS to. So for example, in the mental health system, this is the paper form that we are using, but when we were trying to convert that into DHS to what we actually did was we optimize the data form and at the same time, we had the opportunity of modifying the existing paper based data collection form so that it kind of added to the quality of data that is currently so that now, for example, now this is the modified paper based form that we were able to create because like when we went one step ahead, this is the DHS to base the data collection form. So to minimally confuse the users and at the point of generation of data, we tried to use the same data collection form as that we have with the DHS to. So what we did was we kind of modified the existing paper based form to be optimized paper form. This is another example of the same mental health system where we have optimized the data collection forms, right? And sometimes, of course, in this ERHMI system, there were requirements such as like when we wanted to collect this WHO MNH indicators, we did not have a paper based form to start with. So there, of course, we had the luxury of designing our paper based form also from the scratch, which was kind of easy because this is the DHS to dataset or rather the event form and that one based on that one. So when designing that one, we were able to follow the best practices. And then we came back and we designed a new paper form for the collection of data because most of the institutes that I involved with our MCH system, they do not capture the data, the electronic system at the point of generation. So usually the data entry happens on a paper based form and then that is copied into the DHS too, right? And I mean, for the expanding on this one thing that we had to make sure, like when we are having this digital transformation from paper system to the electronic system is to minimally interrupt the workflow because the thing is, I think that should be the context of most of the countries because you may have a well-established paper format and when you are trying to introduce a digital system, we should try to minimally disrupt the existing data flow. Just like the government's motive is to provide better health, not only data. So if you are just trying to enhance the data, I mean, the quality of data by introducing the digital system and at the same time, if you are kind of disrupting an existing data flow, that will have a lot of resistance during the implementation from so many stakeholders. So the other thing that we tried to do with this is that in our paper based system in Sri Lanka, we had this different approval levels. So those were inherent data quality measures that were there at free levels. So when we were introducing DHS to what we did was we did not try to capture data to the digital system just at the same time. So we preserved the paper based system and we kind of asked them to copy the paper, the data that is captured in the paper based system into the digital system and also the approval workflows that were already there. We tried to implement it DHS to so that there was one step verification before the data goes higher up along the high rock, the health high rock so that the data approval workflows was a major contribute in factor for that assuring data quality in the CRHM IS or the digital MCH system. We conducted the training programs. In fact, like what you're seeing here is one of our field level training program and the person who is in the front is the one who is leading this entire MCH system. Dr. Kaushalya, I think she's a participant in this academy also. So much grateful to her and her team for implementing all this data quality practices in her system. So here the data quality measures and how they should capture data and at that point, how to ensure data quality was an inherent component during this training programs that we conducted at field level. So what we did was we had a national team who were kind of master trainers and we were able to replicate the training programs using our trainers who are at field level. So all these training programs had a special emphasis on ensuring data quality like, for example, especially at the point of generation, point of capture of data. There were so many data quality measures, which I will be showing you in the next few slides. So they have a special emphasis that we put on data quality during this end user training program. And one other thing about ensuring data quality is to have proper user manual. So it's so peace. The thing is like the DHI is to has its own extensive user guide, which like I'm sure that that may confuse many end users if you just ask these end users to refer the DHI is to use a manual which is available online. So that of course didn't work and we realized that. So what we did was we created the end user manual, which is of course, it's a very comprehensive end user manual with the screenshots and also like explaining everything in very simple, simple manner, which could be comprehended by end users. So this end user manual was something that we handed over to the end users during our training programs, which they can keep at their workplaces and they can refer. But the thing is like this is a kind of a comprehensive user manual, which may be a bit too, bit too much when you are just trying to find something real quick when you are entering data at your desk. So for that purpose, we had another type of manual, which I will show in a while. And the other thing is there were so many SOPs that we created before we try to implement the system, because like we understood initially that when you are having this digital transformation from paper based system, which was kind of well established into the digital system, there may be a lot of resistance that that we might encounter because for the simple reason that users don't like too much of change. So we try to simplify everything so that to make sure that the end users don't see this learning curve as a very difficult one. So this is where we had different levels of SOPs. So this one that you are seeing is a checklist that all the facilities should have before entering data into the system. So here we try to explain everything in a simple way. And this we have asked them to keep by the side of the computer that they're using for data entry. But the one that we have asked them either to keep at all times next to their computers, or else to paste it on the wall where they are entering data is these ones. So these are simple, single page end user manuals or SOPs that we have created, just specifically to given tasks. So these kind of, I mean, like how this, these helps in data quality is that the thing is like, if you perceive using a system as a kind of a difficult one, then there is high chance that I mean, that's this is what we have seen that there is a high chance that they might make mistakes. So having these specific SOPs and the user manuals with the end users kind of really contributed to ensuring the perceived ease of use of this system, which of course contributed to the data quality in the end. So these kind of manuals or SOPs we had for each of these tasks. And this is exactly what we followed in our COVID implementation, which was kind of rapid implementation from the learnings that we had from our MCH system. And this is another single page user document that the MCH system is using. And then we also tried to use remote administration software, so remote desktop softwares like TeamWeaver, because like how it really contributed us in ensuring data quality is that what we saw was that when we are trying to provide support to end users, there were some issues where they cannot actually explain even over the phone. So if they actually cannot solve their problem, they tried to enter data, which might be erroneous, right? Because like if they would assume that this would be the proper way of using and which is not exactly the correct way. So here we are using TeamWeaver as a tool where we can connect from national level to the end users computer and assist them in troubleshooting. So with this, we were able to in fact observe the data quality, I mean, like how they are entering data, what are the mistakes that they're doing, as well as it helped us in troubleshooting when they're having issues. So for this one also, we had separate SOPs that we created. And the other thing that really helped us in ensuring that data entry is timely, and like data entry users are having minimal issues, is that the user support. So for the to provide the user support, we had this three tier model. So here what happens is we had this, the initial communication method, which should happen using packs or the hotline or else we are using this instant messaging platform such as Vibe. So there, of course, they can reach our initial help desk for so for simple troubleshooting, let it be public health related issues or else very technological issues like this will be attended at tier one. And then in case they cannot settle it there, they'll be directed to the tier two where we have a medical officer in public health as well as our ICT officer will be attended to these issues. And if they can't also solve them, we have the consultant community patients who are attending these technical matters related to public health issues, and the medical officer in health informatics will be who is going to attend to the very technical or technological issues. So this way it really helped us in ensuring that all the issues the end users are having are attended. And when it comes to specific data quality measures that are already there in DHIs too, we are mainly using in most of these MCH data sets, the validation rules and compulsory data elements. In addition, we are also using the data. But we are in the process of in the incorporating min max predictors, WHO data quality tools, core cards and legends in these MCH instances. But like, why, why there is a delay is that we are trying to match all these tools that are there into the existing the workflows. So that is why it is taking some some time. But this is our plan in next over the next one year's time to incorporate all this into the MCH data flow. And when it comes to review and feedback, we identify review and feedback as a major measure in ensuring data quality. So what we do is we have created dashboards which and provided our end users instructions on how they should be following and adhering this dashboard. So they are supposed to use this monitoring dashboards at the facility level, district level, provincial level and at national. So they are having clear instructions that we have provided to the end users on how to use these dashboards. And in addition, we have the national level staff who will be drilling down and and conducting desk reviews from time to time, make sure that whether there are any data quality or any major data quality issues which are happening at the level because like this is kind of like a secondary level of monitoring for all the district level staff, which we are doing at national level. And to complement this, we have regular reviews, which is happening at the phm is the field level public health midwife level, or medical officer of health is the is kind of like sub district level, our DHS is the district level and at the country level in these frequencies that we have mentioned. So how this happened is that now this what you are seeing here is a kind of like a district level review, we are all the the medical doctors who are in charge of all these sub districts will come to this place along with few public health field level staff. And we are doing a comprehensive review. So these and one key thing is that is that for all these reviews, we are using the DHS to base them see a system. So we'll be displaying the dashboards and they have to comment on any of the data quality issues that that that the expert panel from national level are asking. Now these reviews did not stop even during the COVID pandemic. What actually happened was like it was another opportunity for us to further explore what are the avenues which are available to conduct these online. So what we do did was we organized the Zoom platform that we have and through the Zoom, we were able to conduct this monthly and quarterly review at district level using this the remote technology. So none of these review activities have stopped even during the COVID 19 pandemic. So what you are seeing here is the district level staff who are joining through Zoom and the national level staff who are kind of monitoring whatever they are presenting. Right. And these data that we have gathered through our MCH system is produced as an annual report from the Ministry of Health. And we also have a MCH quarterly report for all these reports, the data is coming through the DHS to base system. And this is why, like once these reports are public, are made public, the Institute and the Ministry has to hold the responsibility. So that is why we are trying to ensure data quality in our DHS to base system because the reports are actually generated through this system only. Right. So these are some data quality measures that we have ensured in our MCH system. So with this, I'm going to move on to our second example, which is the nutrition monitoring system. Right. So this of course was was a requirement which is coming from the presidential secretariat level of Sri Lanka, where the requirement was to identify children who are having nutrition related problems at fee level, and to obtain multi sector support to address them at fee level. But then the information had to be collected and verified and transmitted all the way up to the national level and shared with the multi sector stakeholders. So the major functionality of this system was to register the children with malnutrition at fee level. And then they wanted to routine the monitor and track the nutritional parameters such as height and weight and household risk factors. So these are the routine monitoring components and then obtain household geolocation and also perform analytics and the mobile users and to synchronize the data which is collected at fee level with the central server and then share with the stakeholders. So to achieve all these requirements, we felt that it was the mobile technology that we have to use and the traditional use of laptops and computers to enter data will not really work because the data had to be captured at the individual level at fee level. So this requirement of course came around four to five years back. So at that time, the DHS to Android native mobile application was not that mature because nowadays it is having so many useful features. But those days there were some restrictions to fulfill these functionalities that were that were required by the program. So what happened was this public health midwife who's the fee level health staff health worker will collect data at fee level, which is then synchronized into the DHS to base central server and which could be accessed by their supervising officers and all the staff at district and national level using the web interface. And once this data is sent by the midwife, they are immediate supervising officer who's the medical officer of health or the public health nursing system should validate that data and the data had to be returned back I mean this validation. Once it happens only, the data could be shared with all the stakeholders who are involved in the process. So to do this, of course, what we did was to have this design this custom Android application due to the obvious reason that some of these requirements were not be able to meet by the DHS to Android application at that time. Right, so let's quickly look at few data quality measures that that were there in place in this Android mobile application while capturing data. So the first thing is like there was some there were a few simple interfaces. And now as you can see here, these are the these are few interfaces that we used and the same and simplifying interfaces itself, both identified as one major factor to enhance the compliance of the end users who are using the system is filled and stuff and also to enhance the data quality because like what we found out from our study was that when you when you have very complicated workflows, and very crowded interfaces, it contributed adversely to the data quality because the end users can make a lot of mistakes, because they just become confused due to the complexity of the interfaces. So using simple interfaces one was one major practice that we followed. And secondly, because they were collecting this height and weight values of the children with malnutrition, what we did was we provided them a visual verification of the data as they entered. So if you can focus in on this height and weight values that are entered here, you can see that color of this one is greenish. So that and this one is kind of like a yellowish amber color. So what happens is now when someone starts typing height, what will happen is like, there will be some age verification based on the so we have the date of birth of the child and today date and with this, we'll be able to know which which is the age and there'll be a kind of a verification with the comparison of this WHO. There are standard ranges, the reference ranges for different standard deviation values for height and weight for that particular age. So if the the height of weight value is okay, it will be the kind of a greenish color. If it is slightly bad, it will be amber and if it is really bad, it will be red color. So that way, there'll be a visual verification that happens at the point of data entry, so that it will give an idea about if there is a major mistake done by the end user at that point itself. And then of course, to further assist with data quality as well as to support the interventions, what we did was we provided some longitudinal charting. So these are kind of like analysts analytic outputs that we integrated into the system, which contributes to enhancing data quality. So here, what happened is like, in the mobile app itself, the end user after entering the height and weight values can verify by observing this height and weight charts and see whether there is a I mean, kind of a significant abnormality in any of these values that have been entered. So there'll be a kind of this kind of like a dual verification of the data that was entered by the end user. And in addition, they have a simple reports and simple analytics which are available to the end user in the mobile application itself. So it kind of provided them a very brief overview about say like here, what you can see is like how many children of each of these nutrition problem where they are under her care. So that kind of analytic outputs are available for the end user. Right. So once this data was captured at the field level, then next there was some some validations that happened at facility level. So for this one also, there was a custom web app which was designed mainly to cater the requirement of data approval of tracker data, because this is tracker data, right. So this, even though it's collected by a custom Android application, the data is sent to the DHS to system. So that means all this has to be in compliance with the DHS to data model. So this tracker data that is entered here will be verified by their supervising officer. So the supervising public health system has to go through this data, and they will be the ones who will be marking the mess completed after ensuring that they are adhering to the, I mean, at least like these these are plausible values. That's what they can maximum, I mean, at maximum level they can do. So once this is done, there will be a data approval, which happens at the medical officer of health, the moist level. And once the data approved only, the data will be visualized and shared with other stakeholders at provincial and national level. And in addition, there were several data quality measures that happened at various other levels, which kind of existed in this entire process, because like the data had to be shared with the multi sector stakeholders. This was a crucial step for validation of data that is captured at the field level. Right, so I think I have consumed almost 25 minutes so slightly over, so I will stop it at there. And if there are any questions, I'm willing to take it. Thank you so much. Great. No, thank you so much for that was an outstanding presentation. I really appreciate how you touched on so many various elements of data quality that people don't always think about. We're always so you're exactly right. We're always so concerned about checking the data once it gets into the system and the different tools that we have in DHS to, but you're you hit the nail on the head, so to speak, when you talked about like form design, standard operating procedures, training processes, you know, all of these things are absolutely critical to data quality. And I'm really we tried to make that point yesterday. And I'm really glad you were able to drive it home today. We did have one question from the community. And again, everyone, we are taking your questions on the slack channel, just like we did yesterday. So please, if you have a question, post that to the slack channel. There was one question. They are asking if you faced any challenges or you had to change anything during the implementation of these various reporting forms and data quality checks. So once you started actually implementing, how did you were there any changes that you had to make in the process? Well, okay, so as I mentioned, one crucial fact was that we had to minimally disrupt the existing workflow. So one major thing, I mean, like, this was not a very kind of abrupt implementation. So there was, I mean, especially my other colleague from Israel, was the one who was greatly involved with this MCH system. So there was like, at least more than one year of planning in in assessing all the workflows and paper forms to ensure that I was and there was, I mean, the other thing that I did not mention here is the testing and piloting that goes into the entire process before you actually implement the system because like, most of these data quality measures should be captured at that point, if the system is not capturing data, I mean, like, there are errors in data quality that you observe during this piloting process, those have, I mean, you have to come back and reverse whatever the workflows or the data sets designs that you are following. So there was a lot of planning. And that's how I mean, after doing that only, we went for the first round of implementation. So there were a lot of issues that we found at that phase that were corrected. And then what happened was, during the initial round of change training that we did all around the country, there were some feedback that were coming related to different issues with the data sets and accessing the system workflows, things like that. So during that time, we accommodated a lot of changes that came from the from the end users. But after the first round of implementations, we kind of like what we did was we received feedback from the end users, but we did not go on changing the formats, the data formats from time to time. But what we informed them was that if you need us to accommodate something in this, the designs of the forms, so this will only be done, maybe at the start of next year. So we kind of like, if they are not really critical, we kind of collected all of all these things. And we discuss internally within our team. And maybe at the start of next year only, we will accommodate them into the DHS2 based forms. But if they have a major changes that required, maybe collect, I mean, like to decide whether this data item has to be collected and all, that of course had to go to the public health specialist, and there has to be a decision that is that is required, which is of course, totally out of the scope of the DHS2 implementations. So yes, the other challenges, but we were careful not to entertain and include all the changes made by the end users, but we reviewed and incorporated them on time-to-time basis. I think that's quite clear. Thank you for that answer. A couple of more questions. I know you're a busy man, but if we can just keep you around for a few more minutes. Question from Lisa Grout. In the application that where you're tracking individual children, do you do the validation child by child? Yeah, good question. Yes, the thing is, it's a very difficult thing. I mean, like this is one reason like why we try to incorporate all these, the color verification and things like that at the point of data capture, because otherwise it's going to be a very hectic task for the supervising officer to go through all the records. So what we have, I mean, like, we have made provisions in the system to verify child by child, but when we are implementing what we have asked them is like, because the thing is, these are kind of like when you try to have these systems implemented in a couple of districts or provinces, the infrastructure could be even technology, mobile, I mean, like the network, as well as human resources, they tend to greatly vary from district to district or even from health facility to health facility. So what we have instructed them with, like, you always need to like, this is, I mean, the system has a potential, but like, it really depends on the capacity that you have in your health facility. Like, if ensure, I mean, like ensuring data quality. So there is always a kind of a fine balance between data quality and actually capturing data, because like, if you have minimal capacity and you try it and your, your priority number one is ensuring data quality, then no data will come out from that capacity. So it's kind of like we have given them the choice. So this is what the system has. This is the best practice, but it's totally up to you to decide on what you should do. But we have given them some hints as to like how to capture if there are any major data quality issues. Like, like, I mean, we have asked them to observe the data entry practices of the end users. And if they identify from the preliminary study that there are a couple of end users who really do these mistakes over and over again, then you can either focus on them when you are validating the data or as to provide them kind of refresher training and guidance to ensure that they are not doing those mistakes. So there are I mean, like the simple asks is like, we have not kind of strictly enforced these rules and we have asked them to, I mean, we have make it so flexible so that the end users can, the health facilities can decide which is the best option for them. Right. Okay. Thank you very much for that. The questions are coming in quite quickly now. It seems like people are really intrigued by your presentation. Just two more questions for you before we need to take a break in and move on. We have one question. I'll just go ahead and ask both of them now. We have one question is, how long did it take to implement these various data quality checks? And then the other question is, what happens if suspicious data is found in the review process? How do you go back and make the change and how to keep track of the changes that you've made? Yeah. So I will take the MCH instance for this example. I mean, the two questions are, so like the MCH instance was initially set up in 2017 and now we have passed three years of implementations. So data quality, ensuring data quality was not the number one priority during 2017. So it was in 2017, it was mainly about addressing user issues in accessing the system, entering data and to ensure the completeness. Even those are data quality measures, but like we didn't go into this finer data related issues in the first year. But then from year 2018, 2019, these years we focus mostly on the data quality. I mean, like things, especially we put a lot of emphasis on during the reviews because these reviews used to be happening based on the paper data that was I mean, previously before 2017. So then there was some transformation required from 2018 onwards to use the system for this review meeting. So it took some time into that. So it was like, so what I would like to say is like, if you see our plans for future, so we have more plans like compared to what we have done so far. So it's a work in progress. So we have achieved some level as of now, but we have not to do. So we can't say like we have ensured data quality in these last three years. We have done, we have taken some major steps, but there's not more to do. So it's really about like what you try to achieve. And sorry, what was the second question? What happens if you do find suspicious data in the review? How is it changed and how do you keep track of the changes? Yeah, the thing is like now that of course, like what happens is like they try to understand what is the issue with this data during this review. So for example, now I mentioned about these desk reviews that our staff is conducting at national level. So during the desk reviews, if they identify some some data quality issues before the reviews, they will contact the health facility and advise them. And if for some reason, if the same mistake is happening over and over again, they really try to talk to the health manager or the public health doctor and identify why it is happening and try to like if they identify the issue that this may be happening recurrently. So then there may be some training gaps, which if we identify, we even like what we try to do is if we feel that in a particular district that there is the same mistake that is happening over and over again, we try to organize a refresher data analysis training where we also talk about data quality during that process. And then the other thing is sometimes like, I mean, to ensure the timeliness, we lock the data after a particular given period of time. So then we have to put what we do is we will put lock exceptions and after accepting that it's a mistake and when they have like the end user, I mean, at the facility level, they accept that it's a mistake which has been communicated to national level, then only we will put lock exceptions and unlock the data for it to be correct. So those kind of measures are what we do when we find out there are some data quality issues. Okay, that's that's very clear. And again, thank you so much Pamud for the wonderful presentation. There were many quite a few questions that I don't think we have time to get to we