 So welcome everyone for our session. So we'll be looking at DHS2 and the country digital architectures. And so we'll have presentation from various teams from South Africa, also from West Africa, and also we'll have also a presentation from PEPA and also another presentation from Tanzania and also the last presentation will be from Rwanda. And so the presentation will be around how different integrations and how the different, yeah, how the different implementations are, they fit in within the broader systems architecture or the digital health architectures maybe within the the countries they're implementing or within also the regional they're implementing or within also the global scale based on how the different things are tied in. So we'll be able to, yeah, look and if you have any questions kindly just we'll give you some time and it'll be able to respond. So without much taking time I'll invite the Firstly Centres will be able to start us off. Yeah, thank you Sean. Right, thank you very much. So Sean and Elmerie will be doing this together and those of you that were in the last session on AI will see that this is a little bit of an overlap from the AI machine learning session into into architecture. So we're from history Africa, which is by far the oldest and the largest his, which means we've made the most mistakes and we can tell you a lot about how not to do things. And hopefully a little bit about some of how to do stuff right. We'll start with this picture. I studied medicine is where I started my career and looking around the room aware of a similar age but there are a few that might have been in primary school when I studied medicine. And what's been very interesting for me over the years is to see how thinking around certain diseases has changed so for example when I was at medical school, we didn't know what caused cervical cancer. We knew women died from cervical cancer. We knew that it had something to do with having a lot of sex and the more sex you had and the more partners you had the more you died of cervical cancer. But the idea that it was caused by a virus and that you could cure that virus with a vaccine and therefore women wouldn't die of cervical cancer was not part of our thinking in those days. So how did we get to figuring that out well there were lots of randomized control trials and there was a whole lot of science and medicine that helped us get to that point. So that's for a patient if we thought the thing about our health systems. In many ways, we want to do the same, we want to be able to understand our health systems in a way that we can manage our health systems better and make better choices so that our health systems get stronger. But health systems not a person. You may have heard them described as complex adaptive systems and because they're so complex you can't do a randomized control trial of health systems. So how do we start figuring out how to make better decisions about health systems and I put this picture up so so that feels got a very diverse and very landscape. You have some environments that look more like this and some environments that look more like like Oslo. You get across those environments you need to start making decisions about what do I need to do in this health system that's going to work. What does this health system needs today and more importantly what's this health system going to need in five or 10 or 20 years time. And this is where the technologies like artificial intelligence and particularly machine learning allow us to do some predictive work that can be can be pretty powerful and help us hopefully start being able to diagnose what our health system needs before it needs it so we can take action. Alright, so that's just the intro. These are the things we're going to talk about. I'll make some introductory comments and get into a little bit of architecture and then I'll hand over to Elmerie and she'll get into an actual machine learning example. So part of what I started talking about in that introduction was was this landscape that what we do most of the time with DH is to somewhere around the bottom here. We look at what has happened. We try and understand why it happened. We try to figure out and get into what will happen next and associated questions around well how can we make it happen if it's good and how can we stop it happening if it's bad. So we largely stuck at this at the second step. What we want to do is to get very much into those into those next two steps. And what we believe absolutely is in the next decades, public health practitioners and people working with data like DHS data are going to be working at the top two steps not at these levels if we do our jobs right. So we're going to talk about a particular use case which comes from our human resource information system project in South Africa where we built a health workforce registry. And then nine things we want to do in this use case, we've done the first two. We have a sense of, of how far we are on this. We started this journey more than two years ago where we decided data science was important to us we needed to start thinking about artificial intelligence and machine learning and we needed to start employing the kinds of people that could do this work. And we spent the last two, two and a half years getting past those first two steps. So the big message for those of you wanting to get into these kinds of projects is start investing in people start investing in getting the right sorts of people are going to be able to help you on this journey. And today we will talk a little bit about those first two steps and then get in summer into some of those next ones. So I'll spend a few minutes on this very busy architecture diagram because that starts speaking also to the core subject of this session. What we're also discovering is that our work with DHIS is so fundamentally different to what it was 10 years ago. We're 10 years ago with the DHIS alone you could do a lot. You can still do a lot. But if you're wanting to get into the kinds of health systems planning and starting to prepare our health systems for the future and for the challenges that they're facing. There's a lot of other things you need to be doing alongside the DHIS. And what this project around a human resource information systems recognize immediately was that the DHIS routine health information system data was going to be really important because we need to know where our burden is going to be. But it's going to need to be combined with a whole lot of other data. In the top box there we need a lot of data that describes the health workforce. There are nurses who are doctors. Where are they coming from? What's the pipeline like? How many are we training? How many started medicine this year? So in six, seven or eight years time they're going to be coming through. What does that pipeline look like? So there's a whole lot of other data that we're going to need to pull into our environment. We won't go into all of those data sets. If we go into that environment, then there are different things that we're going to need to do with that data. And the red boxes describe the work that we're doing around this specific use case, which is to look at attrition. And so the question that was asked for us, asked of us with our human resource information system is, is it possible to predict in five years time one, where you'll need which categories of staff. But two, how those staff will be changing down to the point that if there's a nurse in a facility working in TV, what's the chance she's going to be there in five years time. There's a chance that she'll resign down to the absolute specifics around specific categories in specific areas. And that's the, and that's the challenge that we face. Is this a good point for me to hand over to you. Do a bit more on the architecture. Thank you, Sean. Okay. Yeah, so we, as you can see, there's parts of DHS to we're including a fire registry, which looks at use the MCSD profiles for staff. And then these red boxes are the HR is representing the HRH planner module that we sort of as a group, which enables the health manager or the HR manager in the Department of Health to plan for universal health coverage, as you all know that there's the WHO targets to achieve universal health coverage by 2030. And these components is what we have used in this use case to do the machine learning. So that module is specifically use the for the machine learning using Apache Python. The front end of that is a streamlet application which we can then allow users to run the back end machine learning. And then we have analytics feeding through a web portal, DHS web portal, through Apache superset. Thanks Sean. So, in our machine learning model we follow the quantitative approach of data collection or the raw data from DHS to we have HR and financial systems that dump data onto a cloud environment. We're using PG admin and to a dimensional data model to extract that data out to store the data and we extract it from there. One of the big parts as well, more or less 60% of your work is cleaning that data describing and visualizing the data, understanding what data you actually have. And then to do the modeling of the data, looking at the actual versus predicted data. And lastly, and deployment through pushing the data that we have generated back into DHS to through streamlet. And so, as you could see one of the big things that we have done so far is we are able to predict who is going to leave the Ministry of Health employment in the next six months. And we, what we are using for that is the act of headcount up to the latest data and we've got data for a few years and we're using termination data from 2020 understanding why people left, mainly looking at permanent employees and contractors leave because they contract ends. So that's not really giving us a reason why, and you don't intend to keep them but other people, are they leaving because they resigned because they were disciplined and fired or they might have retired or died or whatever. And we're looking then at, at all levels of termination voluntary and involuntary. And we also look at the educational level of the employees that are leaving the source systems we are taking that from HRH data, education data that we have modeling through the streamlet and Python tools and then use superset for data extraction and data visualization features, looking at demographic information of the person, the job information job title the reasons for the termination and where they are actually located. And then for the machine learning we're using machine learning algorithms which I'll talk about now more and advances the SQL analysis. So, what we have done is, in our model accuracy random forest and logistic regression was fitted, and below other results for both models so random forest was 801,9% and logistic regression performed better, and was used for this modeling. And so what you can see here is the features of importance in the highest drivers of deciding whether you will, a person will turn or not. So age, employment title, the tenure how long they have been in employment, the turnover rate, the salary level, number of qualifications, and so on up to raise generation country of citizens shopping less important in deciding whether people would turn. So from the streamlet app. It allows you to filter different predictions and deciding what you want to filter on, and also basically tuning your results in terms of precision, ensure that you don't overfit the model. And then you can see here just a screenshot of the results so we've taken out the individuals identifying information. But basically we then get a list such as this to say that these individuals are likely to turn in the next six months. We do this on a monthly basis and give that to the to the Ministry of Health. And so what we will do now is to also know that we are able to show them this we are, we are able to convince the ministry to give us longer data back than 2020 for us to more accurately predict. And so, and we are making this list of high attrition probability available for the provinces to work with. And that list can be used to for retention, retention interventions, and conducting these people finding out maybe doing surveys satisfaction surveys, or else planning for replacement of those if it's going to be basically unavoidable that they. Thanks. So in the Prolonga province, we have done a visit to them. And there are also specific data quality improvements that we have been able to work with them on on so that the data is better sometimes they don't capture the reason for leaving. So, or the HR system, some of the data is quite poor. So in looking at the output of this we are also feeding back better data quality into it. And now we are starting to work with him on expanding the model to include what is the current staffing needs for universal health coverage. Our next steps is to maximize the attrition benefits through looking back and then these interventions and measuring the impact of those interventions. Then to determine the current needs for you to coverage and predict the future needs for you to coverage. And that is essentially the big part of the presentation here is a user satisfaction survey chatbot code that we use on all of our systems where we survey users to see are they happy with the performance of the system the how the system. How easy it is to use. So, you're also welcome to run through that. If you have telegram installed on your phone. And that is essentially our presentation. So, we just need to thank one of our colleagues is online, Christopher, he's done a lot of the technical work behind that so well done Christopher, if you can if you can hear us. And what we believe is this is one tiny little use case that is going to begin to be part of a whole collection of use cases are going to be part of how we intervene very directly in our health systems. So, we're trying to get a very large generic programs we're going to get down to very specific information like, always likely to leave in six months and what am I going to do about it. Thank you very much. Thank you Sean and Marie for the presentation so as we set up the next presenter, Damola sheriff from West Africa Health Organization. Yeah, so we can maybe take one or two questions as they set it up. No one question, no one question. Yeah. Yeah. So this is to something you mentioned very early on in the architecture is a triggering word for me is MCSD. So how are you using that in your architecture and how did it play out into the learnings you had there was any correlation and just something that was available in DHS to that you leveraged. The registry is set up with MC is using MCSD profiles profiles for practitioner and practitioner role. Because of that we are able to interface with other systems that need to use a provider registry in a sense so, and the bigger aspect of this HRS system is to have not just the Ministry of Health data but to a comprehensive resource and a resource of all HR data. So it's part of the architecture in terms of individual data. So we don't have the individual data in DHS, the DHS stores the aggregation aggregated results. So that's how those profiles could be into the, into a fire registry. Yes. Okay. Thank you. Thank you very much. I think so this is working up is going. It's okay. Okay, thank you. Yes, we are coming from West Africa. That's the West African health organization. So we, I just want to invite you to see this presentation follow this presentation, not only in context of one country, because was the colleague I just showing us now what is happening in one country so that's the Wahoo or in Yokoas community is among 15 Yokoas countries. That's all these 15 countries, how to put the system. And this experience was requested from us when the experience very bad experience with Ebola in 2014-2016. And then the minister of health ask Wahoo to put in place one platform that can allow any country to know what is happening in terms of the epidemic situation, the epidemic prone disease to be put in place. And the challenge is so how to work with the different 15 countries that need to build their own system to put work and then how the regional system can be bullied on the data. This thinking, this experience we will show you here. Analyze it in this regional context, not in one country. That's the more you can go. Okay, thank you. So, to me just describe who we are. In summary, this is the outline of our presentation. This is 15 countries, approximately 400 million people single economic block and we have free movement of people. So epidemics happening in one country is usually not limited, as Thomas said, from my experience with Ebola, it could be spread across four to five countries. We're giving a mandate to set up a regional platform so that countries can know what is happening in the region. And the objective of our regional platform is to enable timely access for health data in the region to all our stakeholders. After the mandate was given to us to put in place the platform. We had a lot of trial and error failures along the way. And over time, we learned, and we took all these lessons and we finally adopted DHS2 as a platform for the regional data warehouse in 2014. And since 2014, we've been building on this platform, even building things like public dashboards and everything on this platform. To us, our internal objective of having like a regional health observatory for the ECOWAS region. So this is where we want to go. So we want to have like, we want our regional platform to be part of our information systems architecture, where we have data coming from the countries from external data sources, and passing through data pipelines, formatting them for ingestion into the regional data warehouse, which is based on DHS2. And in the future, having things like machine learning and artificial intelligence based on the data we have. And also having things like data explorer for the, for researchers and other people, and also being able to store unstructured data in a way that is accessible to our stakeholders. And we want to want this platform to be a collaborative platform for our member states and to enable timely availability of quality data. And for us to be able to develop health information products to communicate situations to our stakeholders, we just don't want to collect data, want to be able to process the data, analyze the data and communicate back to our stakeholders. So, in order to do this, we collect for epidemic prone diseases, we collect data for epidemic prone diseases from member states, and we have a lot of challenges with that. And one of the challenges we have is that we used to collect it manually. The people have to log in from the 15 countries weekly and enter the data manually. And we know that our countries have internal systems for information systems management so we started talking with them how we could get this data into the regional platform by minimizing manual processes. And we discussed with them about interconnection and everything, but there were a lot of concerns and one of the primary concerns was privacy and data sharing concerns, like on what framework would this be built upon. And who is going to take responsibility, there's a problem. So there was a lot of skepticism from the countries, and we have to do a lot of advocacy to bring them on board, especially in terms of the buying. Then, another challenge was, sorry. So another challenge was in terms of the manual data reporting systems, there are a lot of turnover at the country level so the person entry data can be reassigned tomorrow and we'll get data from a particular country. And finally, even if we're going to do like an automatic integration, different countries were using different systems. And three major systems are Somers, DHS2, and MACPY, like totally different systems for information management at the country level. So one of the, so for this challenge is what we did was to address the challenges one by one. For the concerns, we developed the data governance policy at the ECOWAS level because in terms of mandate, we have a mandate to collect data. And in terms of the concerns of the country was how we managing the data. So we developed an internal data governance policy for WAO that specifies how we collect the data, how we use the data and how we disseminate the data and how we secure the data at the regional level to assuage the concerns of the member states and other stakeholders. And in terms of the automatic process, we realize that we cannot just jump from manual to a fully automatic system. So we developed this, like we call this semi-automatic. Basically, the countries compile Excel sheets summary every week. So we built an app where they just upload the Excel sheet and we extract that into the DHS. But that still wasn't, you still needed someone to go and upload an Excel sheet. So and finally, we developed like a completely automated process. And the process we went through is we did like a review and assessment of the tools of our capacity internally at WAO. And we decided on a particular framework we're going to use and also platform and everything. So the platform we use is Apache Airflow. And what we did was we set up the Apache Airflow and we did what we call operationalizing. We call the countries for like a regional meeting in batches, batches of five, because there are 15 of them. And at this meeting we created what we call mappings, org unit mappings, between the org unit at the country level from the district and the regional platform. We call it data element mapping from what they call data element, what are they call it indicator data elements, what are they call it? So we saw the line in in mark by or whatever platform they were using. And when we add this mapping, we're able to create like DAGs. Airflow has this concept of DAGs, where you write the code for execution. And this DAGs is customized for each of the countries based on the on the mappings. And after that, we execute it. And one of the reasons why we set it on Airflow is that Airflow gives off instrumentation capacity and constantly when the automatic process has run along with through whether it failed, why it failed, how often it fails and whether we need to do something about that. So basically this is the process I just described. So, for each DAG, we fetch the country data, we load the mapping for that country, we load the data elements mapping for that country. We do some merging, and we format it in DHS to import format, we send it to the DHS to, and we log what were failures or successes, and that's the end of the process. So then we have agreement to member state on when these DAGs can run. So right now, for automa for data transfer in the Equals region for epidemic prone diseases, this is our overall architecture, overall data flow. So we have 13 of our 15 countries sending well we're pulling data automatically from 13 of our 15 countries weekly at different days of the week. This is based on agreement we have with them. And two other countries are still using the old semi ETL process where they upload Excel sheets on our platform. So, and at the end of the week we have almost all the data from all the countries, and our problem is no longer data set completeness, usually more about data element completeness. So based on this information that is now on our platform, we're able to now view public public public dashboards, and other information products that we used to communicate with general public. So these are two examples that will put in our presentation we have the outbreak dashboard, which shows for COVID-19 data from 2020 to 2023. We have for the other ideas are ideas are epidemic prone diseases cholera anthrax AFP polio, and the likes. So for the next steps. We're trying to scale up this process to other programmatic areas like HIV and tuberculosis, like program data from the countries want them to be reporting automatically so this is more or less like a pilot for us to see if we can do the, if you can solve the human problem. And the technological problems that we cannot replicate the same process with the programmatic area so we're currently working in malaria, nutrition and I else. So, so we're working on all these people trying to bring the data into the regional platform. I'm also looking for how we can bring even survey data from public data sources, because for us at the regional level we are not really tracking program indicators. So we are more or less trying to see information and extract information for policy action and policy intervention at the regional level. So there are a lot of information out there. Survey data, socio economic data and all of that. So we're trying to see how we can bring all this as well, using this platform into the regional data warehouse, so that when our statisticians internally need information they don't need to be doing manual data extraction. So with the internal analysis capacity of the DHS to they can be able to do some basic analysis then extract data for further analysis in a simpler way. In conclusion, so even though our countries all use different systems internally, they all have different administrative processes internally, we've been able to make progress, at least 13 of the 15 countries are connected automatically. And this is a stage for us to one in the surveillance system to build a digital warning system, because we're going to continue to work on this while we scale to other areas. So we're going to work on this to do automate automated notifications, so that when a district in the cross border area report some suspected cases of epidemic prone diseases. We're going to modify neighboring countries to just to take notes, because right now this is done manually but since we have this data now, we can automate all this process so this is the next step for us. So, I want to say thank you. Yeah, one of the points we need to take in consideration. This, thank you. Thank you. This is line to say so this context of regional. Any sentence you are writing working at it was need to be sure so you are in three languages. Because we have three speaking a different language. That's all. So anything we need to do it three times. So be sure. That's all. Thank you. Thank you so much for that presentation. So we'll have large set up as we take one question. And also if you have any question you can write it down and we'll be able to have it and you can also be able to reach out to them. Anyone with a question before we have the next presenter. Yeah. Yeah, let's just give it to him. Thanks. So you mentioned that 13 countries that are contributing data to the system itself. From your workflow look like you took the responsibility of connecting to those systems and pulling data out. Is there agreement with the providers of the systems to maintain your ability to have access to that were you working to a common format. What happens in three years time when so much upgrades. How are you planning for that. So one of the things we discovered during our initial assessment is that the capacity at the nature of health keeps changing and fluctuating. So we can depend on that. So it's easier to build this capacity at the regional level and maintain this capacity in terms of technical capacity to do API implementations for example. So, when it changes our team will do the technical work to make sure because the connection will fail. So we'll go see why it has failed. We'll call the countries and we'll call the country and we'll try to work with them. So we also have forums where like we meet with the countries every year. Next week we're going to meet with all the all our 15 countries in what got to go. Later this year we'll meet with them again. So we usually meet with them. So when things are changing at the operational level, we are keeping track and we keep contact with the new people. We also have a WhatsApp group that we used to communicate information with the countries and they also used to communicate back with us. So it's a combination of people, processes and technology. Thank you. So just take and maybe we can. Thank you for the presentation. So I'm quite interested about the regional agreement and the policy you develop. So the current data that has been shared within the region is it indicators like statistics or it's including the individual, like a case data or how does it relate to that the whole IHR focal point and yeah, how does it relate. Thank you. And the date the hosting is where Let me take the last one first. The hosting as its own that way that server center in Burkina Faso there. We also have cloud providers. We use etna in the European Union because European Union GDPR is a bit compatible with our data protection policy that we have that wow then in terms of the data we collect it is aggregate data is not patient level individual level records, but the protocol that sets up our institution is based on a treaty among the 15 countries is very strong so even in the future if we want to go to that level, we can go there. Thank you so much. Is that one too. Oh, okay. Hi everyone. I'm glad she is really I'm with the ICF, I'm supporting PEPFAR and I hope all of you are familiar with PEPFAR is everyone likes to ask that it used to stand for inner city fund but it's just an acronym those days that's known as ICF only we're just a contractor on the PEPFAR program. Alright, so this is South Africa and echo us, it's actually like one level higher so we started with a national we went original and PEPFAR is a global program. So it is HIV, mainly so we don't do a lot of different disease areas but it is a global epidemic that we're trying to fight and PEPFAR needs a lot of information in order to make decisions in this process. So, as part of this decision making process, we need to get the data from co host countries and a lot of times what happens is that the data is reported to the ministers of health to PEPFAR to other different organizations, and there's a lot of duplication of the data and we don't want to ask people to report the same data many many times so we have this collect once use many times paradigm that we're trying to push forward and the DA is really a view of national programs trying to figure out you know what's happening within each of these countries. PEPFAR supports over 70 countries, however, you know, for the MOH alignment we started small we start a little over five years ago with 11 countries from the PEPFAR priority list. PEPFAR priority country list is 23 countries which make about, I think, 80 over 80% of PEPFAR support. And now we support 23 countries through the DA alignment. The short term goal was to take the data identify the reporting challenges where the discrepancies are between what the Ministry of Health has in their data reporting versus what PEPFAR gets directly from the partners that support PEPFAR. And then to use this data to improve the processes and overall program efficiencies. The long term is, you know, to actually take the data from MOH instead of collecting the data directly from partners. So as a part of this MOH alignment activity there is some parallel reporting. From system to system and electronic exchange of data as part of the MOH alignment so we're not asking people to manually enter data twice, but what we're trying to do is find this flow so that we can replace the reporting to PEPFAR directly with the data being sourced from the Ministry of Health. So, I'm just going to do a non diagram overview of architecture. A lot of the thing that Echo us mentioned actually applies to us as well we're doing something very similar. We went for DHS to platform for a couple reasons, main being that we already use DHS platform for datum, which is our big hub for collecting the Mer data which is the HIV indicators, but we also have other data streams that are all collected in datum. We're not familiar with it, not just our users the PEPFAR users but also the Ministry of Health user users and others. And DHS to is quite mature when it comes to aggregate data collecting, you know, the other parts of DHS to might not be as good, but the aggregate is definitely there and thanks Lars. But it cannot do everything. I know, you know, DHS to is supposed to be a Swiss Army knife, but it's not. So, in order to achieve some of the shortcomings with other global goods to supplement those. Echo has mentioned facility list harmonization, we have the same issue. Now we have our DHS to with those 70 countries. We have our unique identifiers for all those facilities. Ministers of health that have their DHS to is with their unique identifiers names don't always match because sure there's master facility list but does everyone actually always use the master facility. So what we did is we used global open facility registry go for tool, which actually is a global goods developed by Interhealth, which bolts on into DHS so to say there's an app that you plug in into DHS to it uses the metadata from DHS to and it allows you to do side by side comparison, and then mapping of the facilities so that you can use later this mapped data to do the data atl and then reporting. Similarly, we have metadata that needs to be mapped similarly to echo us. For that we use open concept lab the OCL the terminology service, but OCL when we started using it didn't have any front end that would be easily usable in our environment so we actually use the DHS to platform to create an application, which is another structure of DHS to that makes it easy to use it. And then there are back end processes that OCL for example it's not just the front end it does require back end integration. There is the metadata mapping OCL integration go for integration in order to do all of this we use open him and the custom mediators to do this. There's no route API is coming to DHS to so we probably will look at it if we can replace it because, you know, there is a reason to scale down if it's possible so if you have a new feature that can replace the system that's sitting there, why not you know, just use one system. There's no analysis outside of DHS to as well. Again, DHS to can do a lot but not everything so we use our shiny where users are able to look at the data from a perspective that is not available in DHS to so we have a facility level data so it's pretty granular. And smaller set of indicators we don't do a lot for this activity, but you want to do comparison side by side and our shiny which is a web version of our code gives us that. And then I popped in the architecture diagram there I don't know how many of you are familiar I hope all of you are. So, obviously we don't use every single icon on there but the ones that we're using are basically there there's the interoperability through the green bar there is the HMS there is terminology service there's facility registry and so on. This is just screenshot of the apps we developed so we have that in indicator mapping up we have the gopher. There is the import app that is used to upload the data there is the dashboard that we use for monitoring what's happening with each of the countries and you know these are all running within the DHS to so you know people do not have to use multiple different platforms to do all of this stuff. And this is really the only slide I have left I'm trying to make this short but you know there is lessons learned in this process obviously. How far is data hungry, we do like a lot of data, unfortunately. I know the first year I came here college was in the room and you know, like how much more data do you guys want. You make decisions to you know you unfortunately need data and if, for example, you're treating HIV for decades. The age bands now that you want to access get finer because now there are people above 50 years that are leaving. And in order for you to actually address the issue, you are not satisfied with 50 plus to group them all together right you want 50, 55, 55, 60, 60, 65. And, you know, you introduce that that means you're introducing new data points and new metadata. But just saying that I want, you know, 9500 age band, because you think you need it is not a good idea right you start with analytics. And you go backwards saying like, what do I need from the analytical perspective to drive what the metadata is so that I don't just ask for 100,000 different data points and I only really need five. So, multinational program, you know, like regional program multinational program requires unique approach you know you need to actually collaborate with all the different players it's not one country where you have one ministry of health, saying that you know you do this. Different countries have different timelines that have different processes that have different maturity levels. Some are technically savvy, some are not you know some need a little more support in this process. You have data, much higher granular level some don't. So that goes into my next one, the data, but then it's to be a validation process because you're getting data from different sources. Validation is important so that you just don't end up with garbage. If you just allow anyone, you know, 23 countries to send whatever, and you lose the picture of things. And as I said, flexibility is necessary. Different countries with different timelines different technical skills. So, as you ask for more data, you need to realize that you cannot be inflexible. And then communication and capacity building is the key so support the teams that give you data so that they'll be happy and you'll be happy. Thank you Vlad so we can take a question as we set up the next presenter. Thank you for the presentation. So, in terms of the meta data from pepper, has it been like published and then in terms of the granularity is it down to like data element the most atomic sphere. I mean, atomic level of data element and you mentioned about originally thinking about OCL but now it's not, but where will this meta data be hosted, because I think within the country we do I know the pain. Yeah, whenever I got a new requirement and then we have to try to configure it so if it's possible to automate the whole process and we can generate whatsoever people want and that will help a lot. Yeah, thank you. Thank you. Thanks Joseph. Yeah, I'll try to answer that quickly so that the irony has a chance to continue. So, the image alignment indicators are actually quite limited so it's not as bad as paper, you know, paper has hundreds of data elements and thousands of category option combos. So for image alignment we only have seven indicators that we collect, and they're all published. So, I can give you the link if you want. The OCL has metadata sharing platform that's out there. I don't know if you're familiar with it. I know in Malawi they were demonstrating. John was there and John. It's machine readable. So, if you want to use OCL you can definitely get the PEPFARs both the MER and the Ministry of Health alignment indicators, they're all out there. Yeah, the data element category option combo and indicator so the metadata sharing platform is this really cool thing that they've developed where it's actually if you take the MER guidance that PDF. It's actually in a form where you can use it almost like a wiki where it links and you can see the connection between different indicators and what data elements make up an indicator. Vlad. So, we'll continue on and also at the end also we'll have a, we're doing well with time so we'll have some time for questions. Hello. Hello. Do you hear me? Hello. Yeah, so my name is Zahran Kalungwa. I'll be presenting this presentation improving disease surveillance data availability in Zanzibar through integrating ESR with HMIS. Maybe I should speak about Zanzibar. Zanzibar is a semi-autonomous region in Tanzania. Tanzania is a union of two countries, Tanganyika and Zanzibar, but Zanzibar is semi-autonomous meaning like they have their own Ministry of Health and other ministries, only few ministries which are like the union ministry. So Zanzibar, they have their own Ministry of Health. So I'll be speaking about Zanzibar today, not like Tanzania United Republic, but only Zanzibar. So my name is Zahran as I said, I work with CDC Tanzania, but I'm lucky to be presenting on behalf of Ministry of Health Zanzibar today. So for this project, these are the collaborators. We have Ministry of Health Zanzibar, which provided overall leadership in implementation of the integration between ESR and HMIS. We are here at CDC Tanzania. We provided technical lead and advisory in the implementation. And then we have MDH, which is an implementing partner funded by CDC. They are doing coordination and on the integration between EADSR and HMIS. But then also we have UDSM, the technical development partner, and luckily we have representative from the University of Dar es Slam, who are the technical implementing partner. So if there's any technical question, I'm well covered because just for to be able to respond to them. So the background of how this came about. So the revolution government of Zanzibar has adopted WTO EDSR third edition, which required immediate reporting of epidemic prone diseases or condition. So before the new improvement of EDSR, Zanzibar was only reporting aggregate data weekly. They had 36 diseases, which they are reporting weekly, but only aggregate. So, however, the before the portion reporting weekly aggregate data was conducted through HMIS system so they use the normal HMIS system for them to be able to report. They didn't have the epidemic unit didn't have their own system for them to be able to report. So as I said, they had 36 diseases and condition where aggregated and reported in HMIS weekly from the paper registers. But then they opted to develop electronic integrated with the surveillance system and response in 2019, following the COVID-19 outbreak. And it was funded by us CDC through partner MDH because of the need of immediate case case based reporting for epidemic prone diseases. It was aimed to address the scarcity of surveillance data previously collected as aggregate and entered manual in the HMIS system. But then also the need for the weekly aggregate reporting to HMIS was still requirement by Minister of Health Zanzibar, because there are two units within the ministry. One was HMIS, which they have the mandate to report weekly data, but in aggregate form. But then also there is epidemic unit which they needed for some diseases, 26 of them, to be able to report immediately in case based. So they were like conflict between the two because HMIS was saying, if you are not going to be able to report to us on aggregate for these six diseases, then we don't need to support. We are not going to support you developing the EISR, which is case based. So then like they had to find a way for them to be able for this HMIS to get the information but also epidemic unit to be able to monitor and also to detect diseases by case space. So that's why they had to work on the integration between the EISR and HMIS for critical to ensure that the automated process of aggregate weekly reporting from EISR to HMIS for all 36 diseases and improve the quality of data. Then that's why this project came around. So the methodology, they first initial engagement of all stakeholders because there are so many stakeholders within that you have HMIS, you have ZSU, which is epidemic unit, and then you have malaria, you have RCH, all of them. So they needed to have this initial engagement for them to be able to agree of what is going to happen. And then also they had to identify key data element. And then they did mapping between the EISR and HMIS, but then also they has to do mapping on facility list, because HMIS was using different facility list, but then also EISR was also using different facility list. They had a Zanzibar Health integration interoperability layer, which was already developed, but then they had to agree on the payload, etc. And the architect, the diagram show the architect, we are one party of EISR, but lucky for them, both system were using DHS2 platform. For EISR, they are using both tracker as well as the aggregate reporting which was there, because for 26 diseases which were immediately portable, they are reporting as case. So they are using tracker, but then within the system, they are also reporting on the weekly for seven diseases as aggregate. So they had to use both, but luckily for them, both HMIS and EISR, both of them are using DHS2 platform, which was made a bit easy for them to do the integration. And then the integration layer we're using open him. And one, luckily also, one of the use case was the integration between EISR and HMIS. So the results, they managed to do the integration through the EISR, through the ZHIL, and so far, 21 weeks worth of data has been submitted. About 326 data points are updated weekly across the 374 facilities, they are facilities in Zanzibar. They only have 374 and about 135 data points, evaluated from immediate reportable data through program indicators. And then reporting of immediate cases in EISR is automatically submitted to HMIS on weekly basis, they no longer need to do the manual aggregation and then submitting it manually. So in conclusion, we found that the system integration is critical for improvement of health services in Zanzibar. The automation ensure the accessibility of accurate data because before people do it manually, then you're not even sure of how accurate your data is. But also integration can also potentially improve work efficiency because people now spend less time on doing aggregation and reporting manually. It also reduce the cost since it eliminates manual review and entry of data. As the integration improve these benefits are expected to be realized in Zanzibar. So the team, I would like to thank the team, the MOH team in Zanzibar, CDC, MDH and also DHS2 UDSM team implementer. Thank you very much. Thank you Zaharani for showing us how data is to fit in in the country level architecture. So we'll take a couple of questions before we set up the other presenter. So yeah, thank you. Thank you for the presentation. And it actually shows how difficult data reporting in countries. So I just want to know now, you have successfully trying to integrate EIDSR, which has case based data and the aggregate data of HMS. But at the end of the day, all these cases are generated at the same health facilities. You see the cases which aggregated come into a particular administrative level and a health care center may be a primary or secondary. And then they're the same cases that, you know, get details on the, in the case investigation forms, whether they're measles or polio, etc. So at the, at the health facility level, how does reporting happens and why people cannot report aggregated data at the end of the week to the same system and case based data to the same system. So there's no need to integrate at a higher level. Thank you very much for that question. So basically, the way it has been implemented. Everything happens at facility level. So the immediate case based reporting is done at facility level. They were wherever there's a case which there's a disease or condition which need to be reported immediately. They have to within the facility. They've Android app, they report that notification to the, to the higher level they do other other thing to confirm that like yes there's this disease. So once that happens, it goes to the different levels. So what is happening now, the facilities are not required to report to the HMS the aggregate number. The system. The integration. It does that aggregation for them. And that information is being sent to the HMS from the facility itself, because that process is done automatic instead of doing the paper based and then like reporting again it's it should be the lubricate if you do that and they didn't want that to happen. So they do everything in the facility using the tool which has been developed and then the tool. So every Monday, it does the aggregation for them and it send that aggregate data to the to the system so they no longer required to report for those 26, 36 diseases or conditions. They don't need to do the, the manual process of reporting. It's just like the system does it for them. So, when you report case this data, is it a probable case or a confirmed case, because then the accounts differ. Thank you. Okay, so I'm for HMS. When they were before, before the year this are they used to have a year this are but aggregate form, meaning that like they had module within HMS, they call it. But reporting aggregate. So all the statistics diseases, they were the same, like you have it in the HMS as a desire but reported as aggregate. So what the year this are came to do the improvement was only like to take for those 26 proud diseases to make them as a case based. So means like everything which you have previously you have it only that for the 26 the the aggregation is a little bit different because you're reporting them as case based but for the seven which was remaining, you just like the same mapping like the same the HMS will be used in the year this are for them to be able to report. So they're the same 36 for for the idea star. And then the second was on the. Yes. So the way the year this are function for the immediate, the first reporter suspect. They report a suspect, once it is suspected, like we have this disease. Then that goes to the next level which is a district, the district people come to do verification, to do the investigation, once they verify and confirm that this is this particular disease, then it is confirmed so they have both suspect and also confirm depending on the investigation which has been done. So it's first it goes as suspect, but then when the first investigation is done lab test is being done and then confirm, you have both suspect and confirm so for those which if they confirm it's confirmed for those which they haven't done the investigation is she should be it's continue to be suspect. Thank you and also we'll have a brief session after this, I will be able to have questions. Thank you. Thank you very much. Good afternoon to everyone. Allow me to share with you our experience in Rwanda, linking up to systems, the civil registration and vital strategic system, statistics system, and our routine organization digital registry to improve the health services, especially trying to to have better coverage and also a bit of service in the health facilities. So, this is a project coming just to, to, for the country to, to reach the global health coverage by ensuring that all new born in Rwanda. We've got the vaccination services. And we don't have those economies, those are unsupposed. So as a quick background for Rwanda Rwanda is one of the early adopting countries for the DHS as a routine health information system, whereby as we talk about the system covers private and public or programs integrated into one HMS system. That happened since 2012 but towards 2019. So when this electronic registry package was developed, so the API program ought to have this digital solution to support the digitalization of children information, especially throughout the routine vaccination schedule. So that includes normal routine vaccination, but also the HPV vaccine administration. So as we talk then so technology keep on improving in 2020. The government of Rwanda ought to have a digital CRS system. This is a homegrown software developed at the country level to capture civil status and health events. That has been rolled out nationwide to administrative and health facilities as you know, our vaccination services are being, are being, are being done in the health centers in Rwanda. And I think that's how it's done across another countries. And also, most of Rwanda, the 95% births are happening in the health facilities. So this, to some extent, the birth is being done at the health facilities, the big percentage, and also the vaccination services are being done inside the health services. So, integrating these two systems were seen as a strategy to make sure that we improve the coverage. Talking about the electronic universal system Rwanda and also the vaccination services in general, Rwanda is not the big country, but we expect approximately 370,000 children in the program per annum. And the electronic registry was rolled out countrywide in all vaccination posts, approximately 515 health centers using the registry. So as you know HR is supposed to be comes to the health facility as we fill out the normal paper registry so a typical digital registry has been put in place to support the process. As I said, 94% deliveries happen in the health facilities are based on DHS 1920 and also based on the same survey. 96% of children as of that month receive all basic vaccination from the health post or from the health services. Health facilities, sorry. So, since all health facilities are supposed to fill out paper registries and also electronic registries. So as you understand it's more time consuming and duplication of efforts. In 2020, borrowing a read from what happened during COVID where but the Minister of Health has experienced the immediate digital registration of COVID cases using the choice. So I thought why not trying the same for for routine immunization and go to Paris to ensure that clinical staff at the health facility can focus more on on delivering services and use DHS as a digital registry for the vaccination services. And records events for every vaccination visits done by parents. Talking about the CRS as I said the CRS ideas came in by in 2020. Of course, as I said, this is was to increase the birth registration and the certification to 95% and to 90 for this certification by 2025 that it is this is the global targets for the country. So the CRS digital solution came in to help the country to keep track of progress but also to support a growth services. So among all the steps for the CRS. This is just like to give you an overview of how the CRS works. The system development took these three steps and covered almost 416 civil registrar at the health sector means like means the children who are born in the community can still be registered at the sector's office but those who are born and the health facility. Around the 612 civil registered facilities also been trained to report children born in the facilities as you understand. The civil registration is happening at his facility and also the vaccination services are happening in the, in the health facilities. So, as we talk in your upper right you can see in your in your right you can see the phases and the events. How it is done in 2020 that's when the birth and this registration module was brought in, which has triggered an idea of starting thinking of how can we link these two systems to reduce the workload, but also because the CRS system provided the national identification number for under 16. So that was also another opportunity to leverage on so that we can easily identify newborn and also reuse the information from the system. Since birth and vaccination services are happening at the health facility. So how did we thought of for implementing this as you understand the our digital, our digital registration registry was built on the HIS of course it has a profile page where you record names and information, but we didn't have of course the teachers can offer possibility of having the unique ID, but that unique ID wasn't cost cutting or other government systems. So, using the CRS system to use because like when a child is born his facility parents are required to fill children child information, names, names, details, parents, information, location, and so forth, but but also still have that national identification number for the child. To do that interoperability, we thought a situation whereby data are going to be fed from the CRS system and be brought to the HIS instance, but we thought it would be a good idea to have if I can say a similar a similar registry in the HIS just like as a staging as a staging program to be able to have like a quick check to see if all information the CRS are transferred from one point to another. So part of the information that we transferred from the CRS of course person profile data and also the national identification number which is unique for every newborn in the in Rwanda. Of course this information during the registration parts, the NASA vaccine is seeing a child simply has to search throughout the HIS. I think, you know, you are familiar with the HIS such from that registry and retrieve the information. So by retrieving a child, the child comes with all demographic information and from there it continues with registering our visits and the rest of information as the child is brought back for vaccination services. So after doing that implementation, so the aim of this the purpose of this was the purpose of this assessment was to to ensure that we've done our scripting, we are making sure that data are moving from one side to another. But we want to be sure that how quality data we are getting from the CRS reaches to our the HIS instance. So we have selected number of some of the indicators just to help us to monitor by doing the data counting of and comparing like information pushed from one side to another, but also comparing what has been reported in the monthly report. Because by that time we did how the country hasn't stopped yet the HMIS aggregate reporting because in the HMIS aggregate we have already section for vaccination where the report on a monthly basis those are children who received all those are demonstrated by reports on vaccination services. So among the the variable we looked at is the total the total deliveries that are reported in HMIS and live birth from the HF facilities in the HMIS. And also we looked at what the number of birth registered in CRS and those one transferred to our DHS staging registry before we start recording or even unrolling HR into the immunization registry. This is we added on like the BCG registered in the immunization registry it means like after retrieving information or pushing information from the CRS and unrolling HR the vaccination program. We also went ahead and through a program indicator generate this value on monthly but we were a bit interested to see on monthly basis how many children have been vaccinated for example BCG because this BCG indicator was to some of our analysis being used as a proxy denominator when it comes to coverage. So we just in the case some who prefer using BCG as a denominator and in our immunization registry how is it what the number we have so that we can easily address also denominator of circumstances that program have been facing. So as I said that part of the methodology we of course we developed a script that of course on daily basis that pushes data from the CRS and also create a staging birth registry in our DHS systems and also retrieve or review monthly aggregate data vaccination routine aggregate data from the routine immunization services and also monitor on monthly basis children recorded in the routine in our digital immunization registry and then perform some quality checks and of course document on to see how our interoperability is performing and doing the job. Some of the findings that of course we done number of checks but here for the sake of this presentation, we did this exercise last year in March trying to see of course how far we are, we are with the government with the national target of having 95% CRS bus registered by 2025 by 2025. So as you can read from this bus registration you can see the total deliveries we have in the HMIs they are 29,000 this is the mastery data. And we had by that time 28,900 children registered. So you see there were some discrepancy because by that time that's when the CRS was introduced but also that's also the time that also started checking. See, is CRS good enough to to replace the aggregate. And when you look at the live bath from this facility, this is the 29,554 but when you look at in our registry, the interoperability is what has been pushed by that time in CRS. No, from CRS to our data systems, you find they're still a very big gap because like it was like it by trying to see how our interoperability is quality is doing quality works to have similar data on both side. So there were some technical transfer issues, especially from CRS for the HMIs. Of course, but you can see you can see from the CRS, the registration of newborn, the quality of data is quite high. It's quite high but despite we have less data in our data systems. So, last year towards the end of the year, so that we had the ministerial order enforcing going paperless for all his facilities in Rwanda that to some extent has and whereby all his facilities were required to to first of all register each other before demonstrating vaccine. Of course, if you don't have your national ID number, they still register you and vaccinate but one of they have been a change in the process of delivering or delivering vaccination services. And that has come to some extent having quality data you can as you can read from the the the the orange the orange bars. Some big improvement has been achieved, even though there are still some some gap. So, for example, looking at the number of people just children just in the service has increased, but and also when you look at the number of here. I'm not sure if this mouse. Oh, where is the pointing. Okay. So, when you look at the, the, the Syahirah basically just at the bus and also the DHS bus a push from Syahirah basically just you find there is a quite big number of improvement but still there are some challenges, some missing information that to some extent affect the services because to be able to vaccinate the child it has to that the child has to be registered has to be in the system, or so that you can utilize the need provided for great from the Syahirah. Part of the challenge of course. Apart from because when you're having that introduction when there's a need to official maybe in between you synchronization. Something can happen that I don't synchronize don't cause as you wish, but also some organization unit to mismatch among the two systems, some new facilities may be introduced. Or even you find organization unit don't doesn't exist at all. So, those are some of the challenges that we say so when that they don't exist, and the system pushes so data is completely is automatically lost. So, that's the conclusion. We have seen that this interview to some extent is a facilitating registration and the follow up of children, despite the challenges mentioned, and as we talk 99% of children registered service automatically available for our electronic ministry for routine additional services and follow up visits. Of course, that also to some extent have to identify a zero dose children, and also locate those who have missed some doses. And also, it's, I can say it has solved the denominator issue but to some extent, it's helping us to see the gap, especially if you decide to use the BCG as the data we decided to use, because we have almost 90% of children of newborn and our system. So, depending on projected population based on the census. It's a, it's a, it's a choice. Of course, some improvements are still recommended. Thank you very much for your time. And patients. Thank you adults. And we have all these presentations on the website. So you can easily be able to access them and actually download and go through them, but we have some four minutes remaining for any quick questions that we may be there from the start we looked at it from how we were looking at the details to and how it relates to the country architectures we started from. Yeah, looking from the first presenter how with the machine learning, how does it fit in, then we went to regional program, like a course then we went to a global program like pepper. And then we're able to look at now the country specific implementations and the integration like what he has just ended with the, the CRBS and also what he presented about the, about the ideas are. So if you have any questions, let's have the questions and then let's go into response I'll start with someone and then go. Thank you adults. I just want to ask I saw in one of the challenges you indicated that the script retrieves data every hour. I've just failed to see how it's a challenge. But also, maybe even to try to digest that. Is it a performance issue. Is it that there's a real time need of data from CRBS to the destination system, or have you guys looked at another alternative ways to instead push data into the destination system that you want to go. So, so when I add a new born baby in the CR in the CRP system that the system pushes data, instead of being retrieved over. Thank you for the summer so we'll take a couple of them and then they can be responded together. Thanks Dennis had maybe his touches a little bit on what Steven was asking as well I think that off one of the really interesting things about the scenario that you described is maybe less about some of the technical details but that is hard enough to do architecture within Ministry of Health. But now you're talking about a different ministry. Let's see RBS presumably that's that's home affairs or something similar. They curious to know, what is the negotiation process been like how do you, how do you work out things like the data sharing agreements, even mundane details like data formats and things like that, when you have to negotiate across ministries, rather than just within Ministry of Health. Thank you Bob so I don't know whether you have any other questions so let him respond. Okay thank you very much maybe let me start by Steven. Is it concern or clarification. So when it comes to the challenges. The service system as I said it's a it's a it's a locally developed homegrown solution, supporting the service processes. So the way the interoperability is done so they have API that we expect to push data to our DHS registry. But actually we are going to fetch from the service automatically on a given plan or plan job that are pushed, or after every two hours, data is being pushed to from the service to our DHS staging program. And then from the DHS staging program that we are every time when a NAS are demonstrating for the first time, for the first time, he has to, he or she has to to search from that staging base registry to register a child into the international registry. That's actually how it's been done so but plan to maybe still on that. So this is the government system. They are the ones who are giving you options. So they are actually because like it's the database that govern the whole population registry. So they are the ones who are giving you options on how data can be forced. As one of the newly agreed way of having data is doing that live streaming of data, but it's something that has been tested but not yet implemented. Talking about what Bob said, of course, government to government institution. Sometimes it's easy or difficult depending on who manages things. So you can't say it's always a spirit one process but in our case, because this was actually a request for the two programs they were like, they were vaccination program, but also a unit in charge of this certification. So they all wanted data from the service because that's where you can have this information. This certification and also but so we have received the supporting data from the Minister of Health requesting that there are a bit of information. Of course, the selection of variables to be to be pushed has to be where I think we did have like almost five, five, five, five letters requesting changes of API because every time there is a change in your API list. So you need to have the letter expressing or like just find the needs of change. So that's how maybe Andrew is there. Thank you. Actually, on the data sharing what happened, the government appointed one institution. Want me to stand. Oh my goodness. Okay, good. Thank you. So the government appointed one entity called the National Identity Agency to coordinate the service implementation. So there was a win-win. Because you know, for us in the Minister of Health, we are in charge of health facilities. And for the live birth to be registered, they needed our police to support. Then on the other side, we needed them to send us the live birth with their, they call it national identification number for us to be able to vaccinate. That is what I always Bob always emphasize on when you know that you are gaining from integration of sectors everyone will be involved so there was that support me, I support you then you implement. That's why the data sharing is not a problem we are looking at the benefits of the citizens rather than the data sharing. Thank you. Since there is one burning question so let me just allow him is from the Ghana. Thank you just to add an additional perspective we are doing a similar thing. In our case we have so it's a try the Ghana Health Service we have the national identification authority, and then the birth and death register. Now we've experienced two sides of the coin where the integration with the national identification authority was pretty smooth. And that of integration between Ghana Health Service and the birth and death registry, a lot of bottlenecks. One of the bottlenecks being contributed by the technical teams that were supporting these other government agencies. And so those are some of the things you are likely to face so technically or at the managerial level, there might be these agreements to share data. But if the technical implementations people are not in sync with the idea and what do you all want to do then you have other bottlenecks introduced and I just get the whole process slowed down so it can be very complicated sometimes. Thank you. Let's appreciate our presenters. Thank you all. Thank you all for joining. Yeah, and feel free to reach out to them through the different sessions. So thank you.