 Well a very warm welcome to all of you online here and as you will have noticed as you were signing in we are recording and this is a on so that you can play it back and look at it after you leave it or else just send on the link to other people who may not have been able to join us. My name is Elaine Byrne and I'm with the HIST programme in Oslo and part of the implementation and research group there and we have a great 90 minutes plant where we'll actually go through the particular apps, we'll look at a short demo and we'll also go through some country experiences and I thank you all for putting in when you were registering you know your interest areas. It gives us a good idea about your background. I'm not sure we'll be able to address all your queries because it's a wide range of people from those who are looking at the kind of setting it up and the technical aspects of the applications, as well as those who want to be able to use it, but hopefully will spark your interest in your curiosity and learning more about these apps and we'll post a lot of material and resources that you can follow up on afterwards. I'll just do a brief introduction to each of the presenters. We'll leave questions and discussion till the end but feel free to post in the chat any questions you have as the presentations are ongoing. The reason we leave it to the end you know it's kind of easier for timekeeping but also we're kind of hoping that as we go through the presentation some of those queries would actually be addressed. At the end we'll give the presenters a chance to look at questions that were on the chat if they haven't replied to them or summarize the responses but then we'll also open it up to the floor for further discussion. And as I said there's plenty of resources that you can look at later on if you want more information or you can post on our community of practice to get more information. I would like to introduce the first presenter Maria Muniz who is a health systems strengthening specialist based in the UNICEF ASARO office. Maria will introduce the history of the apps and the use of these data use apps. Thank you Maria and as you've done it's quite nice if you are presenting if you can with the bandwidth open up your presentation so we can see everyone's lovely faces there. Maria over to you. Thank you so much. Just confirming that you can see my screen and that you can hear me okay. Yes that's perfect. Thank you very much. Great. Thank you so I'm really pleased to be here today and wanted to thank the University of Oslo for organizing this information sharing session to all the speakers today and to the participants for joining this webinar. I'm really looking forward to hearing from the speakers for what I know will be an informative session and I encourage participants to feel free to interact and engage throughout using the chat. Feel free to introduce yourself and ask questions and we will also allow some time towards the end for you to share your observations and reflections during the Q&A portion. So our aim today is to gather as a community of practice and as Elena said to share a suite of information use applications that support the synthesis visualization and analysis of routine data in DHIS2 which are ready for your use. We'll have a demonstration of some of the features of the applications by the app development team and implementation support teams in his Tanzania and Uganda. And our speakers will then take you on a learning journey to Yumbe district in northern Uganda and in the context of the TV program in Tanzania to hear experiences from the field and using these applications. We also want to hear from you as a community how we can improve the current applications and what you'd like to see in the future. So to start us off I wanted to give a bit of history on these applications. So UNICEF and his have been working closely for the last couple of years to develop and support the in-country implementation of the three data use applications scorecard bottleneck analysis and the action tracker which we will be learning more about today. And a driving force behind the development of these applications has been to strengthen from a systems perspective and in an integrated non vertical program manner. The synthesis visualization and use of health facility and community level routine data for purposes of supporting performance planning and evidence based performance reviews and decision making. Conversations about data that drive local actions to improve equity and coverage. So a scorecard application is available and that is configurable in national DHIS to instances and it supports the visualization and use of data to assess progress on key indicators and to benchmarks of national performance. There's also a bottleneck analysis tool that supports the use of health systems data so data on supply and demand side factors that affect coverage to identify health systems constraints, diagnose root causes and develop local solutions. The visualizations helps stakeholders engage in conversations around performance, ask questions, you know why coverage is low and facilitate discussions about solutions to address bottlenecks and inform health operational plans. The app also supports documentation of the outcomes of the conversations as you will see. And then once the solutions are planned, their implementation and effectiveness in addressing system constraints and enhancing coverage can be monitored through an action tracker. And the idea is we've often get asked about these apps and how they complement each other. And the idea is that they work together to support this different aspects of the information cycle. So the scorecard supporting presentation and communication around data, the bottleneck analysis around root cause analysis and interpretation. And once decisions and actions are taken, the action tracker can support monitoring and implementation of those actions. And to start the whole cycle over again and assess if you know how performance is or is not improving after those actions have been made and again to diagnose what are some of the remaining challenges in the system. So there are many exemplary aspects about the app development process for these three applications, including participatory design, rapid prototyping and user feedback from many of the people that you'll hear from today and perhaps others in the audience. But we know that their success is really going to be contingent on if and how they are used and the difference they make or not in supporting development of evidence based action plans and improving the lives and wellbeing of communities and progressing towards the universal health coverage. So in addition to demonstrating the app today, we did want to give a chance to hear and share from some early implementation lessons, and it's something that will continue to document and learn from the future. And so before I hand back over to Elaine, I wanted to acknowledge that these applications would not be possible without the hard work of his Buganda and Tanzania team developers. The users in the field who contributed requirements and gave feedback, the University of Oslo UNICEF staff in the East and Southern Africa region and to Gaby who has supported this work through its data strategic focus area. So thank you very much and over to Elaine. Thank you very much Maria for putting that into such a comprehensive and very short context. As Maria said, please post your questions within the chat as we go along. And I'm going to hand you over to a presentation on the apps and I must do what I've asked everyone to do is to make sure their video is on when they're talking. So I just want to hand you over to two members of the teams that Maria mentioned. So we've got Eric Munya Mabazi, who's the lead monitoring, evaluation and learning from his Buganda. And Siddiq Mashaume, who's the business analyst with his Tanzania, and they will do a presentation on the apps in terms of the uses and the implementation. And then we'll also hopefully have a short demo on that. So over to you, Eric and Siddiq. Thank you, Ellen. It's always nice to meet wider community that is interested in data use. Just shortly, we will split our presentation into two. We'll make a brief introduction on the applications, and then we'll later on have a bit of a short demo as you have elaborated. I'm sharing my screen. Would you kindly confirm if you can see my screen please? Yes, that's great. And we can hear you very well. Thank you very, very much. First of all, of course, just like Maria said, I would love to thank the team, the Investors of Oslo team and the wider community for organizing this webinar. It's good to share information and get feedback. These applications are some of the best DHS to best data use applications and it's always nice to see that more people are interested in using them. I will make a brief introduction on the applications and then Siddiq will probably do the short demo. The screenshot you see is that screenshot was taken from an East African community meeting where members were meeting to look at the scorecard and particularly on heads of state commitment on maternal and child health interventions. And the scorecard appeared as one of those tools that would facilitate providing real-time information to be able to make decisions so that I thought I would very speak a bit on that background image you're looking at. So I will just highlight a few things on up implementation and use and then we can probably like Sid receive some feedback and also questions in the chat room. Yes, you, I know you can see me, hopefully, but I work with this Uganda and like you said, I lead monitoring and evaluation and support project implementation, mostly on performance as well as quality assurance. As a way of overview of what we are going to be discussing in this presentation, one will look at a few objectives for the webinar and why really we thought it's important to share some of the knowledge and also some of the processes that we have gone through to put together the applications. We'll also look at some of the application use cases both in country and in other places and also some of the wide knowledge base that we've picked, but also share in terms of resources and community support for these applications. So these are broadly thought about objectives for the webinar and the main, of course, the other one is most important is to share this knowledge by way of orienting partners on the applications and to basically discuss and be able to get feedback, explore how the DHS to data use applications, add value to our national and subnational planning. Maria already in a way highlighted how these applications feed one another in terms of planning, in terms of analysis interpretation and so on. So we also think that from here, from this webinar, we should be able to develop some, you know, competence in using the applications to do some basic analysis and also generally pick interesting seeing how they can be able to support data analysis at national and subnational level in our respective places of work as well as countries. But also to look at the implementation requirements, what goes into what is required really to have these applications running, how can we support them and so on. So, broadly, that's how we will be having this discussion and hopefully that we can get the most of these objectives. So why the webinar again briefly to guide users, sorry, why the applications briefly to guide the users and managers on some of the best practices for data analysis. Again, the applications come together as a way as a way of getting user needs from different environments of work from different experiences, and then putting them together but also to see how to improve some of the features that we have within our different applications that we use for analyzing data, especially healthy data. Then also to enable comparisons in terms of performance at national and subnational level, where we look at different indicators on different programs on different interventions, health interventions, but also to spot areas of strength and weakness to guide prioritization and response. One of the key and very useful applications that we'll be looking at and discussing is the bottleneck analysis, the bottleneck as the way it actually sounds, we are looking at gaps in implementing different interventions. So the app helps us to support those areas where we have challenges and be able to guide in terms of prioritizing areas of focus in terms of planning and also response both at national, subnational and also at facility level as we will be discussing. Lastly, the apps are really mainly to support tracking of actions. Once we have our priorities set in place and we've put them into a plan, how do we monitor implementation of this plan over time through tracking of these actions. And this is where the action tracker is really, really very, very beneficial. So, quickly some few highlights, largely picking out on some of the few processes that the team has gone through and put in place to have the applications work the way they are working. One is already Malia highlighted on the fact that this was user specific and user driven where different perspectives were picked from different countries. Kessin Point Malawi, Tanzania, Kenya, Uganda, Randa and Ethiopia and to put together really the perspectives on how the squad should come together, but also using some of the earlier discussions on say for example Alma, you probably some of you had about the Alma squad. Yeah, so this was really user driven to put together a traffic light based kind of analysis that says I am if I'm in red, largely I'm not doing very well if I'm in my performance in yellow, I'm progressing but not maybe so well. If I'm in green, then maybe I can have a smile and then move on. So it was really largely user driven but also very simple in a way that we need to use our data. Two, to mention that the application is again mainly based within the DHIS2 and also largely dependent on metadata that is inherent within the DHIS2 system. So if in this country you have DHIS2 already implemented at the national level, it is much easier to have the application run within that environment that's already set up especially with routine data from facilities or even from district. Number three, there are cases for external experience where we have found that data resides in different environments and in some cases you find human resource data residing in a different system, logistics data residing in a different system. So it's important to again bring together some of these different data sources so that you're able to have a comprehensive analysis, especially if you have an intervention that has indicators that are cross cutting. Number four, important to engage in a consensus building discussion that looks at indicators from various programs so that you are able to identify and define them together because again analysis is for everyone, both at national and sub-national level. Five, we also noticed that it's important to set up conventions for example for indicators and also for data use so that it's able to have standards that we are able to have standards that you rely on while looking at this data across, especially where we have data coming from different systems. Six, we also found that grouping of metadata makes analysis very easy but also makes access to configuration and using the applications very, very easy. Then seven, which is really not the last but least is engage different program teams in terms of review and validating these visualizations to ensure there's quality. So for example, if you have indicators that you have picked from the DHS to system and you have used them to customize a scorecard, we found that it's important to verify that the data that you're looking at is what it says it is and that it can be verified and approved by programs in different aspects. So that's the overall of what the scorecard entails and how it functions in terms of setting up but also analysis. The second one, again, like I mentioned, I'm giving highlights of these applications. We may not have enough time to look into the details but the second other implementation approach for the bottleneck analysis is one. We mentioned the user specific needs to diagnose under performance in selected health interventions. And this, again, is based on the Tanahashi framework, which we probably will discuss a little bit more later, that looks at performance. That looks at performance in different interventions and then identifies bottlenecks in those interventions based on a framework that considers supply and demand of different health services at different levels. So this, again, gives us the idea of where to focus and how to focus our interventions. Number two, just like the scorecard, it also utilizes existing metadata within the national DHIS instances, then also uses the knowledge of stakeholders and experts to estimate which this number three is actually very, very important. For example, where you have cases of data that is not available, say denominators and indicators, important here is to note that stakeholders and experts need to help in terms of putting it together. Then I've already mentioned a bit on the data. Number four, attention to numerator and denominator and some basic calculations is important because it's easier to miscalculate and then get a wrong outcome on the chart. Actually, BNA, again, is very contingent on good enough data. The idea is not to have traffic, but data that you can work with that is representative of the interventions that you are implementing at both sub-national and national level. Then second last important to identify data quality issues that need to be corrected. This is really, really important so that you can take care of them through the process of putting together this application. There will definitely be potential data quality issues, but it's important to plan ahead so that you are able to mitigate these quality issues. It's rare to find data that has no data quality issues. I think most important is how you plan to mitigate these data quality problems. Lastly, again, it's the action tracker I can mention. This is around really planning and monitoring actions that go into our plans based on a concept called causality analysis. Causality analysis picks again from the bottleneck chart where if we have found a bottleneck in our intervention, we are able to say what is causing the bottleneck and that's where the name causality comes from. We are able to trace and say what caused underperformance on human resource and so on. You are able to cascade the problem and look at what exactly is causing this problem. The action tracker deals mostly with this and once we arrive at what the problem is, we are able to document that problem, plan for the problem and then plan to mitigate the problem and then monitor our plans. Briefly, that's what really the action tracker is about. Here, number one, identify the real cause of the problem, support the tracking of actions taken to address the causes of these bottlenecks in our interventions, analyze previous periods planning and tracking for the future. This number three basically says that if I am going to plan, I need to analyze data. If I am analyzing data, I don't analyze data that I collected in the future, but I analyze data that I collected in the past. Let's say last year and then plan against that data. So our plans are in the future. Our analysis is for the past data. Then number four, plan multiple actions for each suggested solution is to say that we don't have one solution that fits all, we can plan multiple solutions and then implement them accordingly. Then measure actions taken by looking at performance of key indicators. So in terms of looking at some use cases, overall, we basically consider areas like program performance reviews. Again, the programs, you can use this on any other health program. We can use this to do evidence based planning where we have, let's say quarterly district reviews or quarterly district healthy management meetings. We can use this to do data quality reviews over time. We can also have this to do district performance review meetings where we are able to document what comes with the outcomes of those meetings and then plan against those outcomes for monitoring. Then health facility monthly data review meetings. Again, to look at different aspects of the data that is generated at the facility. These are some of the few use cases, but again, the apps are generic and can be explored to support any other health or non-health related interventions as we will be discussing. Lastly, some few pointers on the resources that can be used here. One is there is documentation that is freely accessible online for the user guide, technical guides. This can be found online. The apps have been reviewed and quality assured and can be downloaded from the DHS to App Store. Some additional information on app releases and also communication generally on support can also be found on the community, on the community of practice. Again, for DHS2 on that link, the demo and training instance. If you really want to experience and use some of these applications can also be picked off the demo. The HIST.apps.dhs2.org slash training demo instance. And then if you find maybe it doesn't fit your particular environment and you have a feature that you want to suggest that can be added. We also have JIRA for DHS2 that we use to again log and then document a lot of the features that go into these applications. I wish to, yes, I wish to thank you very much for the audience and hopefully that these applications can contribute to the work that you're already doing and hope that we can hear a little bit more feedback from you, the team. Thank you very much and over to you, Elaine. Thank you very much, Eric, there for your presentation and overview of the apps. And we're now going to go into a quick demo. And I know we're running a little bit behind schedule, but it's fine in the sense that we're going to have them to overview so we have plenty of time at the end for questions. So, as I mentioned, his Tanzania, if you'd like to start the demonstration, and I see people have started to post a questions in the chat, which is great. So if you have any questions as presentations go on and continue in the chat. So over to you with your demonstration. Thank you very much. So I'll just take you through the apps. The four apps that we have, Spokad, BNA, and Action Tracker, where in particular for the Action Tracker, we have two versions, Linked Action Tracker, and Standard Action Tracker. So since I'll just be focusing on the demo, I'll just turn off my camera for a while and take you through the apps for a few minutes and then we'll have a discussion afterwards. So as presented by Maria and Eric, so basically in terms of the apps that we have, the focus is on three main components. So we have apps that are focusing on presentation and communication, particularly the Spokad. So how do we want to present data and how do we want to communicate the data that we are collecting to identify issues and then come up to make actions that we want to make. And then we have the analysis and interpretation apps. And lastly, for decision and support, we also have the Action Tracker, which can be used to track document actions and track follow-ups that you want to make. So we'll start with the Spokad, which is basically the Spokad in the BNA app, the key parts, the ones that we use to identify issues, do the analysis and interpretation, and then we use the Action Tracker to document the actions that we have observed from the Spokad or the BNA. So I believe most of you are familiar with GCS2. If you're not, basically, if you want to access the app, we have the app menu option here for write, so you can just search and find the app. So we'll start with the Spokad. So we have the latest Spokad is called Interactive Spokad. So some of you might be familiar with the current Spokad region that we have, which is the normal Spokad, but we have updated the Spokad and explain why in the next few minutes. So the Spokad that we are going to present is an updated Spokad, which is already available on GCS2 App Store, and we'll take you through some of the improvements that have been done on this new Spokad region. So basically that's step number one, you go to the Apps Trail and then select the Spokad app for you to access it, and I'll take you through the different features that are there and how you can navigate through the Spokad. Yeah, so the landing page of the Spokad, I'm accessing it as an admin or user with privilege to create some of the Spokad, and of course the Spokad behaves like a traditional dashboard that you have on GCS2. Basically, for example, you'll see I have a dashboard already created here called LeagueTable RMSHA, which I can view, edit or delete, but I have another Spokad here that I can just view, but another Spokad here that I can view, edit and delete. So that's the first aspect that we have, which is the normal in terms of Spokad development in the past and this new version as well. Once they are Spokad created, you will get a view of those Spokad that have been created and it will give you options based on how you've been given privilege to access that Spokad. You can either view, edit or delete if you are given that privilege. But as part of the app, so you have the option here to add a new Spokad. So I'll just quickly click that to show you what we have done in terms of adding a new Spokad. As Maria and Eric said earlier, in terms of Spokad, we want to give you an idea to quickly present and communicate the data that you have. So what we have done with the new Spokad is make it a bit easier now for user to configure the Spokad and this is part of the user feedback. It's now step by step where you have the general settings where you can put the name of the Spokad. For example, you say immunization and then if you have a subtitle, you can put a subtitle of the Spokad, you can put a description and then you define a period. So basically, as you are defining the Spokad, you can define under what period do you want your data to be analyzed or to be visualized. Is it on a quarterly basis or on a monthly basis and these are from the traditional DHS to periods. So this is a bit similar to what you can do with the standard Spokad that most of you are familiar now. And then you can put a header if you want for the Spokad that you are creating. And the crucial part now is the translation. So as I said before, the Spokad is used to communicate and interpret data, present data in a very easy manner. So you have some targets that you can define and set for which ones are breached. What colors do you want for Spokad for the indicators that have maybe met the target or not met the target or they are not on track. So by default, we have green for those that have reached the target. We have yellow for those that are on progress and then red for those that are not on track. And then if you want additional levels, it gives you the ability to define. So although these are standard levels that we've used, but you can as well use your own levels, you can as well use your own colors. If you click on the color option here, it gives you the chance to create your own color code if you have different ways you want to use to translate. So this is part one where you just provide the basic information about the Spokad. So what we have qualified as the general information and then it allows you to go to the next step. So the next step is the data configuration. As we say the Spokad now allows you to put together a number of indicators that you want to be visualized. So the data configuration, another interesting feature is the ability to provide into add indicators in terms of groups. So you can provide different groups and define how you want to configure or select the different indicators that you want. So for the sake of the demo, I'll just put some of the indicators. So if I'm adding item here, I can save any access to A and C. I add that as an indicator if I want to add another item. So I have my group here. So if I call it, let's say A and C as a group, then I can add multiple indicators related to that. A particular group. So if I have another A and C related indicator, I can just add it there. And then I have another option to add another group. So if I want my Spokad to track multiple groups, I can define a second group, let's say I call it PMDCT, then I can add items related to that. So all my indicators that I want to track that are related to PMDCT can be added as part of that group. So I can select add item and then add another PMDCT indicator. So basically this gives you the chance to define such groups for one Spokad, you can have multiple groups and you can add indicators for each particular group. But another interesting aspect that we have added in the new Spokad is what we define as highlighted indicators. So sometimes you have maybe 10 indicators that you're tracking, but you want your regions or your districts or your facilities to pay attention to a particular indicator when they're viewing the Spokad. Now you have the chance to go in and define highlighted indicators. So this is not a must, but if you have indicators that you think among the 10 that you're visualizing on the Spokad, you want people to pay special attention to. You can easily select and add it as an indicator of interest, maybe skill delivery, for example, just picking it randomly. So now as you're viewing your Spokad, this will become an indicator that is visualized in a different manner because you want people to pay attention to that. So that has become step number three. So you have done your general settings, you have done your data configuration, you have done your highlighted indicators. Then there's an access aspect. So for this particular Spokad, who do you want to have access to. So you can link it to organization unit, specific organization unit. And then if you want some sharing aspect, you can either share it in terms of public, how should people view it, or you can share it with specific people. So if you have certain users that you want them to access, you can easily search and select a specific user that you want to be able to access the Spokad. So that's another aspect that we have added in terms of how you want to add access. So if some of you have used the traditional Spokad, all this was available on a single screen. But what we have done now, we've made it a bit easier in terms of step by step definitions of what you can do. And then the last part is the visualization aspect. When the Spokad loads by default, you have the options to define how should it be viewed, what legends, should legends be seen, title, item number, et cetera, or the average columns. These are things that users can manage later, but you decide as a default. So basically, that's what we have done in terms of the step by step configuration of how you can configure the Spokad. But for a Spokad is already configured. So for example, if I pick this, the RM Sage Spokad, it loads with multiple options. So we have the top layer here, which gives you information about the organization units that the Spokad is related to. So for example, this one is for the whole training land as a country level. But you also have the period here for which data visualization is done. And then you have a section here for that particular location. It shows all the districts and the color coded. This is what Eric also highlighted earlier in terms of how you easily view data in the color coded manner or the traffic lights manner, where you can easily see the red is not performing well or yellow is not performing well in the progress or green that have already achieved the target. So in terms of how you can visualize the Spokad. So you have, for example, the groups, as I was doing the configuration, I took you through the group. So you have, for example, here ANC group, best group, each with two indicators. So as you're configuring the groups, this is how they come to be visualized. So you can have a single Spokad with these multiple groups and each group you can put as many indicators as you want. Another aspect of interest here is the changes. So you have your increase from compared to last period, has it improved or not? So you have some pointers here. If data is visualized over time, you can easily see it could be on progress, but it's going down. So maybe it needs you to have an action to address that concern. So basically, that's it on the Spokad. So the new Spokad has been improved. You have a chance to do the step by step configuration. And you have visualization, which is a bit similar to the traditional Spokad, but we have added some few aspect in terms of how the groups are visualized and how you can access it in different manners. And then in terms of download, for those who want to use the Spokad in different versions, maybe you want to print it. So we have also added, we have multiple options here. You can download this Excel, PDF, CSV, or in terms of Alma, you have multiple options. You can download in terms of JSON or just the metadata that was used to configure the Spokad. So this is the first app as part of the apps that we have done for you to promote data use for presentation and communication. As you can see, the presentation or communication aspect is covered in the red and green and yellow colors. We can easily communicate issues or the performance of the indicators. So in addition to the Spokad, as part of the development that we have done with his few Ganda, UIO, and UNICEF, we have also developed the BNA. So the BNA is fairly new, and Eric explained it a bit. So accessing the BNA, you go back to the bottleneck and up to the apps tray option, like how you access any apps in DHS2, then here you have the bottleneck analysis app. Select the bottleneck analysis app. It loads, by default, some intervention that have been configured. So as it was mentioned by presenters earlier Eric and Maria, so the bottleneck analysis is based on the Tanahash model. So basically, it's another useful tool that allows you to visualize data of multiple aspects. You have indicators related to communities, human resource, service accessibility, then initial utilization, continual utilization, and effective coverage. So basically, in one chart or one visualization, you have six different components, and you can have data from multiple aspects. So the BNA makes really use of the analysis ability and interpretation, because you can analyze data from multiple components, but you can also easily interpret data in seeing, for a particular intervention, what could be the problem? Is it commodity related, human resource, geographic accessibility, and how is it related to utilization? So initialization of a particular drug, maybe, what about the continuous utilization and effective utilization. So in terms of the app now, as it loads, you get access to the multiple interventions here on top. So if there are a number of bottleneck analysis interventions already configured, you'll have them accessed at the top here. If there are more, you have the option to show more. But if you have access to add interventions, so there's a button here to add a new intervention. So for example, if I say I want to add a new intervention. So what we have done, we have configured also what we call template interventions. So for new users that maybe are not familiar with the HS2 or bottleneck analysis app, but I want to make use of the interventions that are already configured. So you can select from some of the templates, so ANC services, for example, ARVs, etc. But you can also add your own intervention if you want. So you have the option to either choose from a template and customize it, which makes it easy for you to understand if you're still learning on how to use the app. But if you're familiar with it, you can just select new and then you are able to create the new intervention. So if you say you want to create a new intervention, it gives you the chance to name the intervention. So I'll say let me just reduce the size there. So as you select it, is it visible, Eric? Can you still see it in the team? It's a bit blue on my side here. Okay, so let me go back to the DPD3 immunization, for example. Is it still visible? Yes, it's still visible. And just to let you know, you've just have about eight minutes left. So for example, for the intervention on DPD3, so you have the option to manage your settings similar to the scorecard. If you go to settings, it allows you to change or select different indicators that you want to be part of the scorecard, but you can also manage your color code. So if you want to use different color codes for data interpretation, you have the options to select from there. So in terms of the layout, so the sections that we have on the bottleneck analysis app. So we have the first part, which gives you the charts based on the six components that we have, the commodities, human resource accessibility and utilization. But you also have the bottleneck sub-level analysis. So for each of the indicators that you have configured, you can have it analyzed at the lower levels. If you want to analyze in terms of district or facilities, similar to the scorecard, it gives you the performance now for those particular indicators that you have configured. So although the first part shows you in terms of the tanner hash layout, this sub-level analysis goes back to a layout that's a bit similar to the scorecard showing you the actual performance of these different areas. But another part that we have, which is going to be relevant to the next app, is the root cause analysis. So at the bottom here, as you analyze in this data, it allows you to document what could be the root cause. So for example, for this intervention, we'll say, yeah, maybe I want to add a new root cause analysis. So if I click at the bottom here, new, it gives me a chance to say for that particular intervention, what area have I seen as the root cause? So I'll say maybe commodities. And then what indicator for that particular aspect in terms of commodity is the one that I think is the root cause. So I say maybe availability of DPT and then what could be the possible root cause. So maybe I'll say delays in delivery, something like that. So that becomes the way that you're documenting your root cause. So you have a new analysis, you visualize your outputs, but you can immediately document what could be the issue. And at the end here, you have the possible solution so you can as well document what could be the possible solution. So maybe add more tracks. You're saying if you add more tracks, that could be a potential solution. That's what you can do with the bottleneck app. So you can add configure indicators, visualize it in terms of layout that is a bit similar to the scorecard in terms of performance, the color coded performance, but you can as well document the root cause analysis. So in addition to this, so these two apps gives you that chance to easily configure indicators and communicate and show what is the performance. But now if you want to act on the issue, so you've seen maybe there's a lot of red for a particular indicator of a particular location, that's where now the action tracker apps comes in. So the action tracker apps, we have two of them. We have the linked action tracker and we have the standalone action tracker. So for the sake of this time that we have, I'll show you the linked action tracker. So we have here BNA action tracker. This is basically the app that is used for users to document actions related to what they have observed on the BNA app. So for example, in the initial presentation that I've just done, I tried to document the root cause for a particular output that I saw. So on the action tracker now, you can follow that app and continue documenting actions and follow-ups if you want to send budget, responsible pass-on, etc. So that, for example, was for the bad districts, so I'll select the district here. So the intervention was immunization, so we have full immunization there. I'll double-click it to select it. It was DINA district rather than BNA district, so I'll select DINA district there. And then the intervention was full immunization. So the year as we are visualizing the data was 2020, but how the action tracker is set up, it allows you to document actions for the following year. So the logic is very simple. You have done your plan, your BNA intervention set up tracking, for example, what happened in 2020. So in 2021, now you are documenting actions to address what happened in the previous period. So although the intervention for the BNA was for 2020, as we are doing the action tracking now, acting on what we observed, it's on the following period, just like when you're doing your quota review. So if you're reviewing quota one, then your actions will be done on quota two. So basically, that's how you see this linkage between the period of the BNA and the action tracker. So for here, I'll select 20, period will be yearly and I'll select 2021. Yeah, so just after selecting 2021, it gives me some of the documentation that was done on BNA. So for example here, I have the DINA district full immunization bottleneck was commodities indicators of availability of vaccine dbt and then delivery funds calling system and then budget for one repair. So this documentation was done as part of the root cause analysis on BNA, but on the action tracker now it allows me to define actions and starts acting on it to resolve what I was observed as a bottleneck. So if I say I want to add an action here, so I can now define. So based on that bottleneck, maybe hire new drivers. So this is the action that I want to take. For when do I want to start working on this action so you define your action date. You can put it the end line and date if you want who's responsible. So you say, maybe john is responsible for that and he's the head of drivers. So he's this responsible to act on this on this action. So you have your outline on the budget if require some budget so you can define based on your parents that you want. And then if you are, you want to link it to the BNA target so for example if you are acting on this action, how much of an impact you expected to have on the on the indicator or the BNA analysis that you just observed you can set a target there if you want and then. You can take that as an action so you have done where you've taken your bottleneck and now you can track it as an action. So on this part for the action planning it allows you to add multiple actions so if that is the first action that you want to take you can add another action as many as you want. And then later, maybe after a month, you want to track what happened so you have the option here for the action tracking. So the action tracking now you'll have your documentation of what was the bottleneck what was the challenge. What did you define as the action that needs to be done, and then you have for each quarter or each period, you can now document what was the status. So maybe status number one, you can still stay in progress. So what was the action so we can say maybe interviews were conducted. And then you want maybe to review this action after three weeks to see if it's still in progress or has been closed so you can say maybe I want to review it on the 17th and save that so that now documents. Based on what you analyze as a bottleneck what you defined as an action and what you want to follow up and this can be done on a quarterly basis so we have added another level in terms of tracking or visualizing your data so you can you don't just visualize data and communicate, but you can as well start tracking them so basically the full set of the apps DNA scorecard and the action trackers allows you that ability to identify visualize configure identify something and then act on it to make sure you can resolve the matter and you can as well go back as Maria said it becomes a cycle you can go back to the scorecard to see what is the impact of the actions that you're taking. So the last part is the standalone action tracker, but I think I'm not sure if we have enough time for that but the concept I can skip I can skip the standalone action tracker. Yeah, unfortunately we're running out of time and I'm sure you could take a whole day. Yeah, but thank you very much for going through that at such a rapid rate. But what we're going to do now is we've kind of had the overview on the history of the apps we've had Eric present in terms of the uses and the implementation and said it has run through at a very fast rate, each of those applications so I'm sure you have plenty of questions so please keep them going through on the chat and I see people are busy in the chat responding to you as well. And so I'd like now to look at some, you know, examples from people who have used these apps and a few minutes in terms of sharing their experiences. And so the first person that we have is a manual in the league, the data analyst at the Ministry of Health, Community Development, Gender, elderly and children in Tanzania. And he will look at the scorecard in the TV program. So Emmanuel, if you can share, if you can, yeah, so Emmanuel, yes you're there, we can see your presentation and I'll hand over to you to go through your experiences. I'll have challenges to connect so I'll be presented on his behalf, but we'll listen to him. Okay, thanks very much. I was wondering there's a picture. That's great. Thanks very much. Okay, thank you very much for the assistance. As I have been identified, I'm Emmanuel Kiregi from the National TV and Depos program of the Ministry of Health Tanzania. Sadiq, can you show the, on the slide show? Sadiq. Yes, can you see that? You're good to go now. Yes, that's correct. Yeah, I am working with the Ministry of Health, the data analyst for the National TV and the Depos program. So I support the data correction analysis and also prepare reports for the TV and the TV program. I also connected the development of the DHST PROACA system, which we named it DHS2 ETL, means DHS2 electronic TV and the lip process system. Okay, Sadiq, let us move to the second slide. Yeah, we started using the score in 2000 and 2010. Yeah, just two years as opening up the DHS2 ETL system. The score is made on the case-based data or individuals. And the program with selected 90 indicators are there with the index as the indicator, which can be generated in a period. And it does not depend on the size of data outside the system. So under the scorecard can be generated in all levels, national, regional, and even at health facilities. So to develop the scorecard, you see, we made it as one of the tools used by the media. And also the legion is some of the legions they have adapted to as one of the staff appraisal. Those staff related to the management of TV and the lip process. And also for the legion, some of the scorecard is selected for the medical officers in the case of the TV and the lip process management. For the national level also we use the scorecard as one of the tools to select the best performance legion, which we awarded by giving certificates. The challenge is that the scorecard you have to select the period. And for some of the TV indicators, you need to, for example, for the treatment indicators, you need to select the data for the old period or the previous period as you are assessing the treatment of the patient. There is no change who have been in treatment for more than 60 or more days. So let's say you can't select the scorecard, for example, if you wanted to know the performance of a certain legion or a tattoo. So the treatment indicator needed to select the data of the same quarter, but in the previous year. So it is one of the challenges which is removing the indicator as one of the options. So what we learned is that the use of the column like the quarter meeting to present the scorecard output is a tool for the performance promoted the use of the scorecard. And the author, they're using the tool at the selection for the best performing legion as also for individual performance appraisal. And the scorecard also is used to assess the data quality issue, particularly for the indicator that could be the different view when the data are not entered or updated. The scorecard also has made it easier to monitor the implementation of national and international policies for example, maybe the TB testing or the initiation of the charity for HIV or CIP cases. Also, the scorecard output informed the planning for supportable supervision, which you can know at the grants, which in Canada is a failing well in each of the districts or the legion. So the suggestion is the inclusion of the indicator to track data from different periods in the same scorecard. So thank you very much and that is a presentation of the scorecard for TB and the resource. Thank you very much, Emmanuel. And I think it illustrates there the use of the scorecard at different levels within the health system, also how you were able to adapt it. And I think quite importantly in terms of, you know, facilitating those kind of conversations and comparisons between the different, the different levels and the different facilities. I'm not sure you'll have some questions to come, but the last presentation and person I'd like to introduce is Nansamba Farida, the biostatistician from Yombe District Local Government in Uganda. So over to you Farida for the final presentation. Okay, we can we can see the presentation. I can't hear you. You're muted at the moment. You're unmuted. We can see your presentation. Thank you very much, Elaine. As introduced here to present to you the Yombe District experience for the DHIS2 scorecard bottleneck analysis is used. As introduced by Elaine and Farida, I work with Yombe District Local Government in Uganda as a biostatistician with the responsibility to collect, analyze, process, store and retrieve data on health. The use of the scorecard and bottleneck analysis applications was introduced in 2016 to the district health management team. This was following the new planning guidelines by Ministry of Health that included the use of contextual data in work planning and the use of the scorecard bottleneck analysis, cash analysis, including management analysis. The expectation was that the guidelines would enable data use at different levels, district and sub-district and indeed develop evidence-based work plans. Both applications have been used for evidence-based planning in the annual district toward health plan, program performance monitoring of those different programs as listed. They have also been used in the district performance review meetings and the whole facility monthly data review meetings too. Through identification of the list performing in CATAS, we conducted bottleneck and cash analysis to identify the possible causes and possible solutions to the bottlenecks. Varietization, action planning and implementation tracking followed suit. So before us is the first data use case where we used the scorecard in evidence-based planning and we see the reproductive maternal neonatal child adolescent health scorecard for the last financial year. The district health team as one of the list performing in CATAS that was identified was a pre-district neonatal mortality rate. And in the conversations that team had by then the possible causes were lack of resuscitation equipment, 40% of healthcare workers not trained in comprehensive emergency obstetric and newborn care in the last 12 months. Limited scope to offer delivery services. Some of the possible solutions and interventions discussed by then. Lobby for resources for procurement of resuscitation equipment. Conduct trainings for health workers on comprehensive emergency obstetric and newborn care. Conduct on job managerships. Advocate for additional funding to upgrade the health center 2 to health center 3. Progress meant so far we've had UNICEF support. The procurement of resuscitation equipment to the district. Through other implementing partners like JAPAEGO we've been able to train health workers on comprehensive emergency obstetric and newborn care and also conducting on job mentorships. And the district council has written. Has placed a submission to upgrade 7 health center 3 health center 2 to health center 3. The second data use case we see is a program for pharmacy monitoring, mainly the integrated community case management program. And by then we ran a bottleneck analysis chart for the last financial year. That's the feed that we got, but specifically on the demand determinants. We saw a bottleneck around commodities and 62% of the villages had took out days of amoxicillin or ascending rapid diagnostic tests for malaria and anti-malarial drugs. Further still we see in the sub-level analysis about 62% of the sub-counties still having a stock out. Possible causes inadequate by monthly supplies of ICCM commodities. Irrational use of the commodities by some of the village health teams. Possible solutions. Standardize the medicine and supplies kits. Conduct quarterly support provisions to the VHTs. Conduct a refresher training for ICCM. Progress so far. We have a commitment from the warehouse to standardize the medicine and supplies kit to the whole facilities in the district. And through implementing partners like TASO, the aid support organization in Uganda. We've been supported to conduct quarterly support provisions to the village health teams. And we have an upcoming refresher training for ICCM for the district health facility and village health teams. Data use case three in the whole facility market data review meetings. We see that together for SRH program monitoring scorecard. In the district we have four model facilities implementing the integrated program. And routinely on a monthly basis we extract and share this data with the facility teams. So that they can have conversations on the list performing indicators, but most importantly. Possible causes and possible solutions on how to better the indicators. Data use case for the district performance review meetings. We have that together for SRH key performance indicators that we shared with stakeholders in the annual sexual reproductive health. HIV, GBV meeting. And at the end of that meeting, we had an action planning session. Which had a range of stakeholders ranging from the district health team, whole facility teams, sub-county leaders and implementing partners. Challenges first during implementation. The limited number of interventions both for the scorecards and bottleneck analysis charts. Mainly for the scorecard. What we are able to visualize in DHS2 as of now are three scorecards to speaking to together for SRH program. And one speaking to the reproductive maternal neonatal child adolescent health scorecard bottleneck analysis app. As of now we can only visualize. Seven interventions, mainly antinatal care for adolescents. Deworming in children. Institutional deliveries. Interpreted management of childhood illnesses. Postnatal care or postpartum care. Routine immunization and vitamin A supplementation. So with above observation, we had to do some manual development. Many what we have in front of us is visual speaking to the elimination of mother-to-child transmission program. So on the left hand side, we had to look at the nine process indicators for the EFT CT program for the district in the last calendar year 2020. And we had to compare what was achieved with the target. And also look out for the gap on the right hand side. We had to run a bottleneck analysis, mainly speaking to the HIV exposed infants testing. And what we saw on the demand side, we had a bottleneck around physical access. And on the supply side, sorry, on the supply side, we saw a bottleneck around physical access. And on the demand side, we saw a bottleneck around HIV exposed HIV infants testing by that PCR. So lessons learned. The use of these applications attracts wider stakeholder participation in planning, implementation, monitoring and evaluation of health services. We kindly suggest that we have more intervention-based scorecards and botonic analysis charts in DHS2. Have the action tracker up in DHS2. We still need to have the metadata dictionary for the different protocols. Thank you for your kind attention. Over to you, Elaine. Thank you very much, Farida, for that kind of comprehensive review. And I think looking at evidence-based planning, a bit like Manuel's case, looking at performance and comparing between facilities and facilitating those kind of conversations around data. I think as you point out, what was really interesting there is that you have improved engagement with the stakeholders now that you have those apps. And, Emmanuel and yourself have also offered some suggestions on improvements, which is great. I think now we're open now to kind of questions. I see there's been a number going on in the chat that have already been addressed. So in terms of, from the start, looking at online and offline applications, there's been suggestions on, you know, what can be used offline as well. There's been some very positive comments there to all of your presenters, particularly to the development and implementation teams. Some suggestions in terms of improvements as well in terms of maybe the kind of periods, if you say kind of period for action, it might make it a bit more clear that it's the kind of future period. And I think one of the questions that is left is whether it's possible to have actions formulated against risks before they become issues within the bottleneck analysis. But otherwise, I think most of those questions have been addressed. So I don't know if anyone wants to answer that question in terms of, you know, possibly identifying those risks before they become issues on a way around this. But also they would like offer, you know, okay, so Dick, if you want to address that question, and then I offer people if you want to just either put your questions in the chat or just raise your hand. But I'll hand over to Sadiq just there. Yes. Thank you, Elaine. So I think that was a question from Ochunu Ochogi, right about documentation of actions against risks before they become bottleneck. Yeah, that is possible. As I said, I was doing the presentation. So we have two action tracker apps. So we have the linked action tracker. So the linked action tracker is the one that that is actually linked to BNA. So that requires you to first identify the bottleneck, then start tracking actions. But you also have a more flexible standalone action tracker, which does not necessarily limit you to the BNA. It's more free for you to identify issues on your own way and then start tracking actions and follow-ups. So for that I'll say, please check out the standalone action tracker app because it's free. It's not linked to BNA. So you can devise your own ways to say this is how I identify issues and these are the actions. So that is more open-ended. So I'll say possible, but please use the standalone action tracker rather than the BNA, the induction tracker app. Okay. Thanks very much. And then, Jimmy O'Glove, if you've raised your hand there, do you want to, is it a comment or a question that you have, or are you going to share your experiences? Thank you very much. I would just like to add briefly what my colleague from Uganda said. The truth is that using the bottleneck and SCOCAD, we have taken over, not as the means of everything, we have now been training the local government leaders to use bottleneck analysis as an evidence-based planning tool. It's a managerial tool and SCOCAD has been helping us so much to monitor these progress in some of the implemented indicators. Now, the challenge, we have some few challenges. We do what we call bottom-up approach planning and we start right from the lower unit like a parish level, a health facility level. But one of the challenges we have been having with the bottleneck analysis is the denominator, which we should use for some of the indicators. It's very hard to determine facility-based denominator, which has been hard to allow us to populate facility-based bottleneck. Now, also, we have the data limitation in the six determinants of the bottlenecks. We have effective coverage, we have adequate coverage. All these six determinants, some of them we may not have the routine data to populate them. Some data requirements may need the surveys. So that's why I think we still have the limitation, which we need to improve on our routine data collection, which is still also a bit. And so in line with that, it's mostly on quantitative data. We don't have the qualitative aspect of it. For example, in talk of access within five kilometer radius, that's acceptable. But there's some other accessibility, which may be qualitative, which also we are limited to use in our bottleneck analysis. So maybe you see the bottlenecks look at the two building blocks. I will remember we have the three, the six building blocks in our healthcare service delivery. So it has left out building block like leadership, health information and financing. It's not captured into the bottleneck, yet they contribute heavily on our healthcare service delivery. So this is a bit of limitation, which we still have to build within the bottlenecks. Otherwise, it has supported the country's message through FDN planning and budgeting. Thank you very much. Thanks very much. And as you say, I mean, a lot of these issues are, I mean, as Maria pointed out in the very beginning, the kind of tools to assist in the information cycle. But there will still be challenges within, as you said, the routine health information system around kind of denominators. And the app is not going to overcome those who still will have to address those issues in terms of the routine health system. And additionally, as you say, I mean, it just focuses largely on the quantitative indicators, but we would assume that, you know, other assessments would be going on within your health system. And then that information could also help you within terms of your kind of action tracker. I'm not sure if anyone else has raised a hand or would like to contribute. I can hand back over to the presenters if they would like to have a final word or if they would like to expand on any of the responses that they've given within the chat. So I'm not sure if Maria you would like to say anything, Eric, Setic, or if we rushed you a bit in terms of Emmanuel and Farida, if there's any points you feel that you would like to still add. Just raise your hands. Okay, Eric. Thank you, Elaine, and thank you, Tim, for the great feedback and for really being very attentive and contributing to the discussion. I think personally my experience with the apps is that they are really very, very good and great applications. They are some few inherent challenges in terms of data availability, but those challenges are really on case by case basis, and they can be solved and the applications have really proved to be very, very good. And for participants interested, please feel free to reach out to anyone of us so that we work together to really improve our data use experiences in our own countries. There was a question earlier on sharing the presentations. I think, Elaine, we may have to add a bit on that on how the participants can get access to the material shared today. So what we do is we will share the recording of this session so that within that you will be able to play back and look at the presentations. So this, as we said, it's been recorded and you will be able to look at it on the YouTube link that we'll send out to you. So a few more closing remarks by Setic and then Maria. Yeah, so maybe I'll just echo what Eric said. So thank you everyone for taking your time to attend. You know, this was a bit brief, but we managed to cover what we could cover and we appreciate your feedback. Maybe just last comment based on from the presentation that was done by Mr. Enkelegi. So he mentioned that the TB use case, they have used individual level data. So it's very important to also let the participants know that traditionally we are creating scorecards from aggregate data. But we have a use case in Tanzania that you can actually start tracking individual level clients using the tracker module of BHS2 and then have a way to create a special scorecard for that, which is not used in a very common manner. So I think those are the things that the apps are evolving in the field and it's good for the teams out there to know that you're not limited to aggregate data but you can as well have scorecard as when you start from collecting individual level data like what the TB team in Tanzania has done. Okay, thank you. I mean, I think that's quite important kind of illustrating the flexibility in terms of the levels, but also you're saying from aggregate to individual data and Maria. Thank you, Elaine. I just wanted to really thank everyone for their contributions to this webinar and to highlight also some of the implementation challenges as was highlighted that, you know, even though the applications have evolved, they have been designed with user input. As you can see, we have great inputs from users today so we're continuously looking at how to improve and add the functionality to respond to users. Also that the apps have really provided a space as you've seen for conversations about data, highlighting data quality issues, data quality gaps. As Jimmy highlighted, to do a full comprehensive BNA, you do need to take into account the enabling environment to have a facilitated discussion that brings in different perspectives. There are some aspects of access, financial, other barriers that may not be captured with the routine data in the system. So it's about bringing all these pieces together. But one thing why we thought that these apps could make a difference is for them to automate some of the analysis and also the ability to drill down. So you saw some examples of facility level use and being able to look at not just the picture at district level, but what's happening at facility. And so by automating some of the analyses and highlighting those data quality issues and gap part of the idea is to allow for more time for conversations about data for the action planning that deep root cause analysis and we saw some of the examples of that in Umbi district. So I hope that we've highlighted a bit how the apps can help with bringing forth these conversations about data that in order to continuously improve and go through that information cycle will need to be addressed for these apps to be effectively used and looking forward to continuing this conversation I know there are other countries that represented here on the call just to say that also as Eric has highlighted there's documentation. We've been working with the University of Oslo team to embed some of this learning into the academies curriculum and also to put some videos so we also look forward to hearing from others what we can do to support you as a community and using and learning from these apps. Thank you so much. Thanks very much Maria and just to reiterate to thank all the presenters today but also the teams behind them that they represent. And so hopefully you'll pass on those congratulations and the positive comments that we've had here, as well as you know them sharing the challenges. And thank you all for participating and I hopefully we've watched your appetite to go and look at those resources, but also we'd encourage people who are actually using these apps to also share their experiences, but you know in terms of how they've adapted it and you know some of the challenges and maybe the ways in which you've innovated and overcome them. So we welcome any of those stories and through the community of practice or through direct email to people that you know when the team. Thanks to all of the people behind the scene there's always people behind working on the technology, preparing to make everything goes very well so thank you all to those behind the scenes for organizing and making sure this webinar ran very smoothly. So we just finished there and hopefully the conversation will continue. And as Maria said that these applications are continuously evolving so we're listening to the stories. So we take into account the the requests the suggestions and the experience shared to continuously improve these apps. So thank you all very much, and I hopefully you have enjoyed the last 19 minutes. Bye.