 So let me welcome everyone. I see the number of people connecting is increasing. We'll just start because we have a very interesting and heavy agenda. I'm very pleased to welcome you all. I'm Karine. I'm working at the Gavi at the Vaccina Alliance and we are organizing this webinar on the real time planning and implementing and monitoring of vaccination campaign is kind of serious. We had a webinar a couple of weeks ago to present you a guidance on this priority topic for us. And this is a deep dive in the experience of Bangladesh. And we will see what they have done at a very large scale and we'll also hear from them what they have learned on very interesting aspect that can be useful for the COVID-19 real time planning and monitoring. I want to thank all the presenters who will be there. Many it's very interesting work bringing a lot of different people and teams together. And I want to particularly give a big thank you to the government of Bangladesh who are present and we know that everyone is busy but you are really kind enough to give you time to share your experience. So I'm passing over to Raquel. She is representing UNICEF. UNICEF have been heavily working on the stopping and coordinating all the presentation. So over to you Raquel and thank you. Thank you, Karin. Good evening, good afternoon, good morning everyone. I would like to thank you very much for joining us today for this webinar. My name is Raquel Wexler and I serve as headquarters coordinator for technology for development within UNICEF's Division of Information and Communication Technology in New York headquarters. And I'll be serving as moderator of the panel today. Please feel free to use the chat and cue a question and answer features to ask questions and we'll look to respond to them during the course of the webinar and there'll be a question and answer segment towards the end of the webinar as well. So you can see the names of the presenters before you. We have many presenters today and I will ask each to introduce themselves at the very top of their remarks. Next slide. So in terms of the structure of the webinar, just to give you an idea of the overview. So we'll go into the background and overview. We'll be discussing a little bit of the digital health center of excellence. We'll be taking a deep dive into Bangladesh real-time monitoring experience with presentations by government and by UNICEF and his speak Bangladesh. We will have a presentation from the University of Oslo looking at DHIS-2 toolkit and its adaptation for COVID-19 and then we'll go to the audience Q&A. Next slide. I also wanted to mention that this work has really been an ongoing work of the GAVI Alliance, UNICEF, UNICEF's immunization team, the Information Communication Technology Division with the support of the digital health team which is supporting the digital transformation of programming and innovation in UNICEF. We've been working for the past year on the consolidation of lessons and practices on the use of real-time monitoring approaches and tools for immunization campaigns across countries, across different campaign places and use cases. As Karina mentioned in February earlier this year, GAVI and UNICEF, we released a publication that you can see before you, the use of digital technologies and approaches for real-time monitoring of supplementary immunization campaigns which is available and I would encourage you all to refer to this document for additional lessons. Of course, we are tech-agnostic. The country cases look at the deployment of a variety of digital tools and solutions in Pakistan, Uganda, Indonesia and Zambia in addition to a literature review which looks at many other country cases as well. The emphasis of this work is ultimately to support the strengthening of national systems and capacities using digital approaches and tools. It's critical that we align our work in digital programming and real-time monitoring with national systems and infrastructure which is currently in place. Now, I also want to say that this work would not have been possible without the technical and financial support of the GAVI Alliance. We also think the World Health Organization, our national government partners and UNICEF and many other interdisciplinary teams across immunization health and ICTD who at country, regional and headquarters levels who have enabled this work. Of course, as COVID-19 with vaccinations underway and the need to accelerate learnings and uptake of digital tools and approaches for real-time planning and monitoring for COVAX rollout and delivery, we really recognize the importance of consolidating lessons rapidly as countries learn. And having learned of Bangladesh's recent and ongoing work using DHIS-2 for the 2020 mass, measles rubella campaign and for COVID-19 planning and monitoring delivery, we're very thankful to have our Bangladesh national EPI partners, Hispe Bangladesh, the University of Oslo and of course our UNICEF colleagues and others to share their experience with that. And with that, I'd like to pass on to Karin Kalender. Karin, can you introduce yourself? Thank you. You're muted, Karin. Sorry, it's too early for me. Thank you, Raquel. I'm Karin Kalender, I'm a senior health specialist in UNICEF's health section in New York and I'm also leading the work on digital health. And I wanted to speak just very briefly about a new initiative that has been established during the last couple of weeks called the Digital Health Centre of Excellence and it stems from... Yeah, you can go to this. Next slide, please. I wanted to frame... Sorry, previous slide. There we go. So I wanted to anchor some of the conversations that we're going to listen to today and some of the work and how that links to some of the work that we have been discussing and sharing together with W. Schoen-Gali for the COVAX delivery and how we, in program division in UNICEF, see this as a priority and how it can link to our other work that we are doing for health and immunization. And two of the priority areas we have been sort of working towards working around for the COVID vaccine delivery is around micro-planning monitoring and specifically working around sort of establishing standardized approaches to support countries around these two areas of work. And if you go to the next slide. And please, can participants mute? Thank you. So from a programming perspective, I think those who work in immunization are very familiar. Those who work in immunization are familiar with this cycle of reaching every district, which is a micro-planning cycle which includes a number of steps to sort of understand who the target population is. And this is mainly working for, this is mainly being put in place for immunization, routine immunization programs. And involves a number of steps that is usually typically in route many countries of paper-based process working to quantify target population mapping through the mapping the catchment population where children live and sort of working towards understanding how best distribute vaccines to that population. And then it involves a number of red arrows here which are the monitoring part of the cycle. Now this is usually an annual cycle and there is quarterly work plan updates and monthly data recording. Now we do see that there is a lot of utilization for improved data for these types of cycles to be more optimized. And when we talk about campaigns, we go to click one more. When we talk specifically about campaigns, we can click one more to get the next. Yes. What we see for campaigns typically is that this last part of this cycle is something that for campaigns need to potentially happen instead of happening on an annual basis we need to have this data and adjustments happening on a weekly basis and maybe even on a daily basis to be able to sort of much better optimize the distribution. And this is where we see the use of digital tools. There's a number of them like Raquel said that can really function to better track progress against targets to work with communities to track defaults, et cetera. And we also have a number of GIS based solutions that can also be better used to plan and we will hear from Bangladesh how they have utilized sort of micro planning to lead up to the point of delivery. So just to say that for the Digital Health Centre of Excellence the DICE that we are launching there will now be targeted support offered to countries for technical assistance to both these areas of establishing sort of the population estimates that will be that will be needed for understanding the population that will be needing the COVID vaccine as well as this for the distribution on how to monitor the data more rapidly and using that to inform the campaigns. Next slide. So not to go into too much detail so DICE just to mention is a multi-agency consortium UNICEF and WHO are co-hosting this but UNICEF running the day-to-day operations. So if you want, if you have an interest in a country, if you're working in a country and you know this government interest in and that there's a need for technical assistance to to institute or implement some of these digital solutions for micro-planning or for real-time monitoring and vaccination monitoring do send an email, we have an address and we can see if there is a way that we can channel some of that technical assistance through the DICE. Next slide. Over to the next slide. So yeah, I'm handing over now to Dr. Mullah from Bangladesh. Thank you for the opportunity. Now I am going to present real-time monitoring for Surabella campaign 2020 in Bangladesh. Can you hear me? Yes, we hear you well. We see you and everything is fine. Thank you. Now I am going to present real-time monitoring from Bangladesh. This is Dr. Mullah. Nationally for middle Surabella campaign Bangladesh conducted last middle Surabella campaign in 2014. 85 to 95 percent of children received first dose and 80 to 85 percent received second dose of middle Surabella vaccine from 2016 to 2019. Surveillance data indicated measures incidence increased from 1.6 per million in 2015 to 29 per million in 2019. Similarly, number of lab confront measures outbreak increased from 4 in 2015 to 82 in 2019. Due to COVID-19 pandemic situation this tissue postponed middle Surabella from December 12, 2020 to February 3, 2021 with revised strategies. Next slide. Next please. Just one second. Thank you. Why DHS-2 as a platform for RTM? Since 2013, EPI is using DHS-2 for routine EPI reporting for MR campaign planning and reporting are already existing DHS-2. Planned to use the same platform of routine EPI micro plan for MR campaign planning and reporting. In 2017, the DHS-2 was used as a platform for MR campaign planning and reporting. In 2017, the DHS-2 was used as a platform for MR campaign planning and reporting. RTM in MR campaign online micro plan daily vaccination reporting including vaccine and logistic use sexual supervision through Android app house to house visit by first-line supervisor through Android app rapid convenience monitoring by second-line supervisor through micro planning and real-time monitoring for MR campaign ensure overall quality of the campaign ensure equitable high coverage through close monitoring of achievement against target by session fixing actual target as per micro plan continuous monitoring the quality of campaign and immediate action to rectify and improve find out miss children and ensure vaccination of all miss children through mob obsession daily vaccine and logistic management plan and monitor vaccine and logistic stock online micro plan and real-time monitoring help to conduct quality campaign when physically monitoring option was very limited due to COVID-19 pandemic situation major partners technical assistance and implementation EPA DGHS heavy unicef WHO system development and server maintenance MIS DGHS unicef University of Oslo and DHS2 community next please thank you monitoring of MR campaign using real-time data monitor campaign quality and miss children through supervision app and household visit app rapid convenience monitoring app used to monitor session and community awareness and covers all level managers monitor the campaign through review and analysis of micro plan report and data from supervision house to house visit and RCM app national monitoring cell daily monitor campaign covers and analyze data from supervision house to house visit and rapid convenience monitoring app conducted division wise online review meeting with sub division level managers using real-time dashboard and provided necessary instruction to the specific unit special team visited field to validate reporting data when mismatch of data found during real-time monitoring next please lesson on challenges and successes challenges commitment from all partners for successful implementation of any innovations capacity of some agent workers to use new technology shortage of dedicated person for data entry at sub-national level sub-national capacity to provide software and server related support successes strong leadership and commitment from national level multi-stakeholder involvement especially University of Oslo details to community and MIS DGHS successfully introduced all six innovations and completed campaign maintaining quality and covers 104% during COVID-19 additional MIS children reached with vaccination building confidence to introduce technology based innovations next thank you all bye now I would like to request University of Bangladesh to come up with their presentation thank you very much Dr. Mahler and warm greetings to everybody from UNICEF Bangladesh and the team here and thank you so much Dr. Mahler for that excellent overview and the exciting piece of work that was done in Bangladesh and sharing that I think our job is really to see the that how was it done and how did we actually use DHS to for real-time monitoring but also more importantly how we're using that experience now to monitor some of the work happening with COVID-19 so I'll be joined with two of my colleagues and my team Jahid Shahid as well as Masood Parvez in doing this presentation next slide please so the first big piece that is on the slide in front of you is what exactly did we achieve and we've heard already from Dr. Mahler but there was a real success in terms of the daily monitoring that we could do and because of this real-time monitoring system we were able to identify the areas that were having the miss children and quickly do the mop ups we were able to also monitor the session quality and address any gaps as well as identify miss children for the routine immunization you just do another click we'll see the numbers and we can see the impressive reach here over 36 million children vaccinated you can see the numbers of sessions, household visits rapid convenience survey that was done as well as the miss children that were identified for routine doses so pretty impressive results I'll hand over to Shahid to tell us how exactly this took place Shahid please sorry I thank you Dr. Sanjana Jaisan Shahid working as a health officer at UNICEF so as you seen we used DHS2 as our main function system for the real-time monitoring of this campaign so we have been micro-planning and daily vaccination reporting tools using the aggregate data sites and the data visualizer apps we used for the preparing the daily vaccine and logistic distribution plan and for the supervision, household visit and RCM apps we used the DHS2 capture apps and there another one is we developed a public dashboard since it doesn't need any authentication repair so for the real-time monitoring which is developed through a PHP frame 1 next thank you so as I said earlier we developed the online micro-planning tool using the aggregate data sites three types of micro-planning data sites you see on the right side of the micro-plan form the three types of micro-plan data server developed for the outreach one, the fixed center including the outreach and the high-risk areas so through this online micro-plan data set following information we have collected the session site name, date the number of target children for the vaccinators, their names, supervisor porters and the volunteers name and their mobile number and also you see there is a use of this micro-plan they just need to enter the micro-plan target and all the other vaccine and logistic required for the session sites are prepared on fly so it was actually the DHS2 indicator to prepare in these these vaccine and logistic requirements for the calculation and the next the vaccine and logistic distribution plan we use there is a data visualizer within just a few clicks the EPI person get this required number of vaccine and logistic information sorry the earlier one yes this one so earlier this was a months-long laborious work for the vaccinators and the EPI persons to get this required number of vaccine and logistic information but this time they got this information fly they just need to enter their number of target children you see there are more than 400,000 campaign sites micro-plan data submitted using these micro-plan datasets and for the daily vaccine and logistic distribution plan and when prepared through the data visualizer apps the user just need preparing these vaccine and management plan within a few clicks so as you know these vaccine and logistic management plan also a very laborious and months-long work for the EPI person but this time these plans prepared by few clicks at the local level and for the daily vaccination information such as the coverage the vaccine and logistic uses the reporting and instantly they see their coverage and the vaccination and the logistic logistic distribution so these real-time monitoring tools the decision-making and the planning of the corrective actions was possible even the national sub-national health managers were able to observe the campaign performance progress up to the lowest unit in real-time next please so another feature of the supervision and monitoring we observed the quantitative aspect by the session observation through these supervision apps so during this campaign the session quality issues such as the way that their educated HR is available or not are the following non-test technique what is the condition of the core chain all of these are monitored and addressed during this campaign through these supervision apps and another one is the household visit app the first-line user were contacted the household visit app to find out the mis-children and coverage of the heart to and high risk areas we also monitor the campaign quality coverage the community awareness and also find out the mis-children and the campaign including the routine emulation through the RCM apps is conducted by the second-line supervisor from national to sub-national level health managers so the independent observers and the development partners so these supervision and monitoring data helps local and national level health managers to ensure the campaign quality and achieve the coverage of this campaign and you see since all of these are android apps so these works in both online offline so all the supervision and monitoring can conduct it where there is no network and low network connections including the heart to which area however this is first time we are using these apps for the campaign supervision and monitoring previously however this is the first time we are using these apps but all the national and sub-national managers use their android apps for this campaign with supervision and monitoring and previously where a paper based checklist format so there was no scope for this campaign monitoring and vision making in real time next please the lesson learned so one of the key lesson learned is the use of existing technology is the key of success it helps us also to strengthen our management system and the health of the organization the second one the government ownership and support from the all sector and the development partners such as WTO UNICEF also a key of success and the most important the user friendly apps to develop user friendly apps the comprehensive interactive training and also the motivation of the user can generate technology based innovation without providing any additional devices such as in our MR campaign supervision and monitoring conducted by the all sub-national national and the field supervisors use their own devices so now and also we could say that now these these apps are well tested also we believe that these DHS200 apps can be used at large scale implementation of any program either routine or campaign and but the last not the least the dedicated technical team must be required for this continuous troubleshooting the service maintenance systems and the service upgradation so thank you all now I would like to request Mr. Masu my colleague to continue the rest of the slides thank you Mr. Shahid and good afternoon everyone so when EPI program decided that they will use DHS200 MR campaign EMS department from DG Health has provided all the required infrastructure support they provided a virtual server with backup functionality then we configured the server and fine tuned it for the optimum performance specially for the large one current connection next please the technical architecture of the MR campaign system as you can see all the health worker from public health facility uploads their regular micro plan and reporting data through DHS200 web application and first line supervisor second line supervisor and the observer will use their mobile application through DHS200 mobile app to upload data for supervision, household visit and RCM this mobile application definitely supports the offline functionality which helps us a lot because in hard to reach area where there is no internet or the internet connection then still user could upload or user could enter the data based on this data all data goes to the connection goes to the proxy server which is ngnex we used it for the connection load balance we developed two server one is for training and one is for live we used updated DHS200 and PostgreSQL for optimum performance one of the main challenge was to create a large number of organization and user which is required for MR campaign as you can see we have created half a million organization in it and 16,000 user using automated script there are database backup functionality and sub monitoring functionality also applicable there next piece Bangladesh is the first country to use DHS200 MR campaign monitoring in large scale so when we started the campaign we faced some performance issue because of large number of organization in web version as well as android version University of Oslo technical team from day one they are also monitoring the system so when we discuss with them about the problem they come up with a new solution with two patch release one is for web version and one is for android version within a very short period of time as you can see in my below graph when we uploaded the patch in the server after 14th and 16th December everything went normal and server performed as expected next piece so country learned a lot through this large scale MR campaign implementation using DHS200 all the health manager and statistician got training during this campaign period so based on this training and our long history of development of community level individual tracking system COVID-19 surveillance system implementation took no time at all so within two or three days we started to get the data then several bug fixing from the MR campaign experience also introduced in the COVID-19 surveillance system which includes the server performance based on the successful experience of MR campaign now digital health is thinking about to use this android based application in COVID-19 surveillance system aiming to improve the completeness of data next piece so this is the COVID-19 surveillance system there are several integrated data system which collects both of daily hospital facility information etc more than 2 million individual record is there with clinical assessment lab test request, lab test result and health outcome information verification and online reporting is also available vaccine logistic management information system is incorporated this initiative is financially supported by UNICEF and technically supported by his knowledge next piece so UNICEF also supported digital health with two public dashboard one is COVID-19 surveillance which collects data from different sources like DHS-2 a CMP portal for logistic management information system and DG health control room database from citizen and all the citizen and journalist also use this web portal to extract data for daily trained analysis especially for COVID-19 confirmed cases and death by geographical location gender and age it also has some public API so that other system could use this data for their reporting especially android app and other web portal as well next please so this is another dashboard for the COVID-19 vaccination purpose where we can monitor the daily coverage of the phasmos and segmental vaccination especially gender based and geographic location based vaccine logistic management information monitoring system is also there and vaccine wasteage rate calculation is still under processing thank you thank you very much and so now we'll proceed with the presentation for the University of Oslo Rebecca can you kind of introduce yourself yes thank you so much so I work with the University of Oslo part of the HISP group the health information systems program and I'm our technical lead for the global health content obviously with UNICEF and with WHO on our standardized designs and metadata packages for DHS to support the global guidelines so today I'm happy to share with you a bit of the consolidation of some lessons learned from the Bangladesh campaign experience as well as some earlier experience from Uganda and some implications for the COVID-19 vaccine delivery that's underway in many countries so just to give you a bit of background so DHS2 already has quite a large global footprint so 61 countries are using it as a national scale HMS and we know that 48 countries are using DHS2 for their routine EPI programs and then in addition to that 34 of those countries have actually adopted the WHO standard EPI module for DHS2 so the footprint among DHS2 use for EPI is already quite large and this is the foundation that we've been building upon next slide please so I'm providing here a bit of an overview and a little bit of a mapping to a maturity model that we had developed so we have collaborated with WHO for the last since I don't know four or five years since around 2017 to develop a DHS2 toolkit that supports the WHO's EPI program guidance and so this has continued to expand over the last couple years we started with the core EPI module which was around routine immunization program data that also included facility level logistics data reporting and since then we've been working to expand a number of apps and tools and solutions and so we've categorized these as optimized which means this is WHO approved DHS2 solution and it's been replicated at national scale in many countries so we consider it to be quite optimized and fit for purpose there are some solutions that we say are still developing so they've been developed based on WHO guidelines or have been approved but they're still in the process of being rolled out or scaled up in countries so we're still kind of learning what that means in terms of replicability and then in terms of defined are some sort of novel use cases that are still representing more innovation with the platform and not yet at a place where we can say we're fully optimized so within this toolkit we consider right now the mass campaign use case is a developing one and I will share a little bit more about what we have learned so far next slide. So in terms of implementation considerations for campaigns a lot of these items they look quite familiar but how we manage them for these supplemental immunization activities has a bit of unique considerations for different components so in terms of the planning and ensuring that the planning is working across these different stakeholders but also with the budget considerations for frequency of data collection for hardware for the supervision required and how is this going to work in an SIA activity versus the routine use cases that many countries already have in place I'll discuss in the next slide a bit more about design but also really providing dedicated time for testing as part of this planning processes so that would include the user acceptance testing as well as performance testing trying to stress test and simulate what's going to happen with your server what's the kind of scale of data that you can manage before you might need to make some improvements and in order to do that helping to have sort of a rehearsal or a dry run also really helps with that the training for field teams for data collection to the supervisors for monitoring trying to bring that sort of first-line troubleshooting support close to the ground as much as possible and establishing these strong field support mechanisms during the campaign so that as issues arise there can be sort of an escalated approach to being able to solve these small issues as close to the ground as possible and being able to escalate any issues that arise so if it's just users not knowing how to log in versus potentially having some challenges with their server performance which would go higher up and the last component is really planning out what are the kind of operational mechanisms and who are these command centers who needs access to this real-time monitoring and data for action throughout the campaign I will also just mention there's still a little gap around how do we take this SIA campaign data and potentially be able to store it for example in the HMIS for future reference next slide in terms of design configuration I've summarized some of the things that we have learned by close engagement with both Bangladesh and with Uganda and in terms of the metadata structure so a lot of the core data that's being captured and monitored around doses administered in target population groups the sort of stock on hand at that site level the metadata structure in DHIS too is actually very quite similar to the routine system but what is really changing quite a bit is terms of the frequency of data reporting with the data model so for example in the Bangladesh case they had the aggregate daily reporting these sort of data sets of the core immunization data in terms of the doses administered for example but they also added a layer of individual level data capture for their rapid convenience monitoring so they actually combined these different types of data sets we also know that there are considerations there are changes from the routine facility reporting hierarchy that needs to be considered and designing this according to the operational structure of the campaign so whether our sub-district campaign teams these rural versus urban delivery sites and if that model looks different in the delivery as well as somehow being able to account for community outreach sessions and things like that so we do recommend that if it can be aligned as much as possible with the HMIS hierarchy for example down to districts that's going to make data analysis and integration in the future much easier but then once you get below that district the types of organizational or administrative units they look a little different than the routine system in terms of dashboards for real-time monitoring we have seen that there's a customization of dashboards so creating a custom dashboard app is often desired so it still uses the core functionality of DHIs too and particularly a pretty robust model for being able to disseminate dashboards according to user access at different levels so a district site we'll see their district level sub-district data whereas the National Command Center we'll see have visibility across all of the districts in operation and then we've also seen that special attention needs to be paid to user configuration as well and really thinking of that in terms of the design process there can be a huge number of users we need to consider the access and the roles and I think sometimes we are tempted to make shortcuts like sharing user accounts one per district but we've realized that that can create some problems down the road when we're trying to troubleshoot or kind of visit some audit trails or also just identify users that were experiencing issues and being able to isolate that problem next slide please in terms of scale so some considerations for scale as you saw from Bangladesh they're working at hierarchy because of this operations of the campaign they have more than 500,000 org units so that's really quite large many countries in Africa for example they might only have to a couple thousand a couple thousand org units in their HMIS so 500,000 plus is really quite large in terms of the number of users so you need to anticipate for not only the number of users but also the number of concurrent users so many times you might have a huge rush of data entry happening at the same time during the day so if everyone is submitting their reports between 4 and 5 pm at the end of the campaign day you need to not only know how many users but also how many are trying to access and push to the system and include that in your performance testing I think those considerations for scaling aggregate versus scaling tracker or individual level data so what type of data model also needs to be considered of course in an individual level model there's a lot of data being synced all the time so it has more pressure thinking about the mode of data capture so DHIs2 Android is quite optimized for offline support but also recognizing when is all of that data going to come online and push to the server right away the analytics processing so for example DHIs2 now has sort of a real time analytics table generation but that should also be tested and making sure that the analytics tables are working at the time when all of the real time monitoring dashboards are being accessed as well and the last component to really think about is this planning that being able to plan for the performance testing and using some of the server monitoring tools that UNICEF Bangladesh had shown some examples from and building up that capacity to be able to make real time adjustments during the campaign next slide please we'll pay a little close attention there's server management the way that server management works in different countries some work with an MOH data center or another data server within country some are working with cloud hosted systems but making sure there's that strong communication between whoever is sort of managing those servers and providing the hosting and having a dedicated relationship throughout the length of the campaign also being able to plan ahead in terms of server specs we need a good quality fast SSD disk but we also found that it's really good to plan for excess capacity so when you have kind of cloud based hosting it's very easy to be elastic and scale as needed but if you're relying on hardware in country you really need to think through that as the campaign gets going you don't know everything right away and you want to be able to reprovision some extra resources rapidly if you find that they're needed and of course the last component that we had mentioned was the server monitoring tools that we used to provide application profiling so this was really useful in terms of being able to connect the DHIS2 developers and also with some of the UNICEF Bangladesh staff to be able to troubleshoot some complex issues but without necessarily giving them access to the system or the data that they should not have access to next slide so this is an example of this is actually Uganda's real-time monitoring dashboard so this is actually a custom app of DHIS2 to give them the user interface that they really wanted to have so this component of being able to customize the dashboard app but use the functionality of DHIS2 it seems to be common between Uganda and Bangladesh designing those dashboard metrics they should really be driven by these key campaign metrics and how are they going to be monitored and what we have seen is a strength is to be able to have this multi-level access to support so for example from Uganda they had a national command center but also some national levels and we expected this to be common across many campaigns again with continuous analytics and considerations for scale but also being able to integrate this dashboard and linking that with the command and operational decision-making structures and using this in a way that day by day some decisions can be made to optimize that campaign so from here I will revisit so just to give a little bit of an intro and Hanan Khan a colleague from his Bangladesh will follow up we have been working with many countries with WHO and UNICEF to support countries to use DHIS2 for monitoring their national COVID-19 vaccine delivery plans so at this point we have 24 countries who are operational most of them have some sort of combination between an aggregate kind of core reporting and analytics module and a tracker based electronic registry as well as adverse events reporting and we have another eight probably more countries who are in development so this use case is already gaining traction and we have some early lessons to learn in terms of where have we drawn some linkages between the campaign use case and this COVID-19 vaccine delivery which seems to sit a little bit almost in the middle between supplemental immunization activity like an MR campaign and this sort of routine distribution through health facilities so for us we're seeing there's some similarities and lessons learned from both models of delivery next slide so here's where I will revisit our maturity model in terms of COVID-19 vaccine delivery and some of our early learning so we do believe that DHIS2 is quite optimized for real-time daily reporting and monitoring of aggregate data we really do believe this kind of scaling up this aggregate module is probably going to be quite feasible in most countries and contexts in terms of the dashboards I have this listed as somewhere between optimized and developing because we do believe a lot of this can be achieved out of the box with the system functionality that's there we do support for example calculating indicators on the fly we do support multi-level analysis but we also recognize that there's user interface design and customization that many countries are asking for to really support this use case in terms of mobile aggregate data collection I would also say this is developing I think this really has the potential to scale but in addition to DHIS2 considerations this is also around having seen so I think if I'm interpreting Bangladesh's experience correctly a lot of the core data entry was happening at web and then there was a subset substantial number of users so it was really quite at scale but these types of users doing the rapid convenience monitoring so they were doing this kind of hand in hand what we have not seen yet is a fully DHIS2 Android at 100% scale happening we'll have to see what countries are looking for and the last piece is that individual level mobile data collection so as you saw from the Bangladesh case this was used for rapid convenience monitoring and some components of this deployment but where we have not seen this at scale is for example there's not really been any effort nor there might not ever need to be but being able to report this really kind of individual level kind of case by case persons being registered as having been and vaccinated as part of this campaign but we also have some learnings from malaria bed net campaigns where some individual household level surveys have been done so we're continuing to explore and learn from this use case I believe this will be my final slide yes and so just a summary of our lessons learned so one thing I wanted to emphasize is how much our developers have been able to learn from Bangladesh and Uganda both by having some close collaboration we've been able to learn a lot about the use case and the requirements and we're very grateful to the ministries of health to UNICEF Bangladesh for actually spending some time with us to explain the design how they've managed these configuration decisions but it's also been incredibly useful for us to have a focused performance testing and being able to test DHIs too against these really real-world use cases that are very well defined and targeted and we've been able to make what we believe are some significant performance improvements across real-time monitoring dashboards as well as the tracker data model and some other things so we hope that those improvements are going to benefit any country who wishes to use DHIs too or Android in the future but we will just leave with a few reflections that is to plan for testing if there's a mobile data collection component to ensure that there's dedicated preparation for this component being able to thoroughly test the configuration before rollout building up these tools and plans for a detailed monitoring throughout the campaign to ensure an optimized performance and this simplicity of the real-time monitoring dashboards seem to have had a huge impact so designing those dashboards in the beginning with key campaign metrics and those data users is something that we would recommend so thank you very much for letting us join you and summarize a few of our observations and learnings and I'm very happy to pass over to my colleague Hanan Thank you Rebecca Good afternoon and good morning to everyone so I am Hanan Khan the leader of his Bangladesh so his Bangladesh implementation for long and I am personally involved with the DHS implementation since 2010 so in COVID surveillance system in Bangladesh is actually quite large and it starts with March 22nd March 2020 when actually the director general asked an earlier week in the beginning of March to start with the DHS2 for the surveillance system so what we have actually we took the University of Osco the core generic module for DHS2 surveillance system and customized for our local requirement but it's day by day it's become quite very large and it's actually not only the surveillance system is actually the whole combination of multiple systems to join together so if I so if you see the diagram so I think it will help you to understand so this is actually the main system so as a nutshell our COVID surveillance system right now today we have COVID surveillance system right now we have the total number of suspect cases 4.2 million and the total test done is 4.44 million and confirmed positive cases 619 thousand total active user is 1,746 so it's quite a big system so not only is the COVID test data case based and aggregate system is also have vaccine reporting vaccine logistics and real-time hospital status and if you see that we have several system integration and other links with the integration with the multiple other system so first we we start with the public app with the service app for the COVID diagnosis and then we go to facilities all dashboard from the same single source so Mr. Massoud already show several dashboards and so all dashboards are actually feed from the NHS to systems not only that the all labs including government and private facilities are actually sharing directly either directly to the DHS2 system or a middleware system where they have an entire SL like system those are not familiar with the DHS2 or not want to enter in the DHS2 those have the other system they also can send DHS2 to the national corona care network so this is actually the total overall architecture of the whole of the COVID surveillance system and we day by day actually accommodating the new requirement for example and now we are incorporating two requests one is for the self-service COVID testing requirement and another one is the hospital system integration so this too integration is on the way and hopefully next week it will be deployed so this is actually the overall picture of the whole COVID integration total COVID integration to show actually take longer time and from March we started the COVID surveillance system and UNICEF we got UNICEF technical support from the 1st of August and thanks to the UNICEF because otherwise for containing to the supporting the government and the COVID surveillance system would be difficult for us so we actually for the to optimize the server and the helps us a lot to make the server optimized and make operational so this is I think all my quota for three minutes thank you thank you very much so let's just go I think it's not I'm not showing the good screen so let me retry sorry for that I just wanted to say thank you but quickly I just wanted to say that we are going to we plan to stay 15 more minutes for question and answer and I just wanted to do a quick point on the GAVI priorities and funding opportunities so actually real-time monitoring has a short name for real-time implementing and planning and all of that so the real-time monitoring intervention among our digital health information priority for our new strategy for GAVI GAVI 5.0 strategy and also is among the priority for the COVID-19 vaccine delivery and this can be supported via GAVI funding for operational cost and technical assistance and I just put some reference about the application guidelines we are just changing strategies so all our material is changing but you can keep yourself aware of all our documentation in our GAVI website that will be revamped in a couple of weeks so this is just for reference and we are going to share the slide so you will have all the links there so we are going to do the question and answers and I will ask Rakel to come in for the moderation we can stay a bit longer because there is a lot of interest exactly well first of all I would like to thank all of the panelists for their very rich presentations today there was a lot of content shared and I see that there were a number of questions that were posed and answered in the chat so far so but I did want to ask our colleague Matthew Tomas I see you had mentioned that some of your questions had been answered and I wanted to see if you would be able to unmute yourself to pose any remaining questions before we go on to the next person thank you Oh thanks very much for the excellent presentations extremely valuable to hear this context particularly of a successful case I'm so honored to be presented I'm I just add some questions in chat and I'm grateful for people responding to them in line on the details on how the real-time monitoring component actually supported changing or adapting the program on the ground as well as I think some colleagues referenced on lessons learned for what could potentially be done with national COVID-19 vaccination campaigns and the opportunity for you know building out for product traceability or systems because it just seemed a lot of data collection and analysis so as this supports better decision-making that's great and just really curious about other ways to support management of immunization campaigns so thanks for the opportunity to listen and to pose these questions to an extremely skilled group thank you so much thank you very much colleagues online could you kindly mute if you're not speaking thank you just put back your mic on I've muted everyone Raquel okay very good thank you so well perhaps we proceed with Hassan Ulbari who has raised his hand in the chat my question is like during the Bangladesh MR campaign they have mentioned that they use the app Android application so it is something already there or they created or developed this new app thank you for your question perhaps Bangladesh colleagues can respond and in potential University of Oslo as well thank you okay thank you Mr. Hassan we are using the DHS to capture apps which is available in the Google Play Store and also you will get the latest version of the Android apps just recently updated by the University of Oslo so these we are using the Android second version provided for the DHS to capture apps for this application thank you great Rebecca did you want to come in and compliment can you please repeat the last 10 seconds of the question I was getting some connection issue my question was like did they use any existing application or did they develop any new application for these activities okay very clear sorry about that so in Uganda for the mobile data collection they actually use ODK because the users already had ODK and were quite familiar with it so the Hispuganda team sort of set up a solution where they could they had to pay very careful attention to make sure that they were structuring the data properly in ODK because it's very easy to kind of create some messy data structures in ODK which then leads to data that's really impossible to analyze but being able to set up and structure that data collection and align it to kind of monitoring and indicators metadata structure in DHIS2 actually ended up working quite well so that's actually how they did it in Uganda and in Bangladesh they use the standard DHIS2 Android app so this is a completely you know it's a generic app just like DHIS2 software is quite generic until you define it so they use the core app but the DHIS2 Android app is natively integrated with DHIS2 so whatever you design and configure in DHIS2 it is immediately available for those data entry users on mobile so as far as I know there was no customization involved in that this is just one of the core global public goods that we support from University of Oslo so I hope that answered the question Thank you very much now I would like to ask Santush Guru to unmute yourself. Thank you. Hi, thank you. I'm Santush. I'm with the EPI team in the blue chat. Thank you all for the very informative presentation just wanted to kind of get one clarification and I put it in the chat for the COVID but I just wanted to kind of get an idea. There is a micro-planning stage and there is a reporting for the RTM that I'm kind of figuring at this point so at least for the COVID thanks for the clarification in the chat but I think for the SIA is that also sitting with two different ministries like the ICT and the Ministry of Health for the micro-planning and for the monitoring that's my first point maybe just for clarification and the second thing is more of a detailed question in terms of just the micro-planning and the reporting is there a phase when the micro-planning is happening and then we are also doing some sort of a pre-registration like a pre-registration of the target groups are we going to that level of details and also the registration process when the people actually come to get vaccinated right so just wanted to kind of understand what's the impact because specially there is so much of the actual coverage that's happening during the campaign and there is the RCM and there's the house to house visit I just wanted to kind of get a feel on how that connection is being made over thanks. Thank you very much so we have a couple of questions there around ministries and roles and responsibilities and around pre-registration and registration at the time of the campaign can our UNICEF Bangladesh or EPI partners of Bangladesh Kylie respond thank you Rekul may I answer the questions please so before going to Shantos question there was another question from Matthew about the how the real-time monitoring help to rectify the campaign to answer this question I just want to put two example in we did the session with the app real-time app this provision app where the quality component was there and just one example that non-test technique are followed or not so now from the provision through app a manager can check that which union the non-test technique is not followed so he can he took the action through the I mean on job training of that of the back center that you are not following the non-test technique so you can follow this type of quality rectification done through online real-time monitoring and I want to give another example of real-time monitoring how we improve the coverage that for the household visit app any supervisor or manager from any level sub-national lesson level they can they could monitor that there was miss children found in household visit app so they plan for the mob of session and cover those unvaccinated children this is how the real-time monitoring help to rectify the campaign quality and got the desired coverage and now come to the question to the Shantosh yeah the registration we plan the campaign in the February of 2020 the COVID pandemic that time all the registration of target children done by the worker visiting each household of the attachment area but when due to pandemic the campaign suspend and we reschedule the campaign in December 2020 that time household visit was not possible due to the pandemic situation but they reviewed that registration list and project with their connection through the guardians and made the target for the each session and they put this target children in their micro plan data set and they completed the data entry in the micro plan data set in details to before campaign started before the campaign start so once they completed their micro plan data set we block the micro plan data set so after that they cannot change the micro plan micro plan target so when the session actually happened they only entered the report reporting data that how much vaccinated how much children is vaccinated so the micro plan and the reporting all in the DHS2 and target children were fixed from the previous registration and these RCM household visit and supervision these are linked all linked because in DHS2 every site was there for the micro plan reporting as an organization unit and the session and the household visit was done in that site so which was in the organization unit so we could easily link the RCM supervision household visit findings and the reporting so I think and the ministry I mean for the COVID vaccination our ICT ministry developed a app Shroka app through this app all the target people are registering by themselves from mobile app in the Shroka apps so they are registering but after their registration the vaccination plan is done at the facility level from for each facility they have one admin who can check how many people are registered for their center center I mean while registration every individual choose the center from where he wants to get the vaccine so when he choose one center the center admin people can see that yes these people are registered for our center for vaccination then he can he plan that these days these people these final people will be vaccinated and he plan physically in the app and send the SMS to those people that you come on that day this is done by the ICT ministry but the MR campaign and the COVID surveillance COVID reporting all are done by the ministry of health which is done under with the help of DHS too I think I can clarify to Shantos thank you Thank you very much for clarifying and responding to those questions I would like to ask Sir Wodru Hi Rekha just hi Rekha give me one minute to supplement this earlier question to Mr. Shantos what the replied by the our ETI colleague is fully correct I want to add one just one point for the vaccination actually is done by Shurokha but to support the vaccination we have some additional questions to one is vaccine reporting and vaccine law sticks management which we include in the DHS too as a national COVID surveillance system so that's a two addition to his point thank you Thank you very much now I'd like to ask Sir Wodruz as Wednesday to unmute yourself and ask your question thank you Thank you so much for giving me the chance to ask the questions My name is Sir Wodruz I am a consultant for COVID vaccine and production from UNICEF I'd like to say congratulations for the Bangladesh team for your successful working and digitalizing the system especially on campaign and at the same time I really appreciate the progress of the DHS to what University of Buffalo is doing because I use this DHS to system across two countries and I learned a lot from the system how much it's important comparing with other data management system or tools like Power BI and the like for example in my experience I had the chance to create dashboards even at district level in for this data utilization at their level so it is so useful especially in monitoring the routine immunization of refugees and nowadays even you are able to use at SIO campaign and also for surveillance in some countries I had the chance to see also in recruiting digitally by Wodruz tracking of the children who are registered for vaccination and now it's also scaling up for COVID so really this is a useful system I can say DHS to add many many advantages in ensuring a really time monitoring so when I come to my questions I have two questions the first one is to my understanding this DHS system is not almost established in main arrow countries like the North African countries and also Middle East countries so what is do you have any plan to reach these countries and start this DHS system my second question is starting from the national country level up to district in many countries where you are working to establish good system but from my observation and experience at regional level for the purpose of monitoring the implementation of this COVID vaccination or the routine immunization at regional levels it's not common what I mean at regional office level many places we use our BI our other system just to create dashboard and track the activities of countries but as to my understanding from experience DHS is much better what I mean is a person may enter data at lower level say it's in district national level of one country so it's possible to get real time data at regional level but it's not there is some limitation when you use for BI you have to collect the data through different system and update your data at regional level manually so there are such kind of advantages so I don't know what the plan do you have just to upgrade or scale up the system to regional level thank you thank you very much for those questions which I think are very relevant for other countries as well Rebecca perhaps you might want to respond the first question regarding the plan for deployment of DHS2 in countries where there may not be uptake yet and the other is regarding I suppose the advantages and the value of the use of DHS2 particularly around dashboards etc and aggregation of data as opposed to other digital solutions such as power BI thank you so I think on the first point one of the real advantages of having some long-term collaboration with Gavi has been able to strengthen the EPI information systems in countries over years so that they are really starting with quite a strong foundation when they go to do something like an SIA activity or a campaign and I think you may be asked a bit about the EMRO region actually quite a few countries nearly all of the countries in the EMRO region that I'm aware of they use DHS2 as some part of their system I think maybe only a handful are using it routinely for their EPI data so they might not be at the same level of maturity with those systems yet as say perhaps Bangladesh is with their national system based in DHS2 so there is a maturity model not just of our software implementation experiences and the strengthening of that system so we have also worked with countries for example responding to some of the COVID-19 vaccine delivery requests there have been several countries who have never used DHS2 but all they have in places is Excel so for example Mauritius and Surinam we worked with them quite quickly to get a solution going at least to monitor the COVID-19 vaccination because there just wasn't really a strong system in the country so that is also possible to start from the ground up but there's more work to do I think in terms of the analysis so DHS2 they have analytics features and functionality built in but one of the really key things I want to differentiate between Power BI and DHS2 is that Power BI is a data visualization tool and it's not a platform and it's not a data warehouse it's not like a storage solution it's not like you can structure your database it's really just a visualization tool so in that sense we do support the analytics of DHS2 but many countries they might wish to take that data from DHS2 and actually do their visualization in Power BI or using other visualization tools there are examples of Tableau integration and other things like that so there is a robust API to be able to pool data to other places but we do know that some of these these tools are a little bit restricted for countries because they are generally proprietary or required licensing or subscription so we feel committed to at least providing some good analytics solutions within the core software I hope that answered the question I know there was a little bit about regional reporting but perhaps that's something we could follow up if everyone has my contact info I suppose but we have worked for example at the WHO AFRO level around developing a DHS2 based data warehouse at the regional level for vaccine preventable disease surveillance and also they're working on adapting that for capturing the COVID-19 vaccine monitoring so some of those ideas have begun to be implemented and we're happy to talk about any options around pushing up to regional reporting if you'd like, thank you thank you very much Rebecca and thank you Raquel for leading the question and answers I think I suggest that we'll have to end the session we are still very pleased to listen to each other but I think it's time to release everyone and I wanted to give a huge appreciation for all the presenters and actually to all the teams that are really working hard successful campaigns and reaching all the children so huge thank you huge thank you for Bangladesh Government and partners and huge thank you to UNICEF as well you've been pushing us to really work on this area and this became a priority for us as well so really thank you and of course a big thank you to University of Oslo and all the HISP so we are a partner for a while now and we want to continue the collaboration you can find on the slide some contacts don't hesitate to approach anyone if you have further questions, comments do not hesitate to share the slides and we are really learning from each other and again thank you for Bangladesh to have taken from your time to share your experience so we will share the slide and the recording and we can all keep in touch and we'll be happy to share another time you know some other experience I think is the best to move forward thank you again and have a good afternoon all day to everyone bye bye