 Welcome everyone, good morning, good afternoon, good evening, very glad that you could join today's session. We look forward to hearing from our presenters and also hopefully we'll have some time to answer some questions. Today's session is on maturing immunization systems, linking learnings from routine EPI and COVID vaccinations as you know. We will be highlighting the work that we've done on AEFI, a metadata package, and then we will also highlight the functionality for capturing, analyzing and mapping vaccine logistics data, including cold chain reporting. UIO has been working on immunization since 2017 and with the upcoming COVID pandemic, it has given us and the world an opportunity to leverage resources to improve routine immunization programs for the long term. COVID-19 vaccine rollout has highlighted three specific needs of attention applicable to routine immunization, especially that routine immunization needs to support, needs support to maintain, restructure and strengthen the immunization services. I would like to turn the time over and welcome our colleague and friend, Madhav Balakandrashnav from the WHO, who has been working with us and partnering with us on this AEFI module. Over to you, Madhav. Thanks, can you hear me, Kim? Yes, everything's good. What about the screen? Yes. Perfect, excellent. So first of all, thank you very much and I want to sincerely thank the University of Oslo for their tremendous efforts and the work which has been going on for the last, I should say, as Kim mentioned now for the last one, one and a half years, particularly for the AEFI part, where we have worked very extensively with them and then we have also helped them to work with the DHIS too, part of it. Now I'm just, I just want to, since I have approximately only five minutes, I have just two slides to show you and I was going to lay the foundation for the AEFI module on answering one important question and the question is why is this happening and why is this important? Now this is important because like in December 2020, what you see on the screen here is the new global indicator to monitor AEFI. So when you look at adverse events reporting earlier, they were reporting adverse events for immunization primarily to a particular form called the WTO UNICEF JRF, the joint reporting form, which was collecting aggregate data. So every year they used to be looking at this data, probably in the month of June and all that, and then they used to do the performance for the previous years. This is what used to be happening in terms of total numbers. The biggest problem we face from WHO is just by knowing the total number of cases, it does not give information for action. Now what do you mean by this word information for action? We get this just a total number of cases, we just know there is for instance Angola has reported 30 cases of adverse events. But we don't know what is the 30 cases, what are the cases of NFLAXs, are there cases of people, are there cases of conversions and all that, and therefore that is why the Global Adversary Committee, when they met in December last year, they said, let us move to this particular indicator called individual serious AEFI reporting rate in a million population from a country per year. So what is going to happen in the next decade or this decade like from 2021 to 2030 is we are going to be looking at the individual, which means case based data on serious AEFI reporting rate in a million population per year. With this particular data, if you look at the numerator, we are going to be using software like the VG flow, which is the in country software for the regulators. And we are also going to be using the DHIS2 module as one of the solutions to collect this particular data. Now all said and done, you should remember one very, very important thing. The data just because it is collected by DHIS2 does not answer the solution. It has to go to the global database which is located into the Uppsala monitoring center in Sweden. And I understand that pilots are already going on in Mozambique and a couple of other countries. And I'm really looking forward to listening to this conversation today to find out where we stand on that. So this is something which is very, very important. So we are looking at the number of individually documented serious cases for which DHIS2 is a great solution. And we really appreciate the work done by the University of Oslo in developing this as a part of the EPI tracker package. And the denominator is the total population. We are getting this data from the denominator from the UN population statistics. So this is what it is. We have got a target also for one serious case. So this is what we are going to be doing. So what we are moving to, let me tell you what is what is the world moving to because of COVID like what Kim mentioned. One country, one safety data, and we are moving from one from aggregate data to case based data and a face manner, and there should be sharing of data between all parts and all groups, like between the EPI program and the regulators. And if you see the VG base is going to be primarily used for assessing the countries and the perform WHO program for international drug monitoring or the PIDM is going to be looking at the number of cases of countries reporting AEFI cases to VG base. Using the JRF, we are going to be matching that so the number of countries reporting AEFI cases to JRF is also going to be looked at. And what we are looking at is this number in bullet number one, which is coming in the part of the aggregate data, and the bullet number one here for case based data in VG base, both of them should be similar. It cannot be identical, but at least it should be similar. That's what we are looking at. The next thing we are also looking at is number of serious cases reported to VG base. We are also looking at it in terms of in the JRF also. And also this is the global monitoring indicator. In fact, I should have put this up in red. The number of countries reporting more than one serious AEFI case into VG base per million population. This is where DHIS2 plays a very, very important role and with your efforts and also I would like to take the full, I mean I would really like to congratulate the team in the University of Oslo and also the HIST programs, particularly HIST India who were involved in the development of these modules and making it quite successful. I'll stop there right now. Thank you. Back to you Kim. Yes, thank you for that insight and encouragement. I agree this VG base, the global repository is an essential component of this work based on the need to get these AEFI cases notified and in a larger global repository. So thank you very much for your support. I'm just going to share my screen now. And I just wanted to continue to put into context what we've been discussing earlier today at the larger global session and now this plenary session, the importance of these WHO digital data packages. The context of this AEFI work comes from the DHIS2 WHO digital data packages. We are a collaborating center with the WHO and in that capacity we have spent four or five years creating specific configurations of DHIS2 to match WHO recommendations and guidance around data collection and analysis for key health programs. These packages come with the guidance and training materials and can be used by any country. One of the health areas that we have worked on since the beginning has been immunization. We've had grown quickly with the ability to have 45 countries use DHIS2 for immunization data. 30 countries have installed the WHO EPI package. 36 countries use DHIS2 COVID surveillance and response and 35 countries use DHIS2 for COVID vaccine. As you can see with those numbers it did not just happen out of the blue. It was on this foundation of what we have been working with in collaboration with the WHO since 2017. Building on routine needs of immunization packages and programs and growing stronger and learning more with each year and each package. Preparing us for this global pandemic and for COVID vaccine. As of right now, breaking it down further specifically to the AEFI program, 17 plus countries are using the AEFI program and are starting to work on planning on integration with the global repository VIGI base. Because of this collaboration and strong foundation of routine immunization programs, we were able to take the AEFI package that is modeled on WHO reporting and investigation form for AEFI with strong WHO recommended 25 core variables giving a foundation and a blueprint of how to have a strong adverse events following immunization. When we first started on this a year and a half ago, it did not have anything to do with COVID-19. However, once COVID-19 came, we were easily able to change and integrate what is needed for COVID vaccine vaccines. To give you a background and a better understanding of this global package, the AEFI surveillance there is many forms associated with this process. However, within DHS2, we are focusing and we have built the reporting form and line listing form, which is the key forms needed for data collection and therefore analysis. We have taken the paper reporting form that was given to us by the WHO with the 25 core variables and able to put it into tracker program, mimicking what it looks like on paper to give countries an easy flow and integration into the system. We have some very strong goals for this DHS2 AEFI program. We wanted to be able to integrate with routine vaccine programs, digitalize at the lowest level, reduce double entry and errors through system rules, increase reporting speed, provide decision support, provide analytics through the decision chain, bring additional countries to global reporting and to promote best practices and adherence to global guidance. However, this package has not been a plug and play package. We have learned through the implementations that we have seen in different countries, it is always important to have that key consideration of what it takes to implement a digital package. Some of the key considerations for this package is how will this package be integrated and managed within the country's HMS and AEFI surveillance system? What is the existing data flow? At what level does electronic reporting go into DHS2? What types of devices should be used? What type of user groups require access to data data capture and admin? To me that point has been very important to focus on is to have these discussions at all levels to decide how your AEFI package will look for the country use. This AEFI package is a cross program stakeholder involvement. You not only are dealing with the API program, the Ministry of Health and the IT team, but also an important component is the National Regulatory Authority NRA or the Pharmacovigilance team. So this is an important implementation consideration is that you have all the players at the table when implementing to make sure it will flow smoothly. As Madhav has discussed, this package has, we have worked on integration of E2B standard for a VIGI base to be able to, we've mapped the core components of what is in the AEFI package and VIGI base, which is especially important for identifying global patterns, particularly around new vaccines for those like COVID. We have created a working prototype to exchange the required data from the AEFI module to VIGI base and Mozambique is the first country that is working on this. We are learning from this first implementation and will upgrade our tool and guidance based on this experience. We'll now turn the time over to Zephyrino, the lead of his Mozambique, who will give us some more information about the AEFI program within the Mozambique system. Over to you Zephyrino. Thank you. Good afternoon everyone. Good morning. Good evening depending on where you are. My name is Zephyrino. As I said, as Kim said, I'm leading the Mozambique team. So we are going to share now in the next 10, 15 minutes the experience for Mozambique when adopting the Address Eventful Humanization module and also we'll also talk about the interoperability of the linkage, how the prototype that is developed, how we consider that and then what are the challenges and the lessons that we're learning. So in the implementation of this module. So the DHS platform. Hello. Can you, can you see? I can see your screen is not in presentation mode. And I can hear you. Zephyrino, are you there? Sorry, sorry, I was having some issues here. So, so I was saying that the DHS platform has been used by Mozambique since 2015. In the 2020, the Minister of Health adopted the WHO package for COVID-19 for case-based surveillance. And last year, or this year, the Address Eventful Humanization was also adopted, especially when the COVID-19 administration started, was adopted in order to collect or to register the information related to Address Eventful Humanization. And this system is currently in use in the national scale or national wide. And the implementor, the stakeholders that I involved, we do have the National Health, the HMIS unit that is leading the implementation process. We have the Pharmacovigilance which is within the National Regulatory Authority, the EPI program and also partners that are involved in the process. With regard to the flow, there is a beneficial that reports the Address Event through, at the moment, it is done through the manual process, which is that they record and then they fill it in the paper. After that, there is health agents that record that information in the DHS platform. And once the information is reported, there is a notification that is sent to the AFI focal point and also there are AFI focal points at the facility. And those that also receive the paper forms, which they also send this paper form to the national level for the validation process. The challenges that exist during the process, the focus has been mainly on training the statistical digital offices. And there were very low technical or AFI focal points that were trained on the platform. And then there is also duplication, which is what I mean by manual entry of the AFI data in the platform that is entering in DHS too and also for the historical, the pharmacovisionals have been entering data for Address Event data in VG flow. So this process is still ongoing. So at the moment we are having this duplication of the process. So that way there is the development of this interoperability. It aims to cut that duplication of the efforts. So by having automatic reporting, meaning that once the data is collected from the data that is reported in the facilities, they enter it in DHS too. That information can be sent automatically to VG base. So as the challenges now, from the experience, as it was mentioned, the universal horse law, they have been developing this prototype that they aimed at sending this data from DHS to VG base, the database. There are some vision challenge that I faced. For example, the lack of coordination during the implementation of the AFI module here in Mozambique, which leads to the low involvement of stakeholders such as the National Regulatory Authority in WTO in the implementation. Of course they have been involved in designing the tools and everything, but when it comes to the implementation, there is very low involvement in the process. So this is impacting on the quality of data that is sent and impacting on the data that we do have at the moment some records that are not reported directly into DHS too. So and then this is also related to the less. So for example, the second bullet here, we mentioned that the data reporting focus more on vaccine delivery and less on the IF reporting. So because the training, it was only focusing on the statistical officers that are the ones that were trained to report the data. So there were less AFI focal points trained on the entering data into DHS too. Also there were challenges related to the devices for data collection and going to give it. This is overall for the immunization and also special for COVID and also for the, which is impacted the adverse event for immunization data reporting. So what are the strategies that the country has adopted or is adopting? So there is a building capacities of the AFI focal points. So that's still been discussed, discussed between the pharmacophage lens with WHO, there are some resources that we are going to use to train the focal points in order for them to be able to report the data. There is also focus on data validation at national level. So this which is going to also be linked to the development or at least creating capacity of the national level, especially the pharmacophage lens team, engaging with the new stakeholders. For example, the call center that will be involved in the reporting of AFI modules because they are used now to get to interact with the client or beneficiaries on the COVID surveillance. So the idea is for them also to be able to report, to register that information in DHS too, because they are using DHS too. And those mentioned here, interoperability team have been, we have been, we did assessment of the data that was reported in DHS too, and then identified the site that with more accurate data. And then based on that, we is where the pilot, for example, for the interoperability is happening. And also there is this local engagement between the local team. So the engagement of local team with them, when I'm talking about local team is the pharmacophage lens in South Egypt or Mozambique with the Uppsala and the University of Oslo developers. There is also as part of the strategies, developing or integrating within these, the EFI, the possibility of integrating USSD and SMS platform in order to get information from the public so that they can, this can be processed and then later on shared with the rich flow of the database. So as a key activity at the moment, this is a process that is going at the moment, which is doing the data validation. All the reports, as I mentioned, the data that is collected is also sent by PEP at national level. So there is a team that is sitting at national level doing data validation. Check looking at what is also reported in DHS to going through all these reports so that if there are something that is missing, can, can, can, can, can be corrected. And also if there are, this is going to guide the training. So the idea is, as I mentioned, there's a trend plan training that is going to happen to the EFI focal points. So that are now based on this validation we are identifying what are the gaps, and then these gaps are going to be used to talk to for targeting the training so that people that are able to report correctly. So because this is going to be the basis, once it's corrected, correctly collected in DHS to so then they will be send automatically to the video to the video flow. So that's what's the situation at the moment here in Mozambique with regard to the implementation of these interoperability in the EFI. I will hand now to Breno to continue the presentation. Thank you very much. Seferino, thank you very much for that very interesting use case in Mozambique. And I think it highlights the importance of having multiple stakeholder involvement and also understanding having specific EFI focal points. I think the work that you have done to work on this to make it work has been very interesting adding the EFI to the call center, having additional trainings, continuing to work on collaboration with communication to get those and to get the report correctness in place. So I appreciate that and hopefully we'll have some more time for questions. I'd now want to turn the time over to Breno, who is our new DHS to logistics lead, and he will be talking on also another interesting and complex component of immunization management which has to do with stock management and mapping of this stock. So I turn the time over to you Breno. Thank you. Okay, thank you, Kim. Thank you to the other presenters and to all the participants. It's good to be presenting here. Let me just share my screen. If you can just confirm when you can see that. Yes. Okay, great. So as Kim said I'm the LMS technical lead here at the DHS project since January, working also closely with George McGuire who's the LMS technical advisor who's also in on the call so it's been very good to be here and be presenting on this. I will go over quickly the LMS use case and then go over to the triangulation of health and stock data and showing some example dashboards of that work and what has been done. Also I'll be showing some mapping of cold chain equipment and cold chain monitoring and some of the work that's been done there and then end with some conclusions and a way forward. If you allow me first just a quick digression then into the LMS use case just to sort of frame the rest of the presentation and sort of the work going forward that we foresee a continuum or a spectrum of the logistics and supply chain management in country. So on the left you have your upstream system down to the facility level stock management. And it's in this far right facility level that we see DHS to best being used to capture stock data to help with order management to provide them performance management dashboards which is the dashboards which I'll be showing you among them which I'll be showing you. Later on, and also the cold chain management and some other functionality in the upstream system so you could do your full scale logistics management using a full scale logistics or ERP system, and really DHS to will be suited at this end user level. And there's no intention to sort of build it out to a full scale system but to really maximize this use at the end user level. So just to illustrate that, if I can just show then on the left you have your central warehouse regional district warehouse, supplying medicines down to facilities. So your hospitals your health centers and community health workers. And it's at this level that we foresee then the use of DHS to on a mobile device providing digitization of data collection, again dashboards and analytics for these end user and user health providers and others, and also providing temperature data monitoring features which I'll also describe later. And then this data being connected upstream here for specifically for logistics to a full scale ELM is but also DHS to at a higher level to allow for other levels of analysis of data. Alright, so this is then the last slide then on the LMS use cases is just the end user stock management so this is the data entry form capturing stock data and this is the data which is then used for the dashboards which I'll be moving to now. So it's really capturing based on reporting periods, your stock on hand stock received and so on and other logistics data which then can be compared and analyzed against health data to provide a higher level of analysis. Alright, moving on to triangulation of health and stock data then. So when we speak about triangulation we're referring to then synthesis of two pieces of data to address relevant questions for program planning and decision making. I think for logistics one thing that we often talk about is having actionable actions coming from from the insights allowed by the dashboards and the analytics. So that's one point to emphasize. So then the focus here was to triangulate data from routine aggregate health reporting and also the stock data and compare those so usage wasted rates and coaching data and compare those and bring those together. So the indicators were based on recommendations by WHO UNICEF and the CDC. And then the implementation team work to develop these indicators and visualizations and then test their performance and have feedback from specific countries. So here is a first of a few examples which I'll show when you bring together then on the left side you have a graph showing doses, given in the green line and stock used in blue and comparing, you know, the relevance between them. You see there's a discrepancy there in January and you can lead to then a follow up and investigation to see what the issue is if it's a data issue, and so on. And on the right you have a similar comparison doses given in stock use but here now by district on the left side you have your national level overview. And on the right, you have it by district to sort of see where you have stock issued and consume to compare and identify any potential issues. In addition to that here you have the same chart in the green and blue the stock doses given in stock used. Additionally, perhaps not immediately relevant that the yellow and red lines with ending balance and stock received. But more interesting is the chart on the right side where you have your wasted rates for close vials and your wasted rate for open vials. And there you can go down and identify for the specific district where you have issues of better or worse wasted rates and you can identify an issue which you want to follow up and see what kind of corrective action you may do in order to improve the quality of the usage and the vaccination. Here then an example from Malawi for BCG stock status by facilities so you're pretty much seeing the amount of stock with color coding by facility to see where you might have an issue of over stock, adequate stock or a stock out situation easily color coded and helps you identify error locations and where you might have some follow up action. So again from Malawi the same data with the stockouts can be mapped over can be put on a map to show specific locations with stockouts so here looking at BCG current stockouts in country or in a district and then identifying where you might have on the right side facilities with a adequate stocks or overstocks and this will lead to an action of redistributing stocks within a similar area to sort of optimize your supply chain to reduce also the risk of having a wastage due to expiring items in a location with the overstock and then supplying where you might have a stock out. Here is more of a national level analysis and this is not this is a test data it's not an actual data but of course you have here a chart showing that you have an overstock in 79.5% of facilities of course a huge risk of having items expiring in wastage and then you only have 1.1% of facilities with adequate stock and nearly 20% with under stock or stock out so again the kind of like high level analysis that will help you dig deeper to see where these inefficiencies are happening to improve then the distribution of your supplies. Here an example from Togo children vaccinated versus stock used and this dashboard shows indicators as a proportion of children immunized per antigen over doses used last month. So here if the indicator is above the limit of one it means more children vaccinated in doses used and you're trying to get to as close to one as possible to improve then the efficiency of the vaccination program. And then any kind of issues can lead to a corrective action such as checking for data entry error checking of stock inventory data is correct. Checking how doses are being tallied and then monitor and mentor facilities to make sure they understand how to properly tally and report data so again digging down to identify where the issue is occurring identifying the type of issue and leading to that corrective action. Here's an example again from Togo and this is then identifying under immunized children by by district. So then this would be to see where additional focus or different additional efforts need to to be made in order to increase the vaccination rates in specific areas. Now moving on to cold chain equipment monitoring. Now this was based on examples from both Molly and Togo. And this is an example showing where there has been a cold chain disruption in the last three months. This is based on manual reporting and I'll get to it after but there's some issues with the quality of data coming in but this at least gives an overview of the sites where you had a number of disruptions per month and then identify where maybe you need to also conduct some corrective action. What is the reason is that the quality of the equipment or or misuse of the equipment in order to avoid any kind of damage to to stocks into vaccines. It's similar here identifying showing the same information then over a map and using DHS to map functions to then highlight where you have greater issues and disruptions with the cold chain you have a larger and darker red dot a few of them are very clearly not visible and very light that might be a less significant issue. And of course you can set thresholds for these and then identified this is where I need to conduct some corrective action do an assessment. Maybe change equipment in order to provide proper cold chain capacity and in that facility. There's another type of analysis then for the cold chain and this one here is really showing the temperature alarms over the past 12 months and then it's divided by regions and showing here quite clearly that the more central regions closer to capital have less alarms and more remote region has more alarms and this is kind of confirming an assumption that more remote locations will have bigger issues with maintaining a keeping proper cold chain capacity. Now some issues specifically to the cold chain equipment monitoring it was, as I mentioned with Molly that data was not being captured from all refrigerators in a facility but only one or two so this did not give a full image of of the site. Whereas in total they were reporting on on all items on all refrigerators in the site or whether they could keep vaccines. So to reconcile the data and Molly were to use predictors for the number of refrigerators in use or use the number of refrigerators as a denominator, but this really led to just approximations. And again, the overall challenge here being that there's divergent practice practices around the cold chain monitoring, and it leads to a high customization of packages and anyone context, and which then will connect to some of our conclusions and way forward. So first the conclusions. I mean, it was a relatively generally agreed that the dashboard can be relatively easily developed and easily used by stakeholders for data monitoring and analysis. This was more or less across the board. And also some graphs in the dashboard are regularly used by data managers at the central level as I showed the different levels of analysis allowed them to make decisions and corrective action to dig deeper and find where specific issues were. And then deal with those if it's either a question of data or question of practice that they could remedy those. It would allow data managers to easily spot data coherence issues in the system and determine which facilities need targeted support again. That's what I said there. And more or less the conclusion, and this is straightforward that will look to continue to develop the dashboards triangulating health and stock data it's proven to be a useful endeavor, and this is only sort of scratching the surface of what is possible. There are some possibilities related to predictive forecasting and how you forecast stocks based on health data coming into a site so there's still a lot of work to be done and this is only sort of the tip of the iceberg of what can be done within the field. And then ongoing work as well within the LMS team here now teach us to is development of the automated temperature data monitoring tool through Bluetooth temperature sensors. So it's both guidance and design of the solution where you would then rather than have the manual reporting, rather have automated sensors and automated alarms to manage cold chain equipment. So those are some of the conclusions and way forward. So just, yeah, contacts if there's any questions or comments and also question or comments here. And back to you, Kim. Thank you, Bruno. That was very interesting. And I think I will highlight it well that this is the tip of the iceberg and it's interesting how the devil is in the details. People are working towards a goal but then it's really a refinement process and I think that is the strength of DHS to and the DHS to community is to have this communication to be able to further dive deeper and make solutions for these issues in country and using global information. I would like to open the floor to questions. We have Intel 55 five till the hour and then we will be closing the session. And we will be able to continue the converse and immunization conversation in the next session. So you can stay on the line if you'd like to continue with that. I also like to highlight that Brenno has a session later in the week to dive deeper into the LMS work with DHS to I see Mike has his hand go ahead, Mike. Yeah, thanks. It's really interesting I appreciate the presentations. I was going to ask specifically to Zephyrino and to the Mozambique use case. I know that you were some of the early adopters and have had to be not only the early adopters around the AFI package but also doing this link to biggie base. I just wondered at this point, if you were able to start over again and for some of those countries that are just starting now. What, what would you recommend what would you do differently to maybe avoid some of the problems that you described now that you've learned from this process. Thank you Mike for the question. In fact, we are one of the point point here that I like is the coordination mechanism. I think one of the one when we started in the implementation of the COVID vaccine package in Mozambique, everything. It was not only the adverse event, I think even the vaccination process. The HMIS unit was involved in the process very late. So they were not well organized. They did not know what is needed to successfully implement the package. So I'm talking about implementation, not only the digital but also the tools, everything. So having this coordination mechanism in place where everyone is on board and then they know their role in the process. We will improve the whole entire process. For example, the training, it was during the COVID, the training happened, it was virtual training, it was only in two days training. So, and then the focus it was with, as I mentioned, that entry, the statistical officers, the one that we are using DHS to the, the, the, the adverse event follow the focal point that did not use DHS to before, and then we are not involved in the training in the first place. So the idea was the data was supposed to be entered by the statistical, the statistical office. But when it comes to place, we found that because of the other challenges, they were only focusing on the reporting the number of people, the people that they were vaccinated, not looking at the adverse event. I think this coordination, coordination mechanism that it needs to be put in place. Of course, the tool it was there. And yeah, and then having also the national, all pharmacophysicists or national authority engaged in the implementation also. They were, they were in the meetings, but when it comes to the, having them in the ground, having the adverse event following immunization team reporting that they were not, they were not following that. I think it's training and also having this coordination mechanism in place will be one, one thing that will make things more easier. Yes, we have, yeah, that's what I can, I can say I think that what comes in my mind at the moment. I think that's, that's great advice. We, we of course, so I'm also at the University of Oslo for those that don't know we were always trying to learn better how to kind of make these adoption of the metadata packages and easier process. I've often asked, you know, how, how much time does it take to adopt one of these packages. Usually, I think that question is from people that are wondering about the technology. And the answer to that is very quickly. You can install a package, you know, in a day. But what what takes a lot more time is this coordination aspect and, and it to me this this one in particular AFI where you really need to have both the immunization programs and the pharmacovigilance programs and potentially different Cognizant Congress of workers involved takes, I think, even potentially kind of new connections that maybe you haven't previously had to bring together before in the work. So yes, I think that's a really important lesson to learn it's something that we try to stress now in the guidance for the package and recommendations on how to go about implementing it but But to me, it's very similar to another one of the immunization interventions we've done, which is to try to link the reporting of some of the vital events that are noticed first in immunization programs, so like birth notification. And again, this is one of those areas where you're trying to link kind of the health program to those that are responsible for the vital events. And often these linkages don't really strongly exist previously. And so plan for some extra effort, I think there and trying to make sure that there is coordination and decision-making level people can be involved in the conversation from the beginning. But yeah, anyway, Zeferino, thanks for sharing that. And of course, we're happy to continue supporting you. This is Mozambique's implementation is one we really want to learn from and continue to improve the tools with. Another point that which is also important for example, in my last slide I showed the picture where there is a late sitting with the forms that we ascend from the vaccination points to the national level. What the team is doing is they are taking that, doing the data quality check. I think of course it's linked to the coordination mechanism. But we need, there's a need of having this team that can go there and then revise not only wanting it to report the data, but at some point we find some aspects. For example, that when they are feeling it, they can make some mistake that they select one district, for example, we did find some that they were not selecting the right facility. They were having their facility but which is linked to this different district. So those decisions, when it comes to when they report it in the paper base, you need to go there and then check whether they report it to the right facility or not because sometimes the those that are collecting that are entering the form, they are not the one that are going to enter the data in the system. So you need to have this data quality team. This is coming from the experience that we had in Saint-Domingue. For example, after the first round of the vaccine, then we did evaluation of the process, then we decided within the second to set up a team that could do the data quality check before the vaccination ends because after the first round we did have two weeks just doing data cleaning. So in order to avoid that in the second round, we decided to have a team that could get whenever the data is reported or on a daily basis they can get the forms and then check that everything was reported in the system. They have selected the vaccine, the vaccine dose, whether they have reported the date, the date and the time that the vaccine was administrated so that this information is very key for when you are doing the reporting of adverse event and then in some of the forms they did not have that information. So it was very important to have that on a daily basis because if you leave it and then it will take time, we won't be able to get that information if you take one week or that it goes difficult sometimes to contact the person. So that is just adding some components of having the team that can do the data validation while the validation or the vaccination process is ongoing. Thank you. I think that's a great point as well. So we have the tracker implementation guide from the University of Oslo that talks a little bit about kind of the differences between in evaluation and ongoing monitoring system and maybe some of the key structures to put in place around data validation because these things, especially in the beginning, there's a lot to be learned about how the users are entering data, what they're missing, what is the quality of what they're entering. But you also want to make sure you design a process that will lead to some reduction in the workload. So you don't want to have some very overburdened process that will last forever because that also drives down the use of the system. But meanwhile, you do need to have confidence in the system. You need to know the data coming in are really good. So there is a lot of thought that should go into that, especially when you're rolling this out to begin with. How do you double check? How do you know that the data are quality? Are there people that you can train for that purpose? How long should that approach run on for? Will there be a kind of longer term monitoring approach? So yeah, a lot to be considered there. But yes, thanks. That was really great. I saw also in the community of practice a question from Suleyman. I don't know if Montauv is still here with us. If he'd want to address that, the question was about whether this is a change in policy for WHO to be reporting adverse events from the routine system. Or it felt like previously there was more of an effort placed on mass campaign reporting. I don't know if you want to address that Montauv, if you're still around. Maybe he isn't still around. No, I think he left actually. But maybe you can address it in community practice. Yeah, we'll try to come back with a specific answer from WHO on that. Just to say that actually I don't think this represents a significant change in the WHO recommendation that they already have. At least we did some mapping of the DHS2 countries and the VIGI based countries. And there was an overlap of at least 40 countries that are using DHS2 for immunization and that are expected to report to VIGI based. So this is something that has been recommended for some time through the routine system in addition to mass campaigns. With the idea being that adverse events are actually fairly rare. They don't happen regularly. And any individual country, the data that they have is not quite as strong as being able to bring it together into a global repository. So if you're seeing an adverse event one in 10,000, the pattern doesn't stand out very strongly in a single country. But the global repository is necessary to track down if there's an actual vaccine problem. But yes, we can see if we can get a better answer to that question. Yeah, my last question is to Zephyrino. Zephyrino, shifting gears and going to cold chain mapping. You are part of a small implementation and I think in South, South Tomei. How was it for you to be able to get the coordinates into the maps for the cold chain mapping? I hope that doesn't put you on the spot too much. Yes, thank you. Yes, I was part of that, the team that we did that activity in Tomei and Guinea-Bissau. So the idea was to have this triangulation done with the aim of helping the EPI programs to monitor the situation or the vaccine or the immunization situation to monitor that. And what we did in Tomei before, they did not have the coordinates. So we got that information from the team, the HMIS, they provided that coordinate and then we put it there in the system. Similar from Guinea-Bissau, we got it from HMIS, but especially that was UNICEF, that team that was working on the WASH project that provided the coordinate and then we updated the system. And then based on that, we are able to get some information, because the two systems, they did have or they do have mechanisms of reporting stock situation and also the cold chain, whether the freeze is operational or not. So based on that, we did the mapping in the system and we are able to show them how they could use this mechanism on the day-to-day basis. For example, if I go to Guinea-Bissau, for example now, we are expanding that. Based on that, we are doing what we call, including within the real-time monitoring indicators that will allow those individual facilities to report the stock situation, not only waiting, but they are reporting on a monthly basis. So to avoid that waiting until the end of the month, they will be able to report both the situation of the stock as well as the fridge, whether there is a problem so that they can send that information on real-time. And then the decision can be taken if it is, for example, to move the vacillings from one freezer, for example, to another in the near-neighborhood health facility. Those things can be done on real-time at least. And this will avoid, for example, them losing some of these vacillings that need to be within the freeze for some time. Yes, that was the plan, what you have done, and then what the idea of the objective of doing this triangulation in these two kinds. Thank you. Zeferino, thank you for your sharing your experience. I just want to close this session and say thank you to everyone for being here and for presenting. And it really does take a village and a community to create a mature immunization system. And I thank you for your learnings and sharings and continue the conversation in the COP and we'll go ahead and leave the session now. So stay tuned if you'd like to continue to listen with Anna and team.