 All right, my name is Catherine Chimetero. I am the facilitator for this session, and I have two speakers with me today, Dr. Tarun Singh and Dr. Tanmay Mahapatra. I will start by introducing Dr. Tarun Singh. Dr. Tarun Singh is a public health professional working as the country director for an international NGO, Maternity Foundation. He has 10 years experience and has worked with the Ministry of Health and Family Welfare, IPE Global, University Research Company, UNICEF and USAID funded projects. Currently, he's working towards empowering skilled birth attendants and midwives through digital health platforms. With a team of experts, they have developed the Safe Delivery app. The app aims to create a low-cost digital solution for frontline healthcare workers and to save the lives of mothers and newborns. As the country director, he drafts creative concepts and successfully reaches out to healthcare workers with life-saving protocols to strengthen health systems. He is zealous in bringing about dynamic changes in the ways mothers and newborns are given priority above all. I'm now moving to Dr. Tanmai. Dr. Tanmai holds a PhD in epidemiology from Fielding School of Public Health, University of California, Los Angeles. He has published more than 160 reviewed publications in reputed international journals. He's a physician and public health expert with over two decades experience in research, measurement, evaluation, implementations and innovations in public health and epidemiology. Currently, he's the team lead of Care India Solutions for Sustainable Development at Bihar. He has conducted several research projects and generated evidence-based results to help in policy making. He is proficient in quantitative and qualitative management and analysis using SAS, STATA and Atlas. He also teaches methods of data management, statistical modeling and analysis with interpretation from statistical and epidemiological viewpoints. He previously worked as a medical research director in Kolkata, a staff research associate in California and epidemiologist consultant in Kolkata. Now it is time for Dr. Tarun to start the presentation and then he will hand over to Dr. Tanmai. Welcome, Dr. Tarun. Thank you so much, Catherine, for providing the introduction and setting the tone for the presentation. I also welcome my co-author, Dr. Tanmay Mahapatra, joining us from the Bihar, where the study was conducted, actually. And first and foremost, I cannot miss to congratulate everyone on International Day of Midwife. And thank you so much, VIDM, for organizing this event. So today, we are going to share results of one of our pilot study, which Matanti Foundation in Care India conducted jointly in Bihar. It's on the user experience and effectiveness of safe delivery app in the nurses posted in public capacity of Bihar, India. So first, I will take you through the layout of the presentation. I'll start with the introduction. Then Dr. Tanmay will take you through the evaluation methodology implementation analysis plans. Then again, I'll come back and share the results, keep finding discussion points, and then we can have the questions from all the joining who joined the session. So what is safe delivery app? So this whole study is designed around this particular tool. So it's better for everyone to understand what it is. So safe delivery app is a free mobile application for smartphones and tablets. And it provides instant access to the labor room protocol to all the skill birth attendants or healthcare worker using this application. Currently, the clinical content designed in the available in this safe delivery app is around basic emergency obstetric and neonatal care, which means the key protocols to save the lives of a mother and new one at facility level. Currently, it is available in various languages, English, Indian languages, basically all the regional Indian languages. Apart from that, also it is available in the global languages as well. The beauty of this app is that this app is completely free. We work on the premium model, so no one need to pay for accessing the content or accessing any part, use of any module, use of getting the certificate, nothing will cost you. And also for the peripheral level health settings, we have designed in a way that this app doesn't depend on internet. So even in the areas where internet is not there, they can access safe delivery app. And also once downloaded, they can access all the modules, all the content in a single go. So safe delivery app is a very important tool and being taken up by government of India in various curriculums and various educational curriculums like BSc Nursing. How we started. So Metenti foundation in partnership with University of Copenhagen and University of Southern Denmark developed the safe delivery app. Before going into the field, we have tested the safe delivery app in Ethiopia in 2014, 15, where we have conducted a randomized controlled trial on knowledge and skills on certain modules. And we have found that there is an increase in more than 50% both in knowledge and the skill which actually helped us in reaching out to the other countries. So we have started reaching out to the countries where the need was pretty high for these kind of tools. And as India where you know, maximum number of maternal let's are happening right now because of the numbers, you know, the population size. We have joined hands with government of India and launched the Indian version of safe delivery app. When I say Indian version, it means that it is aligned with the government of India's guideline. In December, 2017 app was launched with the objective to build the capacity of service provider on the latest clinical protocols. Whenever you know, you start any project, you know, there is a need of evidence and we need to test the tool in the real settings. So earlier we, you know, tested it in the Ethiopian settings, but it was very critical to know how it is behaving in the Indian settings. So here India, which here India is already, was already sub providing the technical support to government of Bihar for the health system strengthening. So we felt that as Bihar is an area where, you know, the maternal and newborn health indicators are low and the, you know, we all feel that the internet literacy is a little bit on the lower side. So we, along with Care India, you know, partnered and you know, conducted this pilot study to explore the effectiveness of safe delivery app in Bihar's public health facility. The two aspects were actually seen during the study. The objectives were first was the usefulness of the SDA, how it is improving, you know, helping in improving the knowledge of the staff and also what is the effect, you know, how they are taking it as a learning experience. What are the, is it user friendly or not? How it is behaving? Is it crashing? What kind of technical problems they are facing? What is good? What is bad? So that we try to capture in this study through the questionnaires. Now I'll just hand it over to Dr. Tanmay Mahapatra to take us through the design of this study, Evolution Methodology and the assessment part. Over to you, Dr. Tanmay. Thank you, Tarun and thank you, Catherine and others as well as the IGM for giving us the opportunity of discussing the findings of this study. Just to start with the methodology. The assessment of this pilot was done in an evaluation mode which involved a pre-post comparison of the knowledge of the nurses who are the delivery conductors participating in this intervention. We did a baseline and then after approximately three months of the intervention completion and end line. The, as Dr. Tarun mentioned that here in India, partnering with government of Bihar was conducting a delivery service providers mentoring program since long. It was called an Amanat program and later on it was actually evolved into its Amanat Jyoti part. So in this program, different facility-based labor room post-aid nursing professionals were trained to deliver better, to give a better experience to the mother-child diet. The facilities were for some reason, the Amanat program was not running where the universe of this particular assessment to ensure that there is no such contamination of information coming from any other sources other than this particular app, this intervention. Thus, 33 basic emergency obstetric and newborn care level facilities belonging to six districts of Bihar were selected for this particular assessment of the study. Now, in these particular districts altogether, the study was implemented by during the pre-intervention assessment. The data collection for the assessment was carried out by a staff, a cadup called Knaps Mentoring Supervisors who were the principal implementer of the mentoring program as I mentioned. And they used to visit the facilities in pairs for mentoring the staff. They in these non-mentored facilities in six pairs of NMS made an exposure visit to the selected facilities to give the introduction about the app. During the process, they also administered a paper-based questionnaire, a tool comprising of 24 different questions covering all the selected for domains of this app selected out of 11 all together. Those were active management of the third stage of labor, neonatal resuscitation process and management of maternal complications mainly two, one reserving the postpartum hemorrhage, the hemorrhage that happens after delivery, person risen, a major life-threatening aspect for the women delivering, as well as another condition where higher blood pressure and protein passing through urine, proteinuria actually found to constitute a kind of syndrome or syndrome complex called preeclampsia or in its advanced age, eclampsia. Regarding these, the post-intervention assessment, the same respective nurses revisited the facilities and followed the same procedure as it was followed during baseline. Based on the responses provided by the nurses regarding their knowledge, regarding all these aspects, the information provided through the app, they were scored in a scale of zero to 24 overall, whereas zero to eight points scale were used for the neonatal resuscitation and active management of third stage of labor, a zero to four scale were used for the management of those two maternal complications, the post-delivery hemorrhages, as well as during delivery higher blood pressure and proteinuria. Based on the distribution of this score, overall knowledge-based question answering performance of these nursing professionals were categorized into either poor if the score was less than 50% or average if the score were between 50% to 75% and good if it was beyond 75%. All together, there were selected 268 nurses listed to start with, but it was required that they should actually be participating in the pre-test question answer session, the post-test question answer session, as well as through intervention throughout the in-me-twee period. Some issues were there, for some reasons, some technical glitches, either the pre-test or the post-test or together data could not be actually partially or completely collected. After removing all of them, there were 229 altogether subjects whose IDs could be matched during the intervention pre and post assessments also. Thus, this particular assessment came up with certain findings and I request my colleague Dr. Turun to take us through the findings and the discussion of it. Thank you. Thank you, thank you so much, Dr. Tanmay for simply finding the complex study protocol for everyone. So now we will see how nurses have performed after the three months of the uses of the safe delivery app. So as shared by Dr. Tanmay, in earlier slide, there were two criterias. One was we will be going to review the scores on the scale of zero to 24 and second one will be, we are going to talk about the categories that were poor, average and good. So these are the results for the overall scores, you know, the scale of zero to 24. So the maximum marks for 24 for the overall questionnaire, but we can see in the baseline that the average score was seven. And when we compare it to end line assessment, we can see in the end line assessment, the score was improved to 9.10, which reflects that, you know, there is improvement in knowledge, which is also statistically significant in point of view of statistical, you know, calculations. If we further subdivide and see the, how each module wise they have performed, we can see there was a upward trend in AMTSL earlier it was 3.06, but now it is 3.80. Similarly, in other module also, we can see the upward, you know, shift in terms of the knowledge, which actually strengthen the evidence that app actually improves the knowledge of the healthcare workers if they use it on frequent basis. Second set of results was the around the categories. So we have divided the participants into three categories, who have scored less than 50%, average was scoring between 50 to 75%, and the good who scored more than 75%. In the baseline assessment, we have seen that across all the module, including overall performance, there is high number of participants were categorized under the poor category. So we can see in the overall score, it is 95% of the people, you know, participants were scored less than 50%. When we implemented safe delivery app for three months, we can see there is a shift from poor to average. So it means their knowledge is improving. So in overall score, we can see the poor category is now reduced to 75%, which means only 75%, people are scoring less than 50%, which is earlier 95%, which is a quite high number. But now we can see that the average scores have increased. So from 5%, now average scores are increased to 21%, which means the people are scoring more between 50 to 75%, which is a good sign. And earlier during the baseline assessment, no one has, you know, came into the good category and no one scored more than 75%, but now 4% of the participants, you know, are there who has, who has scored more than 75%. If we further break it down, the AMTS and Neonatal Recess Citation, PPH and Pre-eclampsia-eclampsia, similar kind of trends we can see where there is, you know, there is a decreased number from the baseline, there is a reduction in the number of the poor category and we can see there is a shift from poor to average and average to good in across all the category, which is, which was very encouraging to see. And again, all the results were statistically, you know, significant in pre and post. As mentioned earlier, we have also checked the feasibility of safe delivery app with the positives and the challenges. So I'll share a few positives and the challenges with you, which were shared by the participant. First was it is available, safe delivery app is available in the multiple languages, which is a good thing, which is shared positive thing shared by the participant because in India, you know, various regional languages are being spoken. So to scale any application or any digital health solution, especially related to the training and the content, it needs to be available in all the regional language. So it is already available in eight languages, which they found very useful and very easy to, you know, scroll the content as per their preference. Second was the audio visual aids actually helped to make concept more clear and learning more interesting, make learning more interesting. So they have found, so in safe delivery app there are various, you know, features available one and one of them is audio visual, you know, animated videos. Other are action cards, practical procedure and drug list, but one of the mostly used feature is animated videos and they found participant found it very, very helpful because we have from our studies, we have seen that, you know, that what you see will retain better instead of just reading through the material. So they can see the animation and they can retain the knowledge for longer time through safe delivery app. Third was helpful for self-learning and knowledge enhancement. So this app is different from other app which are available in India because most of the app are collecting the data and collecting the data for, you know, monitoring and evaluation and also for the monitoring of the healthcare services. But this app is for them advice for the nurses which don't monitor them, it just, you know, they can use it for the self-learning and knowledge enhancement, which they have also felt. I will take fourth and fifth point jointly. So participants have shared that it has a certification process and they can also do a self-knowledge assessment. So Safe Delivery app has a My Learning platform where a user can assess their knowledge on their own and also can get the championship certificate which is signed by highest authority Indian Nursing Council and also carries continuous nursing education points which makes it actually a win-win situation. It is free, it is offline and also it gives you a C&E credit hours which actually helping them in renewing their licenses. Sixth point was content for simple and easily understandable. We have tried to, you know, make it very simple content is to the point, not lengthy, not very elaborative so they can, you know, quickly revise everything. Seventh was comprehensive drug list helpful for reference. It has a drug list feature which is available on the home screen. So quickly in case of any emergency you just click on the drug list and you get all the information about the particular drug, you know, doses indication, contraindication adverse reaction which they have found very useful. Eighth was question changes with each round of tests which help in brainstorming. So app is very dynamic. We have a pool of questions and pool of responses with changes after each use, each, you know, level. So it actually, you know, help them in brainstorming and think more deeply when they were going through the championship certification. Two more positives help to clarify doubts and correct practices as staff nurses come from different part of India and they come from different colleges. Save delivery app actually help them in coming on the same page on the labor room protocol because we have, you know, different teaching methodology across all colleges. So save delivery app actually, you know, take them to, you know, on same page on the, on this front, labor room protocols. Tenth is helping revision during laser time. So definitely app, as I mentioned, is free and can be used anywhere they want. So they have used whenever they are free which was a good thing. Helping identification of complication. So it has animated videos where we have shown them how any complication looks like. So they can, you know, as I mentioned, they can retain it better when they are seeing the complication because regularly they are not experiencing the complications in the facility. These are, you know, very less in number they are coming to the facility. So they, it actually improved the confidence of the healthcare worker on identifying the complication through the app intent. It is helpful in preparation of short theory sessions, clinical discussion, can repeat the test and check one's progress each time they log in. So it has a interactive dashboard where they can see their progress, how they are performing on the save delivery championship certificate path. 14th is very interesting. No special training or trainer needed for the operation. Very simple application. You need only one hour or two hour of orientation and you are ready to go to use and explore all the modules and features of the app. So it was quite encouraging for the digital point, digital health point of view. It gives intermittent notification or reminders to the user of the app. So through our channels, technical channels, we know that, you know, if some users are not using the app regularly, we provide the notification and push notification and reminders to make, you know, to improve the engagement of the user on the application. We use an action card for management of complication seems to cover all essential points. As I mentioned to the point information is keeping and only essential information is being given in the app. Keeping Bragg as homepage icon is easy to access and quickly draw our attention. Step by step information make it easy, make it a fun learning. So they have enjoyed, you know, the learning platform a lot. It's a pocket diary which can clarify doubt in user language whenever necessary. So our idea behind designing the safe delivery app was that it is, it is kind of a body for all the nurses, which, you know, they get the safe delivery app in their pocket, in their mobile phones and information is there in the cloud. And whenever they open the mobile phone, they can access all the information which is there in the cloud for them and to, you know, make them more empowered on these labor room protocols. There are challenges as well as because it is a digital health tool and very, when we started off, it was in very early stages. So there were few challenges, you know, shared by them. I'll quickly take you through those challenges. First was the agent nurses are not interested so much. So it is around the behavioral change kind of problem where, you know, some, there was a blockage of not learning new things, but definitely there was a challenge that, you know, agent nurses were not that interested knowing the safe delivery app and they were seeing as a extra burden for them. All the PC nurses are not having Android phones. It is in the low resource setting, it is a quite huge challenge where, you know, Android mobiles are sometimes not available for the PC nurses. But government of India is also providing and more tablets in the field for each and every facility. So we have tried to integrate the app into the more tablets in various state to overcome this barrier as well. Sometimes safe delivery have become monotonous. So adding frequent educational games in between would help to refresh their minds. So it was a very good suggestion we have received from the feed from the participants that the safe delivery app follows a standardized structure, but they need more games, more, you know, kind of interactive platform embedded in the safe delivery app to, you know, make it more interesting and, you know, would help them to refresh their minds. So we are working on that. The theoretical or conceptual part need more elaboration. So safe delivery app right now work as a, you know, provide to the point information at the time of emergency or a time of, you know, mentoring or trainings. But few participants have shared that, you know, more elaborative information is required around theoretical and conceptual parts, especially physiology and anatomy part, which was well taken by us. The app shuts down after repetitive use. It is a technology related issue. So many have experienced that, you know, that app is getting shut down after repetitive use. If any question goes wrong, then the user has to re-attempt all the questions from the beginning, which is pit-time consuming. So for us, a labor room protocol is the thing where you have to score 100% because it is related to saving the lives. So we have kept the standard of 100%, but many have, you know, few of the participants have shared that it should be reduced to 70 or 80%. So that, you know, it will be easy for them to get the expert level and ultimately achieve the championship certificate. The rational of the few questions was not up to the mark. It created confusion. So in the language versions, you know, many have, few of the participants have shared that the rational was not up to the mark because of the translation purpose. But after this study, we have rectified that issue. And now, you know, it is available in various languages. The last challenge was few guidelines were different from what they have learned. So they have also given that example. But as I mentioned earlier, the app is entirely, you know, on the GUI protocols. And whatever they have learned during, during their pre-service education may be a little bit different from what GUI protocols are there, because those are the updated one. So they have also, you know, shared this that, you know, few guidelines were different from the pre-service education. These are the key findings. We have seen a broad shift in the knowledge, domain-wise and overall at the end line. And it improved significantly as opposed to baseline, both in the absolute scores and also a broad shifting, individual knowledge level from, we can, we have seen the shift from poor category to average category and from average category to the good category. Still domain-wise and overall means scores remain quite less than the maximum achievable or what we were expecting. But it is an unfinished agenda, which we are going to cover in coming studies and coming, you know, programs in India. For a specific module, Neonatal Resuscitation, we have found that it, participants have performed, you know, poorly as compared to the S3 modules, which is again, area where we need to explore more through research studies. For all the participants, you know, to better know the app, you just need to go on Google Play Store or the iTunes Store and just write safe delivery app and you can see the icon of, you know, the nurse and you can download the app and connect the right language version as per the region and you will be able to access all the modules of safe delivery app along with the features. Again, I want to thank VIDM and Catherine for supporting us in this whole process. And also I want to thank Dr. Tandem Mahapatra, Team Lead CML, Team Care India Bihar and Dr. Aboli Gauri, Team Lead Capacity Building Care India Bihar and Government of Bihar. Without them, it was not possible to conduct this research in the difficult, one of the difficult part of India. So we are very thankful to Care India as well. And now over to you Catherine. Now we can move forward with the questions. Thank you, thank you, thank you very much. Dr. Tamai and Dr. Tarun for that insightful presentation. I am tempted to download this app immediately, though I'm not in the midwifery fraternity. And it's interesting to know that the knowledge of the people using the app has gone up. The user experience also is good and it's still work in progress. You're ready to take back the feedback that you've received so far. Before we go to taking questions, there's a question that was asked in the chat box by Begum. He said, is the application suitable for iOS and Android? And the question was answered by Dr. Tamai. He said yes. At times with some specifications updates, flexibility is also there to adapt to other platforms. And two of the participants, Jane and Linda, confirmed that they've already downloaded the app and they're using it. Since Linda is with us, may she tell us how high experience is using the app? Linda, are you there? Maybe she's not there, so we can continue. Let me turn off the recordings now. I ask a question, this is Jane. Yes. Thank you, and yeah, I've had experience with the app and it works very nicely. Thank you, I've been using it in Florida, USA with the residency program that I currently teach in. So I'm going to... My question, gentlemen, was really a lot of work. Thank you, it's very user-friendly. But as an educator, I agree that sometimes the operationalisation of these apps can be the issue. And I am very interested in that you specifically identified NeoNatal. And as a person that does a lot of simulation, have you tried to integrate this within schools of nursing and midwifery where they do their learnings, maybe on the app, and then they go to the models, the technical simulation models of the moms and the babies to facilitate their learning? Thank you. Thank you so much, Jane, for asking the question. It is very relevant and thank you for asking this question. Yes, so we have integrated the app in many ways. So in India, it is being used as a job bit. It is used in the skill-based training and also it is being used as a learning tool for the educators and the students. So we have recently integrated the app, safe delivery app in the formal BSE nursing curriculum where the app is mandatory for all the students and they need to complete the certification to get their degrees. Apart from that, we also sharing the teaching methodologies which we are having from the global experience which we are having. In that, we are integrating the app into the skill-based training where we are replacing the PowerPoint presentation with the videos. And also during the skill-based training, we are asking them to use the app as a reference tool because all the OSCE related steps are already there in a step-by-step manner in the app. So we are also using it as a buddy kind of thing where they can see the app and perform the OSCEs at the field level, at the facility level. We have the experiences from the Ethiopia and also in the India, we have just started the skill-based training and in June, July, we will start those teaching methodologies in India as well. Hope I am able to answer your question, Jay. All right, thank you. We are going to take these two last questions from Sheila. Sheila asks, for an international audience, please explain what education preparations nurses have. Have they studied midwifery obstetrics on your technology? Yes, Sheila, so thank you for the question. Yes, in India, we have a BSE nursing program where they go through all these specific skills, the curriculum you have mentioned. For the midwifery, so earlier in India, midwifery was not there. It was just a BSE nursing program, but recently, Government of India in 2019 launched a midwifery initiative where they have created on the international ICM standards, we have created the midwifery curriculum. And through that curriculum, we are creating NPM educator, national professional midwives and also midwifery educators in the field. So midwifery is a new thing for India in terms of the, you know, in terms of the cadre, but all these practices were covered in the BSE and BSE nursing already by the NPM nursing council and Government of India. Thank you very much for that response. Sheila, I hope you're satisfied. The next question is from Anne-Marie. She's asking, was the poorer response to neo-nazal resuscitation due to the resuscitation being more practical skills-based, e.g., getting the chest to rise? Yes, absolutely. I guess Anne-Marie has, you know, explained it very well. Definitely neo-natal resuscitation is a little bit complex process. And sometimes, you know, the staff who is working in the labour room is not, you know, that skilled enough because they are not exposed to neo-natal resuscitation on frequent basis because in India, we have a vertical training around that. But yes, it is a more skilled work and more, you know, a complex than other complication managements. All right. Thank you very much for answering those questions. I think it's time for us to move on.