 Hello everybody. Welcome to this e-seminar with Dr. Nara Winters and Dr. Judith McCool. And this e-seminar is the first in a new series of seminars we'll be holding over the next few weeks. And their topic is going to be mHealth in LMICs. So I can see from my screen people joining the seminar. So I think as the start time was four o'clock we'll see a few more people joining as we go. And what I'm going to do is just keep everyone muted. And if you've got a question you should see a little hand icon. So just tap on the hand icon and that will mean that you're raising your hand. And then I can hand over the microphone to you to speak. Or what might be easier because there's quite a few people who are attending is just to post your message in the chat box which should be down on the bottom right of your screen. And also if you just use the chat to say if you're having any problems. So things that might happen might be that you can't hear us speaking because we've accidentally muted ourselves. And the slides aren't moving on or something like that just if you could just post that in the chat, then I can hear you. And you can also just type in the chat to say hello. If you can hear me and if you've got any points you want to make. So let me just introduce our speakers. Let me just check that we're recording and everything's going okay. Okay, let me introduce our speakers. So first of all, talking for the first part of the seminar is going to be Dr. Nile Winters and Dr. Winters is an associate professor at the University of Oxford in learning and new technologies in the Department of Education. And he's also a fellow of Kellogg College. And if you want to follow Nile on Twitter. You can see his end when this is Twitter handle below, and I'll hand over to Nile in a minute to give you a bit more of an introduction to his talk. And then after Niles finished talking for about 15 minutes and we've had a few questions. I'll hand over to Jude McCool. Judith McCool is an associate professor at the University of Auckland, and she leads postgraduate courses in global health delivered within the Masters of Public Health and the Masters of Health leadership programs at the University of Auckland and her Twitter handle is Jude McCool there. So you can follow on Twitter. So without further ado, I just wanted to just quickly mention that many of you will have seen the advert for this through the Health Informatics Forum website. So hopefully if everything goes to plan will be recording and the videos will be uploaded to YouTube and I'll post a message on the forum to let everyone know they're available. But you can use the forum for if you've not signed up already. We've got lots of free online courses. You can network with other members, and we try to post news about initiatives in digital health. So now I'll just hand over to Nile. And he can start his talk. So let me just change presenter. And I'll just need to unmute him as well. Let me just unmute now. Okay, so you should be able to unmute now. Yeah, that works. Thanks Chris, and thanks for organizing for the invite to present. Hi everyone. So today they're using technology to support the training and supervision of health workers in my seas. And I'll talk about mainly focus on two areas community health workers and training nurses and both collaborative projects and Chris and show me as well who's here been involved in. So, let's get started. So by way of background, and at the intersection of end diagram there. So what I'm interested in is the intersection between technology training and global health. So essentially, looking at the development of new technologies, be they mobile virtual reality technologies are most likely using artificial intelligence to support the training of health workers. And from a theoretical perspective, we use a social justice and development framing for this, but I won't be talking about that today. So just focusing really on the intersection. And a lot of our work focuses on working with the most marginalized in LMICs. One thing to keep in mind when working with the most marginalized is that digital technology, particularly them help is often presented as a way to support their health care needs. But there's lots of work, particularly in education and communication and in internet studies. That shows that if you're not careful about how you design and develop these technologies, they can actually lead to increased marginalization, particularly for people who are already disadvantaged. So what happens is instead of reducing an advantage, you actually end up adding to it so people become multiple disadvantaged. So I've just been in a number of references there, people want to read them on that background. So what we're trying to do is ensure that any intervention we develop, any technology we develop, does not increase inequality in any way. Okay. So that's the first thing to keep in mind. So just by way of background, so where do we work? We work mainly in East Africa, primarily in Kenya. And you'll see some photos there of the work we've been doing on the left in rural areas in Kenya and in the middle there in informal settlements, mainly Kibera. The photo on the right, I'll come back to you in a second because it's from the project I'm going to talk about. Okay, so just very, very briefly, I talked about working at the intersection of technology training and global health. And so one of the ways of doing that is to take into account approaches to learning and training. So theoretically, from an educational point of view, what way can we think about how learning happens? So we can think about it as information dissemination, which is essentially developing things like training manuals, putting PDFs on tablets. And that's on the educational research to not to be that effective, particularly for practice based interventions. The other way is to think about learning as sort of problem-based, situated or exploratory, where a teacher or supervisor is often working as a mentor, facilitator, and Italian contextual and learners are engaged in often case-based work. So this is shown to be pretty effective. Another way to think about learning is as a conversation, so almost like a one-to-one tutorial. And this is important for supervision, which I'll come on to now. And so you don't need to understand this diagram, but I just want to highlight the reference. This is learning theory, thinking about learning as conversation. And this is a conversational framework developed by Dana Lara-Lard that we use to inform the design of our learning technologies. Okay, so that's learning. So what about supervision? Well, you probably read a lot about supervision of community health workers, for example, which has been a timely topic of research for a number of years now. And basically what the research says is if a community health worker has ongoing regular support and feedback, i.e., good quality supervision, you can see some more references there on the screen, they tend to do a pretty good job. Okay, so it's kind of bluntly obvious, I guess, if you think about it. So it's amazing to think about actually how then can technology support supervision that is regular and allows supervisors to give feedback to community health workers. Okay. There's a number of ways in which you can do that. I won't go into that here, but essentially you want to ensure that the supervision is of high quality, that it is supportive, and that it basically aligns with learning as a conversation, the theory I mentioned earlier. So how have we done this in the past? Well, one project I'll talk about is a community health worker project run with the NGO Amref Health Africa, whom some of you may know, headquartered in Nairobi. And this was a mobile intervention to help community health workers assess stages of childhood development in under fives. I'll show you a number, link to a number of papers at the end, you can find more details about this. But really, in an image like this, in a situation like this, which is where a community health worker is visiting a mother with a young child, a young baby at home, what is the role that mobile technology can play in terms of learning and supervision in this type of context? Right. And what we did was we developed a mobile application that allows community health worker to assess the stages of childhood development to record them and to share them with their supervisor. Okay. And so that's graphically represented here by the arrow in the middle. So you can imagine the community health worker using the app on the left, the supervisor using the app on the right. And through this interaction, the supervisor is able to provide timely and tailored feedback to the community health worker. Because instead of a community health worker having to reflect on their own work and try to talk about that, they can actually share the data that they made to make their decisions. And so the supervisor has not only the data upon which decisions may have been made by the community health worker that they can then use to tailor their supervision either through the application or face to face. Okay. In the interest of time, I won't go into too much about the methodology of how we developed this, other than to say we support taking a highly participatory approach. So we use an approach of participatory action research in which the design and development and implementation of the mobile training application is done with the community health workers from day one. And there's a couple of slides here on the details and methods you can use. And to do that, I'm happy to take questions on those at the end of people want more details. But in the interest of time, I won't go into these and any substantive detail here, but also if you want to follow up with me after this lecture, I'm happy to share this information with you. But I just wanted to highlight the methodological approach taken. And so that's the first project. And the second one I want to talk about is actually one that's one by Chris here that we collaborate on, which is looking at the role of virtual reality. You can see here, and HEC headset by headset on the left, and a mobile game that the team have developed here on the right. So what we're interested in is what learning theories best support training in this context. So this is for nurses working on, in this case, neonatal resuscitation. And I'm going to talk about ongoing work. It's actually quite early work on using artificial intelligence to support the provision of appropriate feedback for the nurse for the learner. So what happens here and you can see on the on the left, you have the mobile game and on the right, you have the virtual reality environment. And in both of these situations, the game or the life platform is asking the learner to make certain decisions. So one is what questions you might ask of a departing colleague on the left on the right. It's to choose from a selection of items before they undertake resuscitation. And in these spaces, what we'd like to do is to be able to provide appropriate feedback if the nurse needs help in answering these questions. And we want to do that in a naturalistic manner. Excuse me. And one way to do this is to analyze what the nurses might say, for example, if they're in a virtual reality space, what questions may they ask and recognize that they're asking those questions and provide appropriate feedback. And we use AI algorithms to do that. Okay, let me skip that. And what we're doing is instead of having to build these machine learning algorithms ourselves, we're using open source machine learning libraries in this case, and TensorFlow.js, which one's in a browser offline and TensorFlow light, which is designed for mobile phones and both provided by Google. And so what we want to do just to give you one example is to answer questions that come up in the virtual reality world. Okay, so for example, you can see some nurses here on the left and some of our research participants using the virtual reality, the life platform, and in the virtual world, they're going through the scenario of resuscitating a newborn baby. And the system may want them to answer the following question, what equipment do you want to have ready to resuscitate the baby? Okay, so they may say something like, well, what do I need to resuscitate the baby? What equipment do I need to resuscitate the baby? Tell me what I need to get ready. And what we want to be able to do is recognize all those as paraphrases of the same question that want to be answered. Okay, I'm not going into the algorithm that does this. But basically in bold there in yellow, you see the highest one is a ranking of how we recognize each of these questions. So for example, the one in yellow, what equipment do I need to resuscitate the baby is most similar to the question, why do you want to have ready to resuscitate the baby? And if you scroll down, I don't know if I can move my mouse here, you can see the third one from the bottom at 0.34% is not a good match. So tell me what I need to get ready. The system won't be able to recognize that because the match isn't good enough. And this is just one little example of what we're working towards to be able to recognize naturalistic interactions in virtual reality spaces in order to provide answers to or feedback to the learner that's appropriate and time sensitive. And that's going pretty well at the moment is again it's ongoing work. And so in summary, before I take questions and then hand over to Judith, I tried to give you an overview of approaches we've been taking to training and learning, highlighting the fact that it's an interdisciplinary approach. I don't think anyone field can address this on their own thinking about the fundamental role that learning theory needs to play from the beginning of the project. We believe that AI in the simple example of showing you about paraphrasing would have strong potential and in global healthcare context, but always keeping in mind and digital inequalities and drawing on AI for education research where appropriate. And I've got a selection of papers there if you want to follow up on any of these areas. I just want to end by thanking everyone, including the whole team and our Twitter handles, etc. are there. So thank you very much. Okay, let me just. Yeah, I think it's working there. So people should still be able to hear me. So that's a really good question. Chris and it's actually very, very timely. So one of the slides I just said over very, very quickly was this divide between the AI for good agenda that you've seen with the AI for good global summit, for example, in Geneva, but it's been a big push towards the idea that AI will support global healthcare in a positive way. But actually, on the right of the screen here, there's a number of books and this is just a selection that shows the problems with AI. But mainly developed by social science and internet researchers, thinking through the ways in which bias, for example, is embedded in AI and how it could potentially lead to further inequalities. And so one of the ways we've been trying to do that is looking at the data sets we use to train our algorithms and making sure they're reflective of race, gender issues, for example. So I think it does have a potential to not do good, let's say, to do bad if you're not careful. But if you approach your design and development of your intervention in this informed way, I think you can work towards addressing that problem in a pretty good manner, I would think. But it's ongoing work, as I say. Okay, thanks very much now. And we've just got a question from Claudia. So I'll just try and unmute Claudia so we can hear her. Hello, Claudia, can you hear us? Hello. Yes, I can hear you. Thank you very much. My question was picked up actually in your last answer. I don't know if you want to read it out, Chris. Let me just say it says thank you AI has a lot of potential to augment and support healthcare and LMIC. But it can be controversial. How do you propose to reassure users and also how affordable scalable is this technology. Thanks Claudia for the question. I just had to unmute myself there. So how affordable and scalable it is. But this is why we're trying to work through this AI for all agendas. So one of the things we're interested in is with the availability of open source machine learning libraries that are built into Chrome, for example, and are also built into increasingly built into browsers. What are the ways in which organizations such as ourselves can utilize these libraries? Okay, so what we're not talking about is having access, for example, like some hospitals in London who maybe work with DeepMind, for example, to millions and millions of images or have access to a big data institute. There's definitely a place for that. But that's, as you say, massively resource intensive. So what we're thinking about is ways in which we can still work at the cutting edge of innovation, but by leveraging a lot of technical work that's already been done. So that's how we're trying to address that. And the first question was how do you reassure people? I think the only way is to do this is the proof is in the pudding. So I think as we work through this, we need to be able to show, and actually it's been developed in a fair and equitable way. And that's why I didn't go to the point in the top, but we tend to take a authoritarian view on the design and development of our interventions. And so that's the underpinning view, rather than a utilitarian one or rather than thinking too much about scale from the beginning, because often the needs of the most marginalized can be overlooked if that's the driving intervention. I'd be happy to talk more about that, maybe offline because I'm aware Jude needs to kick off as well. But I hope that at least partly answers your question, but happy to follow up. Yes, thank you. Great. Thanks now. And I think it was just another couple of questions actually. So one from Abigail says, thank you, Dr. Now concerning the project for community health workers, was there an ICT training of the health workers. And then there's another question from Alu and he said, thank you, Dr. Now how do we ensure the maximum usage of AI in remote areas are developing countries considering scalable limitations in my country, Nigeria. So yeah, so about training local personnel and reaching remote areas. Thanks. Thanks both. So the training, that's a really good point. So we did spend some time and training community health workers with AMREV with the local NGO who already have a program in place for facilitating training. So we're building on that. So we're building into existing approaches to training and programs. And we did need to balance that though again. So we gave time for people to sort of get used to and discover the technology for themselves. So the idea was also to look at ways in which people use or appropriated the technology and build on on that. So for example, one of the ways we had it a few years ago now, so we had built our own chat engine into the into the app and it wasn't really used. And then WhatsApp started to become really popular in Kenya. And so one of the researchers from Kenya suggested we use that and that took off like wildfire. So I think been flexible and open to emerging uses on the ground, rather than thinking you have to take all the training boxes upfront. And I think is a good approach. So it's this balance between developing technical capability, if I could call it that, around training and use of the tools and then open to thinking about as new technologies emerging, new uses of technologies emerging the ground, how you can leverage those at the same time. And second approach, the second question going back to scale again. I think again this depends on such a, almost a bigger question than projects of the size we'd be involved in. So essentially, we're leveraging, for example, the uptake of an usually Android, and a little bit with iPhone but increasingly Android based devices. So the ways in which Google and others are embedding AI assistive technologies into their into their phones and we're building on that. So this research project focus on AI, for example, would not be concerned with the use of feature based phones. So in that sense, when we're doing a research project is not a development project. We're looking at where we think the field may be in three to five years time. So that's what we're thinking about when we're thinking about scale we're not thinking about as an implementation project for today. We're just saying, hang on, marginalized communities or people in rural communities, what types of technologies might they be able to have access to in three to five years time. And how can we design interventions now that will be appropriate for them as the as the technology develops. And just just a last point on that. So, for example, in Kenya, a few years ago when we started probably late 2000s, feature phones before the popular most popular phone. And now it's probably cheap, low end smartphones, you know, you're 30 35 dollar. And I'm I've did a survey looking at, and at least in in key bearer, I think, maybe put more broadly, I have to check with them. And to say the cost of a smartphone. Now, versus the cost of what people are paying for feature phones 10 years ago is roughly the same. So, um, at least when you're developing a research project, I think you don't want to be blinded by the technology people have in their pocket now. And development projects, I think, more specifically, you do need to focus on that directly. But as we're thinking about where we might be in three to five years time. That's where we're our focuses. So I hope that answers your question. Okay. Thanks everybody for the questions. So we're just about half past. So we'd better just pause for a second while we hand over to Judith. And we'll start again in just one minute's time. Once we've got you to set up. Thank you very much. Okay. Hi everyone. And again, thanks to Chris for organizing this event. This is quite exciting actually I think this is a great idea. So I've brought you one together, the three of us in the room here and sharing some of the ideas and some of the projects we've been working on without using carbon. So this is hopefully going to be a useful afternoon for everyone. What I want to talk about particularly is how we are looking at using mHealth or broadly speaking digital health to support universal health coverage in small island developing states. So I work at the University of Auckland in New Zealand. And there I may, as Chris mentioned, teach on the global health program. Okay. So this first slide I like to show because it really like I guess it's helpful in positioning where New Zealand is in global map. So it's really nice to fall off, slip off the bottom and the Pacific region particularly often gets draped around the edges of a world map. So it's really nice, I think to demonstrate a show of map that has Pacific region, Pacific Island region really front and center. But also recognizing that I'm talking about other small island developing states that include small islands within the Caribbean, the Africa and Asia region as well. I'm particularly going to talk about the Pacific region because that's where I'm based and where I do most of my work. So these islands, there are 22 Pacific Island countries and territories, and like most small island developing states face considerable challenges but also opportunities in terms of how they seek to develop and of course grow their economies but also improve health of their populations. And a recent meeting and focus on small island developing states or otherwise called SIDS priorities. They recognize three main areas that are really important and of course the fisheries are particularly important to the Pacific and other parts, other small island developing states. So it's been really important for both health and nutrition but also for sustainable development security as well. Also really important area is water and sanitation. These are certainly in the context of climate change becoming increasingly fragile and certainly as interest in malnutrition in all its forms or non-communical disease and sexual reproductive health remain really important concerns for health. So thinking about some of the really fundamental drivers and risk factors for health are particularly for other regions, water and sanitation. And as I mentioned earlier, climate change resilience is really I would say one of the front in foreground issues for SIDS. These are absolutely critical challenges in terms of how to manage, adapt and mitigate climate change effects on health in particular. So a lot of the work around developing interventions has been focused on climate change and Fiji in particular has taken a leadership role in this field. So in terms of small island developing states and what they do collectively, what they can do, working together to achieve their health outcomes and goals, there's been a really over the last 10 years and certainly in the last three to four years a focus on partnerships. And I'll talk a little bit more about this later, but they really want partnerships to be the foundation for development. And this will refer to particularly with mHealth or digital health. So they want some of the partnerships to be really SIDS specific. So really working together across the regions to see where they can share ideas and resources, capabilities and in some respect capacity. They want these initiatives and partnerships particularly to be measurable and monitorable. So we want to see what the benefits are working together. They want to be achievable and accountable in terms of what they're aiming to achieve in this collective approach, resource based and focused. And they want to have very clear timelines for implementation and transparency by all parties. So this idea of actually these small island developing states share common risk factors, common challenges, particularly around not just only climate but other factors. And there's value in working together to collectively share resource and achieve health specific targets and goals. The recent World Health Assembly, there was a strong focus on in particular to gender items on small island developing states in relation to the broader theme of the World Health Assembly on universal health coverage, leaving no one behind. And I know particularly that there was a lot of discussion around how to improve information systems, how to use digital and other technologies to enable individuals and communities to identify what their health needs are, to actively participate in the planning and delivery of these services. And a particular interest in helping and working with communities to maintain their own health and wellbeing. So this light really captures I think one of the themes that really transit a lot of the discussion at the World Health Assembly a couple of weeks ago. So just focusing particularly on information communication technology status in the Pacific region. I'm not speaking to other small island developing states and other parts of the world, but in the Pacific. I think it's interesting and important to think about when you think about digital or in health, and this understanding that mobile phones are ubiquitous. That everyone has access that everyone has a phone in their pocket or access to internet. But in actual fact in the region there are huge gaps and the information we gather from the ITU or GSMA the Global Monitoring Services Association gloss over some of the deeper issues or access questions in terms of who actually has access to mobile. But Pacific region has been one of the, I guess for one of the better words, slower developers or adopters in mobile technologies and ICT partly because of the remote and remoteness of the region. Some parts of the region have only recently been connected by satellite submarine cabling and so forth. And it's been a slow switch over to digital. So there is absolutely there is access to ICT there is definitely a role for using mobile in the region for health and in other benefits. We see some really good value emerging from that, but I think it's important to recognize in terms of thinking about equity and ensuring access to the benefits of M health for all but particularly the most vulnerable that keeping an eye on the developments and the gaps in terms of ICT coverage. It's not quite as complete as it appears on some websites and some provider information. And we see here in some of the countries of the Pacific Islands region just mentioned on this slide so we know that sort of Papua New Guinea, Solomon Islands and Fiji being the largest Pacific Island countries have the highest rates of the highest proportion of the population subscribing to mobile and the highest rate of connections. We go down to some very small islands such as Nui and Tokalau and Tuvalu which which internet and subscriber penetration is as low as 11, 17%. So this is really important when you think about scaling up and enabling access to mobile technologies for health and who's likely to benefit so across the region huge diversity within countries also considerable diversity that's likely to be patterned not just by geographical location some remote highland certainly difficulty in access but by socioeconomic status and by gender. So internet use for example in the Cook Islands we're also working is very expensive. So I want to talk very briefly about one initiative we developed in partnership with the Ministry of Health and Samoa that gives us or gives an opportunity to explore some of the challenges and some of the real gains we got from working with an bilateral partnership between New Zealand and Samoa. And in this case it was it was an initiative designed to adapt a mobile cessation tool so this is a text message based program to help people who want to quit smoking to give up. And it was designed and trial in New Zealand and in the UK and I found good evidence of effects there was a doubling of quit rates in these two other jurisdictions. So we're curious to see what it would help would perform so to speak if we adapted it for Samoa. Would it help people quit. Would it reduce in terms of their overall health targets would it reduce tobacco use, but also we're really interested in the process what what happens when you adapt a tool that's been developed for a high income setting where there's an established tobacco control infrastructure and program and adapted for an environment that is very different in terms of both culture and teams of infrastructure and and priority in terms of tobacco. So I just thought this was useful to demonstrate a little bit so the program was funded we wanted an MC session program tailored for an SS Samoa and we also started to work with American Samoa. We wanted to as part of this process rather than just take this tool and translate it linguistically, we wanted to establish a governance network and advisory group people who would work with us to advise us on all stages of that adaptation process. And importantly one of the outputs we want to see how much this would cost. We wanted to be able to be confident when we sat down with the Ministry of Health that we had very clear evidence that this would be as effective as potentially other interventions to reduce tobacco. So not prioritizing in health because it's new and exciting, but actually because it's evidence based, and it actually supports the whole system as well as just supporting people to quit. In terms of outcomes we wanted to describe this process so what we went through to adapt this tool, because we thought that would be really useful for other settings, other initiatives, whether it was other SMS based programs. And we've published a work which I can share later on that so that was a very, I have to say quite a time consuming process, very fastidiously undertaken in partnership with the Ministry of Health and Samoa, the WHO and civil society groups. So we adapted linguistically, culturally and in terms of references for the local setting. We also wanted to, at the same time, strengthen capacity and local leadership in MC Sation because we didn't obviously want to rely on our team, obviously working with colleagues in Samoa, but it was important to build knowledge around how to adapt and run these programs both within the Ministry of Health, but also in the local telcos. So the Blue Sky was their partner organization. We wanted to work with them so they knew how to and knew the strategies and the troubleshooting requirements, the technical requirements for running a mobile based program. We also, as a broader long term outcome, we wanted to make progress towards the country's health sector plan, which specified a reduction in tobacco use in line with WHO targets around reducing NCDs. And we wanted to enhance, and this was a requirement of the funding. So I know about, and it's been discussed in another publication, I guess the sort of challenges, we had a high level agreement to enhance service delivery partnerships between New Zealand, the United States, which also contributed some funding and Pacific. I'm showing this to say that both on the kind of ground level, there was a focus on adapting a tool that we knew worked elsewhere, testing whether we're working in Samoa, but alongside making sure that there was a broader focus on strengthening capacity and capability to enable this program to be scaled up if it was deemed to be useful at a population level. And to ensure that there was ongoing skills within the Ministry of Health to maintain the program. So this is the program, was it successful? So just very briefly, there was a 39% reported seven day abstinence at one month follow up. And it's a very short follow up. We were able to follow up over a longer period of time. But these results were consistent with what we found in New Zealand and how and this program also found when it was delivered in the UK. So we're pretty confident and other evidence suggests that people enjoyed receiving the messages in terms of providing the valuable support. It was respectful. And it was one of the factors they felt may help them quit. So in terms of objective outputs, we had a program that was tailored, it showed an effect. In terms of systems level value, we remain, I guess, had some questions about how useful this was over the longer term. We really learned a lot in this process. So one of the things we really felt very clear about when we evaluated this process, there's some real practical realities of working in these environments. So these are small islands with low resource. So ensuring that the stakeholders were identified and meaningfully involved. Now that sounds very clear, but that was very difficult. Even though we've been working in Samoa and had good partnerships and relationships within the Ministry of Health and across other sectors, identifying the key people to be involved. That would ensure that the program had credibility and integrity beyond its trial was really important. We're difficulty in terms of international funding and you can see in terms of the work plan that we had some very high level requirements about outcomes that were perhaps misaligned with the priorities that Samoa would have set. So I think it was really important to align those priorities. And of course we had difficulty with the funding, the time the funding took to be transferred into the Ministry of Health and Samoa was very lengthy and obviously in that case quite costly difficulties and some challenges around employing local staff and this isn't from an internal challenges. This is basically the requirements for international funding dictating how these staff would be employed and under what conditions etc. Again building local technology expertise was important absolutely from the beginning because when the program was running and people would stop receiving messages we needed to have prompt response and expertise to be able to respond to that. We had to spend time again in terms of working very closely and the New Zealand team didn't play a particularly strong, you know, large role in this. Samoan colleagues ran the adaptation process but really important to get the language, the nuances of the tone, the formality of the language and any imagery accepted and tested. So this isn't just tested by the end user who may be the smoker but acceptable at a Ministry of Health level according to their policies. So they'll be comfortable in sending that out. And then importantly but really difficult we realised in terms of measuring and monitoring how well this program was integrated into the health system. So ideally what we wanted to be able to do was have a line in the health budget particularly around maybe NCD prevention. That included him cessation if the evidence was strong enough and was considered cost effective. That was more time consuming and needed to be started right from the concept rather than has the evidence we see it can work. Now let's talk about integration. Again further lessons we as I mentioned earlier that access to mobile isn't absolute or seamless or necessarily equitable. Phones are shared a lot and low income or low resource settings and not only because due to low resources it's just a cultural practice to share phones and share messages. And there's actually some value in that that we could explore in terms of using different types of modalities. Perhaps looking at social networking sites to support behaviour change rather than the very didactic one person to one message to one phone method. We understood the real the importance of really strong governance systems and local commitment right at the very beginning if we think about scaling and sustainability and how that could contribute to broader benefits around universal health coverage. So how did supporting people quit to quit smoking support other objectives that the health sector was working towards and rather than seeing it as an adjunct. An interesting one at the time it was relatively novel but actually if it doesn't add value across the border system when you really need to question its value to begin with. I think equity in terms of benefits like Nile talked about is incredibly important so you're not exacerbating inequities and this can only be achieved from my perspective I guess through conscious intent it just doesn't happen on its own. You have to really think about how you make sure those that support is reaching the people who may benefit most. That comes right from concept and engagement with end users where they may be in design and evaluation so you really are carefully measuring who's benefit and how benefiting and how do we know. I think it's also really important to look at these tools that are already existing existing around monitoring and evaluation we know there's the digital health guidelines have been released and there's monitoring and evaluation guidelines. So there's a lot of really useful tools. Some more user friendly in terms of adapting in a local context and others. But they're becoming they're increasingly available and and I think will provide really good support and guidance. We also need to know realize that evaluating evaluation tools need to be really pragmatic. So RCTs are not necessarily the most useful tool to determine whether this program supported people to quit. We're pretty sure based on other evidence that we don't need another RCT. What we needed was to understand that process at what stage we've done something differently to ensure that that value of this initiative could have been better embedded into the health system and supported it. We need to know and have these evaluation tools reflect context and these context are very specific. And I think it's also important to establish realistic locally and regionally relevant goals and this is particularly important for small and developing states who are collectively working towards outcomes. The Pacific certainly worked towards the regional goal of tobacco free Pacific 2025 as well as country level goals of reducing NCDs. So I show this really a slide as an indication of the representation I guess of the activity that's going on and under the ban of the sustainable development goals around securing and developing these partnerships of small island developing states. Whether it's in relation to fisheries in terms of blue ocean resources in terms of land use or health. I think this value and exploring what these partnerships actually look like. And I guess in summary for digital interventions. I think these partnerships might at have a particular place rather than digital interventions or in health interventions being established in an ad hoc way and individual countries across the region that there is a consideration of sustainable financing across the region. For these initiatives that there's a collective interest in building capacity both human and institutional knowledge around digital interventions. And that there's an environment and enabling environment to that fosters new partnerships. These partnerships need to be inclusive in terms of who can be involved, but also importantly include people that have important have expertise. That's relevant both now but also looking into the future about what might be needed to deliver, adapt and potentially scale up useful practical digital technologies for health. I think trust is really important, particularly in small island developing states. These are very close communities. There's a lot of collaboration. There's sometimes competition, but trust is really important so information can be really shared, both when things work and when they don't work for mobile health. Again, legal institutional governance structures are really important around regulating and developing platforms and governance structures for mobile health. And then feeding back in results of studies and trials and valuations back into these systems so without strengthening and growing. I think we need to invest in monitoring internal monitoring of these partnerships and what impact they have where they are adding value. It's a challenge across the Pacific region is around some practical measures that are consistent, practical across the region for monitoring and valuation frameworks to assess progress. Not just in partnerships for digital health, but actually on the programs and the interventions themselves. We also need to increase access to data for knowledge transfer and again that goes back to feeding back into the system. Evidence from trials that have been undertaken evidence from other forms of evaluation that help build a broader comprehensive understanding about the value of mobile technologies for achieving universal health coverage in the region. And with that, I'd like to thank you all and acknowledge my colleagues and our partners in the Pacific region. Thanks. Great. Thank you so much. That was a fantastic presentation. So we've got our first question. Can I have a display on your screen. Yep. Okay. Thanks, Claudia. So your question is you're interested in the point about social factors and social media. People often believe that their friends say more than what their doctors, teachers and the government say. So you're asking whether I have any thoughts on how public health can best use utilize digital social networks and influence in a way that is solicited. Claudia, facilitating and not manipulating. I might read the rest of that. So just to answer the first part of your question, I think it's really, I think, I mean, it's an issue that we've been discussing a lot and the work that we're developing in the future. I think social networks are really important and we just had some formal discussion in this room about for about these of what's happened and particularly in the Pacific Facebook for not sharing information and connecting families in the Pacific diaspora population across the Pacific region and Australia and New Zealand, but potentially supporting whether it's behavior change or I could use or look at social change. So I think there's really important opportunity in that in the sense that these mechanisms are organically developed. We're not imposing or kind of using sometimes not always the most trusted or credible sources of information in some ways. And so I think there's a scope to to develop some work and seeing how we can work with those platforms. There are some risks and certainly in the context right now around concerns about privacy and and use of data. These studies we've just we're finding out are very difficult to undertake and get funded. So I guess once the question I think it's it would be its potential scope. I think it's really good to use what people prefer you sending text messages to people about behavior, individual behavior change in the Pacific is not it's culturally slightly misaligned. People engage collectively they think about their health and a collective more collectivist way rather than personal individual behavior change. And I think these platforms and these mechanisms are better suited to that. I think it would be really good to learn out for certainly for I'm interested in learning about social network analysis to a better understand how these messages are being transmitted and moving through communities how resilient they are and whether they do shift perceptions collectively and and behaviors so. But I think you're I think if I'm getting your question correctly I think it's it's a really sensitive area I think it's really important to be really cautious around how we develop that not to to break trust in these really valued networks Facebook is absolutely critical to Pacific Islanders who connect back to their own countries. Okay I think we've just got time for one more question and there's one popped up on the chat. Okay so the question from is what in your experience are the best practices we can learn from that can help shift interventions from piloting phase to being institutionalized appropriated and sustained by local governments. And this is a great question. So I certainly know what we wouldn't do. And that number one would be I think one of the other way one of the best practices is is when an initiative has come from the country itself we're working with the Cook Islands at the moment who approached us the Ministry of Health has approached our team based on the work we've been doing with the Ministry of Health and some are wanting to collaborate with us to develop a mobile and in cessation program for the Cook Islands. So we've only very early early stage on this initiative but my sense is the beginning is good. So this isn't us coming to a country going we've got a great idea for you guys and it's going to look a bit like this what do you reckon. And something an initiative that comes with funding and is at this stage with some all was kind of new and looked like it would have an effect and everyone suppose he has a phone. I like this the beginnings of this project so the initiative has come from within the country or the key institution who will be rolling it out implementing it longer term. That's really important. And I think again as we progress throughout the year we we will probably spend less of our resource. We've got a relatively small budget which is absolutely fine. Less resource precisely adapting every word and every nuance, but more resource making sure that it is going to have the mechanism for distribution. Once it's it's tested thinking about that right from the beginning. Okay, well thank you very much, Jude and Nile and thank you everyone for your questions and your participation. So we're going to finish up now is it just had an hour's long e-seminar and we'll be sending out an email to everybody who attended which will contain a certificate of attendance and a link to a survey as well so if you could give us some feedback on how you found the e-seminar then we can make some changes for the next one to meet everyone's needs. But thanks everyone for attending and we'll be sending out if you are a member of the health informatics forum will be sending out a notification of the next e-seminar and the next couple of days. So look out for that and you can register for the next seminar. Okay, thanks everyone.