 Africa welcome so lovely to see you very nice to see you too Africa you're our first our first keynote speaker of this conference so you have the responsibility of the big opening and breaking the ice so we're looking forward to listening to you all yours I will do my best okay well thank you thank you organizers okay well thank you Elena so it is really a pleasure to to be here today so so indeed I mean the last tech conference I participated before pandemic it was big things so I have very very very good memories of that so hopefully next year we can see each other again so okay the subject of my of my talk is about Ben Shiai okay so Ben Shiai is a non-profit organization funded by the Bill and Melinda Gates Foundation as you said to to bring the latest AI technologies to the most underserved communities in the world so let me let me talk a bit about the about the name okay so Ben Shiai is um is a Japanese word it's a Japanese profession that doesn't exist anymore so it's um it's a profession that when the cinemas uh when the films didn't have a sound they were only with subtitles so these uh the Ben Shiai were the people who was explaining what was happening to the film to those who didn't know how to how to to read okay so this is what we want to want to do okay so we want to provide our machine learning platform our algorithms bring those technologies to read what the data say in those uh in this sector so and also I mean most of the of the leadership and and members are coming from Japan working there before so it's a it's a piece of our roots so so yes as I said um our focus our vision is to to reduce um um health inequalities with AI technologies so more concretely or specifically is um to provide real-time and just in time personalized incentives and recommendations to frontline health workers and patients so basically provide adaptive interventions on how we are going to do this let me let me put some examples and let me start with um um with some background about the usage of um of mobile and m-health applications in low and middle income countries so the um when we are talking about um about the utilization about data about machine learning in low and middle income countries settings so the first word that appeared is mobile okay so mobile is used for everything so for instance just one example so the the payment apps and were not invented in us or in china it was the first one was in kenya okay so the what is happening um in low and middle income countries about this is is big okay so there are many apps the first from the apps that are running in high income countries so like for instance in terms of that they're able to run um without internet connection and also they adapt to latency of data okay so um and then I mean because of this landscape because of these situations the number of mobile health apps is is very big it's been bigger than high income countries so and it's used of course to to improve health so it is used to strengthen the health systems and the living for online services it is used to build capacities of online health workers and also to address behavioral change and and this is what Bench is trying to do so okay so these apps generate a massive amount of data that is not used whatsoever to to improve health outcomes and we believe that this can be a kind of half a tremendous impact so let me put some some context and some examples about the the projects we are we are currently working so and some of the of the try I mean of the global health um challenges we are trying to address or to support with technology so maternal and neo neo world mortality is the greatest disparity between low and high income countries every single day more than 800 women and 40,000 babies die from causes that are fully preventable um for reasons related to maternal care and 99 of these happen in low and middle income countries midwest safe lives okay we need to empower them so we need to to help them first they are not enough and second their training can be definitely definitely improved so so this is why I mean there are many organizations trying to address this and mobile phones and mobile apps is the is the communication channel um and to to solve this and this is why we're partnering with organizations like maternity foundation to improve um the the skills to build the capacities of midwives to to help on this on this aspect so um another sample okay so another sample is malaria okay so for instance in 2018 there were more than 40,000 400 sorry thousand deaths and it's one of the greatest global health challenges so with 1.5 million malaria cases since 2000 okay again there are many ups that they are trying to to help and to and to reach areas that they are very difficult to otherwise to to that health or work as to go so be able to survive to to provide some violence and also to um address and to support prevention guidelines so this is um this is also um there are a lot of data generated with this so normally I mean there are rapid tests to that for malaria that if we make a picture and support um the if the test is positive or negative so we can have much more information about what is the situation of if there is an outbreak of malaria and again support with recommendations and predictions when is it more likely another outbreak happens and also how to support the health and work is in the area and also the the the citizens to to take prevention measures another sample that we we work is related with HIV okay so HIV is all behavioral okay or most of it is behavioral so starting for the prevention guidelines also to adherence to the treatments to antiretrovirals once you get a positive and also how to avoid if you get a positive to to transmit the disease to other people so again so the rapid test there are ups that make a picture and also connect with the behavioral um analysis and behavioral interventions to try to improve the situation to try to improve the that the patients pick up the test when the ones they do and to help definitely with all these to the healthcare systems in the in those areas sorry another one that we are currently working is with supporting pharmacists pharmacists are crucial so pharmacists are the first point of contact and sometimes the only contact with the healthcare system with many people in the world so supporting them with with information about which medicines are available which medicines you should start ordering because are going to it's going to take time until the supply arrives okay support pharmacists to provide the the right guidance to the to the patients and also to avoid the rational use of medicine is something that we are also working again through ops that connect um pharmacists that support and help with um with the supply of crucial medicaments and medical um devices so for instance Abel Duccio um prepares a list of the of the of the of the drugs that they're very important that every pharmacy in the world or drugstore has so to assure that the essential medicaments are always um are always available for them in every area of the world is is a must and also we we have partnered with the world diabetes foundation to support full-time health workers in low-income countries to um to support to diagnosed and to take um early treatment uh with diabetes three out of four people with diabetes are now living in low-income countries so it's also very important that uh that this primary care is aware and and support them to with the right information with the right recommendation with the right intervention to to to help to with the diagnosis of this disease and for this purpose we have uh we have built a data-centric and behavioral machine learning platform for low- and middle-income countries so let me be more specific so why we say data-centric okay so of course we have realized that one of the one of the most important points and this is uh I want to say I don't want to say anything new is the quality of the data we are talking about digital data so the the quality can be can be the best okay so this is why we have developed an SDK that uh help our partners to to track and to label the data properly for machine learning purposes okay so automatically they integrate this SDK in the code of the app so automatically it's going to track as as I say the level um with machine learning purposes and then the second step is that this information is going to fit the data governance the data pipeline governance model governance that is available in our machine learning platform so automatically I mean they will fit as a features the different machine learning models that are in production behavioral machine learning models and also advanced experimentation like enforcement learning so all this with one goal with the goal of empowering frontline health workers and to provide them with real time and just in time adaptive interventions so as we saw before and to close the loop okay so what the kind of recommendations the kind of predictions and interventions that we provide to the frontline health workers are different goals so the first one for instance building the mid-wise capacities so a well-trained midway is able to to save two-thirds of the current ads so supporting the primary care LA diagnosis like with what we do with world diabetes foundation or prescribe the right medication at the right time and again be able to support with other diseases epidemiological diseases like tuberculosis malaria or HIV this is just some screenshots of our platform so what we want to show you and also have one video demo included in the presentation is that I mean our vision is like of course there's a strong backhand with the SDK and all the production models and so on but we have an intuitive and actionable front end that every partner has access to their data they can download if they want to do any tailor further analysis they have full transparency about the models that are in production the curiosity the features that I introduced all all the experiments that they are running and the and the and the input they are having on the outcome of interest so trust I mean being transparent is essential for us being to so they can have full access but the idea and why we're building this is that what we want is that every single organization in lower middle income countries have access to this battery of technology both from modeling recommendation experimentation so they can start I mean of course we help them but also these allow us to scale allow us to that the organization with I mean maybe one or two data analysts they can have access to those technologies that before they never had access so this is another another analytics because we also include analytics because what we observe is is also important even if it's not the core of our platform is that analyzing the past showing the results and the campaigns the effectiveness of the campaigns they performed before is also very interesting for them so this also included okay here I have the video just in order to have a look about what we have built we are going to release a new version this month so well just again and well you have access to the summary of the results analytics of every partner has access to it how many experiments has run also the data ingestion how is the the data that is coming the models that are in production so as I mentioned before with very strong model governance so every single change in the I mean if there is a new training set for running every parameter or new feature that is included is also visualized also the accuracy of the models here we see like yes users at the individual level and also the experimentation engine where we have like clinical trials every testing we have also micronomized trials and enforcement learning in production and also you can have access to the analysis of the impact of the different interventions or not just that that every vanization have sent and also with the fact that has an individual level so personalization is key for us and I think that's all okay so so which are the data I mean which data we are we are working with okay which are the what is the information that that we track with our SDK and also we include as a external information no but it's included in our in our platform so the first and main source of information of course as I mentioned before are the logs from the apps okay so we have a lot of front-line health users from front-line health workers sorry and also sometimes also passing that introduce information and also healthcare workers introduce passion information embedded into their apps so all these logs all these records all this information is what is track okay so this is the main source of information in terms of patients health profiles that are introduced by the healthcare workers we also analyze the the trajectories of health and disease and also how the actions so different health workers or even the same health workers can have two different passions so it's not the same so we personalize that at the two levels we can say and also contextual information what I refer with this we need to put context to the recommendations we give so we also partner with public health and government and government's governmental institutions to have access to demographic information environmental information cultural religion very related as well with nutrition that can derive to different kind of diseases as well or country or some of them climate if it's rainy or dry season it's also fundamentally we want to recommend for instance that the person is sent to a clinic and also epidemiological status okay if there is an outbreak of covid of malaria it's fundamental that all this information is included as obviously the the recommendations will change so as I mentioned okay so our goal is to provide useful and actionable information to first understand past behavior okay understand how providing to our partners with this information about what happened in the past the decisions were were right and which ones were not so successful and also predict future outcomes so who is more likely that get a certification in the online learning app who is more likely that stopped using the app and the connection is going to be broken who is more likely that has complication in the in the current disease so you need to increase the the the amount of basis the frequency of the basis and all this information is to take action and not behavior so so once we know for instance that is is is likely that I mean Amidwab is predicted that is not going to get a certification but it's going to be close to get it so how we can motivate her to to be able to to achieve the goal so this is the the core of our work and as I mentioned before as well I mean the personalization is is fundamental for us so we we of course personalize at the level of user okay so every behavior that this person does and and and all how the contextual information affects to to this particular user and then moving from the user behavior to collective behavior so we it's much easier to to be able to to to understand the different profiles also we are talking about collective behavior and also when something that we do as well is that even if we focus on digital information we also perform users interviews on on site okay so it is important as well that that we are able to determine different profiles so at the end I mean we we want to check if this this analysis is correct with personalized interviews and also surveys also personalized at passion level as I mentioned before the actions that that the user does um can have a strong impact different impact on different passions and also personalized at the at the level of pharmacy clinic of point of current okay so it's not the same a pharmacy of a clinic that is in a rural area in a remote area that is if even if it's low income settings in a in a city so the kind of recommendations we also perform in terms of supply chain or availability of drugs or also the the the passions that go to that to that clinic or pharmacy are completely different so the recommendations also adapt to to this level of personalization let me explain a bit how the the work cycle okay of the of our platform works the first point is the is the SDK so as I mentioned so there's the case integrated into the code of our partners app okay so this allowed two communication the first one is like a real-time sync of the of the logs and also way back okay so not only like the data is coming the data is going so we are also able to send interventions not just not just and track the the impact of that so okay so with SDK we receive all the information about the what is what is the the actions and how is the interaction of the user with the app the second one is that automatically is going to be labeled and the data pipeline differentiating between metrics kpis traits and features okay when this is just the individual time series behavior kpis aggregated traits are different characteristics of different behaviors of users and the features is what is going to be included as some as a feature of the machine learning models and then this information that is organized and classified in a very rigorous manner is going to fit the the model management okay and here is where we have a we have a product that organize and keep track of every single change that mode I mean that we do in the mobile that they are in production that we put in production if we are running the model with a longer historical data this is also being controlled okay or if we are changing a parameter of course if we change the algorithm both from algorithmic side and from data side this is going to be tracked into the model management product and we focus on three main kind of models three blocks we can say the first one is the user behavioral predation this means that we are going to we are going to predict at individual level who is more likely that gets the certification or not who is more likely that is stopped using the app who is more likely that is going to have a complication with a with a pregnancy or with the the current disease so these are the kind of user behavioral predictions that that we perform for every single user and then for instance once we know that for instance a person is going to stop using the app or a person is going to to ask for a particular drag in the e-commerce app so we can recommend what additionally we can I mean can be content can be products we we can we can provide to the to the user to to take a better to improve our healthcare so in the case for instance that what is the what is the right information we need to send them I mean in order that that does a better diagnosis for instance and then forecasting okay forecasting is also very important piece first to to forecast the contextual information contextual information sometimes is not updated okay so it's difficult to keep track of all these demographic environmental information so we also perform projections and forecasts to to have more realistic information for the contextual data and also in terms of forecasting of course we include um trajectories of health and disease but also the supply chain supply chain is important that with the limited resources that they have so it's very important that they don't spend too much time making plans of purchases and so on in a very dark and hidden supply and change system that sometimes is very difficult for them what is going to be available and what is not so helping in optimizing the the supply chain helping in in keep things easy and and send reminders about certain materials or medicaments that needs to be purchased in advance is also very important so with all this information okay and and again I mean our goal is to to not behavior to take action it's where we go to the notice service in the notice service we choose to whom we want to perform an action and the action itself which kind of of recommendation if it's machine learning base or just message if it's a post notification we want to perform to to the different people for instance if we imagine that we want to send some reminders so from the head workers that work in in Nigeria okay but you want to focus in the Lagos area only okay so so for instance this is a place where you can choose even as upset of the results that they are coming from the from the from the machine only modes so and then the the next step is the experimentation engine okay XP engine experimentation experimentation experimentation so this is this is extremely important so it's the experimentation is the only way to to to analyze casualty and is is fundamental to to understand what is the impact of our actions are having to the to the different end users okay so this is why in this in this product we provide classical techniques but also my randomized trials and reinforcement learning in a in a basic and more advanced ways to help to give recommendations on the fly and first experimentation then we're sure we can do full adoption and everything both experiment experiments and full adoptions are done through the SDK sorry we also do research in in house okay and and these are the main pieces that the main the main research areas that we focus causal inferences of course one of them as I mentioned before behavioral prediction like working with the with the most advanced behavioral prediction algorithms working with a state-of-the-art and just published algorithms trying to transform these theoretical approaches and put them in production but also in variations of the main algorithms to to be able to to adapt to to to our needs or challenges forecasting also as I mentioned reinforcement learning of course and also synthetic data generation so it's very important that we test everything carefully before creating production with with our partners but this is why we try to simulate synthetic with synthetic data actual users behavior um we have I mean we only have been alive for one year okay so that we are happy that we have two three three main publications um uh I mean I said I mean three or three papers after the publication sorry so the first one was in KDD and we just got two papers after for new reaps in the in the worst of all the first one is in the Andrew and G of data centric AI and the second one is the public health worship in new reaps and we were also very happy that uh that the one that we um that we sent to KDD and it was a word with a based reward in the healthcare worship and we are doing in-house research but we also have key collaborations with um with uh with the best teams in the world in terms of what concerns us so we work with um Harvard University with the statistical reinforcement learning lab there is a specific lab that's focused on on applying reinforcement learning for m-health applications um we work as well with the University of California Santa Cruz with them with game design elements for healthcare okay so we are coming from the from the video game industry and we also feel that we also know that um video games in terms of natural behavior in terms of motivating they are unique so also bringing those those elements to the to the healthcare work can be can be extremely impactful and the last one is the University of Tokyo that it was just recently signed agreement so and the focus of this is um is collaborated with fundamental research on reinforcement learning so at the end what we are doing is adapting algorithms that they are working extremely well in um in high-income countries but uh but what we want is that also develop our own algorithms that work specifically for our settings for low-amount income countries so that is what we are doing with um with um with the lab of machine learning of the University of Tokyo this one of the biggest if not coming I think the biggest in in Japan in terms of machine learning research and before continuing I want to just stop one second and um and talk about the team okay because because we are working restless and uh and they deserve definitely acknowledge um for for the work that we are doing so it's uh I mean part of the leadership uh we're working together with some of them since uh 2015 and and back in Japan and they moved we moved together here to Spain and um well I mean I don't want to say anything new that uh all of you know that for data science teams diversity is fundamental and um for us that we work in in in low-income countries settings is is is definitely a must okay we won't succeed if if uh if we don't have a truly diverse team so we have people from China from Singapore from Iran from Pakistan from Haiti from Nigeria from Korea and and along etc and we are really working very hard on this and and this is no trivial okay so because it requires a lot of effort from immigration's point of view and but definitely the results are better and and it's worth it okay so I want to transmit this also for everyone and also okay so now um I move to the piece of the collaboration we have just a case study with the maternity foundation maternity foundation is a non-profit organization that is the headquarters in Denmark but the focus um the work is mainly focused on Africa and also India this is one of the samples that I'm showing here so the focus of of maternity foundation is to provide with digital tools and with apps that increase the capacities of of midwives okay so don't power them so so basically it's an online learning tool okay so that they try to cover um I mean the most important skills that they need to to reinforce or to to to be fully updated so so for instance in the first figure we see the number of save delivery apps is there is the app that has been developed by maternity foundation so save delivery app users in India okay so here we see a distribution and then for instance we see like the district level poverty in a different in a different dimension so we can also see that the number of users and the number of of I mean the and the and the level of poverty in India also correlate and also when we have a look not only for the users that have to load the app but also when we we see the level of engagement that those users have with the with the save delivery app it also correlates with the areas that they're poor in in in India so some some of the some of the analysis okay just to put a couple of examples of the of the work that we are doing with maternity foundation so one of the goals that um that we work together is to increase certification okay to to get a level of knowledge and exam through a series of personalized and contextually tailored interventions okay so so the the results I'm showing on the right are predictions on learning progress among save delivery users in Ethiopia okay so so here we see different professions okay not only our midwives so other skill birth attendants that that use the app and here we can see so that is there is a a change of behavior below and above the level five of the app okay so and the ones that I mean once you pass the level five is much more likely that you're closer to get the certification so we can see the different distributions of the different of the different um professions and we can see that the students they are not so I mean so keen to continue using it but why is the big need this if we meet wives but for instance if we go to the next slide so also we can see the different distributions of progression and at the end getting the certification so again in the level five we see that um they are the ones who are less likely to progress into the into the learning app and uh and and this at the end this information will tell us is different levels of of course personal life interventions we need to perform to to in order that they continue using the app and continue learning okay and um and with having the the right level of engagement that this is the goal it's not like uh with other apps that the more they use it the better okay I know that it's something that supports the learning curve of the of the skill birth attendance so so for instance the more use the uh in case if if if you spend that these three hours using the app you is predicted that you go below the the level five so be able to provide with personalized content not only to pass the certification for those that that they're going to be close and and and it's very likely that with some uh information that they get it but also to be able to to send information that engagement content modules that that that they enjoy in order that they can also learn the ones that they enjoy less okay so be able to to recommend um personalized for every person for every circumstance and if if and it can be much more aggressive it is very likely that the person will stop using the app that is for instance it's in an area that is more likely than naturally happens or it's very close to get it and and also to finish okay so the last piece that I wanted to talk is about enforcement learning okay because enforcement learning is a fundamental piece in in our platform and uh and something that we really want to bring the state of the art um to to lower middle income countries so enforcement learning consists on an agent that learns through interaction with environment what is the best action to maximize your work for a given state okay so in our case um the agent is going to be bench a platform where the algorithm is in production the action that we perform is intervention okay with a specific um healthcare role so the state sorry I forgot so the state can be any information of any environment representation can be any any information about the user behavior or the the information about where this person lives demographic status etc and the the environment is also the the behavior of of the user the passenger contest and the reward is the user behavior and at the end passing outcomes okay so depending on if imagine if the goal is that the healthcare worker provides some information before doing diagnosis or other that includes some additional tests to the patient okay so if at the end this person did it this would be a reward if they not it's not okay they were not successful that that intervention okay in this uh in this slide I included um a slow machine picture okay because I wanted to introduce the multi arm bandits so um slow machines are sometimes for us I mean I'm not I know most of you know as one arm bandit okay so why because the the old ones had this um this arm okay that you can pull to start playing and also bandits and this is the where this name is coming is because they typically rip your money okay so um and the enforcement learning can can be represented I mean the simplest problem okay is uh is called multi arm bandits okay because it's uh it's equivalent to having a series of slow machines and uh you can decide that each time which arm you should pull to try and maximize your reward or at least to to to to minimize the losses so this is because I wanted to talk about the multi arm bandits so this is the simplest version of the reinforcement learning problem where the actions are picked by the action right Africa sorry sorry for interrupting Africa but we run out of time uh I know you started a bit later so we have to start finishing if you if you may if you can that'd be so kind thank you I can I can thanks okay so basically um let me just summarize quickly okay so um well we use uh a starting for multi arm bandits we move to contextual bandits when information is online so we have also information about the state and we use reinforcement learning okay for first personalized interventions so with other experiment type of advanced experimentation is not possible to take it into account online information again and that the systems um um adapt to learn uh even if the best action changes and um and also just put some context about the different kind of experimentations that we also have in production too so like every test in randomized control trials MRTs where everyone can be in A and B for different in different days okay to study I mean how the how the intervention can kind of how intervention can change depending on the hour that they they sent and moving to contextual bandits okay so basically multi arm bandits are a smart operation of randomized control trials or MRTs and contextual bandits even a smart operation of reinforcement of um randomized control trials or or MRTs okay and uh we are working of course with synthetic data research and also the direction we are going is towards collaborative interactive recommenders where not only based based on actions also focus on sequence of actions um yes the summary so our goal is to empower with information based on data from like health workers in non-medal income countries performance learning is extremely powerful and putting together information about analyzing past behavior predicting future behavior Africa Africa I suggest sorry for interrupting again I suggest our viewers take a picture of this summary so they will remember you don't have you have been talking and you're thirsty take a picture because we have a few questions uh we don't have time Africa to I know you start a bit late and I apologize for that we have a few questions but just a quick one before we say goodbye to you first of all thank you so much congratulations on the amazing job you've done in so little time and uh one of the questions I'm asking is just the one they say yes or no answer uh whether you are working on any uh you mentioned some of the global health challenges uh HIV malaria diabetes of obviously birth rate and mortality at birth what about COVID-19 are you working or will you be working on any of the in this yes or no and uh very quick tweet answer yes we will be working as well excellent so we have to stay tuned then okay Africa I have to thank you so much for this fascinating talk amazing job congratulations Arigato gozaimasu and we'll see you very soon africa beria yes thank you very much