 Right, so I'm very happy to introduce Simon, who's going to be speaking about humanitarian systems and humanitarian aid and, yeah, so, yeah, thank you very much, take it away. Thank you. Hello everyone. So towards the end of last year, a data engineer from Texas, a scientist living in Seattle and a consultant from Mexico, all found themselves meeting for the very first time in Geneva to give a presentation about a tool that they'd been working on for the humanitarian community. Now in this story, I'm actually the consultant from Mexico. I don't look or sound very Mexican, but if you swing by workshop two a bit later on, I can talk to you a little bit about that when I'm making some tacos. But anyway, onwards with the talk. So my name's Simon Bedford. This is the first time I've ever been to an EMF. And when I first came across the concept and the website, I was just so fascinated and wanted to ensure my place here that I put in a couple of proposals for talks and workshops just in case I wasn't able to buy a ticket. Scarily both of them were then accepted, so I had to spend the past few weeks trying to make sure that these were ready to go. And today I am here nominally to speak to you about using machine learning for humanitarian aims. Now this isn't really going to be a very technical talk in the sense that it assumes no knowledge whatsoever of machine learning. What I really want to do is explain a bit about how I got involved in this, paint the picture with regards to what the problem was that we were trying to solve. And talk a little bit about some of the techniques that we then used to create this platform. And painting just how computers and how kind of particular type of algorithms can be extremely beneficial to a certain problem set, I guess. And I also want to just, before I get started, share a couple of reflections I think I had from the course of doing this project. This was something that I did as a volunteer, so in my spare time, over about six or nine months last year. So I'm by no means a professional. I'm definitely not qualified to give a really technical talk about machine learning. But I hope to be able to share an interesting story with you guys. So the very first thing I think I'm probably preaching to the choir here at EMF, but really, all of this started for me as just wanting to learn something new. Probably as a product of having always had jobs that seem to mainly consist of answering emails and creating PowerPoint presentations. There's something really nice at the end of the day about going home and having a personal project to work on. And something that can actually give you a kind of tangible feeling. You can create something. You can produce something. And you can point at it and you can feel like you've actually done something. So about 10 years ago, I had all these ideas of things I wanted to create. And I just kept on saying, no, there's absolutely no way I could do that. I don't know how to do that. It all sounds too hard. But actually, once I chose something and I chose an idea and I knew what it was I wanted to build and just started figuring out how to do that, everything became a little bit easier. So I started getting into computer programming a bit, playing around at the end of the day and on weekends. And then after a while, I started to find myself really jealous of all the people who are working on really awesome sounding things like self-driving cars and autonomous robots and everything like that. But once again, I just kind of treated this as a learning problem. So what should I learn next? What should I try and get into next? To at least feel like or at least experience some of these different elements and start to get involved. So I started getting into data science. And again, this was a learning experience. I took a course. But really what made the difference for me was producing my first proper machine learning driven system and a deep learning system that was used for the very useful problem of classifying pictures of food. That would probably be a pretty good subject for a talk separately. But after a while, I was looking for something else to do. And I knew I couldn't do this, or I felt I couldn't do this professionally. And that's when I came across an organization called Data for Democracy. Has anybody heard of Data for Democracy? Just raise your hand if you have. Okay. So I came across this group of people, I guess, at the end of 2017, which had been a pretty difficult year, I would say, politically speaking. And my Twitter feed seemed to be just full of angry people getting angry at each other and a lot of despair. And then this idea came into being at the end of 2017, which was really just an idea for technical people coming together to collaborate on things remotely and try and solve some problems. So this could be computer scientists, designers, software engineers, data scientists, just a whole range of people who just, I think, all felt the same way that they really wanted to be getting stuck in and solving problems. And they just wanted an outlet for this. So sometimes, I guess, as an individual, it's a bit hard to know where to start or you can't quite get traction with organizations. And the beauty of being part of a bigger organization is that you're able to collaborate with people and people know people and it's much easier to start to get some movement. So when I say it came into being, it really did come into being. It wasn't a case of we are formally launching this big organization. It was just a group of people chatting to each other over Slack and starting to contribute to things and working on code on GitHub. So I signed up because it sounded interesting. And I think the second thing I learned from the whole experience was often you just need to kind of jump in, even if it's something that scares you. So when I signed up, I was encouraged to join a project, which I did. But again, there was always that niggling feeling of, what can I do here? There's all these professionals, a lot of these people are software engineers in their day job, they must know what they're doing. I'll probably just end up breaking something. It'll take them more time to fix it. So there's always that self-doubt and not really wanting to push yourself because maybe you feel like you won't be able to make a difference. And I think if there's one thing I've learned from this, it's just absolutely worth just jumping in and getting started. At the end of the day, the project that I got involved in as an early participant, I never believed it would get as far as it did. And I certainly never ever believed it would end up creating a tool that is actually used by a humanitarian organization to try and improve their efficiency and effectiveness. And on that point, and just before I get to the main body of the talk, if you are looking for a project, if you are looking for something to work on, if you feel like you need that tangible problem or you want to be contributing more, the nonprofit sector is awash with opportunities. There are so many organizations that just in the course of this, I've spoken to where they are desperate for help, to be honest. A lot of them, you know, a lot of the problems I've helped to work on are just teams of people trying to process vast quantities of information or just trying to get a lot of menial work done. And with a couple of hours of throwing together some pretty awful code, you can actually end up saving them weeks of work. So absolutely keep an eye out here. I think if it's somewhere you're interested in doing something, definitely get involved. Okay, so what was the basis for the project? Well, every year tens of millions of people are forced to leave their houses, evacuated from their homes around the world due to things like floods, hurricanes, fires, war, other types of violence. Now, some of these people will actually end up making their way across an international border and they then become technically known as refugees. But there's a whole other group of people who they are no longer in their homes but they're still within their country of origin. And these people are called what are known as internally displaced persons or IDPs. And I know it sounds like really it's the same problem but actually there is an important difference between the two groups because refugees have a certain recognised standing under the United Nations conventions and they have access to certain types of aid and they have access to certain types of assistance which isn't to say that their lives are easy but there are opportunities out there and people working to try and help them get back to their homes and deal with the problem. Whereas IDPs can often end up be stuck in a situation without much hope because particularly if their country of origin is undergoing severe political turmoil it's very unlikely that the government itself will be doing anything to help them. So this is very much about a group of people who are often not very well-known in the media they're not very talked about. Refugees is a much more common term. But there is an organisation based in Geneva which is actually part of the Norwegian Refugee Council and it's called the Internal Displacement Monitoring Centre. And their specific goal and their specific task is to produce really high quality, believable data and analysis about this problem, about these people. The idea being that if you can actually get numbers out there and you know that they're trustworthy you can take these to governments, you can take these to other bodies and really hope to try and raise the profile of this group of people and inform future policy changes, investment, etc. However, getting really good data on the problem is hard because ultimately they have a team of human analysts whose job it is to try and figure out how many people are being impacted around the world. And one of their key sources of information is newspaper articles. Now you can imagine around the world if you take all of the local news sources plus international news sources and all of the hundreds of thousands of articles being produced all the time, these are basically, this is a team that is overwhelmed with information. And at the moment, or before we started this project, it was really just down to these human analysts to be clicking on these articles, trying to skim through them to see which ones might actually be relevant and extract information from them. But the good thing about the IDMC is they're a very innovative organization, so they came up with the idea of knowing that they needed a tool and they knew exactly what they wanted that tool to do. And really what they were looking for was an automated solution to make their analysts' lives easier. They could take in an article, decide whether it's relevant to the problem at hand, decide whether it's about a natural disaster or conflict or other some type of driver behind the displacement, as well as try and summarize from the article, what is it that happened? Where did this happen? How many people were impacted? And when did this happen? And the other innovative thing that they did was rather than just try and build this themselves, they decided to try and crowdsource for a solution. So they really opened this up just to the broader community. They did this via a part of the UN called Unite Ideas, which puts out competitions that people can participate in and submit solutions to. And this was one of the problems that they put out for, which was to build this platform which would be capable of performing the steps I previously mentioned in an automated fashion. So where does machine learning fit into this? Well, I want to start actually with a simpler approach that you might take to the problem. So let's, for instance, imagine that we're faced with the problem of, given a newspaper article, how do we know whether it's relevant or not? One of the first things we could do is maybe think about particular types of words that we should be looking for in the article. So hurricanes are obviously a big natural event that can cause a lot of people to be evacuated from their homes. So we could look for the presence of the word hurricane in the text. But when you start to think about it, you realize, okay, but there's a lot of different ways people might be referring to these types of storms. So your list of vocabulary that you're looking for starts to grow. But then you might just be picking up really interesting meteorological articles and they might be saying nothing about people being displaced. So you also have to start thinking about, okay, we need this article not just to be mentioning terms about storms and hurricanes and the like, but also about people being evacuated. And then when you start digging into evacuated, you realize that the English language or any language is difficult. And there are hundreds of ways of saying exactly the same thing. And you end up looking for references to people being left homeless or even people just being affected. One additional step you could take is to look at maybe the grammar of some of the sentences. So you could say, all right, it's not just enough to look for the presence of words, but you might also want to look for how the words are connected to each other. And if you have people and you know that within a sentence it's talking about those people being left homeless or being evacuated, then that sentence is more likely to be relevant. But the trouble is, the more you study articles, the more that you study the way that they try and do this, you realize that if you were trying to come up with a complete rule book that you could just apply in this way for finding out for deciding if a article is relevant or not, you're gonna drive yourself crazy. So this is where we turn to a field called machine learning. And to paraphrase a definition of machine learning, really what we're looking for is computers being able to learn from examples without us having to very explicitly give them all the rules ourselves. So once again, just to illustrate this with one of the problems that we were faced with and going back to this idea of choosing if an article is relevant or not. The very first place we need to start if we're going to apply a machine learning approach is we need to start with some representative example articles. And what we then do is we get a very nice person to sit down and classify them. So this person has to manually read through them and they decide if they're relevant or not. In this case, the very nice person was my wife who helped me do this. And then once we have these examples that somebody has already classified, we need to figure out how we're gonna show this example to the computer. So one thing you could say is all right, I will just feed in the words in a numerical format. But actually with a lot of machine learning algorithms, the underlying article is not necessarily the most useful format to use. What we want to try and do is we want to try and summarize this article in a number of different ways. We want to try and extract certain types of summaries or representations of this article that we can then use when we show it to the computer. Now these representations or summaries are technically called features. This is often more of an art form than a science in that it can take a lot of, a really detailed knowledge of the problem at hand and then also sometimes some trial and error to figure out what's going to work best. But just to give you an example and a really oversimplified one, what we could for instance do is we could say right, one of the features we could look for in the article is the presence of the word people. And another feature we could look for is the presence of the word evacuated. And if both of those words are present, each get a one because computers really work with numbers rather than words. But then we could also not just look to see if they're present, we could count the number of times that they occur. So another feature that we could use is the count of each of those words. So for instance, that might be people occurring six times and the word evacuated occurring four times. And finally we could also maybe count the presence of both of the words as part of a common phrase so where we know it's talking about people being evacuated. And there are many, many different ways and approaches we could do this. This is really just a simplification. But once we have these pre-human tagged examples and we've converted them into these representations, we have algorithms where we can feed this in and the computer will on its own figure out based on what the examples are, what the best rules seem to be that it can use for the classification. So for example, without oversimplified features what it might decide is that if both people and evacuated occur and the word people occurs more than four times and the word evacuated occurs more than twice and the phrase itself occurs more than once, then we will classify that article as being relevant. And if we've done a really good job with this and if we've chosen the features well and if we've chosen a good algorithm, then hopefully what this should be able to do is take in totally new unseen articles that the computer and the algorithm have never seen before. Once again, we would need to convert them into the same representation. But we should then be able to feed these back into the same computer with its now constructed internal set of rules that it will use. And with a pretty good degree of accuracy it should be able to help us to automatically classify articles like this. So this is a very simplified summary of what machine learning is, but really it's the fundamental underpinning of what we did to build this tool and solve this problem. And going back to what it was that the organization was looking for for deciding whether an article was relevant, we really just relied on machine and machine learning algorithm to do that. And then for figuring out what sorts of displacement it was, whether it was a disaster, a conflict or something else, again, we used a machine learning algorithm for that. In the case of the information extraction, this is actually quite a different type of problem. And in the end what we found worked better was some combination of a few artisanal rules that we put together based on some particular words and grammar and sentence structures. But in combination with some machine learning algorithms we were actually able to further increase the effectiveness of that particular approach. Now one of the good things about working with the open source community is that you don't have to start from scratch. So there are amazing libraries out there that do a lot of the heavy lifting for you. And really this is just a brief summary of kind of the overall environment and type of libraries that we were using. But a lot of these are amazing. So Spacey, for instance, just is pre-trained on hundreds of millions of different types of words and articles and things. And it just gives you a lot of power and ability to work with text. And it makes the job of having to put together this sort of tool so much easier. Because if you had to rely on paid solutions that really wouldn't be an option for the non-profit community. So, oh and probably the two most important tools that I should mention were Slack and GitHub. Because as I said, this was a truly collaborative effort. This was people working on this problem as part of a team around the world in different time zones. We'd never met until we actually went to present the solution. At one point we almost had 24 hour coverage because we had somebody in the UK, somebody in America, somebody in Australia. And it almost felt like as I was going to bed, the chap in Australia was getting up and carrying on with the project. And if we hadn't been able to have these sorts of tools like Slack and GitHub to collaborate, it would have been impossible to interchange messages between us. As a result of this, this was, as I said, a competition. So we submitted our solution and then to our massive surprise, two months later they told us that our solution had won and that they were planning on implementing this for their team in Geneva. Which is when they invited us all to fly out and present it. This is where we met each other for the first time. We got to actually speak to the analysts and get a far better understanding of their day-to-day work that helped us to further tweak the tool and make some changes which made it even better for them and easier for them to use. But probably the most satisfying thing is just knowing that this side project that I started working on because I really just wanted to learn something new and learn something more, ended up helping to address a real problem and helping to ensure that a non-profit team can be more effective and focus their efforts more appropriately. So just to close, I just wanted to summarize a couple of what I think were the key points of success of this. The first one was having defined goals. I think if the organization had said, we want to build a tool, come up with ideas, it would have been a lot harder for people to put something together. Whereas just the very fact that they really knew what it was that they wanted this tool to do, meant that when we stepped forward to try and put together a solution, we actually had something to work towards. The second piece I think which is absolutely vital for any machine learning project is a real understanding of the problem, what is often called domain knowledge or domain expertise. But really, this is just actually having access to somebody who really knows what is at the core of the problem, who really understands what the data is, understands what it means, understands the nuances, understands how it's collected, understands how it can be wrong. Because if you don't have that understanding, then whatever you can build from an algorithmic perspective, whatever you can build from a machine learning perspective will just amplify any errors that you put in. And I think the final thing was just the approach of collaborating with the volunteer and open source community. So again, this organization, they have a budget and they have people who donate money to them and they could well have actually just gone to a private contractor and said, okay, build me this. But I think their approach was really interesting because not only did it end up with them getting a solution, but I think it also just helps to engage people because when you put these sorts of competitions and problems out there, people get excited, people work on them and they become more engaged. And then finally, again, on a personal note, I think once again, I'm pretty sure I'm preaching to the choir, but if you're ever on the fence, just do something. If you wanna learn something new, just choose whatever and get stuck in and start working on it. The first attempt, and normally we'll drive you crazy, but after a while, I think it's the best way to make progress. And obviously have fun. Don't waste your time trying to hack away in your spare time at things that you absolutely hate, but try and enjoy what you do as well. Anyway, so that's my talk done. If there are any quick questions, I'd be happy to answer them. Or if you want to just grab me at any other point after the talk, I'd also be happy to talk more. Hi, I was wondering what you did if you used articles from different languages. And if you did that, how you dealt with that? Because I find that a lot of machine learning models work really well with English, but not necessarily with other languages. Yeah, we were lucky in that when we were asked to put this together, it was constrained to English. Just really as a proof of concept, now that it has been implemented, one of the next steps that the team is actually looking to do is expand this to be applicable to other languages. Obviously, as you say, it's a very different problem. You need to retrain the models. I think from what I've seen, so for instance, with Spacey, I think that their language sets do become more advanced as time progresses. But yes, I don't think there's quite as much out there for other languages as there are for English. But fortunately, we didn't have to solve that problem. Have you at all tried using formal ontologies or any kind of knowledge representation way of capturing the knowledge independently of the words? We didn't get into anything quite that complicated at the time. I think what we did, we came across all of these really fascinating looking papers that looked like they had ideas that could be applicable and got overexcited sharing these between us. I think at the end of the day, we just had to be pragmatic in this situation. As we were all working effectively in our spare time and just trying to come up with a solution that we could submit to this competition, which was ultimately based on some pretty simple machine learning algorithms, then that was good enough at the time. One of the things we did build into the solution was the ability to switch out different backends from a machine learning perspective. At the moment, it had these simple models that we put together, I guess is the version 0.1, recognizing that both as time went on and the analysts themselves would be making corrections to some of the predictions that the algorithms made, but also as people did more work on this, more sophisticated and better models could be produced. So we built in the ability to just easily switch between models so that that piece can constantly be refined and upgraded. I was just wondering if you'd done anything in terms of sentiment analysis on the text to understand where the author was coming from when they wrote the paper. For example, if you've got something from Fox News, it's gonna have a very different story about refugees to something in the Guardian and that's what I did. Yeah, again, it was one of those things that we played around with and we definitely discussed that as a possibility. One of the, just in general, when I can envisage a lot of questions about very different machine learning approaches that you can try for natural language processing, one of the challenges we did face was just a lack of data as well for any particular approach. So we were given about a couple of hundred articles as the input data set and there were some approaches that would clearly have needed more data and even the data that we were given was still pretty messy. So we also had to spend a decent amount of time just trying to clean up the data that we were given. I think, again, there's clearly a lot of opportunities in here for a lot of things you could do. We didn't unfortunately get around to any of that. Did the algorithms go on to learn from themselves from future learning and if so, how did you think about managing learning from bot-created content? So no, this is really a tool not to replace humans but to support them. The idea being that we want to take advantage of both worlds and they weren't looking for something to replace the analysts. They were looking for something to make them more productive. So the way this was implemented was they get 100,000 URLs coming in every single day. The tool processes these and assigns it basically a score as to whether it thinks it's relevant, what country it thinks it's about, what it thinks happened. And then on a daily basis, you will have, for instance, maybe an analyst with responsibility for certain countries in Africa. And they may have identified a possible event from another source and they're trying to now narrow in on what actually happened, how many people were impacted. They choose what it is that they look at, what this tool is giving them is the ability to filter out what is probably a lot of irrelevant information and then they can focus on looking at that information and deciding if it's something that's from a trustworthy enough source that they will use because ultimately they remain responsible at the moment for making a decision about how many people they think were impacted by that event. What we did build in was again the ability for them to give feedback. So they could say, oh, you classified this article as being about conflict but really it's about violence or you classified this article as being relevant but it's not. And they can give that feedback and that human feedback can be used to continue to refine the algorithm but now there was no intention of having the machine learn for itself as it were at the moment. Yeah, so I have a question about the domain knowledge and defining problems but not in the case of trying to learn a new skill but more retraining. So you mentioned the IDMC and indeed in Geneva where I live there's lots of these opportunities for non-profit interaction. Also it's completely awash with data scientists that a lot of which are out of work and looking for new ways of interacting with these things but a lot of them have a lot of domain specific knowledge of how to apply machine learning algorithms and data inference and things like this which they just don't understand how this applies to something else. When I work with bankers who also work in futures and things like this they don't understand how we do this in particle physics and vice versa. How to get people from one way to apply their knowledge, their expertise to problems that they're good at they just don't know how they're good at them. Yeah, I think it's a great point. I mean in this case really the driving force was the IDMC themselves and they have a data team they have a very energetic data scientist on that team. I think him and that team in general they were very broad thinking about knowing that they could get this whether or not it would be from the exact people with the exact right expertise. I think they're just very willing to engage with people knowing that what as an organization they can provide the data domain expertise as long as there's a way to collaborate. So I don't think it's a problem with an easy answer. I mean ultimately the people who have the problems I think one of the things that can help and has helped is giving talks like these so one of my other collaborators he's given several talks both in the US and in forums in England. So just a broader group of people from the humanitarian sector and just introducing the types of things that data science and machine learning can do putting it out there so that the people who have the problems can be more aware of the sorts of skill sets that they could insource for that. Yeah absolutely. I think I don't unfortunately have my own organization yet but I think yeah there's a lot of things that you could do there and I think they just need to know a lot of this is really new to people. I think we even, my colleague gave a talk and everything was just new to people. So I think it's just getting the word out and getting people to understand what the options are both for skills and collaboration and how to get these sorts of things done. Okay no that's great thank you very much. Yeah for time if anyone's got any more questions please yeah just outside but otherwise thank you very much indeed. Thank you.