 Hello, everybody. Good afternoon, and welcome to this sixth webinar on open source verification. I'm so pleased to see everybody here today. I'm always pleased to see our brilliant speakers and our brilliant audience, but particularly in this case, we've had some technical issues and it felt a bit touch and go at various stages over the morning. So just to remind everybody, this event is being recorded mae'n gweld yr amser yn gwneud o'r canol ystod yn y dyfodol. Yn amser yw Henry Ffdor Wilson. Efallai'n fawr o fe wneud am gweithio, ddim yn llwyddiolol i'r Llamadeu Samuel o'r proj gamer o ffairfyrthau ymlaen, a hynny'n gwneud i'w ddysgu'r wyf, i ddyn nhw i ddyn nhw, a i ddyn nhw i ddyn nhw i ddyn nhw. Dyma fyddech chi'n gweithio'r ddyn nhw'n gwneud i'w ddyn nhw i ddyn nhw i ddyn nhw, o'r byddiadau sydd wedi gweithio'r ffordd. Fydden nhw'n ffrindio Christine O'Varriel ar RuCie, y Fideol Serban ar y Cymru Coddau Dŷlachol, Garry Simaeth, ar RuCie'r project Sandstone a Christian Trebert Elin, gweithio'n gweithio'r ffordd. A ydw i'w wneud i'r wybodaeth y seir, ydych chi'n fwyfyrwyr cais 26 ydych chi'n ffordd, mae'r cyfnod o'r cyfrwyr ar gyfer gwir. A'r cyfrwyr yn anulwyddiadau'r ffordd. Felly, mae'n gweithio'n gweithio. Mae'n gweithio'n gweithio, mae'n gweithio'n gweithio. Ond yna'n gwneud fyddfa i'r gweithio'n gwybod bod yna'r gweithio ar gyfer gwir. ac yn siarad o'ch clywbodaeth o'r gynllun mor gennych. O bryd o'r chyncif arno, mae'r ffarairen y ffordd, a o gyfnod y ffordd o'r cyfnodau o'r siarad cyffredig, o'r cyfefion sy'n gwneud cyffredig cyflodieth gyda ddata o'r methau. Mynd i ddif iawn, y gwirioneddau cyfrifiadau. Ydyn ni'n golygu, mae sicrhais ffynifol yn gwleisio sy'n eu gweithwyr o'n cyfrifiadau mor hynod. Nid yw rhaid i chi'n go i gyd, i een ddweud o'r wneud, ond wedi gefnodd o'r awmhничio erbyn hyn sy'n gyntafír awthangos fod unrhyw am hyn yn dda, yr hyn yn grafyn i'r gweithio'r peif, deci vennydd y maen nhw a'r lei wahanol peirioneddau ar y tîl homo, mae'n gar slightest gwybod sydd yn gofynnig perthynau, ond yn gwneud hanes eu cyfwynhauum yn cerdd, rodd, mae'r methu a'r cyfwynhwys yn dyfodol, darlod o askys nesaf yn dyfodol, a'r ddaeth ar y ddaeth ymlaen nhw yn eich hwn ac, roedd ydych yn ffordd yn maen nhw'n ddaeth, mae'r ddaeth'r ddau'r ddaeth yn yn ffordd. Mae'n ddiweddar i'r ddau cyfawr o gyfer sydd y ddau'r ddaeth, y gallwn o arfer y ddau'r ddau. Ond mae'n meddwl o'r ddau'r gwybwyng o ddeddych yn ei wneud o'r ddau'r ddau'r ddau'r ddau'r ddau. Yn ei wneud, mae'n meddwl o'r ddau'r ddau'r ddau, Mi'n mynd i gael y gallu'r unrhyw o bwysig o'i perffodus eu llwysmaint iawn, i'n profiad o'r hyn o'n hysgu ac yn ddweud yn ystod yn y dyfodol i'n ddiadol. Yn gyfnod i'r amlwg, ond hyn sydd wedi bod i'n meddwl i'r eistedd freshmanτau pawb am gyfnodol. Yn gyfnod i'n meddwl i'r give arall, ond, i'n meddwl i'r meddwl i'r meddwl i'u meddl, i'r meddwl i'n meddwl i'r meddwl i'r red communalau, a capac they have networks of local communities who are able to collect information in a different sort of way and who know the local terrain in a different different sort of way. More than that it feels like there's an appetite for more community building between people engaged in open source research. So that's a really interesting idea for us at SOAS to think about and explore as well through the different webinars. There's been this sense that open source research datblygiad llydrim iawn,that the information- Taking S held r sy f dependence on the information kaOver Findings that is can be used in a lot of different ways and are being used in a lot of different ways. But there are also limits to what it can do, there are limits in the sort of data that people can get. There are limits in what they can do with that data sometimes there's so much data is very hard to make sense of it. a'r ddau hyn yn ystod mae'n dweud i ni'n meddwl am y cyfnodol y byddio'r argyfwyrnod, mae'n ffordd a'r bydd o'r hynny, mae'n meddwl a'r bydd y gallwn yn mynd ysgrifennu, mae'n meddwl a'r byddau bod yn blynyddoedd. Mae'r ddechrau ar gyfer hynny, mae'n ddau'n gwrs ar y cyfnodol yw'r cyfnodol, nid oherwydd mae'n meddwl i ni'n meddwl i ni'n meddwl i ni'n meddwl i ni, ond hynny, nid o'n meddwl i ni. Dwi'n cael ei gweithio, fel ymddangos fel four, dda'u ffordd o gwaith y panel. A os ydych yn ffrindio'r ffantafwng sy'n fynd i'r ffordd o irrwyntio'r cyfnodau. Roedd ydych chi'n cwestiynau o'r gwaith? Roedd y pwy ffwrdd o'r cwestiynau i ddod i'r ymddangos y cwestiynau. Roedd ymddangos fel ymddangos fel ymddangos o'r gwaith. Roedd o'r pwysig o'r cyfnod o'r cyfnod o'r pwysig o'r cyfnod o'r 3 o ddoedd. We will be keeping the webinar space open until 3.30 p.m. So please everybody feel free to stay if they can through that whole time. And that after three we'll move to more informal chat. So if people want to speak and turn their video on, that would be really interesting. It's about generating thoughts and conversations and exploring answers. So thank you very much for listening to me. I'm going to hand over to Christina now to kick us off on her talk. Christina works for Lucy. I'm delighted that she's able to join us. Thank you, Christina. Good afternoon, everybody, and thanks firstly to Henrietta for inviting me to to contribute to this series of webinars. I've been following them when I can, and it's been a really interesting set of discussions across the events so far. And your reflections actually tee up nicely what I was going to cover today in my remarks, a couple of commonalities there. So it's it's nice to see. And as I'm sure those of you that have tuned into previous events in this series are aware, we've heard a lot about open source intelligence and remote sensing capabilities in relation to nuclear weapons programmes and missile programmes and how these might be applied to either better understand a proliferation activity or understand the options for verifying and monitoring those activities. And the project I'm speaking about today, it's part of a consortium project that involves Lucy Vertic and the Centre for Non-proliferation Studies in US CNS. And the project is looking to undertake an assessment analysis of North Korea's nuclear weapons programme by using advancements in remote sensing tools and techniques and building a picture to analyse where we see North Korea's nuclear programme developing. The idea is to then take the analysis from the open source and the remote sensing inputs and plug that into a piece of commercial software that's often used to model civil nuclear reactors to help us better understand the flow of material through North Korea's nuclear fuel cycle and to help us and better understand where there might be particular nodes or trade points that would be interesting for maybe more creative verification opportunities. I get the really fun part of this project. Actually on that I should say that I think Grant Christopher who works for Vertic and I collaborate with on this project is one of the consortium partners. I think he presented on the nuclear side in a bit more detail, I think maybe the first or second webinar. So definitely go back and check that out. But where I plug into this project, I personally think I have the most interesting part because I get to look at how we can apply that methodology to North Korea's chemical weapons programme. I only half say psychastically that I think it's the most interesting part because it is incredibly exciting and interesting and looking at how these tools and techniques apply to chemical weapons programmes is actually quite under-researched and there's not much activity, at least publicly, looking at chemical weapons programmes specifically. Obviously the work done in relation to Syria and chemical weapons use in Syria, but there hasn't been as much of an undertaking or effort to look at the infrastructure and the programmes in places like North Korea that might actually underpin a chemical weapons capability. However, there was a reason I only half say it's psychastically is because there's a reason that these activities and these tools aren't necessarily applied to chemical weapons programmes more broadly. So I think before I kind of dig into what it is we've been doing, I wanted to just lay out a couple of points that kind of lay out where we are in terms of why this is more challenging and why this necessarily hasn't been done as much in the chemical space. I've kind of got four things to just briefly outline. The first one is the fact that a lot of the capabilities that underpin chemical weapons programmes often have dual uses. So many of the processes and the chemicals that are used in weapons production also have legitimate civilian uses. So therefore, many of the processes look the same, so it's hard to discern between what's a military purpose and what's a civil purpose and a legitimate purpose, especially when we're talking about remote sensing capabilities. This actually directly relates to the second challenge. And that's that many of the processes that are needed to engage in these activities just aren't visible in the way that we might think about visibility in terms of a nuclear weapons complex or programme. So, you know, we can be looking for particular industrial facilities, but it's quite hard to discern what's going on inside those facilities or to know what the intent of those activities are just by looking at a rooftop or a standard industrial building, for example. So there's a lot of nondescript components of a chemical programme, whether that's military or civil. In the North Korean context, this is further complicated by the fact that North Korea deny any existence of the chemical weapons programme. So any activities that are current or present will obviously be heavily concealed, again, adding to that kind of difficulty and challenge in what's visible and what we can see and know. The answer specifically, but it's probably applicable more broadly, is that we don't actually know how active North Korea's chemical weapons programme is. We have guesses, we have estimates. I am getting a little pop-up that my internet connection is unstable, so I hope you can all still hear me. I can see it up great. And so it's really hard to know whether or not a facility may have had a different purpose at a different time. You know, North Korea's chemical industry really expanded in the 60s and 70s, and the intent for the facilities that we see built then might have actually had a different purpose to what they do now. And those changes may have occurred. Well, the changes in intent might have resulted in technical changes that just aren't observable because they're taking place under a building that we don't have access to. It could also mean that actually the programme has shrunk in size or no longer is this. So we're trying to prescribe an activity to an industrial base. That activity just isn't there. We're trying to prescribe something that doesn't exist. And that's an incredibly difficult thing to try and figure out. So this leads me to the fifth challenge and raises questions around how confident we can actually be in using this approach or whether or not the error bars are too wide and there's too many unknowns to actually allow us to conclude that using tools and methodologies are they have any value in a way that is actually just not heavily caveated. And then we have to question whether or not that's that's all. So given all of those kind of challenges and hurdles I've just laid out, you might be thinking, well, how on earth are you approaching this issue? I should reiterate that this is a feasibility study. And although we're getting some interesting engagement with the project, it's still possible that we might actually conclude that this approach doesn't have that much utility for monitoring and verification in relation to chemical weapons, especially in the North Korea case. I think it's also important before I go into a bit of detail about what we're doing to just reiterate again that we're not looking for a kind of a gotcha moment. We're not looking to pin an ongoing chemical weapons programme on North Korea but looking to explore what capabilities might underpin a chemical weapons programme or chemical weapons capability in a kind of a creative and under-researched way, utilising the tools and approaches that have become available through commercial means to first and foremost improve our understanding of the chemical weapons risks that might be present in North Korea and then go from there rather than trying to say, OK, well, we can see that North Korea has X type of chemical facility and therefore it must have a chemical weapons capability as well. So it's very kind of exploratory and investigative rather than descriptive. However, one of the things that at least a large scale military chemical programme will need is a chemical industrial infrastructure. Again, this can be purely for the purposes of weapons production or it could also be purely for the purposes of simple chemical productions or both. So, but because the chemical industrial infrastructure can be quite large, this is where we've decided to start and North Korea has multiple chemical industrial sites especially for things like the production of agricultural goods and fertilizer as long as alongside many other activities as well. But because the chemical industry has such an important role in other areas of North Korea's domestic politics and economics and some of these chemical sites are actually quite visible and they get talked about a lot to state visits there, which results in ground imagery and. However, it does also mean that there's been lots of speculation around the activities of some of these sites and some of them, a few of them have actually been recognised in open sources by from governments and non-governmental researchers that there is a risk of chemicals and processes that are relevant to chemical weapons production being present in North Korea. But because many of these sites are different they all have different chemicals present, different processes present trying to think about applying remote sensing capabilities broadly across North Korea to all of these different sites is quite challenging and it's still quite broad. So in order to narrow it down more specifically and kind of develop a test bed for applying these approaches, we've taken a case study approach. So we've started by looking at one particular complex, the Namhung Youth Chemical Complex. There's a few reasons we've started this site. It's large and it's been a central part of North Korea's civil chemical industry since around the 1970s. It's still a very active site today. It gets on to state media attention, as I mentioned, lots of visits from key personnel all three leaders have actually visited the site. This then results in on-the-ground imagery that gets published in state media alongside more data-driven information, production capacity estimates, that kind of thing. And that can all provide information that we can assess and look at in relation to what's going on at this particular site. But in looking at this site in particular as Henrietta already alluded to, it's important for us to not just look at what we think is related or relevant for a military purpose or a chemical weapons purpose. But actually it's just as important to be able to make sure that we're thinking about the site as broad as we can, looking at all the activities, because in order to understand whether or not there is a risk of a chemical weapons capability present at this particular site, it's important to understand as much about the complex as possible because this will help us identify what we don't know, but it will also help us identify what we do know and whether or not, you know, understanding that doesn't have an important chemical weapons. It's just as important as finding something that might do because it helps us rule things out. And so far we've identified around 12 chemical processes and activities that have been publicly acknowledged as being present at this site with having dual use risks as well. So what the next stage of this research is for us is to take these activities, the chemical processes, and try to map them out in general terms. So ignoring the North Korean context for a moment and trying to understand what needs to go, what inputs go into these processes and what outputs come out. This will help us get a better understanding of how a site should be laid out or how a site might be laid out by being able to understand the processes and the activities that are going on. What we hope to do is to do this from a purely civilian perspective as well as a military perspective to try and see probably that there's a lot of similarity and overlap, but also pick up on subtle differences as well. We'll also then try to work with people who have expertise in these sort of processes, for example, in producing different types of fertiliser in civilian industrial sites to help us understand what the activity might look like from the ground and then also what it might look like remotely. For me, this is actually the most interesting part of the project because we get to draw on experiences and expertise of people who, like I said, for example, might have operated a fertilizer plant with processes that are similar to now hung on paper. It helps us contextualise the site we're looking at. It helps us think about what practices should be in place. It's not necessarily just about machinery and equipment and the technical things that might be in place, but what other things might you expect to be taking place, whether that's things like security culture, that you won't necessarily be able to get from remote sensing, but it can help you understand the broader context around a particular site. That's where we're going with this. It also means that we get to draw on material and information that I personally wouldn't usually get to draw on looking at nuclear programmes and chemical programmes. That's things like fertiliser needing capacity. Thinking about estimates, for example, that the world food programme might have is how much chemical fertiliser in North Korea actually needs. Then comparing and contrasting that with things like the size of the fertiliser production plants in North Korea, we can do that based on looking at how large a plant is, how active it might be, but also through the information that North Korea gives us about its production capacity, obviously taking that with a pinch of salt. That way we can gather all these different pieces of information that aren't necessarily directly related to or immediately obvious to understanding a chemical weapons programme and help build this broader picture of what's going on. Again, it's not looking to prescribe a particular chemical weapons activity to North Korea or looking for that gotcha moment to say North Korea is absolutely producing chemical weapons at the facility or it's not, but to try and help us walk through some of the pieces of information that are available that might help us understand these activities and the utility of monitoring and verification. I will stop there. I think I could talk more about this project, but in the interest of time I will finish there. Christina, thank you so much. What an amazing overview of a really complicated set of tasks that you're doing. I was really interested by the insight you gave. This kind of mirrors ideas that I'm getting from other talks about how creative and innovative the open source research can be and that you're really exploring different ways of tackling a really complicated problem. Nevertheless, some of the difficulties you're encountering are fundamental to the verification and monitoring space and the dual use problem is key to that. I could ask you lots of questions, but I'm actually going to move us on to the next speaker because not least because we started a bit late today and I don't want to lose anybody too quickly. So thank you, Christina. I'm going to move on now to a video serban from the Imperial College Data Science Institute and I'm really looking forward to seeing what you're going to talk about. Thank you. Thank you. I will start by sharing my screen and if that works, then we can actually. Okay. Apparently I can't share my screen. I'll see if I can do anything about that video. Let me just. Henry, actually that project that you can share your screen just wants to be sure that occurs have the right permissions. Sure. No, I think it's something that has to do with my laptop. Okay, I see. See, let's see now. Okay, I seem to be able to. Okay. Thank you. That was the first problem. Okay, so I work in machine learning and actual English processing. So this is a slightly different perspective from what you've seen and probably what you've seen in other talks. And I currently work in the Data Science Institute where we try to process large data sets. So on this particular project, I'm going to talk about the potential of using social media users as sensors to detect different kinds of events and how can we process this in real time? And hopefully by the end of the talk, I will be able to convince you that this is actually possible. So everybody knows social media and it's this large delusion of information and there are certain platforms available, some of which are more popular and the content is spread around in either text form or image or video or any other mean that people can imagine. And we asked ourselves a very, very early question in this project. Is this information actually useful? Can we do anything with it? And can we process it at scale, especially within an academic context? So in order to confirm this theory, we started the project on biosurveillance where we try to detect symptoms of diseases and symptoms mentioned, as I said by people on Twitter. So the challenges that we had to overcome was to first find out what was the most appropriate vocabulary to detect these symptoms of diseases because as you probably imagine, being in a very, very informal environment, people are not going to use the medical terms to describe their symptoms on social media. We were particularly interested in using text primarily for this particular project, but I know of other projects who started using image and video feeds as well, especially for security context. And yeah, this is something that I think it's absolutely possible. So in order to be able to do this at scale and just to have an impression of the scale we are talking about, we were with Twitter, if you subscribe to their public free feed that it's given to you in real time, that's about 2 million tweets per day in US alone. And if you want to have, and that's about 10%, so in order to process everything in real time, you'll have to have the capability of processing 20 million tweets in real time. And no human being is, or yeah, collection of human beings is able to do that at this scale. So in order to do this and to make this project possible, we decided to go with machine learning and this is something that I work on. So taken the Twitter stream, we are able to extract first the text the timestamp of the tweet, the unique identifier and the location metadata. And this is fairly important because we want to pinpoint this observations on the ground for to find out if they are reliable or not because if you make an observation about an event happening in Los Angeles and you are in on the other coast, it may be less relevant. And then we extract the city and the country from this metadata and that's, I'm not going to go into much detail but the Twitter information is a bit convoluted so we had to process that as well. And then we had to do some natural language processing and also create a machine learning classifier to distinguish between the tweets that are health related and also detect the symptoms within the tweets. And when we looked at this originally people if you look at work in marketing and other type of social media analysis they look primarily at hashtags and trend analysis to see how things are going but in our work we decided to go beyond that and we analyzed the text we analyzed the context because if you say, okay, I have a fever you may have an actually symptom of a disease but you can have also a baby fever or a Bieber fever or whatever and this is something that is not related to a medical condition and we have to filter that out. Okay, and then we have these daily counts aggregated by state and symptom. What we also wanted to do very early on on this project in order to increase the confidence in the results is to compare these results with news sources. Now, bear in mind that we are processing the Twitter feed in real time as the tweets come in whereas the newspapers may report something about an incident on a few days later. And then because we wanted to make forecasting and predictions about future events we had to have our gold standards and ground truth and then we collected data from the CDC which is the Center for Disease Control in the US on all the diseases that they are currently publishing statistics on and this is on a weekly basis a group by disease location and count. Now, there is a bit of latency for publishing these results and the reason why we decided to do this processing as well is that we realized that official data comes two or three weeks later. For certain types of reports and for certain types of work you would like to have results even if they are noisy and biased you would like to have some information and some sensing on the ground earlier than that whereas with Twitter you can do that in real time with CDC data and other national healthcare statistics you can't do that in real time, unfortunately. So this is what we've done. Now, because this project was primarily meant for analysts that will use this data stream and compare it with other data streams they will find some event relevant that will send someone on the ground to investigate a bit more. So we have the evolution of the event on the chart on the top of the screen. The part highlighted in orange is the current event so the analysts may decide to investigate that particular event. Then you have the location or some of the places where these tweets are coming from. Then you have a collection of tweets that we display there are more but for this particular presentation I had to crop the image. A word cloud which may be interesting to give you more confidence and an idea of what are people talking about in this particular event and then we also compute the most relevant tweets for this particular event. Okay, and now we can keep this data for ourselves and we can publish it as an academic partner but I also decided when I joined the data observatory at the Data Science Institute in Occupational College London is to create a version of this data set that is available in the data observatory and this is a very social place where you can have large scale visualisations and you can discuss about your data sets and your results and literally in this particular project you are surrounded by data and what you see around you is the timeline of the data that is available. Okay, and we have different kind of views. This is an intensity map with all the events that we detected and brighter states are states where more events are available and this is the timeline I was talking about and what I want you to realise about this timeline is that there are multiple events. Each event is colour coded by their symptoms and then you can see that some events are spreading over multiple days some are just half a day. So, yeah, as you will see in any PI surveillance application. And then you can go into more details. So what I'm showing here on the screen is one particular event, the evolution of that event. Then the orange area is the current event that the analysts may decide to investigate and then when we go into more details we can see a zoomed in version of the previous slide with the timeline and the event details. And then these are a couple of tweets that we selected. And yeah, just to give you an idea of what kind of data we are dealing with. And all the scores that we see on the screens are what we think as confident or what we describe as confident score that these tweets are relevant or not. So, yeah, this is the data set I was working on and to move beyond this we are doing the same for political events and also for different type of surveillance applications that can benefit from social media streams. Thank you, Avidio. Really, really interesting. I don't know if you can hear, if you can pick up from... It's just started really, really raining, really heavily here, so I hope that's not interesting. Avidio, that couldn't be more topical. Tracking disease in real time. I mean, you must be watching events unfolding and thinking a lot of things. I, like with Kristian, I'm not going to ask you a question to yourself, but we've had a couple of questions via the chat function and I want to give each panellist a chance to respond to them, not least because I know some of you need to leave at three. So, I'm going to invite you, in turn, in speaking order to reflect on these. If you've got no comments, that's fine. You can move quickly through them. But there are two comments about guarding against some of the threats that open source research might pose. The first question I have to scroll up was from Suria. Thank you very much. Saying, is there a way that threats against sovereign states can be addressed? And then Sarah Stanley moved beyond this and was considering threats to individuals if they can be identified by some of these resources but by some of these methods. Sorry, how do we keep the people doing the work safe? Are there any sort of codes of conduct that people are aware of that feed into this sort of work? So, as I say, speakers, don't feel any pressure to answer if this is something that goes beyond what you've thought about. But, Pristina, do you have anything to comment on this? In the interest of time, I think there's others on the panel that are better placed to comment than I am. So, I will keep my thoughts to myself on this one. Okay. Thank you. A video, your turn. I don't think I have anything to add. Okay. Thank you. This is very quick. Gary. Sorry, I've just got to unmute myself. Now, this is a very interesting question, particularly in terms of what the sort of subject matter that we have to deal with with Project Sandstone in terms of what we need to go in terms of, for us, when we look at say the illicit shipping networks, which I'll be talking about after answering this question, we have to be very sure in terms of who these individuals are and in terms of say, for example, the owners of the companies who happen to be sort of operating or owning these ships engaged in alleged illicit activity and we have to, and for that, we're not exactly, when we don't say outright that these people are engaged in illicit activity, but their names are attached to these companies which are attached to these ships that we know are and we are ending up in say North Korean waters, for example, though. But I agree that I think going forward for this, it would be a good idea to, for like the sort of an osyn community, have a code of conduct in terms of what we can do to try and protect the identities of people who may not exactly be involved in these, who may not necessarily be involved in these activities and may be misidentified, for example. But I don't think I've got really anything else to add to that. Thank you. That's really, that's a really clear outline of the problem and, you know, feeling way towards a solution. So Christian, do you have anything to comment about these questions? Yeah, I'll add something to this, to Sarah's question because the question on security challenges to sovereign states is obviously not something I'm involved with because I work for a newspaper. But individual security, I mean, just one point, like just to give you an example, yes, I think it's very important to be aware of this. Just to give you one example earlier this year, Iran shut down a civilian airliner and we were the first to break the news with visual evidence that they actually shut it down, which they were first denying. And basically the key to showing a video that says like, hey, this video shows this civilian airliner being shut down. Requires that we kind of explain how we verify the video. That we're not just showing a random video and accuse Iran of shooting an airliner, right? However, what we do as part of the verification process is geolocation, finding out exactly where the video was filmed and chronolocation is basically based on visual clues, finding out what approximate time the video was filmed or around what date, right? So that you are sure that it's recent. However, you can imagine if you're published the exact location where a video was filmed from, this is evidence. This is before Iran admitted they shut it down. So if you publish at exact location, you may endanger the person that filmed the video because you're basically helping Iranian security services to like, okay, this is the house we need to visit, right? So this is a problem that we run into, I would say every second time we are working with sensitive videos and it's not like a CCTV camera, but someone who filmed it. And I think it's really important to judge it from a case-to-case basis and a group that has done great work on it is the Human Rights Center at the University of California, Berkeley. So the Human Rights Center at Berkeley is anyway doing really well with open source investigations. And I would suggest to check their work out. Great, thank you. And that really echoes comments that we've had from other participants involved in those very real political processes where people contributing the data can put themselves in harm's way if they're known to do so. So thank you very much. We have got another question directed at Christina from Paul Schulte. So I'm going to invite Gary to give his tour, but Christina, if you want to be thinking about that, that would be great. Gary is going to tell us more about his work of Project Sandstone. We are getting very tight for time and I know that Gary and Christina need to leave quite promptly. So yes, thank you very much, Gary. Okay, excellent, right. So first off, let's share my screen. Hopefully that will work. Okay, can everyone see that? Excellent, right, so let's... Right, so yeah, my name's Gary Somerville. I work on Project Sandstone, which you might have already heard before. One of my colleague, Joseph Byrn, was on one of these webinars last month and gave a pretty good overview of the work that he does on it. But I'm just going to go over this briefly into what Project Sandstone is for those that didn't see that webinar. So Project Sandstone is an open source initiative that focuses on mapping investigating North Korea's illicit networks. And our main objective is to generate open source data on the illicit sanctions evasion activity and the information that can be this limited to the court law and also to build a capability for open source data streams that can be used in a variety of different scenarios. So we are looking at, specifically, North Korean sanctions busting activity that can work on illicit wildlife trafficking, for example. And essentially, we want to create information that is actionable and that can be also submitted to a court of law if necessary. So just to give a bit of background on DPRK illicit shipping, as many of you already know, is that the North Korea is already under stringent UN Security Council resolutions in response to its nuclear misalblistic missile tests, which also includes a lot of sanctions against its exports as well, including coal and iron ore, which it generates a lot of revenue. And a lot of that revenue is controlled by the military and a lot of that revenue then goes into funding when you can invest in missile tests. Why we focus on, say, the shipping aspects is that because, of course, a lot of world trade is waterborne, the vast majority of it. And so it's not just North Korea that uses this means of transportation to conduct illicit activity, but also many of those states and criminal organisations across the world. Based on our own research, we know that, of course, the stringent UN Security Council sanctions have made it far more difficult for North Korea to get hold of parts that would be used for its WND programmes and ballistic missiles, but it can still occur there. They are, of course, adapting to try to get around the UN Security Council sanctions. So for this programme, we use a number of open source methods. Of course, Joseph has already talked about the work that he does in the previous webinar, which I highly recommend that you watch as well, where he uses commercial satellite imagery to identify ships, usually in sort of the hotspots of where they gather. So, for example, Nampo, Wonsang, Tongjin, and then using AIS data then sort of is able to overlay that and track the ship and also using machine learning tools. And then on the other side of that, there's also the network investigation as well. So we're looking at, say, the companies that own these ships, and that is what I specialise in. To give a bit more context on that, I'm a Chinese, my background is in Chinese studies, so I'm a Chinese specialist first and foremost. Therefore, my language skills are utilised every day to conduct these further investigations into these networks because a lot of them are located in China, Hong Kong and Taiwan. So I'm going to be going over a couple of the methods and techniques I use, some of the particular peculiarities of, say, doing sort of open source research in Chinese language and using Chinese websites and also some of the challenges that that also poses for someone that's based on the other side of the world. So first of all, looking at maritime data, because we need to know who owns these ships and there's a number of resources that are available. Open source one of our preferred one is a paid for subscription portal called the IHS Market Maritime Port, which provides a wealth of information. And what we're looking for here though is in terms of, as well as like sort of the general ship data information, but also who are the owners, operators, the ship managers behind the ship at the time when it was conducting the elicit activity. And from there, we can also then use that to then look into the company as well. And from that, we can identify, say, contact details because what we're looking for here are identified. So either named individuals, their contact details, information on the parent companies or subsidiaries, and also possibly any other ships that that company may have owned as well, because sometimes they are, a lot of their ships in a fleet under a company will engage in elicit activity as we've seen. In addition, there's also the Techia MOU, Port State Control as well, which is also quite useful, which is also free to use and provides also similar information, but also records of whether a ship was, particular ship was detained for and inspected or detained for certain ship deficiencies such as safety. So for a lot of these companies like that are, for example, based in China, there's a number of where we start is we look at, say, the corporate databases in China. Now, there's the official one, the National Enterprise Credit Information Publicity System. But in my experience of using this, it's not it can be temperamental if you're trying to access it from outside of China, where it will suddenly give you an error message and you get a lot of 404 errors. However, there are plenty of third party websites, which database a lot of the information from that's the official database and actually provide a better visualisation of all that information, which would have been gleaned from ending statements and putting it into something that's far much more readable, and from here we can look at all sorts of information such as the corporate officers, their shareholders, the contact information of the company, their business scope, including historical information as well, such as when has the company changed addresses? Does it have any former names? Who are the former shareholders? Who are the former directors? And you need essentially to collect all this information as best you can. Now, of course, sometimes one of the issues of sort of coming out this from the angle of say having say an English name of a Chinese company, but they may not necessarily use that name openly if you do Google search. And in terms of the sort of number of Chinese characters or what it could be that name, it's very difficult. So usually one of the easiest ways to do that is then using like say, use the like say and the address that they register like say IHS market on those maritime databases to then do a search and see which companies come up. And if any of those Chinese company names match up with the English one and then you've got your and then you found the company that is that you were looking for. In addition, we don't just look into into into into China as well. There's also a similar database and registries available in Taiwan and Hong Kong as well. So Taiwan's official government registration system database works quite well and you can use that to easily find Taiwanese companies as well as Hong Kong's as well, which is also a paid for it's free to register but you have to pay for like say annual statements if you want more information on that. Of course, the caveat being that if you're searching for these for the company names in Chinese for this, you need to be doing it in traditional simplified otherwise you're going to get no results. There's also an and on the Hong Kong corporate registry, one of the interesting things that you can sort of pick up into the the director of the company would have to put on the and on the annual statement either a passport number or China ID number. And the interesting thing about the China ID numbers is that they are that they do reveal certain bits of information about the individual, such as where they're where they're addressed to, which as you can see at the bottom there, the address code then followed by their data book and then the order code of which the last digit of the order code indicates whether or not they're male or female as well. So again, from that just from that bit of information, you can sort of lean a bit more more identifies of the individual that's sort of operating say the company in Hong Kong, which by the way, and I think Danielio in his presentation, he did the last webinar he did here where he alluded that the use of Hong Kong front companies, which there's there's a lot of these listed shipping networks do make heavy use of Hong Kong front companies. And one of the other platforms that we we also use in Sandstone is Sierra, which is a data intelligence platform which is designed to assist the fight against financial crime and improve corporate transparency, particularly in high risk locations. And so it provides essentially just gathers all this sort of information on corporate records, in export records, property transactions, tax records, et cetera, and puts into a much more easily searchable database. And I mean, it's not something that you actually have to have, but it does make things so much easier though when when carrying out these investigations. So from there, we have a good idea in terms of who the company is behind who the companies are, all their over identifiers and the shareholders and the corporate officers involved in it. And there's a good and this these sources that we use here are not exhausted, but these are some of our like main go to sources that we use from time to time. So we make use of like core rulings as well. So some core rulings are made available online on a Chinese government website where you can search and then find information in terms of whether or not these individuals or this company has been involved in say lawsuits, which fully an awful lot of these, a lot of these companies and individuals are. And what it gives you and it gives you an insight into like say their business practices and also who they do business with as well. And also identify over bits and information in terms of how they and how they're how they conduct their activities. I mean, say for example, one company was sued by a lot of the crewman of a ship simply because they didn't pay them. Tywin also, there's also a similar sort of database as well in Ty one. And also in addition, because not all the court rulings are made available on this website, the Chinese one, but you are able to sometimes on some of the Chinese corporate databases are mentioned in the previous slide and that the information is sometimes available on that company's webpage on that on that corporate database. So in terms of other tools we use, there's also a particularly biting maps as well, which is essentially by his version of Google Maps as you'd expect, which I prefer to use rather than Google Maps for the simple fact that Google Maps does not always if you're searching for an address saying in Chinese, it doesn't always come up with the it doesn't give you a pinpoint result. And sometimes doing it in biting maps can usually yield that result. And in addition as well, because you let those of these like used like say Google Street View, Google Street View does not work in China, but there's no such thing. But the idea does have an equivalent called panoramic, which again can be, I mean, if we don't necessarily have to use it that much, but the option is there if you want to sort of get a good idea of what is located at that particular address. Gary, I'm afraid I'm going to have to interrupt you. This is all so interesting. I'm giving a really big sense of how your work involves collecting painstaking levels of information and this going back to what I said earlier, actually the sense of the context really matters. Now it's three minutes to three and I know you have to go. I can probably overrun of a little bit. Well, I know Christian is maybe waiting to come in but if you'd like to wrap up quickly, that'd be great. Okay. Thank you. Apologies. Okay. And then, of course, use of social media. No need to apologise. It's great. Of course, use of social media, LinkedIn, Facebook, of course, they provide you some information, but not so much for mainland China because Facebook, of course, is banned and they have Weibo and WeChat, which are also very difficult to access from outside and also have certain risks because essentially they want to know exactly who you are if you're using Chinese social media and that poses a risk to anyone that's doing this sort of work. And, of course, also quick searches on, say, of a, you know, just on Chinese search engines like using the other identifiers, maybe someone's been posting up ads online and that where they're giving away more information which could be, again, useful for us going forward. So taking all this data, then we need to sort of database and map out the network. So we use this network mapping software we can use. We use Mortigo, which is free to use, but has certain limitations unless you pay money for the other stuff, which there's a customized ontology to create entities and also makes it very versatile in that way and allows you to visualize that data into graphs which is very useful to understand all this data. From that, what we're looking for is overlapping identifiers. So corporate officers, shareholders, do they own any of the company? So are they connected to anyone else? Contact details do some of these companies that connected, of course, share contact details and as well as overlapping registered addresses or whether or not they're in sort of a, whether some of these companies in these networks are located, like say, for example, in the same building. Then, of course, there's cross-referencing that data, all that information as well with other list activity of sanction entities. So, you know, there's a number of places to look for this and how the experts report UN designation list, UN natural designations list, which is like what's published by OFAC or the EU and as well as other open source reporting. So just a quick example here of cross jurisdictional, the importance of doing this sort of cross jurisdictional mapping of networks, as you see, they operate across many jurisdictions. You can read this in our fourth report, which Chinese companies are using UK registered companies to own ships that we're going to in North Korea. And same another one here where you've got a Taiwanese-based company that's had a registered Hong Kong company that connected to another ship which had been doing a listed activity and going into North Korea as well. And another technique we also use as well as a timeline analysis where we're looking for here for overlapping events and patterns of behaviour as well. So you can do this on Excel, but it's usually just, we have a bit of software which we use called precedent which allows us to visualise it and we can make more sense of it. And yeah, so I mean from this we can also sort of understand patterns of behaviour such as a recent report on the North Korean ships that were sort of in Chinese waters that we published earlier this year is that we were seeing an overlap of like say certain Hong Kong front companies that were being registered in at around the same time as each other to operate different ships and even though these ships are also showing similar patterns of behaviour and movements to one another that suggests that they are being centrally coordinated. So just briefly outline some of the challenges that we do face though in this as well is that because of where we're based it's very difficult to access certain sources in China such as some of the full database or paid databases which is not possible. As I mentioned earlier there's also the problems of getting access to Chinese social media particularly now. And one of the other interesting things we found out as well with the use of the Hong Kong front company is that they've been before where a known Chinese individual that is linked to say a company in China is registered themselves as the director of the shareholder in the Hong Kong front company. However, what we're finding now is that a lot of these Hong Kong front companies operating these ships are alleged to be involving the listed activity or not. The individual is the correspondence address they use is in some remote village somewhere and what we suspect is that they're using the somehow someone's getting hold of the ideas of these people and using them to register the companies and they can't be traced back to them from open sources and this is one of them. And I mean there's how they get hold of the ideas we can only guess but I'm sure you can use your imagination. But yeah, that's the end of my presentation. Sorry about having to rush through it. No, that's brilliant. Thank you very much, Gary. Really, really interesting and you're right. It followed on very nicely from Daniel Hughes presentation at the last webinar giving this sense of piecing together different bits of the jigsaw. Thank you. Now, I know you might need to leave so I'm going to say goodbye to you and thank you very much. And I know Christina has already left. So thank you to her as well and Paul I'm going to respond with some comments to you from her. But now I'm going to hand over to Christian Triebert. Thank you very much for waiting. Christian is from the New York Times Visual Investigations Unit and has a background with Bellingcat amongst lots of other things. So thank you, Christian. Yeah, of course. I'll keep it very brief because we're already way over time. So I'll just keep it like I'll just do a very brief case study. Let me see. We'd love to hear anything you've got to say. Please don't really have to rush. It's going to be great. You can hear me and see my screen, I think, right? Yes, that's great. Thank you. Perfect. Let me just see if I can remove your faces somehow. Not sure how. Smaller. This should work. OK, so yeah. I work at the New York Times Visual Investigations Unit which is basically a new form of accountability and investigative journalism being pioneered at the time. So basically combining traditional reporting with more in-depth way of digital investigation. So I think of open source investigations using satellite imagery. We've heard it from the three different speakers. Basically company record, ship tracker, you name it, right? We use it to investigate. So the team is now we have around 15 to 20 people that includes like editors and senior producers and so on. We mostly produce video formats and it's mostly focused. Well, a lot of it is internationally but more recently we also focus a lot on use domestic issues. The top right here we see the shooting in Las Vegas which is obviously already a while ago but we focus a lot on police brutality as well. In the top bottom left corner you can see that. And just to give you a bit of a sense of the variety of investigations we do. I mean, it's police brutality in Hong Kong. It's Venezuela. It's the murder of Khashoggi in Turkey. It's the NYPD in New York, the city where I live, a black driver, marijuana boss, and a body camera that turned off. And it seems that an NYPD officer is planting evidence in the car. This one, one bomb 11 children killed and evidence implicates US. I will talk about it in a second because their open source arms tracking was vital. It was actually the story. The story was built around open source arms tracking. An Israeli soldier killed a medic in Gaza. We investigated the fatal shot. This is very similar to what Gary talked about actually how Kim Jong-un smogled luxury Mercedes in North Korea. So in case you want to see a very complicated investigation with company records, satellite imagery, AS data combined in a five minute video. This is one of the very few videos of us that actually kind of more a less heavy subject. I mean, it's still about smogling but it's just about Kim Jong-un's luxury cars which is like a slightly lighter subject than most of the stuff we focus on. How George Floyd was killed in police custody, we basically tracked the whole incident second by second. And this one, one Pulitzer was part of a body of work at one of Pulitzer this year. Russian bomb force here in hospitals. We have proved basically we obtained thousands of air force recordings and we basically analyzed them to investigate the Russian air force in Syria. Basically a modus operandi is like something happens and we collect and analyze evidence, right? Use these videos, think of scanner audio, think of photos, witness testimonies, you name it. And basically we tried to get as much information as possible. So what we're seeing here, this is the Palestinian medic that was killed in Gaza and basically it's second by second because there's so many people with mobile phones, right? You just try to put them all on a timeline. Another thing is like verifying the visuals, right? This is a process we call geolocation. So on the left we have the source image and on the right we have a reference image which is a satellite image in this case and we try to link the visual features that we see in the source image to a satellite image, for example, or another image from a ground perspective, for example, to make sure it's from the same location. Ship and plane trackers. This is an investigation into Italian bombs being shipped to Saudi Arabia and then eventually being used in Yemen. If you're interested in that, definitely check it out. It explains the methodology. It's in a video format. We use treaty reconstructions as well, often working with the group Forensic Architecture based at Goldsmiths University in London. This is in orange is the Palestinian medic that was killed that I talked about earlier and basically it's every other individual that was around her just to make sure that we have an entirely clear view of what was going on when the fatal shot was fired. Obviously what we're here for, identifying weapons is a big part of open source investigations. I mentioned or Henrietta mentioned that I used to be part of the group called Bellingcat and definitely check out the work of Bellingcat and definitely we can do it after the questions but I myself come from the kind of Twitter niche network. I had worked on the ground in places like Iraq and Syria but what really brought me to a newspaper like the New York Times is literally online investigation, literally investigating weapons and tweeting about it. So the community around Bellingcat is very encouraging. There are a lot of people there that move on to other places but also just the fact that so much can be done online. This by no means to say that on the ground research is not important by no means but sometimes you do not have access to the area and the only way to get a sense of what's going on is by open source investigation and in this case because Henrietta specifically asked to talk about arm striking obviously because of the project it's arms identification as well. So I wanted to show you this case study about Afghanistan. One bomb, 11 children killed and the evidence that implicates the US. So this was an instance where I mean it's Afghanistan. There's a lot of things going on but there was an alleged airstrike on a family home in Wardak province in Afghanistan. And what we usually do when we get material like this, you just look over it a few times, you can see people going through what appears to be the aftermath of some kind of explosion. They claim it's an airstrike, they claim it's the US and the claim here was that 11 children were killed. So there was a family living here and their cousins were staying with them as well and there was only one survivor which was the father because he was working in Iran and was not home at the time. So we have allegedly 12 people killed, 11 of them being children. So I'm not going to show you the pictures but just to give you some background, right? Like we also have pictures of the bodies that were recovered and we had some pictures of the kids as well while they were still alive. So this is a gruesome process to basically compare their faces and features to make sure that we're highly looking at the same kids. But once we establish there some kind of level of confidence to say, okay, hey, this really seems that a lot of children died in this strike as well as a woman. It's like, okay, let's take a further look at this strike. Now, the father who survived the strike, he already seeked answers, right? Like who is responsible for this attack, which was suspected aerial airstrike. But there were very little answers for him. So we teamed up the Bureau of Investigative Journalism, which is also based in London, and we decided to investigate this airstrike ourselves specifically because in October 2018, the Pentagon told us that they found no connections between directions and the claims of non-combatant casualties, which were these 11 children and the mother, right? The 12 casualties in total. So they basically said in October 2018, we don't find any connections, right? And later on, they would go even further when we asked about it again, like we do not even have a record of an airstrike being conducted on September 23, 2018, in the Ychydig District of Water Province. So what we did is like, okay, it's interesting because we have a flat-out denial from the Pentagon that they would be striking, that they even conducted an airstrike in this region. So we thought, okay, we clearly see that this house is damaged. We do seem that indeed casualties, they were living here. We know they died. So what is going on here, right? So a very important thing is that we always try to, like was said, a geolocation, right? So we had this photo of the house and we tried to map it out. Now, an interesting aspect is for anyone who is focused here on Palestine, on Iraq or Afghanistan is that, for example, if you use Google Maps, you won't get the best satellite imagery because it's either downgraded in quality or it's very outdated, right? So you need to find other sources for satellite imagery. You can fill in the reasons, I mean, you know, why are those areas where there is not high-res satellite imagery publicly available most of the time. But we were able to find the exact location. And an important part to say is that we knew that a raid was going on in the village that time. So there's a Taliban person here where Afghan soldiers were being held captive. And Masi's house, the father, the family was staying here. And we indeed could see with before enough satellite imagery that the house was bombed between the time frame that he said it was on the 23rd of September. So there was another part. But then we come to the most important part, which is like when either online photos or images appear and these are alleged weapon fragments. It's obviously very important to make sure that you are sure that these photos or the fragments were indeed found at location and are brought in from somewhere else. But once you know, you can really distill a lot of interesting clues usually from the weapon fragments, right? So in this case, there were this pattern of four bolts which you can see here. And we have, we can see them here as well on a different part. And we talked with weapons experts inside and outside of the New York Times that said basically these are the steering fins, the till fins of a G-DAM, a guiding spot for usually unguided bomb. So a G-DAM basically makes an unguided bomb a guided bomb. And the thing is this is very important because by talking with a variety of experts as I said outside and inside the New York Times as well as from our own experience, we know, okay, this is a very clear indication that this is G-DAM munition was used, guided munition was used. Now, there was another fragment which we found which had a number or characters on it which led back to a US company Woodward. So we had those two fragments besides other fragments that were really keen investigation for us to believe, okay, a guided munition was used, or a bomb with a G-DAM which makes it guided was used. And this was such an important part because for us it was like, okay, we know these fragments are from the location, but the US is flat out denying that they connect this airstrike, which is interesting because we are thinking like, hey, are there maybe other air forces active above Afghanistan that could have launched this strike? Now the thing is basically only the Afghan air force and the United States air force are conducting airstrikes above Afghanistan, specifically the water province in Afghanistan. So we know for a fact that the Afghan air force at the time was not capable of carrying G-DAM munitions. So that left us basically, it was very simple, left us, it must be the US. So what we did is we are like, okay, either we don't know what's going on, but the US is denying a strike that we have evidence for that it was clearly them that conducted it. So what we did is we provided them with the exact coordinates of the house, of the approximate time the day and these munitions basically saying it cannot be any other way. Then it was, we confronted them with this evidence and then they got back to us and they completely changed their story and said that after review, it's our assessment that only combatants were killed. Now we can go for an hour, hours, hours, an hours and discuss how the Pentagon reviews civilian casualties. I won't do that right. Say a week later that the building was stuck and self-defense because there was a raid going on on this Taliban prison and they said there was firing coming from the house where they were staying. Now we weren't able to confirm this, but this is just a very small example to show how important, even if you cannot visit the location, we weren't able to visit the location because it's under Taliban control, how you can still investigate incidents without being on site. And I think, yeah, this is a very small example, but yeah, please check out the work of the New York Times for this investigation team if you're interested in learning more, but also definitely check out Bellingcat and other online arms investigators. There's a guy called Calibra Obscura. We have Abraxas Spad. There's all these guys on Twitter that are really good at arms tracking. For those, I saw there were some questions about the OPCW and so on. Make sure to check out our investigation into the chemical attack on Duma as well, which includes all of the stuff I talked about before as well, but also show some of the limitations. I guess I should stop because it's late. Amazing, what an amazing talk to incorporate all sorts of things that have been mentioned. Thank you very, very much. Really interesting and amazing to see those visuals as well. I was really struck, Christian, that your project has the potential to be truly global. You're looking at what's happening in New York as well as what's happening overseas, which is very interesting. As you pointed out, Paul Schultz has posted a question. He directed a Christina Viola, but I think it has a broader significance, although you may disagree with me. Paul's question to Christina was, how could open source conclusions be integrated into OPCW and UN actions? Regimes at the international level that are set up to deal with international problems, is there a way that results could be fed into those sorts of processes, bearing in mind that all sorts of information is challenged by various states? Paul, I can see you've appeared. Do you want to talk for yourself? I wondered. I was going to put it to both Christian and the video because I recognise that your research might not be aiming to impact on political processes, but I wondered if you considered, clearly, Christian, you have considered how to hold a government to account, but if there's wider considerations about how to feed into bigger international processes. I'll start with Paul and then take the speakers in turn. Thank you. Well, we know how much the Russian and I think the Chinese government do not like billing cat and associating activities. They are going to be able to say that almost everything you produce can be synthesised. So they will be questioning just the sheer replicability and provability of the sort of information that you deal in. Now, of course, it's useful for early warning. It's useful for sanctions. It's useful sometimes for bombing places, which is one of the reasons they don't like it. But how usable is it going to be for politically and legally mandated processes which are subject to Security Council veto or even OPCW council veto? That's my worry. The world's getting very good at looking at stuff, using the compound eyes of humanity, but much of what is being revealed will not want to be seen and will be ruled out as unacceptable by very powerful international actors. That's a recurrent problem for a number of the subjects of these seminars. What is to be done about that? So I'm going to... Thank you, Paul. I'm going to link that and I'm going to ask the video first. Just bounce it back if you want to. I'm going to link it to the question that Paul gave to a video about how easy it is to spoof the data you have. You already said it's very easy and that's why you recognise that. But my reflection for what it's worth is that the sorts of work that a lot of open source researchers are doing is not looking for that one piece of evidence. It's about combining multiple pieces of evidence. So in that case of videos, your work could be really feeding into different processes, but have you got any comments for Paul or this wider political question? Yeah, so I always have seen my work as an early warning signal and not the only signal that can confirm or deny the presence of an event. So even in bias surveillance, as I mentioned, the system that we've been designing and we've been building was only meant to be one of the data sources. And then the analyst would compare these signals with something else they will have. They will compare this with newspapers and other data sources they have access to. And they will confirm if it's something that is worth investigating deeper. Now related to spoofing and data distortions, I know that when you are dealing with automated systems and specifically with machine learning, this is very, very good balance to have, to have cleaner data that you can rely upon and then have noisier data that you need to confirm. With spoofing, I've been working on spoofing applications and spoofing detection on cryptocurrency more recently because that's very, very prone to spoofing. And then the same principle can be applied to social media analysis. And also more recently for coronavirus, we've been looking in more, I can't call them cleaner data sources, but more aggregated data rather than individual tweets that I described in this project. And then, yeah, when you aggregate more data and you have data from multiple sources, then you increase the confidence in the data that you're dealing with. So, yeah. Great. Thank you very much for the video. That was very clear. And I'm going to hand straight over to Chris John because I know he has to go pretty soon. So thank you, Chris John, if you want to comment. I mean, yeah, it's a good question, but I don't think I'm really in a position to say anything about it, right? Because it's my job as a journalist to find out the facts. And basically that's what we're doing, right? So bigger questions as to like, does it fit into the OPCW? I mean, it's, I mean, yeah. I mean, obviously we're looking at, okay, hey, what is the impact of our work and so on. But bigger question is like, okay, Russia or China are like obstructing independent fact finding missions or stuff like that. I mean, we can see it, we can see it with Israel doing it, US doing it, right? So, I mean, yeah, I guess it's a good question, but I don't really know what we could have a very long discussion about it, but I guess I'll just, for now, just stick to finding the facts. Because I mean, we still need to continue doing that, right? Because saying, okay, because like, sure, it may not have consequences, right? What you're saying is like, okay, hey, there's a chemical attack and there's like basically zero consequences for it. In the case of DOOM, of course, it was not the case, but it doesn't mean that we shouldn't investigate the facts anymore. I agree completely. Don't stop doing what you're doing, but we've got to be careful in jumping to conclusions that that's going to transform international action about illegal activities, proliferation and war crimes. Yes, and I think we've heard these sorts of conversations in the other webinars, and I think there's also a sense that it's maybe not clear everything that open source research findings can do yet, but they are, as people have said, the transparency in and of itself is really useful and interesting and valuable in different dimensions. And even though we might not know everything that might be possible in terms of how things could feed into international processes, there are nevertheless some very clear cases where open source research findings have made a difference. I think in research conversations with people, there's been several examples of people being in court charged with crimes they have clearly committed on the basis of open source research. So things, there are consequences that can come out of this. It's difficult. It's not straightforward, but things happen. So there are two minutes left, and I've got some more questions. I know Christian has to go. Please, anybody go that needs to, and I'll see if we can get through any of these questions. One more comment from Christina Paul that she made just before she had to head off. She said that similar work in the nuclear side does get fed into the IAEA and cited by government in cases where private intelligence can't be used. So things that it is being used, I totally get what your point would be is maybe being invoked, but it's not being heard because there are ways of closing down conversations around it, but it is getting to different places. We've also had some questions following on about how to protect practitioners and how to look out for the people that are doing it. And there was a comment about there may be lessons to be learned from work on drone strikes. And I think that's a really interesting question to explore. So I am actually going to stop the questions now because we are out of time, but as always, I'm sure we could have gone longer in all of this. It's been really, really interesting and has given me even more to think about than I generally have at the end of these webinars. A video is so fascinating to hear about disease collection. I'd have loved to go further with you about the instability of search terms and how Twitter may be not particularly satisfyingly objective data source to research. But that's for another time. Thank you very much, Christian. That was fantastic to see how this is being used by journalists. I wonder if it feels very much as though this field can only get bigger. You know that this is the start. So it would be very interesting to think more about that as we go. But thank you to all the speakers. Gary and Christina were also fantastic. Thank you to the questions and for everybody for coming. And I hope very much we'll see you next time. The next webinar is on the 18th of November. We were scheduling for the 4th of November, but we realized there was a very big event the day before. So we've changed it to the 18th of November. And on that day, we've got Melissa Hannan from the Open Nuclear Network and Hans Christensen from the Federation of American Scientists and Alan Hill from Ridgway. So it's going to be another very interesting one to come to. Thank you very much, everybody. And I hope you have a great rest of your day. Bye bye.