 Well, some impressive work you did at the ICTY, both of you. So going back to our discussion, now that we've seen some clips from it, what would you say were some of the key features of the ICTY that distinguished it from, say, the domestic criminal procedures or other international fact-finding missions? Maybe then I can answer first and then follow by Patrick. Some of you might remember that in 1993 when the international tribunal for former Yugoslavia was founded by the UN Security Council of UN, not General Assembly. It was greeted by liberals across the world as something very significant. And there was no doubt at that time that it should be an international court and that it should be placed somewhere outside of the region. For obvious reasons, there was a war waging in the former Yugoslavia. And the court was created and became operational in the Hague, the Netherlands. So when it started to work properly, with many cases going on, and we are talking now at the end of the 90s and early 2000, many victim witnesses were asked to come and testify. For some of them, in the Milosevic trial, for example, the villagers from Kosovo would need their passport issued for the first time in their life because they never left their village before. So it actually opened up a discussion. What is the purpose of having such an alien court in a faraway country for the constituency of victims that per definition ascribed the great expectation and meaning to the court? And I remember participating in some of these discussions with arguments against an international court with proceedings held in English. Yes, translated, but you know translation is always different than a direct engagement that victims can follow it. And I was sort of bouncing and shifting my positions on that for a very long period of time in this discussion, going from yes, international criminal court was the only workable and possible solution at the time and then regretting the end physical, end mental, and cognitive, and even emotional distance it created to its biggest constituency in the region itself. And I must say that bouncing from one argument to another, I completely came back to my initial understanding why an international court was important. And not least for the reasons important for the title of this conference, if you would have had national courts, they would never have the power international UN court had in compelling member states to produce needed documentation. And now in retrospect, the importance, and you might remember, some of you might even remember a huge discussion with criticism of Yugoslavia tribunal being a UN court because UN forces were forces on the ground when the most of gravest crimes were happening. So the whole idea was how could UN found a court while UN forces were directly or indirectly or implied in the crimes. But when it comes to collection of documentary and other evidence from states such as Serbia, the best evidence coming from state institutions came because of Yugoslav tribunal as a court was founded in chapter 7, chapter on the security of the UN that compelled every member state to cooperate. So when it comes to differences in international and national jurisdictions, my argument for international would prevail not least because of excess and possibility to compel states to cooperate. This cooperation, of course, didn't go very smoothly. And states would always find a way to avoid cooperating fully because, as you know, states still are sacred cows of international criminal system. And a non-cooperative state will be, by default, protected by other democratic states because they might find themselves in the same position and be forced documentation in other criminal procedure or mass atrocities. So there was also the understanding among states that they needed to protect each other from international courts such as ICTY. But still, prosecution would have never succeeded in assembling so many documents from state archives, not just from Serbia and the states who were directly involved in the conflict but from other states as well. So this would be my most compelling argument for international courts. There is that definite advantage in terms of collecting documentary evidence precisely because the ICTY was a UN charter-based court. Okay. And Patrick, you also have many experiences in other different countries as well. So could you also perhaps compare your experience from the previous experience, from the ones you had at the ICTY? So I've been in five trials and I supported a sixth one. I was one of the co-authors of the expert report on the sixth one. And three were domestic. All three of those were in Guatemala. Then there were two trials at the ICTY and then one in front of the extraordinary African chambers which tried Hashan Abe, the former president of Chad. And my experience has been that as an expert, I have no ability whatsoever to comment on the jurisprudence or the quality of the legal reasoning or anything like that. But as an expert, I found it striking how variable judges are in terms of whether or not they care what the answer is. So when I give talks about statistics, I often ask at the beginning of the talk, I say, hey, we all need to make graphs. How many of you care if the answer is right? And some hands go up and other people chuckle nervously. But many times people want a graph or a statistic, but they don't really care if the answer's right. Honestly, in the five trials I've testified in, in only one of them did I have the sense that the judges didn't really care what the answer was. In the others, it was striking how thoughtfully and carefully the judges questioned me. They had, when they didn't understand something I was saying, they would either interrupt me or during their question period they'd say, look, what are you trying to say here? What's the point of this? And they'd ask very subtle, very thoughtful, often very provocative questions that I had to answer. Those were the really good trials. And they were also, I think, the trials that produced, well, of course, in the Milosevic, we don't have a finding, but they produced the best outcomes intellectually. We got two really good answers. But if the judges don't care what the answer is and the prosecutor is unable to persuade them to care, you can get a bad outcome. And that's also a problem. Even if the judge's legal reasoning may be very, very high quality, if they aren't willing to do something that's uncomfortable for them, and science is often uncomfortable for people who have not studied it, they will have a bad outcome. In this trial, the judge joked, five minutes into my testimony, he told the joke, ha ha ha, Dr. Ball, Dr. Ball, wait for a moment. You know, some of us studied the law because we were poor at math. Patrick, may I interrupt you? It certainly wasn't, Sir Jeffrey Niskey was not judge as well, but he went to Oxford first to read math. And he never studied law at Oxford. So this elaborate, I know that this is what I noticed, this elaborate way you were allowed to present your methodology comes from his genuine interest in mathematical abstraction. Sir Jeffrey got that when I explained this stuff to him, I have told the anecdote many times, I have never explained a complicated statistical argument to a non-statistician who understood it more quickly than Sir Jeffrey did. He got it in an instant and began asking very good questions right away. And I think that was what led to it, but also the judges in this case, all three of the judges, Judge May, Judge Quan, Judge Robinson, all asked me challenging and thoughtful questions in the most which case. And so that I think led to a good and rich common understanding of what I was trying to argue. In this other case where the judge dismissed the math in his finding, in fact, he made a series of findings that were unsupported by any actual scientific reasoning. He, the defense had another expert who said things that were absurd, that were completely non-scientific and would not have passed even the slightest bit of review, but that turns out to be the judge's finding because he didn't really want to think about the math, he wanted to dismiss it. So you have to think when you're in a judicial context, when I said as part of my preparatory materials for this, your audience are the judges, that's right. And if you can't persuade them or if they don't want to be persuaded as an expert, there's very little you can do, you're stuck. But you know, I'm four for five. So in four of the five cases, we've had judges that really did want to know the answer and ask really good questions. I'm especially struck by this most recent case in front of the extraordinary African chambers where the argument was somewhat subtle and the judges asked, all three judges, asked terrific questions. And in their findings, they wrote very useful interpretations. They linked the scientific evidence to some of the legal and eyewitness evidence and argument in a very thoughtful and interesting way that I had never foreseen before. So you can get really good outcomes, but it's, you know, you can also very occasionally, one time in five, get a bad outcome. Yeah, well, it's interesting listening to Patrick and because you know, when I was not expert witness, but I worked as a non-lawyer with the lawyers and very often they would tell me and my colleagues from a team that was called leadership research team that was part of the prosecution. And we were asked for different trial teams to help lawyers in another team of non-lawyers and non-investigators with military analyst teams. The military experts who would guide lawyers and investigators in understanding military component, which was a huge component when it comes to implementation of the plan and commission of mass atrocities, obviously, because military infrastructure. So it was interesting, Patrick was saying he was trying to convince judges, it was his audience. We from my team never tried to do it. I even, I didn't know the names of the judges. They were so far from me, I needed, I only needed to persuade my lawyer. And it was lawyer's skill then and it was completely different skill to convince judges. And I would lose patience when we say, but judges wouldn't understand it. They say, what do I care? I understand it. And I'm telling you what I understand. So use your legal skills rhetoric to convince them. So if you felt that you were communicating with judges, you were missing a very able lawyer in between to translate your language through his or her questions to the judges. So I wouldn't go immediately to address judges. I would first go and say, hey lawyer, you are having as a prosecution lawyer. It is, you are soliciting testimony from me. I'm your witness. So you should know what you want judges to know. And that was fascinating and interesting filter for all of us who basically had the same goal that the most important arguments were heard in the courtroom. But without very skillful legal technical language between the witness and the judges, there is no success. So Patrick, you mentioned you were kind of focused on the judges when you were making your presentations, but I assume that you definitely discussed the matter with the lawyers, to say the prosecutors before you made the testimonies. And how did that work out? Different court contexts have different ways that that happens. In the context of the Milosevic case, I worked pretty closely and I had several meetings with Sir Jeffrey and with others in the team and I explained everything and people got it very quickly. People understood the argument. And indeed, in court, I was very ably led in my direct testimony. That's not always how it works. In other legal contexts, I've had very little discussion with the prosecutors, maybe only 10 or 15 minutes. And it's not clear to me if they understand what I'm saying or maybe they do. Well, I mean, it's both not good and good. I mean, in some sense, as a scientist, my job is not to support the prosecution. My job is to present the science. And so I'm actually comfortable with that context. Now it leaves me very, very vulnerable if the defense starts saying things that are nonsense. When the defense starts, no, like Milotinovic. I mean, Milosevic said crazy things, but Judge May was having none of it. And I don't really think that was a huge risk. But in Milotinovic, the defense said crazy, ridiculous, nonsensical things and the judge just took it as though it was reasonable. I mean, it was disastrous for me. I mean, I understand there was a good legal outcome, but it was disastrous for the scientific quality of the reasoning. And so that was the one that really failed. In the Guatemalan cases, it's actually not actually the practice that you work that closely with the prosecutor. The prosecutor gives you a document that says there are hypotheses are. And then as a perito giving a peritaje, you're presenting scientific findings that are independent of the prosecutor. And I like that model. I think that model's good. And so there my job is not necessarily to persuade the prosecutor. The prosecutor decides to call me. But my job is to then present the scientific expert results as science to the judges. And so my job is to persuade the judges. In the extraordinary African chambers, it was also similar. I had relatively little contact, a little bit, but not very much with the prosecutor. And in fact, because of that, it was not an Anglo-American adversarial system, but rather continental system. My connection was with the investigative judges rather than with the prosecutor. And so the investigative judges who were also a neutral party in the trial were the ones who said, look, these are the things that we want you to speak to. And I said, okay, I can speak to this, this, and this, but not this, but how about this? And they said, okay, yeah, do these pieces. And we agreed on the general structure of what I would present. And then I sent them the expert report and then I showed up in court a year later. So it's, I think different court structures have different rules about the way the expert and the prosecutors work together. Yeah. I'm also curious, like, for example, like crimes against humanity has certain legal definition and prosecution has the burden to prove that those elements were fulfilled. And when either of you were actually working on collecting evidence or presenting evidence, did you actually have that kind of legal issue in mind or was it something that was just left to the lawyers to figure out? Before, well, one of the thing is when you start working at a court dealing with mass atrocities and if you are a non-lawyer, you come to work there without any pre-knowledge. There is no training you once get the job there. You get all the tasks and you have no idea where you're heading at. I remember my first day coming to the office in May 2000. I got a chair and computer and they say you are working on a Bosnian part of indictment for Milosevic and I said, okay, no guidance, nothing. So this widespread systematic, which I later understood just because I was so bluntly honest about my ignorance, I would send email to a team of 150 people and say, what is that? And people would come to me and say, weren't you embarrassed? I said, no, I would be more embarrassed to work without knowing it. So I asked and I got the answers. So for non-lawyers, it's a hard way to learn these things but then it's easy because you learn for practice. You hear things at the meetings and you are sent in the field and you have to know for yourself because you can overwork. You can be all over the place while you can focus yourself. So talking about widespread is systematic with test of proof beyond reasonable doubt, by the way. That's what I've learned immediately as well is the standard of proof for crimes against humanity opposed to genocide where you have to prove intent to destroy in part or in whole group of ethnic religious and so forth. So what do you do with this? Well, to connect it with Patrick's expert testimony, I remember when he came to testify or when he went to visit several times, my team was called to listen to his lecture. So he had a lecture and it was of course, not mandatory, anyone could come working on Milosevic case and of course I came. I was in the audience, it was a small audience. Remember this room in the circle and I remember him talk and I remember talking to Sir Jeffrey and other lawyers about it afterwards. He gave us, from all these interesting statistics you now have seen, probabilities and so forth, he actually gave us very concrete dates where he saw the huge movements of Kosovo population leaving Kosovo and that stuck in our minds and this is why this widespread and systematic comes from many, many places and you as a lawyer and non-lawyer, you then very easily recognize collaboration of this evidence from other places. So Patrick and I talked this morning about development of evidence and during the defense part of the case, there was a colonel, Vlatko Vukovic, who was then part of Yugoslav Army and we got hold of his wartime diary and this widespread and systematic that we already knew from statistics without Patrick even being there at all these different places, we saw the movements of people and then you read it in the wartime diary where this colonel is mentioning exactly the same places, exactly the same dates and at the end of every page, he would write 1,000 expelled. Next page, 1,500 expelled. So yeah, widespread and systematic comes not only from one place, it comes if you know what you're looking for, you recognize it in almost every evidence including of course victim statements, testimonies and so forth, but another thing what Sir Jeffrey was saying and I think that is maybe what Aitzen was referring to in the first place was how do I prove it? How much is enough? For Milosevic case and I think Judge Kwon is mischievously grinning about that because I do think there was a huge, there was difference in approach that judges would have liked to see from Milosevic case because as Judge May said, you can't have so many witnesses, you can't have so many crime scenes, but we had to. Why? Tomorrow I will show you maps of counts. In Milosevic case, there were 66 counts spread over three different conflicts in time and geographic area. And if we needed to prove and widespread and systematic and intent because we had genocide charges as well, and if we wanted to show that all three wars are part of the same criminal intersection to create post-Yugoslav-Serbian state that extended its borders to the West in the areas where Serbs were never majority, but that these territories had to be conquered, we needed to include as many counts as we could because geographically any time we had to prove that was not one time mistake that people were killed, but that was premeditated plan. So we were including all counts that would cover this geography. But then the question would be, each count had sometimes 15 charges. And that becomes then a problem. If you want to prove all the charges within one count, you would have for each count each geographical place 20 witnesses. So we lost some charges, but we hardly lost count. And this was our calculation. Do not lose count, lose charges if it's necessary to lose. So I hope it's not too technical to complicate it. And I hope Judge Korn would be able to reply to this. Perhaps we should invite Judge Korn to our short explanation, but no, it's all right. So Patrick, did you also have this kind of prior legal knowledge that helped in putting together your evidence? No, not at all. And to this day, I don't really understand most of the law. And I don't really think it's my job to do so. What my job is to do, and this is very often the position statisticians are in, is that statisticians very often are working with substantive experts in other fields. We're working with biologists, we're working with survey people, we're working with people doing medical experiments. And our job is to figure out what is the question these people are trying to figure out? What is the fact that our counterparts, our partners, need to understand? And so we have to listen to them, listen through their substantive complexities and figure out what's the question. So I'll talk, I'm not sure if it's tomorrow or in my other session, I'll talk about what did genocide mean to a statistician? Now genocide has all kinds of complicated legal meanings. That to a statistician, genocide means, okay, you have this area and a lot of people got killed, but not just a lot of people. People of this kind did get killed and people of that kind did not get killed. You need a big difference in the rate of killing between these two people. Now that's not a legal definition by any means. In fact, I've heard lawyers argue to me that you can have a charge of genocide without anybody getting killed. Maybe so, maybe not. But for a statistician, this notion of the difference in the killing rate, what a statistician would call a relative risk, is for us a useful index or indicia of genocide. So in the case in Guatemala, we showed that for an indigenous person living in the three counties in which the prosecution alleged that the army had committed genocide, the probability of being killed was eight times greater than for a non-indigenous person living in the same place. So the relative risk is a factor of eight. Now that took something like 11 years of research to come up with, but that little statistic is not proof of genocide. And I would point out that none of the statistical reasoning I've presented in any of these cases is proof. It's all circumstantial. It's all the kind of pattern we would expect if the prosecution's hypotheses are true. If what the prosecution is alleging are true, these are the statistics we would expect to see. We see them. That adds a little bit of weight to what the prosecution is saying. But it is not proof. And it's not proof because there may be some other pattern that we have not considered which could explain the same observations we make in reality. This is a very complicated sort of epistemological problem. And in fact, as I've reread the transcript from the Militinovich case several times, it was the heart of the misunderstanding between the prosecutors, the judges, and me that led to such a bad outcome in that one case. The judges thought I was there to prove something. What I was really there to do is disprove something. Statistics is excellent at showing that an explanation could not be true. Indeed, it could not be the case that the killings in Yugoslavia were caused by NATO. It could not be the case that the killings in Kosovo, excuse me, were caused by the KLA. Those explanations we can reject because we can show that the patterns of bombing and the pattern of KLA activity were completely inconsistent with the patterns of homicide. So it was impossible that those two actors caused the killing. We did not prove that Yugoslav forces committed those killings. We could observe some very interesting coincidences. So I hope you'll stick around for the afternoon and I will in fact make those observations. But we can't prove things. And statistics is not really in the business of proving things. So we're in the business of disproving things. And I think that's also very useful in prosecutions. And when we're at our best, we're able to convince the prosecutors and the judges that, hey, this is a way of narrowing down what the debate can be. Years and years ago, back in the 90s, a guy who was then a scholar of transitional justice and has since become a Canadian politician, Michael Ignatiev, once wrote that the purpose of truth commissions is not to establish the truth. The purpose of truth commissions is to narrow the range of permissible lies. And I think that very often, expert testimony in these trials is of the same kind. We are there to narrow the range of permissible lies. And this enables the prosecutors to get much closer to the truth. We narrow the kinds of nonsense that the defense can throw out. So I add that as a kind of complex, a roundabout way to make that point. But statistics should never be the only thing prosecutions are presenting. And of course, in none of these trials, is it the only thing. I think that statistics in human rights work is a footnote. It's not the headline, it's not the lead paragraph, it's a footnote. But it's a really important footnote, and it has to be right. It's a footnote that sets a big picture that narrows the range of permissible lies that gets us close to the story. As Nanna said, it gave them a starting point that then they could investigate more deeply and show something like finding one of the perpetrator's war diaries. Or saying, hey, we're looking for people who were expelled from Kosovo between the 10th and the 17th of April. That was a huge wave. Let's go look for those people. Because there weren't so many people before the 7th. In fact, from the 4th to the 7th, no one left Kosovo. 4th of 7th of April, 1999. So I hope that's somewhat eliminated. It's very interesting that you say statistics is a circumstantial, somewhat negative, it provides a negative explanation for basically disproving things that some of the claims made by the defense or other human rights abusers. Nevertheless, I would say, as we've seen from the movie clip, the defense will try to argue counter your arguments at court and outside. And how did you prepare yourself for the cross-examination from Milosevic or Militinovic or other defendants? Well, in Milosevic, because I was very well-prepped by the prosecution team, I had a good sense of the kinds of things that were likely to be said. And I thought I was very, very well-prepared. And the next trial I was in, the Militinovic case, I was much less well-prepared. I was not prepared for people saying things that were scientifically meaningless or ridiculous or nonsensical. It had not occurred to me that we were going to leave the bounds of logic and say things that were nonsense and that the lawyers would say these things with earnest, reasonable voices and straight faces. And they were mathematically meaningless, absolutely absurd. And I was unable, during the course of that, to explain how absurd and meaningless the things that were said were. And then the defense brought a defense expert who said things that were completely and meaninglessly absurd. How do you respond to that? I'm not actually, even to this day, what, 11, 10, 11 years later, I'm not entirely sure how to respond to it. But I think that, and it's right, you have to coordinate with the prosecutor. You need to enable the prosecutor to cross-examine that kind of nonsense more effectively than we did. And so if I've learned something, it's that I have to work more closely with the prosecutor when I can. It's not always possible, as I said, in the Guatemalan context and in the extraordinary African chambers. It wasn't really available. That's not really how those courts work. So we got lucky. So Nina, when Patrick was taking stand and we lost, which basically told him, like, it's a fabricated figure. Did you expect that or, I mean, did you prepare for that kind of situation as a member of the prosecution? Well, in such cases, even any accused representing himself goes beyond legal narrative and language. Prosecution hardly needed to intervene because it was so obvious that judges would make up their mind. But different thing is when a defendant is represented by professional lawyers and if you are prosecutor and you have, we call it testimony in chief, if we call an expert, is prosecution called an expert, testimony in chief was so important because it lays a ground for important factual determinations. So you don't leave the other side ruin your witness or credibility of the evidence so it would not be an expert witness, like Patrick and any other, should be so confident sitting in his chair or her chair, knowing that prosecution would object and lead the evidence back into the court and so that the judges might have easier way to make determinations. So we were never too concerned about Milosevic going too far because it was so obvious what was he doing. And I remember Sir Jeffrey always telling us, don't fret if I sit and do not object too much because if I object too much, it would be like Goliath against David. Let him go and make all this nonsensical observation himself, I would sit in my chair. So he would warn us because it was definitely his strategy but it wouldn't be his strategy if the accused would be represented or many accuses in Milutino which case by different lawyers than prosecution would try to go back to the very strict rules, what is permissible, what is not and not protect a witness but protect the integrity of the evidence because testimony in chief is very important because it shows what the strategy, the case of prosecution is what factual basis prosecution is relying upon. But to go back to Patrick's mixed feelings about his expertise at two different courses, ICTY, if I may I would make a very general observation starting with again yet another disclaimer that I'm a huge admirer of demographers who worked at the ICTY and certainly an expert witnesses and our permanent prosecution team consisting of a formidable lady, Eva Tabor, because what happened with all of them and I worked very closely to them to Eva and I will show you in my presentation some of immensely valuable discoveries about conflict just done by rigorous scrutiny of available data on demography is that demographers experts moved the statistics and demographies and human losses during the armed conflict to a completely new dimension and as Eva Tabor would inform me and we talked a lot about it, they now understand that there is no understanding of the statistics without contextualization and at the beginning I think Patrick but certainly not Eva didn't even dare to put any of their data in a political and historical context. At certain point, you all realized that it's necessary and this is a huge contribution which might not be readily understood and available to people who are not intimately part of the developments, how do you present the best figures not to the judges, to the victims and basically to world citizens because now we all care what's happening with mass atrocities and what statistics brought in the through legal process. I don't think the statistics would have been developed in such a pace if it were not for international criminal trials because they were suddenly asked. We had three or four statisticians demographers working at the prosecution for now 25 years. It's a huge human capital that now needs to be made available to the world to understand what it is and I think it puts development of statistics at a higher and good level to be and to be shared with non-mathematicians as well. So that was I think one of the huge impacts and results of criminal procedures we had. And I just mentioned about the world citizens and how they view about the trial as well. And I kinda wanna pivot to, we've been talking so far about the things that are happening inside the court but I guess you also had in mind the court of public opinion outside the courtroom and I think Patrick you said at the end of the clip very end of the clip that the judges and the audience are the ones who that you care about and you specifically mentioned the audience and Jeffery Nice also referred to the kind of evidence that will stand the test of time which kind of something that Judge Kwan referred to this morning citing Robert Jackson from Jürgenberg. So when you are presenting or preparing the evidence how did the, I'm sure you had the public opinion in mind as well. So how did that figure into your preparation and presentation? Maybe Nina can go first. Once you work at such a complex and high-profile cases Lovona Milosevic you don't have time to think even about judges let alone audience because you are always working on yesterday's deadlines. So that does not concern you at this point very much but what was so interesting when the trial finished by a premature death of the defendant I suddenly had more time to look what's going on because when you work when things become so obvious to you you think that the whole world understands them and the whole world follows it. And then I understood first of all that the international criminal justice system criminal justice system was so readily accepted by liberal world citizens not just the victim communities immediately but they had huge very unrealistic expectations. I don't know where these expectations came from but official ICTY sites certainly didn't help to temper them because in their first days and years they announced the end of impunity, end of mass atrocities and so forth, et cetera. That was a goal actually. Yeah, but then they changed it obviously because if you read any justice department and national jurisdiction what they amend it is they would never say we are going to extinguish crimes ever forever. The only expectations we can have is to control crimes because political violence is with us since people form some sort of political entities so we won't extinguish it. So whatever we do we need to find ways to control it but this language, this rhetoric actually made everyone I expect miracles and then very quickly. So this was the first thing that I realized that audience expects too much. Second thing that I noticed when I started reading articles and reactions of people that that there is such a legal illiteracy among the general audience and certainly victims audience. For example, if you would tell associations of victims like mothers of Srebrenica former prisoners of war in Bosnia that of course some of the people were acquitted and should be acquitted because of procedural issues and breaches or I don't know what. They would never accept it. They said we don't care. The law can be broken. The regulations can be broken. He or she is guilty. We want him on her to hang. So there was some sort of expectation of expediency that would bring result and not respect our highest norms of a legal system. So, and then thirdly what was interesting for me is that the longer I followed the works of ICTY you do understand that slow justice might not be justice at all because after 25 years you have so many people who already died waiting for interesting verdicts. Secondly, you have verdicts that are utterly unsatisfactory for the audience who is there. So, coming back to if we would be able to start all over again, I think from my experience the good remedy would be to make outreach program of the ICTY as big as numbers and endeavors as the prosecution. It was almost like each prosecutor should be shadowed by outreach officer to explain what has been done and that the communication and interaction between defense and prosecution and judges should be displayed in a much more transparent way towards the audience. When I worked for the prosecution we were so scared off to talk to defense people of judges that I hardly dare to agree them because I thought I was breaking some sacred law. It was nonsense. You can be very professional talking to defense lawyer without breaching any law but it was some sort of spastic fear of anxiety that we should be so confidential that it actually hurt the institution in a way. There could be possible room for conflict of interest as well, so I don't know. Yeah, but these happen anyway behind the closed doors. If I have a dinner with the defense lawyer and everyone can see me, I prefer that than lack of transparency and then bad things happening anyway. So transparency is a huge, huge achievement of our democracy and it should not be shortcut for some cosmetic or artificial purposes of objectivity or impartiality because you certainly can work on outreach and you can work on outreach only when you include all the parties. Outreach shouldn't come just from the prosecution. Outreach shouldn't come just from the judges. It should be a joint effort to understand what is actually happening in such institutions. So the outreach would be my first attempt to improve the system. But also when you're actually presenting evidence in the courtroom or the instances where you had not just that you or Joffrey Nye said, not just the judges, but also the larger audience in mind as well, I can think of like, for example, some video footage that was played during one of the cross-examination. Well, you are talking about the Scorpion video. We are going to talk it during my presentation in greater detail, but this is a fascinating story and this story was certainly not done to influence outside world politicians. It was done for many internal reasons. And I don't know whether it's a good moment to tell you the whole story. Okay, we can save that for a later session. But Sir Geoffrey Nye showed that video without even informing the chief prosecutor at that time. And the only person who used it for self-promotion was Chief Prosecutor Khaled El-Ponte, but she even didn't know what the team was doing. And the reason why Sir Geoffrey used it is following Milosevic defense case started at the end of August, 2014. At 13 of September, we registered this video in our evidences and how we registered it. So we got it after prosecution part of the case. So we couldn't present it in chief through a witness, prosecution witness. So if you do that, you are in trouble as prosecution because such an important evidence cannot be used just like that in a defense part of the case. Why? Because you always have to show authenticity of the evidence, integrity of the source where you got it or originator, all sorts of things. And in cross-examination, if you use it, you don't have time to establish the basis for it. So we knew that judges would never accept it as evidence. And yet, because we got it in such a late stage, Sir Geoffrey thought and knew if he doesn't show it, it will stain the integrity of the trial because it was at the ICTY. And the story how we got it is a fascinating story because one of our investigators got a telephone call from a former witness, a lawyer, Bosnian Muslim lawyer from Tuzla in Bosnia Herzegovina. And he said, I have a client who wants to sell you a piece of evidence. And ICTY is not allowed to buy evidence. So the investigator goes back and he says, well, we can't buy anything before we see it. So the lawyer gives him the tape and he sees what's in it and he goes like, oh my God, what is this? So he makes a copy, gives it back to the lawyer, said, no, we are not interested. So we didn't buy it. But we have it in database. And this immensely important general, Obrac Stevanovic, a MOOP general, Ministry of Internal Affairs of Serbia who has made his whole career in Kosovo. And during the Kosovo war, he is very high up in the hierarchy being in charge of different units, of a unit for anti-terrorist actions. And he brings about 300 files as defense evidence showing all sorts of crimes committed by so-called KLAs, they call it, so-called Kosovo Liberation Army. And Sir Jeffrey comes to us and he says, he and another general, Boži Dardelic, could actually undermine our Kosovo case. So we need everything possible to attack the credibility and as a prosecution, we can use so many documents, so many things we didn't show in prosecution case for undermining credibility. So this was our path to show it. So first of all, Sir Jeffrey thought we cannot possibly face the world after the end of the trial when it appeared that this video was already in our database without showing it. Secondly, yes, we can use it completely to shake up Obraste Vanovic because he can undermine Kosovo part of the case. So we did it. But what happened later outside of a courtroom was beyond our power because chief prosecutor that year Karla Delponte was nominated with another, with Natasha Kandic as a Nobel Prize candidates for peace while she saw it for the first time from her office, watching it on the TV screen. But for whatever, we of course, it never became evidence in Milosevic case because we, judges rightly said that they would be needed to establish foundation. But what was so important that that single video triggered the most important post-conflict debate about responsibility in Serbia for the crimes in Bosnia-Herzegovina. So in a way, I call it transformative value of trial evidence that sometimes these sort of trials are important just because they're made for general public known the details who and how committed the crimes. And I think that was absolutely immensely important value of this piece of evidence. Thank you for all the behind the scenes story as well. For the audience, the Scorpion video that we mentioned, it's actually about a two hour footage of basically Serbian militia forces massacring people in Serbia Nietzsche that were initially played at the court and then it was played by, it was aired in the Serbian TV networks which caused this national debate in Serbia about its responsibility during the conflict. So Patrick, did you also have, did you also, was the public perception a big factor in your preparation or presentation of your evidence as well? You had experience in the truth commissions and other as well, so it's something, yeah. I would like to talk to that, but I wonder is that my close or are we about to end or do we have a bit more? Okay, then I would like to ask if you could punt to her and I'll excuse myself for just a second. Oh, sure. Okay. All right. Sure, now let's go to the move on to the next question. So we've talked about the public perception, but we also have a lot of civil society organizations, peoples from NGOs seated here. And I understand that they also have the NGOs or civil society has some role to play in the collection or preparation of evidence as well. And what will be the things to consider when the NGOs have this come across this kind of evidence that might play crucial later on? Or maybe perhaps something, things that they should not do if they come across such evidence. NGOs played very important role and for the establishment of the International Criminal Tribunal for former Yugoslavia as all other legal institutions later on. But as for collection of evidence and for Kosovo and for Bosnia, the reports by Human Rights Watch were used extensively. And why and how these people were already working on investigation and recording of the crimes on the ground before tribunal started. So we are talking about contemporaneously recorded crimes. They did extensive interviews with eyewitnesses, victims, even perpetrators, which were safely stored. And some of them like Fred Abrahams from Human Rights Watch was very important fact witness for the crimes happening in Kosovo in 98 and 99. But more importantly, why Human Rights Organization's work was important for defendants like Slobodan Milosevic, politicians in charge, is that every time they produced a report of breaches of human rights, they actually officially send it to relevant ministries and very often to Office of Slobodan Milosevic. And that was very important for prosecutions for evidence, notice evidence, putting politicians on notice. Jerry Labor was another person whose evidence was immensely important for us who would say that they warned politicians in Serbia what was happening in Bosnia and Herzegovina. And once politicians were put on notice and if nothing changes, if the crimes continue to happen, that was a great evidence in the court for the prosecution. So their reports were used, human rights researchers were used or asked to be fact witnesses. Sometimes they were just used to help investigators in the office to sort out evidence. But once prosecution would get, for example, a report in evidence, in the footnotes you would see the protective names of the witnesses. So if we wanted to go back to the original file of as we needed to ask special permission. So the confidentiality of research of human rights organization needed to be upheld because of their future work. Because when they work with the people on the field, everyone who would speak to them should know that in no circumstances their identity would be disclosed. And these are very complicated, very technical rules that needed to be respected from all possible sides. So are there certain things that the NGOs or civil society will have to keep in mind when they collect evidence? I guess this question kind of goes to both of you. I mean, so that it is actually presentable in the court that it is something that it will not be challenged by the defense or basically. So what will be like the basic rules that the NGOs will have to keep in mind when they're preparing this kind of evidence for later trials? In several of the cases that I've been in, we've gotten a kind of special ruling from the judges that what we're presenting will not have the same kind of chain of evidence that other kinds of evidence will have that we're putting together a story that involves collecting thousands or tens of thousands of individual stories, which we may not have available in some sort of way that could be interrogated by the defense. And so the challenge by the defense has to be a more technical challenge rather than a case by case data by data, page by page kind of challenge. And I think that that's probably appropriate for statistical evidence. It doesn't strike me that it's going to be feasible to have every deposition in a giant crate behind the expert when he or she presents the results that's just not really going to work out. Instead, the kinds of critiques that a defense expert could bring would be about the methods that were used, about whether or not the methods are appropriate, whether or not the results say what the expert, the prosecution's expert actually says the results say statistical evidence may be subject to complicated interpretation. And so maybe it doesn't say what the expert's saying but says something else. One kind of analysis that I would also recommend that both prosecution experts and defense experts who are using statistical claims to you is called sensitivity analysis. And in this, I did quite a bit of this in my expert testimony in all the cases in which we say, well, you know, we have the evidence here of say in the Guatemalan cases, we are using something like 22, 23,000 individual testimonies are part of what we're presenting here. Some of them are probably lies. They're not all true. I'm sure they're not all true. How many of them have to be wrong before the result changes? That's the statistician's claim to the concern. The statistician doesn't ask, are they all perfect? I know they're not all perfect. Some of them are wrong. Most of them are right. Where's the line? Now I'm making a claim, going back to the Guatemalan case, you guys may recall that what I was arguing is that the relative risk of being killed for an indigenous person was eight times greater than for a non-indigenous person. And I found that to be consistent with the prosecution's claim that acts of genocide were committed in these three counties of Guatemala. How many of the testimonies that we used as the foundation for our report would have to have been falsehoods in order for that result to change in any meaningful way, okay? And so we could subject that by deleting some of them, by changing some of them, by perturbing the data is the way a statistician talks about it. How much can the data be perturbed and still get to the conclusion that is consistent with the prosecution's claim? So rather than fighting about any individual case, we say, what is the big picture? How much do we think is plausible? Do we think that 20% of these are lies? Are 30% of them lies? And it turns out because of the way the math works and because of the way the whole calculation works, most of our findings are robust, which means we make the same finding even if 20% of them are lies, which is quite extraordinary, right? Because the results are not a close call for the most part. So we can have enormous amounts of garbage in our system and still get a clear answer. And we can make different kinds of assumptions about what kinds of garbage it is and still check to see if the answer continues to be solid. And that's the kind of way a statistician approaches this problem. Again, I think it's quite different than the way a journalist or a field analyst or a lawyer might approach it where you're worrying about each case, each story. We're not worried about each story. We know some of the stories we get are not true. Does it matter? Well, it depends how many of them there are. Let's check, let's test that, let's measure it, let's re-run the model, see if they still hold. I guess the big numbers or the methodology helps in that aspect to get it the more fuller picture of the events. The analogy that I used in court is that doing statistics is like listening to your friend talk to you in a crowded bar. People are talking and lots of people are talking and there's lots of noise. Can you still hear your friend? Can you still make out what your friend is saying? Well, the answer is maybe, right? It depends how loud the bar is. It depends how clear your friend's voice is. It depends if you have a pretty good theory about what your friend's saying. If you kind of know what your friend's saying, it's easier to hear what she's trying to tell you. There's lots of things that go into your ability to make out the signal from the noise and that analogy is actually quite good. There's ways that that represents many of the problems of understanding a signal, you know, a statistical pattern in the midst of noisy and partial data. Okay, so in the remaining 20 minutes, we will take questions written. Question that I punted before? On audience? Sorry, because I ran off. Maybe now is the time to, yeah, for... That's great. I'll return to that one. When scientists do a project, we don't generally do it once. We'll do it over and over and over again, trying to figure out if additional data or a different method or some other approach yields the same result. And for most of the cases that we've done, the case that I present in court is usually the third or fourth or fifth version of that project I've done. And I may do more after that case. Each of those projects probably has a different kind of audience. And so some of those audiences are truth commissions, some of those audiences are public reports, some of those audiences are chapters in the book that Human Rights Watch wrote about Kosovo after the trial. I mean, they're a series of outcomes. And so when we think about an audience, the specific way we frame a piece of expert testimony is to try to address some point that we understand the prosecution to be arguing. And we frame it in a way that we think is appropriate for court as best we can. But we may also write a scientific article about it and publish in a scientific journal or we'll make a public report with lots of glossy pictures or a website or an interactive, some sort of interactive tool or in the Kosovo case, we published the data sets. And this turned out to be crucial because the data sets we published for Kosovo have been used in many, many academic studies and dissertations and academic scientific articles and so forth. And so there's lots of different ways, different kinds of audiences when you're doing scientific work in this context that you can present it. And I think they're all valuable and we shouldn't give up on any of them.