 Okay, so it's just five after four. I think we're going to get going. Someone just kindly came up to me and said don't worry about the numbers in here because there's lots else going on but I just said this number is perfect for me, so But we'll keep the door open at the back just as people come in But we have until five. I'm not planning to take an hour. So hopefully we'll have lots of time for Q&A at the end So let me just I guess begin by thanking you guys for having me here today My colleague Alexios Mansourlis who's the director of the international fact-checking and network which is at the Pointer Institute Was unable to be here, but I'm happy to be his Canada-based minion here instead So my name is Dana Wagner, and I'm the co-founder and editor of fax scan the Canadian political fact-checker. I and I Like I said, there's gonna be I'm not gonna take up the hour There'll be lots of time for Q&A at the end and really please interrupt me with questions as we go So I'll just begin by quickly telling you a little bit about set fax can so we're a non-profit We're independent and we're non-partisan and we focus on Canadian federal politics So we've been operating since February 2015, which was the lead-in to the last federal election And we rate checkable claims on a scale of true misleading and false We also have a score of farcical and withholding judgment, but we tend to deploy those two scores more sparingly But today really what I'm going to talk about is bigger than fax can and it's bigger than Canada It's about the past year in fact-checking And I'm going to give you a bit of a lay of a lay of the land of what's been happening globally because it's really been a Very interesting year So I think everyone's probably familiar by now with the phrase that we're in a post-truth era I certainly reject this idea and I think a lot of you guys here would too I Really think that fact-checking is as relevant as ever I think that Kind of the casual relationship with facts that we've seen over the past year or two Combined with the ease of spreading misinformation Makes it really critical that we're exploring new ways of catching False information that we're exploring new ways of correcting it and especially disseminating that Which I'll talk a little bit more about But I'll really keep the focus on three areas Which is our understanding of how fact-checking works. So what the research tells us about our impact in fact-checking Some of the types of problems that we're speaking that we're seeing especially how and and why misinformation is spreading and Some of the ways that fact-checkers globally are kind of tackling these newer problems So to begin does fact-checking work? What is our impact and I think this is a good place to begin because we actually do know something about this We have a very good idea of what happens when humans meet facts And one important study that was published earlier this year tells us a lot about the way that we process political misinformation and corrections So before I explain to you what's happening here in this graph the main takeaway from this study is that fact-checking Changes our minds and our beliefs but not our votes So this is still a very good and positive finding So what these researchers did here Sawyer Thompson and and her colleagues? I they worked with just over 2,000 participants and they got them to rate for factual statements by Donald Trump and for inaccurate statements And they got these participants to rate their belief in these statements on a scale of zero to ten So zero being I don't believe ten is is I believe So some examples of the inaccurate ones included claims that like employment in the US is as high as 42% That vaccines cause autism like those types of inaccurate claims Participants were then asked after they were asked to to Rate their belief they were shown corrections Or or confirmations if it was a true statement, and they were asked to rate their belief again So here's where if I could just ask you to look at the the lines on the bottom of the graph here these are the inaccurate statements, and you'll see that if someone scored their initial belief at around like four or five After they were shown a correction They expected how we want people to behave how we expect them to and kind of rationally how we we think humans should behave after shown A correction is that their belief in that incorrect statements decreased now the interesting thing that the researchers found was that That the effect of this was more pronounced in the immediate So they also tested people's beliefs in the same statement one week later and found that there had been a bit of a movement back to the initial belief Not huge and this is something that the researchers has have said they're going to continue testing But there still is a drop in that belief So this overall is a good thing that facts do change people's beliefs and Another interesting thing that these researchers found was that partisan preference didn't matter so you have the same Ability to correct your belief whether you were a Democrat Whether you were a Republican who supported Trump or whether you were a Republican who supported like another candidate So again, that's a very positive finding So another thing we know about fact-checking is that it does not backfire and I'm curious here has if you could just show me With show of hands has anyone here heard of the backfire effect? Okay, so a few people. Okay, perfect So for those of you who didn't know who don't know a backfire is this comes from a pretty significant study on the relationship between facts and partisan belief And this study on backfire was done by Brendan Neon and Jason rifler and this was in 2010 and this study of theirs focused on a single issue And what they did was kind of similar to the previous study. They tested participants On a statement about weapons of mass destruction in Iraq and these were conservatives who they tested and after showing These participants the fact that in Iraq there were no weapons of mass destruction What they found was that after this correction people actually Believed more in the fact that there were weapons of mass destruction in Iraq So they found that fact-checking didn't improve understanding it reinforced mistaken beliefs hence backfire and this was cited Way too many times. We took it as fact. We reexamined really like how we were operating Funders were questioning whether fact-checking was working. So it was a pretty depressing study But that entered these two other researchers Porter and would and what they did Ethan Porter and Thomas would they tried to replicate this study and they couldn't they could not replicate backfire And so again before I decide explain what's going on here What they did was they tested a much larger participant participant group of over 8,000 people and they tested across 36 different topics And they really tried to find backfire. So they were looking at statements not just from Trump. They were looking at controversial Democrats controversial Republicans more neutral ones across the political spectrum and the results are graphed here So you'll see the uncorrected claim on top Decreases and belief score once corrected and this again is regardless of ideology which is across the x-axis here So whether you're liberal middle of the pack are conservative again People are behaving rationally their belief in untrue statements decreases. So overwhelmingly There's no backfire and this tells us again that people are influenced by factual information Even when it challenges there like previously held partisan commitments And I like this quote from Ethan Porter one of these researchers So again talking about this common perception that we're in a post-truth political era that the American public is immune to factual Information we conducted study after study. We found that no one was exhibiting backfire across any issue So what about the politicians does fact-checking work as a deterrent? We definitely need more research on this question as far as I'm aware of there's only one kind of supply side analysis And this was a study that was done back in 2013 and interestingly it was done by the same researchers who did backfire Neon and rifler, but they're actually quite respected academics as well. So this study that they conducted they sent Letters to I think it was about 1200 State legislators in the US and these letters reminded the state legislators about fact-checking and reminded them about the reputational impacts of fact-checking and if they had their credibility question what that might do to their reputation and they found that the legislators who Got these letters versus ones who didn't and ones who got like a placebo letter Were more likely to score better on accuracy indicators and that was measured through various ways one of which was like whether they had a claim In the coming year that was questioned by political fact or Washington Post or other fact-checkers So again, this is promising. It shows that politicians really do pay attention Now there's also the question of how politicians behave after the fact and really I haven't seen any studies here So after a claim has been debunked. What do politicians do? Do they correct themselves in here? We just I think really have anecdotes at this point But they're fun to go through so what I did was I asked some fact-checkers just globally about what they thought some of their most significant fact checks have been over the past year and Consistently they forwarded me some of the work that had prompted a politician to retract a statement that they had made So I'm just going to show you a few because really I think they're quite glorious So first from Doral up higher, which is a fact-checker based in Turkey This was a check that they just did in December and this comes from President Erdogan Who was quoting an OECD report and he said that according to this OECD report Turkey is now a high-income country but Erdogan took that information from a newspaper and the editor of the newspaper in Turkey had actually Misinterpreted the OECD report and what it did was project for scenarios for the future for 2030 What would happen and only in only one of those scenarios was Turkey going to be a high-income country? So they corrected this and Erdogan actually retracted it and he came out and said You know what Turkey might become a country that is high income in 2030 if all goes well This example here comes from Pajela Politica in Italy And they had a great segment in the spring where they actually had on national TV the former Prime Minister Matteo Renzi who you see here with his back towards us And so he was on one of their live fact-checking episodes And Pajela debunked a claim of his on-live TV in front of him And this claim was that during his tenure as Prime Minister The scale of GDP growth in Italy was unheard of so he claimed that it was a Scale that was larger than seen across the entire eurozone and this was in fact not true There were precisely five other countries that had a larger leap than Italy as Spain Ireland Cyprus, Slovenia and Greece in that period that he was PM So it was just like a fascinating episode to watch him kind of cringe uncomfortably in the chair, but he corrected himself He actually did something interesting though, and I think a lot of politicians do this and it's something interesting that we have to watch He started saying like okay, you know, I stand corrected But you know Italy really can't be compared to those other countries because they came from a much deeper slump beforehand. So, you know Essentially what he did was apart from those five countries that prove me wrong I'm right and kind of went on from there But anyways, he changed the parameters, but he still corrected the record And finally just one from Lupe Lupe in Brazil They caught this is a more recent one They caught a series of contradictory statements by the ex-president Cardoso who really remains still very influential in Brazil and he had for a long time supported the current president Michel Temer And after these series of contradictions that Lupe exposed just about kind of his wavering support for some things that the government was doing Cardoso publicly changed his position of support for Temer And this was really the first time that he came out and publicly did this And I don't know if anyone knows but Temer currently is Being accused in a pretty large-scale way of corruption in Brazil. And so this loss of support was a pretty significant hit to him now Moving on to the second topic and I'll return to some examples but I want to just quickly talk about some of the problems that we're facing and in particular some of the problems that have been kind of rampant over the past year and a Lot of this really circles back to fake news And I can't take credit for spelling it this way this comes from a colleague, but I think it really reflects that fake news Like it is an awful term and I think that's because like it's not precise And we've also really seen it be deployed by politicians as a way to really undermine the credibility of critical reporting So what do we mean when we actually say fake news and others who have spent a lot more time thinking about this Have tried to kind of shift our language talking about misinformation and disinformation So misinformation is kind of tends to be defined as more inadvertent sharing of false information So what like you don't necessarily do it deliberately. You might not know you're doing it You hit retweet and you're part of that misinformation ecosystem So even if you remember that earlier example of president Erdogan in Turkey, right? So he's sharing information that he didn't really deliberately try to mislead people Versus disinformation which would be the more kind of deliberate and really often kind of online LinkedIn networked form of sharing misinformation or sorry false information And one of the better taxonomies that I've seen for the types of misinformation comes from a colleague at first draft news Claire Wardell and Claire argues that there are seven different types of of this mis and disinformation And she puts them on this scale. That's kind of it's loosely measured on here Along a scale of someone's intent to deceive So you'll see like from satire and parody on the left of here going over through false connection misleading content false context imposter content Manipulated content and then over to fabricated content on the more deliberate end of the scale on the right and Intent here is key Because it's really linked to the ways that content is created and shared So for instance like Claire argues and I I tend to agree that one of the most concerning forms of misinformation That we're facing are these deliberate campaigns Typically driven by political influence and these types of campaigns are the one that were the ones that we're seeing again linked to like bot Networks and troll factories I probably don't have to tell many of these people Many of you guys hear about what these are, but if you want to understand more about trolling I point people to there's there was a fantastic episode By this American life in in January that they had interviewed some people at this Kind of celebration for for people who call themselves trolls in the US and one of them Was boasting about the fact that they had memed Trump into the White House so these are fairly coordinated and and sophisticated movements and Part of the reason that they are so dangerously sophisticated is because I think of the way that our brains work So these coordinated and consistent messaging Forms really do fool our brains so with like Overwhelming amounts of information out there like our brains do rely on shortcuts and when we see multiple messages Saying the same thing and reinforcing the same thing we tend to attribute more credibility to that piece of information So picture yourself sitting there seeing the same thing like five six times throughout the day You're going to be more likely automatically to attribute more credibility to that to whatever that piece of information is So how are fact-checkers responding to this this new kind of misinformation environment? And by the way, there are about 126 active fact-checkers as Counted by the Duke reporters lab here. I know there are many more kind of that we were talking about informally But these are the ones that are a bit more kind of formally organized as as distinct organizations So I just wanted to touch on about three responses that I think are pretty cool from fact-checkers Some neat forms of collaboration Automation in fact-checking and then standards So one really neat form of collaboration that we've seen over this past year I Comes from France and this was leading up to the French election Where media outlets were watching the rise of Marine Le Pen, but not only that they were watching this really a new form of Seeing like videos and images Deliberately circulated to the public that were completely false often manipulated content So 17 French media outlets decided to fact-check together during the election So they created this website called cross-check where they would actually publish decide to publish together instead of on like a site like Liberation they'd publish here on cross-check and You'll see that splash of logos on the side are all the media outlets that had verified this piece of information together and In so after the election and a bit of like a post mortem some of the journalists were talking about like why they think this works So well and and really there was consensus that this was a very smart way for them to have done business during the election One they think readers likely got more coverage when the media outlets were pooling their resources Because they had found that before they created this they were just completely duplicating their efforts like Across multiple platforms they'd be debunking the same videos and just all those Reporters that they'd be putting towards that obviously only one person could do it a few others would verify and they'd all be fine putting their logos on the story So more coverage and then second for from a reader perspective like seeing all these competing news outlets Kind of put their sample of approval on the same thing does lend more credibility to the story So just very briefly again because these examples are interesting What this story here is debunking In the video just on the story. You'll just kind of see like a freeze frame of the video and what's happening here This was something that was a piece of manipulated content that really mattered in this election campaign and in the shows Macron Washing his hands and this Initial claim was put out by a Twitter account and it declared that Emmanuel Macron washes his hands after he shakes hands with workers And it comes from a real video of him actually like using a hand wipe to wipe his hands But the longer video actually shows him at a separate meeting where he's handling eels like literal eels And then he goes from that meeting into a car and starts wiping his hands so It and it's just interesting because that rumor again like really followed him So there were subsequent like media interviews with him or just you know at different campaign stops There was just one notable example From a factory and a woman came up to him like just someone who was working there and just said oh well mr. President like are you are you good enough to shake my hand and and he would be facing this kind of thing constantly and it all started from this Twitter account and Again, like I just like this example because it goes to show just like this manipulated content is a really kind of Different form of misinformation that we're seeing spread So in a similar vein like another example of this collaboration comes from Norway and Norway has an upcoming election in September And outlets here were really anticipating the same Problems that we're facing other countries elections including France So manipulated content and just other forms of misinformation and three outlets here including the two largest online news outlets and the public broadcaster launched I'm gonna get factisk Which means in a region like factually or actually and the rationale for For putting aside competition in this instance like it really wasn't obvious to me like you start I'm just trying to pitch your Canadian news outlets doing that type of thing and I to be honest I can't so it was interesting to hear the rationale from the CEO Halle Solberg who Who was quite candid about this that for a long time Herself and she she's a seasoned journalist and other news outlets really recognized that they were not each other's competitors anymore That their competitors were actually Facebook and Google and other publishing platforms And they were competing with just other forms of information including misinformation another Just kind of I guess like innovative way to respond by some fact-checkers has been automation And the leader here, I think really is full fact in the United Kingdom Which has been developing some niche prototypes They also put out a report last year with just a snapshot of where we are globally with automation I know the type is small here But this chart comes from that report which was just released at the end of 2016 And they show kind of what's available right now and what's not yet possible just given our current tools And some of the things that are available right now that are actually being deployed by different fact-checkers including fact scan Our tools that monitor claims That can spot specific claims so not not actually debunk a claim But actually help us like isolate or isolate checkable claims from like really large amounts of text So if you consider like the hands-on transcripts or like long speeches we can throw that in a Tool claim buster for instance is one that we use and it actually gives us claims that that are verifiable So that just like and the development of this tool actually comes from MIT in the US And they had been talking to the founder of politifact and asking like what do you need? Like what can we help you by building and his answer was interns We just don't have enough people to actually go through and like find all the like political claims that are out there to help us check So that's one that we use And then some very simple checking of claims like with verbatum Text and things that have already been kind of checked before So some things that are not yet possible would be spotting claims that are paraphrased so non verbatum things and Definitely like non-text iterations of things so things that we'd be hearing on radio and TV and that kind of thing And to just illustrate some of the automation that's happening or under development right now So full fact is developing there's one tool called live that they're developing And what it would do is work in in real time to recognize a claim Including on TV or radio and it would be able to source a structured database and produce a mini analysis And so really the end user of this type of tool I've heard them talk about they envisioned journalists being able to use this So the idea is that they'd be instantaneously able to make a better judgment as they're listening to something So if you can picture like a politician being interviewed on live TV and they're talking about Anything like poverty for instance and then you know with the help of something right in front of them like on an iPad or TV The journalist is able to say Okay We know that there are like two different Datasets right now that are very popular to talk about poverty to statistics one tells its poverty is going up One tells us poverty is going down. Why did you quote the database that you just quoted versus the other one? Tell me about that And so really it's just trying to cut down the time between a claim being made and some being someone being able to like instantly rebut it Another tool that they're developing is called trends, which aims to track repetition So the claims that have been debunked, but are not going away. So really persistent ones. So That vaccines cause autism. Why is that continuing and and like who is perpetuating that? the third area here a bit of a plug for the international fact-checking network, but something that I think is is pretty significant and One of the ways to really respond to what's happening right now because there are a huge number of players players out there Conducting fact-checking and fact-checking some legit some not including politicians that we've seen use those terms as reality check on liberals blah blah blah reality check on conservatives And so the philosophy behind standardizing some of the work that we do is that fact-checking can be a pretty powerful Instrument of like accountability journalism But if it's biased or if it's partisan that really starts to undercut everyone's work And it can have the opposite impact and just increased distrust in media in general So we're now at 29 verified signatories and a number of other fact-checkers that kind of in the queue in the vetting process And I'll just end here with that kind of some questions about what's next and these are really open-ended questions Like for us at fax can and for a lot of our colleagues internationally One of the interesting things is that we've seen a lot of research out there that people seek out facts that are consistent with their Ideology, so we know that people do that. We don't yet know how to intervene So how do we get like that? We also know that there's a perception out there that Fact-checking as a form of journalism is more of a left-leaning thing that it really caters the audience and that There's a certain agenda that hasn't been borne out by any Research looking at kind of like the types of claims that we checked and the scores that we attribute to different folks But that's still an important perception. So how do we overcome that? There's there's a philosophy right now that That fact-checkers should Maybe try to pivot away from the focus that we currently have on politicians So at fax can what we do right now is we check claims by individuals So we say Trudeau said this false Andrew Scheer said this true What if we're doing ourselves more harm than good? So the idea here would be that the alternative would be something called issues-based fact-checking where instead We say oh, that's an interesting thing that Trudeau said about wages going up We're just going to do a fact-check about wages in general. We're not going to attribute it to a person It's going to be about the issue as a way to increase your readership and get over that kind of partisan block that might have someone either read a fact-check or not just depending on on the focus another Area here is like passive information consumption. So how to outsource the debunking that we do and this really comes down to like I guess it's about skills development and and should we have a better role in in trying to help people Have tools themselves to just increase skepticism increase critical thinking and that type of thing So not necessarily to go independently to source a database Although certainly that would be kind of an ideal type of world But even just to be more skeptical like how do we how do we teach something called emotional skepticism and? One big advocate of this Craig Silverman who's the media editor for Buzzfeed has talked about this and just How this seems to be something that we haven't quite understood that well, and we certainly don't know how to tackle where we're not Being skeptical of our emotions So I'm certainly guilty of this like if you can picture yourself sitting at home And you read something on Twitter, and you're just I rate you read this headline You're just like it just drives me crazy and you reach read it and you share it and The idea would be to instead to just pause and recognize that that might be a trigger for you Like just a really forceful reaction either positive or negative And to pause and to be skeptical of of your own emotions And then just finally another big topic here And I would certainly welcome feedback from anyone in this room because this area to me These are really big on it sort of questions right now in terms of what is the role of? The Giants out there Google Facebook snapchat. I was too nervous to put Wikipedia, but Wikipedia What is their responsibility? Some of the things that that really are up in the air at this point is Like should should we be expecting that a platform like Facebook actually remove content? Some content they already are removing some they're not Should we be expecting more help with tracing misinformation? So if anyone out there can do it can actually trace these little atoms of misinformation that go out there and kind of spin through like Networks like person-to-person connections because that really is how misinformation is spreading right now if anyone can trace it It would be like the Googles of the world How do we how how can fact-checkers be enabled that our connection or our corrections can kind of follow that same pathway? You know should we have help doing this or not? and just to give you an example of like this this Issue of whether to remove content or not like this is a fact check that we did recently We checked a petition that the Conservative Party has up online on Facebook, which is still up online and This is about the issue of I won't go into detail, but just so everyone's on the same page here Omar Cotter is a Canadian citizen a little bit of a more controversial figure in Canada. He was involved in a disputed firefight And then he was jailed for the better part of a decade in Guantanamo Bay and elsewhere and then finally he was brought back to Canada and There has been a settlement that has kind of changed over time, but Or sorry that there was a lot of civil lawsuit that him and his lawyer launched against the Canadian government for alleged participation in his imprisonment and mistreatment in Guantanamo And this lawsuit was just recently settled by the federal government Who had been relying on some major Supreme Court decisions in favor of Cotter and just Verifying this mistreatment of his as a fact so Whether or not you kind of agree that a settlement should have happened how large the settlement was There are just some pure facts in this case And one thing that this petition has or it currently says is that Omar Cotter didn't apologize neither should the Canadian government That's incorrect. So Omar Cotter has apologized. He apologized while he was going through a military Commission hearing in the United States. He apologized as this settlement was happening just much more recently so we rated this false and At the time that we checked this this petition had over 3,000 likes that had been shared over 600 times as Of yesterday those numbers had roughly doubled And so I'll just end here because this is a problem. So something that we check it's still up online And whose responsibility is it? Is it Facebook? Is it Google? Is it the conservative party? These are still questions that we really are grappling with right now. So I'll end it here and with these open ended questions. I Invite to your questions now, too Hello, I just want to say first of all, I think fact-checking is incredibly cool possibly one of my dream jobs That being said how are you and or how is your organization trying to reach out to people and organizations who are Not otherwise interested in fact-checking who aren't we're sort of apathetic to whether or not something is true And they're more interested in who has said it Yeah Yeah, thank you for that and I share your passion obviously for this field So we Certainly are not the leaders here in terms of the outreach that we've done We've recognized as a major gap something that we are looking into just how to reach people who are outside kind of our typical demographic and We certainly haven't gotten it right one strategy that we are watching right now is Something that's politifact in the US is piloting. They just got a pretty big grant to I do something that It's kind of like gorilla fact-checking. So they've got just gotten a big grant to go out into more traditionally conservative smaller communities Especially towns are not major cities and especially kind of rust belt in southern US and what they are planning to do is just Hold a lot of consultations a lot of town halls a lot of teaching kind of really the basics of debunking and fact-checking and Trying to understand if and how they can increase the readership and especially their engagement by Folks who are outside of their typical readership and so we're going to be watching the results of that Kind of pilot project pretty closely to see if there's anything in terms of even even the way that So one of the parts of this project that they're doing They've brought in some kind of independent consultants to look at the types of words that they use in there in their fact-checks and how things are positioned and portrayed and if there's anything like a I don't know certain words that that are more that Democrats might be more sympathetic to versus Republicans, right? It could be in anything in terms of the presentation and what we're doing So I think a lot can be learned from things like that, but we certainly haven't cracked this at all It was interesting that the Norwegian Co-operative was of news organizations Which might be for profits how in general are these fact-checkers funded? Okay, I don't have the exact figures, but yes the majority of us we just had a Similar to wikimani, wikimani are equivalent to this is called global fact and just in July we had global fact four So it's we're such a nascent industry. We've only had four annual conferences And we there was a survey that was just done of like Everyone's annual budget and where our funding comes from and the majority of fact-checkers Annual operating budget is under $10,000. So the majority of the 126 Existing fact-checkers so are working with very small budgets meaning a lot of them are volunteer run non-profit The majority get their funding from foundations. So it's not user-generated revenue or you know other profitable Enterprises, but that's not to say that it's not possible. So again, PolitiFact is is really one of the pioneers in a lot of areas and in in terms of revenue generation as well Like they just in the last year launched a subscription campaign and already have made I Don't want to get the number wrong but have made far more than they were initially counting on already just through a user subscription So it's really shown them that what they're doing people are actually going to pay for so I think that there's a lot of ways that we can explore different like revenue generation models that we have have not just done yet Have I did I answer your question properly? Yes, yes Yeah, and and and just to quickly touch on here. They're certainly not all like that. So the Washington Post for instance has kind of a Sidekick fact-checker, you know, but but it's part of its main business. So, you know, there's all different types of models I wanted to ask you when you first started talking about how even correcting a fact doesn't actually change someone's voting going back to that original thing Can you speak I guess that there's this a hypothesis, right that over time If your facts are consistently wrong your authority your credibility goes down and it doesn't seem like that's happening Right, someone can be wrong once or twice. Everyone makes mistakes But somehow this Even with fact-checking. Are you actually seeing people change their behaviors over time or their their beliefs and things or their Perception about whether this is this person is trustable or not Have you seen that kind of play out in fact-checking or is that too nascent to see now? So you're not asking me about the behavior of politicians You're asking me about the behavior of of the users of readers and the public Yeah, I guess I'm interested in that like facts fact-checking is great for like a point in time, but Misinformation builds over time and so to correct misinformation or disinformation You kind of have to do this at a consistent ongoing basis so that Whoever that person or authority is is no longer viewed as credible. And so have you seen that play out? That the effect of long-term fact-checking actually leads to something like that or is it too soon to see yeah No, excellent question. And and I think it is definitely too soon to see so just One way to answer this is that I didn't really go into too much detail, but you're right that one the first study that I mentioned just on That was analyzing solely Trump claims what they found was that yes fact-checking and corrections changes our beliefs, but not our votes and Just kind of one thing to say about that is that that's It sounds depressing, but it's actually not a bad thing because I think that as fact-checkers Our role is not to change votes It is to change beliefs and so I don't think that that's a valid end goal for us either others may disagree with that but I think that voting aside What is more important is is having informed policy discussions and policy debates And I think that there can be a legitimate and and complex and real policy discussions You know regardless of which way people are voting and of course within a party and among people who favor different candidates and that type of thing So if we're contributing to a more informed discussion To me that is a valid end goal and we don't have to go and we shouldn't necessarily want to go further than that and change votes In terms of what we know about If fact-checking can change votes, it's not necessarily that they found that it couldn't it's just that the research didn't bear that out So after corrections people were still unwilling to change their votes But an interesting thing and where the researchers in that particular study have said they want to go further is is Whether is if they can show people a more proportionate and realistic View of the scale of misinformation put forward by one candidate So Trump of course being the example here a good example What they did in this study was show people four statements That were factual and four statements that were incorrect So it's kind of like a balanced picture and and that could have influenced the participants Voting intentions just by you know kind of thinking like oh well on balance if half are right and half are wrong Maybe that's how all politicians behave that type of thing That's still an unanswered question if there's a more realistic portrayal of someone who is an extremely misleading Politician maybe that would change, but I think the answer is we don't yet. No Now to steer a whole nation let's say into a misbelief Takes a very skilled maybe team or experts very professional people and I guess they would be behind top politicians but I'm wondering is there also a group that is well. It won't be equally well funded like your group is not well funded I presume Even the big newspapers you were mentioning and the Toronto Star, which is one that I get to highlight it It's my preferred paper and I still read the paper version and I see all these corrections by one of the reporters on Trump but still it's It just seems to me that There's such a power behind this misinformation at the top political levels that even you guys can't Counteract it find out who's doing it for sure. I would agree with you like until we One of the questions that I've had since we started in 2015 that I still haven't I don't have an answer to and I Haven't seen an answer is how do we how do we raise the costs of misinformation? So until we make it more costly for a politician in terms of their reputational impacts and just you know Whatever might be considered as a cost for them and until we understand that calculation I don't know that I don't know that the deterrent effect is going to be borne out. So Unfortunately, I don't have an answer for your question, but it's a good one Hi I'm you did and I'm Norwegian and so I was just browsing Disks and now as we were speaking and I looked at The facts that have been checked here and I saw that most of the politicians that haven't gotten feedbacks that their facts were Facts actually they have corrected themselves, but there were also a lot of wrong information from Online magazines and webpages and what I saw there was most of them did not reply did not Did not correct themselves. So the politicians at least in the Norwegian context. They do Respond to this, but other sources do not Yeah, just very quickly. Did you know about fast fact is beforehand? Oh good. Good. Okay So this is really again promising like you're saying that the politicians are able to correct, but Yeah, in terms of whether or not other media outlets do I don't have like a I Can't cite anything that's you know kind of points to whether or not they're more likely to and what might make them more likely to But one strategy that I've seen that has been really effective in the United Kingdom like full fact They're they're quite a good resource team and they actually have dedicated I think it's a team of two on So on their fuller staff, they've got two full-time staffers Dedicating to chasing misinformation and getting that getting things retracted and corrected So I think that for instance is an excellent strategy like we know just in I'm not trying to Excuse anyone in the media, you know for keeping something up that might be misinformation, but we do know that There are limited resources Across different newspapers and you know and just online content takes people power to to correct an update So having a dedicated team has actually been really useful in the United Kingdom for not only kind of chasing politicians and not kind of You know in the nicest way getting them into a corner and getting them to retract but also doing the same thing whether it's like like full fact goes after like Cherries and nonprofits who release different like statements and news releases and studies and that kind of thing and gets everyone to kind of Correct what they've put up online. So that's one strategy I have a question at the end of this but at a related conference We had a speaker from a society professional journalists who was trying to make some kind of cooperation with the Wikimedia World to certify that journalists or journalistic organizations were doing fact-checking. We're reliable sources not just to have a conversation about it, maybe statistics about it and Here too, it seems like there could be a cooperation with the media wiki platforms or the other network of issues that it whether a Fact is true is associated with who said it did the journalist check it is that journalist at its institution that does checking So broadly I wonder if your organization is likely to cooperate or network with Wikimedia or other organizations of this kind like professional journalist association Yeah, definitely. So thank you for that question the It doesn't matter that one of the slides that I had on here Which was that code of principles which was five different kind of you know high-level but more it goes into more detail these are Principles that I Didn't explain the process well, but there there is a full process behind verifying fact-checking outlets that are signatories to this code So there there there is a structure behind this in terms of independent Verifiers going in and actually checking out every individual organization that applies to be a public signatory to the code of principles so this is one way that yes there is a formal way to verify and Can I envision cooperation, you know with a partner like Wikipedia and you know affiliates? Absolutely. This has already been happening with I Haven't given them enough credit. I'm more than happy to we're running out of time I talked to anyone who wants to know more about what some of the things that Facebook Google and others are doing Google for instance has been piloting. They've just recently expanded their pilot outside of the US, but they've been partnering with the international fact-checking network and actually any of the verified Signatories on to this code of principles Can now be tagged on Google as a fact-checker and what they've also been doing is flagging content that has been Challenged by one of these verified fact-checkers and Google is also a few different things, but including they'll elevate fact-checks so if someone types in Codder payout Facts can will be one of you know the the elevated Sources on there and we're tagged as a fact-checker. So yeah that type of collaboration is definitely happening And I think it needs to happen more and more so I think we're maybe If there's like it one more question There's also someone who has not asked a question yet just to be equal opportunity Maybe you can we can chat afterwards Do we see a cultural shift in the West where? facts of being seen as less Importance in a culture and we're aligning ourselves more as a culture with the African adage that Relationships are more important and facts can can bend to suit it Sorry, I'm not sure I understand the question So so is is fact-checking inherently a cultural and political position on its own Because some cultures do not value fact-checking for example like some some Sub-Saharan African cultures and some East Asian cultures consider Relationships more important than than facts. Are we as Worst in world moving towards that direction or is it just blip that we need a lot of fact-checking because politics is weird as of these few years I Would be interested in seeing any research that you have to back that up because I haven't myself seen anything that says that One culture is more likely to be influenced by facts or not. I think Intuitively I could maybe see that the presentation of facts and how Authority is challenged certainly would be different across cultures, but I can tell you there are very active Colleagues of ours fact-checking like in South Africa. There's a big outlet Africa check and Japan has a lot of their news outlets are Piloting fact-checking operations right now. So it is something that really is taking off like globally I think a lot of my Examples here are we're certainly biased to certain parts of the world And that's my my blind spot and just where where my closer colleagues are to pull examples from but if you do have any sources like that I'd be interested in seeing them afterwards So anyways with that I think we have to wrap up because we're in plenary right now But thank you so much everyone for your time and excellent questions and please do be in touch