 afternoon thanks for sticking around long enough. I am here to talk about a particular use case but a little bit more than that and the work I've been doing the last couple of years has been in the world of the information ecosystem and in particular the polluted part of it. Recently someone asked me what I what I was working on I said misinformation and he said oh on which side somebody who deep skepticism which I think is really valid about everything that we see in the world that we have to sort it out and we have to do it a whole lot better and journalists are a key constituency in making this better I hope that they will do it but not just through better journalism some other things. I want to make the case that an annotation as a tool for transparency is a fabulous tool but I want to give a little bit of background and also point out that people actually think that there's value in transparency they're more willing to pay for stuff. Journalists didn't realize that till some surveys came out. Part of it is journalism's traditions and norms which are pretty arrogant and top-down and always have been at least until recently and this notion that these are things you'll hear from journalists when you ask them about their work often. My favorite least favorite is our work speaks for itself. Well no I don't think that it really does. And what do we do about convincing journalists to change their ways? Well projects we're working on are actually helping and I'll get to that in a minute. I got to make do you reminder that misinformation is not new. We've been dealing with it for a long time and that what is new is this coalescing of technology, malign intent and scale and a whole bunch of things that were not the case before and that it spans a great variety of kinds of misinformation. I beg you here never to ever use the expression fake news. It's long in every possible way. Two in particular one is that by definition it's wrong and second is that people who lie all the time have taken it over to describe things they don't like that are often true so please don't use that. And if you follow this field at all you know that it's gonna get worse. The so-called deep fake phenomenon of having people saying things they didn't say it's gonna be a problem. And what the work that I've been doing for a while now involves not just thinking about how we upgrade journalism and information but how we upgrade ourselves on the demand side of information. The supply-demand equation requires us to have understanding on the demand side and we don't have a lot of that at the moment. One of the ways to get there is through what people call media and news literacy. My colleagues and I have come to favor an expression that came from the American Press Institute news fluency literacy having a kind of negative connotation for people outside of the Academy so we think of it that way. But it's really a combination. It's a whole bunch of literacies that we have to help people get better at and this is increasingly a holistic situation. For journalists one way to do this to help their communities be more news fluent news literate and to give them more tools to combat the misinformation that's just pouring over them all the time to understand better what's real what's not. By the way I'll make a copy of this available for everybody is that is transparency and there's evidence that it works and what is it it's these things that I've listed here and and other lots of things that involve techniques and practices that have not been part of the norm for journalism. One of our project partners is the McClatchy company which has 30 newsrooms around the country and we've been working on a bunch of experiments with we think some pretty great success and trying to make it go further. One example of the transparency is having the journalists explain how they did the journalism and this was a case where a reporter in Kansas City did something he had never done in his life which is to explain in a podcast and a story on a wonderful series he did and it's changing the culture and we think that's a wonderful thing and I've not yet gotten people to think of annotation as part of that except in some senses they're already doing it and I'll give you some examples in this case this is another McClatchy newsroom in Idaho the Idaho statesman did a long and involved and really quite good story on childcare in the community and McClatchy has built into their content management system this thing they call it the I forgot the title they've given it but it's a it's a transparency plug in they can answer questions about the journalism and they can move it anywhere they want in the story and they have deep metrics associated with this so they're figuring out what people will actually look at and thinking about using that in other ways this is an annotation method it's not the kind that hypothesis does particularly but it's really a breakthrough no another no other news organization I'm aware of is doing this and they're moving ahead on this project pretty quickly other news organizations have done transparency and annotation in other interesting ways this is actually pretty old from the New York Times basically they link from the footnote or the number one to their response to that point in a letter to the editor after a scathing piece they did on a local congressman and I would have liked to see that conversation go further let him respond to that or his staff and see where it goes but it was a good thing to do lots of things to use this for in including you know like you want to select out a really astonishing fact to say well here's why this is true usually there's attribution but not always the context you know going deeper on things the invitation for the audience the readers the listeners viewers to come back to the journalists with better information because one of my personal cliches when I was a journalist was that my reader knew more than I did and this was not like you know any kind of breakthrough concept it was just obvious they had to and there was great advantage for me the transparency is not the only use of annotation that I can think of and there's some really fun examples that people are doing this is from Josh Marshall who writes talking points memo political well he started as a blog it's now really robust media company and he takes an apple pen on so I forget the software on his iPad and mark stuff up this in this case Josh who is a scholar before he was a journalist is doing some of the college professor markup saying no God this is terrible stuff and it should embarrass the people who did it this is my personal favorite when the Republican House after not long after the Trump election passed a repeal of the Affordable Care Act Obamacare a bunch of middle-aged white men got together with Trump on the White House lawn to celebrate their passage of something it would destroy the Affordable Care Act and Josh goes in and he goes gets the data and says okay here's how many people in each of their districts is going to lose health insurance this is brilliant surfacing the data it's journalistic values enormous and it's in a weird way entertaining and there's no reason journalism has to be derived this is great stuff and more people should imitate it hypothesis works with climate feedback on interesting ways of understanding bad journalism boy there's a lot of bad science journals out there and of our other projects is involved in that there's another kind of annotation that's embedding metadata for transparency in the CMS in the data in it's the trust project it's fairly complicated process but we strongly believe that this is a good idea and that organizations that do this are going to have a real advantage by demonstrating that they are deserving of more trust than not no one's deserved complete trust but we different people deserve more than others another annotation form that I love in journalism is the cry on on cable news which I otherwise don't like because it's just people yelling at each other and generally in stupid ways but this is actually quite valuable to say no he didn't say that this is another one you notice I'm pointing to people who like to use the expression that I won't use and these there's value in this it's sort of instant corrections I think it's a mistake by the way to put people who lie on television in the first place I would ban anyone who's a known serial liar from my programs that would be my policy it would be fine people who want to defend the president but who don't lie maybe difficult but it's worth trying so I that's that would be you know if I become czar of journalism that's going to be one of the rules you don't give allowed speakers to water I've given you examples of things that are kind of read write or more so than not and I read only I think it read right is where we ought to go with it though it's hard and an example that I just posted on disappeared okay so these are things that I don't know if you can see them there they're not very prominent here but I added things that I think would be journalistically valuable to what I did the first one is it basically says you know why am I doing this is because I believe in free of expression okay that should give you context for why I'm writing this and then you know some other things that are context including down below where I say here's what they could do the platform companies to help on fixing misinformation and I've made a few a list of a few of them you know nope you know not much progress you know mixed at best and zero and then added one which I would love to see more often which is maybe you have other things that you'd like to see them do and why don't we thrash that through or just email whichever this is I think annotation like this in journalism would be highly useful recognizing that journalists a are in a business that is crumbling financially and have no time so everything we asked them to do they say well what do you what do you want me to not do that's that's a problem but I think it's worth asking and I think that the results are worth having I want to ask all of you for some help in the annotation community and this comes from a conversation with John Udel who knows more about this than anybody I know and he has suggested I truly love this idea of an evidence layer that would live in the journalism but that would be interoperable with other people's journalism so we're all pointing to the same source material and talking about that and we don't have to constantly reinvent wheels that building this into content management and to journalism would be astonishing and be such a service to journalism and of course the key thing is it's got to be easy don't we can't burden journalists with stuff journals are not technical for the most part but it would be great to have this evidence layer and I'm excited by the idea I I think it's obvious but I'll say anyway this is not just for journals any organization in the world at this point that wants to be trusted has to think about ways to do it because everyone's under attack at some level companies not just politicians not just journalists and we may be transparency is one of the ways I believe it is that they can add to trust from the rest of them so that's a quick look through some ideas I'm I really came here to listen more than talk and annotations one of those things that is absolutely dazzled me from the minute I saw it even that horrible thing that lived on the web back in the 1990s they that pissed off absolutely everybody third voice third voice yes and but the hat when I saw it I thought glad this is this is a really interesting idea they didn't do it right but I think you're all doing important work and and thanks for helping me learn questions or comments for Dan so this is not related to my day job like everything else I've talked about here but but politics news media is kind of a hobby of mine so I try to stay off top of it so this is fascinating to me in terms of the evidence layer that you were talking about how do you how would you foresee that not getting skewed in any particular direction and staying kind of pure evidence because if it's you know being you know curated by the general public it's easy for those kinds of things to go off track so how I mean would there be some sort of regulatory something or how would how would that even work in your imagination if I understand the basics of it I think John's better equipped to answer your question but I think this would be done by the journalists at many different organizations and it would you get into all kinds of fraught territory with who actually gets to be there is this a club of the you know all the usual suspects or do we invite people or not the traditional journalists and I would go for the latter but there's a governance issue but I think it's well worth the exploration and the work to do it right so you could bring in you know historians other scholars people who are I think domain experts would be among the best participants because journalists don't tend to be domain experts so I think there's rich opportunity for forgetting that to be at least useful and maybe crucial yes I found this really interesting especially your comment about how there's a lot of bad science journalism out there as someone who works at a science organization I feel like it's a really hard balance because you want science to get out into the general public and you want to get it out the right way that's really difficult this is a little bit more of a comment but at triple yes we have this initiative called SILINES the idea is that you're directed that you are connecting journalists with scientists and so that as you said an evidence layer almost being like an expert layer so that different experts could let could you know lay additional levels of information on there I know you know when I was working in student journalism at least I would have a I would interview a professor and then I would say what I think the professor said in an article and then the professor might email me and say hey you didn't quite get that right and then I might have to issue a correction or something like that that probably would never necessarily be seen or wouldn't it was taken out of context is hard to understand so I think the idea of an expert layer would be maybe a way to do that as well not just so journalists can comment because as you said time is limited and that sort of thing but yeah when I was complaining about science journalism I wasn't complaining as much about people who specialize in science as the fact that most science journalism is done by people who don't know much about science and one thing I would beg science journalists to do would be to pick this mantle up of helping their audiences understand how to read a piece of science journalism that that's part of their job I believe now I've not made any headway in doing that we we are working on another project that I mentioned to you is we don't think news literacy news fluency should be a subject by itself though it is we think it should be embedded into topic areas and the first place we're working on that is science and health where in the case of health being misinformed you know can hurt you in a really serious way as we're seeing with this new outbreak of measles so we we we have to work on these things together but again I think it's not totally nothing wrong with asking somebody who knows well did I get this right when I was a columnist on technology I used to call up people and say I'm reading you a section that's a little technical tell me if I'm right wasn't like I'm gonna read you your quotes and you you know edit me I'm saying tell me if I've got this right and not invariably but often enough to be important at any time any number over one or over zero is enough to be important they would say well you're you're close and how about this that's journalism that's not anything except that so I encourage that and I encourage non-specialists to do that a lot I didn't thanks for that do you think that so I wanted to kind of get your take on the difference between had supported media and subscription media or public media with respect to how they were report science news do you think that the there's different incentives to maybe be more sensational in the different categories and what can we do about that yeah I can't do anything about financial motives twisting the output I accept try to persuade people that's a terrible idea if it's gonna matter but you know we all see this every day you know yesterday coffee is really bad for you today it's actually going to save your life tomorrow it's somewhere in the middle and then you know a day two days after that coffee plus chocolate will save your life I mean get wine well wine I that one I have confirmation bias on that one on red wine is good for me that's the end of this discussion but take it for me I don't think it's this audience driven engagement driven will tend to be more sensational we keep in mind that tabloids have been far more successful financially since they've existed and then you know serious journalism and it was probably worse a century ago when first got us into a war with lies you know so far that hasn't happened now I think we have to ask everybody who's doing it to care more about the mission that they allegedly are in but a lot of people creating media they're in the mission purely to stir things up make money I can't do anything about them except to point out when they do it so it this is again why I'm proponent of the demand side being worked on at least as hard as trying to convince journalists and people doing quasi journalism to be responsible I think we have to be more responsible I just wanted to echo what you said about how the solution has to be easy in this in the climate of journalism as it has to exist now and so I wanted to make a pitch to you know people out there who are coders programmers data architects visionaries design architects whatever that we need help with our kind of management systems and so on and if we can make annotation or inputting data as part of the workflow they make it make it actually a way to it's that collecting that data in an organized fashion actually helps us and it saves this time in long run and then there's easy painless ways to have it flow through this is you know something I learned from Johnny Adele that that's gonna that's how it's gonna work you can't ask people to do the extra work like you said but if you know if the system could keep track of what websites I use while researching a story and you know or I could very easily keyword stuff and and then I could very easily see it at the end and and or it could recognize phrases that I picked up from where and so on and then and that that that's gonna happen at some point it's just what happened you know in five years what happened 50 but I hope what he said so I like many people in the room was very interested in the idea of the evidence layer and so my question was just more to learn a little bit about more about how you see that is that kind of like a cannon for journalism that we kind of have these different categories and as an example it's say that we collectively agree this is a matter of opinion that an egalitarian society a more egalitarian society is a happier society from the bottom to the top so then we as journalists like we're not gonna kind of spend time and spend space arguing for that we're just gonna kind of cite to that research and go on ahead and do that in separate categories or do you see it like a different way I was curious about how you conceptualize it I confess that I don't know that's why I put on that slide ask John I'd like to see people explore and I don't I don't like the idea that something becomes the all encyclopedia of citations I think the idea that people would if they're pointing to the same stuff then they can argue about and debate about the same thing whereas today we're not necessarily doing that and if it lives in a system where people can flag it for others and where others can flag inconsistencies or outright mistakes and things of that sort and I don't have a good analog to it I think the talk page on Wikipedia on any robust Wikipedia article gets out some of that where people thrash out the article which is bigger obviously annotation but but there's a there's a process of arriving at something that's in theory neutral and valuable and I think Wikipedia is one of the great achievements of our of our age however flawed it is my my personal page there has been wrong for five years and Jimmy's an old friend and he sort of laughs when I tell him I said well my problem but it's I'd still think it's one of the great resources and the fact that people can hash out the changes they want to do and site then go pointing to other things that evidence layer it's put into Wikipedia would be pretty astounding too where they that would save them a lot of time I'm I do I'm thinking of this and I think John would have adamant about this this this thing would be a great time saver for journalists who wanted to get stuff right that alone is valuable by any standard when I when I'm in newsrooms these days I realize how cushy my job used to be and how hard they work I want to tell you that people doing journalism especially local journalism the young people they are they care more about journalism than my colleagues used to care because they're doing incredibly hard so help them out yeah so just a quick fall to that do you see the evidence layer more as being a group of resources or pieces of data that we've kind of acknowledged as being fairly robust and then whatever analysis we make whether we agree whatever kind of determinations we make we can just refer back to a piece of data that we do have trust in yeah I think the key thing is we're referring to the same thing that this is going to go this is while you know interoperable in the best kind of way for the people using it and that at at the very least we're having conversations and citations from the same item and not just web pages and things but w3c spec of going deep in links that go to the fact level or the sentence level things like that this is the potential for this is just huge and it John's expressions for this is to break down silos you know way that's never been done before so that's one of the reasons I'm enamored of this because I'm always in favor of breaking down sorry if I'm not giving you a good technical answer because I don't know I completely understand and you're fully forgiven for that