 Hello, I'm Sean Roberts. I'm one of the engineers for on the hypothesis team Yeah, whoo And I like to drop things so My group was over here and we focused and scoped our discussion on fact-checking Not on fake news specifically on fact-checking and even in that area That is fact-checking means a lot and nothing at the same time so we had to scope that even down further and Where we landed was what questions can we inspire to maybe Lower the barrier to entry How can we you know, it's one thing to try to solve fact-checking for yourself, but how we get other people to To learn how to do fact-check checking themselves and then What are there? What are the other options for? You know fake news and like de-incentivizing so the question so basically my notes are What are the questions that we came up with with in terms of fact-checking it is the first question is In terms of lowering the barrier to entry something like how I think it was Michael, but it was like wiki sub posts where when Wikipedia really started getting a lot of traction it was You know, they took data that they already knew about for a given article Such as like senators and things like that and they stubbed all the information They knew and so that lowered the barrier entry for people to Start and then give the information that they can contribute after the fact and so with annotation Is there something useful that we can do for taking this current page that we're on and Kind of seeding it with useful information Going further things such as auto-suggesting older or other references To improve the quality of the post like auto-suggesting information about authors Information on the website in general because there's you know after a while once You know websites get some sort of popularity or longevity They build a reputation for themselves and that reputation is interesting when you know If you're looking at a very specific article, and this is the first time you've been here You kind of want that summary so things like that would be very useful for people. We think You know auto-checking tools for like Facebook and things like that to You kind of move, you know, this isn't really an annotation base, but Move the conversation into where people are so on Facebook so when an article is suggested is there a way that we can Pull quick information that is static about that website to kind of seed or start the discussion And then information on the website in general such as longevity things like that so Some of the things that came up in terms of how do you help other people learn how to do fact-checking the biggest You know where we focused a lot was could Education in curriculum and schools be changed to help You know teach people better ways to you know have critical thinking and better reading skills in terms of You know actually consuming the content and then following up and investigating Going further in terms of the references The people and then just going backwards. So just introducing it you know like Take what you know, we all kind of collectively know and understand as a good path for fact-checking and see if you can Standardize it into a curriculum that people can take and base out So and then there's the other options such as De-incentivizing fake news in general The it was brought up several times like can we take the data that we know about any given page like it could there be a platform that aggregates credibility signals about websites or articles And provide them to people who give them an incentive such as their you know monthly income So can we de-incentivize? You know advertisers for example from giving them You know paying them to do this And then on the other side of the spectrum getting people to annotate and come up with fact-checking Can you incentivize them more? Is there rewards that could be given? Is there some sort of gamification that could be applied to? Chit you know it kind of with a fun talk. Can can you? Make it more interesting and more lower the barrier entry and make it interesting to Get more people to do it and then maybe there's a viral or cascading effect from that So lots of questions and lots of interesting conversations from that group, but that's what we talked about. Thank you publications annotation group and we had a wide-ranging discussion that started as Discussions in academia tend to be about credit and how to get credit and how to turn annotations into credit of various sorts So one thing that we talked a little bit about is various efforts and turning the amount of annotations and possibly type of annotations. I promise this wasn't me Into altmetrics course and ubiquity press actually has a has an ongoing grant funded project On this this is altmetric as the concept not as the company We also talked about Kind of a little bit about the flip side of this How can we get people credit for good annotations that they make and that our contribution to Scholarly discourse which we didn't get very far on except that it's difficult but that led us into an interesting discussion about Assigning DOI's to Annotations which there was some pushback because DOI's cost money it might be overkill It might also not be necessary because we already have your eyes But DOI's do have you know an interesting infrastructure including the metadata behind them So there is a good case to be made for them We talked about the fact that DOI's by themselves, of course, don't guarantee permanence It's a promise of permanence and then depending on who assigns them that may be more or less Credible and is particularly tricky for annotations where you have as opposed to data or an article You have two groups that need to be permanent You have the person that the organization that serves the annotation and the Organization that serves the object that's being annotated so the permanence is extra tricky here and well the robustness group didn't take place but they have Big project in front of them one thing that we talked about that it might possible to kind of Request the DOI for select annotations say if you want to cite an annotation formally you could request the DOI that keeps the costs and the amount of DOI's To a reasonable limit and that's also a model. That's already kind of at least discussed in the context of granular data citations That kind of then took us into the other big topic that we would discussed which was various issues and pop post publication Commenting and I'm gonna try to somehow group them one question was kind of the various Ways in which they can be interesting and add value and something that I found interesting was this idea for Textbooks etc that they can be a signal to authors on passages that need refinement or are particularly Used heavily right if students annotate a page a lot with help. I don't understand this you should probably rewrite it We also talked about annotations as a possible feedback mechanisms from research participants, so if you study People in a group they can read your article and can you know annotate it and say We didn't say this or this is a crappy representation of who we are And that led us into kind of a tangent on making this research accessible to the groups That you're talking to about etc We talked a little bit about the NYU project Where annotations are mainly for for kind of a general post publication conversation But they also kind of create interesting indexing links between They use the index of indexes of books, and I'm hoping saying this correctly Multiple books bring them together and create interlinkages between books as a sort of Annotations that way and then we talked as the last bit about Making this a vivid in terms of like having things like life annotation sessions With the author right so you publish an article maybe high profile, and then you have instead of having a reddit AMA you have a life annotation session on on the article And I think he life is playing with that proposal and there's you know some authors who are terrified by the idea but apparently some Excited and those are some of the things we talked about. Thanks Okay, I was in the section around Storing annotations for data visualization and we pretty much talked about the whole life cycle of all of the metadata That's involved and one of the big assumptions that you make it that I was making in this situation was that in A URL if you go to it whenever you go to it it renders the same page Which was just a caveat that I wanted to call out that I think you know with public pages that you don't have that Guarantee and so we thought about that and just I think the solution We came to is when you are adding an annotation at as much context as you can and store that and Basically don't conflate storing the annotation with then when you render the annotation how you want to show it Meaning if you have an annotation that is you've stored really specific scope You could then write logic later on when you then render it of maybe making that annotation at a showing it at a more general scope Then it was originally like if I wanted to annotate a specific point on a chart Maybe that means that it actually shows up for multiple charts when it's actually rendered so separating that logic as well as having Actually different sources of annotations user generated annotations is really just one type There's also maybe a machine generated annotations from anomaly detection There's also annotations that could be made on a specific data set Which is not how it's rendered but just the data that it came from and so our group talked a lot about Storying and separating that from actually how you then show it and the different ways you could use the annotations That's it. Thanks Okay, so we were doing the opt opt-in or whatever session and So what we were trying to do first is to what we want to do is try to describe the problem space and orient and so what we did was we laid out a situation with has fewer constraints moving towards the top and more constraints moving towards the bottom and then Remy had the brilliant idea of mapping these different options that we Started to lay out against statements that had been made like as their statement number one her principle number one is I own my own content her Statement number two was about freedom of speech and her statement number three was kind of a Let the market decide many moderation entities will emerge We also have similar statements that were made by other people So Audrey waters last week wrote a blog and then another person tweeted in support of that Which was effectively like my blog my rules. So this is a universe that has more constraints and then William Gunn paraphrased the Alternative stance which might be my browser my rules as kind of the more libertarian worldview so The different options that we can think of that different people have articulated and this is kind of where We got to but I'm also super interested for other people who Either now or later might come up and go. I've got another option another part of this problem space, but The first one is kind of the do-nothing Which is kind of I would say where we're at now in other words There is a that hypothesis or how I would characterize the way the genius plug-in currently works, which is that there is a public channel in both of them and You know, they will moderate the the provider will moderate extreme You know abuse and things like that or moderate abuse or whatever But that otherwise, it's not moderated and there are no you can't opt out as a as a side owner Other people have articulated and it's the extreme opposite Are people that have suggested that the internet ought to be opt-in to annotation in other words You can't annotate in public anywhere unless that site owner has said I don't mind being annotated a Another Kind of an extreme Option would be what Audrey waters did which was to deploy a drop JavaScript piece There are several different examples of these, but they're like shrapnel. You can deploy in your page it's an active JavaScript countermeasure that will defeat annotation tools by by attacking the way that they insert Annotation anchors into the DOM into To the the browser so that kills all annotation including the use of personal notes And or private groups so it's kind of a kill it all type approach, which is fine, right? I mean my browser my rules their blog their rules if they want to deploy that JavaScript That's that's fine, right, but you know are there other options that might you know Give them some of what they want so that they don't pull, you know the ultimate push the ultimate nuke button But anyway, that's that's a part of the opportunity space Another option might be a you know something annotate dot text thing that would say Don't annotate me But if and that would maybe Show at the top of the sidebar when you are getting ready to annotate this blog has requested that you not annotate them But if you determine that you think it's in the public interest you could override that But maybe that would draw an immediate Moderator to take a second look and make have a little scrutiny To see whether you were behaving yourself. It wouldn't solve everybody's Concern about this, but it might you know get interest a little bit better. Maybe maybe not depends on your perspective We're about to launch a feature that where publishers or blog owners would be able to create default groups over their content that would Help address some of this by creating the space that they intend The discussion over their blogs to happen and that they have moderation control If you want to bring a group from outside you'd still be able to do that But you would have to know That that was an option and you know be a member of a of an annotation group so Those are that was our what we talked about these were some of the this was the kind of the map that we came up with and Happy to talk more about this with us other folks later All right, let's go so we already have the notes for the fun session Annotated inside the Google doc and there's a photo of that page So if you don't listen to anything else I say you can check it out there Okay, so we were going through the fun session which meant a lot of what we're talking about was game design motivation for why people do things Both online and offline and then we also kind of went through some historical Aspects of annotation and what it might look like in the future I think the main aspects were Brooke Allen back there Defined fun as pleasure with surprise and I think if we were to look at How annotation was going mainstream the quote was Games are voluntarily obstacate overcoming unnecessary obstacles And so while you may have a scientific community or children doing homework that are required to annotate Once it becomes a broader game to annotate the web. We might have more progress Interesting parts that came up were that wikipedia's originally were 13 to 22 year olds mostly male mostly truth-finding Individuals and they they gained a sense of satisfaction Disfaction and and completion and status I suppose out of out of annotating wikipedia and also You'll find this in rap genius as well Rap genius came up a lot with wikipedia We spoke about how in gaming models you have Challenge and skill on axes and you want to try and find a flow state between You stress mastery or flow and exhaustion and making sure that you stay in the flow state We we made some meta points about I guess tweets being a form of annotation and also The daily show I guess is a form of annotation as well Other other stuff we went through is that children reinvent society every generation and that you have this axis Where we either interact with or act upon the world and we either we either interact or Act upon something and the something is usually people all the world And so you have achievers that might be people trying to build the new world and then you have killers which might be By definition people training other people to do something And so the teachers training people to annotate would be considered killers Whereas the people building the annotation software itself would be considered achievers So we went through a lot the notes are all there But I think the one thing that got to me was that if you can make it fun and you can make it a game That's excellent and also for the early stage. You'll probably see it's a lot more of the the Adolescence that'll be building the annotated web awesome