 So thank you very much for the introduction. Yes, very recently a PhD graduate, very new to the Scholarly Communication Society. I remember I served on a panel on repositories and knew nothing about them, which I think tells you a little bit coming from the research and perspective. So I wanna start talking today about this tweet. And so I think this tweet is interesting. Does anyone know who this is or what this refers to? So this is by the author of a very infamous nature paper around arsenic life. And the tweet somewhat makes sense. She's defending discussions happening only within quote unquote scientific venues, largely because she's being attacked in very public venues such as Twitter, Facebook, blogs and probably a few others. And so I think this is interesting because I agree with points of it, but I also disagree with a lot of it. And I also think it's telling to how scientific communication occurs. So the fact that we think public should be separate from that, or as this is kind of a normal thought, shows largely how traditional publishing occurs. So I'll be talking about gray literature and social media as open review to further muddy the waters of open peer review. I think most people when they think of open peer review think of the traditional peer review system and then they're made public. So I'm looking at things in quite a different perspective and we're approaching things at the window or in quite a different perspective and this is to really empower gray literature and even social media, if you will, as new forms of open peer review. So this is maybe what you think of when you think of social media. It's what I think of, cute cat pitchers. Social media and social networks are highly effective at sharing cat pitchers. It's probably a bigger industry than scholarly publishing, but it's not just that. So there's a lot of other things going on that I hope to share with you beyond this funny or cute cat. Gray literature for this audience, most of you know this, but I think as a PhD research in biology, I would not have known what gray literature was and I would have thought something along the lines of this and I'm not gonna say anything else about that slide. And so going forward, I like to define these, so we're all kind of on the same level. Social media are applications that enable users to create and share content or to participate in social networking. Sounds kind of like what scholarly publishing should be. You'd like to enable users to create and share content within a community of their peers. Gray literature is some overlap with social media, so a report or manuscript circulated or published by unconventional routes. And we at the Winnow are largely focused on publishing gray literature in that we give them a DOI in permanent archival for a blog. Recently I've started to reach out to listservs to even curate emails, Reddit AMAs, really everything that's being written out there that most traditional publishers ignore. We're saying, okay, this should be count not only for the scholar, but also for the scholarly record. So how big are these two entities and how are they used? Most of you probably know. A lot of you are on Twitter right now, maybe criticizing this talk, which I encourage you to do, but how are they used? I think it's somewhat unfamiliar because a lot of us do use it to share pictures of our dogs or cats. I know I certainly do. So I'll highlight a few of these, not all of these, but some of these I think will be new for this community. And I think to show that there is value beyond the sharing of the cat pictures, I'd like to show some of the use cases. So these are metrics from Reddit, from the channel science in Reddit. So Reddit can be a good and a bad place. It's basically a place where you share links. People can upvote or downvote them. Recently they released, they're trying to be more transparent as well as journals are a report on their metrics because people were saying they were being too restrictive actually. And so this is from one channel alone and there's multiple academic channels on Reddit, but as you can see, it's quite big. There's a lot of people there, a lot of people commenting 100,000 to 125,000 comments per month. This are comments on traditional publications as well as new events that they're hosting themselves. And then a lot of page views. So this is driving traffic traditional journals as well as fostering discussion around traditional journals. And so here's what it looks like. And you can see these are links to publications that appear in traditional journals as well as an AMA which is an Ask Me Anything. PLOS has formalized this where every Wednesday they have one of their authors will basically be available for a Q&A with the community. And we are actually now formatting these as white papers or as PDFs, assigning them a DOI and archiving them. So this is great. I think it's one of the biggest intersections between the public but also amongst researchers. And you can see sometimes they've introduced Flare so you may not know who the person is. They may have a funny screen name but there's actual verification that they're a PhD in whatever. Twitter, this is probably very appropriate since a lot of you are tweeting as I speak. I had a hard time finding actual statistics on academics using Twitter. I still think it's very small. I didn't use Twitter as a PhD student until I had the Winnor and now I use it every day multiple times a day. Most people in my lab didn't and I would argue a lot of academics are still not using Twitter. Twitter is big. There's 320 million monthly active users and this graphic over here is showing how academics are using Twitter. So this is from a survey of over 3,000 researchers. And I'll just quickly point to it because it's somewhat hard to read I think. But Twitter is used largely to follow discussions to share content and along with that sharing content you can discover recommended papers and discover peers as well. And I've actually discovered a lot of people that I've met through Twitter that I've never met in person are actually in the audience. So I hope to actually meet you guys face to face. So how else is it used? So here's someone shares a link and you can see this is subtweeted which is really just kind of a funny word for review. These are peer reviews. These are scientists or scholars looking at this and the study is saying social media exposure does not have a significant effect on downloading citations. Well this may be true in that case because it's a very tiny sample of all paywall journals. So sharing things on social media that has a paywall is not gonna go very far. And in fact there's an entirely new website also started by a student to get around this problem which I guess I'm not supposed to name. Facebook is somewhat not or is not used as heavily as these other ones for actually engaging with academics. I use this largely to connect with my family which is across the United States and to be friends with people from high school that I don't know anymore. But that I think is changing in certain groups. So if you look at astrophysics and I just quickly stole this from a blog there's a lot of groups going on with very vigorous discussion around astrophysics. They're actually closed so you need to ask permission. Some of these groups are restricted to only astrophysicists. This is for true of science journalists and other things as well. There's nothing in my own field that I've found to be so worthwhile but that doesn't mean that things are not happening out there that are great and I think this is one of them. There's social network sites that are specifically geared towards academics so academia.edu and research gate. These are growing quite a bit or they've grown quite a bit. I believe I'm on both of them. This is largely in case you're contacted so I think the engagement on these are somewhat low but I think they're trying to change that. So they're introducing discussion mechanisms around these endorsements and different things there. And this is usually commenting on the papers that have been published through the traditional route. Pub peer, I don't know if you guys have heard of this. This is bills itself as kind of the online journal cub and the way this works is you can take any DUI or URL, put it in there, it'll pull the metadata from Crossref I believe and you can comment anonymously or with your name on any piece of work. This has been used to take down a lot of fraud in science. There was a big legal case around this where they took down a paper and a researcher lost his job or his new job offering and they got sued. They won that and they're continuing on and I think what is interesting here, I show the numbers not just in terms of comments per month for pub peer but also for PubMed Central. So PubMed also has a commenting function on PubMed papers but it requires you to use your name. You also have to have a paper on PubMed to be able to comment on there and the numbers are quite low. And so this points to the fact or I think opens up the discussion that we're gonna have later is how do we get people to peer review in a way that's not just saying this is fraud, I wanna take this down and to be proud of what they're saying because people ultimately want the content out there. That's why they're taking the time to do this. They care about the scientific record but why do they not wanna put their name next to it I think is an interesting question and I'm not sure just a DUI will help as I can tell you from our own experience. Journal clubs, so this is a little bit of self promotion. We publish or we're trying to encourage groups to publish their journal clubs. So journal clubs happen differently around the world and lab to lab basis. From my own experience every week, we would discuss one paper that was relevant to our group's research. It would be led by a graduate student and they would go figure by figure through the paper saying okay here's what we agree with it, here's what we disagree with it and I think this is really a tremendous way to learn how to review papers. The first time I ever sat in these, I read the paper and said oh this is great, it's in this big journal, it's amazing and then my boss who is leading the journal club that one time pointed out all these significant flaws and how to look at the papers and the weaknesses and it's really where you kind of develop a critical eye and I think become a good peer reviewer along with actually reviewing articles itself. And so we're trying to encourage groups to publish their journal clubs by blurring the line a little bit between a publication and a blog. So we give a DOI an archival via Portico and it looks nice as an altmetric widget, so on and so forth. And this quote I take from a paper recently, I forget the name of the journal but there's a journal that basically published a paper where they found their peer review process failed so someone pushed a fraudulent paper through and it was very nice that they went back and looked at where they had failed to address these concerns so I appreciated that transparency and I highlight here initially where these flaws were found and this was found during journal club discussion as part of the training for pharmacology residents. Blogs, again the numbers on these are somewhat hard to come by but there's thousands of them. I think altmetric has at least 8,000 on their list. Some are very active, some have a lot more views than the actual original research. I highlight research blogging which basically are blogs, anything that refers to a traditional publication would be included in this or aggregated in this and these are not necessarily used as a form of review where it's this is wrong, this is right but maybe highlighting it for a lay audience which I also think is important. And this again is a bit of self promotion but I think it shows that blogs are actually being used in different instances so this researcher had published one of his blogs with us and now it's been cited I think 18 times which is more than a lot of his traditional publications. To submit with us is very easy for a blog you copy and paste a URL and that's it and then you can leave it open for review or you can close it immediately, it's led by authors. And then I kind of conclude this survey of all these different social media platforms by taking a quote out of this recent paper that after 14 months of informal post publication discussion, the hypothesis was refuted and this refers to that original tweet. So I believe that a lot of this informal post publication discussion, maybe some of what we're having now is important and so we should not ignore it but we should actually figure out how to make sure it's done in a better way that's preserved and that also can be validated and tested and it's not perfect either. I think this also asks the question if we're noticing a lot of these errors in post publication peer review, how good is classical pre-publication peer review? I feel like I may have to duck now after the last few slides because this is where I get a little tacky. So the name, the winner and the idea, the winner came from reading The Trouble with Medical Journals which is by Richard Smith, former editor-in-chief of BMJ and these studies I think are very eye-opening to me and so to test peer review, classical peer review, which is really the big difference between a blog and a scholarly literature as peer review, you can insert major artificial errors into a paper and send it out for review if you have a journal, of course, not everyone can do this and so this is what they did at BMJ, they put in nine major artificial errors and on average only two were detected. So this is months, sometimes years of time with results that may not be so perfect or is it what we would expect? This has been repeated at JAMA, again with similar findings at the Annals of Emergency Medicine again and I think this is somewhat scary because it's all in medical journals and then another study looking not just at can peer review detect errors but the bias of peer review sought to test this back in 82 and what they did here was basically take papers that were already published in a journal, change the name of the authors on that and the institutes from somewhere, the names were like Balula, Artass and they would change them from Harvard to Tri Valley Institute and send it back to the journal that had already published it and the majority of these cases were rejected so these were journals that were rejecting work that they had already published not because they wouldn't come out and save because of the names but because of fundamental flaws. They got caught halfway through this and actually there's a typo down there, one went through a tenure battle so his tenure was revoked and they I think moved to Yale after this and so we have, we published the backstory on this which is how I found that out and then last but not least if you look at the agreement between reviewers as to whether papers should be accepted, revised or rejected, it's close to chance and so you would think if there's an objective or how objective the peer review would work it would reflect some agreement amongst reviewers but it does not and so that doesn't mean that I think peer review is bad, I don't, I actually think peer review is great, I wish it was happened out in the open so we talk about open access and this refers to the primary publication but I think we're ignoring stuff that is very valuable which is the experts critically engaging in discussion around the work itself and so this is a survey that we did on the winter war, it's quite small so there's only around 100 responses but I think it gives a good reflection of how people feel about peer review as it is now, how they feel about open peer review and then a few other things and so I would put myself as that I benefit from being peer reviewed and then I think others benefit from peer reviews as well. I think what needs to kind of change in this is that we need to start to make them open, this would be good for others to learn how to peer review, to verify the peer reviewers, to see who peer reviewed it and of course I think it should happen in a post publication manner because we talk about even when peer review works if you find a big flaw and reject it, well there's a lot of journals now we'll publish it and you'll never see that negative peer review and so I think it's very interesting that you publish these discussions and leave them open to see, I think that's maybe not the best thing for selling your journal but for the community it is the best thing and The Winner has the same policy where we will publish things that are wrong, I'm sure there's a lot of wrong things on The Winner, we facilitate those reviews to say okay this is published but this is also wrong and I think the onus is on people to be right, people don't want to publish wrong research but making reviews in an easier way will discourage that and so would making peer reviews open be, I think this title, oh would it change the content? No, not significantly, I think it probably wouldn't, I think the biggest thing that would change would be the tone, so right now peer review, if you've ever seen that cartoon is basically like going through a fight, it is really contentious, there's a lot of, I saw this one YouTube video that is down the impact architecture ladder and the way they portray peer review is they put a blindfold on the person and then they're boxing that person so I think making reviews open makes it harder for you to be an A-hole and I think that constructive criticism is a good thing and this will also transition to ways where we can leave open post publication peer reviews that are not just this is fraud but I think this is a good paper. Would it require a lot of work? I don't think it necessarily would so there's some great organizations working on making peer reviews open that have already existed or that are already out there I should say and I don't think it would take too much work because really the content is not gonna change, it's gonna be the tone or maybe the content will change a little bit where you have to really back up your claim saying that you disagree with this or not just ask for another experiment because it's easy and you have that power to do so and so I'm gonna follow with a few quotes basically from the survey that we did talking about open peer review this is from an editor in a medical journal and basically they actually studied this and they looked at it didn't change the quality of it but it actually changed the tone which was slightly more constructive and this quote follows that as well they take more care and effort to review carefully and thoughtfully and then if it had been published the remarks that are exceptionally rude and are stupid would probably have been avoided by the reviewers and I think this is true no one wants to be super aggressive and mean in the community or if they do there's a small minority of them and I think light is a great kind of sanitizer so how do we get people to start to make their reviews publicly available? Some journals restrict this so if you wanna make your review open and it's been published in a journal that's not open peer review well this is somewhat tricky because the authors didn't know it was gonna be open later the reviewers didn't know and so it can be difficult and so what would make people to leave their reviews open if they were given the chance so you mentioned that most don't put their names on it I think the biggest is if they're valued in terms of their career so if the I always call it privilege promotion and tenure committees started to look at these as opposed to just and they do in a sense they say you put where you review but you're not largely saying how many reviews you did and what the quality of those reviews were and then if more people started to do it so it's always hard to start kind of a movement of people takes a few small brave souls so what do I recommend in terms of the peer review language major revisions and this is largely not just say we need to do away with everything because I agree that there's a lot of great infrastructure there I think it just needs to be kind of shifted around experts reviewing papers is a good thing but behind closed doors that only two people can read I would argue is not a good thing and so the winner was my attempt at that I launched the winner very naively said okay I'm frustrated with scholarly publishing I used to publish on my own a lot I have publications in big name journals cost a lot of money it was slow would be reviewed by people that I knew personally and so I said okay well I'll just fix this I quickly realized how naive that was and you know we've shifted to where we are now from trying to publish more original type research to this great literature and when we first launched I considered ourselves somewhat a mix between scientific publishing and Reddit so somewhat a mix between social media and science publishing I think that is somewhere where we are and I include this because this is cool this is the founder of the Reddit and I thought that was interesting until I saw this tweet which goes back to peer review on social media and this says that really if you're a mix between traditional publishing and scientific and Reddit well that would cost 3,000 pounds to post a photo of a cat eating bacon a year later and so there's aspects that we don't want to take from that and vice versa there's aspects from social media that we don't want to take to traditional peer review although maybe someone would do this so here's the winner we publish yeah all the things that basically the big publishers ignore and we're also trying to include younger generations so undergrads can publish in undergrad journals which to me are not so attractive basically because they're called undergrad journals they also have very low visibility so this is a big initiative that we're trying to push is to get the next generation publishing earlier in kind of a light publishing scenario so that when they go to publish in these more official type journals they have some experience in this and so they have some experience publishing they also have some experience with open peer review so DOI is an archival for Reddit AMAs lists there's even conference summaries to talks people have presented their talks there's now great citizen science projects published on the winner so if anyone's heard of FoldScope these are cardboard box microscopes that people can go around and take pretty high quality pictures of whatever they want to look at well some of those are starting to appear on the winner and this is kind of a group on Stanford as Tony mentioned we've grown quite a bit we have over 4,500 authors and now over 1,100 publications which I think is pretty good for being two years old starting from Josh Nicholson as a graduate student so we didn't have a Nobel Prize to consult us that may have made things somewhat easier but I think I understand the perspective of the researcher very well which I think has helped and we're also playing the same game that traditional publishers are playing so if you think about it a DOI and archival for all these new forms of media really makes us kind of the prestigious venue for this and so I think this has led to people to publish their blogs with us because we're not necessarily easier to write a blog there's WordPress and Blogger but to get these traditional type tools you have to go through us or you can get it a few other ways as well that's not to take all the credit we're not alone so there's a lot of great groups experimenting in this Science Open, I believe it's here Open Air for funding us and funding new initiatives I think this is the critical thing that is missing science publishers have a lot of money to experiment with and they're not doing it we have almost no money to and we're entirely experimenting and so I wish they would give us some and we could do more experimenting too F-1000 Research which is largely very similar to the winter but purely focused on biomedical research arranges reviews whereas ours are author driven and I think that's important because it's hard to get people to review unless you have someone really bugging you but there's still lots to be done and so we are experimenting daily with new initiatives, journal clubs even listservs now we're thinking about curating and publishing we need to figure out ways to incentivize people to participate in these so I think the discussion that we'll have after this will be good for this and then can we improve upon some things that we already do so can we automate a better way to identify appropriate reviewers so I think, I mean maybe I could be wrong here but a lot of journal editors use some of these open source tools or some of them have proprietary methods of finding reviewers can these be improved and can we do it in a sense where we have basically a pool of validated reviewers a way to identify them not just as one person looking but maybe aided by computers so AI a bit and then the invitation of reviewers and so these are more kind of pie in the ideas or pie in the sky ideas going forward but I think we shouldn't be scared to try risky things and then what else can we do to make peer review more robust so we are looking at gray literature and post publication peer review but I'm sure there's other things that we have not considered and I think we should start to think in ways that are more creative than just making this open that you know is the first step well what can we do beyond that can we include more in the peer review process maybe graduate students maybe postdocs I served on a panel where one PI and one editor said postdocs shouldn't be allowed to review which kind of made me somewhat furious because we had received a PhD and back then postdocs you know postdocs are relatively recent phenomena they used to go on and start their own lab but because there's so many of us and because funding is so tight now there's this kind of limbo period where you're still treated a bit like a child despite being 30 plus years old and with that I'll say thanks thanks for listening to my rant hopefully you found some of it informative and I'd love to talk to you guys at the break you can reach me at the respective locations and I'm happy to take questions