 I think it's about time to get started. We have a really interesting presentation today and I want to allow plenty of time for it. Welcome, I'm Cliff Lynch I'm the director of the coalition for networked information and thanks for joining us on this Friday for the some of the last of the first week of the fall CNI virtual 2020 member meeting. A couple quick logistical things this session is being recorded and will be available subsequently. There is closed captioning available if you want it. There is a chat box please feel free to introduce yourself or comment as the session goes along. There's also a Q&A tool. And you can use that to queue up questions for the presenters. After the presentation Diane Goldenberg Hart will materialize and moderate the Q&A. And I think that's everything I need to say logistically so let me introduce our session. We have with us today Katie Oberling and Fernando Hoches LaGuardia, both from the University of California at Berkeley. So in that sense they are colleagues of mine. They are with the Berkeley initiative for transparency and social sciences, which is a really important project trying to advance reproducibility and replicability in the social sciences. And they have a really interesting angle on some of the work they're doing in that they're thinking about how to integrate this is a pedagogical tool as well as a well as a set of good practices for researchers. And I think that's a really important idea that I'm looking forward to hearing more about. So without further ado, let me thank Katie and Fernando for joining us and turn it over to Katie. Thanks Cliff. That was, that was a really great introduction. Yeah, so I won't talk. I won't read this whole slide because I think Cliff has done a good job for me but I'm Katie. Fernando is also here and I'll talk to you a little bit about the the Berkeley initiative for transparency and the social sciences of which we're both apart but today we're focusing on a project that is very much still in progress but is kind of getting ready for to share with the world that is trying to advance specifically computational reproducibility. And as Cliff mentioned also using this as a teaching tool. So, there we go. I'll just give you a quick overview of what I'll be talking about today so I'll just spend a minute or so on who bits is. I'll talk to you about why we care about reproducibility. And then I'm hoping to spend the bulk of this presentation on the actual project itself and at the end I would love to since I said this is a in progress project. I would love to get some feedback and your, your perspectives on what we're doing because we're at kind of a couple inflection points in terms of the design of the project so hoping to hoping to hear from you all later on. Okay, so bits. The, the initiative that both Fernando and I are part of our goal is to promote ethical transparent and reproducible research in order to improve the integrity of science and eventually to inspire better public policy. And we're doing this in three ways so we're generating evidence through meta research. We increase access to open science education. So we run trainings we develop curriculum. We provide funding to, to other instructors who are trying to either create their own curriculum or adapt ours for their contacts. And then we also try to strengthen the kind of scientific ecosystem so a lot of this is through this kind of infrastructural development kind of the project that I'm talking today fits pretty squarely under this, this category. We also it also involves kind of developing open science policy with specific partners and developing protocols for for labs and research centers. Okay, so why we care about reproducibility. I think most of you on this call are probably aware of the kind of replication reproducibility crisis that we've been discussing a lot the past couple of decades in the social sciences. I won't get into what these too much into the, you know, the research on this but I wanted to kind of get us all on the same page about these two definitions that we're working with so for computational reproducibility we're really just talking about trying to get the same findings using the same data and the same methods that another researcher has used in a paper that's been published. There are a number of reasons why this might not happen. You know some of the data might be missing the code might not be well documented, the software that was used originally the version of that software might not be available anymore, or it might not be available in the first place. And so the, the figure on the right is from a recent paper from Gertler Galiani Romero that it was examining 203 papers published in economics journals. And these were journals with data and code availability policies so they're kind of the reasons I just mentioned led to a fairly significant number of findings that weren't reproducible. And on the other end where we're talking about using different data and or different methods to find somewhat similar findings at least you know in the same direction within a reasonable margin of error. And the reason that we try to distinguish routine these two is because while replication, we know is really valuable and really important for trying to understand the credibility of research. And we think that it's, we think that it's useful only to the extent that the original work was also reproducible. You know so someone might not be able to replicate a study. And that could be because the, you know the methods were flawed or they, it was a, it was just just by chance, but there's also a really good chance that that work just wasn't reproducible and in that case you're, it's almost a waste of time and so we are going to get everyone on the same page about starting, you know, putting the, not putting the cart before the horse and starting with the reproducibility part first. These are just a kind of this table is just showing the, the, the kind of rates of replication and reproducibility that have been found in recent years. And some of you have probably heard of the open science collaboration. Other efforts have found anywhere between 30 to 60% replication rates which all things considered isn't so bad reproducibility rates at least in economics are a little bit more alarming. And this is one of the reasons that we're trying to focus on this. Okay, so the kind of solutions and mitigation strategies that already exist. So let's talk about the transparency and openness promotion or top guidelines. So these are a tiered version of a tiered guidelines for journal policies that include anything from pre registration to sharing data to all the way to publication verification. It's a mouthful but basically that just means that a paper, the findings in a paper must be verified prior to publication so that means that the data and the code that the author provide must get the same findings that they are claiming to have found in the paper. I think maybe a couple dozen journals that we know of that are doing this now. Notably the American Economic Association or AEA released a new policy last year that that does this and this was kind of one of the motivations for the project that I'll talk to you about in a minute or so. We're working with the data editor to try to make it easier both for him to verify the reproducibility of the work that submitted to him, as well as prepare the the discipline of economics for complying with this policy moving forward. So the AEA manages nine or 10 of the top journals in economics and so kind of what the policies that they adopt hopefully kind of trickle to other economics journals in the discipline they're pretty influential. And so the idea is to kind of prepare the discipline at large for for being able to do this. I mentioned this list just a few others so that a couple of journals and political science as well as nature methods and meta psychology are also doing this. And then, I'm sure all of you know about these tools but the kind of the other side of enabling reproducibility we see as these, this digital infrastructure so your data repositories, your version control tools like GitHub, dynamic documents. So I've got, I have Jupiter here but in our trainings we generally focus on, you know, the just dynamic documents using state or are which are the software programs that are generally more used by social scientists. So I wanted to show you this is from a recent paper from Garrett Christiansen and colleagues, just kind of showing you the rapid rates of adoption that have happened in the last couple in the last decade or so, especially for sharing data and study instruments. So this is, you know, this makes us feel a little bit better about what's happening in the social sciences, but this is so this is focusing on economists political scientists sociologists and psychologists. But what this doesn't do is doesn't show us anyone who's not publishing in the top 10 journals in the discipline. So, you know, while this is, this is exciting for us and you know kind of justifies a lot of the work that we've been doing for the past nine or 10 years. And, you know, it's not unlikely that the rates of adoption would be lower for for people who are not, you know, publishing in these journals or going to these programs. So, that kind of leads me to my last slide on the motivation for this project. We've been thinking a lot about what scholarship means. A lot of people when they think of scholarship are thinking about the article, the, the main findings of an article, the conclusion. And I think something that we've been a more inclusive and broader definition that we have been working with has been the Clairvout principle that was paraphrased here by Buckhead and Donahoe. I'll just read for you really quickly. The article about computational results is advertising not scholarship. The actual scholarship is the full software environment code and data that produced the results. So what we what we're hoping to do with this project that I'll talk about in a sec is to kind of improve access to all of the data, the code and the decisions that went into creating these this final product is going to be published in the journal. The image we have here on the left is from the Garden of Forking Pads by Borges. And so you can kind of think of a paper as being the red line from point A to point B. But what it doesn't necessarily tell you is how the researcher got to point A in the first place. If they tried all these other paths and you know didn't didn't decide to go down them, or if they did go down them. It didn't work out. All of that is is useful knowledge, but it's not it's generally not going to be published in a paper. So having access to that knowledge can not only kind of accelerate our understanding of the world and of science, as well as supporting the learning of people who are, you know, digging through the materials and the data and the code to use for their own research. We also see is see it as improving inclusion and participation in science. So you can think of, you know, a lot of recent studies about kind of privilege and inequality and academia recently have been revealing that there have been unequal access to opportunities to learn from your instructors to learn from your peers. So you can think about things like homework groups, office hours, knowing when and how to ask for help who to ask help from. And so there seems to be like a growing body of literature about that. Our hunch is that there's also kinds of implicit knowledge that continues into, you know, graduate school and beyond so you can think about things like how to ask for data. How to talk to people about their, their research efforts already been published. How decisions and research are made, you can even take it all the way to how to navigate authorship conversations, but in the context of reproducibility. It's really difficult without the you know the full set of decisions and data and code to know exactly how someone got somewhere. And so a big part of our motivation is, is trying to improve access to that in an equitable way. Okay, so the acre project itself. This project includes kind of for major activities. We're developing curriculum which I'll show you in a sec. We have students and instructors on both how to how to go through the curriculum and how to teach it how to adapt it for their own courses. We're developing an online platform that is going to help us crowdsource reproductions, as well as improvements and then also spur discussion around around this work. And eventually, once we have enough reproductions, we're hoping to conduct a few assessments of reproducibility across journals and fields. Okay, so the curriculum starts out with this guide. The acre guide is a set of step by step instructions for conducting and recording a reproduction and notably it includes chapters on how to choose a paper, how to assess reproducibility, how to make improvements, how to conduct robustness checks, and also how to have constructive conversations with authors. I wanted to show you, I'm actually not sure if the sharing is going to let me do this, but I might have to do a new share. Let's see. Okay, so one thing I wanted to show you this is part of the guide and I included the URL in the slides. What we are are hoping to do with this project is to help people move away from these kind of binary statements about, you know, this paper is reproducible or it's not reproducible, and instead identify what exactly makes it not reproducible, not just at the paper level but at the claim level you know paper might include several claims within within it. And, you know, it might end up being that, you know, some of the cleaning code was gone or it was missing, and the student or the reproducer whoever they are suggests new cleaning code to add to the code that was already shared, or if no code was shared at all they can suggest code and move the paper or the claim from level six reproducibility to level seven. And we're hoping that helps, that helps foster more constructive conversations around this kind of thing and because the idea is not just to say this is reproducible or not it's to it's to move the the fields of social sciences forward. I'll go back to the slides. Okay, this is back on the slides right I'm not I'm not quite sure what people are seeing. Yeah, you're on the slides. Great. Okay. Okay, so I'll talk a little bit about how this has been going. We, this is generally taken up two to four weeks of graduate courses that we've piloted this with. It kind of just depends on how, how far you want your students to get into stuff. We've also piloted this with one undergraduate who who conducted their undergraduate thesis on this and it took them a whole year but they were getting really really deep into the details of a specific paper. So far we have piloted this with 37 reproductions, mostly in development economics courses. And so far people have have seemed to enjoy this, and they've actually found fairly high rates of reproducibility so this is, it's just within one one discipline but so far it's going going alright. So the kind of incentives or benefits for the people doing this might differ from differ depending on where they are in their career. We're, we're kind of thinking that graduate students might be more interested in getting credit for making their improvements or assessing reproducibility, as well as for kind of having language to discuss papers after they've been published, and also learning what the process is or should be for accessing data and materials, how to talk with authors, and empowering them to carry out these exercises as they move forward in their careers. And then undergrads as I as I mentioned earlier, might be more interested in kind of getting a closer view of what research is and what entails. The platform itself. This is this is exciting for me because we we've just gotten to a place where we're pretty comfortable sharing this fairly widely. But the social science reproduction platform or the SSRP is an open source online platform for crowdsourcing these reproductions. Just a couple weeks ago we got our cross surf membership so we'll be able to assign do is so these group productions will be citable. And we're hoping to in the next month or so to integrate the discourse form so this is an open source online form where people will hopefully be able to you know give feedback on other people's attempts to reproduce or their improvements. And then ask questions ask for help. And then eventually, I mentioned this as well, the kind of aggregated results on a on a metric dashboard showing the levels of reproducibility of certain sub fields and journals etc. The figure here on the right is from the platform and I also wanted to show you that so I think I'll do a new share. So this is the platform. I also included the URL in the slides, and I'll share these afterwards. If you want to take a closer look. But this is what it looks like so far. The main thing I wanted to show you I'm signed in here is kind of what the start of one of these forms look like. So I already started one this is a dummy but so it's, it's got a kind of a not not super helpful titles and data but I'll show you anyway. You kind of start out with what the entering the basic information about the paper that you're trying to reproduce. You've got your repack handle. This will be different depending on the discipline but in economics repack is a is a helpful database of papers that have been published regardless of journal working paper status. So the title your journal the year publication the DOI the authors. Importantly, it asks you if there is a package reproduction package already available and if there isn't. It takes you this question of, have you contacted the authors for the package. So you can't move on. So the idea is that they have to then go to the guidelines. We've got a chapter here on on having constructive conversations with reproducers are sorry with original authors. The part that I wanted to show you was we've actually sorry. Here we go. We've developed a few email templates. So trying to make it as easy as possible for people to talk to these authors. So we've got, you know, the, the salutation. I'm contacting you to talk specifically about this paper. So being as clear as possible, who they are where they are, why they're trying to reproduce this in the first place often it'll just be because they're part they're part of course work. And what exactly they, they were able to find and what they weren't able to find, taking that all the way through, you know, requesting specific missing items, asking for additional guidance, responding when people have refused to share, or if they haven't really given a reason, etc. So the idea is that, you know, there has been. Let me get back to the slides. I'm not sure if that was the right one. Here we go. You know, there's been kind of a history, at least in economics and other social sciences of, you know, retaliation when people have tried to conduct these reproductions. Sometimes these conversations can become hostile. And the idea is that we, we hope to try to guide people through discussions in a way that is going to avoid any of that kind of language. You know, make make it make it a learning process, both for the student and for the author in terms of how to actually make their work reproducible, moving it from, you know, level to level eight or whatever it is. The last thing I wanted to mention before I kind of opened it up for questions is that we are hoping to make it easy to for people to build on on top of other people's attempts to reproduce so someone might have spent, you know, a few weeks trying to assess this reproduction, you know, their course sorry assess this paper for reproducibility, their course ends. Someone else wants to go back to that paper and make improvements and instead of having to do, you know, contact the author bother that person again. There's a record of that person having contacted them, what they said, what they were able to provide. And what, you know, how far the original, or sorry, the first reproducer got in terms of assessing or improving and the next person can more easily build on top of that work so saving some time there as well. Okay. The last thing I wanted to talk about was that so I mentioned that we're we're still in progress. There's a lot of open questions that we have, especially as we're getting ready to launch. The main one is if and how to allow anonymous reproductions. So I talked about kind of the the risk to reputations that that people have perceived and felt in the past. So we want to balance that with, you know, we don't want to create a space for trolls and internet there's also been a history in economics of toxicity and some of these online forums. So we're trying to you were trying to balance this this kind of trying to create this kind of constructive space for discussion. Also, de identifying these attempts is definitely not foolproof. The most we can do we think is to give people guidance on how to de identify, you know, their code. But it's, like I said, it's if you're using GitHub, especially, you're going to have to go through the whole thing and make sure your name is gone from all of your commits, which could be pretty tedious. We're considering kind of embargo period so if you if you do a reproduction anonymously than maybe within, you know, two to four years and it automatically becomes non anonymous. And we're also trying to understand how to assess quality for those kinds of reproductions. Other open questions we've got are how to integrate other research databases, you know I talked about repack but there's lots of other ones in different disciplines. It's really just how to get buy in from social science scientists for this. You know how do we demonstrate and communicate the value of this. You know, maybe we maybe it's an encouraging citation or maybe it's more just that we want to improve the field of economics in general. So at this point I'm happy to take questions Fernando is also here for questions but we also would love to just hear from the audience if they have ideas about any of these questions that I've posed, or thoughts about the platform in general. So I'll open it up. Thanks Katie. That's a really fascinating tool really interesting. And it's great to hear about it. I just want to let our attendees know that of course you can certainly type your questions into the Q amp a but I think Katie and Fernando would love to hear directly from you about some of these questions that she raised and other issues surrounding their development in this tool. If you're, if you would be, if you have something to add to the conversation and you would like to be unmuted. Just raise your hand and we can definitely unmute you so that you can interact directly with Katie and Fernando, or type your questions or comments in the chat. I'm more in the Q amp a and I can, I can definitely moderate those here now. I just want to remind everyone that this is a CNI fall 2020 membership meeting will have another webinar coming up at 3pm Eastern time. And that will be the Catholic news archive. I know there are a lot of things going on this week, especially today. Thank you so much for spending some time here at CNI. And while we're waiting for people to jump in. I just want to say that that was a wonderful presentation and I think you've produced a really potentially very powerful platform here. I'm wondering if you can share a little more about the reaction from faculty who have been looking at this as they think about using this in the context of their own classes and in particular, maybe what kinds of classes what level of students. Those sorts of things. Yeah, sure. And so, so far we have presented in in one academic seminar where where faculty were very curious about it. They seemed interested about adopting a tool but there was no specific commitment to who and when they will take it. And we are piloting we're aiming to two pilots with several courses in the spring. And across different universities and the target audience for this would be a second year students of graduate courses and masters or PhDs where they are doing some applied work and and these these these exercises of doing reproductions are somewhat standard part of their curriculum. We just want to give them these these tools to make it a the the reporting process of these exercise tools to standardize it. And the two courses that we've piloted and already have been in Ted Miguel's courses and he's the one of the PIs on this project so it was easy to get his mind but yeah we're hoping to identify a few more this next semester in the spring and figure out how how easily or how hard it is to to kind of adapt this for a full course. I will also say so the day editor Lars Bill Hoover who's also a partner on this project. He's the one who's in charge of running these verifications for a journals. He does this with undergraduate students and his policy is that if an undergraduate student can't understand what you've done then you need to work on it to make it more easily understandable. And so, you know it might take an undergraduate longer to you know do this as part of like a thesis project or something but at the end of the day whatever is published should be should be fairly clear. Yeah, that's part of what I was thinking was this is this is this is an opportunity to provide an amazing educational experience for an undergraduate in terms of really coming to grips with a piece of published research. Yeah, and the guidelines are the guidelines of this project are structuring a way that that in order to do a reproduction you do not need to have in depth knowledge of the specific paper. Basically, the, the idea is that in the initial stages, what do you need to be able to identify is that what's the scientific claim that is being made in the paper. So, what's the set of scientific claims among that set which one would you focus on, and then the other thing that you need to focus is for a given claim you need to be able to find the result in the paper supporting that claim. So this result might be one number in a table in a paper so you provide the location you say that's that's my target. This, this is the claim and this, then you go look at the reproduction materials and how far can you get along into basically being able to run the car so to say to get to get to that point. And, and the last stage is, it's about robustness and that requires a little bit more in depth knowledge so it has to with like how to think of things and to see how, how the results change, but but it's definitely the materials are definitely developed with them with the mindset of a broad audience. That's great it's really interesting. It's a great tool. And we'll look forward to serious seeing what comes out of these questions that you're asking and investigating. Yeah, one, one thing I'm really curious about downstream is this matures is there are many folks in academic and research libraries who actually get quite involved in teaching these kinds of skills and it would be very interesting to see how they begin to make use of or make reference to this body of work for folks that they're teaching outside of the sort of, you know, formal curriculum if you will. Absolutely. Yeah, I think librarians for the past couple years we've realized if our are some of our closest allies on this campus so we'll definitely be working with them to figure out if they can not necessarily, you know, put it in a course but how how they can integrate it into their work with students to I will shut up now I've been advising all the q amp a on this fantastic presentation. Well, thanks Cliff for the questions and thank you Katie and Fernando for your thoughtful responses and this great presentation. We are a little bit past time now. I'll turn it down the public portion of the presentation I'll stop the recording we do have a few attendees still with us here and if, if they're interested in approaching the podium and having a chat with you after we turned off the recording. Just raise your hands I'll be happy to turn on your microphones and thank you so much for your presentation and thanks to all our attendees. Bye bye.