 We are a few seconds past the hour, so I think we will get started. Thank you everyone for attending this webinar about the emergence of grassroots networks promoting open science. We are delighted to have this discussion today for an hour with representatives from a few different networks who are doing a variety of things to try to promote open science in their regional communities. And so the goal for this particular webinar is to have a very open discussion about how communities are trying to engage and shift norms and practices for the research communities that they serve and hopefully provide a breeding ground for some communication and collaboration among us so that we can do this better and more efficiently and share the knowledge and practices that we are pursuing together. So I'll just make a few preparatory comments and then each group of our panelists will share some of the activities from their perspective and the things that they've been working on. You as attendees can have chat during the discussion in the chat window and it appears that John Tennant already got that started. Thanks John. And there is also a Q&A button at the bottom of your screen where if you'd like to ask questions from different members of the panel or for broader discussion, you're welcome to post those there at any time. While they are giving their remarks, I will monitor those and try to provide some stimulus for them to respond to and then after each of the presenters finishes, we will use those Q&A questions to promote some more general discussion. So before we transition to them, let me just make a note that when we use the term open science, there's a lot of different communities that are doing a number of different related activities that fall under that umbrella. Sometimes open science is about transparency. Let's make the outcomes of research more available, open access. Let's make the data or the materials, the code behind that research more accessible, or let's make the process more available, showing that what people have planned in advance versus things that are discoveries after the fact. Sometimes open science refers to rigor. How is it that we can improve our research practices and make them more robust and accessible and reproducible overall? And sometimes open science is referring to inclusivity. How is it that we can make scientific community? Contribution to science more available to everyone and change the kinds of models of how it is people can get can make contributions and get rewarded and acknowledged for those contributions. So that diversity of activities that that embody open science is also diverse in the various grassroots communities in terms of how they're trying to engage others in trying to improve research practices or align the values that we have for science with what the daily practices and what's rewarded. So each of the communities that present today will give their own flavor of the things that they have been working on to try to do these, promote some of these new practices. And what's been most exciting over the last few years is that the emergence of many of these community networks. That really the open science movement is characterized much more by grassroots campaigns. Researchers deciding that we can do better than what the present practice is and starting to work together to make those changes happen in their local communities. But it's also clear that the challenges facing the open science community are difficult ones because they are culturally embedded. And so grassroots communities can leverage each other to more effectively change the cultural practices that make it make provide constraints for researchers to be more open or transparent or more inclusive in their research. So our primary hope with this kind of communication is that we start to see that other communities are doing some of this work and that we can each learn from each other to try to maximize the quality and impact of that work. So I'll make one more note and that is that this is being recorded. It will be available afterwards. And so if you miss things or if you want to share this with others, there will be a link for that at the end. Okay, so let's go ahead and transition right to the presenters that will be talking about their communities today. The first one will be Anita Erland from the Open Science Communities Utrecht. She represents a variety of open science communities that have emerged in the Netherlands, each in a city based format. And so she'll tell us more about the work that she and her communities have been doing. So Anita. Okay, thanks, Brian. Let me see if I can share my screen with you all. Doesn't seem to work, right? Or does it? Not yet. You're not shown. There it is. And I see me. Wait, but this is not the screen that I wanted to share. It's this one. Yes. There you go. Perfect. Okay, thanks, Brian. My name is Anita Erland. I'm an assistant professor at Utrecht University in the Netherlands. And I will first briefly explain why we, Luc Brigman and I, came up with the idea of an open science community and then talk about the different formats that we have in place. So the whole open science movement was started by researchers who thought that we could accelerate scientific progress by making science more accessible, transparent, robust, and inclusive. And policymakers reacted to that. And now you can see that there are policies in place with respect to open science on a local, national, and international level. But having policies in place will not automatically make people adopt open science practices. So that's why we thought if we want to have a large scale adoption of open science practices, we need a grassroots initiative. So some people who we will call the experts are already practicing open science and they form the so-called open science bubble. And then there's a large group of people that are not practicing open science yet. But they are at least interested in the topic, interested in learning more about open science. And in order for them to do so, they need to know what open science is and how and why they should practice open science. And then there is a group of researchers that are more or less ignorant when it comes to open science. They either don't care or are against open science and those researchers will probably only adopt open science practices if they really have to. And that's not a group of researchers that we focus on. We really try to focus on those that are interested in open science and are just outside of the bubble. Because for a large scale adoption of open science practices, it is important that we reach those people that are just outside of the bubble. So those are the target audience for our open science community. We need researchers to want open science if we really want everybody to adopt these practices. And that's why we came up with a bottom-up platform for researchers with an interest in and researchers with knowledge with respect to open science. So this displays that we need both experts with respect to open science, but also those that are interested in. We really didn't want to create another group of experts that talk to each other and tell each other why open science is so great. We really needed a group of people that are interested in open science and we needed them to learn from and hear from those researchers that are just one step ahead of them. So that's why we focus on experts and those interested. And that's also why it can be part of the open science community, even if you have little or no knowledge with respect to open science yet. So we really want to be as inclusive as possible. So then there's a couple of goals that we have for the open science community. First of all, we want to make open science more visible and accessible. There are lots of people working on open science, but you don't always know who they are and what they're working on. So making open science knowledge more visible and accessible is an important goal. We also want to promote knowledge exchange. So those that that have experience with respect to open science shared our experience with those that are not experienced yet. We want to inspire and enable researchers to take the next small step. So don't expect them to go full-blown open science within the day or so, but really help people to take the first small step. And then what is also important is that we want the open science community to identify obstacles and a need for support so that we can inform policymakers what is needed, what their community needs to adopt those policies. And to reach these goals, we have a couple of formats in place. So the first thing that we did was we created a website that lists who we are, what we do. It has an agenda and it also nicely lists the members that we have. So right now we launched our community a little over a year ago and we right now have 160 members across the university and from all different faculties. Being a member only mentioned of means that you are on our website with your name. There's an example here of my membership. You can list open science expertise that you have and that you can be contacted for by other community members. So if you want to know something about let's say cips for example, you can see here that you can contact me for that. If I click on the the tag it will list all the other people that have the same expertise. So that's I think a very nice thing to make the community visible and to make open science knowledge more accessible. Then we have a monthly newsletter that we try to sign up every month that lists all the activities that we had. We also try to highlight one of our members. Let's see. We're active on social media so we do have a Twitter account. There's also a podcast The Road to Open Science that we do not create but that we host on our website. And then the past year we organized six workshops and open science cafes. So every other month we had a workshop that was open to everybody. It did not have to be a member. It was also open if you did not work at the university. And after each workshop there was an open science cafe which basically means an informal discussion on open science topics. And what we realized was that during our last workshop we had those people that just took their first steps and tell them about it to share their experience with others. And that actually worked very well. So you don't want to have an expert saying to everybody how they should do it. But if you have a person that is just one step ahead that tends to work well. And then we have a couple of new formats that we want to launch next year. So the first thing is stickers. People seem to love stickers. So we created our own stickers and people can post it on their door to actually make open science and the community more visible. And instead of hosting workshops that are open to everybody we came up with the idea of hosting half-day symposia per faculty. So that you can go more in depth with respect to the open science issues that are particular to a specific research area. And another thing that I'm really looking forward to is the member initiative. So we want to highlight or stress the community feel by having our members initiate whatever they want. So if they want to get a group of people together to discuss certain open science issues or to work on something, organize a hackathon for example. They can actually do that and we will help them promote it and get a group of people together. Let's see. Yeah and so as I said we launched our community a little over a year ago and now there are seven other open science communities in place in the Netherlands and more to come. We're still figuring out how trying to figure out how this national network should operate but it's clear at least to me that if we join forces a lot can be done. Well thank you Anita that was very exciting to see the overview of it. One quick question before we transition to the next session is you said you already have more than 160 faculty involved in it. What has been the most effective way to actually grow that group of people that are signed on members? Yeah good question. We started out personally reaching people that we knew were interested in open science and now we have those people reach out to the people that they know. So I think it really helps if you do a one-on-one contact with people just personal contacts, writing emails to people, drinking coffee with people. It doesn't help if you say hey this is the community feel free to become a member that doesn't really help. So you really need to put in a lot of effort to personally reach out to people. Yeah that's great. So there's a lot I think of meat in the what it is you presented as the various strategies of making things visible, of having personal contact and another one that really stuck out to me was really meeting researchers where they are. That idea of having a presenter that's one step ahead rather than way out here sort of talking idealistically of that's way too far. I can't do that. Those seem like very interesting things to one pack. So we'll surely return to that in our group discussion. So thank you for that Anita. We will next hear from Marcus Monofo and Laura Fortunato from the UK reproducibility network. Network that is coordinated at the national scale but has a lot of individual sites that are involved in this and will complement very nicely I think some of the things that Anita described. So Marcus and Laura please. Marcus you're still on mute too. There we go. Does that screen look all right to you? Yes. Excellent. Okay so thanks for the invitation to speak today. I'm going to start by talking about the UK reproducibility network generally and then Laura is going to talk a little bit more about how that is realized at a local level at her own institution of Oxford which is one of our more mature local networks and in a moment you'll see what I mean when I talk about a local network. So the the reproducibility network that we've built up in the UK grew out of existing activity around meta research, open research and our interest in the drivers and incentives that shape the behaviour of researchers themselves. So our aims are to understand the factors that contribute to poor research reproducibility, provide all incentivised training and disseminate best practice, support and test interventions to improve reproducibility and critically the point that Brian was alluding to ensure coordination with our stakeholders. So we had an initial meeting in September 2018 where we brought together key stakeholders, funders, publishers and other organisations within the UK and talked about the need for this kind of national coordination and since then we obtained some funding from those research councils and other funders that were present at that meeting and used that to establish the network and we now have local network leads which are the grassroots peer led part of the network at I think currently 43 different UK institutions and we're supported by a range of different stakeholders at the moment we have 25 different stakeholders that range from research funders, publishers, learned societies and then other professional organisations and bodies within the UK principally that support that wider academic ecosystem and the steering group of UK RN consists of myself, a psychologist, Lara Fortunato from Oxford who's an anthropologist, Malcolm McLeod from the University of Edinburgh who's a neurologist, Alexandra Collins from Imperial College who's an ecologist and Chris Chambers from Cardiff who's a neuroscientist. So we have quite a broad representation of disciplines and that's reflected in our funders as well so we have support from the Medical Research Council and the Medical Trust which are biomedical funders but we also have support from the Economic and Social Research Council, NERC which funds research in the natural and environmental sciences, the Arts and Humanities Research Council so it's really quite broad in terms of our ambition because our view is that whilst much of the interest in reproducibility has been in the broadly defined biomedical space including psychology, actually many of the issues transcend that and there are some very interesting conversations to be had about what constitutes data sharing and indeed what constitutes data across different disciplines. So the network is structured like this we have the steering group at the centre which provides coordination but the primary structure is one of this network of local networks that are led by individual local network leads that are these grassroots peer-led self-organising structures within different institutions and they take on their own identities and set themselves up in a way that best suits the needs of that particular local institution and Lara will talk about the one that she set up at Oxford in a few moments. One of the initiatives that we have been well first of all so that coordination of the grassroots activity with the stakeholder group that we have which comprises our funders and our publishers and those other organisations that I mentioned is part of what I think is potentially quite powerful about this structure because it allows us to create a direct connection between the researchers themselves and the organisations that shape the incentive structures that are relevant to the issues that we're interested in so we've already had a number of cases of one of those stakeholders for example JISC that provides digital services to the UK higher education sector sending out a questionnaire to our local network leads in order to essentially sanity check some of the initiatives that they are in the process of developing to ensure that whatever they come up with is actually fit for purpose in terms of the researchers that will be impacted by that that new initiative and similarly when we have ideas for how the the funding system that we operate within for example could be improved we can feed that directly back up from our local networks to the stakeholder group so that connection between the two I think is valuable and important but the next step that we're taking which we're just beginning to develop is to try and bring in that middle part which is the institutions themselves so you have the grassroots of researchers on the ground you have the stakeholders that are providing the incentives at the top level but the organisations that hold much of the research culture and incentives in terms of things like hiring and promotion practices are the institutions themselves so we're now in a process of recruiting institutions to formally join the network and we have a number that have committed to doing so and are just working through their internal processes before they can sign that off but what that entails is creating within one of these institutions where we have a local network a senior role reporting to the senior management team with a title similar to academic lead for research improvement obviously individual universities make shape these in ways that suit their own local structures but Bristol has just created one of these roles and that is intended to sit between the grassroots and the stakeholder group and provide another point of contact with a part of the wider academic ecosystem if you like that is relevant to shaping incentives in the way that I would imagine we're all interested in so that's the state that we are at the moment we have recently completed an exercise where we think about the kind of theory of change that underpins the work that we're trying to do this figure represents that I'm not going to go through it in detail but it illustrates that we're talking about a very complex and interconnected problem and we can zoom in on different elements of that and think about how we can shape those different components through the activity of these grassroots networks through engagement with the various stakeholder groups the sense that I have certainly within the UK is that there's a great deal of interest from many researchers but also from many funders and publishers and others in improving the quality of the work that we do but there's no one coordinating body that allows all of that to be essentially oriented in the same direction and so part of our role we think is to help support that kind of coordination and there are a range of different activities that we've engaged in in year one and we're moving into our second year at the moment we have a range of activities that where we're working with stakeholders so many of these you'll be familiar with registered reports and registered reports research grants or funding mechanisms accountable replication policies at journals editors for better research so we have many journals that we're working with on adopting registered reports for example that's led by Chris Chambers partnering journals and funders through these registered reports research grant scheme and you can see one of the pilots that's underway at the moment between the journal Lichtyn and Tobacco Research which is published by OUP and Cancer Research UK a major funder in the UK and we have some funding from the welcome to evaluate that pilot and feasibility study to determine whether or not it would be feasible to do a full trial of that that model of combined funding in journal peer review and then for researchers for the grassroots networks we're setting up open research working groups we're supporting those networks to establish reproducibility journal clubs we're working on a range of short courses that will provide training in the broader skills around for example how to go about sharing data and what some of the issues are that may differ across different disciplines we're working with institutions on their hiring and promotion practices to incentivize different open research practices for example through my role as academic lead for research improvement at Bristol we recently included data sharing in our promotion criteria and that's something that we're hoping to roll out across other institutions and then we have other projects like consortium based student projects led by Catherine Button at Bath and we're developing a series of primers on open research practices again to allow us to reach out to that outside of that bubble that was mentioned earlier of researchers who are already engaged with these issues to those who might be interested in adopting these kinds of ways of working but need some guidance on exactly how to do that so that's where we've got to it it feels like we're at quite an exciting stage in terms of the sheer amount of interest that's been generated both at the grassroots but also at those higher levels as well so at that point I'll hand over to Lara okay so I don't have any slides but I'll be very happy to talk through what we've been doing here in Oxford and how the local network interfaces with the broader UKY network that Marcus has just introduced so the the network I coordinate here with many others is reproducible research Oxford it's an initiative that started in 2016 officially we applied for some funding from university IT services and the focus then was specifically on provided training in basic skills that researchers need to make the research more open and reproducible so we did that specifically by setting up a partnership with the carpentries software and data carpentries which some of you may know are community based community led organizations that work to provide training in these areas and for the first couple of years we effectively focused on providing that training so we run about seven workshops I think here in Oxford tailored to anybody in the university from undergraduate students all the way to senior professors technicians anybody librarians you might be interested in or involved in providing this sort of training or applying these skills and these were quite successful I think we catered to upwards of 200 learners uh the workshops were free um they were run they were run by volunteers instructors and that was the initial focus another side of that was also to train instructors based serinox for the who could then go back to their own departments go back to their own units within the university and provide training to their colleagues so we run two instructor training events over the two years and those were quite successful and frankly over the subscribe so to indicate that there is actually quite a lot of demand in the university of this size now alongside this initial effort a number of different activities had developed some run in different departments some run by early career researchers for example spanning the whole range the whole spectrum of of things that have to do with open science and open research and say with the creation of of uprn of the eukaryocritic network that marcus has just introduced this seemed like a good opportunity for us to try and bring together all those efforts within the university and create one broader community because it seemed very important that we have a sort of a one direction of travel that we're all speaking to administrators for example within the university and have one message and I'm trying to press particular points that we want to see so effectively the past year has focused on trying to get that community together and coordinated with the uk rn on the one hand but also and I think this is important for for a network the stage that our network is at we've been looking for opportunities to fund somebody's time effectively to run some of the activities because of course everybody is heavily involved in this activity but we're all doing it on a sort of goodwill basis and of course that's an limited resource and also it's it's it's not necessarily effectively used over the long run so we've been applying for funding from the university and that's been a core focus of our activity over the past few years in a university of this size so here in oxford we do have different pots of funding available and this may vary of course across institutions depending on size but I think the one thing that we've learned is that it pays to be creative because different units within the university might be interested in this and might be available to to fund bits of the puzzle so that's where we're at in terms of of sustaining the network for the long long term really in terms of activities I mentioned the workshop that we've been running we are planning a summer school which run already last year so this will be running in September this year in partnership with Berlin there is a very successful journal club that has been hosted by St. Parsons and Amy Auburn in Psychology which is named the early career researchers and that's been going for about a year now I guess the reproducibility journal club so there are many different initiatives that have been happening in oxford and really the focus now is to try and coordinate with that energy and channeling it into one direction of trouble and I guess one final point I would make is that our focus here has been to try and build as inclusive a community as possible in terms of disciplines say Marcus mentioned I'm an anthropologist I'm an evolutionary anthropologist so here in oxford I sit in the social science division and so perhaps not the most obvious target for open research initiatives with that in mind the lesson that we've learned from this is that there is a lot that we can learn from other communities and how they interpret open science and the problems around open science and so really that has been a really interesting and important focus of our efforts to try and bring different disciplines together across all areas of study so not just the part sciences so we tend to use the label open research for that reason to involve for example people in humanities so excellent Laura Marcus thank you for that it's incredibly impressive the scope of things that you have organized locally and nationally with this work one of the most and there's a lot of things to talk about with it but one of the most important things it seems to me that the network really provides is that through way of communication between the stakeholder groups and the individual communities it can be in a lot of these grassroots efforts that at the local level it can feel like it's just fighting a fight that can't be won because because who can change the culture at that scale and so yeah we're training we're learning we're getting all these activities but what's going to happen and then even among stakeholder groups of curious observation and my experience has been that feel very divorced from the realities of the everyday researcher and don't know what to change or how to change it because they don't know what impact it will have on people on the ground so I wonder if you can briefly before we transition to Alexander say a little bit about how you were able to get started in getting that network of stakeholders together and is it now just sort of self-sustaining that they see the value of this connection or are there things that you need to do to cultivate that um I mean it took a long time to get everyone in the room once we managed to get everyone in the room then there was a huge amount of enthusiasm for what we were proposing I think that I just spent maybe five years cultivating networks not just myself but people like Lara and Chris and and others making connections with the relevant stakeholders with the publishers with the funders we each had different connections if you like and we were able to leverage those to to get them in the room but we were also able to capitalize on serendipity because the science and technology committee of our government held an inquiry into research integrity last year and whilst the focus of that was primarily research fraud they did also touch on reproducibility and that focused the minds of the funders in particular but also the institutions and so there was a sort of almost teachable moment that we could use to get them in the room and once they were in the room and they could see just how much need there was for coordination amongst themselves then they were very enthusiastic to support us we're at a sensitive point at the moment because they funded us in most cases for the first year and they want to see what we've done with their money in the first year I think we have a pretty good return to show them on that investment but it will be moving from the model that we have at the moment which is lots of small plots of funding from individual funders through to something that's more sustainable that will be the next key transition but I'll just pick up on something else you said I think it's really true that that there's a need at both of those levels the grassroots level and the stakeholder level for that connection and at our meeting in March one of the powerful things was that we had a meeting in the morning with our local networks and the meeting in the afternoon with our stakeholders and in the middle of the day over lunch they were able to mingle and interact and we ended up with one of our early career researchers who leads one of our local networks sat next to the editor-in-chief of nature talking about reproducibility issues and they exchanged email addresses and have been in touch since and both of them you know found it to be a valuable interaction so I think one of the things that we're really keen to do is to allow people to develop those relationships and to interact with each other and also to promote initiatives that just seem like you know good initiatives that are valuable to the community we've talked about a few of those but those are really not ones that we've generated ourselves so reproducibility for example as Lara mentioned originated from a group of early career researchers in in Oxford Amy Orburn Sam Parsons and Sophia Cruel and it's still very much their project we're just providing a bit of support to be able to you know buy teapots and send those out to our local networks and so on to to just enhance what they're already doing that's excellent yeah you can just really see how once these networks are established they just keep defeating each other like that as a perfect illustration of that there's lots more to unpack there listeners if you have questions that you want to make sure the panelists address afterwards please do drop them into Q&A I see that there are a couple of there already but we'll get Alexander to present about how bits is working and then we'll open up for a broader discussion so Alexander Bogdanowski from bits you're up thank you hi good morning everyone good morning from Berkeley thanks to you Brian and your colleagues at the Center for Open Science for putting this together let me start off with by telling you a bit more about what bits is and a little bit about our origin story so bits is an initiative of the Center for Effective Global Action which is a UC Berkeley based research hub which focuses on international development we've been around since 2012 practically we were established by a group of researchers from across the social sciences I think including you Brian who got together at Berkeley to discuss the use of pre-analysis plans and study registries so the idea back then was sort of facilitate exchange of ideas and best practices across the social sciences given that researchers across these disciplines face similar challenges and respond to similar incentive structure and then this sort of interdisciplinary and facilitating approach has been at the core of what we do ever since in a nutshell what we do is we work to advance the rigor ethics transparency and reproducibility of social science and we see this as means of improving its credibility we interact with not only researchers but then also funders research institutions journals and as I said well in our interactions we mostly act as facilitators namely by convening them and facilitating consensus and network building and also by empowering them to push for change within their environment we're a relatively small team we only have we're essentially a team of four with two scholars namely our faculty director Ted Miguel and Fernando Hostos de la Guardia who's our project scientist and two program staff Katie Horbeling our program manager and myself however we do get a lot of support from our 13 member advisory board and our colleagues at CEA. So what we do has evolved over time but our activities mainly fall under three general lines of work so first of all we conduct and support meta-research so this is including research projects on factors that contribute to poor reproducibility but then also research that develops and evaluates the effectiveness of solutions such as tools and practices for open science we conduct training and then a large part of this a large part of our work in this regard is our is based on our in the work of our catalysts where essentially our network and these are graduate students academic faculty and other researchers who teach and advocate for open science I'm going to talk a bit more about them later and then finally an increasing part of our work is our work with journals institutions donors where we help them develop and implement policies and protocols to support open science so just to give you a general idea of our the scope of our work here are some numbers from our work over the last three years so our network consists of 137 individuals who are based at 100 different institutions so across the world and 33 different countries over the last three years our network and through our training through trainings that we've organized on our own and through our MOOC that I'm going to talk about also later we've reached out to roughly 4,500 researchers our research portfolio consists of 32 different metasize research projects and then in the last three years three to four years we've allocated we've compellingly allocated $760,000 to support research and training activities led by our network so let me delve a little bit deeper into each of these three buckets so in terms of how we support research smart has been our flagship program or that stands for social science meta analysis and research transparency grants through this funding scheme we've allocated roughly six six hundred thousand dollars between 2015 and 2017 to 22 different research projects and then these span from all these span all over what we what we now know as as meta science so from research on researcher and publisher practices such as investigation of publication bias in economics journals or the misuse of covariates to achieve statistical significance in political science to a valuation of tools and methods such as an evaluation of the effectiveness of pre-analysis plans or data use agreements to facilitate data sharing for meta analysis and then some of them are field and topic specific meta analysis or systematic reviews all of them you can you can learn about all of them on our website and then some a lot of them are have also been published on meta archive which is the preprint service that we run so in addition to supporting research we also conduct research and by we I mean mostly Ted and Fernando I want to highlight a current or a recent project which is the bit state of this of social science study or the 3s study which is like we like to say the first representative longitudinal survey of researcher norms and practices across the social sciences and early though this isn't still work in progress and early insight is has provided some good news for the open science movement so just think that over the last decade there has been a rapid expansion of the use of open science practices accounting for sharing data study instruments and registering so based on the responses that we that we got from our sample we learned that over 80 percent of scholars have used at least one of these practices which has been a rapid increase from less than a quarter over a decade earlier another interesting another related project in this line of work is our upcoming forecasting and initiative where we work with Stefan Odell Avigna and Eva Vival and as part of this project we will build a platform to systematically collect prior so this is going to be sort of a public resource and then we're going to test the platform through a dozen half a dozen pilot projects and hopefully that will help us generate insights on how to improve the accuracy of forecast in general and then finally though this is not an actual research output this is a repository we run a meta archive which is we hope or we hope that it will become a go to repository for research and meta science so far we have only published 61 preprints so any one of you who are doing meta science I encourage you to consider posting your your work on this this preprint service in terms of our capacity building efforts our sort of flagship program is a research transparency reproducibility training or rt2 which is a three-day workshop on pretty much all aspects of research transparency and reproducibility so from understanding the drivers of of the reproducibility crisis through various tools and practices for improved transparency and reproducibility such as analysis plans pre-registration data sharing etc we've organized seven of these so far including few international ones in London and Amsterdam and I like to use this opportunity to advertise our upcoming one in Washington DC which is going to take place between September 11th and 13th and we're still accepting applications for this one so in terms of how we support our network to do to build to train others on how to do open science I mentioned the bulk of this is done through our catalyst network catalysts are researchers who teach an advocate for open science and we've we're very proud of their work they they work all over the all of the world and so far they've they've trained around 1500 thousand uh 1500 researchers including uh 950 in uh lower and middle income countries so this sort of stems from the mandate of our host organization the Center for Effective Global Action which which puts a lot of emphasis on on uh including scholars from from from these parts of the world then we also run our uh five-week long BitSmoog on the Future Learn platform and but however it's also available on our website right now and then finally we curate a growing repository with over 100 teaching materials templates software tools as part of our research library so if you if you're not sure where if you like to practice open science and you're not sure where to start I I highly encourage you to check this resource out and then if you yourself have some tools and practice and and and and templates and other resources that you would like to see there please reach out to me and so we can have them included in our library and then a very interesting uh development a very interesting piece of news from what is our recent recently published textbook which came out just last week so uh transparent and reproducible social science by Greg Christensen who's at the U.S. Census Bureau and formerly at BitS by Jeremy Fries at Stafford University and our faculty director Ted Miguel uh it's available to purchase for purchase uh from UC Press uh unfortunately we're not able to provide an an an open access copy however we are trying to facilitate and broaden access to as much as possible so uh if you yourself are teaching uh uh of course in in in in the social sciences and you you would like to consider integrating um uh transparent and reproducibility as as a module uh I encourage you to email me I provided my email address right there and uh ask for a free copy and then finally uh as I mentioned an increasing part of what we do is are collaborating with uh researchers with with uh journals donors and other research institutions to help them develop policies and protocols to recognize and foster uh transparency and reproducibility so this stems from the understanding that research is conducted as a part of a larger ecosystem in which funding collaboration and career incentives also play an important role so some of the are some of the projects that I would like to highlight are our collaboration with the Inter-American Development Bank uh from earlier this year where we worked with research managers at the institution to help them develop SOPs for uh to practically integrate transparent reproducibility tools and practices in all research supported by the bank another interesting project is our uh pilot of registry reports with the Journal of Development Economics which uh from what we know is was the first effort to introduce this format in in in economics it's been largely successful with over 45 submissions so far six in principal acceptances as of right now and the JD has also decided to maintain this this uh this track as a on a permanent basis um another project that we that we work uh in terms of institutionalizing transparency is our work on open policy analysis which is an approach to policy analysis where data code materials and uh accounts of methodological decisions are made freely available to in a way to uh help out with collaboration discussion and views um it's essentially an application of open science tools and principles in a completely new realm um and in this regard we work with research and with policy analysts to help them either develop uh open policy and OPA uh compliant versions of their policy reports or to incorporate tools and practices into their regular workflow so um as a as a research as a as a recent instance of of our work in this in this regard I would like to point out a recent case study that we did with uh Berkeley economist Emmanuel Saez and Gabriel Zucman uh where we reproduce a fully reproducible where we help them produce a fully reproducible version of their analysis uh which is the basis of uh some of their a little bit of progressive well-tax and part of their 2020 uh recent uh presidential campaign and finally um in an upcoming partnership we're we're gonna work with the AEA particularly with the AA data editor where we'll develop curriculum and deliver trainings to support their recently updated data and co-policy which uh mandates uh prepublication verification of computational reproducibility so uh in addition to uh curriculum and training uh we're also going to develop a platform to catalog all uh crowdsourced replications and results from all over economics so uh if yourself are are interested in this particularly if you're interested in replication in economics stay tuned for more updates and finally uh I encourage you to keep in touch here's some like my contact information uh and then make sure to subscribe to our monthly newsletter and then follow our blog also reach out if reach out if you have any ideas on on and you're interested in authoring a blog post. Excellent thank you Alex that was a fabulous overview and he really captured their multiple dimensions at which bits is operating and it seems that they have some connectivity among them their the research work the training work and the social initiatives uh that'll be great to unpack in in some Q&A time if we can I want to be mindful of time and we do have a list of questions that people have raised I'm going to suggest we go to 10 minutes past the hour and if people need to go they can go but I'll launch into some of these questions so that we can get through as many of them as we can and and I'll start with one Alex that came uh while you were speaking that I think is part of the the training and the early thing that you said Emily Farron asks how do you test best practices how do you know what it is that you want to get people to do and evaluate whether it's working yeah that's a great question uh well I guess part of our our training agenda is informed by uh meta research so we we like to say that we're in very much in tune with uh what we've learned from from from uh research evaluating these tools and practices uh and however we're also listening to our faculty for artitude faculty who they themselves are experienced researchers and and can speak a great deal about what has worked and hasn't from their own experience so in in a in a short that that could be the shortest answer I can give yeah yeah this is complex right because a lot of these we have good theory for why it is these are the right way to do things and the evidence base is still we need to actually do it in order to accumulate that evidence base that's great right uh so Jason Cullen Cherry has a question that I'll read in full and I think it probably uh Laura and Anita might be the best place to start with an answer and then everybody from the panel can address and that is what advice do you have for phd students who are interested in following open science practices and implement measures to prevent reproducibility issues but do not have the necessary support and sometimes active resistance from pis and others how is it that these communities might be able to help with some of that should I go first um say uh what what we've observed here in Oxford is that there are communities of uh early career researchers so other phd students maybe not in your department maybe in a different research group in the same department or in a different department and those can be very effective groups for phd students to adopt and learn from sort of horizontally from other phd students so this may be especially useful for people who might feel isolated within the research group because nobody else is interested or maybe the PI is not particularly supportive um so seek out people uh in the in the department or in the broader institution who can help you learn some of these techniques also um of course if there are networks like uh we have here in Oxford or the or the open science uh groups that Anita has talked about in the Netherlands of course those are great resources it goes without saying um uh but then look up also for example the carpentries they have free workshops that you may be able to attend um uh and that may provide an initial uh um uh road into some of these techniques yeah I was also going to say try to seek support um maybe not within your institution but beyond um there are lots of phd students that probably struggle or struggled with the same issues and I think it's very important that you see that you're not the only one and that you can learn from the experiences from others so others that had PI's that were not supportive of open science for example um yeah so I think yeah that's important especially if you don't feel that their support within your institution Alex did you have something you want to say there too yeah I just want to collaborate what what Anita and Laura said uh yeah make sure to join uh a community of practice and uh sometimes what we've learned from from our interaction with catalyst just having some sort of a a network or institutional support can go a long way so um even even just just like a a simple letter of support that we for example have given to our catalysts have helped them a great deal and uh yeah just try to try to join a network uh and seek for for whatever support that you can get that's great thanks that actually leads to another question that Crystal Stelenpole asked and maybe uh Marcus might start uh on responding to this she says I'm curious if the panelists have thoughts about how people at low-resourced institutions or areas might get involved with open science initiatives the examples that we have today are from places that have a lot of resources and in fact that's a sounds like it's been a key part for some of these uh coming to coming together well what is it that people can do to leverage some of these networks or to start them themselves without that same kind of resources uh Marcus or others well the first thing I'd say is that for many of the things that we're talking about you don't necessarily need um a huge amount of resource so for example journal clubs uh principally require time more than anything else um but then I think another level to that answer is that these networks are there to provide that sense of community for those who otherwise might feel relatively isolated so the fact that there is a um a group of individuals across a country or a region or a discipline that um are engaging with the same issues provides you with that wider support structure uh one of the things that we've done through ukrn for example is to set up a slack channel for our local network leads so that they have a means of communicating regularly with each other we also send out starter packs for those who want to set up reproducibility journal clubs um but then another reason for having that connection between the grassroots and the institutions and the other stakeholders at a higher level is that it um identifies those needs that individual researchers or groups of researchers have that we can then use to advocate for uh provision of that resource either by institutions or by um stakeholders at a higher level so for example jist in the uk is a digital services provider and they're interested in things like how they can provide the infrastructure to enable things like data sharing more readily across a wider range of institutions so i think to some extent um you do whatever you can do locally but you try to make yourself part of this um wider network and and there are so many of them now as we've been talking about that there's almost certainly going to be one that you can connect with that then provides you with that next level of support great any others want to comment on that as well okay great thank you excellent uh so uh uh perhaps related to the sort of the joint nature of the two themes that just the last two questions sau chen chen asks some open practices are unwelcome in countries outside of north america new although that may be overestimating how welcome they are within those domains too uh for example public preprints open peer review so asking what advice would you offer to change minds in societies or locations where there's still a lot of resistance or a lot of lack of information and anita you had raised some of this in your characterization even within the institutions that you're engaging of some people are actively opposed or not engaged and others are more interested how do you think about this on a cultural scale well that's a tough question um i think when it comes to preprints for example um i don't see why you could not upload your work as a preprint in addition to do whatever is uh expected from you um i do think that it's hard if if the culture is such that open science practices are not supported at all um and to be honest i'm not really sure how you can actually change that i think that even in uh north america and in europe a cultural change is needed so that we are going to reward open science practices and that's something that is very hard to influence so i think i would try to do as much as you can and want to do uh even though it's not supported without doing the stuff that gets rewarded and is supported if that makes sense no do others that's great do others have comments on onboarding the reluctant or the skeptical uh and experiences that happen locally uh or more broadly uh sometimes i think the the barrier to entry is simply uh not knowing so i would encourage encourage whoever is in such a position to sort of share some some resources and and try to take the lead uh when when they they work with for example like senior faculty who are reluctant or resistant to such practices so luckily there are there is a wealth of resources pretty much for any discipline for for any use and uh i would encourage them to sort of take advantage of that and uh help their colleagues help their peers overcome this this barrier of entry great thank you alex i'm gonna have one more uh general question from amilia zine uh that each of you can answer uh and then i'll just do a couple quick closing comments about next steps for this uh and she asks what is the biggest obstacle that you've had to deal with uh in managing your initiative uh and what are some things that are either you've done to overcome them or still working on uh to try to address that what makes this work and what are the barriers maybe marcus we can start with you and everybody can provide an answer so the biggest challenge personally and i think um laura alluded to this as well is just bandwidth you know i have to keep my own day job ticking over in the background doing my primary research and then um keeping all of these plates spinning one of the things that the funding that we've been given has allowed us to do is to appoint an administrator that just takes a little bit of that pressure off people like myself and laura and um and chris and so on um but then i think that also needs to happen at the local level so one of the reasons why we're trying to get institutions on board is that they will then start to fund this activity locally provide the infrastructure to allow those local networks to grow to allow the senior roles to uh become embedded and so on um as with many things the biggest obstacle is just time anita yeah for us uh it was funding so money uh that also comes down to time uh everything that we did in the first year we did in our own time without any funding um and we have overcome this obstacle i think almost a year in our initiative um i think the university saw that we were actually doing something that we were actually able to reach out to researchers to get their feedback and to inform them about open science practices and that we could also help them to give feedback from the researchers on their policies and let them know what support was needed so i think in the beginning they didn't want to support us which makes sense because well we had to prove ourselves um but after we did um they were actually very happy in helping us getting getting funding because they saw that uh what we did was important and that they actually needed us as well excellent thank you laura i think time and and funding and of course the two intersect to the extent that you are applying for funding to make time for the activities um so obviously we're very um lucky here in the sense that as i said there are different parts of funding um that are available um so i think that's been i wouldn't necessarily an obstacle but as something that has um possibly slowed us down in the sense of a lot of energy and a lot of good will and we want to do a lot of things uh but we have to be realistic and manage what what we take on um so i think that's i wouldn't i wouldn't say that's an obstacle but that's just the reality of of doing this on a volunteer basis um uh and and as marcus alluded you sort of have to pace yourself to make sure you don't don't burn and crash uh early on yeah great and alex um i guess also funding what's new uh i i realized that what we do is is very niche and there's not a lot of funding opportunities uh specifically intended for open sizes especially with visibility and we've overcome this by being very entrepreneurial very creative uh very flexible in terms of uh the things that we work on and and the partners that we seek out but then uh i mean i'll a huge part of our or our time in the in the last year particularly has been dedicated to just like fundraising and being part of a of a larger center such as the center for fickle global action which is a lot of uh program staff on board we've had the luxury to do this and we've received a lot of support so um yeah yeah so thank yeah thank you everybody for those uh responses uh and what's clear from from the range of them is that grassroots communities can have a lot of energy a lot of motivation get a lot done but the that challenge of sustainability how do we make this something that is an institutionalized uh and pervasive and ongoing solution is a real barrier uh and is going to continue to be we're not going to escape that easily or quickly among us uh so given the time let me just close this session first by just thanking our panelists for making time and sharing the amazing work uh that their communities have done and hope that that provides some good context for the listeners about things that they might be able to do and draw from uh likewise the participants question there are more questions in the q and a uh you can and i see people are already answering each other uh that is great uh please continue to do that and i just want to close with a couple of next steps so that this isn't just a conversation uh that ends and goes nowhere else uh and so the the things to point out are our first that this conversation among a few networks is the start of a potentially much better set of communication coordination a lot of what came up in everybody's comments uh about their networks locally and regionally was about the role of convening of communication of having information exchanges so that people could be more effective more efficient and use what little resources are available as far as they can go uh and so there is a uh google spreadsheet that people have been crowdsourcing just populating with the networks they know about that networks list is there at the short url bf cz4 uh you can go there and find networks that might be relevant to you or add one that isn't uh there so that there can be more information about who is out there that can be communicating and coordinating there's also now a google group uh for discussion among network of networks uh and this short the next short url that's r b g i seven not one seven uh and you can join that and see what other networks are doing and have a place to be able to communicate about common challenges or solutions and the last thing that i want to mention is that we have the preliminary work done on an nsf proposal for a mechanism that nsf supports networks of networks to try to help facilitate exactly the challenges that we are discussing here as a group there are some challenges of nsf's funding in terms of where it can be spent particularly it is us centric but they are trying to enable global networks so we're going to try to push the boundaries of that as far as that mechanism will go uh we will be reaching out to everybody that's on that google discussion group and networks list to potentially be involved in that if you aren't on either of those but you have a network please get on that or email me to say you are a network and you want to be part of that and we will be sending out information uh shortly about that uh and then lastly if you got something out of this as a discussion uh one of the things that we are trying to help do is continue to facilitate these kinds of open discussions for the global community and so the address there cos.io webinars is where the upcoming webinars are scheduled and the next one is listed there advocating for change and how science is conducted uh to level the playing field and this will be particularly focused on some of the themes that came up today about incentives and culture change and how do we shift those across the various stakeholder groups in the research community so with that we will close this webinar and thank everyone for their questions their participation and for all of the work that they're doing to try to advance the progress in open science in all of its dimensions so thank you very much